1

SCCC Comprehensive Assessment Plan for Institutional Effectiveness (CAPIE)

Workshop

Spring, 2007

Dr. Tina GoodDr. Allen JacobsDr. Louis Attinasi

2

CAPIE Workshop Agenda

PowerPoint Presentation– Assessment as a Process, Tina Good– Goals and Objectives, Allen Jacobs– Assessment Tools, Lou Attinasi

Hands-on practice developing goals and objectives (with worksheets)

Discussion about developing unit assessment plans and timelines (with forms and templates)

3

Follow-up CAPIE Meetings and Workshops

Meetings with units to assist with development of assessment plans and timelines prior to implementation

Regular contact with units during implementation of plans

Workshops on interpreting and reporting assessment results

Workshops focusing on connecting unit goals and objectives to new strategic plan

4

College Assessment Resources

Office of Institutional Effectiveness Office of Curriculum and Assessment Assessment Advisory Council Strategic Planning Council CAPIE Website

5

Assessment

a process

6

What does Assessment as a Process mean?

Process– The fact of going on or being carried on, as an

action, or a series of actions or events; progress, course.

– Oxford English Dictionary

Process requires– Action– Reflection– Revision

7

What does Assessment as a Process mean?

Assessment– the ongoing process of establishing clear,

measurable expected goals and objectives; ensuring that there are sufficient opportunities for affected areas to achieve those outcomes; and systematically gathering, analyzing, and interpreting evidence to determine the degree to which outcomes match expectations

– CAPIE

8

What does Assessment as a Process mean?

Assessment ≠ Test Process ≠ Product

9

Assessment

The Anxiety

10

Why does the word “assess” cause anxiety?

to assess:– To estimate the value of (property) for taxation.– To charge (a person or property) with a special

payment, such as a tax or fine.– To charge (a foul or penalty) against a player,

coach, or team. 5. – To determine the value, significance, or extent of;

appraise. – The American Heritage Dictionary

11

Why does the word “assess” cause anxiety?

Accountability and Improvement– justifying the investment of resources by

providing substantive evidence and analysis which demonstrates that the investment yields significant results

– CAPIE

– To do something for the purpose of improving suggests automatically a fundamental bias in the process—the assumption that improvement is needed.

12

Assessment as Opportunity

1.A forum for articulating to ourselves and to our constituent groups what we value.

2.A forum for collaborating with constituent groups such as students, faculty, taxpayers, SUNY, accrediting agencies, etc., on what should be valued.

13

Assessment as Opportunity

3. A systematic method for using assessment results

To recognize and celebrate what we are doing well To inform planning and resource allocation decisions To improve programs and services

14

Assessment Plan for Institutional Effectiveness

How well are we doing what we say we are doing?

The effectiveness of an institution rests upon the contribution that each of the institution’s programs and services makes toward achieving the goals of the institution as a whole.

– Middle States Commission on Higher Education

15

Four-Step Planning-Assessment Cycle

1. Defining clearly articulated institutional and unit-level goals;

2. Implementing strategies to achieve those goals

3. Assessing achievement of those goals; and

4. Using the results of those assessments to improve programs and services and inform planning and resource allocation decisions.

– Middle States Commission on Higher Education

16

Evaluation

1.The part of the assessment process which uses professional judgment to form conclusions about the data.

2.Using assessment information in combination with professional judgment to make appropriate decisions about what has been assessed

– CAPIE

17

How do you measure the abstract?

Develop goals first then figure out how to measure to what degree you are achieving them.

Do not confuse assessment measures, tools, indicators (retention, graduation, transfer, and placement rates, financial ratios, surveys, cost analyses, etc.) with assessment processes (gathering quantitative and qualitative evidence to determine the degree to which you are achieving your goals).

18

The Myth and The Metaphor

Love is like a rose.

19

Multiple Measures

Because there is no one perfectly accurate assessment tool or strategy, institutions should use multiple measures to assess achievement.

– Quantitative/Qualitative– Direct/Indirect– Benchmarks

All assessment tools and strategies should clearly relate to the goals they are assessing

– Middle States Commission on Higher Education

20

Characteristics of Assessment Processes

Useful Cost-effective Reasonably accurate and truthful Planned Organized, systematized, and sustained

– Middle States Commission on Higher Education

21

How do we build a Culture of Assessment

Keep it simple Do not assess everything Do not reduce assessment process to measuring

only that which is most easily measured Build on what we already have Develop reasonable plans and timelines Understand there are multiple audiences when

developing reports Link assessment results with planning Keep process open and transparent

22

College of Excellence

Assessment + Planning = Institutional Effectiveness

23

Goals and Objectives…

where assessments begin

24

TERMINOLOGY….

Outcomes Mission Goals Objectives Standards Aims Behavioral Objectives Cognitive Objectives Affective Objectives

Learning Goals Learning Objectives Behavioral Outcomes Desired Outcomes Intended Outcomes Achieved Outcomes Process Standards Content Standards Value standards

25

Terminology in Turmoil

26

Translucent Terminology

Lack of precision Confusion with non-technical uses Overlap in meaning Inconsistent application

27

As a solution…

Keep it simple.

Keep it consistent.

28

The Overall Concept – statements of purpose

MISSION

OBJECTIVES

GOALS

Broad

Narrow

Not Measurable

Measurable

CONT INUUM

29

Action Plan

Steps taken to achieve an objective.

30

MISSION

The broadest statement of the purposes of an institution.

The starting point for programmatic development and assessment.

More than just decoration. The ability of an institution to achieve its mission

= institutional effectiveness.

31

Institutional Mission Statement

Suffolk County Community College promotes intellectual discovery, physical development, social and ethical awareness, and economic opportunities for all through an education that

transforms lives, builds communities, and improves society.

32

Programmatic Mission Statement

The broadest statement of the purposes of an institutional program or unit.

Can be more restricted than the institutional mission statement.

Must derive from or reflect some aspect or aspects of the institutional statement.

33

GOALS

Add some specificity to the aims of the mission.

They retain breadth. They are not typically couched in language

that lends to assessment, i.e., they are not directly measurable.

34

OBJECTIVES

Like mission and goals, objectives are statements of aim, purpose, or outcomes; however, they are narrow in scope and specific.

Objectives link to goals. They are written with language that allows for

measurement. Achieving objectives implies the achievement of

goals.

35

The Metaphor

36

Writing goals & objectives…

...it’s all about the choice of words.

37

Writing goals…

Abstract Can be lofty, florid, emotional, etc. Broad, global, general, etc. Typically, not measurable

38

Writing objectives…

Concrete Can be mundane, unadorned,

dispassionate, etc. Narrow, specific, explicit, etc. Always measurable Time bound

39

The Language of Goals

Appreciation Understanding Knowledge Comprehension Critical thinking Ethical awareness Improving society Communication skills

Competency Happiness Excellence Life-long learning Community service Professionalism Good citizenship Physical development

40

DEVELOPING GOALS: Some Things to Consider…

What do we value? (Goals are important.) Is this a cumulative outcome? (Goals

generally require that several objectives be met.)

Is there a defined endpoint? (Goals are generally ongoing.)

Is it long-term? (Achieving goals usually takes time.)

41

The Language of Learning Objectives

Name Define Classify Explain Use Illustrate Write Demonstrate

Compare Experiment Calculate Create Argue Predict Select Estimate

42

More Language (for non-learning objectives)

Increase Decrease Collect Implement Develop Promote

Distribute Gather Establish Inform Create Communicate

43

The Components of an Objective

Audience: Who?

Behavior: What?

Condition: How?

Degree: How much?

End point: How long?

44

Audience: Who?

State specifically who will be performing…

All StudentsESL Students

All FacultyBusiness Faculty

All GraduatesNursing Graduates

45

Behavior: What?

State specifically what it is you expect the audience to do.

Employ terms to identify behaviors that you can see, hear, touch, smell, or taste, i.e., use measurable verbs.

46

Condition: How?

State the circumstances under which the behavior will occur.

State what can or cannot be used in the performance.

A statement of the condition is often omitted if the condition is obvious.

47

Degree: How much or how well?

A statement of acceptable performance, e.g., In how much time? or To what degree of accuracy? or To what level?

This is sometimes omitted if it is obvious.

48

End Point: How long?

Objectives reference a timeframe in which they are to be met.

Timeframes need to be reasonable and realistic.

49

Kinds of Objectives

Learning Objectives

Programmatic/Unit Objectives

50

Types of Learning Objectives

Cognitive: intellectual learning and problem solving, what the student is able to do (Student will be able to…)

Affective: feelings or emotions, how the student chooses to act (Student will choose to…)

Psychomotor: physical skills, what the student can perform (Student will be able to…)

51

Types of Programmatic/Unit Objectives

Operational – support institutional goals

Strategic – support strategic goals

52

Examples

Goal:_______________________________________________________________________________________________________________________________________________________________________________________________________________________

Objective 1: ( _Cognitive _ Affective _ Psychomotor) ( _ Operational _ Strategic)

Audience (Who?) _____________________________Behavior (What?) _____________________________Condition (How?) _____________________________Degree (How much or how well?) _____________________________End point (How long?) _____________________________

Written Objective:_____________________________________________________________

________________________________________________________________________________________________________________________________________________________________________________________________________________________

53

Learning Objective

Goal :__Students will have a basic understanding of Spanish grammar.______________ ________________________________________________________________________________________________________________________________________________

Objective 1: ( X Cognitive _ Affective _ Psychomotor) ( _ Operational _ Strategic)

Audience (Who?) _Students successfully completing SP11Behavior (What?) _Conjugate irregular ar, er, & ir verbs___Condition (How?) _From a list of 50 infinitives___________Degree (How much or how well?) _With 80% accuracy_________________End point (How long?) _One semester______________________

Written Objective: __Given a list of 50 irregular ar, er, and ir verbs, students who______ successfully complete SP11 will be able to write the conjugations of these verbs_ with 80%_accuracy.___________________________________________________________________________________________________________________________

54

Learning Objective

Goal : _Nursing Program graduates will be proficient at performing venipunctures.____

Objective 1: ( _Cognitive _ Affective X Psychomotor) ( _ Operational _ Strategic)

Audience (Who?) _Graduates of the Nursing Program_______Behavior (What?) _Inject a solution into a vein_____________Condition (How?) _With a 20 gauge, 1 ½” hypodermic needle

_and 3 cc syringe______________________Degree (How much or how well?) _On first try, 9 out of 10 times____________End point (How long?) _Upon completion of Nursing Program_____

Written Objective: __Given a 20 gauge, 1 ½” hypodermic needle and a 3 cc syringe,___ Nursing Program graduates will be able to administer an intravenous injection on their first attempt 9 out of 10_times._______________________________________

55

Actual Example 1

Outcome Behaviors:Upon completion of theatre

workshop, students will have learned:

1) techniques of script analysis in preparation for a production

2) the relationship of your specific discipline to all other disciplines in a given production

3) to develop standards and practices for rehearsals and performances in all disciplines.

4) to sustain the rehearsed production through the performance schedule.

Outcome Behaviors:Upon completion of theatre

workshop, students will be able to:

1) Analyze scripts in preparation for a production

2) discuss the relationship of your specific discipline to all other disciplines in a given production

3) develop standards and practices for rehearsals and performances in all disciplines.

4) sustain the rehearsed production through the performance schedule.

56

Actual Example 2

COURSE OBJECTIVESBy the end of the semester, the

students will be able to: Know their minimum and maximum

target heart rate (THR) Know when they are working within

their THR and be able to do so for 20-30 minutes

Participate in aerobic combinations helping to improve their cardiovascular system

Have a better understanding of weight control, nutrition and fat reduction

Select a safe and appropriate aerobic program

COURSE OBJECTIVESBy the end of the semester, the

students will be able to: Cite their minimum and maximum

target heart rate (THR) Identify when they are working

within their THR and be able to do so for 20-30 minutes

Participate in aerobic combinations helping to improve their cardiovascular system

Formulate a nutritionally balanced diet aimed at weight control and fat reduction.

Select a safe and appropriate aerobic program

57

Actual Example 3

COURSE OBJECTIVESEach student, upon completing

the course, should have an understanding of the following essential skills pertaining to:

I. The topics of techniques of playing the guitar

II. The topics of music theory for guitar

III. Sight-reading music for guitar

IV.Guitar Harmony

COURSE OBJECTIVESUpon completing the course,

students will be able to:

I. Play xyz on the guitar (to what degree of accuracy?).

II. Discuss topics (specify) of music theory for guitar.

III. Sight-read music for guitar (level of difficulty?).

IV. Discuss guitar harmony.

58

Programmatic Objective

Goal:_ Support the food-service industry.______________________________________________________________________________________________________________________________________________

Objective 1: ( _Cognitive _ Affective _ Psychomotor) ( X Operational _ Strategic)

Audience (Who?) The Culinary Program__________________Behavior (What?) Increase the number of program graduatesCondition (How?) _____________________________________Degree (How much or how well?) By 10 students a year _________________End point (How long?) For the next 5 years____________________

Written Objective: __The Culinary Arts Program will increase the number of its graduates by 10 students a year for the next five years.____________________________________________________________

Action Plans: __________________________________________________________________________________________________________________________________________________________________________

59

Facilities Objective (a poor example)

Goal: _Provide an environment conducive to learning.______________________________________________________________________________________________________

Objective 1: ( _Cognitive _ Affective _ Psychomotor) ( X Operational _ Strategic)

Audience (Who?) College HVAC/R personnel___________Behavior (What?) _Install air conditioners______________Condition (How?) _While classes are not in session______Degree (How much or how well?) _In all non-air conditioned classrooms_End point (How long?) _By mid-August of this year__________

Written Objective: _By August 15 of this year, at times when classes are not in session,_ college HVAC/R personnel will install air conditioners in all classrooms that are not currently air conditioned.__________________________________________________

60

Facilities Objective (a better example)

Goal: _Provide a comfortable environment for learning and working.___________________________________________________________________________________________________________________

Objective 1: ( _Cognitive _ Affective _ Psychomotor)( X Operational _Strategic)

Audience (Who?) _All students and personnel______________________Behavior (What?) _Experience temperatures of 68-72 degrees_________Condition (How?) _____________________________________________Degree (How much or how well?) _Whenever they occupy indoor spaces at the College End point (How long?) _By August 15, 2007___________________________

Written Objective: _All students and personnel will experience temperatures of 68-72 degrees, whenever they occupy indoor spaces at the College, by August 15, 2007.______________________________________

Action Plans: a. Install air conditioners in all non-air-conditioned rooms by August 15, 2007.________________b. Repair broken thermostats by August 15, 2007_______________________________________

61

Human Resources Objective

Goal: Promote employee effectiveness.__________________________________________

________________________________________________________________________________________________________________________________________________

Objective 1: ( _Cognitive _ Affective _ Psychomotor) ( X Operational _ Strategic)

Audience (Who?) _Personnel division of HR________Behavior (What?) _Update all job descriptions_______Condition (How?) _In consultation with administratorsDegree (How much or how well?) _To reflect current job expectationsEnd point (How long?) _By end of next year_____________

Written Objective: __In consultation with appropriate administrators and supervisors,_ the personnel division of Human Resources will update the job descriptions of all employees to reflect their current job expectations by the end of next year._______

62

Financial Affairs Objective

Goal: __Promote data-driven decision making by administrators._________________________________

__________________________________________________________________________________________________________________________________________________________________________

Objective 1: ( __Cognitive __ Affective ___Psychomotor)( _ Operational _X_ Strategic)

Audience (Who?) The Office of Financial Affairs______________Behavior (What?) Develop formulas for non-tangible costs_____Condition (How?) By analyzing expenditures over last 5 yrs.___Degree (How much or how well?) Including costs for energy, security, time, etc.End point (How long?) By August 31 of this year__________________

Written Objective: _By August 31 of this year, the Office of Financial Affairs will develop formulas for computing non-tangible costs, including costs for energy, security, custodial care, and_______ administrative time, by examining expenditures made in these areas over the last 5 years._____

63

Assessment Tools

an overview

64

Assessment Plan

What is it? • Agreement about the measures and processes to be

used• Indicates a readiness to act on the basis of a shared

understanding

65

Assessment Plan

Elements • Subject Matter for Assessment• Methodology• Time Line• Use of Assessment Information• Provisions for Administering the Plan• Provisions for Assessing the Assessment

66

Assessment Plan

• The plan will specify an assessment schedule

• Note: No institution/unit can be expected to assess everything at the same time

67

Assessment Plan

Decisions must be made about: • How plan will be formatted• How plan will be evaluated• How much it will cost to implement it• What should be the starting point

68

Assessment Plan

Key Point: • There is a clear order to the assessment process:

formulation of the intended educational outcomes or administrative objectives must precede the identification of the means of assessment.

69

Current Assessment Efforts

General Education Assessment Academic Program Reviews Student Services Program Reviews

70

Academic Assessment

General Education – Achievement of Learning Outcomes

Direct Measures – provide direct evidence of outcome achievement

• Standardized Tests• Local (Criterion-referenced) Tests• Qualitative scoring rubrics applied to written work, oral presentations,

etc.• Imbedded measures• Portfolios• National Licensure and Certification Exams

71

Academic Assessment

General Education – Achievement of Learning Outcomes

Indirect Measures – provide indirect evidence of outcome achievement

• Questionnaires• Interviews• Focus groups

72

Academic Assessment

Academic Program Review

Involves assessment of:• Program goals and objectives (currency, mission-relatedness)• Curriculum & Instruction (currency, effectiveness)• Students (learning; enrollment, graduation & transfer rates; satisfaction)• Resources (library, technology, instructional & office space)• Staffing (level of full-time faculty, professional development of faculty)

73

Academic Assessment

Academic Program Review

Involves the following assessment tools:• Benchmarking with peer institutions (curriculum, staffing)• Standardized tests, local rubrics (learning)• Surveys (curriculum, students)• Institutional data (students, resources, staffing)

74

Assessment of Student Affairs Units

Student Affairs Program Reviews

Have involved the following assessment tools (indirect measures):

• Commercial Self-study Questionnaires• Surveys of Students & Faculty

75

Assessment of Student Affairs Units

Student Affairs Program Reviews

Could involve direct measures, e.g., direct evidence of student development as a result of student affairs programming

• Standardized tests• Local tests• Other demonstrations of skill or knowledge attainment (e.g.,

portfolios)

76

Need to Assess Non-Academic Areas

We cannot know how well we are doing—or how we can do better—in non-academic areas of the College unless we regularly and systematically assess our activities in these areas

Our institutional accrediting body—Middle States Commission on Higher Education—requires assessment of all areas of SCCC

77

Assessment Tools for Non-Academic Units

Direct Measure of Effectiveness Objective: By Spring 2008, the Business Office will have

reduced by an average of five working days the time it takes to process a travel reimbursement request form.

Method: Use of institutional transactional database to determine the average time it took to process a travel reimbursement request form in Spring 2007 with the length of time it took in Spring 2008.

Measure: Average length of time it takes to process a travel voucher

78

Assessment Tools for Non-Academic Units

Attitudinal Measure of Effectiveness Objective: Ninety percent of the Business Office’s

customers during AY 2007-2008 will indicate satisfaction with the overall service they were provided by the office.

Method: Survey of customers Measure: Percentage of survey respondents who

indicated they were “satisfied” or “very satisfied” when asked on the survey how satisfied they were with the overall service they received from the Business Office

79

Assessment Tools for Non-Academic Units

Effectiveness based on Meeting Professional Standards

Objective: By Spring 2008, the Business Office will be in 100% compliance with all standards set by the National Association of College and University Business Officers for business office processes

Method: Gap analysis involving a comparison of Business Offices processes with NACUBO standards.

Measure: Extent of discrepancy between NACUBO standards for processes and status of Business Office processes

80

Assessment Tools for Non-Academic Units

Effectiveness based on Best Practices (Benchmarking)

Objective: By Spring 2008, the effectiveness of the Business Office’s accounting practices will place it among the top 25% of community colleges in NACUBO

Method: Benchmarking against institutions that participate in the annual NACUBO survey of member institutions

Measure: Business Office’s placement among the community colleges responding to NACUBO survey

81

Assessment Tools for Non-Academic Units

Other Units (e.g., Business Affairs, Facilities, Human Resources, Institutional Advancement, Institutional Effectiveness, Workforce Development)

Typical measures• Institutional or Unit Data that demonstrate improved efficiency or effectiveness• User-satisfaction and other attitudinal surveys• Benchmarking against standards set by professional groups (e.g., NACUBO)• Benchmarking against units in comparable institutions (e.g., NCCBP, NACUBO, Educause, ,AACC Digital

CC Survey)• Performance Indicators

82

Validity and Reliability

Two Important Characteristics of Good Assessment Measures

• Validity: the extent to which assessment procedures and instruments accurately measure what they are designed to measure

• Reliability: the extent to which assessment procedures and instruments consistently measure what they are designed to measure

83

Validity and Reliability

Example:

Unit X wants to determine its customers’ satisfaction over the last 6 months?

Which survey item is a valid measure?

Item 1: “Overall, how satisfied are you with the service Unit X has provided you over the last 6 months? Very dissatisfied → Very satisfied

Item 2: “How often have you used services provided by Unit X over the last 6 months? Very infrequently → Very frequently

84

Middle States Assessment Model

Middle States Model for Comprehensive Assessment

85

Questions?