assessment 101: the core curriculum susan hatfield - winona state unc wilmington november 2005...

Assessment 101:The Core Curriculum

Susan Hatfield - Winona StateUNC WilmingtonNovember 2005

[email protected]

Common Reactions to Assessment Initiatives

• Ignoring it• Bribing someone else to do it• Complaining about it • Losing sleep over it• Sitting down and writing it

Assessment is -- first and foremost --

about student learning.

A Little Quiz

Quiz

All faculty need to be actively engaged in assessment for a department or program to really be “doing assessment.”

Quiz

All faculty need to be deeply committed to assessment for a department or program to really be “doing assessment.”

Attitudes toward Assessment

Level of Commitment

Hostile Accepting Enthusiastic

15% 15%

70%

Quiz

The best way to build a campus culture of assessment is for top administration to be prescriptive in student learning outcomes, assessment measures and methods.

Quiz

All departments and programs should be moving ahead on assessment at the same pace, meeting specific goals within specified time frames.

Quiz

Effective assessment programs have eliminated indirect measures and measures of departmental effectiveness (satisfaction, etc.).

Quiz

Assessment is a time-intensive add-on that will be a huge burden to faculty who are already overburdened.

Quiz

Effective programmatic assessment plans have every faculty member assess every outcome in every course every semester.

Quiz

Accrediting organizations expect to see fully realized assessment plans during site visits.

Quiz

The most effective assessment programs are ones in which the students are not aware they are being assessed.

Uncertainty

• Uncertainty related to questions of • How to do it• Why it needs to be done• What to do with the data• How the data will be used• How to find the time to implement it• What support is available

Reducing Uncertainty

• Clear• Understanding • Definitions• Processes• Rules• Resources• Understanding of pitfalls

The Assessment Core Curriculum:

10 things you should know about assessment

Core Curriculum

1. Assessment terminology2. Evolution of assessment initiatives3. Difference between direct and

indirect measures of learning4. Assessment requires exertion and

intention5. Writing student learning outcomes

Language of Assessment• A. General skill or knowledge category GOAL

• B. Specific accomplishments to be achieved OUTCOME

• C. Activities and Assignments to help students learn LEARNING EVENTS

• D. The key elements related to the accomplishment of the outcome COMPONENTS

Language of Assessment

• E. The objects of analysis OBJECTS

• F. Data indicating degree of achievement CHARACTERISTICS

• G. Combination of data indicating relative degree of achievement of the learning outcome INDICATORS

Goals

Organizing PrincipleCategory or Topic AreaSubjects

Goals

CompositionPEHumanitiesFine ArtsNatural Sciences and Mathematical

SciencesSocial and Behavioral SciencesInterdisciplinary Perspectives

Learning Outcomes

Student Learning Outcomes

Communication

Relating Speaking Listening TeamingWriting

Learning Events

Learning Events

• Assignments (in class and out of class)• Feedback on practice• Self evaluation• Peer evaluation• Role Play• Pre Tests• Simulation

Learning Objects


Communication


Sales

Components


Goal

OutcomeOutcome Outcome OutcomeOutcome

Object

Learning events


Communication


DemonstrationVerbal

Nonverbal

Organization

Evaluative elements

Performance Characteristics


Communication


DemonstrationVerbal

Nonverbal

Organization

Indicators


GOAL

component

component

component

Outcome

Object

component

Outcome Outcome OutcomeOutcome

Degree to which outcomeis achieved

indicator

Core Curriculum




Maturing Assessment

INSTITUTIONAL RESPONSIBILITY

DEPARTMENTRESPONSIBILITY

BEGINNING PROGRESS MATURING

Maturing Assessment

DIRECTMEASURES

INDIRECTMEASURES


Maturing Assessment

PROCESS MEASURES

OUTCOMEMEASURES


Maturing Assessment


PROGRAMASSESSMENT

CLASSROOMASSESSMENT

Maturing Assessment

INSTITUTIONALEFFECTIVENESS

STUDENTLEARNING


Evolutionary Trajectories

BEGINNING

MAKINGPROGRESS

MATURING

04 0599 00 01 02 0398

Core Curriculum




Direct Measures of Learning

• Capstone experience• Standardized tests• Performance on national licensure certification

or professional exams • Locally developed tests • Essay questions blind scored by faculty• Juried review of senior projects• Externally reviewed exhibitions performances• Evaluation of internships based upon program

learning outcomes

Indirect Measures of Learning

• Alumni, employer, and student surveys (including satisfaction surveys)

• Exit interviews of graduates and focus groups graduate follow up studies

• Retention and transfer studies• Length of time to degree• ACT scores • Graduation and transfer rates• Job placement rates

Non-Measures of Student Learning

• Curriculum review reports• Program review reports from external

evaluators• Faculty publications and recognition • Course enrollments and course profiles • Faculty / student ratios, percentage of students

who study abroad • Enrollment trends• 5 year graduation rates• Diversity of the student body

Core Curriculum




Exertion without Intention

Intention without Exertion

Intention and Exertion

Core Curriculum




Learning Outcome Format

– Students should be able to <<action verb>>

<<something>>

KNOWLEDGECOMPREHENSION

APPLICATIONANALYSISSYNTHESISEVALUATION

CiteCountDefineDraw

IdentifyList

NamePointQuoteRead

ReciteRecordRepeatSelectState

TabulateTell

TraceUnderline

AssociateClassifyCompareComputeContrast

DifferentiateDiscuss

DistinguishEstimateExplainExpress

ExtrapolateInterpolate

LocatePredictReportRestateReview

TellTranslate

ApplyCalculateClassify

DemonstrateDetermineDramatize

EmployExamineIllustrateInterpretLocate

OperateOrder

PracticeReport

RestructureScheduleSketchSolve

TranslateUse

Write

AnalyzeAppraiseCalculate

CategorizeClassifyCompareDebate

DiagramDifferentiateDistinguish

ExamineExperiment

IdentifyInspect

InventoryQuestionSeparate

SummarizeTest

ArrangeAssemble

CollectComposeConstruct

CreateDesign

FormulateIntegrateManageOrganize

PlanPrepare

PrescribeProduceProposeSpecify

SynthesizeWrite

AppraiseAssessChoose

CompareCriticize

DetermineEstimateEvaluate

GradeJudge

MeasureRankRate

RecommendReviseScoreSelect

StandardizeTest

Validate

Lower division courseoutcomes

KNOWLEDGECOMPREHENSION

APPLICATIONANALYSISSYNTHESISEVALUATION

CiteCountDefineDraw

IdentifyList

NamePointQuoteRead

ReciteRecordRepeatSelectState

TabulateTell

TraceUnderline

AssociateClassifyCompareComputeContrast

DifferentiateDiscuss

DistinguishEstimateExplainExpress

ExtrapolateInterpolate

LocatePredictReportRestateReview

TellTranslate

ApplyCalculateClassify

DemonstrateDetermineDramatize

EmployExamineIllustrateInterpretLocate

OperateOrder

PracticeReport

RestructureScheduleSketchSolve

TranslateUse

Write

AnalyzeAppraiseCalculate

CategorizeClassifyCompareDebate

DiagramDifferentiateDistinguish

ExamineExperiment

IdentifyInspect

InventoryQuestionSeparate

SummarizeTest

ArrangeAssemble

CollectComposeConstruct

CreateDesign

FormulateIntegrateManageOrganize

PlanPrepare

PrescribeProduceProposeSpecify

SynthesizeWrite

AppraiseAssessChoose

CompareCriticize

DetermineEstimateEvaluate

GradeJudge

MeasureRankRate

RecommendReviseScoreSelect

StandardizeTest

Validate

Upper divisionCourse / Program

outcomes

Learning Outcome Rules

– Only one action verb

– Identify single accomplishments

– Focus on students, not faculty or curriculum

Example #1Gather factual information and apply

it to a given problem in a manner that is relevant, clear, comprehensive, and conscious of possible bias in the information selected

of Bias

Example #2Imagine and seek out a variety of

possible goals, assumptions, interpretations, or perspectives which can give alternative meanings or solutions to given situations or problems

Example #3Formulate and test hypotheses by performing

laboratory, simulation, or field experiments in at least two of the natural science disciplines (one of these experimental components should develop, in greater depth, students’ laboratory experience in the collection of data, its statistical and graphical analysis, and an appreciation of its sources of error and uncertainty)

Core Curriculum

6. What an assessment plan looks like

Component

Component

Component

Component

Component

Component

Component

Component

Component

Component

Component

Component

Component

Component

Component

Relate Speak Listen ParticipateWrite

Core Curriculum


7. How assessment works

How Assessment Works

LEvent 1LEvent 2LEvent 3

componentcomponentcomponentcomponentBASELINE

ProcessReflection;Compareresultsagainst

Benchmarks,Standards,

Targets,Past

Performance

New / RevisedLEvent 1




New / Revised LEvent 2

New / Revised LEvent 3

componentcomponentcomponentcomponent

OUTCOME

componentcomponentcomponentcomponent

Cycle 1 Cycle 2 Cycle 3

Core Curriculum


7. How assessment works8. Why you need to define your

outcomes

volume

poise

conclusion

Speaking

teacher5

eye contact

style

appearance

gestures

rate

evidence

sources

examples

organization

transitions

verbal variety

attention getter

teacher4 teacher2 teacher1 teacher3

volume

poise

conclusion

eye contact

style

appearance

gestures

rate

evidence

sources

examples

organization

transitions

verbal variety

attention getter

Can our students deliver an effective Public Speech?

Core Curriculum



outcomes9. How to implement the plan

StudentLearning Outcomes Course

1

X

X

X

Course2

X

X

X

X

X

Course3

X

X

X

X

Course4

X

X

X

X

X

Course5

X

X

X

X

Phase Four

Core Curriculum



outcomes9. How to implement the plan10. What to do with the data

Patterns of Evidence

DP1 DP2 DP3

Interpreting Data

• Consistency - over time

• Consensus - different populations

• Distinctiveness - different situations/ variables / items

Consistency

• Examines the same practice of and individual or group over time

• Key question: »Has this person or group acted,

felt, or performed this way in the past / over time?

ConsistencyHow well are students performing on the

departmental learning outcome measures?

04 0500 01 02 03

Highperformance

Lowperformance

Consensus

• Comparison to or among groups of students » Variation between disciplines, gender, other

demographic variables

• Key questions: – What is the general feeling, outcome,

attitude, behavior? » Do other groups of people act, perform or

feel this way?

ConsensusHow well are students performing on the

departmental learning outcome measure?

Females Males Transfers OTA

Highperformance

Low performance

Distinctiveness

• Examines individual or cohort perspectives across different outcomes

• Key Question: » Does a person or group perform equally

as well on different outcomes?

Distinctiveness

ANALYSIS

RESEARCH

WRITING

SPEAKING

How well are our students performing on the learning outcomes?

HighPerformance

Low Performance

ETHICS

THINKING

Fundamental Question #1

• Do we have enough data from which to really draw conclusions?


• Does the data represent an identifiable trend in the level of activity / achievement / accomplishment?


• Does the data represent an acceptable level of activity / accomplishment / achievement given our mission and values?


• Are the differences in the sub-populations acceptable?


• What can we do about it?

Really Big Mistakes

Big Mistakes in Assessment

• Assuming that it will go away

• Allowing assessment planning to become gaseous

• Assuming you got it right -- or expecting to get it right -- the first time

• Not considering implementation issues when creating plans


• Borrowing plans and methods without acculturation

• Setting the bar too low

• Assuming that you’re done and everything’s OK, or rushing to “Close the Loop”

• Doing it for accreditation instead of improvement


• Confusing program effectiveness with student learning

• Making assessment the responsibility of one individual

• Assuming collecting data is Doing Assessment

Assessment 101:The Core Curriculum

Susan Hatfield - Winona StateUNC WilmingtonNovember 2005

[email protected]

assessment 101: the core curriculum susan hatfield - winona state unc wilmington november 2005...

Documents

assessment measures

realized assessment

campus culture of assessment

learning eventsd

faculty member

little quizquizall faculty

combination of data

uncertainty uncertainty