Assessment of Assessment of Information SkillsInformation Skills

There is more teaching going There is more teaching going on around here than learning on around here than learning and you ought to do something and you ought to do something about that.about that.

Graduating SeniorGraduating SeniorKing’s College, 1968King’s College, 1968

AssessmentAssessment

A systematic ongoing process ofA systematic ongoing process of setting goals or asking questions,setting goals or asking questions, gathering information,gathering information, interpreting it, andinterpreting it, and usingusing it it

to improve student learning.to improve student learning.Barbara D. WrightBarbara D. Wright

Why not use grades?Why not use grades?

““A grade is an inadequate report of an A grade is an inadequate report of an inaccurate judgment by a biased and inaccurate judgment by a biased and variable judge of the extent to which a variable judge of the extent to which a student has attained an undefined level student has attained an undefined level of mastery of an unknownof mastery of an unknown proportion of proportion of an indefinite material.”an indefinite material.”

Bill Boyle & Tom ChristieBill Boyle & Tom Christie

Issues in Setting StandardsIssues in Setting Standards

Falmer Press, 1996Falmer Press, 1996

Grades may reflect many things besides Grades may reflect many things besides mastery of course objectives:mastery of course objectives:

Verbal abilityVerbal ability

ParticipationParticipation

CooperationCooperation

Extra creditExtra credit

AttendanceAttendance

EffortEffort

Criterion Performance Criterion Performance vs. vs. Value AddedValue Added

Myths regarding student evaluationsMyths regarding student evaluations

Pay me nowPay me now

oror

Pay me laterPay me later

Align your OutcomesAlign your Outcomes

Institutional outcomesInstitutional outcomes

Program outcomesProgram outcomes

Course outcomesCourse outcomes

Class outcomesClass outcomes

Dimensions of AssessmentDimensions of Assessment

Formative - ProcessFormative - Process Feedback for improvementFeedback for improvement Corrective & diagnosticCorrective & diagnostic

Summative – Evaluative Summative – Evaluative JudgmentalJudgmental DecisionalDecisional

Dimensions of AssessmentDimensions of Assessment

An individual An individual A cohortA cohort A classA class A courseA course A programA program An institutionAn institution

Quantitative DataQuantitative Data

NumericalNumerical Statistically Statistically

reliablereliable Structured Structured

techniquestechniques ObjectiveObjective Assumes static Assumes static

realityreality

Usually can be Usually can be generalizedgeneralized

Allows for Allows for comparison and comparison and correlationcorrelation

Looks for facts Looks for facts and causesand causes

Qualitative DataQualitative Data

Not numericalNot numerical Provides Provides

“richness”“richness” Allows for Allows for

ambiguitiesambiguities Structured or Structured or

semi-structuredsemi-structured Written, verbal, Written, verbal,

visual, etcvisual, etc

Often subjectiveOften subjective Can be Can be

exploratoryexploratory Assumes Assumes

dynamic realitydynamic reality Looks for Looks for

motivations and motivations and points of viewpoints of view

AssumptionsAssumptions

Vs.Vs.

AssessmentAssessment

AssessmentAssessment

Enhance student learning and Enhance student learning and institutional effectivenessinstitutional effectiveness

Student learning rather than Student learning rather than faculty evaluationfaculty evaluation

Assess the process, rather than Assess the process, rather than the outcome, for the purpose of the outcome, for the purpose of improving the outcome improving the outcome

Measure What MattersMeasure What Matters

Measure what matters, Measure what matters, NOTNOT what is what is measurablemeasurable

Because what you measure Because what you measure becomes what you focus onbecomes what you focus on

AssessmentAssessment

Knowing Knowing what what you are doingyou are doing Knowing Knowing whywhy you are doing it you are doing it Knowing what students are Knowing what students are

learninglearning as a result as a result ChangingChanging because of the because of the

informationinformation

Debra GilchristDebra Gilchrist

Indirect Evidence of Student Indirect Evidence of Student LearningLearning

Surveys, self-reports & journalsSurveys, self-reports & journals Focus groups & interviewsFocus groups & interviews Alumni & employer surveysAlumni & employer surveys Percent of students entering Percent of students entering

graduate or professional schoolsgraduate or professional schools

Limitations of Indirect MeasuresLimitations of Indirect Measures

Provide data on factors that predict or Provide data on factors that predict or mediate learningmediate learning

DO NOT evaluate learning DO NOT evaluate learning per seper se DO NOT NECESSARILY imply that value-DO NOT NECESSARILY imply that value-

added learning has or has not occurredadded learning has or has not occurred(e.g., enthusiasm or lack of interest may be caused (e.g., enthusiasm or lack of interest may be caused

by factors not related to the course)by factors not related to the course)

Oswald RatterayOswald Ratteray

Students’ Reported Information Skills

Ability to Locate, Evaluate, and Select Information 1998 1999 2000 2001 2002 Average College Average 4.30 4.23 4.10 4.15 4.24 4.20 Accounting 4.26 4.33 4.07 4.21 4.32 4.24 Biology 4.13 4.48 4.23 4.35 4.14 4.27 Business 3.88 4.17 3.97 4.24 4.33 4.12 Chemistry 4.67 4.00 4.20 4.83 4.43 Communications 4.26 4.38 4.31 4.26 4.45 4.33 C. I. S. 4.33 4.17 4.21 4.16 4.23 4.22 Criminal Justice 4.50 4.13 3.96 4.00 4.24 4.17 Education 4.31 4.18 4.20 4.22 4.35 4.25 English 4.47 4.27 4.11 4.31 4.45 4.32

Scale: 5 (excellent) 4 (good) 3 (average) 2 (below average) 1 (poor)

Direct Evidence of Student Direct Evidence of Student LearningLearning

Student assignmentsStudent assignments Standardized testsStandardized tests Course embedded assessmentsCourse embedded assessments Portfolios of students’ workPortfolios of students’ work Capstone experiencesCapstone experiences Student performances & exhibitsStudent performances & exhibits Other observations of student behaviorOther observations of student behavior

Direct Evidence of Student Direct Evidence of Student LearningLearning

Rubrics & exemplarsRubrics & exemplars Concept mapsConcept maps Juried/peer review of student projectsJuried/peer review of student projects Performance on a case study or Performance on a case study or

problemproblem Locally devised testsLocally devised tests Commercially produced tests & examsCommercially produced tests & exams

Limitations of Direct Limitations of Direct MeasuresMeasures

Indicate:Indicate: WHAT students learnedWHAT students learned HOW MUCH they learnedHOW MUCH they learned What they DID NOT learnWhat they DID NOT learn

DO NOT indicate:DO NOT indicate: WHY students learn or did not learnWHY students learn or did not learn

DO NOT necessarily indicate:DO NOT necessarily indicate: Whether VALUE-ADDED learning occurredWhether VALUE-ADDED learning occurred

Oswald Oswald RatterayRatteray

Develop Rubrics to Assess Develop Rubrics to Assess Work:Work:

Levels of achievementLevels of achievement

Criteria that distinguish good work from poor Criteria that distinguish good work from poor workwork

Descriptions of criteria at each level of Descriptions of criteria at each level of achievementachievement

Peggy MakiPeggy Maki

Basic Rubric Grid FormatBasic Rubric Grid Format

TitleTitle

Scale Level 1Scale Level 1 Scale Level 2Scale Level 2 Scale Level 3Scale Level 3

Dimension 1Dimension 1

Dimension 2Dimension 2

Dimension 3Dimension 3

Dimension 4Dimension 4

Task Description

Why use a Rubric?Why use a Rubric?

Examine complex work efficientlyExamine complex work efficiently Clarifies faculty expectationsClarifies faculty expectations Communicates expectations Communicates expectations Improves students’ work Improves students’ work Criterion-referenced grades rather than Criterion-referenced grades rather than

normative-referencednormative-referenced Facilitates course or program Facilitates course or program

assessmentassessment

ExamplesExamples

PortfoliosPortfolios

Teaching portfolioTeaching portfolio Documents teaching over time Documents teaching over time

Course portfolioCourse portfolio Reflection & review of a single Reflection & review of a single

coursecourse Student portfolioStudent portfolio

Highlights student learningHighlights student learning

Portfolio ElementsPortfolio Elements

Course design, philosophy, Course design, philosophy, rationalerationale

ImplementationImplementation ResultsResults ReflectionReflection

Portfolio ElementsPortfolio Elements

Focus on the match between Focus on the match between assessment & course goalsassessment & course goals

Use existing assessments & data, when Use existing assessments & data, when possiblepossible

Include a variety of evidence typesInclude a variety of evidence types Be purposeful when selecting evidence Be purposeful when selecting evidence

about learningabout learning

Assessment is not about us, Assessment is not about us,

it is about student learning.it is about student learning.

Assessment is like Assessment is like scotch,scotch,

it is an acquired taste.it is an acquired taste.

Questions?Questions?

Seniors’ Information Skills by Standard : Mean Seniors’ Information Skills by Standard : Mean Scores/Percentage Correct - Fall 2004*Scores/Percentage Correct - Fall 2004*

GroupGroup NN Composite Composite ScoresScores

Std 1Std 1 Std 2Std 2 Std 3Std 3 Std 4Std 4 Std 5Std 5

142142 57.3257.32 60.5660.56 56.6256.62 60.0060.00 58.4558.45 50.9950.99

AA 2222 50.3650.36 52.7352.73 57.2757.27 52.7352.73 43.6443.64bb 45.4545.45

BB 3838 56.0056.00 58.9558.95 49.4749.47 57.3757.37 61.5861.58 52.6352.63

CC 2121 57.5257.52 55.2455.24 54.2954.29 61.9061.90 65.7165.71aa 50.4850.48

DD 1515 58.6758.67 65.3365.33 57.3357.33 66.6766.67 53.3353.33 50.6750.67

EE 2626 60.3160.31 63.0863.08 65.3865.38 60.0060.00 62.3162.31 50.7750.77

FF 1111 61.0961.09 69.0969.09 56.3656.36 63.6463.64 60.0060.00 56.3656.36

GG 99 64.0064.00 73.3373.33 64.4464.44 68.8968.89 60.0060.00 53.3353.33

ababRefers to comparisons within column where the MEAN scores of group Refers to comparisons within column where the MEAN scores of group a a are significantly (Pare significantly (P< < .05) higher than .05) higher than the MEAN scores of group the MEAN scores of group bb..*Please note that in some cases the small group size and number of questions per standard may preclude more *Please note that in some cases the small group size and number of questions per standard may preclude more meaningful statistical comparisons.meaningful statistical comparisons.

Seniors’ Information Skills by Standard : Mean Seniors’ Information Skills by Standard : Mean Scores/Percentage Correct -Fall 2004*Scores/Percentage Correct -Fall 2004*

GroupGroup NN Composite Composite ScoresScores

KnowledgeKnowledge ApplicationApplication

SeniorsSeniors 142142 57.3257.32 61.8061.80 53.2053.20

AA 2222 50.3650.36 50.7650.76bb 50.0050.00

BB 3838 56.0056.00 58.5558.55 53.6453.64

CC 2121 57.5257.52 61.1161.11 54.2154.21

DD 1515 58.6758.67 67.7867.78 50.2650.26

EE 2626 60.3160.31 66.6766.67aa 54.4454.44

FF 1111 61.0961.09 68.1868.18 54.5554.55

GG 99 64.0064.00 72.2272.22aa 56.4156.41ababRefers to comparisons within column where the MEAN scores of group Refers to comparisons within column where the MEAN scores of group a a are significantly (Pare significantly (P< < .05) higher than .05) higher than the MEAN scores of group the MEAN scores of group bb..*Please note that in some cases the small group size and number of questions per standard may preclude more *Please note that in some cases the small group size and number of questions per standard may preclude more meaningful statistical comparisons.meaningful statistical comparisons.

Information Skill of Seniors & Graduate Students from Three Northeast, Pennsylvania

Colleges by Standard: Mean Scores/Percentage - Fall 2004*

CollegeCollege NN Composite Composite ScoresScores

StdStd11

StdStd22

StdStd33

StdStd44

StdStd55

388388 57.2257.22 61.2461.24 54.5454.54 55.3155.31 58.0458.04 56.9656.96

1 (Srs)1 (Srs) 142142 57.3257.32 60.5660.56 56.6256.62 60.0060.00 58.4558.45 50.9950.99bb

2 (Grads)2 (Grads) 2929 56.2856.28 63.4563.45 48.2848.28 47.5947.59 57.2457.24 64.8364.83aa

3 (Srs)3 (Srs) 217217 57.2757.27 61.3861.38 54.0154.01 53.2753.27 57.8857.88 59.8259.82aa

ababRefers to comparisons within column where the MEAN scores of group Refers to comparisons within column where the MEAN scores of group a a are significantly (Pare significantly (P< < .05) higher than .05) higher than the MEAN scores of group the MEAN scores of group bb..*Please note that in some cases the small group size and number of questions per standard may preclude more *Please note that in some cases the small group size and number of questions per standard may preclude more meaningful statistical comparisons.meaningful statistical comparisons.

Information Skill of Seniors & Graduate Students from Three Information Skill of Seniors & Graduate Students from Three Northeast, Pennsylvania Colleges by Standard : Mean Northeast, Pennsylvania Colleges by Standard : Mean

Scores/Percentage - Fall 2004*Scores/Percentage - Fall 2004*

CollegeCollege NN Composite Composite ScoresScores

KnowledgeKnowledge ApplicationApplication

388388 57.2257.22 58.9358.93 55.6355.63

1 (Srs)1 (Srs) 142142 57.3257.32 61.8061.80 53.2053.20

2 (Grads)2 (Grads) 2929 56.2856.28 57.1857.18 55.4455.44

3 (Srs)3 (Srs) 217217 57.2757.27 57.3057.30 57.2557.25

ababRefers to comparisons within column where the MEAN scores of group Refers to comparisons within column where the MEAN scores of group a a are significantly (Pare significantly (P< < .05) higher than .05) higher than the MEAN scores of group the MEAN scores of group bb..*Please note that in some cases the small group size and number of questions per standard may preclude *Please note that in some cases the small group size and number of questions per standard may preclude more meaningful statistical comparisons.more meaningful statistical comparisons.

Five Stages of AssessmentFive Stages of Assessment(from Elisabeth Kubler-Ross)(from Elisabeth Kubler-Ross)

1.1. Denial Denial ““No, not me”No, not me”

2.2. Anger or resentment Anger or resentment ““Why me?”Why me?”

3.3. Bargaining Bargaining ““Yes, me, but…”Yes, me, but…”

4.4. Depression Depression ““Yes, me”Yes, me”

5.5. AcceptanceAcceptance