Fairness, Accuracy, & Consistency in Assessment

NCATE’s recommendations to reduce bias and ensure fairness to students

Fairness“Assess what’s been taught”

Candidates should be aware of the knowledge, skills, and dispositions which are measured in the assessments.

O Was it taught?O Curriculum map – shows where students

learn and practice what is assessedO Do students understand the expectations?

O Syllabi containing instructions and timing of assessments

O Rubrics/scoring guides shared with students

FairnessAccording to these guidelines, are your program’s

assessments FAIR?

O Is your curriculum map up to date?O Does the curriculum map link to all

professional standards and GSE standards and dispositions?

O Does the curriculum map indicate any gaps?

O Do syllabi indicate timing of assessments?

O Are rubrics/scoring guides shared with students when assigning the work?

Accuracy“Assessments measure what they say they measure”

Assessments should be aligned with standards and proficiencies they are designed to measure

O Are assessments aligned with standards?O Content matchO Complexity matchO Appropriate degree of difficulty

O Is there corroborating evidence?O Is there field input on the

assessment?

AccuracyAccording to these guidelines, are your program’s

assessments ACCURATE?

O Do your program’s assessment types (test, observation) match what is being assessed?O Dispositions = observationsO Skills = performance assessment

O Which of your assessments “validate” (relate to) other assessments?O Ex. - Work sample to lesson plan assignment

O Have you had input from working professionals?

Consistency“Assessments produce dependable, trustworthy results”

The assessment results should be reliably consistent regardless of time and rater

O Are scoring tools sufficiently descriptive?O Language to differentiate between components and

between performance levelsO Clear descriptors that promote accurate scoringO Students can tell from scoring tool why they were

rated at a certain levelO Are raters trained?

O Agreement between “what a 3 in lesson planning” looks like

O Raters understand of the consequences of final scoresO Programs have a plan to support/address students with

insufficient performance that alleviate rater-pressure

ConsistencyAccording to these guidelines, are your program’s assessments

CONSISTENT?

O Does your program consistently use descriptive rubrics for major assignments and performance observations?

O Does your program have regular rater training?O Has your program engaged in rubric calibration and

moderation activities?O Are raters aware of how their assessment’s scores

affect the student?O Are raters aware of how their assessment’s scores

contribute to program review?O What is the plan to support struggling students?

Avoiding Bias“removing contextual and cultural bias from assessments”

The assessment itself and assessment context should be analyzed for factors that would affect performance.

O Are clear assessments administered in the proper environment?O Location/equipmentO Clear instructions/questions

O Have assessments been reviewed for bias?O Racial/ethnic/cultural stereotypesO Disability resource center reviewO Assignments that favor one group over

another

Avoiding BiasAccording to these guidelines, are your program’s assessments BIAS-

FREE?

O Have all key assessments been reviewed for clarity of expectations?

O Have all your assignments been reviewed for accessibility?

O Have all your assignments been scrutinized for cultural bias and stereotypes?

O Has your program analyzed student outcomes according to sub-groups to determine if consistent scoring bias exists?

GSE Rubric GuidelinesDeveloped by the Assessment Committee - 2010

O 4 levels of competency: Unsatisfactory, Emerging, Proficient, Exemplary

O In ascending order from left to rightO If numbers are used: 1-4 from left to right

  NEEDS IMPROVEMENT

(1)

EMERGING (2)

PROFICIENT (3)

EXEMPLARY (4)

States values, beliefs and assumptions about priorities for resource allocation.

Statement of values, beliefs and assumptions is absent

Statement of values, beliefs, and assumptions is vague, too general or contrived.

Statement of values, beliefs, and assumptions is clearly defined and specific.

Statement of values, beliefs, and assumptions is clear, specific, convincing, and includes personal experience that promotes clarity.

Rubric ModerationO Process of strengthening consistency

O Develops inter-rater reliability through shared examination and discussion of student work.

O Involves all/many ratersO Process

O Recruit raters for a two-hour sessionO Provide 4 samples of workO Provide rubric and have raters “score” each

sampleO Discuss as a group why the raters chose the scoresO Debrief about what was learned, what remains

unanswered.

Rubric CalibrationO Process of strengthening consistency

O Develops inter-rater reliability by setting expectations of what the scores mean regarding student work.

O Involves all/many ratersO Process

O Recruit raters for a two-hour sessionO Provide 4 samples of work that have been pre-scored (anchor

papers), at least one at a low, medium, and high level of performance

O Discuss rubric areas and expectations for each level and component before scoring begins

O Provide rubric and have raters “score” each sample with your discussion in mind.

O Have raters compare the scores they assigned to the “anchor” papers

O Debrief about what was learned, what remains unanswered.

Next StepsO How is your program doing in

providing fair, accurate, consistent, bias-free assessments to students?

O What work needs to be done in your program to ensure quality assessments are used?

O What do you need to be able to accomplish this?