assessment in education
DESCRIPTION
Assessment in Education. Patricia O’Sullivan Office of Educational Development UAMS. Last Month Dr. Allen described a situation about an individual asking her how to get to Freeway Medical Center. How would you determine how successful Dr. Allen was?. Objectives. - PowerPoint PPT PresentationTRANSCRIPT
Assessment in EducationAssessment in Education
Patricia O’SullivanOffice of Educational Development
UAMS
Last Month Dr. Allen described a situation about an individual asking her how to get to Freeway Medical Center. How would you determine how successful Dr. Allen was?
ObjectivesObjectives Provide terminology and principles
fundamental to assessment Introduce various assessment
approaches Review tips for a common assessment:
Multiple Choice Questions Review elements for a less common
assessment: Portfolio
What is Assessment?What is Assessment?
Appraising performance through measurement to make an evaluation
For what purpose? Formative Summative
What Assessments are Used at What Assessments are Used at UAMS?UAMS?
What Makes a Good What Makes a Good Summative Assessment?Summative Assessment?
Fair and practical Based on appropriate criteria that
are shared with the learners Valid Reliable
ValidityValidity Key to the interpretability and
relevance of scores Do the scores have a plausible
meaning? What are the implications of
interpreting the scores? What is the utility of the scores? What are the actual social
consequences?
Validity Evidence for Proposed Validity Evidence for Proposed InterpretationInterpretation
Content/Construct Evidence Content matches the objectives &
teaching Score is shown to be meaningful
Criterion/Predictor Evidence Score relates to future outcomes
ReliabilityReliability Assessment data must be
reproducible Written tests
• Internal consistencyCronbach’s alpha or KR-20 (values
range 0-1)• Test-retest
Correlation ranging from 0-1
Reliability (cont.)Reliability (cont.)
Rater-based assessments Interrater consistency or agreement
(generally correlation coefficient or intraclass correlation)
OSCE & performance assessments Generalizability Theory
Improve ValidityImprove Validity
Match assessment with objectives Increase the sample of objectives
and content on the assessment Review blueprint (or produce one) Use test methods appropriate to
objectives Ensure test security
Improve ReliabilityImprove Reliability
Clear questions Appropriate time allotted Simple, clear and unambiguous
instructions High quality scoring Increase number of observations or
questions
Norm or Criterion ReferencedNorm or Criterion Referenced
Norm-referenced (relative standard) Use the results of the assessment to set the
standards (e.g. proportion to receive each grade); performance judged by comparison with others
Criterion-referenced (absolute standard) Learners achieve some minimal standard of
competency
Standard Setting ProceduresStandard Setting Procedures
Judgments based on Holistic impressions of the exam or
item pool Content of individual test items Examinee’s test performance
Anghoff Standard Setting Anghoff Standard Setting ProcedureProcedure
Judges are instructed to think of a group of “minimally acceptable” persons
For each item estimate the proportion of this group that will answer item correctly
Sum up proportions to get the minimum passing score for a single judge
Average across judges
Assessment MethodsAssessment Methods ACGME Toolbox Additional methods
Other written assessments• Essay• Short answer and computation• Patient management problems• Modified essay questions
Create a game Self-assessment
Matching Assessment Method Matching Assessment Method and Objectivesand Objectives
Examine ACGME matrix
Alternative Testing ModesAlternative Testing Modes
Take-home tests Open-book tests Group exams Paired testing
Reducing Test AnxietyReducing Test Anxiety
Make the first exam relatively easier
Give more than one exam Give advice on how to study Encourage study groups Give a diagnostic test early
Reducing Test AnxietyReducing Test Anxiety
Before a test explain format Provide sample items Provide a pool of final items with
syllabus
Technology and AssessmentTechnology and Assessment
Computer based testing Use of images and video clips
Computer Adaptive Testing (CAT)
Take a TestTake a Test
Multiple Choice Questions: Multiple Choice Questions: Writing TipsWriting Tips
http://www.nbme.org/about/itemwriting.asp Simple suggestions:
Design for one best answer Place hand over stem. Can you
guess the answer? Use a template Attend to Graveyard of Test Items
Template ExamplesTemplate Examples A (patient description) is unable to
(functional disability). Which of the following is most likely to have been injured?
Following (procedure), a (patient description) develops (symptoms and signs). Laboratory findings show (findings). Which of the following is the most likely cause?
GraveyardGraveyard B-type: matching heading with
words or phrases (can use heading more than once)
D-type: complex matching—two step process
K-type: stem with four options and choices are A=1,2,3 only B=1,3 only C=2,4 only D=4 only, E=all are correct
PortfoliosPortfolios
Case Records
Logs
Conferences
Evaluations
PortfoliosPortfolios
Definition A purposeful collection of student
work that exhibits to the student (and/or other) the student’s efforts, progress, or achievement in (a) given area(s).
This collection must include student participation in Selection of portfolio content The criteria for selection The criteria for judging merit Evidence of student
reflection
PortfoliosPortfolios
Learning Showing progress Showing effort Showing
remediation
Assessment Showing
competence
Case Records
Presentations
Evaluations
Logs
Criteria__________________
Criteria__________________
Criteria
Reflection
Portfolio
• Why selected
• How reflects skills
and abilities
Using Portfolios the Learner Using Portfolios the Learner Engages in AssessmentEngages in Assessment
Making choices Critically self-examining Explaining what they are doing
Well-structured PortfolioWell-structured Portfolio
Menu Selection criteria Scoring rubric Benchmarks Trained raters
Determine Reliability and ValidityDetermine Reliability and Validity Reliability: Generalizability theory
Found needed 6 entries with 2 raters Validity: number of studies
Resident performance across years Resident performance in areas Resident perception Resident performance when program made
an intervention Correlation of portfolio performance and
other areas
10-Step Approach10-Step Approach
1. Define the purpose
2. Determine competencies to be assessed
3. Select portfolio materials
4. Develop an evaluation system
5. Select and train examiners
10-Step Approach10-Step Approach
6. Plan the examination process
7. Orient learners
8. Develop guidelines for decisions
9. Establish reliability and validity evidence
10. Design evaluation procedures