assessment module - wikispacesof... · educational measurement assessment units aschroft and...
TRANSCRIPT
Principles of assessment
Assessment
module
Norbert Boruett
A Pragmatic approach
With Norbert Boruett
Educational measurement assessment units
Pre- amble
Everything that be counted does not necessary
count: everything that counts cannot necessarily
be counted. Albert Einstein, physicist 1879-1955
One test does not improve learning anymore than
a thermometer cures fever. we should be using this
tests to get schools to teach more of what we want
students to learn, not as a way to punish them.
Hubert, professor of education, Columbia
University
Educational measurement assessment units
Assessment- try to visualise
why should we assess?
what should we assess ?
How should we assess and
when should we assess?
Educational measurement assessment units
Effects of assessment on learning
and teaching Assessment influences what is taught and
what is learned. Often teaching is artificially
divide from assessment. The two teaching
and learning are linked by
feedback(Harden,1992).
. We shall discuss more on feedback on
subsequent parts of this module
Educational measurement assessment units
Effects of assessment
Torrance (1993) asserts that changing theories and methods of assessment have been the focus of significant attention for some years now. Curriculum developers have realised that real change will not take place in schools if traditional paper and pencil tests, be they essay or multiple choice, remain unchanged to exert a constraining influence on how teachers and pupils approach curricula.
Educational measurement assessment units
Aschroft and Foreman- Peck (1994) point
out that assessment is a key element in the
effective management of learning than any
component of the learning process.
Curriculum developers have noted that
positive change will not take place in
learning institutions if assessment does not
change (Tornance,1993).
Effects of assessment on learning
and teaching- cont
Educational measurement assessment units
Miller (1976) puts it more succinctly by asserting
that to change the curricular without changing
examination will achieve nothing.
He further observes that changing the
examination systems without changing the
curriculum had a more profound impact upon the
nature of learning than changing curriculum
without altering the examination.
Effects of assessment on learning
and teaching- cont
Educational measurement assessment units
Effects of assessment on learning
and teaching- cont The bicycle is a useful model when
considering assessment and teaching. Teaching and learning are represented by front wheel, assessment by the rear wheel. Disasters may occur when the two wheels go in different directions. If the rear wheel is omitted altogether only one wheel remains: difficulties and problems arise. In the same way problems may arise if a curriculum is designed with no assessment component
Educational measurement assessment units
Assessment and evaluation-?.
Aschrof and Foreman – Peck (2000)
highlight that assessment is often confused
with evaluation. Assessment can be
conceptualised to mean judgment about the
progress (formative assessment) or
achievement (summative assessment) of a
particular course of study or a particular
teaching intervention (Nayar, 1995).
Educational measurement assessment units
Formative assessment
There is a problem with formative
assessment since continuous assessment is
frequently used summatively.
Breaks learning into manageable units
Allows repeated attempts to master the
content of each module
Is not perceived as threatening (low stakes)
Educational measurement assessment units
Summative assessment
Is the end point examination
Can block intended career progression(high
stakes)
Is perceived as threatening
Educational measurement assessment units
Criterion and normative
assessment Assessment can be seen as criterion or
normative based. Brown (1994) postulates that criterion based assessment tells us what has been achieved. He further articulates that this type of assessment should have a description telling us what has not been achieved and the reason behind the failure. Put, succinctly criterion referencing is designed to check what a student has mastered in an area of knowledge or skill.
Educational measurement assessment units
Tools in assessment and
evaluation Brown (1995) asserts that the same tools
may be used to assess and evaluate at the
same time, for example it is possible to use
one tool to evaluate a teaching method and
assess a student. If a student does well in an
examination, it follows then that the
teaching method was appropriate. This
example implies the function as an
evaluation of the teaching received.
Educational measurement assessment units
Conclusion
As a principal objective of medical
education is to produce a competent
physician , then , unquestionably , the
raison d'être for the evaluation process is to
assess the competency and not the rank
order of students. To this end, criterion –
reference testing is necessary and must
become the principal method of evaluation
within medical education( Turnball,1989)
Educational measurement assessment units
Who designs an administers the
assessment ? Traditionally the teacher has been
instrumental in executing this task.
Perhaps you may now be aware that currently there exist assessment methods where students take sole responsibility and much-pronounced role in their assessment
Examples here include portfolio-learning, self-assessment, peer assessment (Brown, 1992; Boud, 1995).
Educational measurement assessment units
Assessing ability- competence
Hubbard adage states that:
There is something more scarce,
Something finer far,
Something rarer than ability.
It is the ability to recognise ability
Educational measurement assessment units
OSCE-as an answer
Lumley (etal., 1999) assert that OSCE
brings a new dimension to assessment of
medical training. Of particular importance
is the ability to examine practical and other
skills in a unified, measurable and
reproductive fashion.
Educational measurement assessment units
OSCE CONT
They are effective in assessing knowledge
and practical skills and ensure that each
student is presented with the same material,
thus providing a uniform evaluation with
the consistent marking of all those involved.
Do you recall the Kenyan experience in
OSCE?
Educational measurement assessment units
Why OSCE’S?
Construct validity is the ability of the
OSCE to differentiate students ability , or
follow a student’s progress before and after
a course of instruction.
Content (criterion- based) validity assesses
the value of the station in reaching its
specified objective.
Educational measurement assessment units
Why should students be
assessed?
Safeguarding the public
Feedback to the students
Certification
Monitoring the programme
Educational measurement assessment units
Feedback
Rowntree has laid down a claim that feedback,
or knowledge of results, is the lifeblood of
learning.
Feedback is valuable – hence it should not consist
of notification of pass or failure. Feedback
should be given immediately and comprehensive
at the end of an examination (Dunnington et al.,
1992). Feedback should be given positively
(Amiel and Lunenfeld, 1992).
Educational measurement assessment units
Guidelines for giving feedback
Timing of feedback
· . Methods of providing feedback
Students checked their answers against the correct ones written on the board.
Students discussed the questions and answers with an instructor.
Students were assigned to textbook materials relating to questions they had answered incorrectly
Educational measurement assessment units
Feedback methods cont
Give student’s checklist with examiners
ratings
Watching video tape of own performance
Discussion with examiner
Watching video tape of expert
Receiving feedback from patient or
standardised patient
Reading computer generated report of
performance
Educational measurement assessment units
What should we assess?
Knowledge (Cognitive)
Skills (Psychomotor)
Attitudes (Affective
Educational measurement assessment units
How should we assess?
Validity- ensure the examination is valid
Reliability- ensure that the exam is reliable
Feasibility- set a realistic exam that can measure what you want to.
Educational measurement assessment units
When should students be
assessed?
During the course of training
At the end of the course
Educational measurement assessment units
Formative assessment
The scoring of formative assessment
should be strictly criterion referenced so as
to indicate to students their level of
performance. Aschrof and Foreman – Peck
(2000) asserts that formative assessment
should support student learning , records of
achievement , learning conflicts, portfolio
record of competence.
Educational measurement assessment units
quip
Examinations are formidable even to the
best prepared , for the even the greatest fool
may ask more than the wisest man can
answer.
Charles Caleb Colton ,1780-1832
Educational measurement assessment units
Concepts associated with assessment
Validity- measuring what you purport
(want)to measure
Reliability- consistency.
Educational measurement assessment units
Types of validity
Face validity-
Content validity
Predictive validity
Consequential validity
Construct validity
Educational measurement assessment units
Type of validity Test facet being
measured
Questions asked
Face validity Compatibility with
educational philosophy
What is the test face
value? Does it match up
with educational
intentions?
Content validity The content of curriculum
Does the test include a
representative sample of
the subject matter?
Construct validity The ability to differentiate
between groups with
known difference in
ability(beginners versus
experts
Does the differentiate at
the level of ability
expected of candidates at
the stage in training?
Educational measurement assessment units
Type of validity Test facet being
measured
Questions asked
Predictive validity The ability to predict an
outcome in the future eg
professional success after
graduation
Does the test predict
future performance and
level of competence?
Consequential validity The educational
consequence of the test
Does the test produce the
desired educational
outcome?
Educational measurement assessment units
Reliability
Inter-rater reliability- which correlates the
consistency of rating of performance across
different examiners
Inter-case reliability which quantifies the
consistency of performance of candidate
across the cases
Educational measurement assessment units
Reliability
Parallel form reliability
Test retest reliability
Educational measurement assessment units
Estimating reliability
Split half
Cronbach alpha coefficient
how can we handle reliability issues?
Educational measurement assessment units
Why use norm referencing tests?
As you answer that - think of the inherent
dangers of norm referencing
Outline some of them
Educational measurement assessment units
Criterion-referenced tests
How is this different from the former? And
what are the advantages and disadvantages
of this referencing visa –vis the former
Educational measurement assessment units
What are the practical issues od
assessment design Utility=
reliabilityXvalidityXfeasbilityXacceptabilit
yXeducational impact- Cees van der
Vleuten 1996
Educational measurement assessment units
References
1. Talpot M. Monkey see and monkey do: a
critique of the competency model in
graduate medical education. Med Edu
2004:38 587-92
2. Wass V,Vleuten CPM van der,Shatzer
J,Jones R. Assessment of clinical
competence.Lancet 2001: 357 945-9
Educational measurement assessment units
Modernisng medical cereers. www.mmc.uk
Southgate L,Grant J. Principles for
assessment system for postgraduate
training.Post medical training Board.
www.pmetb.org.uk
Miller GE. The assessment of clinical
skills/comptence/performance. Academic
med 65 s 63-7