the regulatory landscape - colin robinson smsr1 grading and awarding – improving reliability?...
TRANSCRIPT
The Regulatory Landscape - Colin Robinson SMSR
1
Grading and Awarding – Improving reliability?
Colin RobinsonSocial and Strategic Market
Research(SMSR)
The Regulatory Landscape - Colin Robinson SMSR 2
What is a grade?
“A grade is an inadequate report of an inaccurate judgement by a biased and variable judge of the extent to which a student has attained an unidentified level of mastery of an unknown proportion of an indefinite amount of material.”
(adapted from Paul Dressel 1976)
The Regulatory Landscape - Colin Robinson SMSR 3
“An inadequate report”
Loss of information Combining across components Reducing performance judgments to
grades Imprecise
The smaller number of grades that are available for reporting, the more imprecise each grade becomes.
The Regulatory Landscape - Colin Robinson SMSR 4
“An inaccurate judgement”
Markers have to match the candidates’ responses to the mark scheme or performance criteria
Both responses and mark schemes are subject to interpretation
Some answers will be accepted that might have been rejected and some will be rejected that could have been accepted.
These are not “mistakes” but legitimate differences in interpretation
The Regulatory Landscape - Colin Robinson SMSR 5
“A biased and variable judge”
Marker variability Markers’ judgments vary according to
the time of day and personal circumstances
One marker’s judgment will differ subtly from another’s
The Regulatory Landscape - Colin Robinson SMSR 6
“An unidentified level of mastery unknown proportion of an indefinite
amount of material.”
Guidance to candidates and teachers comprises: The syllabus The grade descriptions Past papers/mark schemes Examiners reports
None gives exact detail of what is required
The Regulatory Landscape - Colin Robinson SMSR 7
The syllabus -1858 styleFrom Oxford Regulations for 1858 Mathematics
Euclid, Books I. II. Arithmetic Algebra to Simple Equations inclusive This amount of knowledge will enable a Candidate to
pass in this subject Questions will also be set in Euclid Books III. IV. VI., in
Quadratic Equations, Progressions, and proportion, Plane Geometry not beyond the Solution of Triangles, the use of Logarithms, Mensuration and Practical Geometry.
From Acland 1858
The Regulatory Landscape - Colin Robinson SMSR 8
The syllabus 2008 style
GCSE Mathematics 2007
The Regulatory Landscape - Colin Robinson SMSR 9
Marking 1858
Each of the examiners was instructed to take 100 as the maximum value to be assigned to the paper, and to mark the work of the each candidate according to that scale.
From Acland 1858
The Regulatory Landscape - Colin Robinson SMSR 10
Marking 2005
Code of practice April 2008
The Regulatory Landscape - Colin Robinson SMSR 11
Awarding 1858 The Examiners for the Preliminary
Examination were requested not only to give marks, but also to reject or pass. But this part of the work was considered so important that it was thought better that the examiners should come to Oxford and confer with the Subdelegacy on the standard to be adopted.
From Acland 1858
The Regulatory Landscape - Colin Robinson SMSR 12
Table showing marks assigned to the Exercises of Senior candidates in the Preliminary Examination. (adapted from Acland 1858)
100 90 80 70 60 50 40 30 20 10 0 X
Arithmetic 12 11 43 71 64 54 70 45 17 9 2 3
Spelling 87 97 88 37 31 18 14 15 6 4 4
Grammar 1 1 4 12 29 61 103 97 57 36
Composition 1 7 28 68 147 86 43 15 1 2 3
Geography 18 14 34 37 49 72 83 38 30 16 7 3
History 2 6 9 31 72 114 98 47 19 3
Handwriting 2 18 33 65 109 129 37 8
The Regulatory Landscape - Colin Robinson SMSR 13
The current awarding process
AQAGUIDETOSTANDARDSETTING.pdf
The Regulatory Landscape - Colin Robinson SMSR 14
Determining the grade cut-off(AQA 2005)
The Regulatory Landscape - Colin Robinson SMSR 15
Applying the indicators to two equally weighted components
The Regulatory Landscape - Colin Robinson SMSR 16
Using the Uniform Mark Scale 1
The Regulatory Landscape - Colin Robinson SMSR 17
Using the Uniform Mark Scale 2
The Regulatory Landscape - Colin Robinson SMSR 18
After the award
The role of the Accountable Officer
Borderline review
Enquiries and Appeals
The Regulatory Landscape - Colin Robinson SMSR 19
Conclusions Awarding and grading processes have a
range of mechanisms to improve reliability Statistical and judgmental evidence are
both essential Borderline review may reduce reliability Regrading on the basis of Enquiries are
justified only in extreme cases Reducing grades is hardly ever appropriate
The Regulatory Landscape - Colin Robinson SMSR 20
References Acland, T.D. 1858 Some account of the origin and
objects of the new Oxford Examinations for the title of Associate in arts and certificates for the Year 1858. Ridgeway London 1858
AQA 2005 A basic guide to standard setting. AQA Guildford 2005
AQA 2006 Uniform Marks in GCE, VCE, GNVQ and GCSE Examinations. AQA Guildford 2006
Dressel, Paul. (1976). Grades: One more tilt at the windmill. Bulletin. Memphis: Memphis State University, Center for the Study of Higher Education
