Grading criteria and marking schemes

Liz NormanMassey University

Validity

• Not a property of a test, but relates to the interpretations or actions made on the basis of test scores

• Eg:

The award of Bachelor of Veterinary Science implies that the awardee has sufficient knowledge and experience to

be a competent veterinarian

Establishing validity

• Requires formation of a logical and plausible argument based on evidence (Shaw et al 2012, Kane 2004)

• The chain of inferences and assumptions in the interpretations of test scores is critically examined for plausibility, threats to validity and the supporting evidence for validity

Inference 1

Inference 1

BVSc awardees have sufficient knowledge, skills, attitudes and judgement to be veterinarians

Inference 2

What we can measure in examinations (content domain) are things that are necessary to be a BVSc (extrapolation)

Inference 3

A particular set of examinations is a measure of achievement over all that could be examined

(extrapolation)

Inference 4

• The examination score is a measure of the level of achievement on the examination (generalisation)

Inference 5

• The passing score correctly separates performances at or above the level of achievement needed by a veterinarian from performances below that level

Validity

The exam must elicit proper evidence of the abilities of the candidate and we must evaluate the evidence properly

Validity

• Different markers need to award similar/scores for the same candidate response

• Markers need to reward features we want to evaluate and not reward features we do not want to evaluate

Purpose of marking schemes

• To help you during Q writing– What content is important– Whether the Q asks what you intended it to ask– Whether it is do-able in the time available

• To help you during Q marking– helps you decide whether an answer is good enough

to be awarded a mark– facilitates reliable and fair marking

Types of marking scheme

1. Model answer – “ideal” answer

2. Point-based schemes

3. Criteria- & level-based schemes

4. Schemes with incorporated principle(s) for discriminating levels

Model answers

• Not recommended as a sole component of a marking scheme– Usually more than would be expected to be given by

any candidate– No guidance on how to assess alternative answers to

the model provided– No guidance on how to award marks

• Can be a useful adjunct to a marking scheme

Point-based schemes

• Points for each objectively identifiable content point

• Does not indicate the relative importance of the points it awards

• Sum can be more or less than the whole• Rewards quantity not quality• Only useful for the quantitative SOLO levels

Prestructural

Unistructural Able to identify, list, name, enumerate but does not describe, explain, relate or elaborate multiple aspects of a response

Multistructural

Relational

Extended abstract

Quantitative change

Qualitative change

Criteria- & level-based schemes

• Criteria – different dimensions of performance• Level – different quality/standards of

performance on a given criteria

Criteria- & level-based schemes

• Generic vs specific for the Q• Explicit vs implicit weighting

Standards Very poor Poor Fair Excellent

Criteria

Analysis and interpretation of resultsTotal 8 marks

0-2 marksInterpretation not provided or incorrect.

2-4 marksLacks one or more key elements.

4-6 marksAdequate interpretation that addresses key elements. Misses nuances of interpretation or uncommon differentials

6-8 marksThorough accurate interpretation of results. Well justified and appropriately prioritised list of differentials.

Quality of planningTotal 8 marks

0-2 marksNo plans provided or plans not appropriate or dangerous

2-4 marksPlans miss some key aspects or overly general

4-6 marksAdequate plans that address all key differentials. Some displaced in priority or not pragmatic

6-8 marksThorough detailed and well-prioritised and pragmatic plan that addresses all defined differentials.

Knowledge of current literatureTotal 5 marks

0-2 marksLittle or no literature referred to or incorrectly referred to.

3-4 marksAnswer refers to some of the key literature

4-5 marksAnswer refers to current literature including controversies and comparative work from other species.

Logical presentationTotal 4 marks

1 mark Answer is disorganised and includes a large amount of irrelevant material

2 marksAnswer is somewhat disorganized and includes some irrelevant material

3 marksThe answer is relatively well organized and contains little irrelevant material.

4 marksThe answer shows a high degree of logical thought and well-constructed argument.

Southern Cross University (2013) http://scu.edu.au/teachinglearning/download.php?doc_id=12921&site_id=301&file_ext=.pdf

Analytical vs holistic schemes

• Both are valid• Analytical (criteria scored separately)

– Better agreement between examiners– Insufficient criteria– Overlapping criteria– Really just lots of smaller holistic decisions

• Holistic (scored as a whole)– Don’t straightjacket examiners– Challenging, especially for longer answers– Less agreement between examiners

Whittem (2013) Guidelines for Oral Examiners, Faculty of Veterinary Science, University of Melbourne

Armstrong et al. (2008) University of Western Sydney http://www.uws.edu.au/__data/assets/pdf_file/0004/449860/Assessment_Guide.pdf

Armstrong et al. (2008) University of Western Sydney http://www.uws.edu.au/__data/assets/pdf_file/0004/449860/Assessment_Guide.pdf

Wood (2012). Macquarie University http://staff.mq.edu.au/public/download.jsp?id=56594

Prestructural Question may be rephrased as the answer; almost completely misses the point of the question.

Unistructural Able to identify, list, name, enumerate but does not describe, explain, relate or elaborate multiple aspects of a response

Multistructural Able to list as well as describe distinct aspects of a response (such as being able to describe aetiology, clinical features, management of thrombotic stroke) but unable to explicitly explain causes for observations; unable to present cause-effect relationships.

Relational Able to describe multiple aspects of a process and additionally explain or elaborate observations into cause-effect relationships; able to compare similarities and differences between apparently distinct phenomena. This level is taken as suggesting that the learner has understood.

Extended abstract

Highly developed; able to explain mechanisms of phenomena and apply this information to a novel context — able to develop novel hypotheses, theories, and deduce principles; creative thinking.

Prakash et al. (2010) Adv Physiol Educ, 34(3):145-149

SOLO levels in marking schemesPrestructural The task may be engaged, but the student is distracted or

misled by irrelevant aspectsUnistructural The student focuses on the relevant domain and works with a

single aspectMultistructural The student provides correct material with discrete, separate

pieces of information that may be combined to provide a composite picture

Relational The student offers an integrated understanding of the information. The whole has a coherent structure and meaning

Extended abstract

Abstract general principles or hypotheses are provided

Scholten I, Keeves JP, Lawson MJ. High Educ 44:233–255, 2002.

Incorporating principles/rules

• Ideal is a specific criteria- & level-based schemes with incorporated principle(s) for discriminating levels

AQA GCSE Science A Physics 1 Foundation Tier Physics 1F Specimen Mark Schemehttp://www.aqa.org.uk/subjects/science/gcse/physics-4403/past-papers-and-mark-schemes

Incorporating principles

Calais has a warmer winter and a cooler summer than Wroclaw. Explain why. (3 marks)

Marking scheme: Looking for answers related to distance from the sea therefore latitude is not credited.

– Land heats up quicker than sea (1 mark)– A clear distinction between land and sea heating (2

marks)Ahmed & Pollitt (2011) Improving marking quality through a taxonomy of mark schemes. Assessment in Education: Principles, Policy & Practice 2011;18:259-278.

Incorporating principles

Marking scheme excerpt:Discussion should focus on strategies adopted to ensure sustainability and an evaluation of these with regard to whether or not or to what extent the Sahel can be sustainably managed. The discussion will depend on the content and whether the overall view is optimistic or pessimistic.

AQA (2013) General Certificate of Education (A-level) Geography Unit 1: Physical and Human Geography http://www.aqa.org.uk/subjects/geography/a-level/geography-2030/past-papers-and-mark-schemes

Quality vs quantity

• More complex and unstructured the Q the more assessing quality not quantity

• In very constrained tasks only judging how correct the answer is

• In very open tasks, “correctness” is less important and its quality that's judged instead

Prestructural Question may be rephrased as the answer; almost completely misses the point of the question.

Unistructural Able to identify, list, name, enumerate but does not describe, explain, relate or elaborate multiple aspects of a response

Multistructural Able to list as well as describe distinct aspects of a response (such as being able to describe aetiology, clinical features, management of thrombotic stroke) but unable to explicitly explain causes for observations; unable to present cause-effect relationships.

Relational Able to describe multiple aspects of a process and additionally explain or elaborate observations into cause-effect relationships; able to compare similarities and differences between apparently distinct phenomena. This level is taken as suggesting that the learner has understood.

Extended abstract

Highly developed; able to explain mechanisms of phenomena and apply this information to a novel context — able to develop novel hypotheses, theories, and deduce principles; creative thinking.

Prakash et al. (2010) Adv Physiol Educ, 34(3):145-149

Content vs connections

Lucander et al. (2010). European Journal of Dental Education, 14(3), 145-150.

Writing marking schemes

• Select and organise the criteria/dimensions• Develop clear descriptions for each

level/standard of each criteria• Need to think about poor answers as well as

good ones

Revising mark schemes in use

• Hopefully all types of answer are anticipated• Sometimes though it is not – can indicate

unanticipated problems with the Q• Marking schemes might need revising after first

few students marked

http://www.slideshare.net/liznorman