Clarity and why its important

Liz NormanMassey University

“Effective item writers are trained, not born … “

Downing and Haladyna 2006, Handbook of test development ,p. 11

Why clarity is important

• We want the candidate to do the task we envisaged, not something else– Validity – measuring what we intend to measure

• We want to minimise irrelevant difficulty– Validity – measuring what we intend to measure

Thinking about thinking

https://www.youtube.com/watch?v=EchfO2pjOrM

Thinking about thinking

• When candidates read a question, they must form a mental model of the task required and begin to plan their response.

• They begin to form a model before they have even finished reading the question.

• The model they form is influenced by their expectations and their pre-existing mental schemas and sterotypes

Elephant RidingJan Kemp

Climbing upthe back of an elephantyou spring intothe toehold of its tailheld in place by the mahoutgrab the ropesstrapped round its belly& haul yourself up.

She risesfrom buckled knees under youmoves like a shipyou’re highunder the hanging ashoka leavesas you flow forwardher fly-bitten ears grey sails flap.she flings the odd young-leaved branchinto her mouthwith her triumphant trunk.

Effects of anxiety and time stress on thinking

• Decreased working memory• Decreased processing capacity• Use of type 1 processing increases

– Automatic thinking– Use of schemas and sterotypes

• Lowered ability to concentrate on relevant information and suppress irrelevant information

Can lead to misunderstanding of the intent of the question and the task asked of them

Schemas and sterotypes

• All of us develop schemas or stereotypes• Help us categorise and process complex

information quickly• Particular features of questions trigger certain

schemas and hence expectations. • Triggered without us being aware of it and

before the reading of the text reaches consciousness

• Affect our interpretation of the question.

Implications for question writing

• There may be problems with questions that you cant see – your schemas are different

• The text at the beginning of a question will have more influence on the schema elicited than the text you place at the end.

• Images have more influence on the expectations developed than text

Implications for question writing

• Anxiety and time stress increase schema use– We might be just measuring propensity to

anxiety or writing speed• Each candidate will have different schemas

– We might be just measuring cultural backgrounds or experience with examinations

• Problems with questions may not affect all candidates equally and therefore may be hard to detect.

Contextualising Qs

• Context is good because it brings relevance and authenticity

• Allows assessment of concrete or specific examples not abstract concepts or generalisations

• Allows assessment of applied learning (doing not just knowing)

All these carry with them a potential for bias.

Contextualising Qs

Potential problems:• More words used • Use of colloquial expressions or

culturally-specific terms• Differential familiarity• Requirement for sifting• May activate schemas that interfere with

thinking

Ahm

ed &

Pol

litt (2

007)

Impr

ovin

g th

e qu

ality

of c

onte

xtua

lized

que

stion

s: A

n ex

perim

enta

l in

vesti

gatio

n of

focu

s. A

sses

smen

t in

Educ

ation

: Prin

cipl

es, P

olic

y &

Pra

ctice

, 14(

2), 2

01-2

32.

Focussed contextualisation

• Contextualisation works best when well focussed.

• The most salient aspects of the context are also the main issues addressed in the question.

• Makes Qs more likely to elicit helpful schemas than unhelpful ones, in candidates with relevant knowledge.

• Does not make Qs easier for candidates that lack relevant knowledge.

Ahmed, A., & Pollitt, A. (2007). Improving the quality of contextualized questions: An experimental investigation of focus. Assessment in Education: Principles, Policy and Practice, 14(2), 201-232.

Ahm

ed &

Pol

litt (2

007)

Impr

ovin

g th

e qu

ality

of c

onte

xtua

lized

que

stion

s: A

n ex

perim

enta

l in

vesti

gatio

n of

focu

s. A

sses

smen

t in

Educ

ation

: Prin

cipl

es, P

olic

y &

Pra

ctice

, 14(

2), 2

01-2

32.

Recommendations for maximising clarity

• Use clear language that is as simple as possible while still being precise

• Contextualise purposefully for what it brings to the task

• Avoid contradicting expectations or switching schemas part way through a question

Wording questions – examples of problems

Name two (2) diagnostic tests you would run next to investigate the cause of this dog’s current illness.

Outline your approach to confirming the initial clinical diagnosis and a management and prevention plan for this problem. This discussion should include an outline on further observations taken about ….

A veterinarian asks your for assistance in designing a protocol for the delivery of a vaccine for cats in their practice. What factors would you take into consideration in designing this protocol?

Are there any clinical features which can help you determine this patient’s prognosis?

State what you believe is your most likely diagnosis.

Discuss commonly found tumours and tumour-like disorders associated with the oral cavity and dental tissues of the horse.

How would you localise the site of the lesion?

Answer provided in the marking scheme:Spinal lesion between T3 and L3

Write notes on a) considerations in the selection of

stockpersons and animal attendants

Describe and give reasons for your further clinical examination (if any), and recommendations to the owner/trainer for diagnostics and treatment. Explain what you think is really important and why. Explain if you think there is any controversy in treatment plans.

Outline and discuss a conceptual framework for differentiating between ryegrass varieties available in the New Zealand market place and for defining possible strengths and weaknesses of a particular cultivar. Your conceptual framework should reflect the various options currently employed by plant breeders in developing new cultivars. (10 marks)

Which of the following statements is TRUE?a. A ventral hernia is classified as a false herniab. A diaphragmatic hernia is classified as a true herniac. Umbilical hernias are classified as false herniasd. False hernias have a hernia sace. Umbilical hernias are relatively uncommon

Briefly outline your interpretation of the laboratory results. Do not just state the abnormalities.

Discuss your approach to the management of large colon volvulus.

A client rings you up to arrange his quarterly visit as he is frustrated by ongoing problems with his breeding herd. Some sows seem to take ages to cycle after weaning, a lot are returning (and at funny times), there is a steady trickle of abortions, and a lot of stillborns. When you get to the farm, the farmer mentions that he has also had a few sows go down at farrowing with a fever, they usually start panting and die, and there’s one right now in the old farrowing room he wants you to take a look at. As you walk through the farrowing rooms you notice that many of the sows, both expecting and lactating, have swollen vulvas. The sow in question farrowed yesterday. She clearly has a temperature and is panting. You also think she looks a bit anaemic and note that she doesn’t appear to have any milk. You suspect what the problem is but think it would be nice to confirm your diagnosis as, although it is suspected to occur in Australia, the disease has never been definitively confirmed. You take a blood sample and make a smear.

On your way home you drop the slide off at the lab and ask them if they will have a look at it for you. Later that afternoon the pathologist rings up and (very) excitedly tells you that after using Wright’s stain she spotted some cocci-like organisms attached to the red blood cells. The pathologist has rung Biosecurity Australia and they are not interested in pursuing this particular finding as they have always considered it to be present.

You ring the farmer to tell him that you have confirmed your suspicion and that he now needs to embark on the course of action you had discussed with him earlier. Write the farmer a description of the disease and outline the short and long-term course of action. Include in your report a brief discussion of what, if any, potential there is for eradication. (25 marks)

Steps for checking clarity• Ensure the wording gives an instruction.• Ensure that the question scope is clear. Specify it if necessary.• Check the question asks for what you are rewarding in your marking

scheme—all of it.• Check that understanding the task does not rely on information that

might not be common knowledge in different parts of the world.• Consider how the phrasing of the question may distract or misdirect

candidates and rephrase if necessary.• Use the simplest language possible while still being precise. This includes

word choice and sentence structure. Remove all unnecessary wording.• Check that the discourse of the question mirrors the discourse of the

response you require.• Check spelling, grammar, and punctuation. Explain all abbreviations

unless knowledge of them is considered part of the curriculum.