3 clarity

TO APPLY PAGE STYLESRIGHT CLICK YOUR PAGE>LAYOUT

SCHOOL OF ANIMAL AND VETERINARY SCIENCES

Clarity and why it’s important

Liz NormanMassey Universityhttp://www.slidehare.net/liznorman

Sharanne Raidal, BVSc, PhD, FANZCVS, GradDipEd

Australian and New Zealand College of Veterinary ScientistsExaminer Workshop, 6 – 7th February 2016

http://www.slidehare.net/liznorman



“Effective item writers are trained, not born....”

Downing and Haladyna (2006) Handbook of Test Development, page 11



Validity

Measuring what we want to measure

andNot measuring what we don’t want to measure

Content domain

Exam 2

OralCredentials

domain

Exam 1

Prac

Practice domain



What we don’t want to measure

• Ability to take tests• Ability to write legibly and fast• Ability to rote learn whole pages of textbooks or review articles

– prewriting• Ability to write down a huge series of unconnected facts in no

particular order• Ability to research examiner’s fields of interest and rote learn

impressive aspects of that • Ability to interpret what examiners are thinking



Why clarity is important

• We want the candidate to do the task we envisaged, not something else

• We want to minimise irrelevant difficulty

• Validity – measuring what we intend to measure



CommunicationThe examination questions are the question setter’s expression of the question setter’s task.

The candidate’s answer represents the candidate’s expression of the candidate’s interpretation of the questions.

The marker evaluates the marker’s interpretation of the candidate’s expression of the candidate’s answer.

The marker uses the marker’s interpretation of the setter’s expression of the setter’s task to evaluate the candidate’s answer.

Modified from Pollitt & Ahmed (1999) New Model of the Question Answering Process. IAEA. Bled. Slovenia



General issues with question wording



Expectations and stereotypes

Examples:•“treatment”•All differential diagnoses vs those only applicable in a particular case•Expectation that Qs will ask about what something is rather than what it is not•Expectation of hard questions



Crisp et al (2008) Tales of the expected: the influence of students‘ expectations on question validity and implications for writing exam questions. Ed Res 50(1): 95-115.



Discuss the diagnosis and management of a cat with both chronic renal failure and hyperthyroidism. (24 marks)



Contextualising questions

• Context is good because it brings relevance and authenticity

• Allows assessment of concrete or specific examples not abstract concepts or generalisations

• Allows assessment of applied learning (doing not just knowing)

• All these carry with them a potential for bias



Contextualising questions

Other disadvantages:•More words used•More intended / unintended demand•Familiarity – schemas / stereotypes•Focus may direct candidates to the wrong aspects•Images can be particularly distracting



You are presented with a poorly controlled diabetic dog. The dog is an 8 year old female spayed Labrador weighing 36 kg with a body condition score of 7/9. The dog has been on porcine lente insulin (Caninsulin) twice daily for 3 months. Various doses have been tried during this time and the dog is currently receiving 70 units twice daily. The dog is still polyuric and polydipsic, with 3+ to 4+ glucosuria.

Discuss the possible causes of poor control which you should consider when evaluating this dog.



List the possible causes of consistently high blood glucose values in a dog being treated with subcutaneous insulin injections for the treatment of diabetes mellitus. Outline your approach to investigating these possible causes.



Question 1A crate of 12 cans of cola costs $4.20. How much do 7 crates of cola cost?

Question 2A ski pass costs $42.0. How much would it cost for 7 days.



Guidelines for using context

• If you are going to use context, use a natural, realistic one

• Contextualise purposefully for what it brings to the task

• Only use images when the question could not be asked without them

• Don’t decorate!



a) After several weeks in a tropical climate the body can adapt by increasing maximum swat rate to 3.5 L/h. at what rate, in watts, is energy being removed when sweating that rapidly. Assume all of the sweat evaporates. (3 marks)

b) If a 50 kg girl stopped seating because of severe dehydration, how long would it take for her body to rise to a dangerous level of 40oC if originally she was sweating at a rate of 3.5 L/h to maintain a body temperature of 36oC? (2 marks)



Don’t write questions, write tasks

What is your diagnosis?

State the most likely diagnosis

State the most likely diagnosis and explain your reasoning.

Discuss the differential diagnoses you would consider in this case.



Instructional verb examples

Compare: to find similarities between things, or to look for characteristics and features that resemble each other.Contrast: to find differences or to distinguish between things. Discuss: to present a detailed argument or account of the subject matter, including all the main points, essential details, and pros and cons of the problem, to show your complete understanding of the subject. Define: to provide a concise explanation of the meaning of a word or phrase; or to describe the essential qualities of something. Explain: to clarify, interpret, give reasons for differences of opinions or results, or analyse causes. Illustrate: to use a picture, diagram or example to clarify a point.



Specify boundaries of the answerSpecies

eg. “in both dogs and cats....”Quantities and amounts

eg. “Provide 5 reasons why.....”With reference to

eg. “With reference to published research from...”Time

eg. “in the first 24 hours”Part of the question

eg. “for one of your differentials....”



What are the clinical signs of hypothyroidism in dogs and how do they arise?

List the three most common owner-observed clinical signs of hypothyroidism in dogs and explain how thyroid hormone deficiency leads to each of these signs.



Guidance

Question 1Discuss the use of insulin for the treatment of diabetes mellitus in cats. (25 marks)

Question 2a) Describe the advantages and disadvantages of insulin therapy for diabetes mellitus in cats. (10 marks)b) Indicate the dose and frequency of administration of insulin you would prescribe to a newly diagnosed cat with diabetes mellitus. (5 marks)c) Describe the recommendations you would make for the frequency and timing of feeding in relation to insulin dosing in cats with diabetes mellitus. (10 marks)



Guidance

You have been contracted by a farmer producing Pacific oysters (Crassostrea gigas) intertidally, in a bay containing a number of oyster farms. The farmer is concerned with the amount of dead shell they are seeing during the current grading. Explain how you would approach this scenario. (20 marks)

Include in your answer how the information you could gather might influence your assessment, what differential diagnoses you consider, and detail how you might further investigate potential causes, and what advice you would provide.



Purpose of marking schemes

•To help you during question writing What content is important Whether the Q asks what you intended it to ask Whether it is do-able in the time available

•To help you during question marking Helps you decide whether an answer is good enough to be awarded a

mark Facilitates reliable and fair marking



Wording of questions – examples of problems



Name two (2) diagnostic tests you would run next to investigate the cause of this dog’s current illness.



Outline your approach to confirming the initial clinical diagnosis and a management and prevention plan for this problem. This discussion should include an outline on further observations taken about .....



A veterinarian asks you for assistance in designing a protocol for the delivery of a vaccine for cats in their practice. What factors would you take into consideration in designing this protocol?



Are there any clinical features which can help you determine this patient’s prognosis?



State what you believe is your most likely diagnosis.



Discuss commonly found tumours and tumour-like disorders associated with the oral cavity and dental tissues of the horse.



How would you localise the site of the lesion?

Answer provided in marking scheme:Spinal lesion between T3 and L3



Write notes ona) Considerations in the selection of stockpersons and animal attendants



Briefly outline your interpretation of results. Do not just state the abnormalities.



Sentence-level clarity

• Simple sentence structures• Grammatically correct• Remove superfluous words



Question timing



How long will it take the candidate?

• Unstructured tasks – open ended• Need to control the demands of the question

carefully

• Question reading time• Writing speed



Reading rates reported by 104 medical students at the Mercer University School of Medicine, 2010

Number (%) of students

Number of pages read per hour

Reading rate (WPM)

13 (13%) ≤ 6 ≤ 5055 (53%) 7 – 12 51 - 10018 (17%) 13 - 18 101 - 15016 (15%) 19 - 24 151 - 2002 (2%) ≥ 25 ≥ 201

Klatt and Klatt (2011) Acad Med 86: 1079 - 1083



Poppy is a 9 year old 12 kg female neutered fox terrier cross who has been on treatment for diabetes mellitus with Caninsulin (porcine lente insulin) for 4 months. She is currently receiving 5 units subcutaneously at 7:30am and 5 units at 7:30pm. At the time of each injection she is fed a mixture of ‘Optimum’ dog roll and Pedigree Pal Meaty Bites. Her last visit was 2 weeks ago when her dose of insulin was decreased by 2 units. Since the last visit Poppy has been happy and active and is eating all her food at mealtimes. Her body weight has not changed since the last visit and her body condition score is 5/9. Her owner reports her to be drinking about 600 mL of water daily. Her owner is very happy with Poppy’s progress and comments that she is her normal self again. There have been no signs of hypoglycaemia. You detect no abnormalities on physical examination and admit Poppy to perform a serial blood glucose curve. Her owner has administered the morning insulin dose and fed her as usual. You have been provided with the results obtained. Based on your assessment of the clinical information and the glucose curve shown, what dose of insulin would you recommend Poppy be given now?

(5 marks)



A client rings you up to arrange his quarterly visit as he is frustrated by ongoing problems with his breeding herd. Some sows seem to take ages to cycle after weaning, a lot are returning (and at funny times), there is a steady trickle of abortions, and a lot of stillborns. When you get to the farm, the farmer mentions that he has also ha d a few sows go down at farrowing with a fever, they usually start panting and die, and there’s one right now in the old farrowing room he wants you to take a look at. As you walk through the farrowing rooms you notice that many of the sows, both expecting and lactating, have swollen vulvas. The sow in question farrowed yesterday. She clearly has a temperature and is panting. You also thinks she looks a bit anaemic and note that she doesn’t appear to have any milk. You suspect what the problem is but think it would be nice to confirm your diagnosis as, although it is suspected to occur in Australia, the disease has never been definitively confirmed. You take a blood sample and make a smear.

On your way home you drop the slide off at the lab and ask them if they will have a look at it for you. Later that afternoon the pathologist rings up and (very) excitedly tells you that after using Wright’s stain she spotted some cocci-like organisms attached to the red blood cells. The pathologist has rung Biosecurity Australia and they are not interested in pursuing this particular finding as they have always considered it to be present.

You ring the farmer to tell him that you have confirmed your suspicion and that he now needs to embark on the course of action you had discussed with him earlier. Write the farmer a description of the disease and outline the short and long-term course of action. Include in your report a brief discussion of what, if any, potential there is for eradication. (25 marks)



Handwriting speed for three writing tasks

Writing taskHandwriting speed (wpm)

Mean SD Range

Handwriting speed test (3 min writing test) 37.83 4.71 27.0 – 51.7

Exam 1 (2 h exam with prior knowledge of questions) 17.75 2.78 11.8 – 25.7

Exam 2 (2 h exam without prior knowledge of questions)

15.37 3.12 9.4 – 23.2

Summers & Catarro (2003) Australian Occupational Therapy Journal 50(3): 148 - 157



Effect of time stress

Time stress •Increases the use of schemas (stereotyping)•Decreases working memory processing capacity•Decreases the ability to maintain relevant information and suppress irrelevant information



Oral (viva voce) Examinations Advantages•A student’s [practitioner’s] ability to verbally explain / communicate on a subject is a good measure of their understanding•Provide a measure of candidate’s verbal communication skills•Use of different (visual) cues•Alternative assessment technique - interactive•May provide a more authentic assessment of ‘real world’ skills – thinking on feet, progressive disclosure, no ‘cues’ in question•Rapid marking turnaround



Oral (viva voce) Examinations Disadvantages•Scarey! [may be influenced by personality]•Shorter time period – may limit the amount of material that can be covered relative to a written examination•Consistency may be hard to achieve•More subjective?•Harder to ‘re-grade’ / defend [College procedures]•Delivered asynchronously (confidentiality / security issues)



Oral Examinations – Design and Delivery• Identify question set according to Subject Guidelines (for each

day examined)• Situate oral exam content within your blueprint• Determine an appropriate way to deliver questions and cues

(images, data, other)• Decide appropriate questions – more colloquial delivery than

written papers, but aim for consistency• Identify / predict need for additional cues

Avoid prompting / coaching candidates during exam



Oral Examinations – Design and Delivery•Prepare a marking scheme (as for written / prac) and considering qualitative and quantitative (content) aspects of potential answers

•Ensure images (or other props) are of adequate quality•Be able to respond to problems during exam delivery - candidate nerves- problems with delivery / marking

•Consider what you will say and do during the exam (be prepared)



Conditions under which Oral Examinations are valid, reliable and fair• Ensure items focus on capabilities required for professional practice

(decision making, reasoning)• Appropriate content (defined by peers, profession)• Exam items are within the scope of professional practice, at an

appropriate level• Adequate sampling of questions • Examiner training (issues and methods)• Inter-examiner variation monitored, addressed• Consider potential for bias (pre- and post-exam)

Memon et al (2010) Oral Assessment and Postgraduate Medical Examinations: Establishing Conditions for Validity, Reliability and Fairness. Adv in Health Sci Educ 15(2): 277-289



Key points



Written papers - checking wording

• Check the question asks for what you are rewarding in your marking scheme

• Ensure the wording gives an instruction• Ensure the scope is clear – may need to be specified• Consider how the phrasing of the Q may distract or

misdirect candidates• Simplify wording if necessary• Check grammar and punctuation



Written papers - checking timing

• Allow reading time of 70 words per minute (only necessary to check unusually wordy examinations)

• Allow writing speed of no more than 20 words per minute (pref 15 words per minute)



Oral Examinations

• Included within blueprinting process• Check quality of images / prompts• Ensure clarity of your questions and marking scheme (as

for writtens)• Consider delivery (and responses) to candidates• Evaluate marking scheme

• CONSISTENCY



Next session’s task:

• Finalise your question wording• Print a copy of your Qs and marking schemes• Exchange your exam with someone from

another discipline• Give constructive feedback



End of the day

• Discussion on progress and challenges• Feedback loops between examiners and

chapters• How best for the BoE to support examiners• How best to administratively support examiners



Question / topic:

CriterionUnsatisfactory Novice Proficient Weighting0 1 2 3 4 5

Domain knowledge Little or no awareness of relevant information; frequent factual errors or omission of key information

Able to recall sufficient factual information relevant to topic; some errors or key information omitted

Well informed, excellent recall of relevant information; few or no errors, all/most key info covered

X2

Application of info Little or no understanding of topic; information presented randomly and with limited application to case scenario

Satisfactory understanding of topic, limited or simplistic application to case scenario

Good to excellent understanding of topic; information intelligently applied to case scenario

X2

Use of veterinary terminology

Unable to use, or limited use of, correct terminology

Able to use correct terminology, some errors in pronunciation

Fluent use of appropriate terminology

X1

Holistic impressionExaminer is uncomfortable with student’s preparedness for professional practice; major deficiencies identified on > 1 occasion

Examiner is satisfied that student can operate safely in a supportive environment

Examiner is impressed with student’s abilities; demonstrated knowledge, critical thinking and communication ability suggest advanced readiness for professional practice

Comments



Oral Examinations

•Mirrors oral communications typical of professional practice•Can test limits of knowledge and understanding (weak and strong candidates)•Good test of:

• clinical reasoning (the cognitive processes which constitute professional thinking)

• Interpersonal competence• Intra- personal qualities (confidence, self-awareness)




Oral Examinations

BUT•Resource intensive•Judgements based on limited evidence•May be difficult to defend on appeal•Unfamiliar (potentially difficult) for both examiners and candidate•Can be difficult to distinguish between what the candidate says and how they say it (discourse)•Performance limiting anxiety


3 clarity

Education