blueprinting and drafting questions liz norman anzcvs 2017
TRANSCRIPT
![Page 1: Blueprinting and drafting questions Liz Norman ANZCVS 2017](https://reader035.vdocuments.net/reader035/viewer/2022081604/5899b5361a28aba11e8b58e1/html5/thumbnails/1.jpg)
Blueprinting and drafting questions
Liz NormanMassey University
![Page 2: Blueprinting and drafting questions Liz Norman ANZCVS 2017](https://reader035.vdocuments.net/reader035/viewer/2022081604/5899b5361a28aba11e8b58e1/html5/thumbnails/2.jpg)
What is a blueprint?
"blueprint, n.". Something which acts as a plan, model, or template
OED Online http://www.oed.comRailway and harbour report, Vancouver, B.C. by City of Vancouver Archives, Attribution License
![Page 3: Blueprinting and drafting questions Liz Norman ANZCVS 2017](https://reader035.vdocuments.net/reader035/viewer/2022081604/5899b5361a28aba11e8b58e1/html5/thumbnails/3.jpg)
Why we need to blueprint
Its important that we sample representatively from the content domain (all that it is possible to examine)
This is so we can generalise performance to the whole content domain
![Page 4: Blueprinting and drafting questions Liz Norman ANZCVS 2017](https://reader035.vdocuments.net/reader035/viewer/2022081604/5899b5361a28aba11e8b58e1/html5/thumbnails/4.jpg)
Constructing blueprints• Table or grid with at least 2 dimensions• Dimensions could include:
– Learning outcomes– Diseases or presenting problem– Body system– Species– Knowledge domain: anatomy, pathophysiology, diagnosis,
management, prognosis, epidemiology, investigations, ethics & law– Skills: interpretation, reasoning, decision making, evaluation, problem
solving– Cognitive level: factual recall, analysis and interpretation of data,
problem solving– Degree of abstraction: principles (theoretical), applied (concrete)– Location of question in the examination components
![Page 5: Blueprinting and drafting questions Liz Norman ANZCVS 2017](https://reader035.vdocuments.net/reader035/viewer/2022081604/5899b5361a28aba11e8b58e1/html5/thumbnails/5.jpg)
Constructing blueprints
• Begin by considering the curriculum• What knowledge, skills, and attitudes are
examinable?• Subject guidelines
![Page 6: Blueprinting and drafting questions Liz Norman ANZCVS 2017](https://reader035.vdocuments.net/reader035/viewer/2022081604/5899b5361a28aba11e8b58e1/html5/thumbnails/6.jpg)
Body system by knowledge domain
Written paper 1
Written paper 2
Oral examination
Practical examination
LO1 X X X
LO2 X X
LO3 X X X
LO4 X
![Page 7: Blueprinting and drafting questions Liz Norman ANZCVS 2017](https://reader035.vdocuments.net/reader035/viewer/2022081604/5899b5361a28aba11e8b58e1/html5/thumbnails/7.jpg)
Body system by knowledge domain by exam component
Pathophysiology Investigation and diagnosis
Treatment and management
Gastrointestinal P1Q1 P1Q1, P2Q4 OQ3
Cardiovascular P1Q4 P2Q2, OQ1 P2Q2
Nervous P1Q3, P2Q1
Endocrine P1Q3 OQ2 P2Q3
Respiratory P2Q5
![Page 8: Blueprinting and drafting questions Liz Norman ANZCVS 2017](https://reader035.vdocuments.net/reader035/viewer/2022081604/5899b5361a28aba11e8b58e1/html5/thumbnails/8.jpg)
Proportion of exam by species
species number of Qs percent
small animal 15 52%
farm 5 17%
horse 4 14%
exotic 3 10%
lab 1 3%
all 1 3%
![Page 9: Blueprinting and drafting questions Liz Norman ANZCVS 2017](https://reader035.vdocuments.net/reader035/viewer/2022081604/5899b5361a28aba11e8b58e1/html5/thumbnails/9.jpg)
Body system by knowledge domain by cognitive level
Pathophysiology Investigation and diagnosis Treatment and management
recall analyse, interpret
problem solve recall analyse,
interpretproblem
solve recall analyse, interpret
problem solve
Gastrointestinal P1Q1 P1Q1
P2Q4 OQ3
Cardiovascular P1Q4 P2Q2 OQ1 P2Q2
Nervous P1Q2 P2Q1
Endocrine P1Q3 OQ2 P2Q3
Respiratory P2Q5
![Page 10: Blueprinting and drafting questions Liz Norman ANZCVS 2017](https://reader035.vdocuments.net/reader035/viewer/2022081604/5899b5361a28aba11e8b58e1/html5/thumbnails/10.jpg)
Body system by knowledge domain by subject of question
Written paper 2 Pathophysiology Investigation and diagnosis Treatment and management
Gastrointestinal Investigation of exocrine pancreatic insufficiency
Cardiovascular Investigation of atrial fibrillation
Management of cardiomyopathy
Nervous Diagnosis of L2-3 spinal neoplasia
Endocrine Management of diabetic ketoacidosis
Respiratory Diagnosis of feline asthma Management of feline asthma
![Page 11: Blueprinting and drafting questions Liz Norman ANZCVS 2017](https://reader035.vdocuments.net/reader035/viewer/2022081604/5899b5361a28aba11e8b58e1/html5/thumbnails/11.jpg)
Weighting questions
Impact Weight Frequency Weight
Less important 1 Rarely seen 1
Essential 2 Relatively common 2
High impact 3 Very common 3
McLaughlin, K., Lemaire, J., & Coderre, S. (2005). Creating a reliable and valid blueprint for the internal medicine clerkship evaluation. Med Teach, 27(6), 544-547
![Page 12: Blueprinting and drafting questions Liz Norman ANZCVS 2017](https://reader035.vdocuments.net/reader035/viewer/2022081604/5899b5361a28aba11e8b58e1/html5/thumbnails/12.jpg)
Drafting questions
![Page 13: Blueprinting and drafting questions Liz Norman ANZCVS 2017](https://reader035.vdocuments.net/reader035/viewer/2022081604/5899b5361a28aba11e8b58e1/html5/thumbnails/13.jpg)
Creating appropriate tasks
• Questions are actually tasks• We need the tasks we assess to be things that
Members or Fellows need to be able to do.• This ensures we can validly extrapolate from
performance on the exam, to performance in the Membership or Fellowship domain.
![Page 14: Blueprinting and drafting questions Liz Norman ANZCVS 2017](https://reader035.vdocuments.net/reader035/viewer/2022081604/5899b5361a28aba11e8b58e1/html5/thumbnails/14.jpg)
Creating appropriate tasks
• Emphasis should be on assessing the candidate’s – understanding– skills in analysing and interpreting information– skills in problem solving– judgement
• Not recall of information
![Page 15: Blueprinting and drafting questions Liz Norman ANZCVS 2017](https://reader035.vdocuments.net/reader035/viewer/2022081604/5899b5361a28aba11e8b58e1/html5/thumbnails/15.jpg)
Recall-higher order classification
Fact recall Questions capable of being answered by reference to one paragraph in a text or notes (or several paragraphs for questions requiring recall of several facts)
Applied (higher order)
Questions that require the use of facts or concepts, the solution of a diagnostic or physiologic problem, the perception of a relationship, or other process beyond recalling discrete fact
Peitzman, S. J., Nieman, L. Z., & Gracely, E. J. (1990). Comparison of "fact-recall" with "higher-order" questions in multiple-choice examinations as predictors of clinical performance of medical students. Academic Medicine, 65(9), S59-60.
![Page 16: Blueprinting and drafting questions Liz Norman ANZCVS 2017](https://reader035.vdocuments.net/reader035/viewer/2022081604/5899b5361a28aba11e8b58e1/html5/thumbnails/16.jpg)
Knowledge-understanding classification
Knowledge knowing about; a body of coherent facts; can be thought of as right or wrong.
Understanding knowing how and why; the meaning of facts; the theory that links facts and provides meaning; how sense is made of facts to enable them to be applied to analysis, synthesis, evaluation; to be able to explain why particular facts or skills are applicable to a particular situation; to know which fact to apply when; to be able to create new knowledge or modify or adapt an idea to a new situation.
Wiggins, G., & McTighe, J. (2005). Understanding by design (2nd ed.). Alexandria, VA, USA: Association for Supervision and Curriculum Development.
![Page 17: Blueprinting and drafting questions Liz Norman ANZCVS 2017](https://reader035.vdocuments.net/reader035/viewer/2022081604/5899b5361a28aba11e8b58e1/html5/thumbnails/17.jpg)
SOLO taxonomy
http://pamhook.com/2012/01/20/creating-solo-taxonomy-symbols-in-many-colours/
Biggs, J. B., & Tang, C. S.-K. (2011). Teaching for quality learning at university (4th ed.). Maidenhead UK: McGraw-Hill.
![Page 18: Blueprinting and drafting questions Liz Norman ANZCVS 2017](https://reader035.vdocuments.net/reader035/viewer/2022081604/5899b5361a28aba11e8b58e1/html5/thumbnails/18.jpg)
Prestructural Question may be rephrased as the answer; almost completely misses the point of the question.
Unistructural Able to identify, list, name, enumerate but does not describe, explain, relate or elaborate multiple aspects of a response
Multistructural Able to list as well as describe distinct aspects of a response (such as being able to describe aetiology, clinical features, management of thrombotic stroke) but unable to explicitly explain causes for observations; unable to present cause-effect relationships.
Relational Able to describe multiple aspects of a process and additionally explain or elaborate observations into cause-effect relationships; able to compare similarities and differences between apparently distinct phenomena. This level is taken as suggesting that the learner has understood.
Extended abstract
Highly developed; able to explain mechanisms of phenomena and apply this information to a novel context — able to develop novel hypotheses, theories, and deduce principles; creative thinking.
Prakash, et al. (2010). Adv Physiol Educ, 34(3), 145-149.
![Page 19: Blueprinting and drafting questions Liz Norman ANZCVS 2017](https://reader035.vdocuments.net/reader035/viewer/2022081604/5899b5361a28aba11e8b58e1/html5/thumbnails/19.jpg)
Prestructural
Unistructural Able to identify, list, name, enumerate but does not describe, explain, relate or elaborate multiple aspects of a response
Multistructural
Relational
Extended abstract
Quantitative change
Qualitative change
![Page 20: Blueprinting and drafting questions Liz Norman ANZCVS 2017](https://reader035.vdocuments.net/reader035/viewer/2022081604/5899b5361a28aba11e8b58e1/html5/thumbnails/20.jpg)
Prestructural
Unistructural Able to identify, list, name, enumerate but does not describe, explain, relate or elaborate multiple aspects of a response
Multistructural
Relational
Extended abstract
surface
deep
![Page 21: Blueprinting and drafting questions Liz Norman ANZCVS 2017](https://reader035.vdocuments.net/reader035/viewer/2022081604/5899b5361a28aba11e8b58e1/html5/thumbnails/21.jpg)
Prestructural
Unistructural Able to identify, list, name, enumerate but does not describe, explain, relate or elaborate multiple aspects of a response
Multistructural
Relational
Extended abstract
recall
application
![Page 22: Blueprinting and drafting questions Liz Norman ANZCVS 2017](https://reader035.vdocuments.net/reader035/viewer/2022081604/5899b5361a28aba11e8b58e1/html5/thumbnails/22.jpg)
Prestructural
Unistructural Able to identify, list, name, enumerate but does not describe, explain, relate or elaborate multiple aspects of a response
Multistructural
Relational
Extended abstract
knowledge
understanding
![Page 23: Blueprinting and drafting questions Liz Norman ANZCVS 2017](https://reader035.vdocuments.net/reader035/viewer/2022081604/5899b5361a28aba11e8b58e1/html5/thumbnails/23.jpg)
Prestructural
Unistructural Paraphrase, define, identify, count, name, recite, follow simple instructions, calculate, reproduce, arrange, recognise, find, note, seek, sketch, pick
Multistructural Combine, classify, structure, describe, enumerate, list, do algorithm, apply method, account for execute, formulate, solve, conduct, prove, complete, illustrate, express, characterise
Relational Analyse, compare, contrast, integrate, relate, explain causes, apply theory (to its own domain), argue, implement, plan, summarize, construct, design, interpret (some senses), structure, conclude, substantiate, exemplify, derive, adapt
Extended abstract
Theorise, generalise, hypothesise, predict, judge, transfer theory (to new domain), assess, evaluate, interpret (some senses), critically reflect, predict, criticise, reason
Biggs, J. B., & Tang, C. S.-K. (2011). Teaching for quality learning at university (4th ed.). Maidenhead UK: McGraw-Hill.
![Page 24: Blueprinting and drafting questions Liz Norman ANZCVS 2017](https://reader035.vdocuments.net/reader035/viewer/2022081604/5899b5361a28aba11e8b58e1/html5/thumbnails/24.jpg)
Specifying tasks• Examples of instructional verbs
– Compare: to find similarities between things, or to look for characteristics and features that resemble each other.
– Contrast: to find differences or to distinguish between things.– Discuss: to present a detailed argument or account of the subject
matter, including all the main points, essential details, and pros and cons of the problem, to show your complete understanding of the subject.
– Define: to provide a concise explanation of the meaning of a word or phrase; or to describe the essential qualities of something.
– Explain: to clarify, interpret, give reasons for differences of opinions or results, or analyse causes.
– Illustrate: to use a picture, diagram or example to clarify a point.
![Page 25: Blueprinting and drafting questions Liz Norman ANZCVS 2017](https://reader035.vdocuments.net/reader035/viewer/2022081604/5899b5361a28aba11e8b58e1/html5/thumbnails/25.jpg)
Specifying scope
• Do you want a general answer about this condition that applies to all possibilities or one that specifically applies to this case?
• Do you want an answer to include – all possible options?– those that are currently available in this region of the
world?– those that are available under cost constraints?
• Writing marking scheme helps you get the scope right!
![Page 26: Blueprinting and drafting questions Liz Norman ANZCVS 2017](https://reader035.vdocuments.net/reader035/viewer/2022081604/5899b5361a28aba11e8b58e1/html5/thumbnails/26.jpg)
Specify boundaries of the answer
• Speciese.g. “in both dogs and cats…”
• Quantities and amountse.g. “Provide 5 reasons why…”
• With reference toe.g. “ With reference to the published research from ..”
• Timeeg: “in the first 24 hours”
• Part of the questionEg: “for one of your differentials….”
![Page 27: Blueprinting and drafting questions Liz Norman ANZCVS 2017](https://reader035.vdocuments.net/reader035/viewer/2022081604/5899b5361a28aba11e8b58e1/html5/thumbnails/27.jpg)
Construct irrelevance
When the skills, abilities or knowledge required to answer examination questions is not part of the domain we are trying to assess.
What are we actually assessing?
![Page 28: Blueprinting and drafting questions Liz Norman ANZCVS 2017](https://reader035.vdocuments.net/reader035/viewer/2022081604/5899b5361a28aba11e8b58e1/html5/thumbnails/28.jpg)
Important sources of construct irrelevance
• Unclear tasks – because the candidate may do something other than what we intended them to do.
• Unanticipated and irrelevant difficulty and demand
• Insufficient time to complete the task (fully)
![Page 29: Blueprinting and drafting questions Liz Norman ANZCVS 2017](https://reader035.vdocuments.net/reader035/viewer/2022081604/5899b5361a28aba11e8b58e1/html5/thumbnails/29.jpg)
Controlling task difficulty and demand
• Tasks need to be at an appropriate level of difficulty
• They don’t need to be “tricky” in order to be difficult
![Page 30: Blueprinting and drafting questions Liz Norman ANZCVS 2017](https://reader035.vdocuments.net/reader035/viewer/2022081604/5899b5361a28aba11e8b58e1/html5/thumbnails/30.jpg)
Some factors that affect demand and difficulty
• Type of operation– Simple steps vs synthesis, interpretation, evaluation
• Degree of novelty
![Page 31: Blueprinting and drafting questions Liz Norman ANZCVS 2017](https://reader035.vdocuments.net/reader035/viewer/2022081604/5899b5361a28aba11e8b58e1/html5/thumbnails/31.jpg)
Question 1:Explain the physiological actions of insulin.
Question 2:Explain the physiological actions of ghrelin.
![Page 32: Blueprinting and drafting questions Liz Norman ANZCVS 2017](https://reader035.vdocuments.net/reader035/viewer/2022081604/5899b5361a28aba11e8b58e1/html5/thumbnails/32.jpg)
Some factors that affect demand and difficulty
• Type of operation• Degree of novelty• Number of components or ideas involved
![Page 33: Blueprinting and drafting questions Liz Norman ANZCVS 2017](https://reader035.vdocuments.net/reader035/viewer/2022081604/5899b5361a28aba11e8b58e1/html5/thumbnails/33.jpg)
Question 1:Compare and contrast the clinical signs of hypoadrenocorticism with those of hyperadrenocorticism in dogs.
Question 2:Compare and contrast the clinical signs of diabetes mellitus with those of hyperadrenocorticism in dogs.
![Page 34: Blueprinting and drafting questions Liz Norman ANZCVS 2017](https://reader035.vdocuments.net/reader035/viewer/2022081604/5899b5361a28aba11e8b58e1/html5/thumbnails/34.jpg)
Question 3:Compare and contrast the clinical signs of diabetes mellitus with those of hyperadrenocorticism in dogs and cats.
Question 4:Compare and contrast the clinical signs of diabetes mellitus with those of hypoadrenocorticism in dogs and cats.
![Page 35: Blueprinting and drafting questions Liz Norman ANZCVS 2017](https://reader035.vdocuments.net/reader035/viewer/2022081604/5899b5361a28aba11e8b58e1/html5/thumbnails/35.jpg)
Some factors that affect demand and difficulty
• Type of operation• Degree of novelty• Number of components or ideas involved• Whether resources are provided or need to be
generated by the candidate
![Page 36: Blueprinting and drafting questions Liz Norman ANZCVS 2017](https://reader035.vdocuments.net/reader035/viewer/2022081604/5899b5361a28aba11e8b58e1/html5/thumbnails/36.jpg)
For EACH ECG, make a recommendation for management of the horse and explain your reasoning.
![Page 37: Blueprinting and drafting questions Liz Norman ANZCVS 2017](https://reader035.vdocuments.net/reader035/viewer/2022081604/5899b5361a28aba11e8b58e1/html5/thumbnails/37.jpg)
Some factors that affect demand and difficulty
• Type of operation• Degree of novelty• Number of components or ideas involved• Whether resources are provided or need to be
generated by the candidate• The question wording and any images, diagrams,
or tables provided
![Page 38: Blueprinting and drafting questions Liz Norman ANZCVS 2017](https://reader035.vdocuments.net/reader035/viewer/2022081604/5899b5361a28aba11e8b58e1/html5/thumbnails/38.jpg)
![Page 39: Blueprinting and drafting questions Liz Norman ANZCVS 2017](https://reader035.vdocuments.net/reader035/viewer/2022081604/5899b5361a28aba11e8b58e1/html5/thumbnails/39.jpg)
Some factors that affect demand and difficulty
• Type of operation• Degree of novelty• Number of components or ideas involved• Whether resources are provided or need to be
generated by the candidate• The question wording and any images, diagrams,
or tables provided• Degree of abstraction
![Page 40: Blueprinting and drafting questions Liz Norman ANZCVS 2017](https://reader035.vdocuments.net/reader035/viewer/2022081604/5899b5361a28aba11e8b58e1/html5/thumbnails/40.jpg)
Discuss the concept of quality of life in terms of biological function, “feelings” and natural existence.
Describe how both classical conditioning and operant conditioning are involved in cows confidently entering the milking shed and letting down.
![Page 41: Blueprinting and drafting questions Liz Norman ANZCVS 2017](https://reader035.vdocuments.net/reader035/viewer/2022081604/5899b5361a28aba11e8b58e1/html5/thumbnails/41.jpg)
Outline and discuss a conceptual framework for differentiating between ryegrass varieties available in the New Zealand market place and for defining possible strengths and weaknesses of a particular cultivar. Your conceptual framework should reflect the various options currently employed by plant breeders in developing new cultivars. (10 marks)
![Page 42: Blueprinting and drafting questions Liz Norman ANZCVS 2017](https://reader035.vdocuments.net/reader035/viewer/2022081604/5899b5361a28aba11e8b58e1/html5/thumbnails/42.jpg)
Some factors that affect demand and difficulty
• Type of operation• Degree of novelty• Number of components or ideas involved• Whether resources are provided or need to be
generated by the candidate• The question wording and any images,
diagrams, or tables provided• Degree of abstraction• Response strategy – simple, stepwise, integrated
![Page 43: Blueprinting and drafting questions Liz Norman ANZCVS 2017](https://reader035.vdocuments.net/reader035/viewer/2022081604/5899b5361a28aba11e8b58e1/html5/thumbnails/43.jpg)
Describe the skull. (30 marks)
![Page 44: Blueprinting and drafting questions Liz Norman ANZCVS 2017](https://reader035.vdocuments.net/reader035/viewer/2022081604/5899b5361a28aba11e8b58e1/html5/thumbnails/44.jpg)
Question 1Discuss the use of insulin for the treatment of diabetes mellitus in cats (25 marks)
Question 2a) Describe the advantages and disadvantages of insulin therapy for
diabetes mellitus in cats (10 marks)b) Indicate the dose and frequency of administration of insulin you
would prescribe to a newly diagnosed cat with diabetes mellitus. (5 marks)
c) Describe the recommendations you would make for the frequency and timing of feeding in relation to insulin dosing in cats with diabetes mellitus (10 marks).
![Page 45: Blueprinting and drafting questions Liz Norman ANZCVS 2017](https://reader035.vdocuments.net/reader035/viewer/2022081604/5899b5361a28aba11e8b58e1/html5/thumbnails/45.jpg)
Some factors that affect demand and difficulty
• Type of operation• Degree of novelty• Number of components or ideas involved• Whether resources are provided or need to be
generated by the candidate• The question wording and any images, diagrams, or
tables provided• Degree of abstraction• Response strategy – simple, stepwise, integrated• Guidance provided
![Page 46: Blueprinting and drafting questions Liz Norman ANZCVS 2017](https://reader035.vdocuments.net/reader035/viewer/2022081604/5899b5361a28aba11e8b58e1/html5/thumbnails/46.jpg)
You have been contacted by a farmer producing Pacific oysters (Crassostrea gigas) intertidally, in a bay containing a number of oyster farms. The farmer is concerned with the amount of dead shell they are seeing during the current grading. Explain how you would approach this scenario. (20 marks)
Include in your answer how the information you could gather might influence your assessment, what differential diagnoses you consider and detail how you might further investigate potential causes and what advice you would provide.
![Page 47: Blueprinting and drafting questions Liz Norman ANZCVS 2017](https://reader035.vdocuments.net/reader035/viewer/2022081604/5899b5361a28aba11e8b58e1/html5/thumbnails/47.jpg)
Pacing examinations
• Reading time: 40 words per minute• Writing time: 20 words per minute
• Marking schemes help you check the pacing