1 blueprinting

TO APPLY PAGE STYLESRIGHT CLICK YOUR PAGE>LAYOUT

SCHOOL OF ANIMAL AND VETERINARY SCIENCES

Blueprinting and drafting questions

Liz NormanMassey University

Sharanne Raidal, BVSc, PhD, FANZCVS, GradDipEd

http

://co

nsul

tatu

torb

log.

files.

word

pres

s.co

m/2

010/

06/e

xam

stre

ss.g

if

Australian and New Zealand College of Veterinary ScientistsExaminer Workshop, 6 – 7th February 2016



Overview

• Blueprinting and drafting questions• Grading criteria and marking schemes• Clarity, timing; oral examinations

http://www.slidehare.net/liznorman

http://toonclips.com/600/4997.jpg



What is a blueprint?

blueprint, n. something which acts as a planmodel ortemplate

OED Online http://www.oed.com https://lnocc.org/wp-content/uploads/2013/01/blueprint1a.jpg



What is a blueprint?

• Specifies what’ going to be in the exam Content / topics (breadth) Level (depth)

https://lnocc.org/wp-content/uploads/2013/01/blueprint1a.jpg



Purpose of blueprinting

• To document sampling• To show that your

examination is representative of all that could be examined and is sufficient

Content domain

Exam 2

Oral

Essay

Credentials domain

Exam 1

Prac

Practice domain



Why blueprint?

• To show that we have a representative sample

• To permit extrapolation to content (practice) domain

• To plan how each component assessed

Content domain

Exam 2

Oral

Essay

Credentials domain

Exam 1

Prac

Practice domain



Content of questions

• Scope of topics• Other relevant considerations - eg. species

body systems• Scope of skills and abilities



Subject guidelines

• Represent the entire content domain (ie. they define all that is examinable)

• Specify both scope (breadth) and level (depth) of the knowledge, skills, attitudes and judgements required

• For Fellowship, some aspects are assessed in the credentials document (and process and training)



Subject guidelines

The candidate will have a detailed knowledge of:

The aetiology, pathogenesis and pathophysiology of cardiac, renal, respiratory, alimentary, musculoskeletal, endocrine, ophthalmological and neurological organ dysfunction in the cat and the dog.



Level of questions; skills & abilities required

Level – depth Levels of knowing and understandingThinking processes required by the discipline

•Surface vs deep•Fact recall vs application•Blooms taxonomy•SOLO taxonomy




Fact recall:Questions capable of being answered by reference to one paragraph in a text or notes (or several paragraphs for questions requiring several facts).

Applied (higher order):Questions that require the use of facts or concepts, the solution of a diagnostic of physiologic problem, the perception of a relationship or other process(es) beyond recalling discrete fact.

From: Peitzman et al. (1990) Academic Medicine , 65(9): S59-60




Bloom’s taxonomy

http://julietovar.edublogs.org/files/2011/05/blooms-taxonomy-1k4snjn.JPG




Bloom’s taxonomy

http://www.psia-nw.org/wp-content/uploads/Blooms_Taxonomy.jpg

Recall

Application



Level of questions; skills & abilities requiredBloom’s instructional verbsCreate: compose, plan, propose, design, formulate, arrange, assemble, collect, construct, create, set-up, organise, manage, prepare.

Evaluate: judge, appraise, evaluate, rate, compare, revise, assess, estimate.

Analyse: distinguish, analyse, differentiate, appraise, calculate, experiment, test, compare, contrast, criticise, diagram, inspect, debate, question, relate, solve, examine, categorise.

Apply: interpret, apply, employ, use, demonstrate, dramatise, practice, illustrate, operate, schedule, sketch.

Understand: translate, restate, discuss, describe, recognise, explain, express, identify, locate, report, review, tell.

Remember: define, repeat, record, list, state, recall, name. Recall

Apply



http://www.pescholar.com/wp-content/uploads/2013/03/Solo-SS.png




SOLO taxonomy

http://litre.ncsu.edu/solowrittencomposite.gif



Level of questions skills & abilities required

SOLO taxonomy

http://www.mbms.org.uk/faculties/science/assets/SOLO/solo-taxonomy-explained-full.jpg



SOLO taxonomy

Prestructural Question may be rephrased at the answer; almost completely misses the point of the question

Unistructural Able to identify, list, name, enumerate, but does not describe, explain, relate or elaborate multiple aspects of a response

Multistructural Able to list as well as describe distinct aspects of a response (such as being able to describe aetiology, clinical features, management) but unable to explicitly explain causes for observations; unable to present cause-effect relationships

Relational Able to describe multiple aspects of a process and additionally explain or elaborate observations into cause-effect relationships; able to compare similarities and differences between apparently distinct phenomena; this level is taken as suggestion the learner has understood

Extended abstract Highly developed; able to explain mechanisms of phenomena and apply this information to a novel context – able to develop novel hypotheses, theories and deduce principles; creative thinking



SOLO taxonomy






Quantitative change

Qualitative change



SOLO taxonomy






Deep

Surface



SOLO taxonomy






Application

Recall



SOLO taxonomy

Prestructural

Unistructural Paraphrase, define, identify, count, name, recite, follow simple instructions, calculate, reproduce, arrange, recognise

Multistructural Combine, classify, describe, enumerate, list, do algorithm, apply method, account for, execute, formulate, solve, conduct, prove, complete, characterise

RelationalAnalyse, compare, contrast, integrate, relate, explain causes, apply theory, argue, implement, plan, summarise, construct, design, interpret (some senses), structure, conclude, substantiate, exemplify, derive, adapt

Extended abstract Theorise, generalise, hypothesise, predict, judge, transfer theory (to new domain), assess, evaluate, interpret (some senses), critically reflect, predict, criticise, critique, reason




Skills and abilities

•What is contained in subject guidelines?•Skills beyond cognitive?•Recognition / interpretation?

•Communication?

- diagnostic images- cytology / pathology- reports - statistics



Blueprinting - process

• Mapping to scope and level




• Breadth by learning outcome

Written paper 1

Written paper 2

Practical examination

Oral examination

LO1 x x x

LO2 x x

LO3 x x x

LO4 x




• Breadth by topic

Patho-physiology

Investigation & diagnosis

Treatment & management

Gastrointestinal P1Q1 P1Q1, P2Q4

Cardiovascular P1Q4 P2Q2 P2Q2

Nervous P1Q3, P2Q1

Endocrine P1Q3 P2Q3

Musculoskeletal P2Q5




• Breadth by topic and depthPatho-

physiologyInvestigation &

diagnosisTreatment & management

recall higher recall higher recall higher

Gastrointestinal P1Q1 P1Q1 P2Q4

Cardiovascular P1Q4 P2Q2 P2Q2

Nervous P1Q3, P2Q1

Endocrine P1Q3 P2Q3

Musculoskeletal P2Q5




• Breadth by species

Species Number of Qs Percent

Small animal 15 52%

Farm animal 4 14%

Horse 5 17%

Exotic 3 10%

Lab 1 3%

All 1 3%




• Content coverage x depthTopic 1

30%Topic 2

30%Topic 3

40% Total

Knowledge of terms 2 5 5 12

Comprehension of principles 4 3 4 11

Application of principles 3 3 3 9

Analysis of situations 3 2 5 10

Evaluation of situations 3 2 3 8

Total questions 15 15 20 50

http://scoring.msu.edu/written.html



Blueprinting

Enables you to plan and check that your exam, or set of exams, covers the content (and skills) it is meant to cover, at an appropriate level.

Without this you cannot extrapolate performance on this exam to performance in the whole domain.

Validity



Controlling difficulty and demand

Should questions be difficult?

Norm-referenced vsstandards-based grading




Should questions be difficult?

Appropriate demandvs irrelevant difficulty

Practice domain

Content domain




What makes questions difficult?

•Familiarity / novelty•Complexity – components, links•Scale – components, links•Resources utilised (provided, generated)•Abstraction – concrete experience, concepts, principles•Task strategy – simple, stepwise, integrated•Guidance

Adapted from Hughes et al (1998) The development of a tool for gauging demands of GCSE and A level exam questions. Brit Ed Res Assoc Conference.




Familiarity / novelty

Question 1Explain the physiological actions of insulin.

Question 2Explain the physiological actions of ghrelin.




Familiarity / novelty

Question 1List four (4) clinical signs of diabetes mellitus in dogs.

Question 2List four (4) clinical signs of feline hypersomatotropism.




Complexity, scale – components, links

Question 1Compare and contrast the clinical signs of hypoadrenocorticism with those of hyperadrenocorticism in dogs.

Question 2Compare and contrast the clinical signs of diabetes mellitus with those of hyperadrenocorticism in dogs.




Complexity, scale – components, links

Question 1Compare and contrast the clinical signs of diabetes mellitus with those of hyperadrenocorticism in dogs and cats.

Question 2Compare and contrast the clinical signs of diabetes mellitus with those of hypoadrenocorticism in dogs and cats.




Resources utilised – provided, generated

Describe the skull.(30 marks)




Abstraction – concrete experience, concepts, principles

Discuss the concept of quality of life in terms of biological function, “feelings” and natural existence.

Describe how both classical conditioning and operant conditioning are involved in cows confidently entering the milking shed and letting down.




Task and response strategies – simple, stepwise, integrated

Question 1Discuss the use of insulin for the treatment of diabetes mellitus in cats.

(25 marks)




Task and response strategies – simple, stepwise, integrated

Question 2a)Describe the advantages and disadvantages of insulin therapy for diabetes mellitus in cats. (10 marks)b)Indicate the dose and frequency of administration of insulin you would prescribe to a newly diagnosed cat with diabetes mellitus. (5 marks)c)Describe the recommendations you would make for the frequency and timing of feeding in relation to insulin dosing in cats with diabetes mellitus. (10 marks)




Task and response strategies – simple, stepwise, integratedGuidance

Describe the skull. (30 marks)





You are presented with a 7 year old male Warmblood gelding with exercise intolerance. You auscultate an arrhythmia with a variable pulse and order an ECG. Define the type and give a specific name for the arrhythmia demonstrated in the rhythm strip below. (2 marks)





You have been contracted by a farmer producing Pacific oysters (Crassostrea gigas) intertidally, in a bay containing a number of oyster farms. The farmer is concerned with the amount of dead shell they are seeing during the current grading. Explain how you would approach this scenario. (20 marks)

Include in your answer how the information you could gather might influence your assessment, what differential diagnoses you consider, and detail how you might further investigate potential causes, and what advice you would provide.



What’s different about oral questions?

• Thinking on your feet• Can’t revisit or edit• Tests verbal communication skills (as well as

knowledge)• Interactive – can prompt and redirect, can ask

candidate to explain their answer



Key points



Blueprinting

• Ensure the examination (as a whole) representatively samples from the domain specified for the examination

• Membership – this is all the learning outcomes

• Fellowship – this is all the learning outcomes, except those that can only be sampled in the credentials document



Blueprinting

• Ensure the examination (as a whole) representatively samples from the domain specified for the examination

• Consider - topics- level- species, body systems, modalities- skills - other things (eg. Dx vs mgt, principles vs application)



Structuring the exam

• Ensure you follow subject guidelines explicitly• Shorter vs longer questions• Scenario vs direct questions• Difficulty, guidance



Next session’s task:

• Check your blueprint - content- level

• Classify your draft Qs according to SOLO taxonomy

• Redraft Qs if necessary



SOLO taxonomy

Prestructural

Unistructural Paraphrase, define, identify, count, name, recite, follow simple instructions, calculate, reproduce, arrange, recognise

Multistructural Combine, classify, describe, enumerate, list, do algorithm, apply method, account for, execute, formulate, solve, conduct, prove, complete, characterise

RelationalAnalyse, compare, contrast, integrate, relate, explain causes, apply theory, argue, implement, plan, summarise, construct, design, interpret (some senses), structure, conclude, substantiate, exemplify, derive, adapt

Extended abstract Theorise, generalise, hypothesise, predict, judge, transfer theory (to new domain), assess, evaluate, interpret (some senses), critically reflect, predict, criticise, critique, reason

1 blueprinting

Education