blueprinting and drafting questions liz norman anzcvs 2015

Blueprinting and drafting questions

Liz Norman

Massey University

Australian and New Zealand College of Veterinary Scientists

Examiner Workshop, 14-15 February, 2015

http://www.slideshare.net/liznorman

What is a blueprint?

"blueprint, n.". Something which acts as a plan,

model, or template

OED Online http://www.oed.com

Railway and harbour report, Vancouver, B.C.

by City of Vancouver Archives, Attribution License

https://www.flickr.com/photos/vancouver-archives/

http://creativecommons.org/licenses/by/2.0

What is a blueprint?

Specifies what’s going to be in the exam

– Content/topics – breadth of the exam

– Level - depth of the exam

Railway and harbour report, Vancouver, B.C.

by City of Vancouver Archives, Attribution License

https://www.flickr.com/photos/vancouver-archives/

http://creativecommons.org/licenses/by/2.0

Purpose of blueprinting

To document the sampling.

To show that a particular examination is representative ofall that could be examined and is sufficient.

Why we need to blueprint

• Its important that we sample representatively

from the content domain (all that it is possible to

examine)

• This is so we can extrapolate performance to the

whole content domain

Content of questions

Content

• Scope of topics

• Other scopes eg species mix

• Scope of skills and abilities required

Subject guidelines

• Represent the entire content domain (all that is

examinable).

• Specify both scope (breadth) and level (depth)

of the knowledge, skills, attitudes and

judgements required.

• For Fellowship some aspects are assessed in

the credentials documents.

Subject guidelines

The candidate will have a detailed knowledge of:

The aetiology, pathogenesis and pathophysiology of

cardiac, renal, respiratory, alimentary, musculoskeletal,

endocrine, ophthalmological and neurological organ

dysfunction in the cat and the dog.

Level of questions and

skills and abilities required

Level - depth

Levels of knowing and understanding but also thinking processes required by the discipline

• Surface – deep

• Fact recall – applied

• Blooms taxonomy

• SOLO taxonomy

Fact recall vs applied

Fact recall:Questions capable of being answered by reference to one paragraph in a text or notes (or several paragraphs for questions requiring recall of several facts)

Applied (higher order)Questions that require the use of facts or concepts, the solution of a diagnostic or physiologic problem, the perception of a relationship, or other process beyond recalling discrete fact

From: Peitzman et al. (1990). Academic Medicine, 65(9), S59-60.

create

evaluate

analyse

apply

understand

rememberRecall

Application

Blooms taxonomy

Blooms instructional verbs

• Create: compose, plan, propose, design, formulate, arrange, assemble, collect, construct, create, set-up, organise, manage, prepare.

• Evaluate: judge, appraise, evaluate, rate, compare, revise, assess, estimate

• Analyse: distinguish, analyse, differentiate, appraise, calculate, experiment, test, compare, contrast, criticise, diagram, inspect, debate, question, relate, solve, examine, categorise.

• Apply: interpret, apply, employ, use, demonstrate, dramatise, practice, illustrate, operate, schedule, sketch.

• Understand: translate, restate, discuss, describe, recognise, explain, express, identify, locate, report, review, tell

• Remember: define, repeat, record, list, recall, name, relate, underline.

SOLO taxonomy

http://pamhook.com/2012/01/20/creating-solo-taxonomy-symbols-in-many-colours/

Prestructural Question may be rephrased as the answer; almost completely

misses the point of the question.

Unistructural Able to identify, list, name, enumerate but does not describe,

explain, relate or elaborate multiple aspects of a response

Multistructural Able to list as well as describe distinct aspects of a response (such

as being able to describe aetiology, clinical features, management

of thrombotic stroke) but unable to explicitly explain causes for

observations; unable to present cause-effect relationships.

Relational Able to describe multiple aspects of a process and additionally

explain or elaborate observations into cause-effect relationships;

able to compare similarities and differences between apparently

distinct phenomena. This level is taken as suggesting that the

learner has understood.

Extended

abstract

Highly developed; able to explain mechanisms of phenomena and

apply this information to a novel context — able to develop novel

hypotheses, theories, and deduce principles; creative thinking.

Prestructural



Multistructural

Relational

Extended

abstract

Quantitative change

Qualitative change

Prestructural



Multistructural

Relational

Extended

abstract

surface

deep

Prestructural



Multistructural

Relational

Extended

abstract

recall

application

Prestructural

Unistructural Paraphrase, define, identify, count, name, recite, follow simple

instructions, calculate, reproduce, arrange, recognise, find, note,

seek, sketch, pick

Multistructural Combine, classify, structure, describe, enumerate, list, do

algorithm, apply method, account for execute, formulate, solve,

conduct, prove, complete, illustrate, express, characterise

Relational Analyse, compare, contrast, integrate, relate, explain causes,

apply theory (to its own domain), argue, implement, plan,

summarize, construct, design, interpret (some senses), structure,

conclude, substantiate, exemplify, derive, adapt

Extended

abstract

Theorise, generalise, hypothesise, predict, judge, transfer theory

(to new domain), assess, evaluate, interpret (some senses),

critically reflect, predict, criticise, reason

Doing blueprinting

Blueprinting

• Mapping to scope and level

Breadth by learning outcome

Written paper

1

Written paper

2

Oral

examination

Practical

examination

LO1 X X X

LO2 X X

LO3 X X X

LO4 X

Breadth by topic

PathophysiologyInvestigation and

diagnosis

Treatment and

management

Gastrointestinal P1Q1 P1Q1, P2Q4

Cardiovascular P1Q4 P2Q2 P2Q2

Nervous P1Q3, P2Q1

Endocrine P1Q3 P2Q3

Musculoskeletal P2Q5

Breadth across species

species number of Qs percent

small animal 15 52%

farm 5 17%

horse 4 14%

exotic 3 10%

lab 1 3%

all 1 3%

Topic 130%

Topic 230%

Topic 340%

total

Knowledge of terms 2 5 5 12

Comprehension of principles 4 3 4 11

Application of principles 3 3 3 9

Analysis of situations 3 2 5 10

Evaluation of solutions 3 2 3 8

Total questions 15 15 20 50

In this example

• Topic 1 represents 30% of the whole curriculum being assessed

so 15 of 50 questions are to address it.

• Four questions are to address comprehension of the principles

of Topic 1.

Scoring office, Michigan State University, Writing Test Items, http://scoring.msu.edu/writitem.html

Blueprints that include depth

PathophysiologyInvestigation and

diagnosis

Treatment and

management

recallhigher

orderrecall

higher

orderrecall

higher

order

Gastrointestinal P1Q1P1Q1,

P2Q4

Cardiovascular P1Q4 P2Q2 P2Q2

NervousP1Q2,

P2Q1

Endocrine P1Q3 P2Q3

Musculoskeletal P2Q5

Blueprint

Enables you to plan and check that an exam, or

set of exams, covers the content it is meant to

cover, at an appropriate level.

Without this you can’t extrapolate performance on

this exam to performance in the whole domain

= one component of validity

Controlling difficulty and demand

Should questions be difficult?

Norm-referenced vs

standards-based

grading

Should questions be difficult?

Appropriate demand

vs irrelevant difficulty

What makes questions difficult?

• Familiarity – novelty

• Complexity – components, links

• Scale – components, links

• Resources utilised – provided, generated

• Abstraction – concrete experience, concepts, ideas, principles, projection in time

• Task strategy – simple, stepwise, integrated

• Response strategy – simple, stepwise, integrated

• Guidance

Adapted from Hughes, Pollitt, & Ahmed (1998) The development of a tool for gauging the

demands of GCSE and A level exam questions. British Educational Research Association

conference, Aug 27-30 1998, Belfast.

Familiarity - novelty

Question 1:

Explain the physiological actions of insulin.

Question 2:

Explain the physiological actions of ghrelin.

Familiarity - novelty

Question 1

List 4 clinical signs of diabetes mellitus in dogs.

Question 2

List 4 clinical signs of feline hypersomatotropism.









• Guidance




Question 1:

Compare and contrast the clinical signs of

hypoadrenocorticism with those of

hyperadrenocorticism in dogs.

Question 2:


diabetes mellitus with those of

hyperadrenocorticism in dogs.

Question 3:



hyperadrenocorticism in dogs and cats.

Question 4:



hypoadrenocorticism in dogs and cats.









• Guidance




Describe the skull. (30 marks)









• Guidance




Abstraction

Discuss the concept of quality of life in terms of

biological function, “feelings” and natural

existence.

Describe how both classical conditioning and

operant conditioning are involved in cows

confidently entering the milking shed and letting

down.









• Guidance




Question 1

Discuss the use of insulin for the treatment of diabetes mellitus in cats (25 marks)

Question 2

a) Describe the advantages and disadvantages of insulin therapy for diabetes mellitus in cats (10 marks)

b) Indicate the dose and frequency of administration of insulin you would prescribe to a newly diagnosed cat with diabetes mellitus. (5 marks)

c) Describe the recommendations you would make for the frequency and timing of feeding in relation to insulin dosing in cats with diabetes mellitus (10 marks).

Describe the skull. (30 marks)









• Guidance




You have been contacted by a farmer producing Pacific oysters (Crassostrea gigas) intertidally, in a bay containing a number of oyster farms. The farmer is concerned with the amount of dead shell they are seeing during the current grading. Explain how you would approach this scenario. (20 marks)

Include in your answer how the information you could gather might influence your assessment, what differential diagnoses you consider and detail how you might further investigate potential causes and what advice you would provide.









• Guidance




What’s different about oral Qs

• Thinking on your feet

• Can’t revisit or edit

• Tests verbal communication skills as well as

knowledge

• Interactive – can prompt and redirect, can ask

candidate to explain their answer

Key points

Blueprinting

• Ensure the examination (as a whole)

representatively samples from the domain

specified for the examination

• Membership - this is all the learning outcomes

• Fellowship - this is all the learning outcomes,

except those that can only be sampled in the

credentials document

Blueprinting

• Ensure the examination (as a whole)

representatively samples from the domain

specified for the examination

– Topics

– Level

– Species

– Other things eg:

• diagnosis vs management

• principles vs application

Structuring the exam

• Ensure you follow the subject guidelines

explicitly

• Shorter vs longer Qs

• Scenario vs direct Qs

Next session’s task

Next session’s task:

• Check your blueprint

– Content

– Level

• See if you can classify your draft Qs according

to the SOLO taxonomy

• Redraft Qs if necessary

Prestructural

Unistructural Paraphrase, define, identify, count, name, recite, follow simple

instructions, calculate, reproduce, arrange, recognise, find, note,

seek, sketch, pick

Multistructural Combine, classify, structure, describe, enumerate, list, do

algorithm, apply method, account for execute, formulate, solve,

conduct, prove, complete, illustrate, express, characterise

Relational Analyse, compare, contrast, integrate, relate, explain causes,

apply theory (to its own domain), argue, implement, plan,

summarize, construct, design, interpret (some senses), structure,

conclude, substantiate, exemplify, derive, adapt

Extended

abstract

Theorise, generalise, hypothesise, predict, judge, transfer theory

(to new domain), assess, evaluate, interpret (some senses),

critically reflect, predict, criticise, reason

blueprinting and drafting questions liz norman anzcvs 2015

Education

city of vancouver archives

vs appliedfact recall

scope of skills

level of questions

harbour report

solo taxonomy http

attribution license

level depth