texas tech university health sciences center school of pharmacy krystal k. haase, pharm.d., fccp,...

T E X A S T E C H U N I V E R S I T Y H E A LT H S C I E N C E S C E N T E R S C H O O L O F P H A R M A C Y

KRYSTAL K. HAASE, PHARM.D., FCCP, BCPSASSOCIATE PROFESSOR

BEYOND MULTIPLE CHOICE

QUESTIONS

OBJECTIVES

• Describe the pro’s and con’s of using different question formats other than multiple choice.• Identify and resolve common problems

when constructing open-ended questions.• Develop standardized grading procedures

for open-ended questions.• Discuss exam length and other challenges

when using open-ended question formats.

ASKING THE RIGHT QUESTION

REVIEW

Blo

om

’s

Evaluation

Synthesis

Analysis

Application

Comprehension

Knowledge Low

High

QUESTION TYPES

Selection Response

Construction Response

Recognition Recall

QUESTION TYPES

Selection Response

True / False

Matching

Multiple Choice

Construction Response

Fill in the Blank

Short Answer

Essay

TRUE / FALSE

• Pro’s• easy to write• easy to score

• Con’s• limited ability to assess mastery• high probability of guessing

• Best Use: • Dichotomous, factual info

Blo

om

’s

Evaluation

Synthesis

Analysis

Application

Comprehension

Knowledge

MATCHING

• Pro’s• Can assess a lot of info in a confined space• Fairly low probability of guessing

• Con’s• Assess recognition not recall

• Best Use: • Knowledge recall• Add additional “distractor” items

to increase rigor Blo

om

’s

Evaluation

Synthesis

Analysis

Application

Comprehension

Knowledge

MULTIPLE CHOICE

• Pro’s• Very versatile• Easy to score

• Con’s• More challenging to write• Assess recognition over recall

• Best Use: • Factual, conceptual, or

procedural information Blo

om

’s

Evaluation

Synthesis

Analysis

Application

Comprehension

Knowledge

SHORT-ANSWER

• Pro’s• Assess unassisted recall• Relatively easy to write

• Con’s• Only useful if you can give a

short answer.• Must be worded carefully to

avoid scoring problems

• Best Use: • Assessing information that you

expect to be memorizedB

loom

’s

Evaluation

Synthesis

Analysis

Application

Comprehension

Knowledge

ESSAY

• Pro’s• Can test higher complex objectives• Can test process / reasoning• Realistic tasks

• Con’s• Take longer to answer• Hard to grade fairly, consistently• Take longer to grade

• Best Use: • Assessing highest level objectives

Blo

om

’s

Evaluation

Synthesis

Analysis

Application

Comprehension

Knowledge

www.siop.org/workplace/employment%20testing/testformats.aspx

SUMMARY

• Each question type has pro’s and con’s• Question type should be guided by the

learning objective to be assessed.• Limited options for assessing higher

taxonomy

CONSTRUCTING OPEN-ENDED QUESTIONS

WELL-DEVELOPED ESSAY QUESTIONS

• Mirror well-defined learning objectives• Assess most appropriate content types• Require content recall, evaluation, and

reasoning • Are clearly written• Provide boundaries• Have well-defined grading criteria

http://testing.byu.edu/info/handbooks/WritingEffectiveEssayQuestions.pdf

APPROPRIATE CONTENT

• Content that justifies high-level mastery• Construction, higher-order taxonomy• Analysis (analyze, compare, contrast, interpret)• Evaluation (evaluate, explain, justify)• Synthesis (develop, construct, modify)

• Complex, multi-step thought processes• Simulation of real-world processes• “given a patient-case scenario”

If content / processes can be assessed by methods other than essay questions, they

probably should.

KEY ELEMENTS

• An ideal essay question requires students to: • Recall facts• Make an evaluative judgment or develop a

novel solution• Explain reasoning behind response

• The question should include:• Task• Problem situation

WRITING FOR CLARITY

• Ensure your question requires higher-order thinking• Make sure the task is defined and focused• Make sure the problem situation includes

adequate detail

SETTING BOUNDARIES

• Increasing structure prevents• grading problems• bluffing

• Avoid indeterminate questions• Students can redefine and answer with info the

know well

• Give time / space limits• Establish rules for answers

ESTABLISHING GRADING CRITERIA

• Must have specific criteria for grading identified a priori• Create a model answer• Assign point values• Peer review

• Identify essentials in response• Determine whether partial credit is

allowable and how will be awarded.• Grade blinded

DIFFERENT GRADING APPROACHES

• Comprehension / Understanding• Screen responses for key elements• Assign points for each element present• Scores for presence of content only (potential

bluffing)

• Reasoning / Complex Processes• Must assess complete response (time consuming)• Consider a rubric approach to limit subjectivity

• Problem – Solution• Encourage requiring students to show work• Can grade multiple steps in process

GRADING TIPS

• Grading scheme should be easily interpreted by other graders • Self-explanatory• Point values that can easily be tallied

• When multiple graders, encourage frequent communication and comparison of results• Before grading, screen a sample of

responses for consistency

OTHER ISSUES

• Exam length• Addressing unidentified “correct” answers• Partial credit or all-or-none• Practice examples• Multiple examples• Should not be the same questions as on exam