Download - Writing test questions
Writing Assessment Questions
12/14/09
Melissa Chiaramonti
Questions Improve the Learning Process
Slide 2
Know
led
ge/S
kill
Time
Reduce
Forg
ett
ing
Reduce
Forg
ett
ing
Cre
ate
s In
trig
ue
Str
ength
en
s M
em
ory
Questions….
Slide 3
Dr. Will Thalheimer research:
Learning Benefits from Questions Learning Benefit
Min Max
Pre-questions (to focus attention) 5% 40%
RepetitionsInitial
Subsequent30%15%
110%40%
Feedback 15% 50%
Retrieval Practice 30% 100%
Questions Spaced Over Time 5% 40%
100% 380%
Guiding Principles for Evaluation
1. Match the right question type to the right skill
2. Carefully construct your questions
3. Provide feedback to guide learning
4. Organize questions in a centralized item bank
5. Randomize choices and questions
6. Align delivery environment to assessment stakes (*)
7. Accommodate participant needs: blended delivery (*)
8. Analyze results and sharing with stakeholders
Questions to Ask Yourself
• What objectives will (should) I be assessing?
• What types of items will be included in the test? – 2i– 2a– 2b– 2c
• How long will the test be in terms of time and number of items?
Common question types
• True/False / Yes/No - the participant selects one of two potential choices
• Multiple choice - the participant selects one choice from a group of possible answers
• Multiple response - Similar to multiple choice except the participant can select one or more choices
• Numeric questions - a participant is prompted to enter a numeric value
• Fill in the blanks - participant is presented with a statement where one or more words are missing
Slide 6
Question Types verses Cognitive Levels of Learning
KnowledgeComprehension
Application AnalysisSynthesis Evaluation
2i, 2a 2b 2c
Multiple Choice (MC)True/False (TF)MatchingCompletion Short Answer
MCShort AnswerProblems EssayPerformance
MCShort AnswerEssay
Planning
1. Outline learning objectives or major concepts to be covered by the assessments• Test should be representative of objectives and material covered• Major Learner complaint: Tests don’t fairly cover the material that was supposed to be
canvassed on the test
2. Create a test blueprint – Currently working on
3. Create questions based on blueprint – Why we are here today– Match the question type with the appropriate level of learning
4. Organize questions and/or ideas by item types
5. Eliminate similar questions and organize questions logically
6. Validate the tests
7. Analyze the results
Don’ts
Avoid… Why
Avoid negatives, especially double negatives.
Can make questions confusing to the participant.
Avoid "All of the above" and use of "None of the above" with caution.
Make it much easier for a participant to eliminate obvious choices and “guess” an answer
Avoid vague pronouns (e.g. it, they). Can make questions ambiguous and confusing to the participant.
Avoid unnecessary modifiers, especially absolutes (e.g. always, never, all, none).
“Always” and “Never”, etc., can make it easier guess an answer
Slide 9
Don’ts
Avoid… Why
Avoid providing cues to correct answer in the stem.
Giving the answer away means the question will not effectively measure the skill/knowledge it was intended to measure
Avoid providing clues to the answer of one question in another question.
Giving the answer away means the question will not effectively measure the skill/knowledge it was intended to measure
Avoid conflicting alternatives. Can make questions ambiguous and confusing to the participant.
Slide 10
Do’s
Do Why
Keep stems and statements as short as possible and use clear, concise language.
Provides clarity to what is being asked; minimizes ambiguity
Use questions whenever possible (What, Who, When, Where, Why and How)
Provides clarity to what is being asked; minimizes ambiguity
Maintain grammatical consistency to avoid cueing (Parallel Structure).
Helps to avoid providing cues to the correct answer
List choices in a logical order. Helps to avoid providing cues to the correct answerOptional – in many cases you may wish to shuffle choices.
Slide 11
OVERVIEW OF THE QUESTION TYPES
Multiple Choice Items • Advantages:
– Extremely versatile-can measure the higher level mental processes (application, analysis, synthesis and evaluation)
– A compromise between a short answer/essay and T/F item – Can cover a wide range of content can be sampled by one test
• Disadvantages – Difficult to construct plausible alternative responses
14
Multiple Choice Question Construction
Stem: A 65-year-old man has difficulty rising from a seated position and straightening his trunk, but he has no difficulty flexing his leg.
Lead-in: Which of the following muscles is most likely to have been injured?
Options: A. Gluteus maximus* Answer
B. Gluteus minimus
C. Hamstrings
D. Iliopsoas
E. Obturator internus Distractors
15
Have a Distinctive Shape
Stem
Lead In
Option A (distractor)
Option B (answer)
Option C (distractor)
Option D (distractor)
Option E (distractor)
16
Poorly Shaped MC Question
Short stem & lead-in
a. Long option
b. Long option
c. Long option
d. Long option
Consider this question:
• Which of the following are not disadvantages of hybrid cars when compared to similar gasoline powered vehicles? (select all that apply)
A. Lower sticker priceB. Better fuel economyC. Lower carbon emissionsD. Better audio systems
Slide 17
Consider this question:
• Which of the following are not disadvantages of hybrid cars when compared to similar gasoline powered vehicles? (select all that apply)
A. Lower sticker priceB. Better fuel economyC. Lower carbon emissionsD. Better audio systems
Slide 18
Double Negative
Better wording...
• Which of the following are advantages of hybrid cars when compared to similar gasoline powered vehicles? (select all that apply)
A. Lower sticker priceB. Better fuel economyC. Lower carbon emissionD. Better audio systems
Slide 19
Consider this question:
• The President and Vice President of the United States are chosen by the Electoral College.
A. TrueB. False
Slide 20
Consider this question:
• “The President and Vice President of the United States are chosen by the Electoral College.”– Ambiguous question because...
• This is true to some degree, BUT...• You could make a strong argument for “False”, making the point that the
Electoral College is mainly a formality because electors are generally “pledged” (or “instructed who to vote for”) based on the results of the popular vote in their state.
Slide 21
Better wording:
• “The Electoral College meets every four years to cast the official votes for President and Vice President of the United States.”– There is no ambiguity here.
Slide 22
1. Write the stem first: a) Be sure the stem asks a clear question b) Stems phrased as questions are usually easier to write c) Stems should not contain a lot of irrelevant info.d) Be sure the stem is grammatically correcte) Avoid negatively stated stems (double negative risk)
How to Write Multiple Choice Questions
How to Write Multiple Choice Questions
2. Writing the correct response 1. Use same terms/reading level2. Avoid too many qualifiers 3. Assign a random position in the answer sequence
3. Read the stem and correct response together
4. Generate the distractors/alternative responses
Multiple Choice Items
• Other Tips for Constructing MC Items:– Items should have 3-4 alternatives. – Stem should present a single, clearly formulated problem– Simple, understandable, exclude extraneous words from
both stem and alternatives – Include in the stem any word that are repeated in each
response – Avoid all of the above (can answer based on partial
information)– Avoid none of the above
26
Consider this question:
• The primary purpose of the stam is to remove the– carm– denton– menace– stam bar
27
Consider this question:
• The primary purpose of the stam is to remove the– carm– denton– menace– stam bar
“Word repeats” in stem and answer
28
Consider this question:
• Which of the following pairs has won the greatest number of Abby awards?– Jones and Smith– Smith and Taylor– Smith and White– White and Allen
29
Consider this question:
• Which of the following pairs has won the greatest number of Abby awards?– Jones and Smith– Smith and Taylor– Smith and White– White and Allen
Convergence: correct answer has the most in common with other choices
30
Consider this question:
• How many pounds of pressure are exerted by a callam?
– 2.6– 150– 260– 2600
31
Consider this question:
• How many pounds of pressure are exerted by a callam?
– 2.6– 150– 260– 2600
Convergence: Correct answer has the most in common with other choices
32
Consider this question:
• The stanon is aided by a– instel– immon– octal– port
33
Consider this question:
• The stanon is aided by a– instel– immon– octal– port
Grammatical cues: only “port” follows grammatically from the stem
34
Consider this question:
• The stanon frequently overheats because– all grestels are bilious– no immon are directly fectitious– octals are usually casable– ports are always critical
35
• The stanon frequently overheats because– all grestels are bilious– no immon are directly fectitious– octals are usually casable– ports are always critical
Absolute terms: never found in the correct answer
Consider this question:
36
Consider this question:
• Stammation normally occurs when the– anstels rupture– immon falls and the denton is in place– octal rotates easily– ports pass over the carm
37
Consider this question:
• Stammation normally occurs when the– anstels rupture– immon falls and the denton is in place– octal rotates easily– ports pass over the carm
Longest answer is usually the correct answer
True-False Test Quedstions
• Best suited for testing 3 kinds of info.: • Knowledge level learning • Understanding of misconceptions• When there are two logical responses
• Advantages: – Sample a large amount of learning per unit of Learner
testing time
• Disadvantages: – Tends to be very easy– 50-50 chance of guessing – Tends to be low in reliability
Tips for Constructing True/False Items
–Tips for constructing True-False Items• Avoid double negatives• Avoid long or complex sentences• Specific determiners (always, never, only, etc.) should be
used with caution• Include only one central idea in each statement• Avoid emphasizing the trivial• Exact quantitative (two, three, four) language is better
than qualitative (some, few, many) • Avoid a pattern of answers
Short-Answer Items
• Two Types: (Question and Incomplete Statement)
• Advantages: – Easy to construct– Excellent format for measuring who, what, when, and where info.– Guessing in minimized– Student must know the material- rather than simply recognize the
answer
• Disadvantages: – Grading can be time consuming/difficult– More than one answer can be correct
Short Answer Items
• Tips for Constructing Short Answer Items – Better to supply the term and require a definition– For numerical answers, indicate the degree of precision
expected and the units in which they are to be expressed.– Use direct questions rather than incomplete statements– Try to phrase items so that there is only one possible
correct response– When incomplete statements are used, do not use more
than one blank within an item.
Essay Questions
Types of Essay Questions
• Extended Response Question– Great deal of latitude on how to respond to a question. – Example: Discuss essay and multiple-choice type tests.
• Restricted Response Question– More specific, easier to score, improved reliability and validity– Example: Compare and contrast the relative advantages of
disadvantages of essay and multiple choice tests with respect to: reliability, validity, objectivity, & usability.
Essay Items
• Advantages: – Measures higher learning levels (synthesis, evaluation) and
is easier to construct than an objective test item– Learners are less likely to answer an essay question by
guessing– Require superior study methods– Offer Learners an opportunity to demonstrate their abilities
to: • Organize knowledge• Express opinions• Foster creativity
Essay/Scenario Items
• May limit the sampling of material covered• Tends to reduce validity of the test
• Disadvantages– Subjective unreliable nature of scoring
• “halo effect” – good or bad student’s previous level of performance• Written expression• Handwriting legibility• Grammatical and spelling errors
• Time Consuming
Essay/Scenario Items • Give Learners a clear idea of the scope & direction intended
for the answer– Might help to start the question with the description of the required
behavior (e.g., compare, analyze)
• Appropriate language level for Learners
• Construct questions that require Learners to demonstrate a command of background info, but do not simply repeat that info.
• If question calls for an opinion, be sure that the emphasis is not on the opinion but on the way its presented or argued.
• Use a larger number of shorter, more specific questions rather than one or two longer questions so that more information can be assessed.
Essay/Scenario Items • You might
– Give Learners a pair of sample answers to a question of the type you will give on the test.
– Sketch out a rubric (grading scheme) for each question before reading the papers OR randomly select a few to read and make up the grading scheme based on those answers
– Give Learners a writing rubric – Detach identifying information and use code numbers instead to avoid
letting personality factors influence you. – After grading all the papers on one item, reread the first few to make
sure you maintained consistent standards – Be clear to student the extend to which factors other than content (e.g.,
grammar, handwriting, etc.) will influence the grade.
Essay Questions
• Tips for constructing Essay Questions – Provide reasonable time limits for each question
• “thinking and writing time” – Avoid permitting Learners a choice of questions
• Will not necessarily get a representative sample of student achievement. Only be requiring all Learners to answer all questions can their achievement be compared
– A definite task should be put forth to the student• Critical words: compare, contrast, analyze, evaluate, etc.
Scoring Essay Items
• Write an outline of the key points (use outline to design a rubric)
• Determine how many points are to be assigned to the question as a whole and to the various parts within it.
• If possible, score the test without knowledge of the student’s name – Face Sheet
• Score all of the answers to one question before proceeding to the next question – Consistent standard
Questions?