item writing techniques for surveys, quizzes and tests writing... · item writing techniques for...

Copyright © 1995-2014 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.

Item Writing Techniques for Surveys, Quizzes and Tests

Ivan Forward

Assessment Solutions Consultant, Questionmark

Copyright © 1995-2014 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.

• Founded in 1988

• Offices in US, UK, Canada; Distribution partners worldwide

• Assessment technologies and portal solutions that enable organisations to measure knowledge, skills, and attitudes securely for certification, regulatory compliance and improved learning outcomes.

Background

• 1,000 current enterprise customers worldwide

• Business, Government, Education, Awarding Bodies, Certification and Test Publishers

Customers

About Questionmark


• Assessment Management System

• Questionmark OnDemand (SaaS/cloud)

• Questionmark Perception (installable software)

• Multilingual

• Localized for dozens of different languages and cultures

• Global 24/7 support available

Products and Services

About Questionmark


2016 Users Conference

Slide 4

Get vital info and training on the latest assessment technologies and best practices

Network with fellow assessment and learning professionals

Learn about the Questionmark product roadmap

w w w. q u e s t i o n m a r k . c o m /g o /c o n fe r e n c e

SAVE

THE

DATE

MiamiApril 12-15

http://www.questionmark.com/go/conference


Webinar in “lecture mode” …

Slide 5

To ask questions, use the “Q&A” feature


Define “fair”, “valid”, and “reliable” and describe why they’re important when writing items

Describe the different elements that make up an item

Write a well-written stimulus

Write a well-written choice

Write an effective survey question

Objectives


Questions need to be:• Fair


Questions need to be:• Fair• Valid


Questions need to be:• Fair• Valid• Reliable


Questions need to be:

Slide 10

Well written questions essential to trustable assessment results

Reliable

•How consistently does the test measure a construct?

Valid

•Does the test is measuring what it is supposed to measure?

Fair

•Was there an unfair advantage for any given demographic group?


Reliable: dependable, repeatable, consistent

Valid: measures the specific knowledge and skills that were intended to be measured

Reliability and validity

Slide 11

Figure 1:Reliable but not Valid

Figure 2:Not reliable, not Valid

Figure 3:Reliable and Valid


Job Task Analysis

Slide 12


JTA Authoring

Slide 13


JTA Surveys to Validate Content

Slide 14


JTA in Analytics

Slide 15


JTA in Analytics

Slide 16


Items vs. Questions


Which statement is an identifying characteristic of a mixture?

Well-written stimulus:• Carefully worded – no trickery• Clear – participant knows what’s being asked• Concise – no window dressing• Avoid jargon• Positive phrasing• Use questions whenever possible

What, Who, When Where, Why and How


Well-written choices:• Same length• As short as possible• Avoid keywords• Avoid grammatical cues• Reasonable

• Logical or numerical order• Avoid negatives, especially

double negatives• Independent – no overlap• Avoid AOTA & NOTA• Correct choices evenly

distributed


Which factors impact a presidential election?a. Previous president’s performance,

weather, local economyb. Weather, local economy, national

unemployment ratec. Congress’ performance rating,

weather, national unemployment rate

d. Local economy, weather, Congress’ performance rating


Which factors impact a presidential election? Select all that apply.

Previous president’s performance

Weather Congress’ performance rating Local economy National unemployment rate


Which of the following describes a banana?a. It’s a vegetable.b. It’s yellow.c. It’s blue.d. It’s round.e. All of the above.


Which of the following describes a banana?a. It’s a vegetable.b. It’s yellow.c. It’s blue.d. It’s round.


Yes or No: When loading your snarkleblaster, should you beedlethe diggle or zix the frondle?oYesoNo


Yes or No: When loading your snarkleblaster, should you beedle the diggle?

o Yeso No

Yes or No: When loading your snarkleblaster, should you zix the frondle?

o Yeso No


When putting a child to bed for the night, you should:

a. Blast him/her with a fire hose.b. Create a calm and relaxing atmosphere.c. Parade 3 elephants through the room.d. Paint yourself purple with pink polka dots

and dance a jig.


When putting a child to bed for the night, you should:a. Give the child a cup of his/her favorite soda.b. Create a calm and relaxing atmosphere.c. Run around the house to exhaust the child.d. Turn on some music loud enough to drown out

any noise from any other part of the house.


A chimpanzee’s favorite treat is a:a. Appleb. Bananac. Oranged. Eggplant


A chimpanzee’s favorite treat is:a. An appleb. A bananac. An oranged. An eggplant


When replacing strings on a bass guitar:a.Change the pre-amp’s battery.b.Seat the thick end of the string in the

bridge, lightly tighten the string with the tuning machine in the headstock, then stretch the string before fully tightening and tuning it.

c. Set the equalization.d.Adjust the strap.


When replacing strings on a bass guitar:a.Change the pre-amp’s battery.b.Set the intonation.c. Set the equalization.d.Adjust the strap.


Which of the following is not a characteristic of a friggle in a state of grex?a.Hoomishnessb.Blue ricklesc. Excessive

sninishnessd.Lack of nongs


Which of the following is NOT a characteristic of a friggle in a state of grex?a.Hoomishnessb.Blue ricklesc. Excessive

sninishnessd.Lack of nongs


You are a Questionmark help desk technician. A customer calls to ask for advice about putting together an assessment. You demonstrate the assessment creation process in Questionmark Live. You discuss how to schedule an assessment. When the customer asks about using assessments in solving business problems, which video do you suggest?

a. “Assessments Through the Learning Process”b. “Assessments Through the A-Model”c. “Writing Good Assessment Items”d. “Engaging with Questionmark”


Which Questionmark video explains using assessments in solving business problems?

a. “Assessments Through the Learning Process”b. “Assessments Through the A-Model”c. “Writing Good Assessment Items”d. “Engaging with Questionmark”


When putting a child to bed for the night, you should:a. Give the child a cup of his/her favorite

soda.b. Create a calm and relaxing atmosphere.c. Run around the house to exhaust the

child.d. Turn on some music loud enough to

drown out any noise from any other part of the house.

A chimpanzee’s favorite treat is:a. An appleb. A bananac. An

oranged. An

eggplant

When replacing strings on a bass guitar:

a. Change the pre-amp’s battery.

b. Set the intonation.c. Set the equalization.d. Adjust the strap.

Which of the following is NOTacharacteristic of a friggle in a state of grex?a. Hoomishnessb. Blue ricklesc. Excessive

sninishnessd. Lack of nongs

Yes or No: When loading your snarkleblaster, should you beedlethe diggle?

o Yeso No

Which Questionmark video explains using assessments in solving business problems? a. “Assessments Through the Learning Process”b. “Assessments Through the A-Model”c. “Writing Good Assessment Items”d. “Engaging with Questionmark”


Survey Question Tips


List what you want to know/learn from the survey results

Write down who will use the information and for what purpose

Ensure that the output of the survey will answer the questions and meet the stakeholder needs

Then – write your questions

Deciding on what you want to know/learn from the survey

Slide 39


Objective: Determine the amount of time per week spent reading email.

How many emails do you receive per week, on average?

On average, how many hours do you spend responding to emails every week?

How long does it take to read the average email?

On average, how many hours do you spend reading emails every week?

Ask for the Data You Want!


Were you satisfied with our service? Yes

No

What if I was not really satisfied, but it wasn’t bad enough to be dissatisfied?

What if I was satisfied with the restaurant service, but very disappointed with room service?

Make Sure the Question Can Be Answered!


There was ample interaction in class and the handouts were valuable.1. Strongly disagree2. Disagree3. Agree4. Strongly agree


I routinely refuse to cooperate with my coworkers.

versus

Members of my workgroup routinely cooperate with each other.

Write Questions You Would be Willing to Answer


Lots of scales to choose from but some rules of thumb: Use the same scale for all

questions (e.g., all Likert are 1-5 Strongly Agree to Strongly Disagree)

Don’t make the scale too long as participants usually can’t differentiate

Ensure that the scale is weighted appropriately (e.g., ‘strongly agree’ is ‘5’, strongly disagree is ‘1’)

Consider not using a “cop out” N/A or unsure

Great book dedicated to this subject of scale development: DeVellis, R. F. (2003). Scale

Development: Theory and Applications Second Edition (Applied Social Research Methods).

Scale development

Slide 44


Direction, e.g.: Strongly Agree Strongly Disagree, or

Strongly Disagree Strongly Agree

Some evidence that respondents have a natural desire to want to agree with a statement If “strongly agree” is first, there may be a slight tendency to

select it without much thought

If “strongly disagree” is first, there may be more of a tendency to think before responding, and differentiate more honestly between “agree” and “strongly agree”

Rating Scales: Direction


Keep the survey short and sweet (only ask the minimum number of questions required) This will help get the best

response rate

Put the most important questions first

Ensure that the survey flows (order questions)

General survey authoring tips

Slide 46


Sometimes it takes just one word to bias a question. Avoid using inflammatory words in surveys, such as:

allege, allude, arbitrary, blame, claim, demand, error, failure, fault, ignore, ill-advised, ill-informed, incompetence, ineptness, insist, just, maintain, misinformed, must, neglected, one-sided, only, overreact, peremptory, purport, questionable, rejection, rigid, so-called, unfortunately, unilateral, unreasonable

Avoid “Red-Flag” words

Slide 47


Avoid the problem altogether.

I ride my bicycle to work:

could be:

On average, I ride my bicycle to work:

Rating Scales: Labeling

http://blog.questionmark.com/wp-content/uploads/2013/10/surveys-5-3.png

http://blog.questionmark.com/wp-content/uploads/2013/10/surveys-5-3.png


Space choices evenly.

Provide “not applicable” option if there’s any possibility the respondent may not have an answer. Make “not applicable” different in appearance.

Numbers: low to high in order.

Use the same term at each end of the scale if at all possible:“not at all shy” “very shy” instead of

“brave” “shy”

Rating Scales: In Summary


www.questionmark.com/go/whitepapers

Recommended White Papers

Slide 50

http://www.questionmark.com/whitepapers


Keep in Touch

Slide 51

blog.questionmark.com

twitter.com/questionmark

youtube.com/questionmarkvideo

https://www.linkedin.com/grp/home?gid=1931441

facebook.com/questionmarkgroup

plus.google.com/+QuestionmarkCorp/

https://plus.google.com/+QuestionmarkCorp


2016 Users Conference

Slide 52

Get vital info and training on the latest assessment technologies and best practices

Network with fellow assessment and learning professionals

Learn about the Questionmark product roadmap

w w w. q u e s t i o n m a r k . c o m /g o /c o n fe r e n c e

SAVE

THE

DATE

MiamiApril 12-15

http://www.questionmark.com/go/conference

Copyright © 1995-2014 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.

Thank you for attending!

Ivan Forward

Assessment Solutions Consultant, Questionmark

item writing techniques for surveys, quizzes and tests writing... · item writing techniques for...

Documents