11

QuestionnairesQuestionnaires

Topic PresentationTopic Presentation

Julie StromerJulie Stromer

October 20, 2004October 20, 2004

22

ContentsContents What is a questionnaire?What is a questionnaire? Who invented the questionnaire?Who invented the questionnaire? When to use questionnairesWhen to use questionnaires Information typesInformation types Question typesQuestion types

– Likert ScalesLikert Scales Advantages / Disadvantages of QuestionnairesAdvantages / Disadvantages of Questionnaires Interviews – the alternative to QuestionnairesInterviews – the alternative to Questionnaires The debate: how hard it is to write your own The debate: how hard it is to write your own

questionnairequestionnaire Guidelines for questionnaire writingGuidelines for questionnaire writing Questionnaire writing – by exampleQuestionnaire writing – by example

33

What is a Questionnaire?What is a Questionnaire?

a set of questions on a topic that can be a set of questions on a topic that can be asked of a potentially large group of asked of a potentially large group of respondents. respondents.

Answers can be analyzed and conclusions Answers can be analyzed and conclusions drawn from the responsesdrawn from the responses

Specifically for HCI studies, questionnaires Specifically for HCI studies, questionnaires are meant to collect information from are meant to collect information from study participants regarding:study participants regarding:– BackgroundsBackgrounds– Opinions about different aspects of tasksOpinions about different aspects of tasks– General usability of the product (e.g. interface)General usability of the product (e.g. interface)

44

Who Invented the Who Invented the Questionnaire?Questionnaire?

Francis Galton (1822-Francis Galton (1822-1911)1911)

Was the first cousin of Was the first cousin of Charles Darwin. Charles Darwin.

Explorer, Meteorologist, Explorer, Meteorologist, Geneticist, Psychologist, Geneticist, Psychologist, Statistician, Statistician, Anthropologist, Anthropologist, Criminologist, Eugenicist.Criminologist, Eugenicist.

55

When to use QuestionnairesWhen to use Questionnaires Resources and money are limitedResources and money are limited Participant privacy is necessaryParticipant privacy is necessary To complement information obtained by other To complement information obtained by other

methods (e.g. experimental methods) methods (e.g. experimental methods) Early definition phase of computer system designEarly definition phase of computer system design

– What should the new system be able to do? (but NOT What should the new system be able to do? (but NOT the design of it)the design of it)

During usability study of interface prototypeDuring usability study of interface prototype– May indicate eventual acceptance of systemMay indicate eventual acceptance of system

After system deliveryAfter system delivery– How well system works, what needs to be changedHow well system works, what needs to be changed

66

General Layout GuidelinesGeneral Layout Guidelines

Set context for respondentSet context for respondent– situation, topic, or set of circumstances situation, topic, or set of circumstances

Only ask for information you will directly useOnly ask for information you will directly use Keep the questionnaire as short as possible Keep the questionnaire as short as possible

– Don’t bore or frustrate the respondentDon’t bore or frustrate the respondent Have an underlying reason for every questionHave an underlying reason for every question Group topic areas together to keep the Group topic areas together to keep the

respondent focusedrespondent focused

77

Information TypesInformation Types

FactualFactual– E.g. age group, computer experienceE.g. age group, computer experience

OpinionsOpinions– Respondent’s thoughts and ideasRespondent’s thoughts and ideas– Generally never wrongGenerally never wrong

AttitudesAttitudes– How do they feel about the system?How do they feel about the system?

88

Question TypesQuestion Types

Closed-ended itemsClosed-ended items– Multiple choiceMultiple choice– Rating ScaleRating Scale

Open-ended itemsOpen-ended items

A B C D E

1 2 3 4 5

99

Closed-ended QuestionsClosed-ended Questions

Very particular finite answer Very particular finite answer expected expected

Easier to tabulate and analyzeEasier to tabulate and analyze– Saves time and moneySaves time and money

Two types:Two types:1)1) Multiple ChoiceMultiple Choice

2)2) Rating scalesRating scales

1010

Rating ScalesRating Scales

Questions or statements with a Questions or statements with a numerical numerical valuevalue assigned to the alternatives assigned to the alternatives

Note: the numerical value is internal, and Note: the numerical value is internal, and is usually not presented to the respondentis usually not presented to the respondent

Added advantage: traditional statistics Added advantage: traditional statistics used to analyze the responses used to analyze the responses

Typically use a Typically use a Likert ScaleLikert Scale

1111

Likert ScalesLikert Scales

Characterized by an odd number of choicesCharacterized by an odd number of choices– Odd to provide a clear number of choicesOdd to provide a clear number of choices

Two types:Two types:

StrongelStrongely y DisagreDisagreee

DisagreDisagreee

Neither Neither Agree Agree nor nor DisagreDisagreee

AgreeAgree StrongelStrongely Agreey Agree

ContinuousContinuous

DiscreteDiscrete

X

X

Continuous better suited to most statisticsContinuous better suited to most statistics– Distribution assumptionsDistribution assumptions

1212

Open-ended ItemsOpen-ended Items Collected information is unrestrictedCollected information is unrestricted Much like an interviewMuch like an interview Typically used at the questionnaire end Typically used at the questionnaire end Problems:Problems:

– Answers cannot be easily used in statistical Answers cannot be easily used in statistical analysesanalyses

– Answers have to be hand-readAnswers have to be hand-read– Many readers means the answers are subject Many readers means the answers are subject

to the reader’s interpretationsto the reader’s interpretations– Answers require more thought and time on the Answers require more thought and time on the

part of the respondentpart of the respondent e.g. dishonesty favoring brevitye.g. dishonesty favoring brevity

1313

Advantages of Advantages of QuestionnairesQuestionnaires

Anonymity elicits more honest responses Anonymity elicits more honest responses

Less expensive than other meansLess expensive than other means– a questionnaire can be administered to a questionnaire can be administered to

many people at oncemany people at once Feedback is from the respondent’s point Feedback is from the respondent’s point

of viewof view BUTBUT: If conducted orally, as in an : If conducted orally, as in an

interview, these advantages are limited.interview, these advantages are limited.

1414

Disadvantages of Disadvantages of QuestionnairesQuestionnaires

Additional information and opinions cannot Additional information and opinions cannot be discovered if questions are on paperbe discovered if questions are on paper

– People will write as little as possible, even if People will write as little as possible, even if they have more to say. they have more to say.

Typically designed to fit a wide range of Typically designed to fit a wide range of applicable situationsapplicable situations

– Specifics may be missed Specifics may be missed Often used by themselvesOften used by themselves

– Cannot gather all the information requiredCannot gather all the information required– Are quite subjectiveAre quite subjective– Best used in combination with objective Best used in combination with objective

methods (e.g. laboratory experiment, field methods (e.g. laboratory experiment, field study, etc.). study, etc.).

1515

Interviews – more flexible?Interviews – more flexible?

QuestionnairesQuestionnaires A printed form of questions is fixed!A printed form of questions is fixed! No additional information retrievableNo additional information retrievable

InterviewsInterviews More personal and flexibleMore personal and flexible Novel opinions may emergeNovel opinions may emerge Interviewer can ask for clarification or Interviewer can ask for clarification or

elaboration on respondent’s statementselaboration on respondent’s statements

1616

Questionnaire Writing:Questionnaire Writing:Not for the Novice…Not for the Novice…

Pre-made questionnaires have undergone Pre-made questionnaires have undergone rigorous testing rigorous testing

Questionnaire writing requires a deep Questionnaire writing requires a deep understanding of:understanding of:– Psychological measurementPsychological measurement– Statistics Statistics – Experience administering / interpreting Experience administering / interpreting

questionnairesquestionnaires Other sources offer “easy-to-follow” guides Other sources offer “easy-to-follow” guides

for creating questionnairesfor creating questionnaires So which is it?!? So which is it?!?

1717

Existing QuestionnairesExisting Questionnaires Pre-made questionnaires for HCI studies:Pre-made questionnaires for HCI studies:

Refer to www.acm.org/~perlman/question.html Refer to www.acm.org/~perlman/question.html for more informationfor more information

Acronym Instrument Reference Institution Example

QUIS Questionnaire for User Interface Satisfaction Chin et al, 1988 Maryland 27 questions

PUEU Perceived Usefulness and Ease of Use Davis, 1989 IBM 12 questions

NAU Nielsen's Attributes of Usability Nielsen, 1993 Bellcore 5 attributes

NHE Nielsen's Heuristic Evaluation Nielsen, 1993 Bellcore 10 heuristics

CSUQ Computer System Usability Questionnaire Lewis, 1995 IBM 19 questions

ASQ After Scenario Questionnaire Lewis, 1995 IBM 3 questions

PHUE Practical Heuristics for Usability Evaluation Perlman, 1997 OSU 13 heuristics

PUTQ Purdue Usability Testing Questionnaire Lin et al, 1997 Purdue 100 questions

1818

Purdue Usability Testing Purdue Usability Testing QuestionnaireQuestionnaire

1919

Composing QuestionsComposing Questions

ClarityClarity– avoid colloquialisms, unambiguousavoid colloquialisms, unambiguous– be precisebe precise

Leading QuestionsLeading Questions– Don’t put words in the respondent’s mouth Don’t put words in the respondent’s mouth

PhrasingPhrasing– Child-like vs. childishChild-like vs. childish

Prestige BiasPrestige Bias– Over or under-estimating own abilitiesOver or under-estimating own abilities

2020

Steps to Writing a Steps to Writing a QuestionnaireQuestionnaire

1.1. Defining the objectives of the Defining the objectives of the surveysurvey

2.2. Determining the sampling groupDetermining the sampling group

3.3. Writing the questionnaireWriting the questionnaire

4.4. Administering the QuestionnaireAdministering the Questionnaire

5.5. Interpreting the resultsInterpreting the results

2121

Our ExampleOur Example

This year’s GSA (Graduate Students This year’s GSA (Graduate Students Association) freely distributed day timer.Association) freely distributed day timer.

2222

Step 1 : Defining Our Step 1 : Defining Our ObjectiveObjective

1.1. How satisfied are University of How satisfied are University of Calgary graduate students with this Calgary graduate students with this year’s graduate day timer? year’s graduate day timer?

2.2. Is it useful for them? Is it useful for them?

3.3. How might its design be improved How might its design be improved upon in future editions of this upon in future editions of this publication?publication?

2323

Step 2 : Sample Group Step 2 : Sample Group

This comes directly from the This comes directly from the research questionresearch question

University of Calgary graduate University of Calgary graduate students students

2424

Step 3 : Writing the Step 3 : Writing the QuestionsQuestions

What do we want to know? What do we want to know? What information will we use?What information will we use? Is this item better asked in an open-Is this item better asked in an open-

ended or closed-ended question?ended or closed-ended question? How should we phrase it?How should we phrase it?

2525

Try it Yourself!Try it Yourself!

1.1. Demographic questionsDemographic questions to establish to establish which group this respondent fits into.which group this respondent fits into.

2.2. General time scheduling habitsGeneral time scheduling habits, , establishing the requirements of a establishing the requirements of a day timer for a graduate student at day timer for a graduate student at the University of Calgary.the University of Calgary.

3.3. Day timer useDay timer use in general in general

4.4. GSA day timerGSA day timer

2626

What’s NextWhat’s Next

After the Questionnaire is written, it After the Questionnaire is written, it is evaluated, tested, and revisedis evaluated, tested, and revised

This cycle may be repeated more This cycle may be repeated more than oncethan once

Next the Questionnaire is distributed Next the Questionnaire is distributed to the target groupto the target group

Finally, the responses are analyzedFinally, the responses are analyzed

2727

Thank you!Thank you!

2828

References and ResourcesReferences and Resources1.1. Eberts, R.E. (1994). “Chapter 4: Experimental methodology” in User Interface Eberts, R.E. (1994). “Chapter 4: Experimental methodology” in User Interface

Design, Prentice Hall. (Design, Prentice Hall. (A basic reference, as provided in class, about A basic reference, as provided in class, about questionnaires.)questionnaires.)

2.2. Kirakowski, J. (?). Questionnaires in Usability Engineering – A list of Frequently Kirakowski, J. (?). Questionnaires in Usability Engineering – A list of Frequently Asked Questions. Human Factors Research Group, Cork, Ireland. Accessed Asked Questions. Human Factors Research Group, Cork, Ireland. Accessed September 27, 2004. September 27, 2004. http://www.ucc.ie/hfrg/resources/qfaq1.htmlhttp://www.ucc.ie/hfrg/resources/qfaq1.html ( (A list of A list of answers to questions typically asked by questionnaire designers. Quite useful answers to questions typically asked by questionnaire designers. Quite useful for basic information, and rationale for question design choices.)for basic information, and rationale for question design choices.)

3.3. Wikipedia – the free encyclopedia. “Francis Galton – inventor of the Wikipedia – the free encyclopedia. “Francis Galton – inventor of the questionnaire” Accessed October 4, 2004. questionnaire” Accessed October 4, 2004. http://en.wikipedia.org/wiki/Francis_Galtonhttp://en.wikipedia.org/wiki/Francis_Galton ( (A simple reference identifying the A simple reference identifying the inventor of the questionnaire.)inventor of the questionnaire.)

4.4. Root, R.W., Draper, S. (1983). Questionnaires as a Software Evaluation Tool. Root, R.W., Draper, S. (1983). Questionnaires as a Software Evaluation Tool. CHI’83 Proceedings, December, 83-87. (CHI’83 Proceedings, December, 83-87. (A study of the strengths and A study of the strengths and weaknesses of questionnaires.)weaknesses of questionnaires.)

5.5. O’Brien, D. (1997). Questionnaire Design. Accessed September 20, 2004. O’Brien, D. (1997). Questionnaire Design. Accessed September 20, 2004. http://www.cc.gatech.edu/classes/cs6751_97_winter/Topics/quest-design/http://www.cc.gatech.edu/classes/cs6751_97_winter/Topics/quest-design/ ( (A A list of tips and suggestions for designing questionnaire questions.)list of tips and suggestions for designing questionnaire questions.)

6.6. Mantei, M.M. (1991). Questionnaire Design Studio. Tutorial Notes CHI’91, New Mantei, M.M. (1991). Questionnaire Design Studio. Tutorial Notes CHI’91, New Orleans, April 28, 1991. (Orleans, April 28, 1991. (A tutorial on creating effective questionnaires, A tutorial on creating effective questionnaires, written in plain English with many good examples and indications of what to written in plain English with many good examples and indications of what to do / what not to do when creating questionnaires.)do / what not to do when creating questionnaires.)

7.7. Perlman, G. (1998). Web-based User Interface Evaluation with Perlman, G. (1998). Web-based User Interface Evaluation with Questionnaires. Accessed September 20, 2004. Questionnaires. Accessed September 20, 2004. http://www.acm.org/~perlman/question.html (http://www.acm.org/~perlman/question.html (A web-resource of pre-made A web-resource of pre-made usability study questionnaires and information regarding available online cgi usability study questionnaires and information regarding available online cgi scripts and web-based material.)scripts and web-based material.)