designing questionnaires
DESCRIPTION
Designing Questionnaires. Dr. Roger Harris Visiting Professor Research and Innovation Management Centre (RIMC) and Institute of Social Informatics and Technological Innovations (ISITI) [email protected] http://www.rogharris.org/ @ HarrisrwhRoger September 2013. - PowerPoint PPT PresentationTRANSCRIPT
Designing QuestionnairesDr. Roger Harris
Visiting ProfessorResearch and Innovation Management Centre (RIMC)
andInstitute of Social Informatics and Technological Innovations (ISITI)
[email protected]://www.rogharris.org/
@HarrisrwhRogerSeptember 2013
1. Survey Research
Dr. Roger HarrisVisiting Professor
Institute of Social Informatics and Technological [email protected]://www.rogharris.org/
@HarrisrwhRogerSeptember 2013
Survey questionnaires
• If the researcher is looking to test and quantify hypotheses and the data is to be analysed statistically, a formal standardised questionnaire is required.
• If the data to be collected is qualitative or is not to be statistically evaluated, it may be that no formal questionnaire is needed
• The heart of a survey is its questionnaire; the main means of collective quantitative primary data
• Survey results depend crucially on the questionnaire• Best practices stem from experience and
methodological research
Questionnaires
• A questionnaire is a formatted set of questions for obtaining information from respondents.
• The objective of questionnaire design is to translate the researcher’s information needs into a set of specific questions that respondents are able and willing to answer.
• Questionnaires collect quantitative data in a standardised way so that the data is internally consistent and coherent for analysis.
• There is no scientific principle to guarantee an optimal or ideal questionnaire.
Survey implementation choices
• Direct interview– costly; slow; high quality data
• Telephone interview– Costly; needs expertise.
• World Wide Web– Efficient; cheap; limited range of question types; hard to apply full
range of implementation procedures; respondents need access.• Self-administered by Mail
– Well studied; moderately expensive; can do well• Self-administered by eMail
– Cheaper; efficient; respondents need access• Multi-method
– Complex.
Quiz 1: True or False?
A questionnaire can be used to test and quantify a hypothesisTrue
Survey results depend on questionnairesTrue
There is a scientific method to design questionnairesFalse, best practices stem from experience and methodological research
The objective of questionnaire design is to translate the researcher’s information needs into a set of specific questions
True
Direct interviews are cheap to conductFalse
Telephone interviews are costlyTrue
The 5 most important elements for achieving high response rates
• Respondent-friendly questionnaire• The 4 contacts;
– Pre-notice invitation– The questionnaire– Thank you– (Replacement questionnaire reminder)– Final outcome notification
• For mail – return stamped envelopes• Personalised correspondence• Prepaid token incentive
Three types of survey information
• Dependent variables– Information we are primarily interested in
• Independent variables– Information which might explain the dependent
variables • Confounding variables
– Other factors related to both dependent and independent factors which may distort the results and have to be adjusted for
Characteristics of survey questionnaires
• Prescribed wording and order of questions, – to ensure that each respondent receives the same
stimuli• Prescribed definitions or explanations for each
question, – to ensure interviewers handle questions consistently
and can answer respondents' requests for clarification if they occur
• Prescribed response format, – to enable rapid completion of the questionnaire during
the interviewing process.
Quiz 2: True or False?1. Questionnaires should be respondent-friendly
True
2. There are 5 contacts with respondents during a questionnaire survey
False; there are 4
3. The three types of variables in a survey are; dependent, independent and confusing
False, they are dependent, independent and confounding
4. Dependent variables explain the independent variablesFalse, independent variables contain information that might explain the dependent variables
5. Prescribed wording and order of questions enables rapid completion of the questionnaire during the interviewing process.False. Prescribed wording and order of questions, to ensure that each respondent receives the same stimuli
However
• Given the same task and the same hypotheses, six different people will come up with six different questionnaires that differ widely in their choice of questions, line of questioning, use of open-ended questions and length.
• There are no hard-and-fast rules about how to design a questionnaire, but there are a number of points that can be borne in mind
A well-designed questionnaire should :
• Meet the research objectives• Obtain the most complete and accurate
information possible• Ensure that respondents fully understand the
questions • Be organised and worded to encourage
respondents to provide accurate, unbiased and complete information
• Be easy for the interviewer to record the answer• Keep the interview brief and to the point
How people answer questions; ideally
1. First, they interpret the question and deduce its intent, next ,
2. They must search their memories for relevant information, and,
3. Then integrate whatever information comes to mind into a single judgment.
4. Finally, they must translate the judgment into a response, by selecting one of the alternatives offered by the question.
This process is known as optimizing
How people may answer questions;
1. By automatic compliance; with no intrinsic motivation to make the answers of high quality
2. They may become fatigued, disinterested, or distracted and may be less thorough in comprehension, retrieval, judgment, and response selection
3. Instead of attempting the most accurate answers, respondents settle for merely satisfactory answers
4. Respondents may interpret each question superficially and select what they believe will appear to be a reasonable answer
This process is known as satisficing
Minimising satisificing
• The likelihood of satisficing is thought to be determined by three major factors:– task difficulty,
• question-specific attributes, e.g., the difficulty of interpreting a question and of retrieving and manipulating the requested information
• attributes of the questionnaire’s administration, e.g., the pace at which an interviewer reads the questions and the presence of distracting events
– respondent ability, • the extent to which respondents are adept at performing complex mental operations,
practiced at thinking about the topic of a particular question, and equipped with pre-formulated judgments on the issue in question
– respondent motivation• Motivation is influenced by need for cognition, the degree to which the topic of a question
is personally important, beliefs about whether the survey will have useful consequences.
• Efforts to minimize task difficulty and maximize respondent motivation are likely to pay off by minimizing satisficing and maximizing optimising.
Remedies for satisficing• Minimise task difficulty
– Minimise the number of words in questions– Avoid double-barreled questions– Decompose questions when needed
• Use ratings not rankings• Label response options• Minimise ‘response effects’
– Avoid blocks of ratings on the same scale (prevents ‘straight lining’)– Avoid ‘no response’ options– Avoid ‘agree/disagree’, ‘yes/no’, ‘true/false’ questions
• Maximise motivation– Describe purpose of the survey– Provide instructions to think carefully– Include random probes (“give your reason”)– Keep surveys short– Put important questions early.
Quiz 3: True or False?1. Given the same task and the same hypotheses, six different people
will come up with the same questionnaire False
2. There are no hard-and-fast rules about how to design a questionnaire
True3. The 4 steps in respondent optimising are; interpret, search,
integrate and translateTrue
4. Satisficing includes when respondents interpret each question superficiallyTrue
5. The likelihood of satisficing is thought to be determined by 3 factors; task difficulty, respondent ability and the weather.False. The likelihood of satisficing is thought to be determined by 3 factors; task difficulty, respondent ability and respondent motivation
Example of a survey1. Adoption rates of personal
computers varies among people2. Knowledge workers
3. Computer anxiety4. “Respondents with higher levels of
computer anxiety will be less likely to use personal computers”
5. The Computer Anxiety Rating Scale with 19 items; able to learn, confidence, apprehension, etc.
6. “I feel insecure about my ability to interpret a computer printout “ “I am confident that I can learn computer skills “
1. Review the literature to define the research question
2. Define the population to be surveyed3. Isolate the variables of interest4. Organise the variables into
hypotheses that reflect the research question
5. Operationalise the variables into tangible indicators
6. Develop questions that will enable measurement of the indicators
7. Formulate a questionnaire8. Carry out the survey9. Analyse the data
2. Purpose and Types of Questionnaires
Dr. Roger HarrisVisiting Professor
Institute of Social Informatics and Technological [email protected]://www.rogharris.org/
@HarrisrwhRogerSeptember 2013
Approaches
• Structured– All respondents provide answers to a set of questions
on a standardised questionnaire. Results can be compared to each other and easily summarised
• Unstructured– Allows the respondent to talk about whatever aspects
of the survey topic they wish. Best suited for qualitative data
• Semi-structured– Mixes both. Uses qualitative data to supplement
quantitative.
Administering questionnaires
• Self administered questionnaires may be sent by post, email, or electronically online.
• Advantages– Cheap and easy to administer.– Preserve confidentiality.– Can be completed at respondent’s convenience.– Can be administered in a standard manner.
• Read out by interviewers. Questionnaires may be by telephone or face to face.
• Advantages – Allow participation by illiterate people.– Allow clarification of ambiguity.– More reliable response rate
• The method depends on who the respondents are. For example, university lecturers may be more appropriately surveyed by email; older people by telephone interviews; train passengers by face to face interviews.
Comparisons of data collection modesParameter Interview Telephone Mail/Self
Cost Most expensive Medium expensive Cheapest
Standardisation Depend on interviewer
Depend on interviewer Good & reduces bias
Coverage Depend on personal contact
Depend on personal interviewer
Wide and scattered reach
Observation Listen, watch, assess Listen, assess None
Probing Allow Allow Not possible
Literacy Not required Not required Required
Respondent skills Language Language Not needed
Response rate Highest Medium Lowest
Privacy Difficult Some anonymity Good
Consent Easy to gain Possible Difficult
3. Asking Questions
Dr. Roger HarrisVisiting Professor
Institute of Social Informatics and Technological [email protected]://www.rogharris.org/
@HarrisrwhRogerSeptember 2013
Common Wisdom
1. Use simple, familiar words 2. Avoid technical terms, jargon, and slang3. Use simple syntax4. Avoid words with ambiguous meanings5. Use wording that all respondents will interpret in the same way6. Strive for wording that is specific and concrete as opposed to general
and abstract7. Make response options exhaustive and mutually exclusive8. Avoid leading or loaded questions that push respondents toward an
answer9. Ask about one thing at a time 10. Avoid double-barreled questions (if there’s an ‘and’ in the question,
there are probably two questions11. Avoid questions with single or double negations.
Optimize question order
1. Early questions should be easy and pleasant to answer, and should build rapport between the respondent and the researcher.
2. Questions at the very beginning of a questionnaire should explicitly address the topic of the survey, as it was described to the respondent prior to the interview.
3. Questions on the same topic should be grouped together.4. Questions on the same topic should proceed from general to
specific.5. Questions on sensitive topics that might make respondents
uncomfortable should be placed at the end of the questionnaire.6. Filter (branching) questions should be included, to avoid asking
respondents questions that do not apply to them.
Open versus closed questions
• Open questions permit respondents to answer in their own words
• Closed questions require respondents to select an answer from a set of choices
• The vast majority of survey questions are closed• Closed questions must be grouped into a
relatively small number of categories• This requires the development of a coding
scheme
Open versus closed questions
• Open questions are usually preferable for measuring quantities, e.g. age, number of doctor visits, hours devoted to housework,
• Closed quantity categories for this type of open question (e.g., less than 1 h, 1–3 h, more than 3 h) can produce errors
Comparison of open and closedOpen questions
Advantages• Allows exploration of the range of possible
themes arising from an issue• Respondent is asked to give a reply to a
question in his/her own words• Can be used even if a comprehensive range
of alternative choices cannot be compiled• Reveal the issues which are most important
to the respondent• Respondents can 'qualify' their answers or
emphasise the strength of their opinionsDisadvantages
• Respondents may find it difficult to 'articulate' their responses
• Respondents may not give a full answer if they may forget to mention important points
• Verbatim comments have to be interpreted, coded and reduced to manageable categories
Closed questionsAdvantages
• Easy and quick to fill in, less reliance on memory
• Can specify answer categories most suitable for the purposes
• Minimize discrimination against the less literate (self administered questionnaire) or the less articulate (interview questionnaire)
• Easy to code, record, and analyze results quantitatively
• Easy to report resultsDisadvantages
• Do not allow the respondent to give a different response to those suggested.
• Suggest answers that respondents may not have considered before.
Categorical judgments
• Such as the ‘‘what is most important problem’’
• Closed categorical question can be used only if its answer choices are comprehensive.
• Usual to include other category.• Respondents tend to restrict their answers to
the substantive choices that are explicitly offered.
Open response-option questions
• Both open-ended and also include specific response-options as well. For example,
• What features of this implement do you like?– Performance– Quality– Price– Weight– Others mentioned:
• Eliminates the disadvantages
Types of closed questions format
• Choice of categories, for example, What is your marital status?– [ ] Single– [ ] Married– [ ] Divorced– [ ] Widowed
• Likert style scale, for example, Statistics is an interesting subject– [ ] Strongly disagree – [ ] Disagree – [ ] Cannot decide – [ ] Agree Strongly – [ ] Agree
• Differential scales, for example, How would you rate the presentation?Extremely interesting [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] Extremely dull
• It may also be diagrammatic; • Extremely interesting Extremely dull
Types of closed questions format
• Checklists, for example, Circle the clinical specialties you are particularly interested in:
General medicine Obstetrics and gynecologyGeneral surgery OrthopaedicsOphthalmology Accident and emergencyPaediatrics General practice
• Ranking, for example, Please rank your interests in the following specialties(1= most interesting, 8=least interesting)
– [ ] General medicine – [ ] Obstetrics and gynecology– [ ] General surgery – [ ] Orthopaedics– [ ] Ophthalmology – [ ] Accident and emergency– [ ] Paediatrics – [ ] General practice
Question checklist1. Is this question sufficient to generate the required information?2. Can the respondent answer the question correctly?3. Are there any external events that might bias response to the
question?4. Do the words have the same meaning to all respondents?5. Are any of the words or phrases loaded or leading in any way?6. Are there any implied alternatives within the question?7. Will the question be understood by the type of individual to be
interviewed?8. Is there any ambiguity in my questions?9. Are any words or phrases vague?10. Are any questions too personal or of a potentially embarrassing
nature?11. Do questions rely on feats of memory?
Put questions into a meaningful order and format
• Opening questions– Opening question easy and pleasant to answer– Not threatening– Questions of special importance should be earlier
• Question flow– Flow in some kind of logical order– One leads easily and naturally to the next.– Questions on one subject should be grouped together
• Question variety– Vary the respondent's task from time to time– An open-ended question here and there (even if it is not analysed)– Questions involving showing cards/pictures to respondents can help
vary the pace and increase interest.
The effect of one word
Balanced questions
• Do you support India taking military action against Pakistan?• Do you support or oppose India taking military action against
Pakistan?• If you heard that candidate X supported a higher minimum
wage, would that make you more likely to vote for her?• If you heard that candidate X supported a higher minimum
wage, would that make you more likely to vote for her, less likely to vote for her, or wouldn’t it make much difference?
• Did you happen vote in the presidential election in 2010?• Did you happen vote in the presidential election in 2010, or
not ?
“Don’t know” – offered or volunteered
Using scales to get to “the most”
How questions can affect one another
• Funnelling. Ask general questions before specific priority questions in order to obtain unbiased responses.
• Pay attention to how one question might influence answers to the next:– In considering whether or not to vote for a political
candidate, how important is the candidate’s position on HIV/AIDS in making your choice?
– What do you think are the two most important health care issues for the government to address?
4. How to use Scales
Dr. Roger HarrisVisiting Professor
Institute of Social Informatics and Technological [email protected]://www.rogharris.org/
@HarrisrwhRogerSeptember 2013
Types of scales for closed questions
• The four levels of measurement: • Nominal• Ordinal• Interval, and • Ratio
Nominal scales
• Simplest form of measurement scale • Classifies individuals, companies, products, brands or other entities into categories
where no order is implied. • Often referred to as a categorical scale. • It is a system of classification and does not place the entity along a continuum. • Involves a simply count of the frequency of the cases assigned to the various categories• Example of a nominal scale• Which of the following food items do you tend to buy at least once per month? (Please
tick)1. Okra2. Palm Oil3. Milled Rice4. Peppers5. Prawns6. Pasteurized milk
• The numbers have no arithmetic properties and act only as labels.
Ordinal scales
• Involve the ranking of individuals, attitudes or items along the continuum of the characteristic being scaled.
• For example, if a researcher asked farmers to rank 5 brands of pesticide in order of preference.
• Example of an ordinal scale used to determine farmers' preferences among 5 brands of pesticide.
• Brand Order of preference1. Rambo2. R.I.P.3. Killalot4. D.O.A.5. Bugdeath
• Provides the order of preference but nothing about how much more one brand is preferred to another - no information about the interval between any two brands.
Interval scales
• The interval scale has equal units of measurement, thus making it possible to interpret not only the order of scale scores but also the distance between them.
• Allows the use of the arithmetic mean as the measure of average. • Two respondents with scale positions 1 and 2 are as far apart as two
respondents with scale positions 4 and 5• Cannot be said that a respondent with score 10 feels twice as strongly as
one with score 5. • Temperature is interval scaled, being measured either in Centigrade or
Fahrenheit. We cannot speak of 50°F being twice as hot as 25°F. No fixed origin.
• Interval scales may be either numeric or semantic. • Most of the common statistical methods of analysis require only interval
scales in order that they might be used.
Ratio scales
• The highest level of measurement• Has the properties of an interval scale together with a
fixed origin or zero point. • Examples; weights, lengths and times. • Permit the researcher to compare both differences in
scores and the relative magnitude of scores. • For instance the difference between 5 and 10 minutes
is the same as that between 10 and 15 minutes, and 10 minutes is twice as long as 5 minutes.
• Virtually all statistical operations can be performed on ratio scales.
Likert-type scales• Most often uses 5 points; • “I would use this tool in my work” • [Strongly Disagree ] [Disagree] [Neither Agree Nor Disagree] [Agree] [Strongly Agree]• Semantic scale• “Was the product easy or difficult to use?” Very Difficult [ ] [ ] [ ] [ ] [ ] Very Easy• Scales might have 7or 11 points.• The points offered should cover the entire measurement continuum, leaving out no regions• Points must appear to be ordinal, progressing from one end of a continuum to the other• The meanings of adjacent points should not overlap. • Each respondent must have a relatively precise and stable understanding of the meaning of each
point on the scale.• Most or all respondents must agree in their interpretations of the meanings of each scale point• Offering a neutral midpoint may encourage satisficing, or• Eliminating the midpoint will force people with no opinion to pick a point measurement.• Cross-sectional validity and test–retest reliability increases from 2- to 3- to 5-point scales but
were equivalent thereafter for 7-, 9-, and 14-point scales.
Some labels that work
Bipolar intervals
• Extremely bad• Moderately bad • Slightly bad• Neither good nor bad• Slightly good• Moderately good• Extremely good
• Dislike a great deal• Moderately dislike• Dislike a little• Neither like nor dislike• Like a little• Moderately like• Like a great deal
Unipolar intervals
• Not at all• Slightly• Moderately• Very• Extremely
• Definitely won’t• Probably won’t• Might or might 7not• Probably will• Definitely will
• Never• Sometimes• About half the time• Most of the time• Always
Quiz 5: True or False?1. There are three levels of measurement; Nominal, Ordinal, and
IntervalFalse. There are four levels of measurement; Nominal, Ordinal, Interval, and Ratio
2. Ordinal scales involve the ranking items along the continuum of the characteristic being scaledTrue.
3. In an interval scale, two respondents with scale positions 1 and 2 are as far apart as two respondents with scale positions 4 and 5
True. 4. Interval scales can only be numeric
False. Interval scales may be either numeric or semantic5. A ratio scale has the properties of an interval scale together with
a fixed origin or zero pointTrue
5. Questionnaire Appearance
Dr. Roger HarrisVisiting Professor
Institute of Social Informatics and Technological [email protected]://www.rogharris.org/
@HarrisrwhRogerSeptember 2013
Physical appearance
• The physical appearance of a questionnaire can have a significant effect upon both the quantity and quality of data obtained
• Ill-designed questionnaires can give an impression of complexity, and too big a time commitment
• Unnecessarily confusing layouts making it more difficult for interviewers, or respondents in the case of self-completion questionnaires, to complete
• Keep questionnaires as short as possible• In a rural situation an interview should not last
longer then 30-45 minutes.
Format
• Booklets– In the place of loose or stapled sheets of paper,
make it easier for interviewer or respondent to progress through the document. Fewer pages tend to get lost
• Simple, clear formats– The clarity of questionnaire presentation can also
help to improve the ease with which interviewers or respondents are able to complete a questionnaire.
Design tips
• Creative use of space and typeface– Tendency to put too much information on a page. This is counter-
productive since it gives the questionnaire the appearance of being complicated.
– Questionnaires that make use of blank space appear easier to use, enjoy higher response rates and contain fewer errors when completed.
• Colour coding– Can help in the administration of questionnaires. It is often the case that
several types of respondents are included within a single survey (e.g. wholesalers and retailers). Printing the questionnaires on two different colours of paper can make the handling easier.
• Interviewer instructions– Place alongside the questions to which they pertain. Instructions on
where the interviewers should probe for more information or how replies should be recorded are placed after the question
Distributing questionnaires
• If you’re sending a cover letter, be sure to use letterhead stationary and have the principal investigator or someone of renown from the funding agency sign it to add prestige.
• Your cover and advance letters should include a brief explanation of the study, an explanation of how you obtained the respondent’s name, why it is important that each "sampled" respondent cooperate, and a short assurance of confidentiality.
• Include a stamped self-addressed envelope for the respondent to return the completed form.
Design tips
• Clear instructions how to mark answers:– “Please tick the box that best describes your
answer”• Instructions for each question:
– “Tick one box only”– “Write the number inside the box”– “Circle or underline the choice that corresponds
most closely to your preference”
Design tips
• Questions in a meaningful order; natural flow, progression
• Place definitions in the question• Respondents rarely read instruction booklets• Indicate prominently which questions are
mandatory• Include a “thank you” at the end• Ensure the respondent or interviewer knows
what to do with the completed questionnaire.
Design elements for a mail questionnaire
• Booklet• Descriptive title• No questions on front or back• Size appropriate for mail envelope• Easy to read font; sans serif, 10 pt+, Ariel.
(this is sans serif this is serif)• Colour• Attractive, inviting design• Number questions• Embedded instructions
6. Testing and Evaluating Questionnaires
Dr. Roger HarrisVisiting Professor
Institute of Social Informatics and Technological [email protected]://www.rogharris.org/
@HarrisrwhRogerSeptember 2013
Piloting
• Until it has actually been used in interviews and with respondents, it is impossible to say whether it is going to achieve the desired results.
• Pre-test the questionnaire before it is used in a full-scale survey, to identify any mistakes that need correcting.– whether the questions as they are worded will achieve the desired results– whether the questions have been placed in the best order– whether the questions are understood by all classes of respondent– whether additional or specifying questions are needed or whether some
questions should be eliminated– whether the instructions to interviewers are adequate.
• Usually a small number of respondents are selected for the pre-test. The respondents selected for the pilot survey should be broadly representative of the type of respondent to be interviewed in the main survey.
Testing methods
• Expert review– one or more experts critiques the questionnaire. The
experts are typically survey methodologists, but they can be supplemented with specialists in the subject matter(s) of the questionnaire. Reviews are done individually or as part of a group discussion.
• Pretesting – involves administering a questionnaire to a small sample
of the relevant population under conditions close to, or identical to, those of the main survey. Ask them how the questions worked, which are typically reported at a group debriefing discussion (cognitive interviewing).
Cognitive interviewing
• After testing participants have completed the questionnaire, ask them:– In your own words, what do you think the
questionnaire is trying to discover?– What was the question asking?– What did you consider in determining your
response?– Was there anything difficult about this question?– Do you feel that something important might be
missing?
What am I measuring? Reliability
• Questionnaire reliability has to do with the quality of measurement
• Reliability is a measure of the consistency or repeatability of your measures
• Internal consistency is one type of reliability measure. Cronbach’s alpha measures it by how well a set of indicators measures a single uni-dimensional variable.
What am I measuring? Validity
• Construct validity– The degree to which inferences can legitimately
be made from the operationalisations in the questionnaire to the theoretical constructs on which they are based
• External validity– The degree to which the conclusions from the
questionnaire data would hold for other persons in other places and at other times
Quiz 6: True or False?1. The physical appearance of a questionnaire has no effect on either
the quantity and quality of data obtainedFalse.
2. In a rural situation an interview should not last longer then 30-45 minutes.True.
3. Expert review and pre-testing are two methods of piloting a questionnaire
True. 4. Cognitive interviewing involves asking test participants about the
questionnaireTrue.
5. Reliability is a measure of how truthful respondents areFalse. Reliability is a measure of the consistency or repeatability of your measures
7. Survey Conduct
Dr. Roger HarrisVisiting Professor
Institute of Social Informatics and Technological [email protected]://www.rogharris.org/
@HarrisrwhRogerSeptember 2013
Getting respondents to respond
• Introduce your questionnaire • Motivate people to complete it• Write a few lines to introduce your survey. • Introduction should be short and includes:
– Purpose of the survey – Why it is important to hear from the respondent – What may be done with the results – Promise of confidentiality– What possible impacts may occur with the results. – Person to contact for questions about the survey. – Due date for response
Steps in designing a survey
1. Review the literature to define the research question2. Define the population to be surveyed3. Isolate the variables of interest4. Organise the variables into hypotheses that reflect the
research question5. Operationalise the variables into tangible indicators6. Develop questions that will enable measurement of
the indicators7. Formulate a questionnaire8. Carry out the survey9. Analyse the data
Steps in designing a questionnaire
1. Decide the information required.2. Define the target respondents.3. Choose the method(s) of reaching your target
respondents.4. Decide on question content.5. Develop the question wording.6. Put questions into a meaningful order and format.7. Check the length of the questionnaire.8. Pre-test the questionnaire.9. Develop the final survey form.
Decide the information required.
• Do not start by writing questions• Decide 'what are the things I need to know
from the respondent in order to meet my survey objectives?‘
• From the research brief and the research proposal a small number of preliminary informal interviews with target respondents will give a glimpse of reality
Define the target respondents
• Must define the population about which the researcher wishes to generalise from the sample data to be collected
• E.g., existing users of a product type and/or non-users
• Take into account factors such as the age, education, etc. of the target respondents.
Choose the method(s) of reaching your target respondents
• The method of contact will influence the questions and the phrasing of those questions
• The main methods available in survey research are:· personal interviews· group or focus interviews· mailed questionnaires· telephone interviews.
Decide on question content
• Always ask, "Is this question really needed?“• No question should be included unless the data it gives rise
to is directly of use in testing one or more of the hypotheses
• Occasions when seemingly "redundant" questions might be included:– Opening questions that are easy to answer and which are not
perceived as being "threatening", and/or are perceived as being interesting, can greatly assist in gaining the respondent's involvement in the survey and help to establish a rapport.
– "Dummy" questions can disguise the purpose of the survey and/or the sponsorship of a study
– Validating questions to test the validity of other answers
. Develop the question wording
• Three forms of question,–closed, –open-ended and –open response-option
questions.
Put questions into a meaningful order and format
• Opening questions: – Easy to answer – Not threatening– The first question is crucial as the respondent's first exposure to the interview
and sets the tone for the nature of the task to be performed.• Question flow:
– Questions should flow in some kind of psychological order, so that one leads easily and naturally to the next. Questions on one subject, or one particular aspect of a subject, should be grouped together.
• Question variety:– Respondents become bored quickly and restless when asked similar questions.
Vary the respondent's task from time to time. An open-ended question here and there (even if it is not analysed) may provide much-needed relief from a long series of questions in which respondents have been forced to limit their replies to pre-coded categories. Questions involving showing cards/pictures to respondents can help vary the pace and increase interest.
Quiz 4: True or False?1. Surveys can be structured, unstructured or stratified
False. Surveys can be structured, unstructured or semi-structured2. Self administered questionnaires are relatively expensive
False.3. Double-barreled questions ask about guns
False. Double- barreled questions contain two questions 4. Questions on sensitive topics that might make respondents
uncomfortable should be placed at the end of the questionnaireTrue
5. The vast majority of survey questions are closed.True
6. Closed questions allow respondents to give a different response to those suggestedFalse
8. Automated Techniques
Dr. Roger HarrisVisiting Professor
Institute of Social Informatics and Technological [email protected]://www.rogharris.org/
@HarrisrwhRogerSeptember 2013
Internet questionnaires
• Facilitates – Design; templates available– Distribution– Completion– Analysis
• Lowers costs• Accelerates survey• Only for computer literate online respondents• No face-to-face• Concerns with confidentiality and privacy
Aspects of online questionnaires
• Graphical user interface– Control appearance, colours, fonts for attractive
design• Layout
– Paging from screen to screen– Scrolling simplifies navigation
• Push buttons– Buttons for specified actions and options; ‘Submit’
‘Next’
Aspects of online questionnaires
• Status bar– Visual indicator of respondent’s progress
• Radio button– Circular icon to activate one response choice and
de-activates the others• Drop-down box
– Space-saving device to present choices when they are needed but hides them when they aren’t
Aspects of online questionnaires
• Check boxes– Small boxes next to question that the respondent
clicks on to select their answer; when an x appears in it.
• Open-ended boxes– Boxes in which respondents can type in their own
answers to open-ended questions• Pop-up boxes
– Boxes that appear at selected points containing information or instructions to respondents.
Software aspects of online questionnaires
• Variable piping– Allows variables to be inserted into a questionnaire as a
respondent is completing it.• Error trapping
– Controls the flow• Forced answering
– Prevents respondents from continuing with a questionnaire if they fail to answer a mandatory question
• Interactive help desk– Live, real-time support for respondents with a problem.
Example: Survey Monkey
• Founded in 1999 • In 2013,
SurveyMonkey had 15 million users
• Over 200 employees• Used a lot for market
surveys, and by SMEs
Survey Monkey pricing options
Quiz 7: True or False?1. Construct validity degree to which inferences can legitimately be
made from the operationalisations in the questionnaire to the theoretical constructs on which they are based True.
2. External validity is a measure of the appearance of the front page.False. The degree to which the conclusions from the questionnaire data would hold for
other persons in other places and at other times3. Automated techniques facilitate design, distribution, completion
and printingFalse. Automated techniques facilitate design, distribution, completion and analysis
4. The status bar is a measure of how important the respondent isFalse. It is a visual indicator of respondent’s progress
5. Survey Guerrilla is an example of an online survey tool.False. Survey Monkey is an example of an online survey tool.
9. Examples of Question Design
Dr. Roger HarrisVisiting Professor
Institute of Social Informatics and Technological [email protected]://www.rogharris.org/
@HarrisrwhRogerSeptember 2013
A poor question The problem Improved version
Do you support gun control?
Yes No Don’t know
Unclear question because of vague words
Do you believe that guns do not belong in schools? Yes No Don’t know
Do you believe that guns and knives do not belong in schools? Yes No Don’t know
Two or more questions Do you believe that knives do not belong in schools?
Yes No Don’t know
Whenever violence occurs in schools, weapons are typically found in school lockers. Do you believe that students should keep guns in their lockers? Yes No Don’t know
Wordy or lengthy question Should students keep guns in their lockers?
Yes No Don’t know
A poor question The problem Improved version
Should students pack a 45 at school? Yes No Don’t know
Question contains jargon Should students carry a handgun to school? Yes No Don’t know
Students should not carry weapons and should not have them in their lockers. Do you agree? Strongly agree Agree Undecided Disagree Strongly disagree
Question contains negatives Should students keep guns in their lockers?
Yes No Don’t know
How many times have you seen a student carry a handgun? 0 times 1-2 times 2-3 times More than 3 times
Response categories overlap How many times have you seen a student carry a handgun? 0 times 1-2 times 3-4 times More than 4 times
A poor question The problem Improved version
How often have you seen students carry semi-automatic weapons at school? None 1 time 2 times 3 or more times
Respondent does not understanding to answer the question
How often have you seen students carry a rifle at school? None 1 time 2 times 3 or more times
To what extent do you feel that handguns are a problem at your school? A great extent Some Not very important Not a problem
Unbalanced response option To what extent do you feel that handguns are a problem at your school? A great extent Some extent Little extent
No labels
19 indicators
What does the * mean?
Bit confusing here; age/sex
What’s this?
Instructions
Definition
Overlapping< > ?Two questions
Two questions
Uneven frequency flow
Two questions
Two questions
Two questions
Two questions
Dense appearanceConfusing layout
What this?
Yes or no?
No instructions
What does it mean?Not much space
Comments about what?
No instructions what to put here
No alternatives or graduated response
Wordy
Some observations
• Almost every questionnaire you come across has something wrong with it
• Most are the result of inappropriate approaches;– Starting with the questions– No clear articulation of the purpose– Not following simple guidelines– Insufficient thought put into the process– Underestimation of what’s required
10. Recording and Coding Questionnaire Responses
Dr. Roger HarrisVisiting Professor
Institute of Social Informatics and Technological [email protected]://www.rogharris.org/
@HarrisrwhRogerSeptember 2013
• Closed question answers should be coded in order to facilitate recording.
• Number all questions• Record coded answers in a spreadsheet• For ease of recording, show the code next to the
response:1. Strongly disagree 2. Disagree 3. Cannot decide 4. Agree Strongly 5. Agree
Recording responses
Q1 Q2 Q3 Q4 Q5 Q6 Q71 Respondent2 Respondent3 Respondent4 Respondent5 Respondent6 Respondent7 Respondent8 Respondent9 Respondent
Sample spreadsheet recordingeBorneo Question 1 2 3 4 5 6 Gender Ethnicity Age Education Employment Income
10-20=1 21-30=2 None=N Unskilled=U 31-40=3 Primary=PI Skilled=S 41-50=4 Junior High=J Farmer=F Up to RM250 =1 Lun Bawang=L 51-60=5 Senior High=S Business=B RM251-RM500=2 Kelabit=K 61-70=6 College=C Government=G RM501-RM750=3 M Penan=P 71-80=7 University=U Housewife=H RM751-RM1,000=4
Resp. No. Village F Other=O 81+=8 Post-Graduate=PG None/other=O RM1,001+=5
1 Bakelelan F L 4 J F 22 Bakelelan F L 2 S B 23 Bakelelan F L 5 PI F 24 Bakelelan M L 5 PI F 15 Bakelelan M L 6 J F 16 Bakelelan F L 3 PI F 27 Bakelelan F L 4 J H 28 Bakelelan F L 3 U B 59 Bakelelan F L 3 J F 2
10 Bakelelan F L 4 J F 111 Bakelelan M L 4 PI F 112 Bakelelan M L 5 J B 513 Bakelelan F L 2 S B 214 Bakelelan F L 3 PI B 315 Bakelelan M L 3 C B 216 Bakelelan M L 4 PI F 2
Spreadsheet analyses
Male Female0
102030405060708090
100
62
33
Gender
Gender
Quantity
10-20 21-30 31-40 41-50 51-60 61-70 71-80 81+0
102030405060708090
100
1
25 25 24
127
1 0
Age
Age Range
Quantity
None
Primary
Junior High
Senior H
igh
College
Universi
ty
Post-Grad
uate0
20
40
60
80
100
2
2819 24
9 121
Education
Education Level
Quantity
Unskilled
Skilled
Farmer
Business
Governmen
t
Housewife
None/other
0102030405060708090
100
6 8
35
923
5 9
Employment
Employment Category
Quantity
SPSS for statistical analyses
11. Group Exercises
Dr. Roger HarrisVisiting Professor
Institute of Social Informatics and Technological [email protected]://www.rogharris.org/
@HarrisrwhRogerSeptember 2013
Exercise
• Nomophobia is the fear of being out of mobile phone contact.• A research organisation looked at anxieties suffered by mobile
phone users. • The study found that nearly 53% of mobile phone users in
Britain tend to be anxious when they "lose their mobile phone, run out of battery or credit, or have no network coverage".
• The psychographic variables were:– Attitude towards usage of cellular phones,– Mobile phones dependence, and – Associated anxiety
• Devise a set of questions to measure nomophobia.
The questionnaire focusing on nomophobia had 7 components
1. Duration of having mobile phone with self; 2. Anxiety and stress experienced because of faulty
connections; 3. Loss of mobile and battery discharge; 4. Amount spent per month on recharge 5. Reaction shown to phone ringing at inappropriate
times; 6. Frequency of change of phone/sim cards7. Reactions to inability of using the phone for a
period of one week.
Answers
Physical sensations 1. Physical sensations
Never Seldom Sometimes
Often Always
Accelerated heart beat, heart pounding
Face reddening or hot flashes
Tightness of the chest, chest pain or pressure
Numbness or tingling sensations of arms or legs
Lightheadedness, dizziness or feelings of unsteadiness
Shortness of breath or difficulty breathing
Sweating
Shaking or trembling
Intense fear, panicky feelings, anxiety
Thoughts 2. Thoughts Never Seldom Some
timesOften Always
I worry that something bad might happen to myself
I worry that something bad might happen to someone else
I worry that something bad might happen to the device (getting lost, stolen, or broken)
I worry about others being unable to contact me
I worry about missing important news
I worry that I might get lost
I am preoccupied with thoughts about how to complete tasks without the device
I am preoccupied with thoughts about how I could have acted differently to be able to access the device
Emotions 3. Emotions Never Seldom Some
timesOften Always
I feel lonely
I feel anxious and/or embarrassed because I feel unprepared
I feel anxious because I cannot access certain information
I feel stressed when I am not able to check on certain things on my device
I feel uncomfortable communicating with friends, family, colleagues, etc. without my device (I prefer to avoid communicating face-to-face)
I feel guilty for not contacting someone
I feel guilty because I am unable to do work or other important tasks
I feel angry that I allowed myself to lose access to the device
I feel angry that someone else led me to lose access to the device
I feel sad that I will not be able to use my device
Other 4. Other Never Seldom Some
timesOften Always
I feel bored
I just don’t feel “right”
Review Questions
1. Summarise the qualities of a good questionnaire.2. Where should interviewer instructions pertaining to responses to a
particular question be placed on the questionnaire?3. The textbook says that one does not start by writing questions. How should
the researcher begin?4. What are two occasions when apparently "redundant" questions should be
found in a questionnaire?5. Name the three advantages of open-ended questions.6. What are three reasons why a respondent might be unable to answer a
question?7. What is the recommended duration of interviews carried out in rural
situations?8. What are the key characteristics of opening questions in a questionnaire?