ADVANCED RESEARCH TECHNIQUES

QUESTIONNAIRES

CEMAL ESKİCİ

QUESTIONNAIRES Introduction Questionnaires in language research

What are questionnaires What do they measure Advantages and disadvantages of

questionnaires Constructing the questionnaires

General features The main parts of a questionnaire Questionnaire content and multi-item scales Close-ended questionnaire items Open-ended questions

QUESTIONNAIRES

Introduction One of the most common methods of data

collection in research is to use various kinds of questionnaires. (Dörnyei, 2003)

A survey is a system for collecting information to describe, compare, or explain knowledge, attitudes, and practices or behaviour. (Fink, 1995)

The essential point in research is that it cannot be built on poorly collected data. (Gillham, 2004)

QUESTIONNAIRES

What are questionnaires?

Brown (cited in Dörnyei, 2003)

expresses that questionnaires are any

written instruments that presents the

respondents with a series of questions or

statements to which they are to react

either by writing out their answers or

selecting from among existing answers.

QUESTIONNAIRESWhat are questionnaires?

Dörnyei (2003) states that although the term

‘questionnaire’ is familiar, it’s hard to provide a

precise definition for it. He reasons that many

of the questionnaires do not contain any or

many real questions that end with a question

mark.

QUESTIONNAIRESWhat are questionnaires?

He adds that the other reason is

that the rubric of ‘questionnaire’ has

been used by researchers in at least two

broad senses:

Interview schedules

Self-administered pencil and paper

questionnaires

QUESTIONNAIRES

What are questionnaires?

Aiken (cited in Dörnyei, 2003)

explains that questionnaires are often

referred to as ‘forms, opinnionaires,

tests, batteries, checklists, scales,

surveys, schedules, studies, profiles,

indexes, indicators, or even simply

sheets.’ (p. 5)

QUESTIONNAIRES

What do questionnaires measure? Factual questions- classification

questions or subject descriptors are used to find out about who the respondents are.

Behavioral questions are used to find out what the respondents are doing or have done in the past.

Attitudinal questions are used to find out what people think. (Dörnyei, 2003)

QUESTIONNAIRES

AdvantagesThe main attraction of questionnaires is their unprecedented efficiency in terms of: researcher time researcher effort financial resources. (Dörnyei,2003)

QUESTIONNAIRES

AdvantagesQuestionnaires are: easy to conduct extremely versatile uniquely capable of gathering a

large amount of information quickly in a form that is readily processible. (Dörnyei,2003)

QUESTIONNAIRES

Advantages Low cost in time and money. Easy to get information from a lot of

people very quickly. Respondents can complete the

questionnaire when it suits them. Analysis of answers to closed

questions is straightforward.

QUESTIONNAIRES

Advantages Less pressure for an immediate

response. Respondents’ anonymity. Lack of interviewer bias. Standardization of questions. Can provide suggestive data for

testing an hypothesis. (Gillham, 2004)

QUESTIONNAIRES

Disadvantages Simplicity and superficiality of

answers Unreliable and unmotivared

respondents Respondent literacy problems Little or no opportunity to correct

the respondents’ mistakes Social desirability (or prestige) bias Self-deception

QUESTIONNAIRES

Disadvantages Acquiescence bias Halo effect Fatigue effects Problems of data quality

(completeness and accuracy). Typically low response rate unless

sample «captive».

QUESTIONNAIRES

Disadvantages Questionnaire development is often

poor. Seek information just by asking

questions. Assumes respondents have answers

available in an organized fashion.

QUESTIONNAIRES

Disadvantages Lack of control over order and

context of answering questions. Question wording can have a major

effect on answers. People talk more easily than they

write. Respondent uncertainty as to what

happens to data. (Dörnyei, 2003; Gillham, 2004)

Constructing the questionnairesThe steps required to design and administer a questionnaire include:

Defining the objectives of the survey   

Determining the sampling group  

Deciding on the general features of the

questionnaire

Writing effective items/questions and drawing

up an item pool (Dörnyei.2003)

QUESTIONNAIRES

QUESTIONNAIRES

Constructing the questionnaires

Selecting and sequencing the items  

Writing appropriate instructions and

examples

Piloting the questionnaire  

Conducting item analysis

QUESTIONNAIRES

General featuresLength

Dörnyei (2003) suggests that to

be effective a questionnaire should not

be more than 4 pages and it shouldn’t

take more than 30 minutes for the

respondents to answer all the

questions.

QUESTIONNAIRES

General featuresLayout: An attractive and proffessional design involves: Booklet format Appropriate density Orderly layout Paper quality Sequence marking

QUESTIONNAIRES

General featuresSensitive topics and anonymity

Dörnyei (2003) claims that these are not only confined to explicitly illegal or embarrassing subjects but also include basic demographic items such as age and marital status.

He also states that to diffuse sensitive items we make the questionnaire anonymous.

QUESTIONNAIRES

General featuresSensitive topics and anonymity

Oppenheim (cited in Dörnyei, 2003) suggests that to overcome the issues concerning sensitive topics and anonymity, something along the following line should be displayed prominently on the front of the questionnaire:

«The contents of this form are absolutely confidential. Information identifying the respondent will not be diclosed under any circumstances.»

QUESTIONNAIRES

The main parts of a questionnaire Title Instructions

1. General instructions2. Specific instructions

Questionnaire items Additional information Final «Thank you!»

QUESTIONNAIRES

Questionnaire content and multi-item scalesAppropriate sampling of the content«The temptation is always to cover too much, to ask everything that might turn out to be interesting. This must be resisted.» (Moser & Kalton, cited in Dörnyei, 2003 p.32) The research problem should be clarified. What critical concepts need to be

addressed by the questionnaire should be identified.

QUESTIONNAIRESQuestionnaire content and multi-item scalesUsing multi-item scales Close-ended questionnaire items

Rating scales Likert scalesEx. strongly agree strongly disagree;

51

QUESTIONNAIRES

Close-ended questionnaire items Semantic differential scalesEx. Listening comprehension tasks are:difficult__:__:__:_x_:__:__:__ easyuseless__:__:__:__:__:_x_:__ useful Numerical rating scales True- False items

QUESTIONNAIRES

Close-ended questionnaire itemsMultiple choice items Response options should be short. Responses should be put in a natural

order or in random or alphabetical order.

All the options should be grammatically correct with respect to the stem.

Using negative expressions in both the stem and the response options should be avoided.

QUESTIONNAIRES

Close-ended questionnaire items Rank order items Numeric items Checklists

QUESTIONNAIRES

Open-ended questionsDisadvantages They take up more time and restrict

the range of topics the questionnaire can contain.

They are difficult to code in reliable manner.

QUESTIONNAIRES

Open-ended questions

Specific open questions Clarification questions Sentence completion items Short answer questions

QUESTIONNAIRESReferences

Aiken, L. (1997). Questionnaires and inventories: Surveying opinions and assessing personality. New York: John WileyBrown, J. D. (2001). Using surveys in language

programs. Cambridge, UK: Cambridge University Press

Dörnyei, Z. (2003). Questionnaires in second language research. Mahwah, New Jersey: Lawrence Erlbaum Associates

Fink, A. (1995). How to ask surveys questions. Thousand Oaks, California: Sage

QUESTIONNAIRES

Gillham, B. (2004). Developing a questionnaire. London: ContinuumHopkins, K.D., Stanley, J.C. & Hopkins, B.R.(1990). Educational and psychological

measurement and evaluation (7TH Ed.). Englewood Cliffs, NJ: Prentice HallLow, G.(1999).What respondent do with questionnaires: Accounting for incongruity

and fluidity. Applied Linguistics, 20, 503-533

QUESTIONNAIRES

Moser, C. A. & Kalton, G. (1971). Survey methods in social investigation. London: HeinemannNewell, R. (1993). Questionnaires. In N. Gilbert (Ed.) Researching social life. (pp 94-115) London: SageOppenheim, A. N. (1992). Questionnaire design, interviewing and attitude measurement (New Edition). London: Pinter

QUESTIONNAIRES

Robinson, J.P., Shaver, P.R. &Wrightson, L.S. (1991). Criteria for scale selection and evaluation. In J.P. Robinson, P.R. Shaver

& L.S. Wrightson (Eds.) Measures of personality and social psychological attitudes (pp1-16). San Diego, CA: Academic PressRobson, C. (1993). Real world research: A source for social scientists and practitioner-researchers. Oxford: Blackwell

QUESTIONNAIRES

Sanchez, M. E. (1992). Effects of questionnaire design on the quality of survey design. Public Opinion Quarterly, 56,216-217Wilson, N., McClean, S. (1994). Questionnaire design. Newtownabbey, Northern Ireland: University of Ulster

QUESTIONNAIRES

How would you rate this presentation?

Good Very Good Excellent

Thank you for your attention.

Cemal ESKİCİ