research methodology unit five research instruments (data collection) how will the information that...

Research MethodologyUnit Five

Research Instruments (Data Collection)• How will the information

that is needed to answer the research question(s) be obtained?

Forms of Data Collection (special emphasis on standardized surveys and questionnaire construction)

Forms of Data Collection

• Qualitative methods:• guided interview• group techniques• narrative interview

Quantitative methods:• “face-to-face“ interviews • telephone interviews• written interviews (online, “class room“)

Interview methodssystematic / standardized:

ask the same questionsin the same wayto all participants

A.k.a. “Survey Methods”

collecting primary data

Types of Standardized Surveys

• Face-to-face interviews• Telephone interviews• Written interviews• Possibilities and restrictions in the context of

international development?

Overview:

Types of Standardized Surveys – Face-to-face Interviews (1)

(1) Preparing for the interview

(2) Beginning the interaction / dealing with refusals

(3) Asking the questions

(3) Probing, clarifying, reinforcing the respondent

(5) Closing the interview

General procedure:


Show interest Pause Repeat the reply Repeat the question Ask a neutral question

both verbal andnonverbal signals

are important

Interpersonal skills:


Show interest:

An expression of interest and understanding, such as“I see“, “uh-huh“, and “yes“ conveys the message that the response has been heard and more is expected.

Pause:

Silence can tell a respondent that you are waiting to hear more.

Repeat the reply:

This can stimulate the respondent to say more, or to recognize an inaccuracy.


Types of Standardized Surveys – Face-to face Interviews (4)

Repeat the question:

This can help a respondent who has not understood or misinterpreted the question or who has strayed from the question to get back on track. A neutral preface before repeating a question can help you to avoid sounding too mechanical – for example:• “Yes, but …“• “Can you tell me overall …“• “But in general …“• “We‘re just interested in your opinion …“



Ask a neutral question:

Neutral questions can focus the respondent without biasing the response. Some examples:

for clarification:• “What do you mean exactly?“• “Could you please explain that?“


for specificity:• “Could you be more specific about that?“• “Tell me more about that. What, who, how, why?“• “Which would be closer?“ [repeat response options!]


Ask a neutral question:

Neutral questions can focus the respondent without biasing the response. Some examples:

for completeness:• “What else?“• “Con you think of an example?“

for “don‘t know“-answers:• “You can take a minute to think about it.“• “Can you give me your best estimate?“



For any recording of data, “informed consent“ is necessary, that is, respondents must be informed about the purpose and procedure of data recording, and must be asked for permission before the interview takes place.

Types of Standardized Surveys – Telephone Interviews (1)

(1) Introduction:

your (the caller‘s) name the organizational sponsor a procedure for making sure that the correct respondent is on the telephone the purpose and importance of the survey an assurance of confidentiality the option to participate now or at a more convenient time

(2) Survey interview

(3) Interview ending:

interest in obtaining information about the results?

Types of Standardized Surveys – Telephone Interviews (2)

Phone number Call date Time Result code Comments

Result codes : CS = completed survey CB = respondent contacted ; call back at scheduled time NA = no answer LM = left message; call back RF = refusal

Types of Standardized Surveys – Mail Surveys (1)

• This includes ‘regular’ mail surveys as well as online surveys …

Types of Standardized Surveys – Mail Surveys (2): Procedure

Mail survey checklist:

complete planning process develop questionnaire pilot test the questionnaire develop cover letter & info material develop follow-up postcards send advance notice send first mailing track survey responses send follow-up postcard send second mailing

time schedule

about 1 week

about 1 week

about 2 weeks

about 4 weeks

about 2 weeks

Exam

ple

Forms of Data Collection – Possibilities and Restrictions of Surveys

Choosing the appropriate form of data gathering depends on:

(1) Survey content

(2) Respondents characteristics

(3) Time line

(4) Available resources



(1) Survey content

What types of questions will be asked? (How sensitive, and complex are they?)

Would people be more likely to understand and respond to questions presented in print or orally? Anonymously or in person?



(1) Survey content

(2) Respondents characteristics: From whom do you want to collect information?

What is the easiest way to reach them? Do they have certain characteristics that rule out one

method over another (e.g., literacy skills, no telephone)?



(1) Survey content


(3) Time Line:

How quickly do you need results?



(1) Survey content


(3) Time line


Who will work on a survey applied? Will you have help from outside experts in planning and managing a survey?

What facilities will you use for the processing and analysis of data?

How much money do you have to spend?



(1) Survey content


(3) Time line


What are your experiencesand “rules of thumb“ for decision making?

Country-specific aspects?

Using Surveys – “Rules of Thumb”

You should use a face-to-face survey if … :- you don‘t have a list for the population being surveyed- people do not have a telephone / internet access, or cannot be reached by mail- there is a concern that people might not respond willingly or accurately- the questions are complex and need in-person explanations- the budget is not an issue- Cont. on next slide …


You should use a face-to-face survey if … :- you don‘t have a list for the population being surveyed- people do not have a telephone / internet access, or cannot be reached by mail- there is a concern that people might not respond willingly or accurately- the questions are complex and need in-person explanations- the budget is not an issue

Additional advantages:- flash cards can be used to help people understand issues


You should use a telephone survey if … :- you need result relatively quickly- your target audience has telephones- members of your target audience would have difficulties completing a written questionnaire (e.g., due to low literacy rates, or poor eyesight)- your questions are clear and easily comprehensible- the survey is relatively short- staff is available to do the calling


You should use a mail survey if … :

- you have a complete and accurate mailing list-The people you are surveying have some interest in the survey topic and are likely to respond-Your targeted respondents have adequate reading and writing skills-You want to guarantee privacy / anonymity -You need a large number of respondents


You should use an online survey if … :- members of your target audience are reachable via email / are regular internet users- your survey is short and simple- you need results quickly- you have access to the services of a consultant who can help you with the technological aspects of Web survey design


You should use a “class room“ survey if … :

- you want to capitalize on having respondents available

- you need or want immediate feedback

- you have limited resources-you may not be able to reach the people again (e.g., in case of “mobile populations“)

Continued on next slide …

Using Surveys – “Rules of Thumb”You should use a “class room“ survey if … :

- you want to capitalize on having respondents available

- you need or want immediate feedback

- you have limited resources

- you may not be able to reach the people again (e.g., in case of “mobile populations“)

Additional advantages:- you can personally convey the purpose and importance of the survey to the respondents

- you are available to answer to respondents‘ questions

Group Work – Discussion• What are your own

experiences with surveys?

• Where do you see the overall potential and/or the limitations of this specific method of data collection?

Questionnaire Construction

• Types of questions • Formulating questions• Constructing and choosing answer categories• Effects of questions and answer categories• Index and scaling methods …

Overview:

Questionnaire Construction …

Types of questions (1):

• Open-ended questions:

• Closed-ended questions:

allow respondents to give answers

in their own words

allow respondents to give answers

in their own words

provide respondents with preselectedanswer options

provide respondents with preselectedanswer options

Questionnaire Construction …Types of questions (2):



e.g.: ‘How long did you have to search for your first job? ‘

e.g.: ‘How long did you have to search for your first job?’

not at all, I started working immediately a few weeks a few months one year more than one year

1st respondent: ‘I took a long time…’2nd respondent: ‘It was not too bad…’…

“

These questions are useful if you are interested in information which is comparable between your respondents and which can easily be analyzed statistically.




These questions are useful if you are interested in gettingunanticipated answers or in learning about the world as your respondents really see it.


Possible content areas:

• demographics / background• behaviour• knowledge• opinions / attitudes / values • feelings • sensations / experiences

past present future

past present future

• demographics / background• behaviour• knowledge• opinions / attitudes / values • feelings • sensations / experiences


Possible content areas:

best quality of information!

best quality of information!

• objectivity• validity

Questionnaire Construction …Formulating questions (1):

Preparing questions for your questionnaire:

Make a list of what you want to know Check to see if the information you need is already available somewhere else Don‘t ask a question unless it has a use Think about what you will do with each piece of information As you write questions, try to view them through your respondent‘s eye

e.g.: Will the question be seen as reasonable, will it infringe on the respondent‘s privacy, etc.?

Questions Expected results / hypothesis

(indicator/criterion)

Is there any data already available?

Who can provide any information to the question?

Method of data collection

Sample Who can collect the data?

Time schedule of data collection

Feasibility?

Yes

No

Yes

No

Yes

No

Yes

No

Yes

No

Yes

No

Yes

No

Yes

No

Yes

No

Yes

No

Yes

No

Yes

No

Yes

No

Yes

No

Yes

No

Yes

No


Write down your questions!

Write down your questions!

Clarify relation to evaluation criteria!

Clarify relation to evaluation criteria!


Developing useful survey questions:

• Use simple wording, but be precise • Avoid the use of abbreviations, jargon,

or foreign phrases; use clear wording • Don’t use double negatives • Don’t use double-barreled items• Avoid bias / judgments in questions• Choose appropriate wording for “sensitive” topics • Have your questions proof-read by experts and/or

someone representative for your respondents

Questionnaire Construction …Overall design of questionnaire:

• logical order• Same type of questions/responses throughout a series of

questions• Numbered responses should always mean the same• Place sensitive questions toward the middle, demographic

questions at the end …

•PRINT IN EASY-TO-READ

FONT


Constructing/choosing answer categories (1):

• anchors • forced choice vs. multiple choice• open-ended questions

Answer formats:

closed-ended questions

Questionnaire Construction …Constructing/choosing answer categories (2):

Disagree

Somewhat disagree

Neither

agree nor disagree

Somewhat agree

Agree

Irr1 I have difficulty relaxing after work.

Irr2 Even at home I often think of my problems

at work.

Irr4 Even on my vacations I think about my problems at work.

traditionally consist of five answer options

Exam

ple

• anchors

- Likert-Scales

- bipolar scales

Answer formats:

+++ ++

+ - -- --- Irr1 I enjoyed the training module.

Irr2 I found the topic “Labour Market Analysis” very interesting.

Irr4 I will be able to apply many things in my

country of what I have learned about LMA.


consist of an even number of answer options (usually 6)

• anchors

- Likert-Scales

- bipolar scales

Answer formats:

Exam

ple

no

yes

Irr1 I enjoyed the training module.

Irr2 I found the topic “Labour Market Analysis”

very interesting.

Irr4 I will be able to apply many things in my country of what I have learned about LMA.


consist of only 2 answer options

• anchors

- Likert-Scales

- bipolar scales

special case: Guttman-Scale

Answer formats:

Exam

ple


Disagree

Somewhat disagree

Neither

agree nor disagree

Somewhat agree

Agree



at work.


Exam

ple

• „disagree“• „somewhat disagree“• „neither disagree nor agree“• „somewhat agree“• „agree“

• anchors

- Likert-Scales

- bipolar scales

- labeling of anchors

Answer formats:


Disagree

Somewhat disagree

Neither

agree nor disagree

Somewhat agree

Agree



at work.


Exam

ple

• anchors

- Likert-Scales

- bipolar scales


• „every day“• „every week“• „every month“• „every year“• etc.

Answer formats:


Answer formats:

• forced choice vs. multiple choice

• open-ended questions

• anchors

- Likert-Scales vs. bipolar scales



‘Are you a member of the AU? Yes No’Ex

ampl

e


Answer formats:


• anchors




‘A tachometer is used to indicate: (a) road speed (b) oil pressure (c) engine speed (d) distance traveled’

Exam

ple


Answer formats:



• anchors




- „Briefly state three reasons ...“- „List the five most important ...“

Exam

ple


Effects of questions and answer categories:

Number of questions / items: • the more items, the more accurate the results• at least two items per dimension, (better: 4 items)• for specific aspects: one item sufficient

Group Work – Questionnaire Design• Concerning your own Language

Research project: Start developing a questionnaire …

Preparing questions for your questionnaire:

• Make a list of what you want to know• Check to see if the information you need is already available somewhere else• Don‘t ask a question unless it has a use• Think about what you will do with each piece of information• As you write questions, try to view them through your respondent‘s eyee.g.: Will the question be seen as reasonable, will it infringe on the respondent‘s privacy, etc.?

Things to Consider when Conducting Interviews …

• Influences of respondents and interviewers• Controlling these influences• Recruiting interviewers• Interviewer training

Overview:

Influences of Respondents and Interviewers

respondent‘s behaviour

interviewer B

interviewer A

Recruiting InterviewersJob description

• brief description of the purpose of the survey and the role of the interviewer• description of how performance will be monitored and evaluated and what level of independence is expected• duties and tasks: e.g., screening respondents, by telephone or in person; administering surveys;• abilities, knowledge, and skills required

Interviewer Training (1)

• lecture / presentations / discussion• training manual• practice, practice, practice• observation • supervised field training

Essential training methods:

Knowledge about • the role of the interviewer in conducting the interview• the importance of maintaining neutrality during interviews• the survey project and the objectives of the survey• the principle of confidentiality• rules for handling interpersonal aspects during an interview• techniques for probing during an interview

What should be trained?


Skills to • initiate and maintain a conversation with a stranger• respond professionally to unexpected questions and situations• remain neutral by keeping personal opinions out of the interview process• motivate reluctant respondents• deliver the survey in a flowing, conversational manner• record responses and notes correctly• maintain confidentiality

What should be trained?


Interviewer Training (4) Show interest Pause Repeat the reply Repeat the question Ask a neutral question

verbal andnonverbal signals

are important


Qualitative Interviews …

as an Alternative or Supplement to Questionnaires

Qualitative Interviews - Implications:

adaptation to the communication needs no restriction of possible answersexploration (generating hypothesis)examine hypothesiscreate classifications or typologiesexplanation of empirical facts/processes

Advantages in of Qualitative Interviews …

very close to “every day” communicationget a better insight of the topiccommunication with respondent is possible focus of the interview can be changed“extreme” target groups can be examined

Disadvantages &Pitfalls

time consuming process …

Avoid generalizing!Choose quotes carefully!Address limitations and alternatives!

Further Implications of Qualitative Interviews

interviewers have to be trained very wellreduced interviewer and coder objectivitydifficult analysis...

Examples for open-structured questions

• In what ways did this course (not) meet your expectations? • Do you think you will be able to apply what you have learned in this

course in your home country? • Think about the time frame for the course and this module: was it

adequate in any case? What could have been different in your opinion?• Describe what you liked or did not like about the leisure activities and

social events? What did you think about the site visits and excursions? • Could you please describe how your attitude towards the course

developed over time? (Social life, ´yield´ in knowledge and skills, language performance and whatever else you thought concerned your well-being here)

Semi-Structured Interviews

• based on interview guide • produce (narrative) data in the form of notes, a

summary of the individual’s interview, or word-for-word transcripts

• analysis is an iterative process 1. get to know your data 2. focus the analysis3. categorize information4. identify patterns and connections within and between categories5. interpretation - bringing it all together

research methodology unit five research instruments (data collection) how will the information that...

Documents

verschiedenen logo

logo schriftzug

dunkelblau kasten logo

farben nutzen

farben falls

kleienr punkt

eure folien

masteransicht schlieen