Data Collection MethodsReliability & Validity

Research Methology WorkshopKerman University of Medical

Sciences

Data Collection Methods

• Primary: where the investigator is the first to collect the data. Sources include: medical examinations, interviews, observations, etc. Merits: less measurement error, suits objectives of the study better. Disadvantage: costly, may not be feasible.

• Secondary: where the data is collected by OTHERS, for other purposes that those of the current study. Sources include: individual records (medical / employment); group records (census data, vital statistics)

Data Collection Methods

1. Physiological/Biological Measurements

2. Observational Methods

3. Interviews and Questionnaires

4. Records/Available Data

Physiological/Biological Methods

• Requires specialized equipment & training

• Objective• Precise• Sensitive to pick up

subtle variations• Can be expensive,

intrusive

• Physical– Weight, temperature

• Chemical– Blood glucose, Hgb,

lead levels

• Microbiological– Cultures

• Anatomical– Radiological exams

Observational methods

• Use standardized systematic plan for observing and recording data

• Observer may or may not be concealed – One-way mirror to watch children

• Intervention may or may not occur– Handwashing demo, then observe

• Structured: checklist, rating scale• Unstructured: field notes

Records/Available Data

• Use already collected data – Hospital records– Historical documents– Videotapes

• Data may/may not have been originally collected for research

• May need data abstraction tool

Interviews & Questionnaires

• Both ask subjects to report data for themselves

• Must be clearly written so subjects understand intent

Interviews

• Data collector questions subjects

• Open-ended or close-ended questions

• Provides more personal, in-depth information

• Types:– Face to face– Phone

است قهوه نوشيدن است زندگي قهوه نوشيدن زندگيرسيدن و زندگي تشكيل و سال چند گذشت از پس التحصيالن فارغ از گروهي

از يكي ديدن به قبلي قرار طبق اجتماعي و كاري خوب هاي موقعيت به . و گله به زود خيلي ها آن جمعي بحث رفتند خود دانشگاه مجرب اساتيد

شد كشيده زندگي و كار از ناشي هاي استرس از براي .شكايت استادانواع از تعدادي و قهوه قوري يك با و رفت آشپزخانه به ميهمانان از پذيراييبرخي و ساده برخي كه كريستال و پالستيكي سراميكي، هاي خوري قهوه

. ميهمانان از و گذاشت ميز روي را سيني بازگشت بودند قيمت گران . ريختند قهوه خود براي همه آنكه از پس كنند پذيرايي خود از تا خواست

گفت خوري : استاد قهوه همگي كه ايد شده متوجه حتما باشيد كرده دقت اگراند بوده قيمت ارزان و ساده كه آنها و ايد برداشته را زيبا و قيمت گران هاي

. . است بديهي و طبيعي شما براي امر اين البته اند مانده باقي سيني دراست همين هم شما هاي استرس و مشكلات همه است سرچشمه همين هم شما هاي استرس و مشكلات همه را! سرچشمه ها بهترين فقط شما

خواهيد مي خود .برايرا بهتر هاي خوري قهوه آگاهانه اما بود قهوه نوشيدن شما همه اصلي قصدتوجه نيز داشتند برمي ديگران چه آن به حين اين در البته و كرديد انتخاب

اجتماعي. موقعيت پول، شغل، باشد، قهوه زندگي اگر ترتيب اين به داشتيد . و ... حفظ براي ابزاري فقط آنها هستند متعدد هاي خوري قهوه همان و

. آن گاهي، داشت نخواهد فرق آنها در زندگي كيفيت اما اند، زندگي نگهداريموجود قهوه مزه و طعم اصال كه هاست خوري قهوه متوجه ما حواس قدر

فهميم نمي را آن .در ... ببريد لذت خود قهوه نوشيدن از آن جاي به نشود پرت ها فنجان به حواستان من، دوستان پس

Questionnaire Design

• As a rule of thumb, never to attempt to design a questionnaire!

• A questionnaire is very easy to design, but a good questionnaire is virtually impossible to design.

Four key cognitive stages in responsing

• First, the respondent comprehends the item

• Second, he retrieves from memory the information required to answer it.

• Third, he has to make a judgement about the information needed to answer the question.

• Finally, he communicates his response.

Constructing New Instruments

• Methodological Research• Occurs when no tool available or existing tools

are not appropriate• Requires multiple steps

– Formulating items– Testing items– Developing instructions– Pilot testing

• Tools must be valid & reliable

The key to a successful survey is to

ensure that your questions are concise and easy-to-understand and give you valid and reliable information.

Explain your survey

Use incentives where possible

• A six-year-old might cooperate for an hour in exchange for a treat or game.

• Academics and some business audiences are motivated by the promise to share results.

• In general, everyone is more motivated by personal rewards.

Ordering of Questions

• Initial questions should be non-threatening, easy to answer and of interest to respondent

• Group questions by topic– respondents will answer 20 questions on

smoking if they’re sequenced but will complain if you ask 10 questions on smoking, go on to another topic, then go back to smoking questions.

Interview/Questionnaire Design

•Group similar questions together•Keep sensitive questions near the

end•Don’t put two questions into one• Offering “Don’t know” as a response option

• Offering “Couldn’t say” or “Not applicable”

The first several questions should:

– be easy for respondents to understand

– be important to the study's purpose

– engage the attention and interest of your respondent

Get your respondent interested in the survey at the very beginning

• In cases where the topic is already of interest or importance to the respondent, start with general questions, then funnel to more specific ones.

• If the topic is of low importance to respondents, start with specific questions. This gives respondents a frame of reference; then ask broader, more general questions

• Group questions in sections, and position sections or questions in a logical order.

• Introduce new sections with a sentence or phrase so that participants have a chance to switch mental gears.

• Place questions about sensitive issues such as income, sexual habits or drug abuse toward the end of the document, or section.

position bias

• Consider lists of similar items carefully. For example, you might ask about product satisfaction using 10 different attributes (price, availability, delivery, color, fit and so on). The first items on a list such as this usually get treated differently by respondents than the last items do.

• There are several ways to minimize position bias in a self-administered survey. One good way is to print several versions of the questionnaire, each with a different ordering of the items on the list.

• You can also randomly or alphabetically order list items and indicate in the instructions how they are ordered. This reduces the likelihood that respondents will see the first items as most important.

• Put demographic questions at the end of the questionnaire, if possible. There are at least two reasons for this. First, some demographic questions such as age and income can be sensitive.

• Second, it is better to keep respondents' minds on the purpose of the survey at the beginning, while you have their attention.

Interview/Questionnaire Design

Determine your survey’s length:1. A survey targeting children should take no longer than

5 to 7 minutes to complete2. For most populations, self-administered surveys should be no longer than four pages3. Web-based surveys should ideally enable participants

to answer each question in a single step 4. Phone interviews should run no longer than 10

minutes 5. Face-to-face interviews can continue for an hour or

longer

Interview/Questionnaire Design

• Keep questions short: less than 25 words

• Avoid “double-barreled” questions

• Avoid “leading” and “loaded” questions

• Be specific

• KISS Principle- Keep It Simple, Statistician

• Begin with interesting & easy questions

Avoid double-barreled questions

• Double-barreled questions ask for opinions about two subjects at the same time:

• “Are you satisfied with the amount and kind of information you receive from your benefits administrator?”

Avoid loaded questions

• Loaded questions discredit your results. like:

• “Many doctors recommend exercise as a way of maintaining good health. Do you agree?”

Don’t lead with responses

• With interview surveys, avoid beginning questions with a phrase like:

• “Do you very often, frequently, seldom or never...”

• Try to minimize the number of times the respondent or the interviewer has to follow a skip rule. For example "If the answer to question 3 is 'None of the Above,' skip to question 6."

Interview/Questionnaire Design

• Distinguish questions from answers

• Make your survey look attractive

• Conduct a pilot study: Usually, no more than 75§ respondents are needed for a pilot study

Distinguish questions from answers

Aesthetic considerations

• The presentation of a written survey can either encourage or deter a person from responding. Make sure your survey is inviting by:

• Including plenty of white space so the document does not appear intimidating.

• Including color, if possible, but no more than two colors.

• Using no more than two typefaces.

Borrow from the pros

• You can borrow style from the

large national surveys

Sensitive questions

• Casual approach – “Did you by any chance murder your wife?”

• The “Everybody” approach– “As you know, many people have been killing their wives these

days. Do you happen to have killed yours?”

• The “Other People” approach– “Do you know any people who have murdered their wives?”

Pause for reply…”and how about yourself?”

•Avoid questions that have a “right” Avoid questions that have a “right” answeranswer•Avoid multiple or double-barreled Avoid multiple or double-barreled questionsquestions•Keep questions briefKeep questions brief•Use simple termsUse simple terms•Insure response options are Insure response options are mutually exclusivemutually exclusive•Insure that response options are Insure that response options are exhaustiveexhaustive

• Except in rare circumstances, include “Don’t know” and “Refused” as response options.

• Scale response options to create equal distribution of responses

– always, nearly always, sometimes, seldom, never

• Always pilot test new questions

• If necessary, conduct second pilot• Most importantly, be polite and respectful

throughout the interview.

Wait about six weeks to start analysis

• Six weeks after the initial survey was sent, you should have received about as many completed surveys as you ever will.

Some tips• one would expect to be able to use adult questionnaires with adolescents aged 16 and

up with little difficulty.• ‘Good quality’ means responses are both

validvalid (i.e. accurately reflect what the question intended to elicit) and

• reliablereliable (i.e. would be forthcoming if the same survey were conducted under similar conditions again). And

• contain few if any unanswered questions (‘item non-response’).

Some tips

• it is recommended that negative questions

• should be avoided in questionnaires for children; this is contrary to standard practice with adult questionnaires, where an equal balance of positively-and negatively phrased questions is thought to enhance data quality (Borgers et al., 2000).

• (e.g. ‘do you find it difficult to finish your homework within the deadlines given?’).

Some tips

• Completely-labelled scales (i.e. those where every point has a label) have been shown to produce better-quality responses from children than partially-labelled ones (e.g. where only the two extremes are labelled).

• It has also been suggested that verbal labels are more easily understood than numeric ones (Borgers and Hox, 2000)

• As a general rule of thumb, it seems sensible not to offer an explicit ‘don’t know’, except at

questions where you consider it to be a valid response, e.g. where specific knowledge will be required to answer substantively.

• Similarly, where an absence of opinion is deemed to be a genuine possibility, there may be a case for offering this as an explicit answer option, in spite of the fact that it may decrease the amount of useable data collected overall (Borgers and Hox, 2000).

Some tips

• Cognitive pretest: The ‘think aloud’ technique requires the respondent to articulate what he is thinking as he goes about answering the question. For example, he is encouraged to say what he thinks the question means; identify any words or concepts that seem unclear; talk through the process of retrieving and formatting the necessary information; and explain how he arrives at his chosen response. (De Leeuw et al., 2004).

Some tips

Reliability & Validity

Reliability & Validity

• Validity– Does an instrument actually measure what it

is suppose to measure?

• Reliability– Extent to which an instrument yields the

same results repeatedly

• Should be addressed in Methods section

Validity Types

• Based on Judgments– Face Validity

– Content Validity

– Consensual Validity

Validity Types

• Based on Checks against Data– Criterion Validity

(Sensitivitu&specificity)

– Construct Validity