reliability & validity qualitative research methods
TRANSCRIPT
Reliability & ValidityQualitative Research Methods
ON
TA
RG
ET
Reliability & Validity• Qual vs. Quant “The quantitative study must convince the reader that procedures have been followed faithfully because very little concrete description of what anyone does is provided. The qualitative study provides the reader with a depiction in enough detail to show that the author’s conclusion ‘makes sense’. . . The quantitative study portrays a world of variables and static states. By contrast the qualitative study describes people acting in events.”
(Firestone, 1987, p. 19)
ON
TA
RG
ET
Reliability & Validity• Challenges1.What can you possibly tell from an n of 1?
2. What is it worth to just get the researcher’s interpretation of what is taking place?
3. How can you generalize from a small, nonrandom sample?
ON
TA
RG
ET
Reliability & Validity• Challenges4. If the researcher is the primary instrument for data collection and analysis, how can we be sure the researcher is a valid and reliable instrument?
5. How will you know when to stop collecting data?
ON
TA
RG
ET
Reliability & Validity• Challenges6. Isn’t the researcher biased and just finding out what he or she expects to find?
7. Without hypotheses, how will you know what you’re looking for?
8. Don’t people lie to field researchers?
ON
TA
RG
ET
Reliability & Validity• Challenges9. Doesn’t the researcher’s presence result in a change in participants’ normal behavior, thus contaminating the data?
10. If somebody else did the study, would they get the same results?
ON
TA
RG
ET
Reliability & Validity• Criteria
Some argue that qualitative researchers should consider validity and reliability from the philosophical assumptions underlying the chosen paradigm.
ON
TA
RG
ET
Reliability & Validity• Qual vs. Quant Criteria Quantitative Qualitative
Internal Validity Credibility
External Validity Transferability
Reliability Dependability
Objectivity Confirmability
ON
TA
RG
ET
Reliability & Validity• Criteria
Credibility: the extent to which interpretations can be validated as true, correct, and dependable
- Is the study believable from the perspective of those observed, does it ring true to the people studied?
- Is the data complete?
ON
TA
RG
ET
Reliability & Validity• CriteriaCredibility: Some argue one can never truly discover the reality of a situation and should look for what’s credible instead.
- Validity is relative- Humans are closer to the truth than if a data collection instrument
had been interjected between us and the participants
ON
TA
RG
ET
Reliability & Validity• CriteriaCredibility: Use triangulation to overcome inherent flaws
- data - investigator - interdisciplinary - theory
ON
TA
RG
ET
Reliability & Validity• CriteriaCredibility:
Have adequate engagement in data collection, a.k.a, saturation
- Holistic - Look for data that supports alternate explanations
ON
TA
RG
ET
Reliability & Validity• CriteriaCredibility: Employ reflexivity
Reflexivity: the process of reflecting critically upon the self as the researcher, the “human instrument”
ON
TA
RG
ET
Reliability & Validity• CriteriaTransferability: degree to which the results can be applied to other settings/situations
- Researcher supplies thick (detailed) descriptions- Pays careful attention to the sample
ON
TA
RG
ET
Reliability & Validity• CriteriaTransferability:
“In qualitative research, a singe case or small, nonrandom, purposeful sample is selected precisely because the researcher wishes to understand the particular in depth, not to find out what is generally true of the many.”
(Merriam, 2009, 224)
ON
TA
RG
ET
Reliability & Validity• Criteria
Dependability: concerned with whether or not the findings can be duplicated/repeated
- Describes changes in the setting and how those changes affected the research
ON
TA
RG
ET
Reliability & Validity• Criteria
Dependability: Difficult because human behavior is constantly changing
- many interpretations- no benchmarks or static means of
measurements- similarity of answers does not ensure accuracy
ON
TA
RG
ET
Reliability & Validity• Criteria
Dependability:
More important to ask if whether the results are consistent with the data collected
ON
TA
RG
ET
Reliability & Validity• Criteria
Dependability:Strategies to help with dependability:
- triangulation- peer examination - investigator’s position- audit trail
ON
TA
RG
ET
Reliability & Validity• Criteria
Dependability:
Audit trail: independent readers can authenticate the findings of a study by following the trail of the
researcher
ON
TA
RG
ET
Reliability & Validity• Criteria
Confirmability: the degree to which the results can be corroborated by others
- Results should be well-reasoned- The results of the study vs. the
researcher’s bias?
ON
TA
RG
ET
Reliability & Validity• Criteria
Confirmability: One concern is reactivity
- How the act of observation changes a situation
ON
TA
RG
ET
Reliability & Validity• CriteriaDifferent criteria apply to
different methods i.e. In narrative analysis look for what “tells” a persuasive story
in a narrative way vs. the thick description needed in an ethnography of a cultural group
ON
TA
RG
ET
Reliability & Validity• Evaluating Research 1.Are the methods of research appropriate
to the nature of the question being asked?
2. Is the connection to an existing body of knowledge or theory clear?
3.Are there clear accounts of the criteria used for the selection of cases for study and of the data collection and analysis?
ON
TA
RG
ET
Reliability & Validity• Evaluating Research 4. Does the sensitivity of the methods
match the needs of the research question?
5.Were the data collection and record keeping systematic?
6. Is reference made to accepted procedures for analysis?
ON
TA
RG
ET
Reliability & Validity
• Evaluating Research
7. How systematic is the analysis?
8. Is there adequate discussion of how themes, concepts and categories were derived from the data?
ON
TA
RG
ET
Reliability & Validity• Evaluating Research
9. Is there adequate discussion of the evidence for and against the researcher’s arguments?
10. Is a clear distinction made between the data and their interpretation?
ON
TA
RG
ET
Reliability & ValidityQualitative Research Methods