managing quality in qualitative research

Managing Quality in Qualitative Research

How to Address Research Quality

Standards, Criteria, Checklists and Guidelines

Concepts of Triangulation

Triangulation of Qualitative and Quantitative Research

Focusing on Process and Transparency

How to Address Research Quality

The issue of quality in qualitative research is located at the crossroads of internal needs and external challenges

• Researchers interest • Can I trust the results that I have created? • Are the methods that I have used in this situation

correct? • If I work with others will our results be similar?

• Funding Institution approach • How is one qualitative approach more better than

another?

• Journal Editor / Conference Editor decisions • Can I trust the rigour of this piece of work? • How do I know that this is the best approach?

• Readers questions • Can I trust what I’m reading? • How much detail did the researcher go into when

they were creating this?

It has become a crucial issue with the establishment of qualitative research as a valid method and in competition with other forms of work

The Independent Samples t-test compares the means of two independent groups in order to determine whether there is statistical evidence that the associated population means are significantly different.

What is the qualitative equivalent?

• Evaluation built on Standardisation • One-Size-Fits-All Approach • Criteria Appropriate to Research or

Research Appropriate to Criteria • How to Assess Research Quality in a Sensitive Way

Different potential ways to approach this

The formulation of research ‘criteria’ is only one solution to the problem

Standards, Criteria, Checklists and Guidelines

Reformulating Traditional CriteriaAim here is to make the whole research process more transparent

• Exact and Coherent Guidelines are used • Interviews are transcribed in a certain

way using specific methods • Distinction between verbatim

statements in field notes and summaries are given

• Reliability of whole process can be seen in reflexive documentation

Reformulating Traditional Criteria

Validating an Interview Situation• Are interviewees given any cause to construct a biased version

of their experiences that doesn’t fit with their views? • Script is analysed to look for these specific points and to

see whether the data is reliable

Communication Validation• Participants can be involved in a secondary research study to

assist in validation of data • Participants can reflect on previous answers and then

better articulate their thoughts

Reformulating Traditional Criteria

Procedural Validation• Validation is looked at for the entire research process and the

different relationships that take place: • Relationship between what is being observed and the larger

cultural, historical, and organisational contexts within which the observations are made

• The relationship among the observer, the observed, and the setting • The perspectives that are used to render an interpretation of any

ethnographic data • The role of the reader in the final product • The issue of author style to render the description or interpretation

Alternative, method-appropriate criteriaInstead of using traditional metrics, we use those that are more credible for qualitative work

• Are findings grounded in the data? • Is sampling appropriate • Are data weighed correctly?

• Are inferences logical? • Analytic strategies applied correctly? • Alternative explanations accounted for?

• Is the category structure appropriate? • Can decisions be justified? • What is the degree of researcher bias? • What strategies were used for increasing credibility?

Traditional criteria tend to miss the specific qualities of qualitative research.

Alternative criteria mostly come without benchmarks for distinguishing good from bad.

Evaluating Grounded TheorySpecific suggestions on evaluating Grounded Theory studies come from Charmaz (2006) who suggests breaking this down into four criteria

• Credibility • Originality • Resonance • Usefulness

Evaluating Grounded Theory

• Has the research achieved familiarity with the setting or topic? • Is the data sufficient to merit your claims?

• Think about range, number and depth of observations contained in the data

• Have you made systematic comparisons between observations and between categories

• Do the categories cover a wide range of empirical observations • Are there strong links between the gathered data and your

argument and analysis? • Has the research provided enough evidence for your claims to

allow the reader to form an independent assessment, and agree with your claims?

Credibility

Evaluating Grounded Theory

• Are your categories fresh, do they offer new insights? • Does your analysis provide a new conceptual rendering of the

data? • What is the social and theoretical significance of this work? • How does your theory challenge, extend, or refine current

ideas, concepts and practices?

Originality

Evaluating Grounded Theory

• Do the categories portray the fullness of the studied experience?

• Have you revealed both liminal and unstable taken-for-granted meanings

• Have you drawn links between larger collectivities or institutions and individual lives?

• Does your grounded theory make sense to your participants or people who share their circumstances? Does your analysis offer them deeper insights about their lives and world?

Resonance

Evaluating Grounded Theory

• Does your analysis offer interpretations that people can use in their everyday worlds?

• Do your analytic categories suggest any generic processes? • If so, have you analysed these processes for

implications? • Can the analysis spark further research in other substantive

areas? • How does your work contribute to knowledge, how does it

contribute to making a better world!?

Usefulness

A Framework for Analysing Qualitative Research Data

Research should be: • Contributory in advancing wider knowledge

or understanding • Defensible in design by providing a research

strategy • Rigorous in conduct through the systemic

collection, analysis, and interpretation of qualitative data

• Credible in Claim through offering well-founded and plausible arguments about the significance of the data generated

Quality in Qualitative Evaluation:A framework for assessingresearch evidence

A Quality Framework

Liz Spencer, Jane Ritchie, Jane Lewis and Lucy DillonNational Centre for Social Research

Government Chief Social Researcher’s Office

418771_CaboffQTY_SHORTV3_AW2 11/7/03 12:39 Page c3

Standards will only be helpful if they apply to qualitative research in general and not to specific approaches

Concepts of Triangulation

Triangulation is the method of location of a point from two others of known distance apart, given the angles of the triangle formed by the three points

x

y°

z°

Triangulation includes researchers taking different perspectives on an issue under study or more generally in answering research questions.

These perspectives can be substantiated by using several methods or theoretical approaches,

Triangulation should produce knowledge at different levels, which means they go beyond the knowledge made possible by one approach and thus contribute to promoting quality in research.

Types of Triangulation

• Data Triangulation • The use of different sources of data • Allows the researcher to reach maximum profit when using the

same methods • Investigator Triangulation

• The use of different observers or interviewers • This is not the sharing of work, it is a systemic comparison of

researchers influences on the issue • Theory Triangulation

• Approaching the data with multiple perspectives and hypothesis in mind

• Points are put side by side to assess their utility and power. • Methodological Triangulation

• Different methods are used to look at the same issue

Within-Methods Triangulation

Method Issue of Research

Approach 1

Approach 2

Example of this technique is the episodic interview which combines questions and narratives.

Within-Methods Triangulation

Interviews can be used to understand the everyday knowledge that people have

Semi-Structured interviews can include narratives where participants talk at length about specific aspects that they have encountered

BUT there is the chance for participants to switch from a narrative discussion to a descriptive, argumentative, or other non-native form of presentation

Episodic Interviews

Within-Methods TriangulationEpisodic Interviews

Situation 1

Situation 3

Situation 2

Episodic Knowledge

Episodic Interview

Narrative Presentation

Argumentative- Theoretical

Presentation

Within-Methods TriangulationEpisodic Interviews

Situation 1

Situation 3

Situation 2

Episodic Knowledge

Concept 1 Concept 2

Sub-Concept 1 Sub-Concept 3Sub-Concept 2

Episodic Interview

Semantic Knowledge

Narrative Presentation

Argumentative- Theoretical

Presentation

Within-Methods TriangulationEpisodic Interviews

Situation 1

Situation 3

Situation 2

Episodic Knowledge

Concept 1 Concept 2

Sub-Concept 1 Sub-Concept 3Sub-Concept 2

Episodic Interview

Semantic Knowledge

Narrative Presentation

Argumentative- Theoretical

Presentation

Argumentations

Subjective Definitions

Examples

Repisodes

Narratives of Situations

Stereotypes

Situational Narratives based on different levels of concreteness

Repisodes regularly occurring situations, no longer based on a clear local and temporal reference

Examples abstracted from concrete situations, and metaphor also ranging to clichés and stereotypes

Subjective Definitions which are asked for within sessions

Linked to Argumentative Definitions of terms, explanations of concepts

Within-Methods Triangulation

Between-Methods Triangulation

Method 1

Issue of Research

Choose at least one method which is specifically suited to exploring the structural aspects of the problem and at least one which can capture the essential elements of its meaning to those involved

Method 2

Triangulation of Qualitative and Quantitative Research

In this module we’ve dealt exclusively with qualitative work. However, it is possible to combine qualitative and quantitative methods in order to increase research quality through triangulation

Quantitative

Qualitative

continuous collection of both sorts of data

Qualitative

Quantitative

continuous field research

wave 1 wave 2 wave 3

Qualitativeexploration

QualitativeQuantitativequestionnaire deepening and

assessing results

Quantitativesurvey

QuantitativeQualitativefield study experiment

Linking Qualitative and Quantitative ResultsResults may converge, that is, are consistent completely, in general, by tendency or partially. For example - answers in a representative survey may match with statements from semi-structured interviews

Results may be complementary. Interviews can provide deeper, more detailed explanations to complement results from a questionnaire

Results may diverge. For example, interviews may produce views that are different when compared to questionnaires. This would cause the need for further research.

Triangulation of qualitative and quantitative research is not per se a quality indicator for qualitative research, but it can contribute to overall quality

Qualitative Research Quantitative Research

Data Sets

Single Cases

Triangulation

Focus Groups

Interviews

Questionnaires

Lab Studies

Positioning Triangulation in the Research Process

Exploration Data Collection Data Interpretation

Generalisation

Focusing on Process and Transparency

Quality in research is produced through the whole process.

Questions to ask for selecting a qualitative research method

I. What do I know about the issue of my study or how detailed is my knowledge already?

II. How developed is the theoretical or empirical knowledge in the literature about this issue?

III. Is my interest in more generally exploring the field and the issue of my study?

IV. What is the background of my study and which methods fit with this?

V. What do I want to get close to in my study? I. Personal Experiences of a group of people / social process II. Reconstruction of underlying research structures

Questions to ask for selecting a qualitative research method

VI. Do I start with a focused research question right away or do I start with an unfocused approach in order to develop the question?

VII. Which aggregate do I want to study: personal experience, interactions or situations, or bigger entities like organisations or discourse?

VIII.Is it more the single case I am interested in or the comparison of various cases?

IX. Which resources (time, money, manpower, skills etc.) are available for my study?

Questions to ask for selecting a qualitative research method

X. What are the characteristics of the field I want to study and of the people in it? What can you request of them and what not?

XI. What is the claim of generalisation of my study?

XII. What are the ethical issues to take into account that are affected by selecting a specific method?

Decide and reflect carefully whether you should use qualitative or quantitative research

• Why qualitative research?

• Which reasons do you have for the one or the other?

• What are your expectations for the qualitative research that you are planning?

Reflect on the theoretical background of your knowledge interest

• What is the impact of your setting on the research?

• How open and closed is your access to what you want to study?

Plan your study carefully, but allow for reconsidering the steps and modifying according to the state of play

• What are the resources available for the study?

• How realistic are the aims of your research in relation to the available resources

• What are necessary and appropriate shortcuts

x

y

z

a

b

Plan your sampling carefully!

• What are your cases?

• What do they stand for?

Think about whom in the field you should contact and inform about your research. Reflect about the relation to establish to field subjects

• What can you learn about your research field and issues from the way you get into the field or are rejected?

Think about why you chose your methods for collecting data

• Is it a decision for a favourite method or for habitual reasons?

• What could or would alternative methods provide?

• What are the impacts of the methods you use on your data and your knowledge?

Plan carefully how to document your data and research experiences

• How exactly should you write your notes?

• What are the influences of the documentation on your research and on your field subjects?

• What are the impacts of the documentation on your methods of collection and analysis?

Think about the way that you want to present what you have experienced in the field and found in your research

• What are the target audiences of your writing?

• What is it mainly you want to convince them about your research?

• What is the impact of the format of your writing on your research and its findings?

Plan how to establish the quality of your research

• What are the quality criteria your research should meet?

• How should these criteria be realised?

• What is their impact on your research and your field subjects or relationships?

How to Address Research Quality

Standards, Criteria, Checklists and Guidelines

Concepts of Triangulation

Triangulation of Qualitative and Quantitative Research

Focusing on Process and Transparency

Information in this presentation was

based on…