1 science is the belief in the ignorance of the experts richard feynman, 1965 nobel laureate,...

1

Science is the belief in the ignorance of the experts

Richard Feynman, 1965 Nobel Laureate, Physics

Mark Luborsky MCUAAAR workshop 5/12

Qualitative Mixed Methods And Design Issues

Mark R. Luborsky, PhD

Institute of Gerontology

Wayne State University, Detroit, MI

MICHIGAN CTR. FOR URBAN AFRICAN AMERICAN AGING RESEARCH

Please do not duplicate or use these slides without the express permission of the author.

• “Social science is not a science… they have not yet found out anything, they haven’t got anywhere yet, maybe someday…”

• “If the theory is right, what should happen, and has that happened needs study… in order to do that I must first understand how the answer probably looks… get a qualitative idea of the phenomena before I could get a good quantitative idea… so I have been working with the hope that in the future that rough understanding can be refined into quantified...”

1999, “The Pleasure of Findings Things Out”, Richard Feynman

Mark Luborsky MCUAAAR workshop 5/12 3

m-m M-M

Ascidians (“sea squirts”) are hermaphrodite marine invertebrate and exhibit traits of the earliest ancestors of chordates and vertebrates. … but with a difference


Topics and Take Aways

Formulating best practices for M-M (vs bashing disciplines or any particular project or researcher)

I. Concepts and considerations • Mixed discourse, multiple methods, Mixed Methods• Conceptual heart of project• Crafting the place of your basic scientific question in the body of knowledge,

not the doing (eg aims as: examining, exploring, analysing) mechanical application of sets of procedures and methods

• II. Design considerations

• III. Succeeding in review and publication

Your take aways:• Questions questions questions and concepts• Examples to think with• Pragmatic skills, vocabulary and guidelines

m-m M-M


Envisioning new horizons … the ideal….

• Rich productive collaboration stands on firm disciplinary strengths

• and steps into the shared margins to

discover new opportunities


Multi-disciplinary Collaborative Research …

closer to the reality

9

Is it true you can now get funding …..

• For a qualitative and mixed methods research project?

. . . NO• More easily for epidemiology, physics, and clinical trials?

. . . NO• A study of

an important question, grounded in the scholarship on the topic, with a full and clearly developed design and methods that fit the

aims required to move us forward beyond what we already know.

. . . YES to good scholarship



work to add to our body of knowledge about factors shaping health and effective human functioning in individuals and groups


Bio-monitoring of Persistent Toxic Substances in Michigan Lower-Income Urban Fish Eaters: Venue Based Sampling of Shoreline Anglers Fishing Lower Tittabawassee, Saginaw, and Detroit Rivers” [CDC/EPA/MDCH/ATSDR] Improving community awareness for Detroit River Fish Advisories [Erb Foundation]

M-M. From the laboratory bench to the dinner table.

Multiple concurrent methods

“Developing Meaningful Life: Social Reintegration of Service Members & Veterans with Spinal Cord Injury”[Dept of Defense (DoD) / US Army CDMRP]

QT QL Conjoint

standardized & QL

“Using a System-Wide Database to Reduce Workplace Violence in Hospitals” [ DHHS/CDC/NIOSH]

QT/Epi > QL

“The Meaning of Self-rated Health” (NIH/NIA QT/Epi < QL Conjoint QT & QL

“Downsizing For Household Relocation in Later Life” [NIH/NIA] State-wide survey Michigan Older Adults Relocatoin HRS supplement questions 2011

QT QLQL QT


Secret Lives: Moral Aspects of Core Scientific Constructs for Design & Analyses

• Power• Validity• Reliability• Sample/Sampling• Representation/representative• Control • Dependent variable• Independent variable

The outside “second life” in daily society and talk connotations of these uses bring in enduring cultural dilemmas and challenges for scientists working to achieving adequacy for each of those constructs

Luborsky, M. & R. Rubinstein. 1995 Sampling in Qualitative Research: Rationale, Issues, Methods Research on Aging 17(1):89-113

Luborsky, M. 1994 Identifying Themes and Patterns. In J. Gubrium, & A. Sankar (eds) Qualitative Methods in Aging Research. NY: SAGE

Mixed discourse, m-m, M-MTwo examples of pervasive substantial confounds

Identification of themes and patterns (Luborsky 1994)• Themes • Patterns

• Focus group analysis (Agar & Hobbes 1996)• Themes• Negotiation of relationships and values

• Consider: William Dressler, arterial hypertension in AA,


14

“Self rated health” powerful predictor of mortality in later life across the globe. But, what's this really about


15

code = “Physical Body Only” (Idler & Benyamini 2002)

< ‘chunked’ text to reveal narrative traits (evaluation) and unmet social contract >

1. Eleven years ago I had a mastectomy. 2. I’ve had a very bad case of high blood pressure for many years.3. For 15 years, I’ve had diabetes.4. I have a sciatic nerve problem, which causes my legs to hurt when I

walk. 5. The last doctor I went to wanted to do surgery to remove the nerves

around my spine. 6. I refused and said I’d come back when I couldn’t walk at all. 7. He said if I didn’t want surgery, he couldn’t help me.

[Original] Eleven years ago I had a mastectomy. I’ve had a very bad case of high blood pressure for many years. For 15 years, I’ve had diabetes. I have a sciatic nerve problem, which causes my legs to hurt when I walk. The last doctor I went to wanted to do surgery to remove the nerves around my spine. I refused and said I’d come back when I couldn’t walk at all. He said if I didn’t want surgery, he couldn’t help me.


16

[‘chunked’ text reveals narrative traits (evaluation) and unmet social contract]

a) Eleven years ago I had a mastectomy. b) I’ve had a very bad case of high blood pressure for many

years.c) For 15 years, I’ve had diabetes.d) I have a sciatic nerve problem, which causes my legs to hurt

when I walk. e) The last doctor I went to wanted to do surgery to remove

the nerves around my spine. f) I refused and said I’d come back when I couldn’t walk at

all. g) He said if I didn’t want surgery, he couldn’t help me.


17


18

What is the all noise about qualitative and mixed methods?

Socio-cultural distribution of • forms of knowledge construction, and • of disciplinary ‘ethnic’ tension about it

DNA mapping


Concepts and Considerations

• History not new. Stouffer et al. “The American Soldier” 1940s (Likert, etc)

• Social science M-M is affliction not in other fields inquiry/discovery

• Compatibility and contentions between paradigms within a field, tensions between discipline tensions (NIA/NINR review).

• Motivation – trendiness will not carry your. Establish scientific need, acknolwedge plus and minuses and address

• Just because you can: quantify qualitative, or qualitative describe experience -- doesn’t mean it will adequately capture the construct or provide an real scientific impact


Is your project or article ready for review?


General Considerations

• No one preferred. Help reviewers see multiplicity of approaches

• Some are more palatable to the NIH’s positivistic quantitative approach

• Review panels less biased against a method than incompletely specified proposals

• NIH is a public health agency: it is accountable for improving public health

• Researcher bears burden of proof regarding accountability and usefulness of methods


What criteria do I use as editor of MAQ and NIH reviewer

• I go to the “methods” section to see if quantitative and qualitative data are collected.

• I look in the “analyses” “results” and “discussion” sections for some connection, merging, or embedding of one type of data in the other.

• I look in the “methods” discussion for rigorous quantitative and qualitative procedures.

• I look throughout the manuscript for how the author has positioned the study within the field of mixed methods research and how the study contributes a thoughtful, unique contribution to mixed method literature.

• I look in the “background and significance” and throughout the manuscript for the paradigm stance of the author.

• I look at the references to see if the author cites and is familiar with the mixed methods literature.

• I look in the “introduction” for the rationale for using mixed methods to study the research problem.

• •


A few journals dedicated to mixed methods …

• Journal of Mixed Methods• Field Methods• Quality and Quantity• International Journal of Multiple Research Approaches

(on-line)

• Other journals that publish mixed methods studies

(e.g., International Journal of Social Research Methodology)

• Special issue journals


Practice Targets for Qualitative M-M Applications to NIH

• INTERNAL CONSISTENCY!! Across problem, aims, literature, design, methods

• Clarity (simple), specificity, and justification for methods are key

• “Watch your language” especially shared terms• Attend to sampling, sample collection and maintenance

• Design. Define, then explore strengths and limits• Mixed methods. How are they combined in process and final product

• Budget for expertise, resources, time: add if needed• Anticipate questions and answer them in writing


14 Common Limitations in Proposals With Qualitative Methods

• Aims not specific, err by stating procedures (e.g conduct ethnography, do interviews, observe…)

• No compelling answer to the “So what” question, need to address the significance of proposed findings

• Adversarial and dogmatic vs informed and reasoned discussion about choice for this particular topic and study

• Significance too general, lacks detail on specific factors and topics. Over-developed “Background” relevance

• State of knowledge used to structure proposal biased by only covering studies using your methods

• Methods not well justified: missing pieces, incomplete descriptions - especially “measures”

• Methods are internally inconsistent


14 Common Limitations (continued)

• Limitations of procedures not address (sample, data collection, analyses)

• Sampling decision not explained with sound rationale• Data collection procedures not explained in adequate

detail (when, where, how, by who) and access to population.

• Data analyses superficially described• Fail to demonstrate expertise in methods, including

technology in using software• Mixed methods fail to adequately link research aims,

questions, data, analyses• Unequal attention to qualitative and quantitative

methods when both used


Imigani y’ikinyarwanda / “hardship and opportunities”In season, the lake’s bounty for the fish and edges shrink, once hungry crocodiles

enjoy plenty, until the next rains, when lakes grow and again they are hungry while others have

plentyKofi Kankolongo

1 science is the belief in the ignorance of the experts richard feynman, 1965 nobel laureate,...

Documents