2015-01-26

1

Research in Health Professions Education: Asking Good Questions – Using Appropriate Methods Brian Hodges, MD, PhD, FRCPC Professor, Scientist and Director The Wilson Centre, University of Toronto Richard and Elizabeth Currie Chair in Health Professions Education Research Vice President Education, University Health Netowrk

Overview

•  What is research, what is not? •  Asking a good research

question •  Choosing an approach: the

cycle of research •  Unearthing assumptions •  Making a commitment

Acknowledgements

•  Charlotte Ringsted

•  Albert Scherpbier

•  Doug Buller •  Vicki LeBlanc

Why do research in medical education?

Discuss in pairs Write down 3 motivations for research 1. 

2. 

3. 

Medical Education Research is rapidly developing as a viable and exciting research field

More Journals

More readers Higher quality

2015-01-26

2

More Association and Conferences

•  To improve the experience, knowledge and competence of students and clinicians

•  To improve the care of patients and the health care outcomes generally

•  To deepen the field of health professions education research

Creating Increased Expectations

Health professions education research is undertaken:

•  In naturalistic / opportunistic settings

•  Without a theoretical base

•  With little funding

•  By isolated researchers

•  Published in a dispersed fashion, difficult to find

Why so much poor quality research?

Bordage  2000  

•  The conceptualization and research methods used are limited and narrow

•  Researchers are often isolated, in units with missions for teaching and administration rather than research

•  There is little funding for medical education research

•  Researchers unable to connect their work to a larger “conversation” in an academic discipline

Explanatory Factors

Albert  2005  

1.  Expand research questions and move from local studies to multi-institution and international studies

2.  Replicate existing research in varied settings

3.  Undertaken better controlled studies and support good qualitative research

4.  Created stables sources of funding for medical education research

5.  Formally train researchers

Solutions: According to Bordage

1.  Intensify collaboration between PhD researchers and clinicians interested in education research

2.  Support the diversification of disciplines which play a role in health professions education research

Solutions: According to Albert

2015-01-26

3

•  108 medical faculty with an interest in medical education: survey and focus groups

•  40% had fellowship or masters training

•  Though most involved in a project, few had funding or published their work

•  No significant differences between those with and without additional education training

A masters or fellowship is not enough!

Zibrowski  2008  

Factors for success •  Education research support •  Enhancing colleague interaction •  Ongoing development activities

Barriers •  Protected time •  Access to a context and support staff that

sustains research •  Knowledge of research methodology (Does the

masters/fellowship actually teach research skills?)

Solutions according to Zibrowski

Goals of Education Research

Aims at building on existing knowledge and understanding of learning and education by

–  Studying phenomena, interactions, interventions

–  Formulating models, theories, and predictions

–  Studying what works, why, how and for whom

Related approaches

•  Evaluation, assessment, audit

•  Study of local, concrete problems and phenomena

•  Limitations: –  Do not contribute to advancement of

knowledge or theory building

–  Not part of a larger ”conversation”

Study of local, concrete problems

Evaluation ”What is the effect of this course in my

school regarding satisfaction, usefulness, learning, behaviour?”

Assessment ”What is the learning outcome of this

instruction or course assessed by MCQ or OSCE?”

Audit •  ”How well is our programme working in

our department regarding ...?”

Research

•  Starts with an idea, problem or case •  Relates the idea to conceptual framework •  Develops specific research questions •  Employs research approach, design, and

instruments •  Collects and analyse data •  Interprets results •  Discusses validity and generalisability (or

transferability) of results and practice implications

2015-01-26

4

A research program

•  Problem

•  Research question

•  Study design

•  Data and analysis

•  Understanding refined

Two poles of medical education research

Theory driven research Deepen the knowledge and understanding of learning, teaching, training, education

Applied research Provide evidence of effect, justify initiatives, concepts, technology, expenditure

Players in the complex field Getting published, promoted, famous,

and socialised Albert AHSE 2007 Albert/Rees ME 2010 Slide: Ringsted 2010

Stokes Quadrant Model of Research Not bird watching…

A single study at Kirkpatrick level 1 (Satisfaction) –  We created a course about #fv*7%d –  5 people took it –  3 of them filled out the evaluation form –  They all thought it was great

Sweeping generalization

–  We have shown it is feasible and acceptable to implement a course in #fv*7%d

–  Medical schools around the world should created courses in #fv*7%d

Bohr Quality standards from the primary discipline

Pure  applied  research  

Adv

ance

men

t of k

now

ledg

e

Applicability  

Edison Quality standards for innovation - eg design based research

2015-01-26

5

Use-­‐inspired  basic  research  

Adv

ance

men

t of k

now

ledg

e

Applicability  

Pasteur

Quality research = theory + utility

Eg. Learning in a Simulator

Practical Evaluation

Knowledge Building Research

Process Did the simulation learning program operate well / as intended?

What organizational factors enable the successful development and implementation of simulation learning?

Outcome Were the learning objectives met? Did students learn?

How does simulation-enhanced education impact learners’ self-efficacy and perceived readiness for practice?

From Practical Evaluation to Knowledge Building Research

Asking a Good Research Question

Don’t start with a tool or data

•  “I’d like to do a survey” •  “I want to do a study on OSCEs” •  “I want to do something qualitative” •  “We have a lot of teacher evaluation

data - I’d like to do some research on it”

•  “My chair wants me to do some research on the clerkship”

The complexity of education

Learning task Nature

Complexity

Learner Characteristics

Experience Instruction

Self-study One-on-one Classroom

Setting School

Workplace

Organization Rules and regulations Selection

Teachers Characteristics Qualifications

Curriculum Content Format

Materials Assessment Content Format

Programming

Environment Milieu

Supportive infrastructure

Society Politics Culture

Evaluation Target Format

Consequence

Education Slide: C. Ringsted 2010

Define your problem

•  Think of an idea, problem, issue in education that has been on your mind

•  Write it down

•  Are you passionate about this problem or issue?

•  Explain the problem, and why it

interests you, to a neighbour

2015-01-26

6

Perform a systematic review of literature

Conceptual frameworks

•  Relate your idea, problem, phenomenon to theories of learning and education (or sociology, anthropology, psychology, etc)

•  Identify models that explain underlying mechanisms of a phenomenon or why and how an intervention might work

•  Create a relevant framework for your study –  Appropriate design and valid methods

Find a conceptual framework

•  ”Conceptual frameworks represent ways of thinking about a problem or a study”

•  ”Different frameworks will emphasise different variables and outcomes, and their inter-relatedness”

•  ”Scholars are responsible for making explicit the assumptions and principles contained in the conceptual framework they use in their R&D projects”

Bordage 2009

Learn more about the problem

•  What, if any, investigation of this problem have you done already? –  Reading about it –  Talking with an expert –  Attend a lecture or workshop

•  Has anyone researched this problem before?

•  If you are to pursue a study about this problem, what will you need to know more about?

Generate an answerable question

Change your problem into a question

The nurses don’t allow the medical students to participate in deliveries

Our international graduates

have problems talking to patients

Students are resistant to

feedback about their OSCEs

Why do nurses not involve clinical clerks in the labour and delivery room?

In what ways do international

medical graduates have trouble with patient communication?

How do students perceive

feedback they receive from their OSCEs?

2015-01-26

7

Ensure your question can be studied

•  Too vague: What is the meaning of life?

•  Too ambitious What is the best curriculum to train a professional

physician for the 21st century?

•  The PhD thesis How do methods of assessment of competence

used in countries around the world reflect their history and culture?

•  Reword, refocus, redefine your question as needed

The Cycle of Research

Ringsted’s cycle of research

Experimental

studies Observa

tional

studies

Justifying

Implementing

Translational

studies

Predicting Modelling

Explorative

studies

Conceptual theoretical framework

What is the nature of the problem?

Experimental

studies Observa

tional

studies

Justifying

Implementing

Translational

studies

Predicting Modelling

Explorative

studies

Conceptual theoretical framework

Design Descriptive Qualitative

Psychometric

Descriptive studies

•  Description –  Phenomenon, novel intervention, assessment

method, administrative procedure, organisation, population of individuals

•  Prevalence –  High in young research disciplines – Medical

Education and Emergency Medicine

•  Characteristics –  Reports with or without outcome data –  Make no comparisons

Conceptual theoretical framework

What is qualitative research?

The goal of qualitative research is: –  the development of concepts which help us

to understand social phenomena –  in natural (rather than experimental) settings –  giving emphasis to the meanings,

experiences and views of all participants Pope and Mays BMJ, 1995

Qualitative methods use language-based

rather than numerically-based data

2015-01-26

8

When to use qualitative research?

1.  Preliminary to quantitative research

2.  Supplemental (triangulation) to quantitative data

3.  To explore complex phenomena not amenable to quantitative research

Qualitative and quantitative are not opposites

•  Often artificially seen as opposing poles •  Different, but complementary ways of viewing

similar phenomena Eg. Diabetes and insulin compliance •  Quantitative study: What is the relationship

between non-compliance (as measured by hemoglobin A1C) and disease progression?

•  Qualitative study: Why are patients not compliant? How do they view disease progression? Do they see a relationship with compliance?

Qualitative methodology and method

Methodology •  Historical

method •  Ethnography •  Phenomenology •  Discourse

Analysis •  Grounded

Theory

Method •  Interviews •  Focus Groups •  Case Studies •  Text Analysis •  Observation

• Naturalistic / Interpretive –  Study phenomena in

natural settings –  Meanings participants

bring to phenomena • Specific to context studied • Individuals’ values involved: participants’ & researchers’

Data Collection

Data Management

Data Analysis

Data Sampling

Does it work in a lab?

Experimental

studies Observa

tional

studies

Justifying

Implementing

Translational

studies

Predicting Modelling

Explorative

studies

Conceptual theoretical framework

RCTs & Quasi- experimental

studies

Challenges with RCTs in education

•  Samples –  Size and representativeness

•  Interventions –  Definition of intervention

and control circumstance –  Standardisation of intervention

•  Measurement –  Identification of valid outcome measure –  Appropriate measurement points

2015-01-26

9

Measurement points

•  Before measurement –  Testing might have an effect on learning in

itself –  The test might be remembered

•  After measurement –  Measurement immediate after intervention –  Measurement after a pause, retention test –  Measurement of transfer

•  Another, but similar task •  A more complex task •  Another context

Experimental design

•  Learning is a sustainable change in capacity to perform –  Measure retention and/or transfer

•  Be aware of the testing-effect Randomized design R Ob X1 Oa Or/Ot 4-group Solomon R Ob X2 Oa Or/Ot before-after R X1 Oa Or/Ot retention/transfer R X2 Oa Or/Ot design

Other concerns about RCTs

•  Ethics, blinding, and feasibility – Matched groups, ’waitlist’ controls

•  Validity – RCTs are the best design to find

out if an intervention works – The worst design to find out who

it works for and the long-term effects the intervention

Measurement

•  Test-enhanced learning – Testing has an intrinsic effect on

memory

•  Effect of stress on learning and performance

Roediger HL et al 2006. Kromann C et al. ME 2009 Kromann C et al. ME 2010

LeBlanc AM 2009. Harvey et al. ME 2010 Dieckmann et al. 2007 Stefanidis et al. 2010 Munch-Petersen/ Rosenberg UfL2008

Feedback and learning

•  Training program: Vascular anastomosis, plastic model –  Distributed over 1 week –  Massed in 1 day

•  Measurement points –  Pre-test –  Post-test –  Retention 1 month after –  Transfer, live animal

Pre Post Retention Transfer

Distributed vs. massed training

Glo

bal r

atin

g * * NS

Moulton 2006

Does it work in “real” settings?

Experimental

studies Observa

tional

studies

Justifying

Implementing

Translational

studies

Predicting Modelling

Explorative

studies

Conceptual theoretical framework

RCTs & Quasi- experimental

studies

Cohort studies Case-control Associational

2015-01-26

10

Observational studies

•  Aims at predicting – effect, cause or consequence

– Cohort studies – Case-control studies – Associational studies

Observational studies

+ Exposure

- Exposure

Outcome measure

Cohort studies Start here

Case-control studies Start here

Cross-sectional studies Examine everything

at a certain time

IIIIIIII IIIIIIII

IIIIIIII IIIIIIII III

III

Examples

•  Tamblyn et al. JAMA 2007 –  Prospective cohort study

•  Scores on communication, decision-making predict complaints in later practice

•  Draycott et al. BJOG 2006 –  Retrospective cohort study

•  Effect of training on neonatal outcome

•  Wayne et al. Chest 2008 –  Retrospective case-control study

•  Effect of ACLS training on performance standards

Example: Case-control study

Norman et al. Med Educ 2008

•  RQ –  Does PBL in medical

school predict results of doctors’ performance reviews in practice?

•  Design –  Retrospective study of

1166 Canadian doctors’ performance reviews

108, +PBL

857, - PBL

4%

5%

24%

28%

Result

Concern Excellent

Case-control studies

•  Study the cause or consequence of differences that already exist between or among groups

•  Useful for studies where –  Outcome is categorical

•  Yes/no or present/absent

–  Prevalence of outcome is low •  Rarely observed phenomenon

–  Time delay until outcome is long •  Years

Associational studies

•  Aims at –  Describing phenomena by

identifying the degree of an existing relationship among variables

–  Clarify understanding and seek to predict a certain outcome

•  However .... –  Conceptual, theoretical

framework is crucial to interpretation of cause and effect

r=0.86

r=0.86

2015-01-26

11

Does it work in different settings?

Experimental

studies Observa

tional

studies

Justifying

Implementing

Predicting Modelling

Explorative

studies

Conceptual theoretical framework

Effect studies Implementation

Observation

Translational

studies

Translational research Sung JAMA 2003

•  Political concern –  Scientific discoveries fail to be translated

efficiently into tangible human benefit

•  Three translational blocks (biomedicine)

Basic Research

From Lab to Human

Clinical Research

From study to practice

Happy Healthy People

Block 1

Block 2

Block 3

Effect studies

Practical clinical trials (PCT) Tunis JAMA 2003

•  Approach is based on a need to make decisions •  Select relevant alternative interventions to compare •  Include a diverse population of study participants •  Recruit participants from heterogeneous settings •  Collect data on a broad range of outcomes related to

both persons and systems, including finances

Satisfaction Learning Behaviour Organization Outcome

Jump on and off wherever you like

Experimental

studies Observa

tional

studies

Justifying

Implementing

Translational

studies

Predicting Modelling

Explorative

studies

Conceptual theoretical framework

RCTs and Quasi- experimental

studies

Cohort studies Case-control Associational

Literature Qualitative

Psychometric

Effect studies Implementation

Observation

•  What KIND of approach to you want to use to address your research question?

•  Think about the previous slides, and see if you can identify where in the cycle you wish to be

•  Chat with your neighbour about your thoughts

Unearthing Assumptions

2015-01-26

12

Where do your ideas come from?

•  “It is known that..” •  “Everyone agrees that…” •  “Many studies have shown

that…” •  “I believe that…”

Unearth your assumptions

What is the answer(s) to your question?

Write down what you think it might be (even if you are not sure - even if you just ‘suspect’ them to be true)

It’s OK to do this - really!

• Bias • Perspective • Reflexivity

Making a commitment

Make a commitment

•  You must make a commitment to yourself

•  Write down your research question…somewhere

•  Put it somewhere you will see everyday

Post it somewhere you can see it!

2015-01-26

13

Advancing healthcare education and practice through research

www.thewilsoncentre.ca

References

Bordage G, Dawson B. Experimental study design and grant writing in eight steps and 28 questions. Med Educ. 2003;37:376-385.

Bordage, "Conceptual frameworks to illuminate and magnify", Med Educ 2009; 43:312-319

Beckman TJ, Cook DA. Developing scholarly projects in education: a primer for medical teachers. Med Teach. 2007;29:210-218.

Campbell et al, BMJ 2000. Framework for design and evaluation of complex interventions to improve health.

Cook DA. Avoiding confounded comparisons in education research. Med Educ. 2009;43:102-104.

Cook DA, Bordage G, Schmidt HG. Description, Justification, and Clarification: A Framework for Classifying the Purposes of Research in Medical Education. Med Educ. 2008;42:128-133.

References Eva KW, Lingard L. What's next? A guiding question for

educators engaged in educational research. Med Educ. 2008.

Morrison J. Developing research questions in medical education: the science and the art. Med Educ. 2002;36:596-597.

McGaghie WC, Bordage G, Shea JA. Problem Statement, Conceptual Framework, and Research Question. Acad Med. 2001;76:923-924.

Norman G. RCT = results confounded and trivial: the perils of grand educational experiments. Med Educ. 2003;37:582-584.

Norman & Eva ASME publ. 2008. Quantitative research methods in medical education.

Prideaux D, Bligh J. Research in medical education: asking the right questions. Med Educ. 2002;36:1114-1115.