the quality of medical advice: vignettes (and more) jishnu das world bank
TRANSCRIPT
The Quality of Medical Advice: Vignettes (and more)
Jishnu Das
World Bank
Saving Meena: Story of a death foretold
From the local to the global
Poor Health
Life Expectancies (Ethiopia 53; Kenya 54; Zambia 42 in 2007)
U5 Mortality (per 1000 born): Ethiopia 127 ; Kenya 114; Zambia 174)
Many countriesFigure 2: Infant Mortality Rates in Selected Countries
050
100
Infa
nt M
orta
lity
Rat
e pe
r 10
00 b
irths
Mol
dova
Rep
ublic
of
2005
Vie
tnam
2002
Jord
an20
02
Arm
enia
2005
Phi
lippi
nes
2003
Dom
inic
an R
epub
lic20
02N
icar
agua
2001
Egy
pt20
05
Par
agua
y 19
90
Indo
nesi
a20
02/2
003
Mor
occo
2003
-200
4
Erit
rea
2002
Bol
ivia
2003
Indi
a 20
05
Mad
agas
car
2003
/200
4
Sen
egal
2005
Gha
na20
03
Nep
al20
01
Ban
glad
esh
2004
Tan
zani
a20
04
Cam
eroo
n20
04
Con
go (
Bra
zzav
ille)
2005
Mal
awi
2004
Eth
iopi
a20
05
Ken
ya20
03
Bur
kina
Fas
o20
03
Rw
anda
2005
Ben
in20
01
Leso
tho
2004
Gui
nea
2005
Zam
bia
2001
/02
Nig
eria
2003
Moz
ambi
que
2003
Cha
d20
04
Mal
i20
01
Data from DHS Surveys
IMR: Selected Countries
US
200
0 (A
ll)
US
200
0 (A
fric
an-
Am
eric
ans)
050
100
Infa
nt M
orta
lity
Rat
e pe
r 10
00 b
irths
Mol
dova
Rep
ublic
of
2005
Vie
tnam
2002
Jord
an20
02
Arm
enia
2005
Phi
lippi
nes
2003
Dom
inic
an R
epub
lic20
02N
icar
agua
2001
Egy
pt20
05
Par
agua
y 19
90
Indo
nesi
a20
02/2
003
Mor
occo
2003
-200
4
Erit
rea
2002
Bol
ivia
2003
Indi
a 20
05
Mad
agas
car
2003
/200
4
Sen
egal
2005
Gha
na20
03
Nep
al20
01
Ban
glad
esh
2004
Tan
zani
a20
04
Cam
eroo
n20
04
Con
go (
Bra
zzav
ille)
2005
Mal
awi
2004
Eth
iopi
a20
05
Ken
ya20
03
Bur
kina
Fas
o20
03
Rw
anda
2005
Ben
in20
01
Leso
tho
2004
Gui
nea
2005
Zam
bia
2001
/02
Nig
eria
2003
Moz
ambi
que
2003
Cha
d20
04
Mal
i20
01
Data from DHS Surveys
IMR: Selected Countries
US
200
0 (A
ll)
US
200
0 (A
fric
an-
Am
eric
ans)
Note: Data are based on the Demographic and Health Surveys (ORC Macro, 2007. Measure DHS STATcompiler) for all countries except India, which is based on the National Family Health Survey (2005). Countries that are cross-shaded form the basis of this paper. No data are available for Paraguay after 1990.
Why?
Infectious diseases: HIV/AIDS, Malaria, Diarrhea, TB
Kenya: TB incidence rate is 62/10,000 (one of the highest in the world)
Partly to do with low incomes But see Riley on countries that improved
health outcomes at low-income levels Partly to do with access to care
Particularly a problem in rural areas
Some quotes There is an obvious difference between rural and urban
postings. Working in rural areas involves helping the poor… in urban areas one can learn, have more income, have a good school for one’s children.
Doctor in Ethiopia
It is Siberia!. Doctor in Addis
Ababa There is no plan of development for a doctor in the rural
areas; it is as if you are lost. The lack of career development means that it is as if you are punished.
Doctor in Rwanda Promotion is as important as remuneration because you
cannot stay in the same place forever. Doctor in Ghana
Sourced from Serneels and others
And yet… Figure 1: The Use of Health Facilities for ARI
020
4060
80P
ropo
rtio
n U
sing
a H
ealth
Fac
ility
Ch
ad20
04
Eth
iopi
a2
005
Ba
ngla
des
h2
004
Ne
pal2
001
Rw
and
a2
005
Nig
eria
200
3A
rme
nia
2005
Mal
aw
i200
4
Mal
i200
1
Mor
occ
o2
003
-20
04
Bu
rkin
a F
aso
200
3
Ca
me
roon
200
4B
eni
n2
001
Gu
ine
a2
005
Eri
tre
a2
002
Gh
ana
200
3
Se
nega
l200
5
Co
ngo
(Bra
zzav
ille
)20
05
Ke
nya
200
3P
ara
gua
y 19
90
Bo
livia
200
3
Mad
agas
car
200
3/2
004
Ph
ilip
pine
s2
003
Mo
zam
biqu
e20
03
Ta
nzan
ia20
04
Les
oth
o2
004
Nic
arag
ua2
001
Ind
ones
ia2
002
/200
3
Eg
ypt
200
5In
dia
200
5
Mo
ldov
a R
epu
blic
of
2005
Do
min
ica
n R
epu
blic
200
2
Vie
tnam
200
2
Za
mbi
a2
001
/02
Jord
an
2002
Data from DHS Surveys
Use of Health Facilities for ARI
020
4060
80P
ropo
rtio
n U
sing
a H
ealth
Fac
ility
Ch
ad20
04
Eth
iopi
a2
005
Ba
ngla
des
h2
004
Ne
pal2
001
Rw
and
a2
005
Nig
eria
200
3A
rme
nia
2005
Mal
aw
i200
4
Mal
i200
1
Mor
occ
o2
003
-20
04
Bu
rkin
a F
aso
200
3
Ca
me
roon
200
4B
eni
n2
001
Gu
ine
a2
005
Eri
tre
a2
002
Gh
ana
200
3
Se
nega
l200
5
Co
ngo
(Bra
zzav
ille
)20
05
Ke
nya
200
3P
ara
gua
y 19
90
Bo
livia
200
3
Mad
agas
car
200
3/2
004
Ph
ilip
pine
s2
003
Mo
zam
biqu
e20
03
Ta
nzan
ia20
04
Les
oth
o2
004
Nic
arag
ua2
001
Ind
ones
ia2
002
/200
3
Eg
ypt
200
5In
dia
200
5
Mo
ldov
a R
epu
blic
of
2005
Do
min
ica
n R
epu
blic
200
2
Vie
tnam
200
2
Za
mbi
a2
001
/02
Jord
an
2002
Data from DHS Surveys
Use of Health Facilities for ARI
U.S
. (19
88)
020
4060
80P
ropo
rtio
n U
sing
a H
ealth
Fac
ility
Ch
ad20
04
Eth
iopi
a2
005
Ba
ngla
des
h2
004
Ne
pal2
001
Rw
and
a2
005
Nig
eria
200
3A
rme
nia
2005
Mal
aw
i200
4
Mal
i200
1
Mor
occ
o2
003
-20
04
Bu
rkin
a F
aso
200
3
Ca
me
roon
200
4B
eni
n2
001
Gu
ine
a2
005
Eri
tre
a2
002
Gh
ana
200
3
Se
nega
l200
5
Co
ngo
(Bra
zzav
ille
)20
05
Ke
nya
200
3P
ara
gua
y 19
90
Bo
livia
200
3
Mad
agas
car
200
3/2
004
Ph
ilip
pine
s2
003
Mo
zam
biqu
e20
03
Ta
nzan
ia20
04
Les
oth
o2
004
Nic
arag
ua2
001
Ind
ones
ia2
002
/200
3
Eg
ypt
200
5In
dia
200
5
Mo
ldov
a R
epu
blic
of
2005
Do
min
ica
n R
epu
blic
200
2
Vie
tnam
200
2
Za
mbi
a2
001
/02
Jord
an
2002
Data from DHS Surveys
Use of Health Facilities for ARI
020
4060
80P
ropo
rtio
n U
sing
a H
ealth
Fac
ility
Ch
ad20
04
Eth
iopi
a2
005
Ba
ngla
des
h2
004
Ne
pal2
001
Rw
and
a2
005
Nig
eria
200
3A
rme
nia
2005
Mal
aw
i200
4
Mal
i200
1
Mor
occ
o2
003
-20
04
Bu
rkin
a F
aso
200
3
Ca
me
roon
200
4B
eni
n2
001
Gu
ine
a2
005
Eri
tre
a2
002
Gh
ana
200
3
Se
nega
l200
5
Co
ngo
(Bra
zzav
ille
)20
05
Ke
nya
200
3P
ara
gua
y 19
90
Bo
livia
200
3
Mad
agas
car
200
3/2
004
Ph
ilip
pine
s2
003
Mo
zam
biqu
e20
03
Ta
nzan
ia20
04
Les
oth
o2
004
Nic
arag
ua2
001
Ind
ones
ia2
002
/200
3
Eg
ypt
200
5In
dia
200
5
Mo
ldov
a R
epu
blic
of
2005
Do
min
ica
n R
epu
blic
200
2
Vie
tnam
200
2
Za
mbi
a2
001
/02
Jord
an
2002
Data from DHS Surveys
Use of Health Facilities for ARI
U.S
. (19
88)
Note: The graph shows the % of respondents who said that they had taken their child to a health facility for the treatment of an Acute Respiratory Infection. Data are based on the Demographic and Health Surveys (ORC Macro, 2007. Measure DHS STATcompiler) for all countries except India, which is based on the National Family Health Survey (2005). Countries that are cross-shaded form the basis of this paper. No data are available for Paraguay after 1990. The equivalent number for the US is not easily available. We use instead health seeking behavior among children for pharyngitis as reported in Stoddard, Jeffrey J., Robert F. St. Peter, and Paul W. Newacheck. 1994.
In many countries around the world, people use facilities a lot more than they do in the US
And yet…
010
2030
To
tal N
um
ber
of P
rovi
der
s
19 24 21 36 6 14 26 29 1 12 25 28 30 4 17 32 34 11 15 7 10 13 27 20 23 31 33 38 8 18This is ongoing work
Deep Rural Madhya PradeshHow Many Medical Care Providers
In deep rural Madhya Pradesh, one of the states in India with the worst HD outcomes, there are 11.3 medical care providers accessible for representative rural households who do not live close to national highways or close to urban centers
And yet…
020
4060
8010
0H
ous
eho
lds
per
Pro
vid
er
28 33 4 38 13 8 18 26 31 6 27 15 11 10 23 17 20 36 29 1 25 12 32 7 30 34 19 21 24 14This is ongoing work
Deep Rural Madhya PradeshHow Many Medical Care Providers
This is not because these villages are huge: for these villages, there are around 20 households per provider
New Studies on Doctor Visits
In 5 countries studied, the poor visit doctors almost as much as the rich Kenya: 60% of poorest quintile seek care
when sick relative to 78% of richest quintile In urban India, new survey methods show
that the poor visit doctors more than the rich The difference arises because of shorter recall
periods In rural India, households visit doctors twice
as much as in the United States
Do the poor really go to doctors less than the rich?
Experimental data from Delhi show that when households are asked on the basis of monthly recall, self-reported doctor visits fall by 65 percent for the poor compared to weekly recall. In fact, the poor go to doctors more than the rich. We don’t know what happens in other countries, because weekly recall questionnaires are very rare!
One alternative
Health outcomes may be related to the quality of health care that people receive
Structural Quality
Quality traditionally defined as structural quality—state of infrastructure, availability of medicines. This is clearly informative BUT Its not correct when demand is a factor (medicines) The quality of medical advice may be equally (or
even more) important The relationship between quality of advice and
structural quality is weak (India, Indonesia, Tanzania)
The Quality of Medical Advice: New evidence Since 2002, team working on the quality of
medical advice at The World Bank and the University of Maryland
Basic Idea What can be measured, and how? How do these measurements help us
understand the quality of medical advice? How can the quality of medical advice be
improved?
Where we are
Quality of medical advice can be decomposed into Competence: What does a doctor or medical care
provider know about how to treat an illness Practice: What does a doctor or medical care
provider do when faced with an illness We systematically find that the two are very
different. This has important policy implications Improving competence is about training Improving practice (given a competence) is about
getting doctors to exert greater effort Countries: Tanzania, Urban India, Indonesia,
Paraguay Ongoing: Rwanda, Rural India, Argentina
Remainder of Presentation
Basic facts about competence Basic facts about practice More interesting facts Lessons learnt (thus far!)
What are Vignettes
Standardized mix of (in our case) 5 cases One interviewer is `patient’; other is `recorder and observer’ Child with Diarrhea
“My child has been suffering from diarrhea for the last two days, and I do not know what to do”
History Doc gets 1 point if he asks about last urination (for
instance) Examination
Doc gets 1 point if he asks about temperature (IF asked, recorded responds `98.8 F’)
Treatment Doc prescribes treatment: treatment is graded by
independent raters in South Asia and the US
Scoring Vignettes
The questions that doctors ask are compared to a checklist of essential procedures
An aggregate “index” is compiled that accounts for the different “difficulties” of different checklist items
We call this index “competence” We normalize mean = 0 and standard-
deviation=1 Moving from 0 to 2 moves you from 50th percentile to
95th percentile
Vignettes: Advantages
Advantages Standardized case-load (same 5 cases to all docs) Standardized patient (specify that patient will
`comply fully with all medications and tests’
What they know0
2040
6080
100
Pro
bab
ility
of N
on-
Har
mfu
l Tre
atm
ent
Diarrhea PreEclampsia TB ViralPharyngitis
Leas
t Com
pete
nt
2nd
Quint
ile
Avera
ge C
ompe
tenc
e
4th
Quint
ile
Mos
t Com
pete
nt
Leas
t Com
pete
nt
2nd
Quint
ile
Avera
ge C
ompe
tenc
e
4th
Quint
ile
Mos
t Com
pete
nt
Leas
t Com
pete
nt
2nd
Quint
ile
Avera
ge C
ompe
tenc
e
4th
Quint
ile
Mos
t Com
pete
nt
Leas
t Com
pete
nt
2nd
Quint
ile
Avera
ge C
ompe
tenc
e
4th
Quint
ile
Mos
t Com
pete
nt
Delhi DoctorsWhat doctors know
Public or Private?0
.1.2
.3.4
.5D
en
sity
-2 -1 0 1 2C om petenc e
His togram K ernel Dens i ty
P ub l i c--A l l M B B S
0.1
.2.3
.4.5
De
nsit
y
-2 -1 0 1 2C om petenc e
His togram K ernel Dens i ty
P riva te --M B B S
0.1
.2.3
.4.5
De
nsi
ty
-2 -1 0 1 2C om petenc e
His togram K ernel Dens i ty
P riva te --Non -M B B S
0.1
.2.3
.4.5
De
nsit
y/P
erc
ent
-2 -1 0 1 2C om petenc e
P ubl ic P roviders P rivate--MB B S
P rivate--Non-MB B S
A l l P rovide rs
Distribution of Competence by Qua lification
An MBBS is the formal Indian medical degree—roughly the equivalent of an MD in the US
Distribution of Competence
-.5
0
.5
Low Middle High
Competence Across Neighborhood Incomes
Overall Competence Public PHCs
Regression ModelsTable 5: What Explains Quality? (1) (2) (3) (4) Income and
Institution Income, Institution and Qualification
All variables excluding price
All variables including price
% of poor households in community
-0.013 (0.002)***
-0.007 (0.002)***
-0.009 (0.002)***
-0.009 (0.002)***
% middle income households in community
-0.011 (0.003)***
-0.006 (0.003)*
-0.004 (0.003)
-0.003 (0.003)
Public Doctor 0.451 (0.172)***
-0.185 (0.175)
-0.174 (0.171)
0.365 (0.348)
MBBS Degree Holder 1.132 (0.153)***
0.957 (0.156)***
0.848 (0.171)***
Tenure in Locality -0.020 (0.007)***
-0.019 (0.007)***
Usual Price Charged (from census)
0.144 (0.080)*
Controls for Origin of Provider?
NO NO YES YES
Constant 0.616 (0.154)***
-0.194 (0.175)
-0.399 (0.255)
-0.804 (0.347)**
Observations 204
204 194 187
R-squared 0.19
0.36 0.44 0.45
R-squared corrected for measurement error
0.28 0.53 0.64 0.65
Results very similar in other countries• More competent doctors in
urban areas• More competent doctors in
richer areas (within urban and within rural areas)
• More training increases competence
Competence across countries
Measuring practice
Sit in the doctors office
Record details about every interaction between the doctor and patient Time History taking Physical exams Drugs prescribed
Photo Credit: Ken Leonard
Practice: Some numbersTable 3: International Comparisons of Effort
Sample Effort Categories or Country Time Spent Questions asked of Patient
% Who do Physical Exams
Poly-pharmacy (Total number of medicines given)
Delhi
Doctors who exert low effort 1.9 1.36 14 2.13
Doctors who exert medium effort 3.36 2.94 78 2.72
Doctors who exert high effort 6.15 5.32 98 3.05
All Doctors 3.80 3.20 63 2.63
Paraguay
Doctors who exert low effort 5.79 5.33 1.38 1.36
Doctors who exert medium effort
7.90 7.50 2.93 1.55
Doctors who exert high effort 11.34 11.91 3.64 1.65
All Doctors 8.33 8.23 2.65 1.52
Tanzania
Doctors who exert low effort (25th Percentile) 3 2 0 N/ A
All Doctors 6.32 3.96 1.51 N/ A
International Comparisons
Tanzaniac (1991) 3.0 N/ A N/ A 2.2
Nigeriac 6.3 N/ A N/ A 2.8
Malawic 2.3 N/ A N/ A 1.8
UKd 9.4 N/ A N/ A N/ A
Notes: We divide doctors by terciles of effort in India and Paraguay, and the 25th percentile versus all doctors for Tanzania. The data are based on the following sources India: Das and Hammer (2007); Paraguay: Das and Sohnesen (2007); Tanzania: Leonard (Mimeo); International Comparisons: Hogelzeir and others (1993) and Deveugele and others (2003).
Another look at practice
0
1
2
3
4
5
6
7
time questions exams
low effortmediumhigh
Less than 2 minutes Just one question
Almost none!
Public-Private Again
4.22
1.56
4.76
2.28
4.06
2.19
5.39
3.1
5.89
3.26
0
2
4
6
Tim
e in
Min
utes
Lowest Second Median Fourth Top
Privat
e
Public
Privat
e
Public
Privat
e
Public
Privat
e
Public
Privat
e
Public
Got Time?
Public-Private AgainTable 4: Practice and Competence across Sectors
Quintiles of ability Type of provider Total history questionsProbability of. examination
(percent) Time spent (minutes) Fees charged (Rs.)Lowest Private 2.93 70 4.22 21.5
Public 1.72 28 1.56 0
2nd Group Private 3.37 71 4.76 34.6
Public 1.88 41 2.28 0
Median Private 3.55 75 4.06 32.8
Public 2.17 42 2.19 0
4th Group Private 3.67 81 5.39 44.0
Public 3.55 41 3.10 0
Highest Private 4.71 81 5.89 57.3
Public 4.04 72 3.26 .01
Note: This table disaggregates provider practice (observed in the clinic) by competence measured in the vignettes and sector (public/private). History questions refer to the number of questions regarding the illness that the provider asked the patient. An examination consists of any physical contact between the provider and the patient or the use of measuring instruments, such as a thermometer, sphygmomanometer, or stethoscope. Note that an examination only implies that the device was used, not that is was used correctly. Fees charged refers to the total payment at the end of the interaction. Finally, public providers are those who were observed in their public practice and need not be providers who work only in the public sector.
Private MBBS Public MBBS Private, No MBBS
What they know, what they do40% of essential questions asked
Perc
enta
ge o
f Ess
entia
l Tas
ks C
ompl
eted
Lost Training: Tanzania
010
2030
4050
6070
Per
form
ance
(%
of
requ
ired
ite
ms)
0 10 20 30 40 50 60 70 80 90Competence (% of required items)
Individual clinician's competence and performancePredicted quadratic relationship of competence to performance (Public)
Predicted quadratic relationship of competence to performance (Non-Public)
Performance = Competence
Sourced from Ken Leonard
Lost Training: India0
.2.4
.6.8
1W
hat
they D
o
0 .2 .4 .6 .8 1What they said they would do
What they know W hat they Do: PrivateWhat they do: Public
Rotating The Curve
Lost Training: Private
Additional Lost Training: Public
These doctors are operating at the frontier of their knowledge
0.2
.4.6
.81
What
they D
o
0 .2 .4 .6 .8 1What they said they would do
What they know W hat they Do: PrivateWhat they do: Public
Rotating The Curve
Lost Training: Private
Additional Lost Training: Public
0.2
.4.6
.81
What
they D
o
0 .2 .4 .6 .8 1What they said they would do
What they know W hat they Do: PrivateWhat they do: Public
Rotating The Curve
Lost Training: Private
Additional Lost Training: Public
These doctors are operating at the frontier of their knowledge
Training or Effort?
Because of lost training We simulate that the impact of training is very
small relative to improvements in effort
The big caveat?
Is it that public doctors put in a lot less effort because they see many more patients? We find more effort in India in hospitals, which
typically see many more patients Two new and incredibly sad pieces of
research
Case loads and doctor shortage Maestad and others (2009)
went and sat in many doctors clinics in rural Tanzania
“The average doctor sees 18.5 patients per day and total time use is 5.7 mins per patient. The doctor completes 22% of essential tasks”
That’s less than 2 hours a day in an 8 hour day
There is no relationship between caseload and doctor effort!
The Hawthorne Effect35
4045
5055
6065
7075
Per
cent
age
of I
tem
s C
orre
ct
-10 -5 0 5 10 15Number of Previous Consultations Under Observation
doctor observed from t = 1 doctor never observed
Effort jumps when doctors are observed
Doctors significantly increase their effort in Tanzania when they know they are being observed with no detrimental effect on patient (Leonard, 2007) Based on Ken Leonard (2007) in Journal of Health Economics
Improving health outcomes
New research suggests that improving effort could also improve outcomes Bjorkman and Svensson, 2008: Uganda Rwanda Pay-for-Performance experiment:
Ongoing, Rwanda
Summary
The quality of medical advice is key and understanding the levels and correlates of quality is an urgent priority
Measuring either competence or practice is a good start But measuring both vastly increases our
understanding of what is going on and where the policy levers may be
Resources exist to help do this A new website ready soon with all studies and
resources in one site Support from the Chief Economists office and DEC
Lessons Learnt: What worked (1)
The overall methodology is sound and important Sound: Correlates with various characteristics
as predicted by common sense Important: Highlights potential and limitations
of different policy measures The distribution of quality across public/private or
rich/poor The distribution of effort The know-do gap
Lessons Learnt: What worked (2) Initial worry that the
variation in doctors is too large in India (Allopathic, Ayurvedas, Unani, Homeopaths) for a single instrument to capture quality Turned out to not be a
bit worry because doctors were all treating patients using the same (allopathic) medicines
Therefore, they could be graded on the same scale
0.0
0.5
2.6
0.5
0.5
2.5
0.1
0.7
2.7
0 1 2 3
MBBSBIM
S/BAM
S/BUM
S/BHM
S
RMP/N
o Tra
ining
Practitioner Qualifications and Drug Use
Medicines per patient Antibiotics per patientAlternative Medicines per patient
Regardless of the style of the provider’s training, the type of medications dispensed were very similar
Lessons Learnt: What did not workVignettes
We chose the simplest cases possible with no complications
Even then, our vignettes are not good at distinguishing bad from very bad doctors (high standard-errors for less than average providers)
Perhaps adding in a simple set of written questions would help
For instance: “If a child is suffering from diarrhea, what should you give the child?”
-6-4
-20
295
% C
onfid
enc
e In
terv
als
05
1015
Per
cent
-2 -1 0 1 2Com petence
Sam ple Density Upper Confidence Interval
Com petence Lower Confidence Interval
Classification Errors and Sample Density
Source: Author’s calculations based on World Bank-ISERDD (2003). The horizontal axis in the graph is competence, the left vertical axis is the density (in percentages) for the histogram of competence and the right vertical axis shows confidence intervals of competence. The solid line is estimated competence, which is plotted against itself (this would be the 45o line if the scales were the same). The two dashed curves represent the upper and lower confidence intervals at the 95% level of confidence. The histogram underlying the confidence interval curves shows how competence is distributed at values of the index with large and small standard errors.
Lessons Learnt: What did not workVignettes
All the 5 cases we chose did not require any treatment at the primary level. Therefore, the “mistakes” we usually pick up are
errors of commission—doctors doing things that they should not have
Definitely consider including cases that require treatment at the primary level (pneumonia?) This allows us to pick up errors of omission
Lessons Learnt: What did not workPractice
The vignettes standardize the case-mix and the patient-mix Observing real patients does not. This leads to problems
Unobserved patient characteristics could affect inference (for instance sorting)
Use an exit-survey to pick some of this up, if possible
The cases that overlap with the vignettes are limited (rare cases almost never seen)
And may not be perfect overlaps Possibility of using “simulated standardized patients”
Pilot currently underway: if this works out, it vastly improves our diagnostic abilities
Lessons Learnt: What did not workOverall
The studies thus far are mostly “boutique” studies We are working on how to mainstream them We are not sure what the cost of deviation from the “boutique”
approach would be
What I would do (again)
Make sure that you keep lots of time for case development
Pilot the vignettes until all (>95%) questions that providers ask have a predefined answer
Train enumerators until they have memorized the entire vignettes module
NEW: use video-recordings of doctor-patient interactions as training material for direct observation (these are being developed)
Things to do differently
Post-code treatments (the ultimate nightmare) Make some changes to the direct observation form
Patient order Interviewer assessments of patients (to be piloted) Number of questions that patients ask
Have a clear idea of the timeline and the work program (but that’s for any of this work!)
Papers on which this presentation is based