march 2004 university of bristol leverhulme centre for market and public organisation comments on...
Post on 28-Mar-2015
212 Views
Preview:
TRANSCRIPT
March 2004
University of Bristol
Leverhulme Centre for Market and Public Organisation
Comments on “Trying it Out: The Role of Pilots in Policy Making”
Carol Propper Research Methods Festival July 2004
March 2004
• Report very welcome – Facilitate greater understanding of the benefits of trials
and the constraints that policy makers and researchers work within
– Recommendations sensible and timely
• But …– Trials and pilots are not an automatic fix
– RCTs may not be the ‘gold’standard (or more properly, will not necessarily provide easy to find answers)
March 2004
Problem in interpreting trials
• Governments have large set of policy innovations running at same time– For area initiative difficult to find areas in which to
undertake pilots - report section 6.8 - “if present trends continue, the supply of suitable ‘untouched’ localities may soon be exhausted”
– Same issue arises in pilots implemented in organisations aimed at individuals (e.g. Makinson scheme in C+E)
– Participation in trials may be because extra monies are forthcoming
March 2004
Problem in implemeting RCTs
• RCTs difficult to administer in social settings– MTO scheme: ‘treatment = living in (moving to) a
better neighbourhood’
– Some of the treated did not participate; some controls moved because of other policies
– Some of the treated moved back very quickly to neighbourhoods more similar to their own (treatment not for long)
– Some maintained close ties with former neighbourhood (what exactly is the treatment)
March 2004
Problem in interpreting RCTs
• RCT approach may not be appropriate when there are externalities – RCT designed to provide partial equilibrium estimates of the
treatment for a well designed population– If practice by one person/organisation affects outcomes for others
effectiveness of the treatment depends on where the trial is conducted
– If doing high-tech surgery creates externalities which has positive effects on other patients at same hospital/in same area, surgical interventions will perform well in areas which already do lots, but perform poorly in areas where other treatments are conducted
– RCTs do not pick up general equilibrium effects
top related