March 2004

University of Bristol

Leverhulme Centre for Market and Public Organisation

Comments on “Trying it Out: The Role of Pilots in Policy Making”

Carol Propper Research Methods Festival July 2004

March 2004

• Report very welcome – Facilitate greater understanding of the benefits of trials

and the constraints that policy makers and researchers work within

– Recommendations sensible and timely

• But …– Trials and pilots are not an automatic fix

– RCTs may not be the ‘gold’standard (or more properly, will not necessarily provide easy to find answers)

March 2004

Problem in interpreting trials

• Governments have large set of policy innovations running at same time– For area initiative difficult to find areas in which to

undertake pilots - report section 6.8 - “if present trends continue, the supply of suitable ‘untouched’ localities may soon be exhausted”

– Same issue arises in pilots implemented in organisations aimed at individuals (e.g. Makinson scheme in C+E)

– Participation in trials may be because extra monies are forthcoming

March 2004

Problem in implemeting RCTs

• RCTs difficult to administer in social settings– MTO scheme: ‘treatment = living in (moving to) a

better neighbourhood’

– Some of the treated did not participate; some controls moved because of other policies

– Some of the treated moved back very quickly to neighbourhoods more similar to their own (treatment not for long)

– Some maintained close ties with former neighbourhood (what exactly is the treatment)

March 2004

Problem in interpreting RCTs

• RCT approach may not be appropriate when there are externalities – RCT designed to provide partial equilibrium estimates of the

treatment for a well designed population– If practice by one person/organisation affects outcomes for others

effectiveness of the treatment depends on where the trial is conducted

– If doing high-tech surgery creates externalities which has positive effects on other patients at same hospital/in same area, surgical interventions will perform well in areas which already do lots, but perform poorly in areas where other treatments are conducted

– RCTs do not pick up general equilibrium effects