The role of theory in systematic review

Ray Pawson

ESRC Research Methods Festival

2008

Realist Synthesis (in a box)

• Unit of analysis is not the whole programme but a particular programme theory

• Primary data is inspected not just for outcomes but for what it reveals about the programme theory

• Purpose of the review is not to deliver some ‘verdict’ on a family of programmes but to refine programme theory so that there is a better understanding of how it works

Theories: locating the mechanism of change

Manifest – the meansHARDLY AT ISSUE thatReducing speed lessens traffic fatalities, better locks deter burglars, smoking cessation reduces disease, smoke alarms reduce fire fatalities, green bin collection improves recycling, training improves job prospects AND SO ON

Latent – the mechanisms

WHAT IS AT ISSUE is how to we persuade people and communities to act on this knowledge. How do we educate, incentivise, shame, legislate, empower, mentor, control, train, punish, impress programme recipients into compliance with the big idea.

Theories: explanatory propositions dealing with contingencies

• Not empirical generalisations (what works)• Not conceptual frameworks, meta-narratives,

themes, metaphors etc.Rather some version of …• “What works for whom and between

whom, in what institutional and cultural contexts, in what respects, with what staying power, in which applications, with what reproducibility … AND WHY”

Theories: at the appropriate level of abstraction

Realist synthesis generates propositions about a class of interventions, which should be … … ….

More specific than a master metaphor or ‘conceptual kiss’

Less specific than a ‘logic model’ that tries to include all the pertinent forces that impinge on an intervention

At a ‘middle range’ of specificity that identifies a general remedy for a broad class of problems

NAMED

BUT WHO IS SHAMED?

• Poll-tax protesters named in the local newspapers

• Sex offenders under community notification

• Under-performing schools identified in league tables

• Motor manufacturers named in the Car Theft Index

Poll-tax protesters named in the local newspapers

Poll tax non-payment – there were a great many sanctions imposed: fines, wage arrest, court appearances etc. Protesters thus ignored or even celebrated disclosure of their names in the local press. Policy abandoned.

Sex offenders under community notification

Notified Sex Offenders are resentful or scared by public attention. One key result is non-compliance with registration and displacement to other localities. Re-offence rates remain static.

Under-performing schools identified in league tables

• Schools respond to league table positions tactically - increasing resources to marginal candidates (middle grades) and excluding/ not entering hopeless ones (lowest grades). Grades improve.

Motor manufacturers named in the Car Theft Index

Car manufacturers – are embarrassed by adverse publicity and loss of reputation (and downturn in sales) - respond with genuine improvements in vehicle security. Car crime goes down.

Is there a theory to accommodate these differences in outcome?

Attitude toward membership

Eligible for membership

Ineligible for membership

Aspire to belong

Candidate for membership

Marginal Man

Indifferent to affiliation

Potential member

Detached non-member

Motivated not to belong

Autonomous non-member

Antagonistic non-member

Merton’s Typology of aspirations to group membership of eligibles and non-eligibles

Resistance

Avoidance

Complicity

Compliance

Findings: The nature of ‘recommendations’

The theory under further refinement

For N&S to ‘work’ the following configuration should be in place:

• the named party should be an ‘aspirational insider’• the shaming mechanism should be dovetailed with other

mechanism (e.g. market sanctions)• the disclosure should carry intense (but controllable)

media interest• the disclosed data should unambiguous both in

allocating blame and in suggested remedial action• the disclosing authority should have had exemplary

watchdog credentials, which are operated benignly

Reference group theory travels onward

“The grapevine already exists so let’s make positive use of it”

“Let us get hold of the hard-to-reach by using those already there”

One study synthesised

Mellanby A, Rees J and Trip J (2000) Peer-led and adult-led school health education: a critical review of available comparative research Health Education Research 15(5):533-545

…. AS EVER, MIXED RESULTS:

Peer leaders ‘were more successful in establishing conservative attitudes related to sexual behaviour’, whilst … Adults ‘were more successful in their warning about pregnancy and sexually transmitted disease’.

Is there a theory to accommodate these differences in outcome?

Attitude toward membership

Eligible formembership

Ineligible formembership

Aspire to belong Candidate for membership

Marginal non member

Indifferent to affiliation Potential member Detached non-member

Motivated not to belong Autonomous non-member

Antagonistic non-member

RISK AVERSION

STIFILING AMBITION

Revised theory

• The salient reference group for developing risk aversion in marginal locations lies in the category of full members (especially from expert members ones with know-how in relation to the hazards).

• The salient reference group for stifling ambition in marginal locations lies with groups even further from full membership. The inclination to stay put is best fostered by the resilient and the resistant (especially if they outnumber the marginals).

More details in

Ray Pawson ‘Middle Range Theory and Programme Theory Evaluation’

In F Leuuw and J Vassen (eds) Mind the Gap: Evaluation and the Disciplines Transaction Press (forthcoming)