It is impossible to present the philosophy of scientific realism and its application in social science in a 30 minute presentation

It is possible to identify crucial problems in using evidence to inform policy recommendations

And to explain how a realist perspective stresses the need to understand mechanisms and contexts in order to explain outcomes.

The presentation will present examples of research that emphasise the importance of mechanisms.

EMPIRICISM•description and measurement give us the ‘facts’ about the world.

•the world exists independently of us.

•But knowledge is reduced to the level of events.

•theoretical descriptions must be defined in terms of observables (operationalisation)

•relies mainly on quantitative methods

REALISM•objects in the physical, social & psychological world exist & have properties independently of our theoretical concepts and discourse about them.

•maintains the linguistic objectivity of scientific theories.

•layered ontology

•Makes use of qualitative & quantitative methods.

•accepts that theories should be subject to empirical validation.

INTERPRETIVISM•hermeneutic

•social constructionist

•relativist

•everything we know is socially constructed (including science)

•tends reduce existence to texts, discourses and forms of consciousness

•relies mainly on qualitative methods.

From: BYGSTAD, B. & MUNKVOLD, B. E. (2011) In Search of Mechanisms. Conducting a Critical Realist Data Analysis. 32nd International Conference on Information Systems, Shanghai 2011.

Related in various ways in Social Science (e.g. Giddens’ Structuration theory)

Conflation and Reduction abound! (e.g. ‘Brainism’, Rational Choice Theory, class reductionism of (some) Marxism)

Critical Realism attempts to engage with “the complexity, openness, and ambiguity of social action.”

Are they ‘technologies” likedrug treatments orsurgical interventions?

Are they more akin to socialor educational programmes which attempt to persuade people to change their behaviour?

They consist for the most part of conversations – perhaps they are more like rhetoric (the art of persuasion).

The question “what works?” should be replaced by “what works for whom in what context?”

Social programmes (such as programmes to reduce crime, increase health and well-being, or to promote “recovery”) are complex, often with multiple outcome goals.

Notoriously, such programmes are evaluated (including with RCTs) as efficacious in one setting, but disappoint when transferred to another.

“The outcome of a mechanism is contextual, i.e. dependent on other mechanisms. Thus, a mechanism may produce an outcome in one context, and another in a different context. This contingent causality (Smith 2010) is inherent in all open systems.” (Bygstad, B., & Munkvold, B. E. (2011). In Search of Mechanisms. Conducting a Critical Realist Data Analysis.)

Key Principles (from Nick Tilley’s presentation Conducting Realist Evaluations)

• Programs are theories incarnate• Evaluations are tests of theories• Theories need to comprise context-mechanism-outcome

conjectures• Mechanisms refer to the ways in which effects are brought about• Contexts refer to the conditions for the operation of mechanisms.

They are seldom closed.• Outcomes refer to the effects of mechanisms activated in context• Mechanisms generally (though not always) involve reasoning and

resources• Programs work differently amongst different subgroups

“Social programmes, as discussed in detail in Chapter 2, offer resources (material, social, cognitive) to subjects, and whether they work depends on the reasoning of these individuals. Subjects may seek out programmes (or not), volunteer for them (or not), find meaning in them (or not), develop positive feelings about them (or not), learn lessons from them (or not), apply the lessons (or not), talk to others about them (or not). It is within this interpretative process - or mechanism - that the causal powers of programmes reside. Any method of systematic review that omits such a vital agent from its core hypotheses automatically sets up a depleted inquiry.”

Pawson R. (2006) Evidence-Based Policy: A Realist Perspective (Chapter 3)

A realist framework:

Outcome: Transformative change (provisionally defined as durable change in a client’s frame of reference /emotional commentary, evidenced by their metacognitive accounts of attitude and relationship changes)

Mechanisms: causal processes generating change - powers emergent from relations (for example modification of emotional commentary resulting from interactions in a therapeutic relationship)

Contexts: structural powers bearing on the actualisation of mechanisms (including countervailing mechanisms, and powers of institutional and other social structures)

From: Pawson R. (1996) Theorizing the Interview. British Journal of Sociology 47/2, p.300

Koski-Jännes, A. (2005) On the Problems of Randomized Clinical Trials as Means of Advancing Clinical Practice. Evidence Based Practice: Challenges in Substance Abuse Treatment. Helsinki: Nordic Council for Alcohol and Drug Research (NAD).

Morgenstern, J. & Mckay, J. R. (2007) Rethinking the paradigms that inform behavioral treatment research for substance use disorders. Addiction, 102/9, 1377-1389.

Orford, J. (2008) Asking the right questions in the right way: the need for a shift in research on psychological treatments for addiction. Addiction, 103/5.

Tucker, J. A. & Roth, D. L. (2006) Extending the evidence hierarchy to enhance evidence-based practice for substance use disorders. Addiction, 101/7, 918-32.

Tudor Hart, J. (1997) What evidence do we need for evidence based medicine? Journal of Epidemiology and Community Health (1979-), 51/6, 62.

Weak theories:If interventions work through specific effects then some should be superior due to the efficacy of their ingredients.Theory-based matching effects should appear.Very little evidence for these.

Markedly inconsistent findings in efficacy, moderator and mediator effects with all the ‘evidence-supported’ interventions.

Important differences between the outpatient and aftercare arms of Project MATCH.

(Morgenstern, J., & McKay, J. R. (2007). Rethinking the paradigms that inform behavioral treatment research for substance use disorders. Addiction, 102(9), 1377-1389.)

Many trials

Various meta-analytic reviews

Some recent investigations into mechanisms

Apodaca, T. R., & Longabaugh, R. (2009). Mechanisms of change in motivational interviewing: a review and preliminary evaluation of the evidence. Addiction, 104(5), 705-715

Morgenstern, J., Kuerbis, A., Amrhein, P., Hail, L., Lynch, K., & McKay, J. R. (2012). Motivational

interviewing: A pilot test of active ingredients and mechanisms of change. Psychology of Addictive Behaviors. 26(4), 859-869.

Results show significant main effects for student residence, with more drinking and more problems reported by students living in fraternity and sorority houses. For students who live in the Greek system, it is possible that a house-based intervention (rather than the individual interviews conducted in this study) would be an effective prevention strategy. Current research conducted by our group is investigating the effectiveness of such a group-based intervention for these students.

What is the mechanism here? And what context facilitated its activation?

“This is a field where there is no lack of randomised controlled trials. Perhaps it is time to move from only studying whether MI works to also studying how it works, that is to study the mechanisms behind MI.”

(Smedslund et al 2011 p.28, emphasis in original).

Morgenstern, J., Labouvie, E., McCrady, B. S., Kahler, C. W., & Frey, R. M. (1997). Affiliation with Alcoholics Anonymous after treatment: A study of its therapeutic effects and mechanisms of action. Journal of consulting and clinical psychology, 65(5), 768.

Morgenstern, J., Bux, D., Labouvie, E., Blanchard, K. A., & Morgan, T. J. (2002). Examining mechanisms of action in 12-step treatment: The role of 12-step cognitions. Journal of Studies on Alcohol and Drugs, 63(6), 665.

Morgenstern, J., & Longabaugh, R. (2002). Cognitive–behavioral treatment for alcohol dependence: A review of evidence for its hypothesized mechanisms of action. Addiction, 95(10), 1475-1490.

Morgenstern, J., Bux, D. A., Labouvie, E., Morgan, T., Blanchard, K. A., & Muench, F. (2003). Examining mechanisms of action in 12-Step community outpatient treatment. Drug and alcohol dependence, 72(3), 237-247.

Longabaugh, R., Donovan, D. M., Karno, M. P., McCrady, B. S., Morgenstern, J., & Tonigan, J. S. (2005). Active Ingredients: How and Why Evidence‐Based Alcohol Behavioral Treatment Interventions Work. Alcoholism: Clinical and Experimental Research, 29(2), 235-247.

Longabaugh, R. (2007). The search for mechanisms of change in behavioral treatments for alcohol use disorders: a commentary. Alcoholism: Clinical and Experimental Research, 31(s3), 21s-32s.

Huebner, R. B., & Tonigan, J. S. (2007). The Search for Mechanisms of Behavior Change in Evidence‐Based Behavioral Treatments for Alcohol Use Disorders: Overview. Alcoholism: Clinical and Experimental Research, 31, 1s-3s.

Apodaca, T. R., & Longabaugh, R. (2009). Mechanisms of change in motivational interviewing: A review and preliminary evaluation of the evidence. Addiction, 104(5), 705-715.

Longabaugh, R., & Magill, M. (2011). Recent advances in behavioral addiction treatments: focusing on mechanisms of change. Current psychiatry reports,13(5), 382-389.

Longabaugh, R., & Magill, M. (2012). Commentary on Michie et al.(2012): The lid is off the black box. Addiction, 107(8), 1441-1442.

Morgenstern, J., Kuerbis, A., Amrhein, P., Hail, L., Lynch, K., & McKay, J. R. (2012). Motivational Interviewing: A Pilot Test of Active Ingredients and Mechanisms of Change. Psychology of Addictive Behaviors, 26(4), 859-869.

In the last of these, the authors distinguish between “active ingredients”, which are things the programme or therapist does, and “mechanisms of change” which are processes that go on with the client, e.g. changes in reasoning or take-up of resources.

The Palo Alto group (Moos, Finney etc) were historically interested in contexts and have attempted more recently to characterise treatment programmes according to theory-based identification of ‘active ingredients’:

Bromet, E., & Moos, R. H. (1977). Environmental resources and the post-treatment functioning of alcoholic patients. Journal of Health and Social Behavior, 326-338.

Cronkite, R. C., & Moos, R. H. (1980). Determinants of the post-treatment functioning of alcoholic patients: A conceptual framework. Journal of Consulting and Clinical Psychology, 48(3), 305.

Holahan, C. J., & Moos, R. H. (1987). Personal and contextual determinants of coping strategies. Journal of personality and social psychology, 52(5), 946.

Moos, R. H. (2007). Theory-based active ingredients of effective treatments for substance use disorders. Drug and alcohol dependence, 88(2), 109-121.

Moos, R. H. (2007). Theory-based processes that promote the remission of substance use disorders. Clinical Psychology Review, 27(5), 537-551.

Problems in relying solely or mainly on an experimental research design to provide evidence about how alcohol or drug dependence treatment works (let alone recovery-promoting systems of care) have been recognised for 35 years.

The EBM movement encouraged an almost exclusive emphasis on RCTs as the gold standard to try to identify efficacious treatments along the drug testing model. Longitudinal outcome studies have been sometimes grudgingly included but they have problems of their own.

In the view of many (e.g Morgenstern & McKay 2007, Orford 2008,) the results have been disappointing or have reached a dead end.

Morgenstern, J., & McKay, J. R. (2007). Rethinking the paradigms that inform behavioral treatment research for substance use disorders. Addiction, 102(9), 1377-1389.

Orford, J. (2008). Asking the right questions in the right way: the need for a shift in research on psychological treatments for addiction. Addiction, 103(6), 875-885.

*according to Prof. Nick Tilley, realist evaluation “refuses to provide (meaningless) simple (minded) answers that are sometimes wanted” but this of course does not mean that critical realist researchers are against evidence-based practice or reject RCTs as valueless.

Timko C., Moos R.H., Finney J.W., Lesar M.D. (2000) Long-term outcomes of alcohol use disorders: Comparing untreated individuals with those in Alcoholics Anonymous and formal treatment. Journal of Studies on Alcohol. 6(4):529–540. 

Kaskutas L. A., Bond J., Humphreys K. (2002) Social networks as mediators of the effect of Alcoholics Anonymous. Addiction 97: 891–900.

Timko, C., Billow, R., & DeBenedetti, A. (2006). Determinants of 12-step group affiliation and moderators of the affiliation–abstinence relationship. Drug and alcohol dependence, 83(2), 111-121.

Kelly, J. F., Magill, M., & Stout, R. L. (2009). How do people recover from alcohol dependence? A systematic review of the research on mechanisms of behavior change in Alcoholics Anonymous. Addiction Research & Theory,17(3), 236-259.

Kelly, J. F., Stout, R. L., Magill, M., Tonigan, J. S., & Pagano, M. E. (2011). Spirituality in recovery: a lagged mediational analysis of Alcoholics Anonymous’ principal theoretical mechanism of behavior change. Alcoholism: Clinical and Experimental Research, 35(3), 454-463.

Kelly, J. F., Stout, R. L., Magill, M., & Tonigan, J. S. (2011). The role of Alcoholics Anonymous in mobilizing adaptive social network changes: A prospective lagged mediational analysis. Drug and alcohol dependence, 114(2), 119-126.

Kelly, J. F., Hoeppner, B., Stout, R. L., & Pagano, M. (2011). Determining the relative importance of the mechanisms of behavior change within Alcoholics Anonymous: a multiple mediator analysis. Addiction, 107(2), 289-299.

Kelly, J. F., & Hoeppner, B. B. (2012). Does Alcoholics Anonymous work differently for men and women? A moderated multiple-mediation analysis in a large clinical sample. Drug and Alcohol Dependence.

Mediators and moderators do not quite get us to mechanisms but they are a good start. (this is beginning to be explicitly recognised in the discussion sections of some of the papers cited)

Emerging clues:

•Changes in social network

•Self-efficacy (social for men, negative affect for women)

•Spiritual practices, reduction in negative affect for the most impaired (apparently less important for the less impaired)

The “mechanisms” in realist terms are the way in which an individual uses the above to change their reasoning (including “emotional reasons”) and make use of resources.

The “contexts” are the contingencies which inhibit or facilitate the activation of mechanisms (e.g. stigma > reduced occupational opportunities > reduced confidence and sense of purpose)

“Our results suggest that the process of recruiting young drug users into a study and focusing their attention on substances before a brief intervention may be, in itself, sufficient to induce positive reactive effects on behaviour.” (p.1024)

Cognitive/experiential1.Consciousness raising: Gaining information that increases awareness about the current behavior pattern or the potential new behavior2.Emotional arousal: Experiencing emotional reactions about the status quo and/or the new behavior3.Self-reevaluation: Seeing and evaluating how the status quo or the new behavior fits in with or conflicts with personal values4.Environmental reevaluation: Recognizing the positive and negative effects the status quo or new behavior have upon others and the environment5.Social liberation: Noticing and increasing social alternatives and norms that help support the status quo and/or change and initiation of the new behavior

Behavioral1.Self-liberation: Making choices, taking responsibility for, and making commitments to engaging in a new behavior or a behavior change2.Stimulus generalization or control: Creating, altering, or avoiding cues/stimuli that trigger or encourage a particular behavior3.Conditioning or counterconditioning: Making new connections between cues and a behavior or substituting new, competing behaviors and activities in response to cues for the "old" behaviors4.Rcinforcement management: Identi fying and manipulating the positive and negative reinforcers for current or new behaviors. Creating rewards for new behaviors while extinguishi ng (eliminating reinforcements) for current behavior.5.Helping relationships: Seeking and receiving support from others (family, friends, peers) for current or new behaviors