Building capacity in Realistic Evaluation with Student Sport Practitioners

Kevin HarrisSenior Lecturer

Objectives

Can Realistic Evaluation offer a suitable Monitoring and Evaluation methodology for Student Practitioners?

To report empirical findings of research testing a participatory based monitoring and evaluation framework on Student Sport Development Practitioner’s (SSDP’s)

To highlight the strengths and limitations of Realistic Evaluation’s use with practitioners

NOT A REALISTIC EVALUATION EXPERT!

Sport for Development (SFD)

Emerging lack of evidence discourse

•Academic critique upon the field (Coalter, 2007; 2010; 2013)

•Proving that sport works (EBP) versus why and how does it work •Expansion of monitoring and evaluation (M+E) in the field but much of it carried out or positioned by academics (Nicholls et al, 2010; Levermore, 2011)

•Valuing the voices of practitioners and those at the heart of SFD programmes in M and E work (Kay, 2009 ;Nichols et al,2010)

What is it about sport interventions that make them successful or unsuccessful? (Coalter, 2007)

Realistic Evaluation (Pawson and Tilley, 1997)Critical Realism: “people have the capacity to act freely and shape their lives within social

structures. Consequently, any change is a result of an interaction between individuals and the systems they work in” (Abhyanker et al

2013: 13)

Context: What conditions are needed for a measure to trigger mechanisms to produce outcome patterns?

Mechanism: What is it about a measure that may lead it to have a particular outcome pattern in a given context (for example how do resources intersect with participants beliefs, reasoning, attitudes, ideas and opportunities?)

Outcomes:What are the practical effects produced by causal mechanisms being triggered in a given context? (Pawson and Tilley, 1997)

Case study

The Coaching Innovation Programme - Southampton Solent University

• Students to research, develop and deliver their very own Coaching Innovation Project (CIP) in partnership with SDP’s. Growing contribution of H.E (curriculum based)

• Projects developed and delivered over 2 years + with SDP’s

Research design

Training programme devised consisting of two models:

1.Developing realistic programme theory

2.Testing programme theory through realistic approaches

Training programme delivered over academic year consisting of capacity building workshops

Training programme tested via Realistic methods to establish what worked for whom in what circumstances and why?

“what was it about the framework that worked (or not) for the SSDP’s and in what circumstances?”

Methods

•Sample: 3 CIP (similar themed) case studies between Oct 13 and May 14

•Blogs, interviews and an anonymous survey was implemented

•Data deductively coded searching for key codes associated with contexts, mechanisms and outcomes impacting upon the SSDPs

•Stored and managed via NVIVO and categorised into C,M,0s in accordance with key themes

Informing M and E practice – CMO’SContexts Mechanisms OutcomesConceptualisation to application‘Scepticism towards M and E’

‘Limited understanding of M and E and programme theory’

‘Fear of M and E’

‘In experienced practitioners’

‘Disparities in group dynamics and cohesion’

‘The CIP and its broader requirements’

 

Perseverance

Collaborative discovery of knowledge

 

Increased competence in RE techniques

Intent to enact RE in future M and E

‘Working with young people’ Critical praxis of RE Increased reflexivity of RE and its strengths / limitations

‘Unpredictable projects / changes and struggles’

‘Awareness and limitations of sport for development programmes’

‘Open minded perceptions / Willingness to learn about M and E’

‘Optional training programme providing support’

 

Valuing RE philosophy

Theoretical grasping of RE

Practical application of RE

 

Realization of what RE can do to understand CIP (how and why)

Increased competence in RE techniques

“Well, during the CMOs I found that at the start it's

quite difficult because I didn't know what exactly I was

looking for, but obviously after reading up on it and through

the workshops I actually understood the specific things

I wanted to look for. And obviously, with CMOs you

can't actually uncover all of the hidden mechanisms and

the different contexts because say if run your program in one context it won't be exactly the

same for another”. SSDP 2 (Male)

“Well, during the CMOs I found that at the start it's

quite difficult because I didn't know what exactly I was

looking for, but obviously after reading up on it and through

the workshops I actually understood the specific things

I wanted to look for. And obviously, with CMOs you

can't actually uncover all of the hidden mechanisms and

the different contexts because say if run your program in one context it won't be exactly the

same for another”. SSDP 2 (Male)

Contexts Mechanisms OutcomesStructure to enable agency‘Limited understanding of M and E and programme theory’

‘Fear of M and E’

‘Dynamics of SSDP’s doing own evaluations’

‘In experienced practitioners’

‘Disparities in group dynamics and cohesion’

‘Need for support’

 

Interaction and discussion

Structured process

Trust in guidance

 

Increased competence in RE techniques

Independent Monitoring and Evaluator’s

Innovative M and E design and findings

Ownership and autonomy‘Dynamics of SSDP’s doing own evaluations’

‘Open minded perceptions / Willingness to learn about M and E’

‘Optional training programme providing support’

‘Wider parameters and goals for Edumove’

Self and collective control

Freedom to act and make choices

Increased reflexivity regarding the strengths and limitations regarding M and E

Increased competence and capacity in M and E

Increased confidence to enact M and E

Realization of what M and E holds to understand CIP

 

“Yeah, we chose our program theory, what we wanted to test, where we wanted to run it, how we wanted to run it within reason to Edumove. We basically had total control and then we obviously came back to you for some support and some ideas, but even then it wasn't like a force, I think you should do this. It was like oh well, you could do this, but it's up to you. I think that was quite good. We did have control.” SSDP (Male: 6)

“Yeah, we chose our program theory, what we wanted to test, where we wanted to run it, how we wanted to run it within reason to Edumove. We basically had total control and then we obviously came back to you for some support and some ideas, but even then it wasn't like a force, I think you should do this. It was like oh well, you could do this, but it's up to you. I think that was quite good. We did have control.” SSDP (Male: 6)

ConclusionsMethodology underpinning the model / training programme has increased the capacity of the SSDP’s understanding and practice of M and E techniques.

Realistic evaluation (RE) is a suitable evaluation methodology that firstly is suitable for small scale SFD interventions, and secondly useable for the SSDP’s to make sense of how and why their programmes achieved certain outcomes.

Engagement with RE facilitated mechanisms of practical and theoretical application of RE concepts which were applied within the CIP’s. There was clear appreciation and value (mechanisms) afforded to RE

A critical praxis of RE where SSDP’s reflected in and for action on its strengths and limitations as a method within certain circumstances (eg working with young people).

Conceptual complexities and language embroiled within RE presents barriers in terms of engagement and understanding for practitioners.

However…

The participatory nature of the training programme acted as a vehicle enabling structure

Increased capacityReflexivityBut also agency and freedom for SSDP’s to engage and

maintain control over their M and E

Next steps of research

The realistic methodology which was mobilised to test the model has uncovered a range of contexts, mechanism and outcomes that will inform the refinements of the model / training programme and enable further testing in line with the realistic approach (Pawson, 2003).

Refine the model / training programme in accordance with the findings of this research

Test the model on a broader range of CIP’s embedding and involving more stakeholders

References• Abhyanker, P, Cheyne, H, Maxwell ,M , Harris , F Mcourt, C (2013) A

realist evaluation of a normal birth programme. Evidence based midwifery 11 (4): 112-119

• Coalter, F., 2007. A wider social role for sport. London: Routledge• Coalter, F (2010) The politics of sport-for-development: Limited focus

programmes and broad gauge problems? International Review for the Sociology of Sport 45: pp.295-320

• Coalter, F, (2013) Sport for Development what game are we playing? Routledge

• Fetterman, D. M. (1994) Empowerment evaluation [American Evaluation Association presidential address]. Evaluation Practice, 15 (1), 1–15.

• Fetterman, D.M. (2005) “A window into the heart and soul of empowerment evaluation – looking through the lens of empowerment evaluation principles”, in Fetterman, D.M. and Wandersman, A. (Eds), Empowerment Evaluation Principles in Practice, Guilford Press, New York, NY, pp. 42-72

• Kay, T. (2009): Developing through sport: Evidencing sport impacts on young people. Sport in Society 12, (9), 1177 — 1191

• Levermore (2011) Evaluating sport-for-development: approaches and critical issues Progress in Development Studies 11, (4) pp. 339–53

• Nicholls, S, Giles, A and Sethna, C. (2010). Perpetuating the ‘lack of evidence’ discourse in sport for development: Privileged voices, unheard stories and subjugated knowledge International Review for the Sociology of Sport. 46 (3) 249-264

• Pawson, R and Tilley, N (1997). Realistic Evaluation. Sage Publications• Pawson, R. (2003) Nothing as Practical as a Good Theory. Evaluation 2003 9:

471• Prashanth, N; Marchal, B; Kegels, G and Criel, B (2014) Evaluation of capacity-

building program of district health managers in India: a contextualized theoretical framework VOL 2 (article 89) Frontiers in public Health journal

• Wandersman, A., Snell-Johns, J., Lentz, B. E., Fetterman, D. M., Keener, D. C., Livet, M., et al. (2005) The principles of empowerment evaluation. In D. M. Fetterman, & A. Wandersman (Eds.), Empowerment evaluation principles in practice (pp. 27–41). New York: Guilford Press.