Zürcher FachhochschuleZürcher Fachhochschule 1

What to do without baselines?

Not often used methods in media evaluation

11th FoMe Symposium

Focus on Impact: Advanced Methods and Concepts in Media Development

Bonn, 1 October, 2015

Christoph Spurk

IAM Institute of Applied Media Studies

Zürcher Fachhochschule 2

What is Evaluation?

• Systematic assessment of policies, programs and projects regarding

- Concept- Implementation - Results and effects (effectiveness)

• Objectives - Formative Learning - Summative Judgement

Zürcher Fachhochschule

Five Core Approaches in Evaluation*

• Needs Assessment (Relevance)

• Assessment of Program Theory or Design

• Process Evaluation

• Impact Evaluation (Results, Effects, Impact)

• Efficiency Assessment and Cost-effectiveness

*(Rossi et al. 2004, p54 ff.)

3

Zürcher Fachhochschule

Basic Requirements for Evaluation

• Clear Objectives of (Media) Intervention

– Based on Analysis of Context and comprehensive planning process

– Surprisingly often not done in current competitive environment

• Baseline study before Intervention

– Gathers data objectives of Intervention

– Provides insights (Needs!)

– Enables to measure later against this base in a before-after-design = extent of change

• Results chains with indicators

4

Zürcher Fachhochschule

Basic Challenges in Evaluation

5

Zürcher Fachhochschule

What is often done – but should not• Ask beneficiaries what the change is / was

– Most Significant Change Technique as Method- «What was the most significant change through the radio programme?»- Strong influence from evaluator’s side – biased

- Social desirability in answers even more enhanced.

– Posing the most difficult evaluation questions directly

- «What do you think is the contribution of Radio XYZ to democratic participation?»

- «How much has your farming income increased due to the radio programme in the last hree years?»

• Unbalanced scales in answer options - Strongly agree / agree / somehow agree / don’t agree

-

• but please dont repeat it 6

Zürcher Fachhochschule

What can be done? – Is there a way out?

1. Prepare a baseline study with control groups for next phase of project

2. Use the «theory based» evaluation method as an alternative design, but still second best.

3. Story checking as an additional data collection method

7

Zürcher Fachhochschule

I. «Theory based» evaluation

• Assessment of the «plausability of impacts»

• Requirements

– Knowledge from evidence-based research about what works and what not

– Assessment whether program elements in intervention reflect this knowledge or not.

– If program is in line with evidence, it has a greater plausibility to achieve impacts.

8

Zürcher Fachhochschule

Example: Civil Peace Service (CPS)

• No baseline data at the beginning of project • Evidence

• existed from previous research what civil society functions did work in particular phases of conflict

• based on comparative research in various conflict settings

• Assessment: • Most CPS program interventions were working on long-term civil

society functions (socialization, social cohesion) that were hardly relevant in ongoing violent conflict, and

• not on short-term functions that were much in need (protection, monitoring).

• Thus, the plausibility of impact was assessed as low. 9

Zürcher Fachhochschule

Potential for media programs - Example

• Farmer information project – wants to know about the changes in farmer practice (organic farming!)

– No baseline data on farmer practice before project start

• Application of theory-based evaluation– Evidence exists from previous research what type of information small-

scale farmers like and need for adoption:

- Technical plus market and economic information - No top down single solutions but variety of options

– Potential to assess content of programmes regarding those requirements

– Results of content analysis would tell to what extent the information is plausible to be implemented.

– Insights could provide the project with valuable insights for future improvement

10

Zürcher Fachhochschule 11

II. Checking stories as exploration

• SjCOOP: Improving Science Journalism in Africa and the Arab World– Better quality in reporting– Publishing science stories that achieve change (“impact stories”)

- enlightening public debate about science (health, agriculture)

- Contributing to evidence-based policy making or immediate change

• Have stories really achieved an “impact”? – Selection of potential stories – Strong verification process of claimed impact by evaluators

- Timeline

- Context/ History

- Media Echo

- Verification of reactions with sources and experts/stakeholders

Zürcher Fachhochschule

Impact : Science story level

12

Government

Follow-upStory 2 (radio)

Follow-upStory 3 (TV)

Follow-upStory 1

Initial science story

Media Echo

Public Agenda

Public reactions

Local administration

Zürcher Fachhochschule

CD4 Machines in Western Uganda

Problem: • CD4 Machines in Fort Portal, did not work• Patients had to walk 1,5 hour

Story:• Hope Mafaranga publishes articles in New Vision 29/30th Nov.

2011Media Echo:

• Radio stations and Ugandan News Agency pick story 29 Nov. to 2 December 2011.

Impact:• Brand new CD4 Machines two weeks later

Zürcher Fachhochschule

Condoms in Northern Kenya

Problem: • Acute shortage of free condoms in Isiolo region, Kenya• Shortage leads to washing, re-using condoms

or using plastic bags.

Story by Violet Otindo:- Television K24 airs14-minute special on 27th February 2011

Media Echo: -BBC Radio Swahili – 15th March-Citizen and Nation TV -22nd March-Further coverage in Kenya and in Danish and Norwegian newspapers

Impact:- Health Minister admits shortage on 22nd March- Emergency delivery of 100 000 condoms by Red Cross Kenya, 25th March-Kenya government ordered larger supplies with UNFPA- Parliament debated the issue on 29th March

Zürcher Fachhochschule

Checking stories yields more insights

• Typology of stories

– Putting an issue on public agenda

– Short-term impact: immediate change

– Long-term impact : changing policies

– Uptake of science

• Enabling and hindering factors

– Media Echo – sometimes, but not always

– NGO – media – civil society relationship: NGOs take issue up

– Context: Government responsiveness

– Framing of story: shaping a new frame

– Role models of African journalists – change agents15

Zürcher Fachhochschule

Lessons

• Lessons to improve the impact of stories:

– Achieve stronger media echo

– Cooperation amongst media

– Target NGO – media relationships / partnerships

– Take government’s responsiveness into account

• Evidence

– What makes stories more plausible to have an impact

16