how we know what works (and what doesn’t) in wp...university. 2. to promote empiricism and...

Title 1: 26pt King’s Bureau Grotesque Three Seven white outTitle 2: 20pt King’s Caslon Display

How we know what works (and what doesn’t) in WP

Miriam Styrnol and Vanessa Todman (in spirit)

What Works, King’s College London

About Us


Objectives of the WWD

1. To contribute to understanding of what works in enabling people to access and succeed at university.

2. To promote empiricism and innovation in widening participation.

3. To support King’s and the sector to think differently about designing and evaluating their initiatives.

The What Works DepartmentGeneration

• Working in partnership on new research to fill the gaps in existing knowledge and validate international insights in the UK context

Dissemination

• Developing reports, briefings and training on cutting-edge research findings and methods

Synthesis

• Drawing together existing research from the UK and elsewhere on supporting access to and success in university

@KCLWhatWorks 3


4

Find us!

@KCLWhatWorks

https://www.kcl.ac.uk/study/assets/PDF/widening-participation/What-works-project-report.pdf https://blogs.kcl.ac.uk/behaviouralinsights


• We’re talking about impact evaluations – generating evidence of the causal impact of an intervention in changing the outcomes of target participants.

❖There are other types of evaluation (e.g. process evaluations), and other types of research.

❖These are all valid and useful; just out of scope for this presentation.

• It’s important to talk about ‘what works’ and impact evaluations because this is an area that is less covered and researched in an otherwise rich and diverse research field.

Before we begin

How many of you have run a trial?

Title 1: 26pt King’s Bureau Grotesque Three Seven white outTitle 2: 20pt King’s Caslon DisplayWe don’t need to find out: we’re

experts and we already know the answer.

Results can be surprising, and even if we’re right

about the direction, what is the effect size?

We have robust findings because

we apply the intervention and

measure impact before and after.

How do we know other factors didn’t play a part?

What if an improvement would have happened

anyway?

How do we know that the groups aren’t also

different in other important ways?

We show the added benefit of our

programme by comparing different

individuals, groups or areas

RCTs are a “black box” into which

inputs disappear and out of which

outcomes emerge with no

explanation.

A good RCT is led by a hypothesis, informed

by theory and primary or secondary – often

qualitative – research.

Interviews, surveys and focus

groups fit our complex context

better.

Are participants are telling the truth? Do they

really know what they want, and how their

behaviour will change in a new situation?


Example: Encouraging Volunteering


Example: Encouraging Volunteering

http://www.behaviouralinsights.co.uk/wp-content/uploads/2016/09/BIT-Update-Report-2015-16.pdf © Behavioural Insights Ltd

http://www.behaviouralinsights.co.uk/wp-content/uploads/2016/09/BIT-Update-Report-2015-16.pdf

The evaluation cycle


The evaluation cycle

@KCLWhatWorks


Diagnose: Research the topic

Sense of

Belonging

Visibility

One purpose of this intervention was to make tasks such as studying for exams achievable. For

example, our study skills video, gave students quick, easy to apply tips and ended with a student

holding a sign saying “you’ve got this”.¹

Knowledge

Self-Efficacy

Videos were designed to tackle a persistent sense of difference among students. By hearing people

talk about what it means to be a King’s student, or asking them to reflect on this, we hypothesise

we could increase student’s sense of belonging.²

In this task we asked students to think about what they may have in common with their personal

tutor, in line with research suggests similarities across various dimensions can increase

compliance and pro-social behavior.³

We designed a video setting out key study skills for students who do not feel prepared. This video

was designed to be basic but uplifting, with BAME students providing key information and the

aim to boost knowledge of study tactics.4

We identified that an intervention designed for all students that targeted BAME issues might be effective

¹Bandura, A. (1977). Self-efficacy: Toward a unifying theory of behavioral change. Psychological Review, 84, 191–215.

²Johnson et al. (2007) Examining sense of belonging amongst first-year undergraduates from different racial/ethnic groups. Journal of College Student Development, 48(5), 525-542.

³Gehlbach et al. (2016). Creating birds of similar feathers: Leveraging similarity to improve teacher-student relationships and academic achievement. Journal of Educational Psychology,

108 (3), 342-352. 4 Miller, M (2016) The Ethnicity Attainment Gap: Literature Review


Plan: Identifying Research Questions

Causal impact of your

evaluation

Did [scheme] increase [main

outcome] among [group]?

Did YKYS improve second year

attainment among

ethnic minorities?

Primary Research Question

These are overarching question that your evaluation will seek to answer.

They will determine the scope and approach of your evaluation.

Secondary Research Question Process Evaluation Question

Focus on specific groups or

intermediate outcomes

Did [scheme] increase [main

outcome/ secondary outcome]

among [group/subgroup]?

Did YKYS improve second year

attainment among ethnic

minorities in specific faculty?

Focus on implementation

and efficiency of your set-up

✓ Was the initiative delivered

the way we expected?

✓ Are we targeting the right

students?

✓ What was the cost-

effectiveness of the initiative?

@KCLWhatWorks


Outcome measures

“I’ll know [outcome reached] when I see [indicator]”

Observable indicators are

those we can build into

the evaluation and control

out

Example: Demography

@KCLWhatWorks

Unobservable indicators

are those we can’t observe

and therefore can’t build

into the evaluative model.

Example: Social Capital


SecondaryAverage Attainment

after Year 2 of Studies

Measure: YKYS outcomes

Primary

SecondaryKing’s Re-/Enrolment

Questions

ProcessUsefulness of video(s)

and task(s)

75%67%

60%53% 52%

40% 37% 33% 31% 26%

% responding positively to the below statements

@KCLWhatWorks

Williams, D. 2006. On and off the ’net: scales for social capital in an online era. Journal of Computer-Mediated Communication, 11(2), 593–628.


Data Collection

Objective

Comparable

Patterns

Structures

Tendencies

Measurement

Behaviour

Attitudes

QualitativeHow?Why?

QuantitativeHow much?

How many?

What people do or how

they act unconsciously

Self-reported

preferences, recognition

and memories

Subjective

Difference

Meanings

Context

Experiences

Perceptions

Documentary Analysis

Journey Mapping

Observations

Experiment

Causal Data Analysis

Correlatory Data Analysis

Simulations

Eye Tracking

Closed Survey

Structured Interview

Open Survey

Participatory Observation

Semi-structured InterviewUnstructured Interview

Prompts and visuals

Focus Groups

@KCLWhatWorks

v

v

v

vv


Selecting the right research method

Level 1 - Monitor

Level 3 - Identify

Level 2 - Compare

RCT/ quasi-exp.

method

Dosage-response

Comparing with non-

participants

Qual research w/ participants and comp.

group

Rationale/ Theory of

Change

Secondary Research

Tracking

Pre/Post

Qual research w/ participants

only

@KCLWhatWorks


Level 1: Monitor


Level 1: ToCThere is a BAME attainment gap at King s

Reduce the BAME attainment gap using an intervention inspired from behavioural insightsAims

Rationale &

Assumptions

Students will sign up and complete tasks. The intervention won t negatively influence attainment. Faculties willing to take part. Tasks can increase attainment. We can afford the incentives.

• Enrolment survey

• Initial/evaluation

survey

• Staff time

• Incentives

• Video budgeting

• Stakeholder time

• Student advice

• Student time

• Director s input

• Commissioning

board input

• Research time

• Qualtrics

• Comms

development

• Ethical review time

ImpactInputs Activities Outputs Outcomes

• Analysing enrolment

data

• Analysing initial/

evaluation survey

• Sending out survey

• Recruiting students and

faculties

• Sending comms to

students/staff

• Ethical approval

• Creating materials

• Reporting findings

• Stakeholder

consultations

• Support staff recruitment

• Students completing

tasks

• Student opt-in

• Enrolment task

responses

• Movement between

initial/evaluation

survey

• Task responses

• Reports

• Blog posts

• Presentation to

commissioning board

Short:

• Increase in sense of belonging and self-efficacy

• Improved personal tutor interaction

• Increase in social capital

• What Works establishing itself with its 1st flagship project

• Faculty/stakeholder relationships built

Long:

• Increase in BAME students getting a good honours degree

• Improved King s

knowledge of

psychological

factors influencing

attainment

• Reduction in

BAME attainment

gap

Situation1

2

3 45 67

8

Process Impact


Level 2: Compare


Level 2: Confounding variables

Other studies:

• Wellness programmes: Jones, D., Molitor, D., & Reif, J. (2018). What Do Workplace Wellness Programs Do? Evidence from the Illinois

Workplace Wellness Study (No. w24229). National Bureau of Economic Research.

• Online advertising: Gordon, B. R., Zettelmeyer, F., Bhargava, N., & Chapsky, D. (2018). A comparison of approaches to advertising

measurement: Evidence from big field experiments at Facebook. Retrieved from https://ssrn.com/abstract=3033144.© Behavioural Insights Ltd

7.0

4.7

Naïve Naïve with covariates Post-consent randomisation

Effect of ‘Study Supporter’ on attendance rate8.4


Level 3: Identify


Level 3: ‘What Works’ (RCTs)Your King’s Your Success pilot Jan- March 2019

Control (not contacted)

2nd years invited to opt in

Belonging task & video

Belonging task

Belonging video

301 opt in to trial and complete

baseline survey

Personal Tutor video

Study skills video

214 fill out endlinesurvey

Personal Tutor task

Study skills task

Personal Tutor task & video

Study skills task & video

Change in attainment from 1st to 2nd year compared for each

group

February 2019 March 2019 September 2019

@KCLWhatWorks


Consider the context of an evaluationResearch question

• For example, an experimental design can only tell you whether something works, not how or why (mixed methods can help here).

Sample size

• Ideally over 1,000 across treatment and control, 200ish if good baseline data on outcome.

Data

• Collected consistently and universally (administrative datasets are best).

@KCLWhatWorks 24

Fidelity and validity

• Try to maintain protocol consistency within your intervention; this can sometimes be difficult—and can sometimes reduce external validity.

Ethics

• If there is substantial, consistent, high-quality evidence that something is effective, it shouldn’t be withheld from anyone who could benefit.


Other examples at King’s:

• Does sending a text message increase attendance at Study Abroad briefings?

• Do phone calls from 2nd and 3rd years reduce non-continuation among 1st

years?

• Do metacognition exercises increase metacognitive skills, and does this increase attainment?

• Do bursaries improve outcomes for low-income students?

@KCLWhatWorks 25


Lessons from the field

1. Know the data journey

2. Get to know the system and the people

3. Look for SMART interventions

4. Communicate and monitor regularly

5. Use behavioural insights

6. There is only so much you can do

7. TASO is here to help

@KCLWhatWorks 26

Why does this matter?


Securing funding

Doyle, M., & Griffin, M. (2012). Raised aspirations and attainment? A review of the impact of Aimhigher (2004–2011) on

widening participation in higher education in England. London Review of Education, 10(1), 75-88.

Aimhigher

✓ 35% of HEIs attributed increased applications to their

institutions to Aimhigher

✓ Associated with rising GCSE results

BUT no evidence that it actually impacted

progression rates, particularly to selective universities

Discontinued in 2011

The Office for Students (and before that, HEFCE and

OFFA) has consistently signalled they don’t have

confidence in the evidence underpinning institutions’

access activities, both in public statements and in their

comments on Access and Participation Plans.

Policy developments like Augar put WP funding in a

perilous situation. Now is the time to put

methodological differences aside and to start sharing

insights to get our evaluative findings out there


‘What Works’ is a beginning, not an end

‘What works’

doesn’t have to be

‘one size fits all’

Focus on ‘what works’

doesn’t need to narrow

the scope of

interventions to what is

easily measurable.

Knowing what works, for whom

and under what circumstances is

easier with a good identification

strategy (e.g. randomisation)


@Taso_HE


@KCLWhatWorks

Thank You.

[email protected]

mailto:[email protected]

how we know what works (and what doesn’t) in wp...university. 2. to promote empiricism and...

Documents