Becoming Impacts-LedEuroPCom 2019:

Checking the numbers: evaluation of communication campaigns

07 November 2019

Richard AddyCo-Founder & Director, AKAS

www.akas.london 2

The Problem: Most non-profit institutions struggle to track impact. They track outputs & outcomes. Some wrongly label impacts.

Types of metrics tracked (US, 2016)

Output metrics

Reach (Outcome)

metrics

Engagement(Outcome)

metrics

Impact metrics

80% 84% 70% 31%

Source: Forum One survey of 92 employers across research institutions, advocacy

organisations, associations, foundations, academic institutions and federal

government (2016)

www.akas.london 3

A possible approach: An impacts framework developed through working with media, NGOs, foundations & global institutions

Source: AKAS (2019)

www.akas.london 4

1.Strategic Inputs &

Processes?

2. Content &Outputs?

3. OutcomeMetrics?

4. Publics?

5. Influencers?

6. Decision Makers?

7. Ultimate?

CampaignSuccess?

One way of thinking about this is that we are creating buckets which capture your campaign or project successes

The framework tries to capture all campaign & project successes

Source: AKAS (2019)

2. Outputs/ Content

3. Outcomes

6. Decision

Maker-BasedImpacts

5. Influencer-

Based impacts

4. Public-Based

Impacts

“Real World” Impacts 1.

Strategic Inputs &

Processes

7. Ultimate Impacts

Long TermShort Term Medium Term

Mission (Why you exist)

Vision (Where you want to be in 3-5 years’ time)

Campaignobjectives

Reach

Engagement

Volume of content

Type of content

Face to Face Interactions

Strategic

People

Tech & Infrastructure

Operational

Individual Decision-Maker based impacts

Deliberative-based impacts

Substantive-based impacts

Individual case studies/ stories (Qualitative)

Changes in Awareness, Attitudes or Actions amongst influencers

Amplification

Recognition & Awards

Individual case studies/ stories

Changes in Awareness, Attitudes or Actions amongst the public

Collective Action

Individual case studies/ stories

It’s like a menu. You can choose to collect data and stories on some (or all) of the sub categories of 7 framework buckets

Source: AKAS (2019)www.akas.london 5

www.akas.london 6

What we’ve

learned

www.akas.london 7

You need to be clear and systematic about the audiences that you are trying to engage: publics, influencers or decision makers

Source: AKAS (2019)

Wider Audiences Influencers Decision Makers

Readers/Viewers Traditional Elite Media President (Office of)

General Public Mid-market/ Tabloid Media Prime Minister (Office Of)

Informed Public Online/Social-based Media Finance Minister

Concerned Public International Institutions (IGOs, MLOs, IFIs, etc) Cabinet Ministers

Engaged Public Regional Institutions Ministers

Disgruntled Public Foundations Chairs of Parliamentary Committees

Citizens NGOs Members of Parliament

Cut by Gender Women's Rights Groups Senior Civil Servants

Cut by Age Youth Groups Local/ Regional/ State Elected Representatives

Cut by Geography Consumer Associations Local/ Regional/ State Senior Officials

Cut by Social Class Faith-based Groups Government Agencies

Cut by Income Think Tanks Central Banks

Academics Judicial Organisations

Trade/Labour Unions

Business Associations

Hgh Net Worth Individuals

High Profile Individuals

Celebrities

You should also use the full set of tools to track impacts

www.akas.london 8Source: AKAS (2019)

Mainly“Qual”

Mainly“What”

Mainly“Why”

Mainly“Quant”

www.akas.london 9

Mainly“Qual”

Mainly“What”

Mainly“Why”

Mainly“Quant”

Survey-based

Analytics-based

Ethnographic-based

High-LevelObservation –

based

You should also use the full set of tools to track impacts

Source: AKAS (2019)

You should also use the full set of tools to track impacts. Most institutions are focusing only on analytics.

www.akas.london 10

Mainly“Qual”

Mainly“What”

Mainly“Why”

Mainly“Quant”

Example:Attitudinal Surveys

(Proprietary or Syndicated)Longitudinal SurveysOnline Communities

EconometricsTheoretical Frameworks

(At Scale)

Example:Big Data Analytics

Web AnalyticsSocial Media Analytics

Behavioural SurveysBehavioural Science Experiments

Textual AnalysisPropensity Score Matching

Search Analytics

Example:EthnographyFocus Groups

InterviewsVerbatim Analysis

Diary AnalysisMood BoardsCase Studies

Example:Casual Observation

UX LabsBiometric Response

Eye TrackingFacial Analysis

Mystery Shopping

Most ImpactMeasurement is locked in this

quadrant

Survey-based

Analytics-based

Ethnographic-based

High-LevelObservation –

based

Source: AKAS (2019)

You should track the impact of all of your outputs: including publications, articles, posts press releases, speeches and videos

Media Institutions Campaigners,NGOs Projects

Articles

Press Releases

Social Media Post

Video/ Audio

Speeches

Interviews

Publications/Reports

Key meetings/ conferences

Visits/missions

Key outputsto track

Most likely outputs to track’ to assess impact (depending on the client type)

www.akas.london 11Source: AKAS (2019)

R² = 0.4599

0

5

10

15

20

25

30

35

10 100 1,000 10,000 100,000 1,000,000 10,000,000

www.akas.london 12Source: AKAS MLO Database (2019)

No of Twitter Followers(July 2019)

Average no. of

tweets per

month

Your outputs are critical. Our twitter analysis of 116 MLOs found a relationship between twitter posts and twitter followership

Twitter followers compares to monthly tweets (116 MLOs, July 2019)

1Clear

Mission

3Senior-

Level BackingFor Impact

5EngageDirectly

WithAudiences

6Clarify

How YourAre Targeting

www.akas.london 13

8Innovative

DataCapture

7Compelling &

AccessibleContent

9Regular

PerformanceReporting

4Comprehensive

Impacts Framework

10Disciplined

Approach ToCampaigns

2Powerful

Organizational Narrative

But having an impacts framework is not enough – you need to embrace other factors to become an impacts-led organisation

Key Dimensions of an Impacts-led organisation

Source: AKAS (2019)

www.akas.london 14

Summary: Becoming Impacts-LedA 10 step guide/checklist for developing your impacts approach

1. Mission/Vision: Establish why you exist and what you are trying to achieve

2. Decision Makers: Understand the changes that are needed amongst decisions makers

3. Influencers: Work out which influencers are most likely to be important

4. Public: Clarify your desired public impact in terms of awareness, attitude or action change

5. Short term media outcomes: Ask what kind of reach and engagement you will need amongst your audiences in the short term

Source: AKAS (2019)

www.akas.london 15

Summary: Becoming Impacts-LedA 10 step guide/checklist for developing your impacts approach

6. Output: Establish the volume and type of outputs (content, face to face engagements) that you will need to produce

7. Resources: Raise money/resources to deliver your mission/vision

8. Collect: Collect evidence of your success using the impacts framework and the full range of tools once you get going

9. Review: Formally review your campaign or project at key intervals

10. Adjust: Adjust your impacts strategy using the findings from your impacts measurement

1. Mission/Vision: Establish why you exist and what you are trying to achieve

2. Decision Makers: Understand the changes that are needed amongst decisions makers

3. Influencers: Work out which influencers are most likely to be important

4. Public: Clarify your desired public impact in terms of awareness, attitude or action change

5. Short term media outcomes: Ask what kind of reach and engagement you will need amongst your audiences in the short term

Source: AKAS (2019)

Becoming Impacts-LedEuroPCom 2019:

Checking the numbers: evaluation of communication campaigns

07 November 2019

Richard AddyCo-Founder & Director, AKAS