Method The research was conducted through short semi-structured interviews. In total 35 individuals participated in the project, all of whom are SIAA members or part of the wider SIAA network and thus have first-hand experience of SIA. The participants interviewed work within the non-profit, social business, management consulting, higher education, and research sectors. Participants were from 16 different countries and collectively operate in 29 different countries.

Where we are now The Opinions and Practices of Social Impact Analysts

Key Findings

The lack of a formal definition of SIA, and of formal training, mean that approaches to SIA vary greatly. The resulting lack of engagement from the wider professional sphere means that participants lean towards identifying themselves in other ways that carry more professional recognition.

Most participants do not practise just one form of SIA and have an awareness of a variety of different methods that are available. The use of different methods was perceived as dependent on the needs of the organisation or programme.

Most participants think that there is a difference between SIA and evaluation. Opinions on what the differences are were lacking in consensus.

The majority of participants mentioned both the advantages and risks associated with good practice and professional standards. Many participants expressed apprehension towards any guidelines or standards that would be too specific or restrictive.

The research identified some of the common opinions and practices of social impact analysts in three key areas:

1. What is SIA? 2. Is there a difference between SIA and evaluation? 3. Opinions on good practice in SIA and the establishment of professional standards.

This summary report provides an overview of some of our key findings.

In the summer of 2013, SIAA collected the opinions of some of our members and supporters on key issues and questions within the field of social impact analysis (SIA).

A map of countries in which participants operate

Source: BatchGeo.com

2

0 2 4 6 8 10 12 14 16

Cost Benefit Analysis

Cost Effectivess Analysis

Economic modelling

Evaluating outcomes

Four Level Evaluation…

Logical Frameworks

No prescribed method

Qualitative Analysis

Quantitative analysis

RCT

SIA/SIM

SIE

Social…

Social Value Chain

Specific to Organisation

SROI

Stakeholder Analysis

Theory of Change

What forms of SIA do you practise?

Yes

47%

No

17%

Only if

asked directly

18%

Yes,

among other things

18%

Are you a social impact analyst?

When asked whether they would identify themselves as social impact analysts, 47% of participants said yes, including 70% of participants not based in the UK. The early stages of SIA as a career, the lack of formal training and the perceptions of the language used in the wider professional context were highlighted as reasons for the lack of engagement with the term. For these participants, identifying themselves in another way, such as as an SROI practitioner or consultant, carries more recognition in the professional world.

What is SIA?

The research highlighted that, due to the context-dependent nature of SIA, defining the key characteristics is challenging. General themes were outlined including stakeholder engagement, measurement of social value, guidance for future goals, interpretation of data, mission fulfilment, generation of practical discussion of the use of information, and moving beyond simply looking at outcomes. The purpose of the results and how they are communicated were also highlighted as important features of SIA.

Participants were asked to identify which approaches they used for SIA

Despite biases for one approach or another from some participants, it is evidently common practice to use more than one approach (59%). This is explained by the necessity of making SIA relevant to the needs of the organisation. 44% of participants indicated that they used no particular form.

Emma Tomkinson, Freelance Social Impact Analyst, Australia said:

“I don’t use any prescribed methods and it depends on what I am being asked to do. I look at proposals/policies and facts to do with people producing social impact and analyse what could be done better.”

Why is SIA valuable? The value of SIA was placed in maximising the impact of organisations through using an objective approach that optimises resource allocation, improves strategy and engages stakeholders. SIA was also indicated to be important for accountability to funders in terms of evidencing where funding is used and the difference it is making. Additionally, the communication of the successes and failures of a particular strategy and creating a wider understanding of impact were indicated as valuable features of SIA.

Steve Coles, Managing Director of Intentionality CIC, London, said:

“No, we wouldn’t talk of ourselves as social impact analysts – maybe social impact consultants…We are reflecting on whether the word ‘value’ might be more inclusive to stakeholders.”

The research revealed a distinction between two non-mutually exclusive approaches for SIA: quantitative (data-driven) and qualitative (people-focused). For some participants, SIA means a balance between both approaches. Participants suggested that, due to the lack of a formal definition of SIA, the approach taken was largely dependent on the practitioner’s professional background.

“SIA is the difference between measuring if books arrive to a community and if children learn to read from those books.” Anne Hand, Filantrofila, Mexico

“SIA is to capture the social value which is not calculated in the financial value.” Ken Ito, Asian Venture Philanthropy Network, Japan

3

No “The purpose of an analysis is to evaluate where it is going right or wrong.” “Social evaluation of a project is the same as SIA.”

Yes “SIA is focused on outcomes, on demonstrating them, and on communicating them. Evaluation

does not have a particular focus, in this sense it is broader than SIA. It can be used to examine not just changes made but processes used.”

“Evaluation has connotations of happening at the end of the programme. SIA is more commonly conducted at all stages (before, during, and after) than evaluation. SIA is more often used as a forecasting tool.” “Evaluation is often a more generic, established process. SIA is very specific to the organisation or individual conducting the analysis.”

“SIA tends to have a much wider scale, by looking at the impact beyond project level. Evaluations tend to be done on specific projects or aspects.” “Evaluation is more commonly conducted externally by independent evaluators. SIA is more commonly conducted by an organisation internally.”

Is there a difference between SIA and Evaluation?

This question proved to be a challenging and interesting one for participants to answer. The following are some opinions expressed by participants during interviews.

Yes

73%

No

9%

Maybe

18%

Maybe… “The difference is dependent on how we define analysis and evaluation.”

“The terms tend to be interchangeable in some organisations, and their different usage reflects the underlying philosophy of the organisation in question.” “There can be resistance to using terms other than evaluation, some professionals identify very strongly with evaluation.” “There is no definite answer to this question – internal vs external captures some of the divide but evaluation isn’t understood very well either… there is confusion around process evaluation vs impact evaluation.” “It depends on how you conduct the evaluation. Thorough evaluations would look at how the changes happen as well was what change has happened.” “My gut feeling is “yes” but the more I talk to evaluation experts the less I think so…”

SIAA is frequently asked this question, but does not have the answer yet. The research identified that there is not one definite answer and that people’s opinions are differing and often contested. It is interesting to see that although 74% of participants believe that there is a difference, there is little consensus around what that difference is. Some differences stated can be grouped under the themes of use, context and purpose. Those who were more uncertain about the differences provided some interesting insights; the challenges of context, language use and communication became apparent. This topic requires further in depth research in order to draw firm conclusions about differences and similarities.

4

Benefits Drawbacks

Standards would give the profession credibility.

Clients invest significant resources in SIA so it should be done well, standards could help regulate this and raise confidence in work.

It would be possible to draw comparisons between the practices in different sectors/ locations.

Without standards unqualified people can claim to be a social impact analyst.

Having standards increases technical support and backing.

There are a lot of uncertainties and professional standards could create a clearer working environment.

Standards would help to develop training programmes and educate people in SIA.

SIA is too dependent on context to have professional standards.

It would be difficult to enforce standards if there is no clarity about what SIA is.

It could make the profession exclusive.

We need more standardised methodologies before introducing professional standards.

It might make SIA more expensive so clients cannot afford to hire social impact analysts.

There is a risk that standards would not be very high and would do damage to the reputation of the profession.

It is more important to have hands-on experience than professional status.

Good Practice & Professional Standards

The majority of participants supported the idea of good practice in SIA. A lot of participants prefer to use internal guidance developed by their own organisation or country-specific forms of guidance. Some also said that they do not use any particular form of guidance.

Key issues identified regarding good practice included:

Theory vs. Practice Many saw good practice as a balancing act between achieving rigour and practical application of SIA. “We need easier and more practical tools for social purpose organisations to work with that must be a compromise between academic soundness and practical applicability.” Jaan Aps Stories for Impact, Estonia “People need to understand different approaches [to SIA] and have rigorous standards, but the dark side is that there is a risk of professionalisation and mystifying processes that are quite straightforward.” John Healy Independent consultant, Ireland.

Context Dependency It was suggested that the context specific nature of SIA may be a barrier to developing and applying international good practice guidelines. “[Good practice guidelines] only work in a given society, time, place, social, economic environment. It is never possible to take good practice as a whole.” A representative from the Revita Foundation in Hungary “The problem with good practice would be defining something that isn’t too generic, but not too specific either, and can help everyone.” Philipp Hoelscher, Phineo, Germany.

Professional Standards

Most participants recognised the role professional standards could play in developing the sector and increasing the quality of the work produced. However, participants also mentioned the risks and drawbacks that might be associated with the development of professional standards at this point in time.

Some benefits and drawbacks of professional standards: