Measuring the Performance and Impact of Community Indicator Systems:

Key Performance Indicators and Evaluation

Charles Thrift & Bobbie Macdonald

February 17, 2011

Outline

1. Introduction

2. Why Evaluation and KPIs?

3. Peg evaluation framework

4. Challenges & lessons learned

5. Discussion & Next steps

Introduction

Purpose:

1. To demonstrate the value of KPIs

2. To present the Peg evaluation framework

3. To elicit feedback and discussion on KPIs

Key Performance Indicators (KPIs) are quantitative or qualitative measures that assess the performance, progress, and impact of a project, program, or organization.

http://mentat.pineda.ca/blog/?p=1262

Introduction

• Program/Project Evaluation is a formal approach to assessing progress, performance, and impact.

• KPIs provide data and information on which to base findings and conclusions.

• “Action without reflection is wrongheaded; reflection without action is self-indulgent.” - Nancy Scheper-Hughes, Death Without Weeping, p. 171

Why Evaluate a CIS?

• to assess CIS outcomes or impacts• to assess whether CIS objectives are

being met • to find out how CIS is operating• to understand who is using CIS and

how• to find out how well users’ needs are

being met• to improve the CIS (to better meet

community needs, better manage the initiative or make it more sustainable)

• to understand why CIS or elements of CIS do or do not work

Why use Key Performance Indicators?

• Reduce the number of decisions that are based solely on instinct and make decisions based on objectivity and facts.

• Quantify the achievement of goals by setting, monitoring and measuring against a standard or target.

• As your CIS grows it becomes more difficult to remain as close to the operational details as you once were.

• Allows you to focus on facts and see where you need to improve.

Peg evaluation framework

Process:

1. Develop long list of KPIs

2. Develop evaluation framework for Peg

3. Select KPIs directly relevant to Peg

1. Develop list of KPIs:

• Legwork: indicator system contributions + brainstorming = 133 KPIs

• 115 Quantitative, 18 Qualitative indicators

http://www.mynetresearch.com

+ =http://pickettsmill.typepad.com/ http://www.dreamstime.com/

2. Develop evaluation framework for Peg

• Inputs - resources required to undertake project activities

• Receiving - persons receiving our information

• Seeking - persons seeking information related to our work

• Engaging - joint efforts to foster change

• Using - persons using our framework or information for other purposes

• Observed Change - actual change in policies and indicators

3. Select KPIs directly relevant to Peg

• 54 Selected: 48 Quantitative, 6 Qualitative

Exit

Challenges

1. Finding KPIs relevant to community indicator systems

current list not exhaustive2. Deciding which KPIs to use

choosing too few vs choosing too many

internal vs external indicators

http://www.pmthink.com/

Lessons Learned

1. Existence of an evaluation and KPI gap in community indicator systems.

2. High value for labour ratio

Next steps

• 1. Extend KPI list to make it exhaustive.

e.g. environmental sustainability KPIs, more qualitative KPIs, ... .

• 2. Create dialogue to generate participation, buy-in, and uptake.

• 3. For Peg, take selected KPIs to internal groups.

Discussion

1. Is there value in using KPIs for community indicator systems?

2. How do we move forward with the momentum we’ve built?

3. How are you using KPIs now?

Charles Thrift

Project Officer, Measurement & Assessment, IISD cthrift@iisd.ca

Bobbie Macdonald

Research Student, Measurement & Assessment, IISD macdonald-b3972@webmail.uwinnipeg.ca

Comments/Questions?

Peg: www.mypeg.ca