The Value of Rubrics in Messy

Non-Profit Evaluation Contexts

Kate McKeggAmerican Evaluation Association Conference

Washington DC, 2013

2

Aim of this sessionDevelop an understanding of how the use of rubrics in the non-profit sector can contribute to:• Organizational learning• Internal evaluation capacity• Improved client outcomes

What are evaluative rubrics?• An explicit way of defining ‘quality’ and ‘value’

for ourselves and for others:– Identifies and clarifies what aspects are important– Identifies criteria to assess how good is ‘good’– Allows us to make a judgment, using these

criteria, about how good, or excellent (or poor) something is.

Evaluative rubrics - a useful tool• Makes the ‘values’ step

transparent and explicit • A broad-brush way of

transparently defining what excellent, good, (etc) quality, value or performance would look like in practice

• A process and framework for being explicit about: – What criteria are used to

make evaluative judgements

– The types of performance we focus on

General guide for deciding performance(E.J. Davidson 2010)

Mercalli Intensity Scale (1906) for earthquake impacts

8

Data + rubric & evaluative sensemaking = reasoned evaluative judgment

Rating Descriptor

Excellent interpersonal communication

All of the features listed under ‘Very Good’ and in addition, all of the following:

Health and social care professionals listen and respond to people in a manner that enables informed decision making in an atmosphere of genuine collaboration

Health and social care professionals communicate in a way that reduces emotional stress and anxiety

Health and social professionals are sufficiently skilled and supported to effectively communicate with all those affected by cancer, including Māori, Pacific peoples and those from other ethnic minorities (supported by cultural advisors where appropriate)

Very good interpersonal communication

All of the features listed under ‘Adequate’ and in addition, all of the following:

Those affected by cancer are enabled, throughout the patient pathway, to discuss issues and ask questions concerning their cancer, its proposed treatment and potential impacts

The preferences of people affected by cancer, regarding their treatment, support and rehabilitation options, are assessed on an ongoing basis and are documented in the person’s individual medical file and the records of other providers

Communication of significant or distressing news is delivered in a suitable environment, by a senior staff member who has received training in advanced communication skills, or by a senior staff member supported by someone who has received training in advanced communication skills

Health and social care professionals actively seek cultural advisor support where appropriate, when communicating with Māori cancer patients and their whānau, and those affected by cancer from other cultural groups (e.g., for Pacific and Asian groups where the support services exist)

Effective clinical supervision is provided to (and utilised by) health professionals involved in caring for people with cancer, to help them explore emotions and reactions with their patients and to help avoid blocking techniques

Communication skills training is incorporated into all health and social professionals’ training curricula and ongoing professional development training programmes

in Good interpersonal communication

Information resources meet all of the minimal requirements listed under Adequate and in addition, several of the higher quality criteria listed under Very Good.

Adequate interpersonal communication

All of the following:

In all patient consultations, health professionals allow time to discuss a person’s particular cancer condition, its treatment and support options, and any related problems and issues as identified by the patient/ whānau

Cultural support advisors, trained patient advocates and interpreters are available (and are accessed) within mainstream hospital-based cancer services to support Māori and Pacific peoples and their carers and those from other cultural groups whose first language is not English

Communication skills training programmes that include inter-cultural

RubricsWhat does it mean overall?• excellent? • very good?• good?• adequate?• inadequate?• poor?Where are the opportunities to improve?

Report

New data we collect

Existing data

• Interviews• Focus groups• Survey• Milestone reports• Programme data• etc

Clear overall evaluative judgement

Sense makingAnalysis

9

Rubrics and organizational learning

10

Ministry of Culture and Heritage & Minister

Contributing Schools

Auckland Philharmonic Orchestra

Programme Director, mgmt, staff & volunteers

Community Supporters and Organisations

Sistema Families

Is this programme worth continuing to fund?

How valuable is the programme to the community, to schools, to the children and their families?What can we learn that we can share with others and apply in other settings?

How effectively have we implemented the programme How well are we delivering the programme?Are there things we can improve and do better?

How effective is the programme in supporting the aspirations and success of our local children?How effectively is the programme contributing to our community?

How well does the programme influence children’s learning, behaviour and engagement in education?Is it worth us continuing to support children’s involvement?

How well does the programme support my child’s aspirations and wellbeing?

Programme Dimension

Evaluation Criteria

Content and design

• Group-based learning • incorporates child development theory and practice • Reflects the community and its cultural diversity• Standardised principles and use of language and processes• Clear progression pathway• etc

Programme Delivery

• Systems and approaches• Human resources• Finances and sustainability• Relationships• etc

Outcomes of Value• Children• Families• Schools• Community

• Children experience a sense of accomplishment and success• Children experience an increase in their self-esteem, confidence and

motivation • Families are proud of their children• Families and children expand their horizons and their sense of opportunity• Children’s behaviour at home, and with their families is more positive• The programme makes a contribution to the development of a wide range

of new skills, behaviours and competencies• The community has a sense of pride and ownership in the programme• Communities are positively transformed

13

Evaluative Criteria guided data collectionData Matrix for determining data collection requirements

14

15

Data analysis and synthesis using evaluative criteria

16

Generic rubric used for Sistema

17

Participative data synthesis – example data pack info

Reporting on the programme’s performance e.g., design, content and delivery

19

So what happened as a result of using rubrics in this case?

• Clear direction about where improvements could be made

Thank you Kate for this report! It has been great to work with you on this, and I'm looking forward to the next stage (email from Joe Harrop, Programme Director)

• Clear evaluative judgments that could be reported in a way that even Senior Government Managers and Ministers could understand! And this lead to further funding for the initiative by the Ministry of Culture and Heritage

Our thanks… for a thorough report, which confirms the value of this programme (email from Barbara Glaser, Chief Executive, Auckland Philharmonia Orchestra).

“The value we’re getting from Sistema is huge. We would never consider pulling away from the programme.” Minister of Culture & Heritage

20

Rubrics as a mechanism for building organisational and

evaluation capacity & improving outcomes

21

Evaluation criteria & rubrics driving capacity – org + evaluation

22

Evaluative questions – driving data

collection

23

Rubrics provide the organisation with a performance pathway

24

Changes in student outcomes

25

Organizational and Evaluation Capacity

26

Rubric driving client outcomes

Self Managing

Cohesive, Resilient and

Nurturing

Living Healthy

Lifestyles

Participating Fully in Society

Economically Secure

Confidently Participating

in Te Ao Maori

Blood Pressure

Weight

Waist measurement

HBa1c

Cholesterol CV risk

Clinically Secure

Kia Rite Kia Ora – Diabetes Prevention Programme

28

Ladder of whānau change for the outcome Self Managing

29

30

31

Whānau living the dream…

We've come almost to the conclusion of our 12 months

and we are starting to live the dream.

We will continue because this programme has been so

awesome, it's made us draw closer to our whānau (family).

33

Whānau being independent…

Kia Rite Kia Ora is about evaluation, independence, reliability, being

honest within yourself. And this is where they actually help us to grow and strengthen within ourselves to get out there and be independent, and that’s what this programme is

all about.

34

Whānau being whānau…

It's not just about us anymore it's about the new addition to the whānau (family) & our moko (grandchildren). It’s given us

even more meaning to make all these changes for all of us as a

whanau so we can be around as long as possible.

35

THANK YOUContact Kate McKegg: kate@kinnect.co.nzwww.knowledgeinstitute.co.nz