case for outcome focus

7/29/2019 case for outcome focus

1/14

Ch ar i t ies Eva lu at io n Ser vi ces

The case for an out comes focus

Dr Jean Ellis

Charit ies Evaluat ion Services

Sept ember 2009


2/14

Ch ar i t ies Eva lu at io n Ser v ices

Contents

PageI n t r o d u c t i o n 1

The Na t i ona l Ou t com es Prog ram m e 1

Th e w i d e r t r en d t o w a rd s a n o u t c o m e s

approach 1

UK go ve rnm en t p o l i cy 2

Th e Sco t t ish Execut ive 3

Ou t com es i n t h e t h i r d secto r 3

Ou t com es and Va lue f o r M on ey 4

The b ene f i t s o f an ou t com es app roach 4

US f in d in g s 5

UK f i nd ing s 6Som e f i na l p o in t s 8

Fur th er read ing 9

References 10

ISBN 97 8-0-95 58 84 9-3-1 (on l in e) Ch ar i t ies Evalu at io n Ser vices 2 0 0 9


3/14


The case for an out comes approach

Introduction

Th is pape r p laces the ado p t i on o f an ou tcom es app r oach by th i r d sec to ro r gan isa t i ons w i th in i t s w id e r con tex t , and r ev iew s som e o f t he f i nd ings on th evalue to be ob ta ined f r om an ou tcom es focus.

The Nat ional Outcomes Programme

Char i t ies Evaluat ion Serv ices (CES) has received seed fu nd ing to t urn aninnova t i ve and w ide -r each ing ou t com es in i t i a t i ve , p r ev iousl y g r an t - fund ed , i n to

a se l f -susta in in g p rog ram m e. The Nat io na l Out com es Program m e, de l ive redand r un by CES w i th f und ing f r om the B ig Lo t te r y Fund , had a t i t s co r e a two -day t ra in in g cou rse , de l ive red f i r st t o ou tcom es cham pio ns , th en cascadedth r ou gho u t the n ine Eng land r eg ions to f r on t l i ne o r gan isa t i ons. Over thepe r iod 20 03 t o 200 9 , t r a in ing w as de l i ve r ed to 169 ou t com es champ ions andapp r ox im a te l y 1 ,50 0 vo lun t a r y and com m un i t y o r gan isa t i ons.

The p r og r am m e was i ndepend en t l y evalua ted i n 2006 and 2 009 b y the OpenUniversi ty and Tr iba l Con su l t ing respec t ive ly. The Op en Un iversi ty f ou nd th a ta f te r t h e f i r st t h r ee yea rs o f t he p r o g r am m e, con f i dence w as h igh , w i th over

90 pe r cen t o f ove r 1,000 pa r t i c i pan ts ag r eeing o r st r on g ly ag r ee ing th a t t h eyhad th e too l s and un de r stand ing t o i n t r odu ce an ou tcom es app r oach . Them a jo r i t y o f o u t c o m e s c h am p i o n s h a d st a r t e d t o i m p l e m e n t t h e a p p r o a ch i n

the i r ow n o r gan isa t i ons, and the m a jo r i t y o f f r on t l i ne vo lun t a r y and com m un i t yorgan isa t ion s rece iving t ra in in g w ere seek ing t o sh i f t t ow ards, deve lop , o rim p r ove an ou tcom es focus i n t he i r o r gan isa t i ons.

The eva lua t ion o f t he second ph ase, 20 06 -20 09 , a lso show ed sign i f i can ti nc r eases i n u nde r stand ing o f ou t com es and o f h ow to exp r ess ou t com es in a w aytha t stakeho lde r s and p a r tne r s und e r stoo d , and how to co l l ect da ta . Cham p ion s

r ep o r t e d t h a t l e ar n i n g f r o m t h e p r o g r a m m e h a d h e l p e d t o d e ve lo p r e la t i o n sh i p sw i th t he i r f und e r s. The eva lua t i on a l so fou nd tha t , a lt ho ugh i t w as d i f f i cu l t t osepa r ate ou t t he con t r i bu t i on o f t he Nat ion a l Ou tcom es Pr og r am m e ac t i v i t i es

f r om the i nc r easing aw ar eness o f o u tcom es m or e gene r a l ly , t he p r og r am m ehad r a i sed awar eness o f t he r o le and t he im po r tance o f ou t com es m on i to r i ng ,con t r i b u t i ng t o estab l i sh ing th i s as pa r t o f t h e m a inst r eam agenda .

The wider t rend t owards an outcomes approach

Heal th services have led t he w ay in an em ph asis on ou tcom es. Research in tohea l th ou t com es and c l i n i cal d r ug t r i a l s developed w ide l y i n the pe r iod o f p ost -second w or ld w ar pharm aceut ica l expans ion . The accept ance o f ev idence-basedm ed ic ine has had t he r esu l t t ha t som e p r ac t i ces tho ug h t o f as best c l i n i ca l

p rac t ice have fa l len ou t o f favour .

From th e 1990 s, th ere have been inc reasing m oves in t he pu b l ic sec to r in t he USand o the r w este r n econom ies to add r ess ou t com es as a key pa r t o f p e r fo r m ancem easurem ent . Th is has been in par t because serv ices focus ing o n ou tp u t s o f ten

1


4/14


r equ i r ed cost l y r ev iew w hen u se r needs and b ene f i t s had n o t been su f f i c i en t l ycons idered . Advantages cou ld a lso b e seen in th e added va lue th a t cou ld b em ode l l ed and est im a ted i n t e r ms o f pop u la t i on o u tcom es. Fo r exam p le , l ear n ingo r hea l th i n com es im pac ted on im p r oved p r o duc t i v it y , i nc r eased t ax r evenue ,few er d ays lost th rou gh sickness and d isab i l i t y , and so o n 1.

UK government policy

In th e UK, vo lun ta r y sec to r o rgan isa t io ns de l ive r ing pu b l ic serv ices op era tew i th in po l i c y and r egu la to r y r equ i r emen t s w h ich ove r p r ev ious yea r s haveinc reasing ly incorpo ra ted an ou tco m es focus . The 1998 Com prehen sive Spen d in gRev iew in t rod uced Pub l ic Serv ice Agreem ent s (PSAs), se t t ing ou t agreed ta rg e tst h a t a ll d e p ar t m e n t s2 w er e expected t o de l i ver and r epo r t . An i nd ependen tstu dy ana lysed th e ta rge t s in t he 1998 PSAs and fo un d t ha t t hey were m a in lypro cess ta rg e ts (51 per cen t ) and o u t pu t t a rge ts (27 per cen t ) w i th a re lat ive lysm a l l p r opo r t i on o f ou t com e ta r ge ts (11 pe r cen t ) . How eve r, t hese ta r ge ts

becam e inc reasing ly ou tcom e-focused in sub sequ ent spen d in g rev iews.3

A laterr epo r t by th e Na t iona l Aud i t O f f i ce sugg ested tha t by th e 200 0 PSAs, 67 p e r cen to f t he ta r ge ts we r e ou t com e ta r ge ts, w i th 14 pe r cen t p r ocess ta r ge ts and 8 p e rc en t o u t p u t t a rg e t s, w i t h t h e t r e n d c o n t i n u i n g .

The num er ous gove r nm en t depa r t m en ta l and no n -depa r t m en ta l agenc ies p laya key ro le in d e l iver ing pu b l ic serv ices. The ach ievem ent o f g overnm ent t a rge tsthe r e fo r e r equ i r es agency p r i o r i t i es to be a l i gned w i th t hose o f g ove rnm en tdepa r t m en ts, and agency pe r fo r m ance ta rge ts to be cons isten t w i th t he de l i ve r yo f t he PSA ta rge t s. H igh -level ou tcom e ta rge t s a re t ransla ted in to op era t ion a lta r ge ts on t he g r ou nd . One o f t he cha l l enges i s to de f i ne an i nd i v idu a l agency s

d i st i nc t con t r i b u t i on tow a r ds the w ide r po l i c y ob ject i ve so th a t a clear ou tcom ecan be a r t i cu la ted fo r t he agency i t se l f .4

Recent g overnm ent p o l icy docu m ent s have u rg ed com m ission ers o f serv ices topu t ou t com es fo r user s a t t he hea r t o f t he s t r a teg i c p lann ing p r ocess, and fo rth e ach ievem ent o f ou tcom es to be u sed as a key ind ica to r o f success in serv icede l ive ry (Cab in e t O f f ice , Of f i ce o f th e Th i rd Sec to r , 2006 ). Since 20 08 , as ar esu l t o f r e fo r m s se t ou t i n the Dep a r tm en t o f Hea lth W h i te Pape r Our hea l th Ou r care Our say(Depa r tm en t o f Hea l th Jan 200 6 ) ,5 pr im ary care t ru sts and loca lau th or i t ies have been expec ted t o repor t on p rog ress aga inst st ra t eg ic ou t com es.

The Loca l Governm ent Assoc ia t io n has cam pa ig ned fo r a na t iona l ou tcom esf r am ewor k , i n l i ne w i th a m ove away f r om a p r edom inan t focus on i npu ts and

1 For examp le, see a New Zealand stu dy by Pr icew aterh ou seCoop ers (Jan 20 03 :22-30 )

2 There are PSAs for th e m ain go vernm ent dep ar t m ent s, to get her w i th PSAs for c ross-de-pa r t m en ta l a reas o f po l i c y w here a ll t he depar t m en ta l t a rge ts rel evan t t o d e l i ve ry o f t he go ve rn -m en t s ob j ec t i ves i n t ha t a rea a re d rawn tog e the r i n a si ng l e ag reem en t . C ross-depar t m en ta l PSAscan be a use fu l t oo l f o r coo rd i n a t i ng i n t e r -depar t m en ta l and i n t e r -agency work w here severa ldepar t m en ts are i nvo l ved i n de l i ve ri ng a num ber o f re l a ted ou t com es, and th e task i f o f ma jo rim por tance to th e gove rnm en t s key ob jec t i ves.

3 See HM Treasur y , Spen din g Rev iews, up dat ed to 20 05 , h t tp : / /w w w .hm -t reasur y .go v .uk / spend ing_ rev i ew/spend_p lancon t ro l . c fm

4 See und a ted HM Treasu ry paper : Ou tcom e Focused Manag em en t i n t he Un i ted K ingd omht t p : / /w w w.hm _t reasu ry .gov .uk /m ed ia /1A4 /OF/GEP_ou tcom e%20fo cused%20m anagem en t .pd f

5 See a l so Depar t m en t f o r Comm un i t i es and Loca l Governm en t (Oc tob e r 20 06)

2


5/14


pro cesses, to w ards a focus on p ar tn ersh ip and loca l ou tcom es.6 These o u tcom esw ou ld be i nc luded i n eve r y counc i l s com m un i t y st r a tegy and t he st r a teg i c p lanso f o th er pu b l ic sec to r p rov iders. The Loca l Area Agreem ent is p ro v id ing th e foca lpo in t f o r t he co r po r a te p lan and the com m un i t y st r a tegy and r equ i r e de l iver yorgan isa t ion s to focus on t he end resu l t s o f t he i r se rv ices and t he b enef i t s fo r

users.

The Scot t ish Execut ive

The Scot t i sh Execu t ive a lso m ade i t s po l icy c lear in 20 06 :

Our na t i on a l p r i o r i t i es m ust b e abo u t o u tcom es the r ea l im p r ovemen t s tha t peop le see i n the i r com m un i t i es and i n t he i r l i ves be t te r hea l th ,

r educed c r im e and an t i soc ia l behaviou r , an im p r oved env i r onm en t and i nc r eased edu ca t i ona l a t ta inm en t .

I t w en t o n to say:

W e n o w a im t o g o f u r t h e r, a n d t e st o u t w a y s o f u si n g o u t c o m e ag r eem en ts r i gh t ac ross the fu nc t i on s o f a l oca l au th o r i t y o r Com m un i t y P lann ing par tnersh ip .7

Outcomes in t he third sector

The 1990 s saw p ionee r ing w o r k on ou t com es w i th vo lun ta r y o r gan isa t i ons insom e sec to r s. I n 1993 , the Depa r tm en t o f Hea lth dec ided to t r i a l an ou tcom es

app r oach w i th g r an t -fund ed o r gan isa t i ons and A lcoho l Conce r n p i l o ted w o r kw i th a l coho l agenc ies on how ou t com es m on i t o r i ng cou ld b e used . Housingand h om elessness o rg an isa t io ns have been exp lo r ing th e use o f ou tcom esm easurem ent s system s since th e la te 1990 s, d r iven in la rg e par t by th ei n t r o d u c t i o n o f t h e Su p p o r t i n g Pe o p l e f r am e w o r k ,8 bu t a l so by th e Lond on

Hous ing Fou nd at ion s IMPACT capac i ty bu i ld ing p rog ram m e.9 The gove r nm en tm ore recen t ly spo nsored p i lo t p ro jec ts in h om elessness adv ice10 a n d c o m m u n i t yd e v e l o p m e n t .11

The need fo r p ub l ic serv ices to assess th e i r ow n ou tcom es and d e te rm inehow these a r e con t r i bu t i ng to gove r nm en t ta r ge ts i s a l so an i ssue fo r t h i r d

sec to r o rgan isa t ion s, inc reasing ly d raw n in t o t he d e l ivery o f p ub l ic serv ices

6 See ht tp : / /cam paig ns .lga.gov .uk / 7 See The Scot t ish Execut ive (Jun e 20 06 ) Transfor m ing Pub l ic Serv ices: The Next Phaseo f Re fo rm . Chap t e r Ei gh t : St ruc t u res, Ou tcom es and Peop le , pa ras 84 and 86 . See h t tp : //ww w.scot land.gov .uk /Publ i ca t ions /2006/06/15110925/9

8 Supp or t i ng Peop le w as l aunched i n Ap r i l 200 3 , ensu r i ng tha t vu l ne rab le peop le havehe lp and sup por t t o l i ve i ndep enden t l y. A round t w o- th i rds o f t h e p rog ram m e i s de l i ve red by th i rdsec tor o rgan isat io ns .

9 See w w w. l h f .o rg .uk /

10 A p i l o t p ro j ec t se t up i n 200 4 /05 i n respon se to a request f rom the O f f i ce o f t he Depu ty

Pr im e M in i ste r p ropo sed fou r h i gh e r -l eve l ou t com es fo r ag enc ies cove red by t he Ho m e lessnessAdvice Serv ice (Nat io nal Ho m elessness Advice Serv ice, 20 05 ).

11 Th e Co m m u n i t y D e ve l o p m e n t Ch a l le n g e Re p o r t r e co m m e n d e d t h a t g o v e rn m e n t a n dc o m m u n i t y d e v e lo p m e n t o r g a n i sa t i o n s w o r k t o g e t h e r t o e st a b l i sh a c o m m u n i t y d e v el o p m e n tou t com es and ev i dence base (Comm un i t y Deve lopm en t Exchange , Com m un i t y Deve lopm en tFoun da t i on and Federa t i on fo r Com m un i t y Deve lop m en t Lea rn i ng , 200 6)

3


6/14


and fu nded by cen t r a l o r l ocal gove r nm en t o r by gove r nm en t agenc ies. The20 06 D epar t m ent o f Hea l th Th i rd Sec to r Com m ission ing Task Force u rged t ha tcon t r acts and m on i to r i ng b e based on ou tcom es no t p r ocess (Depa r tm en t o fHea l th , Ju ly 20 06 ). Learn ing o u t com es have been a t th e fo re f ron t o f recen t

cu r r i cu lum developm en t and p a r t o f a change i n em phasis f r om teach ing t olearn ing and t o a stu den t - focused app roach . Ind iv idu a l serv ices, fo r examp le

tho se i n the hea l th sph e r e, have been r equ i r ed to dem onst r a te th e i r ou tcom esw it h in t he i r assessm ent an d rev iew p rocesses. The personal isat io n of soc ia l careand i nd i v idua l bud ge ts w i l l a l so st r eng then a focus on ou t com es.

There is a lso inc reased a t t en t ion t o t he va lue o f th e th i rd sec to r in t e rm s o f i t sou t com es and im pac t ove r a ll . A 200 6 Tr easu r y r epo r t em ph asised t he need todem onst r a te t he t h i r d sec to r s im pac t m o r e pe r suasi vel y th r ou gh a st r on ge revidence base , go ing beyond i nd i v idua l examp les o f o u tcom es to i den t i f y i ng

how the sec to r im pac ts on be t te r ou tcom es m or e gene r a l ly (HM Tr easu r y ,D e ce m b e r 2 0 0 6 ).

Outcomes and Value for Money

The 20 07 Aud i t Com m ission Hea r t s and M inds r epo r t em phasised t heim po r tance fo r l oca l au th o r i t i es o f sh i f t i ng t he i r emp hasis to value fo r m oneyin fo r m a t ion f r om vo lun ta r y sec to r p r ov ide r s, r equ i r i ng conside r a t i on o f ou tp u t sand ou tcom es as w e l l as i npu t s. One theo r y o f sta te fu nd in g t ha t g ove rnm en t i s

inc reasing ly app ly ing to pu b l ic serv ices and t o t h i rd sec to r p ub l ic serv ice de l ive ryis th a t o f invest in g t o save , w i th i ts log ic o f b e t t e r re tu rn s and red uced cos ts.Fr om 200 8 /09 , the g ove rnm en t h as i nvested ap p r ox im a te l y 350 ,000 ove r th r eeyears in i t s M on i to r ing Soc ia l Va lue p ro jec t , to d evelop a stand ard fo r m easur in gsoc ia l r etu r n on i nvestm en t .

The soc ia l re tu rn on investm ent m etho do log y (SROI) i s an ou tcom es app roachor ig ina l ly deve loped fo r soc ia l en te rp r ises f rom t rad i t io na l cost -benef i t ana lysis.The m e thod o log y pu t s a mo ne ta r y value on th e soc ia l and env i r onm en ta lbene f i t s o f an o r gan isa t i on r e lat i ve to a g i ven am oun t o f i n vestm en t . The p r ocess

invo lves an ana lysis o f in pu ts , ou tp u t s, ou tcom es and im pac t s lead ing to t heca lcu la t ion o f a m onet ary value fo r t ho se im pac t s, and f in a l ly to an SROI rat io o rr a t i n g .

SROI stu d ies have now been ap p l ied to o rgan isa t ion s p ro du c ing soc ia l re tu rn s,such as he lp ing ex -o f fend ers in t o em ploym ent , w here ben ef i ts can be seen asc l ien ts cease to rece ive ben ef i t s and sta r t to pay taxes, al l o f w h ich resu l t insav ings to t he c r im ina l just ice system . I t i s recogn ised th a t th e m etho do log y m ayno t be su i tab le o r app r op r ia te fo r som e o r gan isa t i ons, bu t t he SROI m e thod o log ycou ld he lp m ake a goo d case fo r p rov id ing cer ta in t ypes o f serv ices and is

espec ia l l y use fu l i f an o rgan isa t ion s fun ders requ i re ou tcom es in fo rm at ion inf inan c ia l te rm s.

The benefit s of an outcomes approach

There a re p ragm at ic reason s fo r assessing ou tcom es. Fi r st , in fo rm at io n ab ou tou t com es and im pac t have becom e pa r t o f r egu la to r y r equ i r emen t s. Second ,inc reasing ly , fou nd at ion s and g ran t m akers a re requ i r ing ev iden ce o f the

success o f p ro jec ts and p rog ram m es and o f m easurab le benef i t s to c l ien t s. The4


7/14


Big Lo t te ry Fun d h as b een a h ig h p ro f i le leader in th is, descr ib in g i t se l f as anou t com es fu nd er , bu t o th er sign i f i can t fu nd ers, such as Lon do n Coun c i ls, a lsoem ph asise ou t com es, w i th g ran t s com m ission ed aga inst spec i f i c fami l ies o fou t com es. O the r i ndepend en t g r an t m ake r s now f r equen t l y g r an t f und p r o jec tsl i ke l y to have ou t com es tha t w i l l re late to th e i r own spec i f i ed g r an t p r og r am m eou t com es, and expec t to r ece i ve m on i to r i ng i n fo r m a t ion on t he i r ach ievem en t .

CES research f ou nd th a t som e fu nd ers exc lud ed fun d in g o rgan isa t ion s less l i ke lyto dem on st rate o ut com es (El l is , J and Gregor y, T, 20 08 ). The ind icat i on s are th atloca l governm ent and o th er agenc ies com m ission in g th i rd sec to r serv ices w i l l

inc reasing ly request o u t com es da t a . Inc reasing ly, loca l au th or i t ies a re m ov ing toou t com es-based com m ission ing .

For th i rd sec to r o rg an isa t io ns, th e im po r t an t q uest io n is w heth er there is anyin t r insic value in an ou tcom es app roach , over and abo ve a respo nse to a fun dero r g o v e rn m e n t d e m a n d f o r i n f o r m a t i o n t o f e e d i n t o h i g h e r l eve l d a t a cr u n c h in gr e la t i n g t o d e p a r t m e n t a l t a r g e t s.

US findings

I n th e US, ev idence f r om the f i e ld c lear l y show s tha t quan t i t a t i ve m easu r em en to f ou tcom es in f lu ences fu nd ing success (The Adv isor y Board Fou nd at ion ,1999 ).How ever , stu d ies have show n th a t b enef i t s a lso go b eyond t ha t o f f inanc ia l ga in .

A 199 9 stu dy by t he US Adv isor y Board Fou nd at io n em ph asised o th er ga insbeyond t hose o f ob ta in ing f und ing . These we r e tha t :

De f i n ing an t i c i pa ted ou tcom es b r i ng s c lar i t y t o m ission and goa ls.Out com es-based evalua t ion c rea tes an in tense focus on resu l ts in c l ien t s

l ives.Ou tcom es f i nd in gs cr eate a s ing le vo i ce fo r t he w ho le o r g an isa t i on .

Out com es f ind ing s becom e the basis o f g r an t p rop osa ls, b rochu res, pu b l icsta t em ent s and persuasive conversa t io n .Prog ram m e w eaknesses a re sta rk ly ev iden t w hen resu l t s a re m easured andeva lua ted , l ead ing t o im p r ovemen t .Organ isat io ns can use ou t com es-based eva lua t ion f ind ing s to im pro ve serv ice,accoun tab i l i t y and r esu l t s by r ed i r ec t i ng a p r og r amm e o r by augm en t ingserv ices of fered .

In January 2 00 0, the Un i t ed Way o f Am er ica car r ied ou t a survey o f 391 p ro jec ts,each op e r ated by a d i f f e r en t agency, i n a system a t i c e f fo r t t o de te r m ine thee xt e n t t o w h i ch p r o g r a m m e s h a d p r o f i t e d f r o m o u t c o m e m e a su r e m e n t a n d t h euse o f th e resu l t s (Un i ted Way o f Am er ica , January 20 00 ). 12

Re sp o n d e n t s a g r ee d t h a t i m p l e m e n t i n g p r o g r a m m e o u t c o m e m e asu r e m e n t w a she lp fu l , par t i cu la r ly in the a reas o f :

com m un ica t i ng p r o g r am m e r esu l t s (88% )f ocusing s ta f f e f fo r t on com m on g oa ls and p u r po ses (88% )c lar i f y i ng the p u r po se o f t h e p r og r am (86 %)

i den t i f y ing e f fec t ive p rac t ices (84 %)

successfu l ly com pet ing for resources (83%).

12 The Un i ted Way o f Amer i ca i s a na t i on a l o rgan i sa t i on o f a na t i on a l ne tw ork o fcom m un i t y -based o rgan i sat i on s.

5


8/14


I n add i t i on , t he r e w as ag r eem en t on i t s he lp fu lness i n enhanc ing r eco r d -keep ing system s (80 %) and im prov in g t he serv ice de l ive ry o f th e p rog ram m e(76%). N ine ou t o f eve r y ten r espon den t s i nd i ca ted th a t t h ey w ou ld r ecom m endt h a t o t h e r p r o g r a m m e d i re ct o r s c o n si d e r im p l e m e n t i n g p r o g r a m m e o u t c o m emeasu r emen t .

An ar t ic le by Plant z , Greenw ay and Hend r icks (199 9) descr ib es 30 lesson s learnedf r o m i m p l em e n t a t i o n o f a n o u t c o m e s f o c u s. 13 M anage r s o f p r og r amm es tha ta l r eady imp lem en ted ou tcom es m easu r es r epo r t ed tha t :

A c lea r de f i n i t i on o f ou t com es in i t se l f p r ov ides focus fo r w o r k .

Unders tand in g ach ievem ent p ro v ides a barom eter t o assess p ro gress and

d i r ec t f u t u r e ac t i v i t i es.O u t c o m e s m o n i t o r i n g p r o v id e s v al u a b l e i n fo r m a t i o n l e ad i n g t o i m p r o ve m e n tAn ou t com es focus i s a pow er fu l m o t i vato r fo r sta f f , w ho o bse r ve the p r og r esst hey a re m ak ing w i th p a r t i c i pan ts i n a consi sten t and t ang ib le m anne r.O u t c o m e s i n f o r m a t i o n c a n b e a p o w e r f u l r ec ru i t m e n t t o o l f o r v o l u n t e er s

Outcom es fund ing he lps pos it i on t he agency i n the com m un i t y as a

successfu l organisat ion.

Find ing s f r om a p r o jec t un de r t aken by t he Ur ban Inst i t u te , W ash ing ton DCand The Center fo r Wh at W orks, Ch icago , Il l ino is was pub l ished as Bu i ld ing a Com m on Ou t com e Fr amew or k to Measu r e Nonp r o f i t Pe r fo r m ance , December2 0 0 6 .

These f i nd ings r epo r t ed tha t b e t te r ou t com es da ta w ou ld r esu l t i n b e t te rben chm arks and com par ison s ac ross p ro gram m es and o rg an isa t io ns, p rov id ingbe t te r da ta on w ha t w o r ks, and i n t u r n l ead ing t o sm ar te r dec i sions abou t

a l loca t i on o f r esou r ces, st r on ge r m anagemen t , im p r oved p r og r amm e designth r ou gh i d en t i f i ca t i on o f e f fec t i ve p r act i ces, a l l con t r i bu t i ng i n t he end to

im p r ovem en ts fo r c l ien ts.

The r e have a l so b een a num ber o f a r t i c l es w r i t t en f r om pa r t i cu la r agencyexper iences. For exam ple , a 2005 ar t i c le spo ke o f t he e f fec t o f using a Benef ic ia lOut com es Appro ach w i t h in p arks and rec rea t io n serv ices. A focus on p rov id in gam en i t ies in th e past had led to u nd eru t i l i sed rec rea t ion a reas, encou rag ingun desi rab le b ehav iour o r c losure , over -u t i l i sa t ion in o th er a reas and m any userpo pu la t io ns ign ored (Marne l l , Jun e 200 5) .

A new app roach concent ra t ing on d esi red ou tcom es and exper iences o f v isi to rs,

r a the r th an on num ber s o f f ac i l i t i e s, inst i l l ed i n m anage r s the need t o i nvo l veo t hers in t he d e l ivery o f rec reat io n loca l com m un i ty leaders , loca l bu sinesses,and rep resent a t ives o f u ser g rou ps as w e l l as lead ing to focus on env i ron m ent a lp r o t e c t i o n .

UK findings

I n the UK , r esea rch i n to the b ene f i t s o f an o u tcom e app r oach have been m or e

l im i ted . A CES research s tud y car r ied ou t in 2 00 7 an d 20 08 (El l i s and Grego ry ,20 08 ) had survey respo nses f rom near ly 700 th i rd sec to r o rgan isa t io ns andcar r i ed ou t ove r 100 i n t e r views. I t f ound tha t a l t ho ugh r epo r t i ng t o fund e r s

13 Th i s and o th e r a r t i c l es on ou t com es a re ava i l ab l e f rom the Un i ted Way o f Amer i ca Ou t -comes Measurement Resources Network . See h t tp : / /nat iona l .un i tedway.org /outcomes/ resources

6


9/14


w as a m a in d r i ver i n o u tcom es-based m on i to r i ng and eva lua t i on , respon den t sr a ted th i s im po r tan t , bu t l ess so t han po ten t i a l i n te r na l bene f i t s and b ene f i t s fo rben ef ic ia r ies. Am on g survey respo nd ent s, th e top e igh t perce ived benef i ts w ere ,i n t h e f o l l o w i n g o r d e r :

be ing c lea r abou t the bene f i t s o f t h e i r w o r k

l ear n ing abo u t w ha t i s w o r k ing w e l l /e f fec t i ve p r ac t i ce

im p r ov ing t he end r esu l t f o r bene f i c i ar i es

b et t e r serv ices/st r a teg ic p lann in gi m p r o v in g t h e w a y t h e y w o r k edt e l l i ng o the r s abou t r esu l t sc o m p e t i n g f o r f u n d i n g a n d r e so u r c esi m p r o v in g r e p o r t i n g t o f u n d e r s.

Tw o st ud ies car r ied ou t by t he Socia l Pol icy Research Un i t (SPRU) at t he Universi t yo f Yo r k a re i n te r est i ng . The f i r st p r o jec t i n 2000 w as to im p lem en t an ou t com es

app roach t o carer assessm ent (Socia l Pol icy Research Uni t , Novem b er 20 00 ),

deve lop in g and t est in g an ou tcom es-focused appro ach to assessm ent andrev iew . The f in d in gs were :A con cept ua l f ram ew ork and t oo ls based on carers view s o f va luedou tco m es w ere use fu l to p rac t i t io ners and carers in exp lo r in g d esi red andac tua l im pac ts o f serv ices.M ost p r ac t i t i on e r s conside r ed t ha t an ou tcom es-focused app r oachenhan ced th e i r und erstan d in g o f care rs c i rcum stan ces and asp i ra t ions .In th e i r v iew , th i s l ed to im p r oved ca r e p lann ing , and m or e c r eat i ve car epackages.

Ou tcom es in fo r m a t ion co l l ated f r om a sm a l l samp le o f assessm en t and r ev iew

records w as cons idered by sta f f to be p o t en t ia l l y use fu l t o in fo rm serv ice

d e v e l o p m e n t .

A second SPRU repo r t in 20 05 a lso focused on a p ro jec t in t ro du c ing o u t com es,in t h is case for d isabled serv ice users (Harr is et a l , Sept em b er 20 05 ) . Beforeth e research b egan, researchers syn th esised f ind ing s f rom prev iou s w ork w i thd isab led serv ice users in t o an o u t com es f ram ew ork . Th is was incorp ora tedin t o assessm ent and rev iew p rac t ices and u sed by a m u l t i -d iscip l in ary team o fpr ofession als f ro m on e Socia l Serv ices Dep ar t m ent in Eng land . Researchers th en

evalua ted th e p ro cesses o f change and th e e f fec t s on p ro fessiona ls and serv iceusers. They repo r t ed t ha t :

The m a jo r i t y o f p r o fessiona ls found the new ou t com es-focused app r oach anddocum en ts to b e use fu l , w o r kab le and an im p r ovemen t on th e o r i g ina l needs-b ased system .Serv ice users va lued t he o u t com es app roach b ecause i t w as c lear and b ecauseth ey had g rea te r cho ice and con t ro l in t he assessm ent p ro cessUsing ou tcom es-focused docu m ent s m eant assessm ent s w ere fa r b ro aderth an w as po ssib le w i th a needs-based system .Pro fession a ls va lued t he ou tcom es-or ien t ed app roach b ecause i t p u t serv ice

use r s v i ew s f i r st and a l l ow ed t hem to be c r ea t i ve and im ag ina t i ve i n th e i rw o r k .

Add i t i on a l ly , t he ou t com es f r am ewor k w as consi sten t w i th th e gover nm en t 20 05Green Paper on Adu l t Soc ia l Care and th e Cab in e t Of f ice repor t I m p r o v i n g t h e Li fe Chances of D isabl ed People, 200 5 , ca l li ng fo r an exp l i ci t f o cus on o u tcom esas a m eans o f d r iv ing u p s tandard s.

7


10/14


These stu dy f ind ing s a re cons isten t w i th t he f ind ing s o f t he Lon do n Hou singFoun da t ion (LHF), w h ich w o r ked w i th 2 00 m em ber s o f sta f f f r om over 50hom e lessness agenc ies to supp o r t t hem in t ak ing an ou tcom es app r oach . In i t sr espon se to a consu l ta t i on o n st r a tegy fo r t he Sup po r t i ng Peop le Pr og r am m e,

LHF em ph asised th e benef i t s th a t had b een ob ta in ed fo r keyworkers by p rov id ing

grea te r c la r i ty and focus fo r th e i r wo rk (Tr iang le Consu l t ing fo r the Lon do nHous ing Fou nd at io n , Febru ary 200 6) . Add i t ion a l ly :

An ou tcom es focus p r ov ided c la r i t y f o r t he c l i en t abou t w hy they we r e the r e .Outcom es too l s sup po r ted c l i en t change by he lp ing c l i en t s to see wh e r e theyh a ve co m e f r o m a n d w h a t t h e n e xt st e p m i g h t b e .I t p r o v id e d a m e an s f o r l e ar n i n g a b o u t w h a t w a s w o r k in g a n d n o t w o r k in g i nc l ien t p ro v ision .I t p r ov ided c la r it y f o r sta f f abou t w ha t the ser v i ce w as the r e to ach ieve and agrea te r fo cus in keywo rk in th ose a reas.

I t w as a m eans o f im pro v ing serv ice de l ivery .

Over a l l, LHF found tha t an ou t com es focus had h e lped to p r od uce a cu l tu r e o fl ea rn i n g a n d i m p r o ve m e n t .

Severa l m a jo r UK char i t ies have em braced a focus on ou t com es and im pac t ,w h i l e the r e m ay st i l l b e som e d i stance to b e t r ave l l ed i n o b t a in ing a d i st i nc t i onbe t w een ou tp u t s and ou t com es and pu t t i ng app r op r ia te system s in p lace fo rassessing th em . For m any th e app roach l in ks we l l w i th a user fo cus. Save t heCh i ld ren , fo r exam ple , say th a t be ing aware th a t th ey w i l l have to answ er the

im pac t qu est i on has the e f fec t o f he lp ing sta f f r eal i se tha t t hey need t o b ec lea r f r om the b eg inn ing abou t the k ind s o f changes they hop e to see , and b e

sure th a t th e ac t iv i t ies th ey p lan w i l l lead t o th ese chang es (Sco t t and M o l t eno ,Oc tob e r 200 5 ) They add :

I n ana lysing a l l t h i s i n fo r m a t ion , p r og r am m e sta f f can l ear n w ha t h as w o r ked and w ha t hasn t , and w he r e the i r ac t i v i t i es have had u nexpected (and som et im es even n egat ive ) imp ac ts. Th is is a v i ta l step in b e ing ab le to constan t l y adap t w ha t t hey do t o g e t b e t te r r esu l t s . By i nvo l ving ch i l d r en and o t he r s w e w o r k w i th , i t a l so p r ov ides a m eans fo r b e ing

a cc o u n t a b l e t o t h e m .

The r e i s an i nc r easing em ph asis on using ou t com es in fo r m a t ion no t on l yin t e rna l ly, fo r serv ice im pro vem ent and d evelop ing st ra t egy , bu t fo r ex te rna lpo l i c y i n f l uence . The i n f l uence o f ou t com es evidence i s tem per ed b y the com p lexenv i ronm en t i n w h ich po l i c y i s f o r m u la ted , bu t t h e r e have been no tewo r thyexam ples. Ev idence f ro m th e Du nd ee Fam i l ies Pro jec t , fo r exam ple , w h ichsupp o r t ed fam i l i es fac ing evi c t i on , pe r suaded a n um ber o f l oca l au th o r i t i es toadop t i t s app r oach . Find ing s w e r e sub sequen t l y taken u p by th e gove r nm en t sRespec t Agenda, ac t ion p lan c ross-go vernm ent st ra t egy t o t ack le an t i -soc ia l

beh av iour (D i l lane e t a l , 20 01; N ixon e t a l , 200 6; Pawson, 200 7) .

Some f inal points

M any advoca t in g t ha t t h i rd sec to r o rgan isa t ion s sho u ld assess th e i r ou t com esem ph asise tha t t h is shou ld no t in vo lve com plex , sc ien t i f i c da ta co l lec t io n , bu tr a the r p r ac t i cal da ta co l l ect i on too l s and t hou gh t fu l qu est i on s.

8


11/14


CES ow n exper ience and th e research ev idence sug gest a num ber o fc la r i f i ca t ion s, o r w arn ing s. These inc lu de :

Co l l ec t i ng o u tcom es da ta a lone h as l im i ted va lue . I t i s on l y w hen i t i s

i n te r p r e ted and t r ansla ted i n to pos it i ve changes i n p r ac t i ce tha t i t w i l l y ie ldim pro vemen ts in th e qua l i t y o f serv ices.

Ef fec t ive ana lysis and in te rp re ta t ion o f ou tcom e da ta m eans th ere is a needfo r adeq uate exper t ise and system s, and t he e f fec t ive use o f f in d in gs toin fo rm changes in loca l serv ice de l ivery .Ou tcom e measu r em en t d oes no t e l im ina te th e need to m on i t o r r esou r ces,act i v i t i es and ou tp u t s to m ake sense o f t h e i n fo r m a t ion .Ou tcom e da ta shou ld b e used to i den t i f y wh e r e resu l t s ar e go ing w e l l andw her e no t so w e l l . W hen no t go in g w e l l , t he o r gan isa t i on needs to a t t em p t

to f ind ou t w hy . Th is p ro cess is w hat lead s to con t inu ou s o rgan isa t ion a ll e ar n i n g a n d i m p r o v em e n t .M easu r ing and im p r ov ing p r o ject - l evel ou t com es does no t b y i t se l f im p r ove

com m un i ty leve l ou tcom es. Except in ra re instan ces, an ind iv idua l p ro jec tdoes no t se r ve enoug h i nd i v idua l s to a f fec t com m un i t y -w ide sta t i st i c s,regard less of ho w successfu l i t is.

I den t i f y i ng app r op r ia te and r e levan t o u tcom es, co l l ect i ng r e levan t da ta ,ana lysing and r epo r t i n g i t , can p r ov ide an essen t i a l and w e lcom e focus on endbene f i t s and user s and p r om p t a r eapp r a i sa l o r r ede f i n i t i on o f o r gan isa t i ona l

o r p r o jec t a im s. Ou tcom es m on i t o r i ng and r epo r t i ng se r ve as im po r tan taccoun tab i l i t y t oo l s and p r ov ide use fu l f eedback to f r on t l i ne p r ov ide r s andm anagers. A deeper leve l o f learn in g , essent ia l fo r rep l ica t in g g ood prac t ice ,pub l i c po l i c y i n f l uence and st r a tegy deve lopm en t w i l l r equ i r e the ca r e fu l

ana lysi s and i n te r p r e ta t i on o f t ha t da ta , se t t i n g i t aga inst an und e r stand ingo f t he use r g r oup and i t s spec i f i c needs, the op e r at i ng env i ronm en t , and t hespec i f i c serv ice de l ive ry appro aches and o rgan isa t io na l con tex t .

Furt her reading

The r e a r e a num ber o f o th e r use fu l p ub l i cat i on s and r epo r t s avai l ab le o rr e fer enced on the Un i ted W ay o f Amer i ca web si te s Ou tcom e M easu r em en tResou r ce Ne tw o r k , dem onst r a t i ng how o r gan isa t i ons have used o u tcom e

m easu r em en t . These i nc lude t he fo l l ow ing :Spi egal , RA (199 9) Accoun t ab le Good : Pr og r am Evalua t i on i n t he Nonp r o f i t Sector . Th is descr ibes th e cha l lenges and b enef i t s o f o u t com es m easurem ent .Edu cat i on a l Ach ievem en t Ou tcom es , excerp t f rom th e Edn a Cla rk Fou nd at io nYo u t h D e ve l o p m e n t O u t c o m e s Co m p e n d i u m .

Out com es M easurem ent : Are You M ak ing a D i f fe rence? (v ideo) (20 00 ) show show to use ou tcom es da t a to inc rease serv ice ef fec t iveness. Sta f f o f fo uragenc ies desc r ibe cha l l enges faced i n im p lem en t ing ou t com e m easu r em en ta n d h o w i t h a s p a i d o f f .Wei l , S and Rud d , P (200 0) Perspect ives on O ut com e-Based Evaluat ion for

Lib ra r ies and M useum s, Mu seum s and Lib rar ies Serv ices, Washin gt on , DC,p r esen t c lea r j ust i f i ca t i ons fo r t he ado p t i on o f ou t com es-based eva lua t i on .

Add i t i ona l l y :Buckm aster , N (Apr i l 1999 ) Assoc ia t io ns be t w een ou t com e m easurem ent ,

9


12/14


accoun tab i l i t y and l ear n ing fo r n on - p r o f i t o r gan isa t i ons i n In te r na t i ona l Jou r na lof Pub l ic Secto r M anag em ent , 12 :2 186 197, M CB UP Lt d . This p aper analysesthe bene f i t s o f ou t com es m easu r em en t w i th a view to encou r ag ing n on - p r o f i to r gan isa t i ons to em br ace the opp o r t un i t y to p r om o te o r gan isa t i ona l l ea r n ing .

References

Aud i t Com m ission ( 2007 ) Hear t s and M inds: Com m ission ing f r om the Vo lun t a r y Sector, Aud i t Com m ission , Lon do n.

Cab in e t Of f ice , Of f i ce o f the Th i rd Sec to r (200 6) Par t nership in Pub l ic Serv ices: An act i on p lan fo r t h i r d sec to r i nvo l vem en t , HM Gover nm en t , Lond on .

Com m un i t y Developm en t Exchange , Com m un i t y Developm en t Foun da t i onand Fede r a t i on fo r Comm un i t y Deve lopm en t Lea rn ing ( 2006 ) Th e Co m m u n i t y Developm en t Cha ll enge , Depa r tm en t o f Com m un i t i es and Loca l Gove rnm en t ,

London . www.commun i t i es .gov .uk /documen ts /commun i t i es /pd f /153241 .pd f

Depa r tm en t fo r Com m un i t i es and Loca l Gove rnm en t ( Octob e r 2006 ) St r o n g a n d Pr ospe r ous Com m un i t i es . The Local Governm ent Whi t e Paper , Lon do n. h t tp : // www.commun i t i es .gov .uk

Depa r tm en t o f Heal th ( Janua r y 2006 ) Our hea l th Our care Our say : a New

D i r ec t i on fo r Com m un i t y Se r vi ces . www.dh .gov .uk

Depa r tm en t o f Heal th ( Ju l y 2006 ) No Excuses. Em b race Par t nership no w . Stepto w ards change! Repor t o f th e Th i rd Sec to r Com m ission ing Task Force . h t tp : // www.dh .gov .uk

Di l lan e, J, Hi l l , M, Bann ister , J and Scot t , S (20 01 ) An Evalua t i on o f t he Dun dee Fam il ie s Pro ject, Research Fin d in g s No 121, Scot t ish Execut ive Cent ral ResearchUn i te , Developm en t Depa r tm en t Resea rch Pr og r am m e, Ed inb u r gh .

El l is , J w i th Gregor y, T (20 08 ) Accou nt ab i l i t y and Learn ing : Deve lop in g M on i t o r i ng and Evalua t i on i n th e Th i r d Sec to r , Char it ies Evalu at io n Servic es,London .

Harr is , J, Foster M , Jackson K and M org an H (Sept em b er 2 00 5) Ou tcom es fo r Di sab led Service Users, Soc ia l Pol i cy Research Uni t , Universi t y of York. h t t p : / / www.york .ac .uk

HM Treasur y (Decem ber 20 06 ) The Fut ure Role o f t he Third Secto r in Socia l an dEcono m ic Regenera t ion : In te r im Repo r t , Cab ine t O f f i ce. h t tp :/ /w w w .hm -t r easu r y .

gov .uk /m ed ia53E/94 /pb r 06_3 r d_sec to r _428 .pd f

HM Treasur y, Spen d in g Rev iew s (a r ch i ved and upd a ted to 200 7 ) . h t tp :/ /w w w.hm -t reasury .gov .uk /spend ing_rev iew/spend_p lancont ro l .c fm

HM Treasur y (un dat ed) Outcom e Focused Managem en t i n the Un i ted Kingd om.h t tp : / /www.hm- t reasury .gov .uk /d /GEP_outcome_focused_management .pd f

M arne l l , J (Jun e 200 5) Benef ic ial Out com es Appro ach Im pro ves M anagem ent in10


13/14


Parks and Recreat io n40(6) :79 Nat ion al Recreat i on an d Parks Associat ion , Uni t edStates.

Nat ion al Hom elessness Advice Serv ice (20 05 ) Hom elessness and Hou sing AdviceO u t c o m e M o n i t o r i n g Pi l o t , Nat io nal Hom elessness Advice Serv ice.

Nixo n, J, Hun t er, C and Parr , S (20 0 6) Ant i -soc ia l Behaviou r In t ens ive Fam i ly Sup p or t Pro ject s: An Evaluat ion of Six Pion eer ing Pro ject s , D e p a r t m e n t f o rCom m un i t i es and Loca l Gove rnm en t , Lond on . w w w .com m un i t i es.gov .uk

Paw son , H, David son , E and Net to , G (M arch 20 07 ) Evaluat ion o f Hom elessnessPrevent ion Act iv i t ies in Sco t land , Scot t ish Execut i ve. w w w .scot land .gov.uk/ Pub l ica t ions /

Plant z , M C, Greenw ay, M T and Hend r icks, M (199 9) O u t c o m e M e asu r e m e n t : Show ing Resu l t s in t he No n-pro f i t Sec to r , Un i t ed Way o f Am er ica On l ine Resou rce

Lib ra r y , A lexandr ia , Vi rg in ia . h t t p : / /na t ion a l .un i ted w ay.org /ou t com es/ resou rces

Pr icewaterhouseCoopers (January 2003) A Mo de l f o r D e l i ve r i ng Be t te r Hea l th O u t c o m e s . h t tp : / /www.rmianz .co .nz /pd fs /BHO%20Repor t /Sec t ion3 .pd f

Socia l Pol i cy Research Uni t (Novem b er 20 00 ) I m p l e m e n t i n g an O u t c o m e s App roach to Carer Assessm ent and Review, Un iversi ty o f York . h t tp : //w w w .york .ac.uk

Sco t t , C and M o l teno , M ( Oc tobe r 20 05 ) In the Righ t D i rec t ion : Exam ples o f t he Im pac t o f Save th e Ch i ld ren s Work , Save th e Chi l dr en, Lon do n.

The Adv isor y Board Fou nd at ion (1999 ) Fi f t een Conc lu sions on Eva lua t ing Nonp r o f i t Pe r fo r m ance , Uni ted St ates.

The Scot t ish Execut ive (Jun e 20 06 ) Tran sfo rm in g Pub lic Servic es: The Next Phaseof Refor m. h t tp : / /www.sco t land .gov .uk /Pub l ica t ions /2006/06 /15110925/9

Tr iang le Consu l t in g fo r th e Lon do n Hous ing Fou nd at io n (Feb 20 06 ) London Hou sing Fou nd at io n Respo nse focussing on t he use o f Out com es w i th in Sup po r t ing Peop le Respo nse to con su l t a t io n on a St ra t egy fo r t he Sup po r t ing

Peop le Pr og r am m e, Lon do n Hous ing Fou nd at ion . w w w .ho m elessnessou tcom es.o r g .uk

Un i ted W ay o f Amer i ca (200 0 ) Agency Expe r iences w i th Ou tcom e M easu r em en t : Sur vey Find ing s, Un i t ed Way, Alexand r ia , Vi rg in ia . h t t p : / /na t ion a l .un i ted w ay.orgf i l es /pd f /ou tcomes /agencyOM.pd f

11


14/14


What services does CES of fer?

In-house trainingCES o f fe rs t r a in ing in m on i to r ing , evalua t ion an d q ua l i t y system s.

Tra in in g co urses can b e run in -ho use and t a i lo red spec i f i ca l l y to t he

needs o f you r o rg an isa t i on .

Open trainingW e a lso ru n a p ro g ra m m e o f t r a i n i n g c o u rse s c o ve r in g m o n i t o r i n gand eva lua t i on and qu a l i t y system s f rom ou r accessib le cen t ra lLon do n venue .

Con t ac t u s by em ai l : t ra in in g@ces-vo l .o rg .uk o r ca l l fo r ou r t ra in in gg u i d e .

ConsultancyOur co nsu l t ancy serv ice is f lex ib le and p ro v ides sup po r t fo ro rg a n i sa t i o n s t h a t w a n t t o u n d e rst a n d a n d i m p l e m e n t m o n i t o r i n g ,

evalua t ion an d q ua l i t y system s.

External evaluations

CES has car r ied ou t eva lua t io ns o f a la rge nu m b er o f o rgan isa t io nssince 1990 as w e l l as w o rk ing w i th f un de rs to eva lua te the i rp ro g ra m m e s a n d f u n d i n g st r a t e g i e s.

Con t ac t u s on enq u i r ies@ces-vo l .o rg .uk o r ca l l fo r m ore d e ta i l s.

Publications

CES p ro du ces a range o f pu b l i cat i o ns on qu a l i t y and eva lua t i on .Please look a t ou r w ebs i te www.ces-vol.org.ukf o r f u r th e r de ta i l s.

Chari t ies Evaluat ion Services4 Coldbath SquareLondon EC1R 5HLRegistered charit y number 803602

Te l : 02 0 7 713 5 72 2Fax: 02 0 7 713 56 92enquir [email protected]

www.ces-vo l .o rg .uk

12

case for outcome focus

Documents