NOVEMBER 2011

FOR FURTHER INFORMATION, PLEASE CONTACT

LAURA LANDRY

303-839-9422, EXT. 170 llandry@omni.org

ERIN INGOLDSBY

303-839-9422, EXT. 116 eingoldsby@omni.org

HB 1451 Collaborative Management Program Year 2 Statewide Evaluation Findings

THISPAGEINTENTIONALLYLEFTBLANK.

Our Mission is to Design and Deliver Quality Human Services that Improve the Safety and Independence of the People of Colorado

November8,2011TO: LloydMalone JulieKrow ReggieBichaFROM:CollaborativeManagementProgram(CMP)EvaluationSubcommittee

OnbehalfoftheCollaborativeManagementProgram’sEvaluationSubcommittee,wearepleasedtosubmitthe2011CMPEvaluationReport.TheEvaluationSubcommitteeiscomposedofrepresentativesfromCMPcounties,stateagencypartners,andafamily‐drivenorganization.Ourworkoverthelast16monthshasbeenreviewedandapprovedbytheCMPStateSteeringCommittee.Whilethereportcontainsagreatdealofinformationontheperformanceoftheinitiative,wewantedtotakethisopportunitytothanktheStateofColoradoandtheDepartmentofHumanServices,inparticular,forongoingsupportoftheProgram.

Asyouarewellaware,thecurrentstructureofsiloedstatefundingresultsinsignificantservicefragmentationacrossagencies.Thechallengesofprovidingintegratedcarearefurthercomplicatedbydatasharingissues,conveningstafffrommultipleagencies,andbarrierstojointserviceplanninganddelivery.TheCollaborativeManagementProgramhasprovidedColoradocommunitiesacriticalframeworkbywhichtoexplore,investinandimproveservicedeliveryprocessesandinfrastructurewithinlocalsystems.Asevidencedinthefollowingreport,thishasledtomoreefficient,effectiveandmoreelegantservicedeliverywhichishavingprofoundeffectsonthelivesofColoradofamilies.

Astheinitiativematures,weareexcitedtoimplementframeworksthatwillallowustomanageprocessesmoreeffectively,disseminatelessonslearned,furtherimplementevidence‐basedservicemodels,andevaluateoureffortswithgreaterprecision.Werecognizethatongoingbudgetissuesmeanthatdifficultchoiceswillneedtobemade.WehopethatthestatecontinuestoseethepromiseoftheCMPinitiativeinbreakingdownsilos,givingvoicetoinvolvedfamilies,andimprovingoutcomesforchildrenandfamiliesinvolvedinmulti‐systemcare.

WelookforwardtobeingastrongpartnerinadvancingyourvisionforCDHSandColoradofamilies.

Sincerely,

NormanKirschAdministrator,CollaborativeManagementProgramColoradoDepartmentofHumanServices

CHILDREN, YOUTH AND FAMILIES DIVISION OF CHILD WELFARE Lloyd D. Malone, Director 1575 Sherman Street Denver, Colorado 80203-1714 Phone 303-866-4365 FAX 303-866-5563 www.cdhs.state.co.us

John W. HickenlooperGovernor

Reggie BichaExecutive Director

STATE OF COLORADO

EvaluationSubcommitteeMembersJimAdams‐BergerOMNIInstituteJustonCooperJeffersonCountyCMPSusanCollingColoradoStateCourtAdministratorsOfficeMargieGrimsleyFederationofFamiliesforChildren’sMentalHealthHeatherHarcourtElbertandLincolnCountyCMPErinIngoldsbyOMNIInstituteNormKirschColoradoDepartmentofHumanServicesLauraLandryOMNIInstituteRachelLambertJeffersonCountyCMPCarlLarsonUniversityofDenverRebeccaLarsonOMNIInstituteKamaLinscomeColoradoDepartmentofEducationKellySchrammWeldCountyCMPMegWilliamsDivisionofCriminalJustice

TableofContentsExecutive Summary.................................................................................................................................................i The Collaborative Management Program (CMP) initiative.........................................................................1 

A.  The CMP state initiative...........................................................................................................................................2 B.  CMP statewide program evaluation......................................................................................................................4 

Legislative Goal I: Develop a more uniform system of collaborative management that includes the input, expertise, and active participation of parent advocacy or family advocacy organizations.......9 

A.  Mandated components of CMPs............................................................................................................................9 B.  Local adaptation........................................................................................................................................................10 C.  Effectiveness of the interagency collaboration established by CMPs......................................................15 D.  Inclusion of the input, expertise, and active participation of parent or family advocacy organizations........................................................................................................................................................................21 

Legislative Goal II: Reduce duplication and eliminate fragmentation of services provided to children or families who would benefit from integrated multi-agency services....................................28 

A.  Coordination of client consents and assessment.............................................................................................28 B.  Integration of service delivery planning............................................................................................................29 C.  Coordination of payment for and provision of services to CMP youth...................................................29 D.  Information sharing practices................................................................................................................................30 E.  Effectiveness of CMPs in reducing duplication and eliminating fragmentation..................................33 F.  Challenges associated with duplication and fragmentation.........................................................................33 

Legislative Goal III: Increase the quality, appropriateness, and effectiveness of services delivered to children or families who would benefit from integrated multi-agency services...............................35 

A.  Service quality...........................................................................................................................................................35 B.  Appropriateness of services...................................................................................................................................37 C.  Effectiveness of services.........................................................................................................................................38 

Legislative Goal IV: Encourage cost sharing among service providers that leads to cost-reduction for the services provided to children and families in the child welfare system, including the foster care system, in the state of Colorado...............................................................................................................40 

A.  Cost-sharing approaches.........................................................................................................................................40 B.  Cost reductions, potential cost savings, and reinvestment of funds..........................................................42 C.  Challenges related to measuring costs and benefits at the initiative level..............................................43 

Legislative Goal V: Lead to better outcomes and cost-reduction for the services provided to children and families in the child welfare system, including the foster care system, in the state of Colorado.................................................................................................................................................................46 

A.  Defining CMP target populations........................................................................................................................46 B.  Defining CMP outcomes........................................................................................................................................47 C.  Plan for measurement of standard outcomes in evaluation year 3............................................................53 

Year three evaluation directions.......................................................................................................................56 Endnotes.................................................................................................................................................................59 Appendices.............................................................................................................................................................61 

Appendix A. Structure of state management of the CMP......................................................................................62 Appendix B: Additional reports and products...........................................................................................................63 Appendix C. Sources of additional funding reported by CMPs.........................................................................67 Appendix D. Frequently used data sources for measurement of CMP performance goals........................68

i

ExecutiveSummaryINTRODUCTION

ThefollowingdocumentisbasedupontheCollaborativeManagementProgram’s(CMP)70‐pageYear2EvaluationFindingsReport.FundedbytheColoradostatelegislatureandadministeredbytheColoradoDepartmentofHumanServices,theevaluationhascompletedthesecondyearofamulti‐yearstatewideevaluation.ThefocusoftheevaluationisonexaminingtheeffectivenessofCMPeffortsinachievingthelegislativegoalssetoutbyHouseBill04‐1451(HB1451).Theevaluationdesignisbeingimplementedinmultiplephases,includingformativeevaluationtodescribeandinformcurrentpractices,infrastructuredevelopmenttolaythefoundationforstandarddatacollection,andsummativeeffortstoassessindividualandcross‐CMPeffects.Evaluationfindingstodatehaveilluminatedawidevariationindefinedtargetpopulations,servicemodels,andmeasuredoutcomes,aslocalCMPshavetailoredtheireffortstomatchtheircommunity’sneedsandstrengths.Asdescribedbelow,individualCMPsdemonstratedprogressonkeyperformancemeasuresofeachlegislativegoal.However,thediversityofmeasurementposedsignificantchallengestoassessingstatewideimpactoftheCMP.Toaddressthesechallenges,theinitiativeundertookaprocessinthissecondevaluationyeartoidentifyindicatorsandtodesignabasicdatamanagementsystemtosupportlocalefforts.Threefutureevaluationdirectionsarerecommended:

1. Continuetodevelopandimplementsystemstocollectandanalyzewell‐defineduniformmeasuresofkeyoutcomesandprocesses,atlocalandstatelevels.

2. MonitorandtracktrendsinstatewidemeasuresofkeyCMPoutcomes.3. ContinuetodefineandsupportdisseminationofinformationoneffectiveCMPinteragency

structures,serviceprocesses,andcollaborativepractices.

CMPBACKGROUNDANDPARTICIPANTS

CollaborativeManagementProgram(CMP)InitiativeIn2004,theColoradoGeneralAssemblypassedHouseBill04‐1451basedontheassertionthat“developmentofauniformsystemofcollaborativemanagementisnecessaryforagenciesatthestateandcountylevelstoeffectivelyandefficientlycollaborate,shareresourcesormanageandintegratethetreatmentandservicesprovidedtochildrenandfamilieswhobenefitfrommulti‐agencyservices.”iThisledtothecreationoftheCollaborativeManagementProgram(CMP)whichsetsforththefollowinggoals:

1. Developamoreuniformsystemofcollaborativemanagementthatincludestheinput,expertise,andactiveparticipationofparentorfamilyadvocacyorganizations.

2. Reduceduplicationandeliminatefragmentationofservicesprovidedtochildrenorfamilieswhowouldbenefitfromintegratedmulti‐agencyservices.

3. Increasethequality,appropriateness,andeffectivenessofservicesdeliveredtochildrenorfamilieswhowouldbenefitfromintegratedmulti‐agencyservices.

4. Encouragecostsharingamongserviceproviders.5. Leadtobetteroutcomesandcost‐reductionfortheservicestochildrenandfamiliesinthechild

welfareandfostercaresysteminthestateofColorado.

ii

TheCMPinitiativewasfirstimplementedinsixcountiesin2005‐2006,withadditionalcountiesaddedannually.Intheupcomingyear,thereare30localCMPsparticipatingintheinitiative,twoofwhicharemulti‐countypartnerships(seemaponfollowingpageforparticipatingcounties.)

NumberofcountiesparticipatingintheCMPinitiative

Year 2004‐2005

2005‐2006

2006‐2007

2007‐2008

2008‐2009

2009‐2010

2010‐2011

2011‐2012

NumberofParticipatingCounties

6 12 15 24 29 31 30 32

In2006,theStateSteeringCommittee(SSC),anon‐legislatedoversightgroupcomposedofparticipatingcounties,stateagencies,andfamilymembers,determinedthatperformancebasedmeasuresshouldbedevelopedbyCountyInteragencyOversightGroups(IOG)infourareasthatmatchlegislativeintent: Childwelfare Juvenilejusticesystem Education Health/mentalhealth/otherhealthservices(e.g.,substanceabuse)

Inaddition,performance‐basedincentivedollarsaredistributedtocommunitiesbasedonademonstratedachievementofoutcomesinthesefourareasandinaccordancewithaformulaapprovedbytheStateBoardofHumanServices.

MapofparticipatingCMPcounties

iii

CMP statewide program evaluation

HouseBill08‐1005authorizedanannualexternalevaluationoftheCMPinitiative.Thelegislationrequiresthatlocalprojectsreportonthefollowing:

Thenumberofchildrenandfamiliesservedthroughtheirindividualizedserviceandsupportteamsandtheoutcomesoftheservicesprovided;

Estimatedcostsandcost‐shiftingorcost‐savingrelatedtoCMPefforts;and Informationrelevanttoimprovingthedeliveryofservicestopersonswhowouldbenefitfrom

multi‐agencyservices.

InJulyof2009,CDHShiredOMNIInstitute(OMNI)toconductthestatewideevaluationoftheCMP.Thefirstyearoftheevaluationfocusedondocumentinganddescribingtherangeoflocaleffortsandoutcomesoffundedprojects.Thesecondyearfocusedonthefollowingefforts:

1. RefinementofCMPprocessandoutcomemeasurement:AworkgroupwasformedtoselectcommonvariablesanddevelopstandardizedmeasurementprocessesfortheChildWelfareandJuvenileJusticedomains;

2. Refinementofsystemstoevaluateandshareprogressonlegislativegoals,including:

RevisionandadministrationofCMPannualreportstoimprovecontentandusability. RevisionandadministrationofCMPcollaborativeeffectivenesssurvey. Administrationandanalysisoffamilyadvocacysurveys. Refinementsandtechnicalsupportoftheweb‐basedCMPportal.

LEGISLATIVEGOALSANDEVALUATIONFINDINGSGoalI:Developamoreuniformsystemofcollaborativemanagementthatincludestheinput,expertise,andactiveparticipationofparentorfamilyadvocacyorganizations.InteragencyOversightGroupsandInteragencyServiceandSupportTeamsHouseBill04‐1451mandatedparticipationofspecificagencies,encouragedfamilyinvolvementandrequiredtheestablishmentoftwoprimarystructures:anInteragencyOversightGroup(IOG)tofacilitatecommunicationandresourcesharingandanIndividualizedServiceandSupportTeam(ISST)toplanandcoordinatethedeliveryofservicesformulti‐agencyinvolvedfamilies.Showninthetable,membershipdatareflectsubstantialparticipationbymandatedIOGpartners:

iv

CMPimpactonstaffingteams

“We’vealwayshadgoodcollaboration,butithasgonetoadifferentlevelsincebecomingapartoftheCMP.Wearemoreeffectiveinourstaffing.Weusedtooverwhelmfamilieswithtoomanypeopleintheroom.ThankstoISSTandwrap,wehaverefinedourprocess.Wehavetherightpeopleintheroomattherighttime.We’vebroughtinmorefundingopportunitiesandprogrammingasaresultofourcollaboration.Andlastly,theroleofthefamilyintheirowncaseplanningisclearerandmoreeffective.”

AgencyorIndividualsRepresentedthroughMembershiponIOG

(n=28)

CountyDepartmentofHumanandSocialServices 28 (100%) 27 (96%)

Schoolrepresentative 28 (100%) 27 (96%)

Probation 28 (100%) 27 (96%)

Mentalhealthserviceprovidercenter 27 (96%) 25 (93%)

Substanceabuseserviceprovider 27 (96%) 24 (89%)DivisionofYouthCorrections 27 (96%) 23 (85%)

Domesticviolenceserviceprovider 27 (96%) 23 (85%)

CountyHealthDepartment 26 (93%) 22 (85%)

SenateBill94representative 25 (89%) 22 (88%)

Behavioralhealthorganizationrepresentative 24 (86%) 20 (83%)Localcourts/judicial 20 (71%) 13 (65%)

Diversion 13 (46%) 12 (92%)

Familyadvocacyorganization 12 (43%) 10 (83%)

Lawenforcement 12 (43%) 7 (58%)

Familyrepresentative 9 (32%) 0 (0%)Localhealthservicesprovider 6 (21%) 6 (100%)

Electedofficial 6 (21%) 0 (0%)

Youthrepresentative 4 (14%) 4 (100%)

BusinessorChamberofCommerce 3 (11%) 2 (67%)

Other 17 (61%) 16 (94%)

NumberofCMPsReportingIOGMember(%)

NumberofCMPsReportingIOG

MemberAttendingatLeast

50%ofIOGMeetings(%)

Note:Legislativelymandatedpartnersareshowninboldtextabove.

IOGsidentifiedacommonsetofcoreactivitiesrelatedtolegislativegoals.Thefivemostcommonfocusareasinclude:

Improvingqualityofservicedeliveryforchildrenandfamilies Establishingand/orrefiningIOGstructure,processes,andprotocols Strengtheningcollaborativerelationshipswithnewandexistingpartners EnhancingCMPpersonnelresources AddressingCMPsustainability

ISSTsplanandcoordinateservicesforyouthinfamilieswiththeparticipationofmultipleagenciestodevelopanintegratedserviceplan.ThefollowinghighlightsfindingsrelatedtoISSTactivities: Onaverage,sitesreportedhavingtwoISSTs,

withsitesrangingfrom1to16.ISSTtypesincludethoseservingallfamilies,aspecializedISSTwithinonebroadservicearea(e.g.,JuvenileJustice),andthosefocusedonaspecificoutcomearea(e.g.truancyISST).

YouthandfamiliesaretypicallyreferredtoanISSTthroughaCMPpartneragency.Thetwomostcommonreferralsourcesarecounty

v

departmentsofhumanservicesandschooldistricts,respectively.Othersourcesincludedomesticviolenceproviders,primarycarephysicians(pediatricians),SenateBill94staff,andearlychildhoodeducationproviders.

Twenty‐sixCMPsindicatedthatatleastfouragenciestypicallyattendedstaffing,withthirteenindicatingparticipationofsevenormoreagencies.

Onaverage,CMPsholdfiveISSTmeetingsperfamily,thoughthisvarieswidely,fromaminimumofonetoamaximumofthirty.

TheQualityofCollaborativeProcessesOverall,IOGmembersperceivetheircollaborativestohavestrongandproductivestructuresandprocessesinplace.Onaverage,IOGmembersindicatedthattheirCMP’seffortswerecharacterizedbyhighqualityexperiencesintermsofstrongleadership,authenticprocesses,andpossessingstructuralintegrity,withsurveyscalescoresexceedingthe4.25markerofcollaborativeeffectiveness(outofa6pointscale,with6indicatinggreatereffectiveness).CMPssharedevidenceoftheeffectivenessoftheircollaborativeeffortsbydescribingkeyachievementsattheIOGandISSTlevelsinFY2010‐11,summarizedinthetablebelow:AreasofCollaborative

Effectiveness ExampleAchievementsEstablishmentofsubcommittees,leadingtomoreefficientuseofIOGmeetings•asteeringcommitteetooversee"day‐to‐day"operationalactivities•anoutcomescommitteetospecifylocalevaluationmeasuresandgoalsFullpartnerparticipationinstrategicplanningsessionsCreationofyouthadvisoryboardEstablishmentofa"virtualvoting"processwithinaruralIOGtoincreaseactiveparticipationindecision‐makingRoll‐outofnewprogramstargetingareassuchaspregnancy,mentoring,youthservicelearningIncreasesinthenumberoffamiliesreceivingspecificservicesIncreasesinthenumberofreferrals,indicatinggreatercommunityawarenessofeffectivenessofservicedeliveryDecreasedtimebetweenreferralandenrollmentininterventionservicesFundedsuccessfulgrantwritingeffortsBlendedorbraidedfundingidentifiedforspecificservicesleadingtomoreintegrationandstreamlinedservicesamongpartneragenciesEstablishmentofprocesstograntseedfundstonewprograms

ImprovedprocesseswithintheIOG

Developmentofnewprogramsor

enhancementofexistingservicedelivery

Supportingsustainability

CMPscontinuetostrugglewithsomecommonchallengesintheircollaborativeefforts.Theseinclude:

Participationandbuy‐infrompartners.CMPsexperiencedifficultyingettingparticipationfromspecificpartners,andhavingpartnersfailtofollowthroughonplans.

CMPstaffing.CMPsreportedthatinadequatestaffingandturnover(lackofacoordinator,ISSTfacilitators)hasimpactedCMPadministrationandservicedelivery.

Financialstability/sustainability.CMPsreportedchallengesassociatedwithageneralshortageoffunding,largelyattributedtotheeconomicdownturn.TheCMPhasincreasedfrom6countiesin2005‐06to31countiesin2011‐12,whiletheavailableearnedincentivefundshavenotincreased.

Informationsharing.SomeCMPshavestruggledwithestablishingpartneragreementregardingconfidentialityrequirementsandconsentprocesses/documentstoshareclientinformation.

vi

However,theColoradoChildandYouthInformationSharingcommitteewasestablishedaspartofaCMPstrategicplanandisengagedinpromotingcrosssysteminformationsharinganddatasharing(seehttp://collaboration.omni.org/sites/1451/SitePages/Home.aspxformoreinformation).

SharedCMPmission.AfewCMPs(3)citeddifficultyensuringclearandcommonunderstandingofcollaborativegoals,activities,andpracticesamongpartners,whichhasbeencompoundedbyturnoverintheIOGandpartneragencies.

FamilyParticipationThelegislationstronglyencouragesCMPstoengagefamilyrepresentativestoensurethatsystemsareresponsivetothefamiliestheyserve.FamiliescanbeinvolvedatboththeIOGandISSTlevels.

IOGlevel.Over40%ofCMPsreportedatleastonefamilymemberontheirIOG,about40%reportedatleastonefamilyadvocate,andabout20%reportedatleastoneyouthmember.

ISSTlevel.NearlyhalfofCMPs(12CMPs,46%)reportedthatfamilymembersarealwaysviewedaspartnersinserviceplanningforotherfamilies,andthevastmajority(85%)ofCMPsreportedthatfamilymembersareseeninthiswayatleast“sometimes.”

Participationinserviceplanning.Themajority(63%,17CMPs)ofCMPsindicatedinvolvingfamiliesimmediatelyafterbeingdeterminedeligibleforISSTservicesandnearlyone‐third(30%,8CMPs)reportedthatfamiliesarefirstcontactedpriortoadecisionregardingISSTeligibility.CMPs(96%)reportedthatprimarycaregiversparticipateintheISSTeitherfrequentlyoralways;themajority(77%)reportedthatimportantdecisionsaboutservicesarerarelyornevermadewithoutthefamilypresent;andthevastmajority(89%)reportedthatthefamilyfrequentlyoralwaysparticipatesintheprocessofdevelopingandwritingserviceplans,andisthengivenacopyofthatplan(93%).

CMPsidentifiedanumberofbarriersbelievedtolimiteffectiveengagementoffamiliesintheirstructuresandprocesses.ThemostfrequentlycitedbarrierwasidentifyingfamiliesappropriateforIOGparticipation(64%),followedbyalackoffamilyknowledgeorexperienceaboutCMPprocesses(39%)andconfidentialityconcerns(39%).Projectscitedeffortstorecruit,train,andretainfamilyrepresentativesaswaystoovercomethesebarriers.TheFamilyVoiceandChoiceCommitteehasbeenintegraltofacilitatinglocalefforts.

GoalII:Reduceduplicationandeliminatefragmentationofservicesprovidedtochildrenorfamilieswhowouldbenefitfromintegratedmulti‐agencyservices.CMPsreportedaddressingduplicationandfragmentationofservicedeliverybyimplementingstrategiesinfourkeyareas:1)coordinationofclientconsentsandassessments;2)integratedserviceplanning;3)arrangingpaymentforandprovisionofservices;and4)informationsharingacrossagencies.

ThemajorityofCMPs(74%)useacommonconsentform,meaningthattheindividualorfamilyneedsignonlyoneconsentdocumenttoenableintervention‐relevantinformationtobesharedacrossallagenciesinvolvedinagivencase.

vii

AbouthalfofCMPs(14)reportedusingacommonclientassessmentorsharingassessmentresultsacrossagencies(e.g.,ColoradoJuvenileRiskAssessment).

WhilenearlyallCMPsreportedISSTserviceplansasbeingintegrated(92.5%)andindividualizedtofamilyneeds(78%),onlyhalf(13)indicatedsharingasinglecomprehensiveserviceplan.Somesitesdescribedthatpartneringagenciesagreeuponservicecomponentsyetcreatemultiple(i.e.,agency‐specific)plans.

Over80%ofCMPsshareservicedeliverycostsonacase‐by‐casebasis,withindividualagenciesdesignatedtopayforparticularcomponentsofeachindividualserviceplan.

LessthanhalfofCMPshaveformaldatasharingagreeements(46%)andprocessesinplaceforpartneragenciestoaccessdatasystemsforservicedelivery(32%)orevaluationpurposes(21%).

EffectivenessofeffortstoreduceduplicationandfragmentationAssurveyresultsshow,IOGmembersperceiveCMPeffortsaseffectivelyaddressingthislegislativegoal,withaveragescalescoresexceedingthe4.25benchmark(outofa6pointscale,with6indicatinggreatereffectiveness).Despiteprogressintheseareas,thefollowingbarrierswerereportedbymultipleCMPs:

Thelackofconnectionbetweendifferentstate‐leveldatabases(e.g.,Trails,Eclipse)makesitdifficulttoobtaincompleteinformationaboutyouthbeingservedacrosssystems.Thisimpedesidentificationoftheyouthwithhighestneedforintegratedplanning.

ManyCMPsexperiencechallengesinaccessingdatainspecificsystems(e.g.,schools,privateproviders)andensuringaccessforkeypersonnel(e.g.,enablingISSTfacilitatorstoaccessTrailsdata),whichslowsdownservicedelivery.

One‐quarterofCMPshavenotyetdevelopedacommonreleaseofinformation;somecitedifficultiesinmeetingallfederalandstatelawsaswellasthepoliciesofpartneragencies.

Theeconomicdownturnhasresultedinhighstaffturnoverinpartneragencies,leadingtodelaysandgapsinservicecoordination.

GoalIII:Increasethequality,appropriateness,andeffectivenessofservicesdeliveredtochildrenorfamilieswhowouldbenefitfromintegratedmulti‐agencyservices.CMPscontinuetoidentifyandimplementservicesmeetinghighstandardsofquality,whilealsoensuringthatthoseservicesarewellmatchedtothestrengthsandneedsoftheyouthandfamiliesbeingserved.

1 2 3 4 5 6

Fewerprofessionalsservingonefamily

Morecomprehensiveserviceplans

Increasedcontinuityofservices

Greaterintegrationofservices

Streamlinedserviceprovision

Reducedduplicationofservices

viii

SeventeenCMPscitedimprovementsinservicequalityasaprimaryachievementresultingfromtheirIOGeffortsinthepastyear.NotedindicatorsincludednewlyestablishedorenhancedISSTmodels,rolloutofnewprogramstargetingservicegapsforspecifiedpopulations,processestoensureearlycasecontact,andincreasedmonitoringandfollow‐upwithfamilies.

IOGmembersindicatedstrong“commitmenttoevidence‐based/informedmodels,”ratingthissurveyitemat5outofa6pointscale.ThisisreflectedinthegrowingnumberofCMPsimplementingtheHighFidelityWraparoundmodelii(7CMPs).

Whilewidespreaduseofthesemodelsreflectspositivelyonthequalityofserviceprovision,onlyaboutone‐third(35%)ofthesesitesmeasurequalityoftheirimplementationefforts(i.e.,fidelity).

NineteenCMPssupporttheuseofotherevidence‐informedpreventiveinterventions(e.g.,Multi‐SystemicTherapy,FunctionalFamilyTherapy,LifeSkillsTraining,IncredibleYears).iii

ThemajorityofCMPsimplementedatleastoneofthestrategiesspecifiedinthetablebelowtoincreasetheappropriatenessoftheirservicedeliveryinthelastyear.

CMPstrategiestoenhanceappropriatenessofservices NumberofCMPs(%)

IdentifiedserviceareathatiscurrentlynotbeingaddressedthroughinformalneedsassessmentwithCMPpartners(n=27)

24 (89%)

IdentifiedserviceareathatiscurrentlynotbeingaddressedthroughformalneedsassessmentwithCMPpartners(n=27)

16 (59%)

Implementedanewprogram/modelthatspecificallytargetsthepopulationandserviceneed/gapthatwasidentifiedbyIOG(n=27)

20 (74%)

Implemented/enhancedservicestobemoreculturallyappropriate(n=27) 11 (41%)

AchievementsrelatedtoCMPeffortstoincreaseeffectivenessOverall,IOGmembersreportedthattheirCMPsimplementpracticesthatresultinmoreeffectiveservicedeliveryandimprovedprogramoutcomes.Averageratingsofrelevantsurveyitemsexceededthe4.25effectivenessbenchmarkoutofa6pointscale,with6indicatinggreatereffectiveness.

“BecauseoftheCMP,weareseeingsystemsbeginningtorecognizewhatisnotworkingintheirprocessandwantingtoaddressit.Insteadofpointingfingers,we’rehavingmoreconversationswithawin/winattitude.Youtharenotbeinglabeledas“YourProblem”,ratherthecommunityasawholeisembracingtheirneedscollectively.Thisisatremendousstepintherightdirection.Trustisgrowingandsystemsrealizenooneisouttotearthemdownfortheirmistakes.RathertheIOGisheretohelpeachothersucceedforthebestinterestoftheyouth.”

1 2 3 4 5 6

Improvedqualityofservices

Committedtoevidence‐based/informedmodels

Easierforfamiliestoaccessservices

Committedtoa"nowrongdoor"approach

ix

Cost‐sharing:Useofblendedandbraidedfunds

“[OurCMP]continuestoincreasethefrequencyofblended/braidedfundingforthepurchaseofspecificallyidentifiedservicesforyouth/familieswhichhasincreasedeffectivenessoftheintegratedplansforclients,aswellasincreasedthestrategicuseofmultiplefundingstreams,andreducingfragmentationofservicesbetweenagencies.Ithasalsoelevatedsomediscussionsandinformalagreementsaroundwhopaysforwhatandwhenasitrelatestoasystemicapproachratherthanagency‐specific.Thereisincreasedcooperationonprojectsbetweenpartneragencieswhenitcomestothepursuitorinvolvementingrants,whichhasalsosupportedmorecross‐systemtrainingofstaffincertainareas.”

GoalIV:Encouragecostsharingamongserviceprovidersthatleadstocost‐reductionfortheservicesprovidedtochildrenandfamiliesinthechildwelfaresystem,includingthefostercaresystem,inthestateofColorado.Thelegislationassumesthateffectiveinteragencyservicecoordination,decreasesinduplication,andpreventionofdeeperandmorecostlyfamilyinvolvementinsocialservicesystems,shouldleadtobetterfamilyoutcomesandnetcost‐savingsovertime.Testingthecost‐savingshypothesisimpliedinthislegislativegoalataninitiativeleveliscomplicatedgiventherangeanddiversityofexistingprojects,processesandoutcomes.However,localCMPshavedevelopedprocessestosharecosts.Inaddition,CMPshavesuccessfullyleveragedtheircollaborativeeffortstoobtainadditionalfundingandre‐investsavingstosustainand/orgrowtheirprogramsandservices. IOGpartnersfrequentlyoralwaysjointlypaidforstaffing(20CMPs)andadministrativecosts(25

CMPs),bypoolingfundstosupportsalariesofIOGcoordinatorsorprogrampersonnel,andcontributingin‐kindresourcessuchasofficespaceandtrainings.

Asnotedearlier,CMPsregularlysharecostsofinterventionservices.Asacomplementtocost‐sharing,seventeenCMPsreportedallocatingnearly$714,000oftheirFY2010‐11incentivefunds(approximately29%oftheirtotalfunds)topayfordirectfamilyservices.

Tenofthe29CMPs(34%)reportedreceivingcompetitivefundsfromfederalandstateagenciesandprivatefoundations.Together,theseeffortsleveragedatleast$1,084,776inadditionalfunding.ManyCMPsalsoreceivednon‐competitivefunds.

CostsavingsassociatedwithCMPeffortsThelegislationsupportsCMPsinre‐investingrealizedcostsavingsinlocalefforts.However,measurementofcostsavingsischallenging,asitrequiresacalculatedestimateofthepotentialcostsforagivenindividualorfamilyhadCMPeffortsnotbeenprovided.Inaddition,whileoutcomesmaybeimproved,itisdifficulttodeterminewhetherthesewereachievedatlowercost,andsomemayarguethateffectivelyservingmulti‐systemsinvolvedfamiliesmayactuallyincreasecosts,atleastintheshort‐term.

Only3CMPsreportedhavingaformalprocesstomeasureactualcostsavings,although12CMPsprovidedmonetaryestimatesofperceivedsavings.Examplemeasuresofsavingsinclude:reductionsintheaveragenumberandlengthoftimeCMP‐servedchildrenwereinfostercareandotherplacements;comparisonofcostsofservicesfamiliesreceivedpriorto,andfollowingCMPservices;andreduceduseofemergencyhousingassistanceandTANFfunds.

x

TheseCMPsreportedatotalof$5.9millionthatwerereinvestedinCMPefforts.CMPsutilizedthesefundsmostoftentopurchaseadditionalservicesortoservemorefamilieswithexistingservices(16CMPs).Otherfrequentlymentionedusesofreinvestedfundsincluded:contributingtocoordinatorsalaries(4CMPs),infrastructurebuilding(e.g.,databasedevelopment),family“emergency”assistance,andcontributingtolocalevaluationcosts(2‐3CMPs).

CMPsdescribedothersourcesofperceivedcostreductions,butwereunabletoquantifyassociatedcostsavings.Theseinclude:fewerfamiliesenteringthechildwelfaresystem;lowertruancyrates(leadingtopresumedlowereducationcostsandincreasedschoolrevenues);reductionsinthelengthofmentalhealthtreatmentandinpatientstays;improvementsincoordinationandaccesstoneededcareandfundingsourcessuchasMedicaidandCHP+;andpreventionofcostlyservicesthroughtargetedprograms,includingpregnancyprevention.

GoalV:Leadtobetteroutcomesandcost‐reductionforservicesprovidedtochildrenandfamiliesinthechildwelfaresystem,includingthefostercaresystem,inColorado.CMPsprovidedestimatesofeligibleandservedpopulationsandperformanceonlocally‐definedgoals,includingchildandfamilyoutcomes.

Eligiblepopulation.Currently,itisdifficulttoobtainprecisenumbers,asthereisnostandarddefinitionofeligibility,youthmayberepresentedmorethanonce(duplicatedcases),andCMPstakevaryingapproachestocalculatethetotal.Thus,thecountsarelikelyoverestimated.

o InFY2010‐11,70,000individualswereconsideredeligibleforCMPservices.Abouthalf(30,000)ofthatpopulationwasreportedlyscreenedforCMPservices.

Servedpopulation.Thetotalservedwerecalculatedatdifferentlevelsofservicedelivery(allincludeduplicatedcases):

o 19,600individualsparticipatedinlocalservicesoreffortsthatwereassociatedwiththeirCMP(e.g.,paidbyincentivefunds,donatedin‐kind,providedbypartneringagencies).

o 7,000individualsreceivedservicesthroughthelocalISSTprocess.o 5,300individualsreceivedservicesthatwerefunded,atleastinpart,byCMPincentive

funds(among19CMPswheretheirIOGfinanciallysupportedspecificprograms).BecauseCMPsemployavarietyofservicemodelsthataretargetedtoindividualanddiversecommunityneeds,localsitestendtotracktheirperformanceinmanyoutcomes,utilizingmultipleindicatorswithinoutcomes.InFY2010‐11,thereviewofindicatorsrevealed:

The28CMPswithatleastoneyearofdatatrackedacombined198performanceindicators. Two‐thirds(134)measuredpopulation‐leveloutcomes(e.g.,childandfamily),while64measured

process(e.g.,servicedeliveryimprovements).Performancewasmeasuredatdifferentlevels(i.e.,rateforcountyversusyouthservedthroughtheISST).

Onaverage,eachCMPtracked7indicators,althoughafewCMPsmeasuredtwentyormore. CommondatasourcesincludeCDHSTrails,Eclipse/ICON,schooldistrictdata,andDBH/CCAR.

xi

Intermsofperformance:

CMPsreportedmeeting78%ofthe198performanceindicatorgoals.Successinmeetinggoalswashighinthreedomains,withlesssuccessseenintheeducationdomain.

Performanceonindicatorsreflectingthemostcommonlymeasuredoutcomesissummarizedinthetablebelow.Themajorityofperformancegoalstargetingoutcomesinthechildwelfareandjuvenilejusticedomainsweremet,withmorevariableperformanceonoutcomesineducationandhealth/mentalhealthdomains.

Domain CommonOutcome %GoalsMetFY2010‐11

ChildWelfare

Enhancestabilityofout‐of‐homeplacements 88%Preventout‐of‐homeplacement/increasereunification 80%

JuvenileJustice

Successfulcompletionofprobation 69%Loworreducedusageofcommitment/detentionfacilities 86%Reduceormaintainlowratesofre‐offense/recidivism 88%

EducationIncreaseattendance/reducetruancy 44%Increaseorimprovestudentachievement 60%

Health/MentalHealth

Improvedleveloffunctioninganddecreaseinproblemseverity 60%Increasepreventionandtreatmentforsubstanceuse/abuse* 100%Decreaseormaintainlowratesofhospitalization/inpatientservices 57%

*Note:OnlyoneperformancemeasurewasassessedinthisoutcomeareainFY10‐11.

AlthoughCMPsreportedonindicatorsrelatedtocommonoutcomes,thetargetpopulationsandspecificperformanceindicatorsforthoseoutcomesweredefinedandmeasureddifferentlyacrossCMPs.Thus,datacouldnotbemeaningfullyanalyzedacrosssitestoobtainapreciseestimateofimpactintheseoutcomesacrossthestatebecausedirectcomparisonisneitherpossiblenorappropriate.PlanforcommonmeasurementacrossCMPsinevaluationyear3ThediversityofmeasuresacrossCMPsledtothespecificationofastandardmeasurementplan.InJuly2011,sitesbeganuniformmeasurementofaselectsetofprocessandoutcomeindicators,whichwillbeaggregatedatthestatelevel.TheprocessindicatorswillfacilitateanaccuratecountanddescriptionoftheCMP‐targetedpopulationandtrackprogressoncoreISSTservicecomponents.Theseinclude:

Numberofchildren/familiesreferredtoISSTs Numberandtypeofagenciesorsystemsthat:familiesareinvolvedinatthetimeofenrollment;

participateinthefamily’sserviceplanning;andprovideservicestofamiliesasaresultoftheISST Numberofcaseswherefamilymembersparticipatedinserviceplanning Numberofcasesinwhichanintegratedserviceplanwasdeveloped

Theoutcomeindicatorswillmeasurecoreexpectedoutcomesinthechildwelfareandjuvenilejusticedomains,utilizingtheTrailsandICON/Eclipsestatewidedatabases.Theseinclude:

Number(rate)ofCMP‐servedyouth:withnewopeninvolvementsinTrailsafterCMPservicesbegan,withnosubstantiatedabusefinding,dischargedtoapermanenthome;andnumberofmovesexperiencedwheninout‐of‐homeplacement

xii

Number(rate)ofCMP‐servedyouthwho:successfullycompleteprobationand/orparole;recidivate;andexperiencerevocationsbytechnicalviolations

Ameasurementtoolandonlineclientleveldatabase(ETO©)wascreatedtosupportCMPdatacollection.InYear3,indicatorsintheeducationandhealth/mentalhealthdomainswillbeselected.

FUTUREEVALUATIONDIRECTIONSInyeartwo,theevaluationcenteredonidentifyingstandarddatacollectionopportunitiesacrossCMPsandestablishingmeasurementsystems,andinformingCMPbestpracticesandprogrammodelsthroughongoingprocessandformativeevaluationefforts.YearthreeevaluationactivitiesareintendedtoincreasecapacityandenhancedataqualityinwaysthatwillallowforanexaminationofCMPoutcomes.Theshiftfromformativeevaluationandinfrastructurebuildingtoasummativedesignwillrequirelocaladjustmentsindatacollectionpracticestoensureconsistentandrigorousmeasurementstatewide,whichmaybechallengingtosomesites.OMNI,CDHS,andtheEvaluationSubcommitteehaveproposedthefollowingyearthreeevaluationactivities:1. Continuetodevelopandimplementsystemstocollectandanalyzewell‐defineduniformmeasuresof:

a. OutcomesacrossCMPsonCMP‐targetedpopulations.OMNIwillcontinuetosupportlocalCMPsintheirdatacollectionofstandardandlocaloutcomemeasures.Mechanismstomeasurestandardoutcomesineducationandhealth/mentalhealthdomainswillbecollaborativelydevelopedandimplementedinFY12‐2013.

b. Outputandprocessmeasures.OMNIwillworkwithlocalCMPstofurtherrefineanddevelopadditionalmeasurementstrategiesincollaborativeprocesses,familyinvolvement,serviceduplicationandfragmentation,quality,effectiveness,andappropriatenessofservices,andcostsandestimatesofcostsavings.Theon‐linestatewidemeasurementdatacollectionsystemprovidesastructureandmechanismbywhichlocalCMPscoulddevelopadditionaluniformmeasuresofCMPprocesses.

2. MonitorandtracktrendsinstatewidemeasuresofkeyCMPoutcomes.OMNIandtheECwilldevelopan

on‐lineplatformtodisplaydatatrendsovertimeinthefourCMPoutcomedomainsatlocalandstatelevels,whichwillraiseawarenessoftheinitiativeamongstateleadersandcommunitymembers.

3. ContinuetodefineandsupportdisseminationofinformationoneffectiveIOG,ISST,andcollaborative

practices.OMNIevaluatorsareleadingorparticipatingineffortstoreviewCMP‐relevantresearchanddisseminatebestpractices.Theseactivitiesinclude:

a. Developmentofa“SystemsFacilitationGuide”summarizingresearchandlearningonsystemscollaborationandCMPcorecomponents(familyinvolvement,integratedserviceplanning).

b. Briefevaluationandresearchreportsonkeytopics.ThesequarterlyreportswillexploretopicsrelevanttostateandlocalCMPstakeholders,suchasmeasurementofcostsavings.

xiii

c. CMPHandbookdevelopment.ThehandbookwillorientnewCMPsandnewstakeholderstotheinitiative,theestablishinglegislation,SSCpoliciesandprocedures,localevaluationandmeasurementprocesses,andcollectivelearningregardingeffectiveimplementationofIOGandISSTstructuresandprocesses.

d. CMPcommunitynetworking.OMNIwillcontinuetosupportCMPinformationsharingthroughmaintainingtheCMPSharePointportal(anon‐lineinformationsharingtool)andattendingand/orfacilitatinglocally‐initiatedregionalmeetings.

e. Familyinvolvementtechnicalassistance.OMNIwillcontinuetosupporttheFamilyVoiceandChoiceCommittee’seffortstoexamineevaluationfindingstoidentifytechnicalassistanceneedsrelatedtofamilyinvolvementacrosstheinitiative.

f. Incentivefunding.OMNIwillfacilitatetheSSC’sdiscussionsregardingthedevelopmentofnewmethodstodetermineperformancebasedincentivefundingallocations.

Movingforward,theevaluationwillfocusoncontinueddevelopmentofmeasurementinfrastructuretocreateasolidfoundationforsummativeevaluationofCMPeffortsstatewide.OMNIwillcontinuetogroundtheevaluationinaparticipatoryapproach,workinginclosecollaborationwiththeCMPStateSteeringCommitteeandtheEvaluationSub‐committee,inanefforttoensurethatresultsarewellalignedwiththeneedsandinterestsoftheinitiativeanditsstakeholders.

iColoradoRevisedStatute,Title24,Article1.9.(2010).Retrievedfromhttp://www.michie.com/colorado/lpext.dll?f=templates&fn=main‐h.htm&cp.http://www.michie.com/colorado/lpext.dll?f=templates&fn=main‐h.htm&cp.iiBrunsEJ,SatherA,StambaughL.(2008).Nationaltrendsinimplementingwraparound:ResultsfromtheStateWraparoundSurvey,2007.In:Bruns,E.J.&Walker,J.S.,(Eds.),Resourceguidetowraparound.Portland,OR:NationalWraparoundInitiative,ResearchandTrainingCenterforFamilySupportandChildren’sMentalHealth.iiiSAMHSA:APocketGuidetoEvidence‐BasedPrograms.(2011).Retrievedfromhttp://www.samhsa.gov/ebpwebguide/appendixA_Schools.asp

1

TheCollaborativeManagementProgram(CMP)initiativeIn2004,theColoradoGeneralAssemblypassedHouseBill04‐1451(referredtoasHB1451)toestablishcollaborativemanagementprogramsdesignedtoimproveoutcomesforchildrenandfamiliesinvolvedwithmultiplecountyagencies.TheGeneralAssemblydeterminedthat“developmentofauniformsystemofcollaborativemanagementisnecessaryforagenciesatthestateandcountylevelstoeffectivelyandefficientlycollaborate,shareresources,ormanageandintegratethetreatmentandservicesprovidedtochildrenandfamilieswhobenefitfrommulti‐agencyservices.”iTheresultingCollaborativeManagementProgram(CMP)isdesignedtoimproveboththequalityandcost‐effectivenessofinterventionsforColoradochildrenandfamiliesinvolvedwithmultiplegovernmentalprogramsandcommunityagenciesstemmingfromhealth,education,childwelfare,andjuvenilejusticesystemcontact.Thelegislationcallsforthedevelopmentoflocalcollaborativemanagementstructuresandprocessesthatbringtogetheragenciesandservicesforat‐risk,highsystems‐usechildrenandfamilies.PartnersinlocalCMPsincludecountydepartmentsofhuman/socialservices,localjudicialdistricts,healthdepartments,schooldistricts,communitymentalhealthcentersandBehavioralHealthOrganizations,parentorfamilyadvocacygroups,andcommunityagencies.Thespecificgoalsofthelegislationareasfollows:

1. Developamoreuniformsystemofcollaborativemanagementthatincludestheinput,expertise,andactiveparticipationofparentadvocacyorfamilyadvocacyorganizations

2. Reduceduplicationandeliminatefragmentationofservicesprovidedtochildrenorfamilieswhowouldbenefitfromintegratedmulti‐agencyservices

3. Increasethequality,appropriateness,andeffectivenessofservicesdeliveredtochildrenorfamilieswhowouldbenefitfromintegratedmulti‐agencyservices

4. Encouragecostsharingamongserviceproviders5. Leadtobetteroutcomesandcost‐reductionfortheservicesprovidedtochildren

andfamiliesinthechildwelfaresystem,includingthefostercaresystem,inthestateofColorado.

2

Thelargergoaloftheinitiativeistoimproveoutcomesformulti‐systeminvolvedyouthandfamiliesthroughcross‐systemserviceplanningandcoordination.Researchhasdemonstratedthatthesecollaborativepracticesyieldimportantbenefitsincluding:

Increasedprobabilityofimprovementinchildandfamilyoutcomes

Maximizationofavailableresourcesfortheprovisionofservices Increasedcoordinationwithinandamongservicedeliverysystems

SharedresponsibilityacrosssystemsandserviceprovidersiiImportantly,thelegislationreflectsalonghistoryofsystemreforminColoradobasedonSystemsofCareprinciples.Coreelementsincludecommunitycollaboration,familyinvolvementinserviceplanninganddelivery,andculturallycompetentservicestailoredtotheuniqueneedsofdifferentpopulations.iiiTheseelementsareusedtoengagestakeholdersoutsidestateandlocalgovernmentinconsensus‐orientedeffortstomanagepublicresourcesandcollectivelysolveproblems.Inpart,communitycollaborationhasbecomeahallmarkofsocialservicesreforminColoradoduetoresearchindicatingitseffectivenessinengagingdiversedisciplinestoaddressissuesthathavemultiplecausesandsolutions.iv

A. TheCMPstateinitiative

TheCMPinitiativewasfirstimplementedinsixcountiesin2005‐2006,withadditionalcountiesaddedannually(seeTable1below).Inthe2011‐2012fiscalyear,thereare29localCMPsparticipatingintheinitiative,twoofwhicharemulti‐countypartnerships.Table1.NumberofcountiesparticipatingintheCMPinitiative

Year2004‐2005

2005‐2006

2006‐2007

2007‐2008

2008‐2009

2009‐2010

2010‐2011

2011‐2012

NumberofParticipatingCounties 6 12 15 24 29 31 30 31

Participatingcountiesvaryinsize,populationdensity,collaborativestructure,servicedeliveryapproach,andstatedoutcomes.Figure1providesastatewidemapofparticipatingcountiesbygeographictype.

“Collaboration…hasbeenseenasameansbywhichcomplexproblemswithinterrelatedcausescanbeaddressed,astrategyformaximizingtheefficientuseoflimitedresources,awayofreducingthefragmentationwithinandbetweenbureaucracies,[and]ameansofengagingcitizensinademocraticprocessofdecision‐making.”

Emshoffetal.,2007,AmericanJournalofCommunityPsychology

3

Figure1.CMPparticipatingcountiesbypopulationdensityandyearjoined

TheinitiativeismanagedbytheColoradoDepartmentofHumanServices(CDHS),withinputfromanon‐legislatedStateSteeringCommittee(SSC),andanumberofstandingandad‐hocsubcommittees.ThestructureofthesegroupsispresentedinAppendixA.SSCCommitteemembersincludeCMPcoordinators,stateagencyrepresentatives,astatefamily‐drivenorganization,andotherkeystakeholdersandpartners.In2006,theSSCdeterminedthatperformance‐basedmeasuresshouldbedevelopedbyCountyInteragencyOversightGroups(IOG)infourareasthatmatchlegislativeintent:

Childwelfare

Juvenilejusticesystem Education Health/mentalhealth/otherhealthservices(e.g.,substanceabuse)

4

Torewardcommunityefforts,performance‐basedincentivedollarsaredistributedbasedonachievementofoutcomesinthesefourareasandinaccordancewithaformulaapprovedbytheStateBoardofHumanServices.CMPsarerequiredtosetspecificgoalsinallfouroutcomeareasandtodescribehowthesewillbemet.CMPsuseincentivefundstosupportthecostsofcollaborativeprocesses(e.g.,coordinatorsalary)andservicedelivery(e.g.,existingandnewprograms).Projectsmayalsorequestwaiversofrulesandopttoreinvestanygeneralfundsavingsintoadditionalservicestofamiliesthatwouldbenefitfrommulti‐agencyservices.

B. CMP statewide program evaluation

In2008,HouseBill08‐1005authorizedanannualexternalevaluationoftheCMPinitiative.Specifically,ColoradoRevisedStatute(24‐1.9‐102.5)statesthattheDepartmentofHumanServices,withinputfromthecounties,involvedstateagencies,participatingstakeholders,andfamilyadvocacyorganizationswilldevelopthecriteriaandcomponentsoftheexternalevaluation.Thelegislationrequiresthatlocalprojectsreportonthefollowing:

1. thenumberofchildrenandfamiliesservedthroughtheirindividualizedserviceandsupportteamsandtheoutcomesoftheservicesprovided;

2. estimatedcosts,aswellascost‐shiftingorcost‐saving,relatedtoimplementingtheCMPapproach;and

3. informationrelevanttoimprovingthedeliveryofservicestopersonswhowouldbenefitfrommulti‐agencyservices.

InJulyof2009,CDHShiredOMNIInstitute(OMNI)toconductthestatewideevaluation.OMNIisanon‐profit,socialscienceresearch,evaluationandtechnicalassistancefirmbasedinDenver,Coloradoestablishedin1976(formerlyOMNIResearchandTraining,Inc.).InOctoberof2009,OMNIbeganworkinginpartnershipwiththeSSCandtheProgramDirectoratCDHStodeveloptheevaluationdesignandanEvaluationSubcommittee(EC)wascreatedtoguideevaluationefforts.MembershipintheECisopentoCMPpartnersandcurrentmembersincludeIOGcoordinatorsfromfourcounties;theProgramDirectoratCDHS;representativesfromtheDepartmentofCriminalJusticeandtheStateJudicialBranch;representativefromastatefamily‐drivenorganization;OMNIstaff;andresearchersfromtheUniversityofDenver.i. Phasesoftheevaluation

TheoverarchingfocusoftheevaluationisonexaminingtheeffectivenessofCMPeffortsinachievingthelegislativegoalsofHB1451(e.g.,increasedfamilyinvolvement;reducedduplicationandfragmentation;increasedquality,effectivenessandappropriatenessof

5

services;greatercostandresourcesharingacrossagencies;andimprovedchildandfamilyoutcomes).Becausetheprojecthadbeenoperationalforseveralyearsbeforecommencingtheevaluation,thedesignisbeingimplementedinmultiplephases.Theseincludeformativeevaluationtoserviceanddescribecurrentpractices,infrastructuredevelopmenttolaythefoundationforstandarddatacollection,andsummativeeffortstoassessindividualandcross‐siteeffects.Eachofthesephasesisdescribedbelow.PhaseI–Formativeassessment.Formativeeffortscollectdataintendedtohelpidentifyandrefineprogramactivities.Thefocusofdatacollectionistoexamineimplementation,identifyingbarriers,observedsuccesses,andotherqualitativeinformation,inordertogainanunderstandingoftheprogram.Informationgatheredcanbeusedtodevelopstrategiestostrengthentheprogram.Formativeevaluationmethodshelptosurfaceprogramneedstobeaddressedinthesecondphaseofinfrastructurebuilding.PhaseII–Infrastructurebuilding.Thesecondphaseoftheevaluationseekstolaythefoundationforstandardevaluationpracticesacrosssites.Evaluationeffortsfocusonthedevelopmentofmeasurementstrategies,implementationofdatacollectionsystems,andbuildingthecapacityoflocalprojectstoparticipateintheevaluation.Formativeevaluationtechniquesarealsousedinthisphasetofurtherinformprogramimplementationandsupportimprovementefforts.PhaseIII‐Summative.Summativeevaluationsexaminecumulativeoutcomesofaprogramorinitiative.Inthisphase,evaluationeffortsfocusonanalyzingdatatoexaminevariationinperformanceoutcomesasafunctionofdifferencesinpracticesandprocesses.Usingthistypeofanalyticapproach,theevaluationcandetermineprojecteffectsaswellasreflectonperformanceeffortstoidentifyeffectivepracticesandopportunitiesforfurtherprogramrefinement.TheCMPstatewideevaluationiscurrentlyinphasetwo.PhaseIwascompletedinthefirstyearinordertodocumentanddescribetherangeoflocaleffortsandselectedoutcomes.Thisworkhelpedtoilluminateawidevariationindefinedtargetpopulations,servicemodels,andmeasuredoutcomes.Suchdiversityposessignificantchallengesforthespecificationandapplicationofauniformmeasurementstrategyandrelatedoutcomeanalysis.Toaddressthesechallenges,OMNIrecommendedthattheprojectdevelopuniformoutputmeasures(e.g.,clientsserved,servicesprovided),asmallsetofcommonoutcomemeasures,andabasicdatamanagementsystemtosupportlocalcollectionefforts.Thisefforttobuildevaluationinfrastructurebeganinthesecondyear(2010‐2011)oftheevaluation,withtheECandCMPstakeholdersworkingtoestablishstandardmeasurementanddatasystemstosupportfutureevaluationactivities.Infrastructurebuildingwillcontinueandsummativeevaluationeffortswillbeginintheupcomingyear.

6

Itshouldbenotedthatthemovetowardmorerigorous,cross‐sitedatacollectionprocessesthatsupportuniformmeasurementofeffortsandoutcomeshasresultedinconcernsbeingvoicedbyasubsetofCMPcommunities.Thisistobeexpectedassitesinprioryearshavebeenabletosubmitdatainwaysthatdonotconformtostandardorrigorousdatacollectionstandardsormethods,andwhichcannotbeverified.Thus,therequesttoparticipateinastatewideevaluationrequiringsomeadditionaldatacollectionmaybeperceivedbysomeasanunwelcomeburden.OMNI,theEC,CMPCoordinators,inconjunctionwiththeprojectdirector,haveworkedcarefullytodevelopadatacollectionprocessthatinvolvesaverysmallsetofvariablesforthispurpose.OMNIstaffwillcontinuetoworkwithsitestosupportcollectionefforts,whilealsocreatingdatareportingtoolsthatwillhelpcoordinatorsandlocalISSTsusecollectedinformationtosupportserviceandmanagementactivities.ii. Yeartwoevaluationmethods

Thefollowingdescriptionoutlinestheevaluationeffortemployedinyeartwo.1. RefineCMPmeasurementprocessesInOctober2010,theEvaluationSubcommittee(EC)proposedtheformationofatemporaryworkgrouptohelpselectvariablesandrefineCMPmeasurementprocesses.Accordingly,anOutcomesWorkgroup,comprisedofkeyCMPstakeholders,wasassembledtoexplorethebenefitsofimplementinguniformmeasurementprocesses;examinekeyassumptionsandintentionsofthelegislationtoselectcriticalmeasurementareas;identifyasetofcommonindicators;anddevelopastandardmeasurementapproachtobeimplementedstatewide.LedbyOMNIandwithguidancefromtheEC,theworkgroupmetthreetimesforhalf‐dayretreatsinNovember2010,January2011,andFebruary2011.ThespecifictaskscompletedbytheOutcomesWorkgroupincludedthefollowing:

Identifiedwaystoimprovemeasurementprocessesthatarebeneficialtotheinitiative,bothlocallyandatthestatelevel

ExploredkeyassumptionsoftheCMPlegislationandidentifiedevaluationandmeasurementtargetsthatarecentraltotheinitiative

DevelopedameasurementapproachtobeimplementedacrossallCMPs SelectedanddefinedprocessandoutcomeindicatorsintheChildWelfareand

JuvenileJusticedomainstobemeasuredinthe2011‐12fiscalyear(andbeganexploringaprocessforlaterselectionanddefinitionofEducationandHealth/MentalHealthindicators)

Surfacedissuesandchallengesrelatedtodatacollectionandmeasurement,andidentifiedopportunitiestoaddresstheseproblems

7

DevelopedaplantodisseminateinformationtoStateSteeringCommitteeinMarch,2011andmaderefinementsbaseduponfeedback

FacilitatedimplementationintimeforintegrationoftheplanintoMOUsforthe2011‐12fiscalyear

TheeffortsoftheOutcomesWorkgroupledtothespecificationofastatewidemeasurementplanadoptedbytheSSCandrolledoutinJuly2011.OMNIidentifiedrelevantmeasurestosupportdatacollectionanddevelopedprotocolsfordataentryintoaweb‐basedmanagementsystem.Theplanisfurtherdescribedinthefinalsectionofthisreport.2. RefineandmaintaininfrastructuretoevaluateandshareprogressonlegislativegoalsTheevaluationestablishedandcontinuedsystemstodocumentandshareCMPprogress:

RevisionandadministrationofCMPannualreports.OMNIfurtherrefinedthestructureandcontentoftheCMPmid‐yearandannualreportstomakereportingeasierforCMPcoordinatorsandenhancetheusefulnessofdatacollected.Evaluatorsusedbothquantitativeandqualitativeapproachestoanalyzedatafromthesereports.

RevisionandadministrationofCMPcollaborativeeffectivenesssurvey.On‐linesurveysassessedIOGmembers’perceptionsofthequalityandeffectivenessoftheirlocalcollaborativepractices.Selectscalesfromthefirstyearadministrationwerere‐administeredinthe2010‐2011fiscalyeartoexplorechangesincollaborativeeffectivenessovertime.OMNIprovidedlocalresultstoindividualCMPsandanalyzeddataaggregatedacrossCMPstoassesscollaborativeeffectivenessstatewide.

Administrationoffamilyadvocacysurveys.IncollaborationwiththeFamilyVoiceandChoiceCommitteeandafamilyengagementresearcherfromWaldenUniversity,OMNIadministeredtwosurveysinSpring2011.OMNIcompiledthesedata,conductedaggregateanalyses,andprovidedareporttotheFamilyVoiceandChoiceCommitteetoinformfutureefforts.

Refinementsandtechnicalsupportoftheweb‐basedCMPportal.Establishedinthe2009‐2010fiscalyear,theCMPPortalisaweb‐basedsystemdesignedtofacilitatecommunicationandinformationsharingwithintheinitiative,providingaccesstocommondocuments,calendars,andresourcesinacentralizedlocation.BasedonfeedbackreceivedfromCMPstakeholders,evaluatorsmadefurtherrevisionstothecontentandstructureoftheCMPportalinthe2010‐2011fiscalyear.

EvaluationmethodstodatehavereliedoncollectingandanalyzinginformationaboutCMPstakeholders’perceptionsandotherself‐reporteddataduetotheabsenceofuniformprocessandmeasurementstrategiesacrossCMPs.Thus,findingsmustbeinterpretedwith

8

cautionastheymaybeinfluencedbystakeholderbiases.Asdiscussedabove,implementationofdatastandardsinyeartwowillsupportanalysisbetweenandacrosssitesbeginninginyearthree.

iii. Yeartwofindings

InformationprovidedbelowdescribeslocalCMPeffortstotargetthelegislativegoalsofHB1451andsummarizesfindingsfromthesecondyearofthestatewideevaluation.Findingspresentedinthisreportarenotexhaustive;additionalreportsanddeliverableshavebeenproducedduringthecourseoftheyear.Thelegislativegoalsprovidethestructureforthepresentationoffindingsthroughoutthisreport,witheachgoaldiscussedinturn.Ingeneral,eachreportsectionfirstdescribesCMPeffortstoaddresstherespectivegoal;exploresevidenceabouttheperceivedeffectivenessofthoseeffortstodate;andconcludeswithon‐goingchallengesorbarriersencounteredandopportunitiesforcontinuedimprovement.Thereportconcludeswithadiscussionofnextstepsanddirectionsforevaluationeffortsinyearthree.

9

LegislativeGoalI:Developamoreuniformsystemofcollaborativemanagementthatincludestheinput,expertise,andactiveparticipationofparentadvocacyorfamilyadvocacyorganizations.ThefirstlegislativegoaloftheCollaborativeManagementProgramsetsforthseveralexpectationsregardinglocalimplementationofacollaborativemanagementsystem.ItmandatesthatthemultipleagenciesaddressingtheneedsofColoradofamilies,youthandchildrendevelopamoreuniform,interagencysystemofservicemanagement.Itfurtherencouragescollaborativemanagementsystemstoengageparentsorfamilyadvocacygroupstoensurethesystem’sresponsivenesstothefamiliesitserves.ThefollowingsectionexaminesthedegreetowhichlocalCMPeffortshavecontributedtocreatinggreateruniformityinthecountysystemsaddressingtheneedsoffamilies,youthandchildren;moreeffectiveworkingrelationshipsbetweenagencies;andstrongerinvolvementofindividualsandorganizationsthatadvocateonbehalfoffamiliesservedbytheseagencies.WefirstdescribethemandatedcomponentsoflocalCMPsandthenexploretheextentofvariationamongprojectsinkeyareas.ThisdiscussionisfollowedbyapresentationoftheevidenceofoverallCMPeffectivenessandidentifiedchallengesorbarriers.Finally,weconcludethissectionwithadiscussionofCMPeffortstoinvolvefamiliesintheirwork.DataandinformationpresentedthroughoutthissectionwerecollectedfromCMPannualreportsandsurveydataoncollaborativeeffectivenessfromthe2010‐2011fiscalyear.

A. MandatedcomponentsofCMPs

Perthelegislation,countydepartmentsofhuman/socialservices,healthdepartments,judicialdistricts(probation),schooldistricts,andcommunitymentalhealthandbehavioralhealthorganizationsaremandatedtoparticipateinlocalCMPefforts.CMPsalsoarestronglyencouragedtoinvolvefamilyadvocacygroupsandothercommunityagencieswithrelatedmissionsandservicepopulations.Thelegislationalsostipulatesthattwoorganizationalstructuresaretobecreatedtosupportcollaborativeefforts:anInteragencyOversightGroup(IOG)tofacilitatecommunicationandresourcesharingacrossagencies;andanIndividualizedServiceandSupportTeam(ISST)toplanandcoordinatethedeliveryofservicesformulti‐agencyinvolvedfamilies.Additionally,CMPsarerequiredtoestablishprocessesthatfacilitateeffectivepartnerships,includingmethodsformakingcollectivedecisionsandimplementingproceduralchangeacrossagencies.

10

Finally,countiesmustsubmitMemorandaofUnderstanding(MOU)withparticipatingpartnersaddressingthefollowingareas:

Definitionofthepopulationtobeserved

Servicesandfundingsources CreationofanInteragencyOversightGroup(IOG) Developmentofcollaborativemanagementprocesses

DevelopmentofIndividualizedServiceandSupportTeams(ISST) Clearauthorizationtocontributeresourcesandfunding

Descriptionoftheprocesstoreinvestmoniessaved Performancebasedmeasures Confidentialityprotectionsandrequirements.

B. Localadaptation

Aswouldbeexpected,CMPsshareincommonmanycharacteristics,includingIOGandISSTcollaborativestructures,anemphasisonresourceallocationandreinvestment,andsimilardefinitionsoflocalperformancemeasures.However,CMPsalsoreflectaconsiderableamountofvariationinkeyareasincluding:

PatternsofIOGmembership,attendance,andareasoffocus; IndividualizedServiceandSupportTeam(ISST)modelsandprocesses;and CMPcoordinatorrolesandresponsibilities.

Eachoftheseareasisdescribedfurtherbelow.

i. IOGmembership,attendance,andareasoffocus

InformationcollectedinthefirstyearoftheevaluationrevealedthatthecompositionandactivitiesofIOGsvariedconsiderablyamongCMPs.YeartwodatacollectionalsoaddressedIOGattributes,includingIOGmembership,attendance,meetingfrequency,andareasoffocusduringthe2010‐2011fiscalyear.Analysisofthesedatashowedthatingeneral,bothmandatedandnon‐mandatedpartnersareengagedincollaborativeeffortsandthatthereexistmanycommonareasoffocusamongIOGs.

a) IOGmembershipandattendance

Inyearoneoftheevaluation,CMPsfrequentlycitedchallengestoobtainingbuy‐inandparticipationfromvariouskeystakeholders,mostnotably,families/parents,schools,andlawenforcementofficials.AsshowninTable2,however,membershipdataforthecurrentevaluationyearreflectrelativelysubstantialIOGparticipationofmanyofthesesamestakeholders,particularlylocalschools.Ingeneral,legislativelymandatedpartnerswere

11

Inclusionofnon‐mandatedpartnersonIOG

CMPsoftendescribedthepresenceofnon‐mandatedpartnersontheirIOGs: Nearlyall(89%)CMPsincludeSenateBill94staffontheirIOGs.

FourCMPsspecificallycitedtheinclusionofnon‐mandatedIOGpartnersasoneoftheirprimarysuccesses.Suchpartnersincludelocallawenforcementofficials,highereducationstaff,localtriballeaders,andothercommunitypartners.

wellrepresentedonIOGs,bothintermsofIOGmembershipandfrequencyofmeetingattendance.Infact,all(28)CMPsreportedIOGmembersfromcountydepartmentsofhumanorsocialservices,schools,andprobation,withthevastmajority(96%)attendingatleasthalfoftheIOGmeetingsintheyear.Ofnote,SenateBill94stakeholdersareincludedin25(89%)IOGsthoughtheirparticipationisnotrequired.

Table2.IOGmemberagenciesandattendance

AgencyorIndividualsRepresentedthroughMembershiponIOG

(n=28)

CountyDepartmentofHumanandSocialServices 28 (100%) 27 (96%)

SchoolRepresentative 28 (100%) 27 (96%)

Probation 28 (100%) 27 (96%)

Mentalhealthserviceprovidercenter 27 (96%) 25 (93%)

Substanceabuseserviceprovider 27 (96%) 24 (89%)

DivisionofYouthCorrections 27 (96%) 23 (85%)

Domesticviolenceserviceprovider 27 (96%) 23 (85%)

CountyHealthDepartment 26 (93%) 22 (85%)

SenateBill94representative 25 (89%) 22 (88%)

Behavioralhealthorganizationrepresentative 24 (86%) 20 (83%)

Localcourts/judicial 20 (71%) 13 (65%)

Diversion 13 (46%) 12 (92%)

Familyadvocacyorganization* 12 (43%) 10 (83%)

Lawenforcement 12 (43%) 7 (58%)

Familyrepresentative* 9 (32%) 0 (0%)

Localhealthservicesprovider 6 (21%) 6 (100%)

Electedofficial 6 (21%) 0 (0%)

Youthrepresentative* 4 (14%) 4 (100%)

BusinessorChamberofCommerce 3 (11%) 2 (67%)

Other 17 (61%) 16 (94%)

NumberofCMPsReportingIOGMember(%)

NumberofCMPsReportingIOG

MemberAttendingatLeast

50%ofIOGMeetings(%)

Note:Legislativelymandatedpartnersareshowninboldtextabove.*Insomeinstances,CMPsprovidedinconsistentresponsestosimilaritemswithintheAnnualReport.Assuch,countsofmembersreflectedabovemaydifferfromsimilarfigurespresentedelsewhereinthisreport.

Additionally,overallIOGcompositionreflectsthelegislativeexpectationofmulti‐agencymembershipreasonablywell.Onaverage,IOGsincludemembersfromabout12ofthe19

12

systemslistedinTable2,withaminimumof9andamaximumof16systemsrepresentedperIOG.Finally,nearlyall(27)IOGsmeetatleastquarterly,withabouthalf(15)meetingonamonthlybasis,andonlyonemeetinglessthanquarterly.

b) IOGareasoffocus

IOGsareexpectedtoconductoversightinseveralareas,includingimprovingcommunicationandresourcesharingacrosspartneragencies.Theyarealsograntedsignificantdiscretionindefiningtheirownpriorities.Inthe2010‐2011fiscalyear,IOGsidentifiedacommonsetofcoreactivitiesthatalignedcloselywiththelegislativegoals.ThefivemostcommonpriorityareasreportedarepresentedinTable3below,inorderoffrequency.Specificactivitiesassociatedwitheachpriorityareaalsoarelisted.

Table3.IOGpriorityareasandactivitiesinthe2010‐2011fiscalyear

PriorityArea ActivitiesIntroducingnewservicesEnsuringearlycontactbetweenprovidersandfamiliesRefiningISSTmodelsDevelopmentofIOGbylaws,clearpolicies,andproceduresdistributionofearnedincentivefundsFormationoftargetedIOGsub‐committeesClarificationofIOGmemberrolesandresponsibilitiesRecruitmentofnewpartners(lawenforcement,domesticviolence,tribalcommunities)CreatingopportunitiesforIOGpartnerstolearnmoreabouteachotherandtheCMPinitiativeHiringnewstaff(coordinators,facilitators)ProvidingtrainingandprofessionaldevelopmentopportunitiesStrategicplanningPursuedgrantfunding

AddressingCMPsustainability

Improvingqualityofservicedeliveryforchildrenandfamilies

Establishingand/orrefiningIOGstructure,processes,andprotocols

Strengtheningcollaborativerelationshipswithnewandexistingpartners

EnhancingCMPpersonnelresources

ii. IndividualizedServiceandSupportTeam(ISST)modelsandprocesses

TheISSTisaserviceplanningandcoordinatingbodythatbringstogetherprovidersfrommultipleagenciesand,insomecases,familyparticipants,todevelopanintegratedserviceplan.ISSTmodelsofservicedeliveryvariedsignificantlyacrossCMPs.YeartwooftheevaluationexploredISSTmodelsandprocessesbyexaminingthenumberofISST’simplementedperCMP,ISSTreferralsources,andagencyandfamilyinvolvementduringISSTs1.Findingsineachoftheseareasarepresentedbelow.

1SomeCMPsreportedmorethanoneISSTmodelorstructure.Thesecollaborativeswereaskedtofocusonone“primary”ISSTforthepurposesofrespondingtosubsequentquestionsaboutISSTcharacteristics.

13

a) NumberofISSTsandreferralstoISSTs

Onaverage,sitesreportedhavingtwoISSTs,withsixteenreportinghavingoneandonesitereportingutilizingsixseparatemodels.ThevariationinnumberandtheexistenceofmultipleISSTswithinasingleCMPreflectstheISSTmodel(s)beingused.Specifically,ISSTstypicallytakeoneofthreeforms:

ageneralISSTthatservesallfamilies;

aspecializedISSTwithinonebroadservicearena(e.g.,JuvenileJustice)thatservesonlyfamiliesreceivingrelatedservices;or

anISSTspecifictoanoutcomearea(e.g.truancyISST)thatservesonlyfamiliestargetingthespecificoutcome.

CMPswithmultipleISSTsusuallyhaveatleastonethatfocusesonaspecificissueorpopulation.

Table4.ISSTreferralsources

ReferralSource(Nvariesbyreferralsource)DepartmentofHumanServices 27 (96%) 40% (1% ‐80%)Schooldistrict 26 (93%) 24% (1% ‐66%)Probation 22 (79%) 15% (2% ‐45%)Mental/behavioralhealth 19 (68%) 9% (2% ‐25%)Court/judicial 13 (46%) 14% (4% ‐50%)Lawenforcement 10 (36%) 2% (1% ‐4%)DepartmentofPublicHealth 9 (32%) 6% (1% ‐22%)Self/parent 9 (32%) 11% (3% ‐25%)Diversion 7 (25%) 10% (5% ‐17%)DivisionofYouthCorrections 7 (25%) 5% (1% ‐8%)Other 14 (50%) 19% (3% ‐48%)

NumberofCMPs

Reporting(%)

Average%ofReferrals(Min.‐Max.)

YouthandfamiliesaretypicallyreferredtothelocalISSTthroughaCMPpartneragencyorotherentityinthecommunity.Table4(above)presentsthemostcommonreferralsources,includingthenumberofCMPsreportingeach,aswellastheaverageshareofreferralsmadebyeachsource.Thetwomostcommonreferralsources,bothofwhichalsomadethelargestshareofreferrals,arecountydepartmentsofhumanservicesandschooldistricts,respectively.Additionally,fourteen(14)countiesindicatedthattheyreceivereferralsfromsourcesnotlistedabove.Theseincludedomesticviolenceproviders,primarycarephysicians(pediatricians),SenateBill94staff,andearlychildhoodeducationproviders.

14

CMPimpactonstaffingteams

“We’vealwayshadgoodcollaboration,butithasgonetoadifferentlevelsincebecomingapartoftheCMP.Wearemoreeffectiveinourstaffing.Weusedtooverwhelmfamilieswithtoomanypeopleintheroom.ThankstoISSTandwrap,wehaverefinedourprocess.Wehavetherightpeopleintheroomattherighttime.We’vebroughtinmorefundingopportunitiesandprogrammingasaresultofourcollaboration.Andlastly,theroleofthefamilyintheirowncaseplanningisclearerandmoreeffective.”

b) ISSTstructureandagencyparticipation

BecauseISSTsaredesignedtoconvenerepresentativesfrommultipleagencies,CMPswereaskedaboutthenumberofagenciesparticipatinginatypicalISSTstaffing.Twenty‐six(26)indicatedthatatleastfouragenciesweretypicallyinattendance,withthirteen(13)indicatingparticipationofsevenormoreagencies.Themajority(59%)ofCMPsreportedthatISSTparticipationisbothstandardandtailoredtotheneedsoftheindividualyouthbeingserved;thatis,theyincludeacoresetofcommonparticipantsaswellasadditionalattendeesbasedonthespecificissuespresented.Justoverone‐quarter(26%)reportedthatISSTparticipationisdeterminedsolelyontheneedsoftheindividualyouth.CMPsalsodescribedthetypicalstructureofanISSTmeeting,namelyhowmanyfamiliesareaddressedduringthecourseofasinglemeeting,howmuchtimeisspentperfamily,andhowmanymeetingsareheldperfamily.Findingsareasfollows:

Projectsarenearlyevenlydividedbetweenthoseaddressingonefamilypermeetingversusthoseaddressingmultiplefamiliespermeeting(42%and46%,respectively).

Onaverage,CMPsdedicateaboutonehourperfamilyperISST,thoughthisvariedfromaboutfifteenminutestotwohours.

Onaverage,CMPsholdfiveISSTmeetingsperfamily,thoughthisvariedwidely,fromaminimumofonetoamaximumofthirty.2

iii. CMPcoordinatorrolesandresponsibilities

Inyearoneoftheevaluation,CMPsemphasizedthevalueofhavingadesignatedCMPcoordinator,ideallydedicatedtoCMPactivitiesonafull‐timebasis.Inyeartwo,CMPswereaskedtoprovideadditionalinformationabouttheircoordinators,particularlyaboutthecharacteristicsandrolesassociatedwiththisposition.Nearlyall(96%)ofCMPsreportedhavingadesignatedcoordinator,withthemajority(63%)servinginthisroleonafull‐timebasis.AbouthalfofCMP coordinatorsarehostedbythecountyDepartmentofHumanorSocialServices.Lesscommonhostagenciesincludementalhealthcenters,localfamily/youthadvocacyorganizations,thejuvenilejusticesystem,andlocalschools.Ofnote,

2WhenreportingtheaverageamountoftimespentinanISSTperfamilyandtheaveragenumberofISSTmeetingsperfamily,manyCMPsprovidedranges(e.g.,between1and3hours;orbetween10and15meetings).Intheseinstances,themid‐pointoftheestimatedrangewasusedforpurposesofanalysis.

15

aboutone‐thirdofCMPsreportedajointcoordinatorhost,oftenwithoneagency“housing”thecoordinatorandanotherservingasfiscalagentfortheposition. ThecoordinatorpositionisfundedfullywithCMPincentivefunds(i.e.,theyearlyfundsallocatedtoeachCMPbyCDHS)in14programs.However,six(6)projects(22%)reportedthattheircoordinatorpositionisfundedonlypartiallybyCMPincentivefunds,andseven(7)counties(26%)reportedthatthepositionissupportedentirelybyotherfunds.

Table5.CMPcoordinatorroles

CoordinatorRole(N=27)

Liaisonbetweenpartners 27 (100%)Disseminatescommunication 27 (100%)Documentsdecisionmaking 27 (100%)ISSTteammember 20 (74%)IOGleadership 19 (70%)Managesfundsandoverseesbudgets 18 (67%)CoordinatesISST 17 (63%)Managesintakeandreferrals 15 (56%)Conductsfollowupwithfamilies 13 (48%)Providesdirectservice 10 (37%)Other 12 (44%)

NumberofCMPsReporting

"Yes"(%)

CMPswerealsoaskedtoidentifythekeyrolesandresponsibilitiesofthecoordinatorposition(seeTable5,above,forthemostcommonresponses).CoordinatorsmostcommonlyprovidesupportattheIOG‐level,withall(100%)servingasliaisonsamongCMPpartners,disseminatinginformation,anddocumentingdecision‐makingprocessesandoutcomes.However,manycoordinatorsalsosupportserviceplanninganddelivery,includingparticipatingonandcoordinatingISSTs,andmanagingintake,referrals,andfollow‐upwithfamiliesserved.Twelve(12;44%)CMPsindicatedthattheircoordinatorhasresponsibilitiesotherthanthoselistedinthetable,suchasservingasaliaisonwithpeopleandagenciesoutsideofthelocalCMP(e.g.,CMPStateSteeringCommitteeandsub‐committees),managingdatacollectionandreporting,andsupportingfundraisingefforts.

C. EffectivenessoftheinteragencycollaborationestablishedbyCMPs

EvaluationeffortsalsoexaminedtheoutcomesofCMPcollaborativeprocessesandstructures.Inthefirstyearoftheevaluation,OMNIworkedwithcollaborationresearchersattheUniversityofDenvertoemploytwomethodstogatherinformationaboutcollaborativeeffectiveness.First,theyconductedkeyinformantinterviewswithasampleofIOGcoordinatorsandstate‐levelstakeholders.InterviewsexaminedprocessesusedtoestablishandimplementtheCMP,reflectionsoninclusionanddecisionmaking,and

16

lessonslearned.Thisworkculminatedinareportdescribingkeypracticesandperceivedindicatorsofcollaborativeeffectiveness(seeAppendixBforalinktothefullreport).Second,togatherabroaderperspectiveoncollaborativeeffectivenessandfurtherexplorethemesfromstakeholderinterviews,evaluatorsdevelopedasurveyforIOGmembers.TheCollaborationSurveyandOverallSuccessSurveywereadministeredto235IOGmembersin22CMPsinthe2009‐2010fiscalyear,andarevisedversionintegratingselectscalesfromthetwosurveyswascompletedby276IOGmembersin27CMPsinthe2010‐2011fiscalyear(seeAppendixBforalinktoreportsdescribingthesurveysinfurtherdetail).Thesurveymeasuresseveraldimensionsofcollaborativeeffectivenessthat,overall,reflecta)perceivedqualityofthecollaborativeprocess,andb)perceivedimpactofCMPsinreachingbothlegislativeandothergoals.AlthoughthemeasurereliesuponIOGmembers’perceptionsofeffectiveness,surveyitemsassessingthecollaborativeprocesshavebeenfoundtobepredictiveofpositiveoutcomesthatpublichealthcollaborativesaredesignedtoachieve.v

i. Qualityofcollaborativeprocesses

Overall,IOGmembersperceivetheircollaborativestohavestrongandproductivestructuresandprocessesinplace.Thesurveyincludedthreekeyscalesassessingthequalityofthecollaborativeprocess:

Structuralintegrity,whichoccurswhentheprocessisperceivedas“fair.”The

processallowsforsufficientopportunityforstakeholderstochallengeandrevisedecisions,butinacontextinwhichallpartnersfeelequallyheardandrespected

Authenticprocess,whichresultswhencollaborativepartnersperceivethedecision‐makingprocessas“openandcredible,”becausetheyseethemselvesashavingrealpowertobothformulateandmakebindingdecisionsvi

Strongleadership,whichreflectstheperceptionthateffortsofcollaborativepartnersareledbydedicatedandeffectivecoordinators.

Onaverage,IOGmembersindicatedthattheirCMP’seffortswerecharacterizedbyhighqualityexperiencesineachofthesethreeareas,withaveragescoresexceedingthe4.25markerofcollaborativeeffectivenessona6‐pointscale,withhigherratingsindicatinggreatercollaborationvii.Ingeneral,therewaslittlevariabilityintheaveragescoresintheseareasacrossCMPs.

17

Figure2.Ratingsofthequalityofcollaborativeprocess

Note:Itemsareratedona6pointscale(1=stronglydisagree,2=disagree,3=disagreemorethanagree,4=agreemorethandisagree,5=agree,6=stronglyagree).N=276respondentsfrom27IOGs.

ii. Successofcollaborativeprocesses

IOGmembersalsoratedtheextenttowhichtheircollaborativeprocessesledtosystems‐levelsuccessesduringthe2010‐2011fiscalyear.AsseeninFigure3,onaverage,IOGmembersperceiveahighlevelofgeneralsuccess,believingthattheirCMPsgenerallyhaveaccomplishedorexceededtheiryearlygoals.Additionalevidenceofcollaborativeeffectivenesswasgatheredinannualreports,inwhichCMPsdescribedsignificantsuccessesrealizedasaresultoftheircollaborativeeffortsduringthe2010‐2011fiscalyear.Themostcommonlymentionedareasofsuccess(mentionedbymorethanfiveCMPs)appearinTable6,alongwithexamplesofreportedachievementsineacharea.

18

Figure3.Item‐levelratingsofoverallsuccessofCMPduringtheFY2010‐2011

1 2 3 4 5 6

Hasaccomplisheditsspecificgoals

Hasachievedmorethanitsoriginalgoals

Hasledtonewprojectsorefforts

Hasachievedextraordinarysuccess

Note:Itemsareratedona6pointscale(1=stronglydisagree,2=disagree,3=disagreemorethanagree,4=agreemorethandisagree,5=agree,6=stronglyagree).N=276respondentsfrom27IOGs.

Table6.CMP‐reportedevidenceofcollaborativeeffectiveness

AreasofCollaborativeEffectiveness ExampleAchievements

Establishmentofsubcommittees,leadingtomoreefficientuseofIOGmeetings•asteeringcommitteetooversee"day‐to‐day"operationalactivities•anoutcomescommitteetospecifylocalevaluationmeasuresandgoalsFullpartnerparticipationinstrategicplanningsessionsCreationofyouthadvisoryboardEstablishmentofa"virtualvoting"processwithinaruralIOGtoincreaseactiveparticipationindecision‐makingRoll‐outofnewprogramstargetingareassuchaspregnancy,mentoring,youthservicelearningIncreasesinthenumberoffamiliesreceivingspecificservicesIncreasesinthenumberofreferrals,indicatinggreatercommunityawarenessofeffectivenessofservicedelivery

DecreasedtimebetweenreferralandenrollmentininterventionservicesFundedsuccessfulgrantwritingeffortsBlendedorbraidedfundingidentifiedforspecificservicesleadingtomoreintegrationandstreamlinedservicesamongpartneragenciesEstablishmentofprocesstograntseedfundstonewprograms

ImprovedprocesseswithintheIOG

Developmentofnewprogramsor

enhancementofexistingservicedelivery

Supportingsustainability

19

PositivecommunityimpactsasaresultofCMPprocesses

CMPswereaskedtoreflecton“whathaschangedinyourcommunitysincetheinitiationofCMP.”“OneofthebiggestimpactstheCMPhashadin[ourcounty]isincreasedcommunicationbetweenagencies.Thepartneragenciesareworkingmoretocollaborate,shareresourcesandreduceduplicationofservices.Asasmallcommunity,relationshipsalreadyexistedandhaveonlybeenstrengthenedbytheirmembershipintheCMP…largeroverarchingissuesgetaddressed…andspecificplanstoaddressareasofconcerncanmoreeasilybecollaboratedupon...”“Thereisgreatercommunicationbetweentheagencies,namelytheschoolsandDSS.Inthepast,therehasbeensomecontentionbetweenthetwoaroundwhoisresponsibleforhelpingthisfamilyandwhowillpayforit.Forexample,ayouthwouldbetruant,andinsteadofutilizinganRTIprocess,theschoolwouldpuntthekidtoDSSandaskthemtofileaD&N.Thisputtheagenciesatoddswitheachother.BecauseoftheCMP,weareseeingsystemsbeginningtorecognizewhatisnotworkingintheirprocessandwantingtoaddressit.Insteadofpointingfingers,we’rehavingmoreconversationswithawin/winattitude.Youtharenotbeinglabeledas“YourProblem”,ratherthecommunityasawholeisembracingtheirneedscollectively.Thisisatremendousstepintherightdirection.Trustisgrowingandsystemsrealizenooneisouttotearthemdownfortheirmistakes.RathertheIOGisheretohelpeachothersucceedforthebestinterestoftheyouth.”“[Our]county’sCollaborativeManagementProgramisembarkingonitsfifthyear.Eachyearhasshownincreasedcollaborationandeffectiveservicedeliveryamongyouth‐servingagenciesasthemodelofcollaborationsupportedbytheCMPgraduallyhasbecomemoreofaculturalnormwithinourcommunity.TheIOGanticipatesthattheCMPwillcontinuetofine‐tuneitseffortsandthatitssuccesswillcontinuetogrow.”“Thoughagencieshavealwaysworkedtogether,thisefforthasprovidedtheresources(coordinator,fundingforsomeservices)thatallowtheagenciestostepbackandlookintosystemsandknowthatthereisamechanismthatwillhelpsolveissuesthatarise.1451hasbecomeahouseholdwordamongouragencies–peopleknowwhatwearetalkingaboutwhenwesay1451.Agenciesknowthat1451isagoodcollaborativepartnerinmeetingtheneedsofhighriskchildrenandfamilies.”“OurcountyhasexperiencedasenseofunityandpurposethatisdirectlytiedtotheCMP.Weareidentifyingweaknessesandfeelingthateffectivestrategieswillbefoundtosolvelocalproblems.”

20

iii. ChallengesassociatedwithCMPcollaborativestructuresandprocesses

Takentogether,theresultsaboverevealedthatIOGmembersgenerallybelievethattheyhaveestablishedhigh‐qualityandeffectivecollaborativepartnerships.Despitetheseaccomplishments,CMPsstillstrugglewithsomecommonissues.Themostcommonchallenges,alongwithstrategiesusedtoovercomethem(ifidentified),include: Participationandbuy‐infrompartners.Eight(8)CMPsciteddifficultiesobtainingbuy‐in

frompartners,bothinCMPoversightandservicedelivery.SomeoftheseCMPsindicatedthatengagingspecificpartners,suchasschools,non‐profits,andcitygovernmentagencies,hasbeendifficult.OthersdescribedsituationsinwhichIOGmembersagreeuponaplan,butthenmayfailtocommunicatethistothespecificstaffmembers(s)responsibleforimplementation.CMPsindicatedthattheyareworkingtoaddresstheseissuesbyincorporatingexpectationsaboutparticipationandinternalcommunicationintoagencypoliciesandcounty‐levelMOUs.

CMPstaffing.Anequalnumber(8)ofCMPsreportedthatinadequatestaffingandturnoverhaveimpactedCMPadministrationandservicedelivery,citingdifficultiesresultingfromthelackofacoordinator,ISSTfacilitator,orotherstaff.SomeCMPshavecontractedwithprivateprovidersoragenciestoaddressstaffingneeds.

Financialstability/sustainability.Seven(7)CMPsspecificallyreportedchallengesassociatedwithageneralshortageoffunding,largelyattributedtotheeconomicdownturn.Inordertoaddressthesechallenges,CMPsindicatedthattheyareundertakingadditionaleffortstoobtainexternalfunding.

Informationsharing/confidentiality.Barriersrelatedtoclient‐levelinformationsharingwerereportedbysix(6)CMPs.Inmostinstances,CMPsdescribedalackofpartneragreementregardingconfidentialityrequirements.StrategiesintendedtoclarifytheserequirementsincludeconsultationwiththeColoradoChildandYouthInformationSharingCollaborative(CCYIS),aswellasgeneraldiscussionamongpartners.

UnderstandingandclarityofCMPmission.Finally,three(3)CMPsciteddifficultyensuringclearandcommonunderstandingofcollaborativegoals,activities,andpracticesamongpartners.OnefactorcontributingtothischallengeappearstobeturnoverattheIOG‐level,aswellasatpartneragencies.Toaddressthischallenge,someCMPshaveconductedresearchoncollaborativeeffectiveness,consultedwithCMPcoordinatorsinothercommunities,andheldregularmeetingstodiscusstheseissues.

Overall,CMPsdescribedtheirlocalcollaborativestructuresandrelationshipsasstrong,andreportedthatjointeffortshaveledtosignificantimprovementsinservicedeliverysystemsandcapacitytoaddresscommunityneeds.AlthoughCMPscontinuetoexperiencesomebarriers,theyappeartobeworkingtogethertoidentifysolutionsandinnovationstoovercomethese.

21

D. Inclusionoftheinput,expertise,andactiveparticipationofparentorfamilyadvocacyorganizations

TheestablishinglegislationstronglyencouragesCMPstoengageparentorfamilyadvocacygroupstoensurethatsystemsareresponsivetothefamiliestheyserve.AlthoughthelegislativelanguagefocusesoninvolvementoffamilyrepresentativesattheIOGlevel,theCMPStateSteeringCommitteehasencouragedCMPstoalsoincorporatefamilyadvocatesandfamilymembersattheISSTlevel.Astatefamily‐drivenorganization,theFederationofFamiliesforChildren’sMentalHealth,hasbeenastrongproponentandfacilitatoroftheseefforts.Familyrepresentativescontributeuniqueperspectivesnotonlytopolicybutalsotothecontentanddeliveryoffamilyservices.Forexample,familyrepresentativesmayidentifypotentialchallengesforfamiliesinserviceplansorpolicies,whichcaninformeffortstodevelopmoreintegratedandstreamlinedprocessesthatresultinbetterfamilyoutcomes.ThissectionpresentsfindingsrelatingtofamilyinvolvementinCMPactivities,includingstrategies,evidenceofeffectiveness,andon‐goingbarriers.Weincludeinformationontheextenttowhichfamilyandyouth,aswellasfamilyadvocates,areincludedinCMPactivitiesanddescribeevidenceoftheeffectivenessoffamilyengagementstrategiesandcommonbarriersencountered.Thisisreportedatthreelevels:

InvolvementoffamilyrepresentativesinIOGefforts; InvolvementoffamilyrepresentativesadvocatingforfamiliesinISSTs;and Engagementoffamiliesintheirownservicedeliverysystemsandprocesses.

i. InvolvementoffamilyrepresentativesinIOGefforts

AttheIOGlevel,familyinvolvementismostreadilyreflectedbythepresenceoffamilymembers,youth,and/orfamilyrepresentativesontheIOGitself.Table7belowprovidesthenumberandproportionofCMPsreportingIOGmembersineachofthesethreecategories.Over40%ofCMPsreportedatleastonefamilymemberontheirIOG,about40%reportedatleastonefamilyadvocate,andabout20%reportedatleastoneyouthmember.

22

Table7.FamilyrepresentationonIOGs

0 17 (59%) 23 (79%) 18 (62%)1 8 (28%) 3 (10%) 5 (17%)2 4 (14%) 2 (7%) 4 (14%)3 0 (0%) 0 (0%) 1 (3%)4 0 (0%) 0 (0%) 1 (3%)5+ 0 (0%) 1 (3%) 0 (0%)

NumberofRepresentatives

onIOG

FamilyMember(%)

(N=29)

YouthMember(%)

(N=29)

FamilyAdvocate(%)

(N=29)

NumberofCMPsReporting

TheextentoffamilyinvolvementintheIOGisreflectednotonlybythepresenceoftheserepresentativesbutalsointheirroleasadecision‐maker.Onlythree(3)oftheCMPsthatdohavefamilyrepresentativesontheirIOGsincludethemasnon‐votingmembers.

ii. InvolvementoffamilyrepresentativesadvocatingforfamiliesinISSTs

Familyinvolvementalsooccurswithinserviceplanningeffortsonbehalfofotherfamilies.NearlyhalfofCMPs(12CMPs,46%)reportedthatfamilymembersarealwaysviewedaspartnersinserviceplanningforotherfamilies,andthevastmajority(85%)ofCMPsreportedthatfamilymembersareseeninthiswayatleast“sometimes.”WhileCMPsreportedasignificantdegreeoffamilymemberinvolvementforthebenefitofotherCMPfamilies,few(22%)indicatedthattheyhaveaprocessinplacetosupportormentorfamilymembersservinginthisrole.Thesix(6)CMPsthatdidreporthavingsuchaprocessinplacedescribedthefollowingtypesofsupport:

orientationprocessestoclarifyCMPmissionandactivitiesaswellastheroleoffamilypartners;

generalsupervision,coachingormentoringprovidedbyotherCMPpersonnel;and assistanceobtainingexternaltraining.

Localhighlight:IOG‐levelyouthadvisoryboards

SomeCMPsdescribedinnovativestrategiesforincludingyouthinCMPoversightactivities.Forexample,oneCMPreportedtheimplementationofaYouthAdvisoryBoardthatworksincollaborationwithitsIOG.TheBoard,whichiscomprisedof10localyouth,wasestablishedduringthecourseofthe2010‐2011fiscalyearandmeetsindependentlyoftheIOG,providesinputonIOGactivities,andvotesonallofficialIOGissues.

23

Lessthanhalf(42%)ofCMPsemploypaidorvolunteerfamilyadvocates(alsoreferredtoasfamilysystemsnavigatorsandfamilysupportpartners)toworkwithfamiliesgoingthroughtheISSTprocess.AdditionaldatawerecollectedfromCMPsthisyeartobetterunderstandtheroles,training,background,andresourceneedsoffamilyadvocates;findingsaresummarizedinthehighlightboxbelow.

FamilyInvolvementSurvey:RoleoffamilyadvocatesinserviceplanninganddeliverySince2007‐2008,theFamilyVoiceandChoiceCommittee(FV&C)hasbeencollectingdataaboutCMPfamilyinvolvement.Previousyears’datacollectioncenteredonfamilyinvolvementattheIOG‐levelwhereasthisyear’seffortsfocusedonlearningaboutfamilyadvocacyactivities.SurveysweredesignedinpartnershipwiththeFV&CCommittee,OMNI,andDr.DonnaHeretick,afamilyengagementresearcheratWaldenUniversity,andadministeredinthespringof2011.Twenty‐fourcoordinatorsand42individualsservinginafamilyadvocacyrolecompletedsurveys.Highlightedfindingsareasfollows: Of24totalcoordinatorsurveyparticipants,42%(13)reportedhavingfamilyadvocatesinvolvedinISSTs.

ThemajorityofCMPswithfamilyadvocatesreportedthattheyarepaid,mostoftenbythelocalDHS,andaboutone‐third(36%)devotemorethan20hoursperweektothiswork.

CMPfamilyadvocatesreportedavarietyofrolesandresponsibilities,themostcommonofwhichwereparticipatinginIOGmeetings;consultingwithfamiliesabouton‐goingissues;referringfamiliestolocalservicesandsupports;andassistinginfamily‐levelprogramorservicedesign.

Overhalf(55%)ofadvocatesreportedcaseloadsof10orless,andaboutone‐thirdreportedlargercaseloadsofanywherebetween11‐30families.

Familyadvocatetrainingmostfrequentlyinvolvesinformalandformalagency‐basedtraining,specializedworkshops;andinformation‐sharingamongpractitioners,bothon‐andoff‐the‐job.

Whenaskedtodescribethegreatestchallengesorresourceneedsforenhancingfamilyadvocacywork,themostcommonresponsesincluded:

o TrainingonprofessionalrolesandresponsibilitiesforFamilyAdvocates(e.g.,confidentialityprotocols,communicationpractices);

o SupportanddevelopmentofFamilyAdvocateswithpersonalexperiencewiththeCMP,namelypreviousCMPclients;

o Developmentofahiringmanualandformalpolicies/proceduresaboutFamilyAdvocatesforCMPs;and

o Additionalfundingforadvocatepositionsandassociatedcosts.

24

iii. Engagementoffamiliesintheirownservicedeliverysystemsandprocesses

AcorevalueoftheCMPinitiativeismeaningfulinvolvementoffamilymembersindevelopingtheirownserviceplans.Ingeneral,CMPsreportedasubstantialdegreeoffamilyengagementinexploredareasincludingthetimingoffirstcontactwiththefamily,ISSTparticipationbyaprimarycaregiver,andthefrequencywithwhichimportantdecisionsaremadewiththefamilypresent.Themajority(63%,17CMPs)ofCMPsindicatedinvolvingfamiliesimmediatelyafterbeingdeterminedeligibleforISSTservicesandnearlyone‐third(30%,8CMPs)reportedthatfamiliesarefirstcontactedpriortoadecisionregardingISSTeligibility.AsshowninTable8,nearlyallCMPs(96%)reportedthatprimarycaregiversparticipateintheISSTeitherfrequentlyoralways;themajority(77%)reportedthatimportantdecisionsaboutservicesarerarelyornevermadewithoutthefamilypresent;andthevastmajority(89%)reportedthatthefamilyfrequentlyoralwaysparticipatesintheprocessofdevelopingandwritingserviceplans,andisthengivenacopyofthatplan(93%).

Table8.ISST‐levelfamilyinvolvementpractices

Always 19 (70%) 6 (22%) 1 (4%) 19 (70%) 21 (78%)Frequently 7 (26%) 6 (22%) 0 (0%) 5 (19%) 4 (15%)Sometimes 0 (0%) 11 (41%) 5 (19%) 2 (7%) 2 (7%)Rarely 1 (4%) 3 (11%) 11 (42%) 1 (4%) 0 (0%)Never 0 (0%) 1 (4%) 9 (35%) 0 (0%) 0 (0%)

ImportantDecisionsMadeWithoutChildorFamilyPresent

(n=26)

FamilyParticipatesinISSTServicePlanning(n=27)

FamilyGivenaCopyofISSTPlan

(n=27)

NumberofCMPsReporting(%)

Frequency

AtLeastOnePrimaryCaregiverParticipateson

ISST(n=27)

Family,Friend,AdvocateOtherthanFamilyon

ISST(n=27)

iv. EffectivenessoffamilyinvolvementinCMP

Toalargedegree,themeasureofeffectivenessofthislegislativegoalisreflectedbythenumbersofCMPswithfamilymembersinvolvedinIOGsandISSTs,aswellastheextenttowhichCMPsreportthatthesemembersarevaluedasactivepartnersinCMPservicedelivery.Thefindingspresentedabovesuggestthatprojectsaremakingstridesinthisarea,withapproximatelytwo‐thirdsofCMPsreportingactiveparticipationoffamilyrepresentativesintheirIOGsandalmostallCMPsindicatingactiveinvolvementoffamiliesinISSTmeetings.Additionally,someCMPstakeholdersspecificallyhighlightedachievementsaroundfamilyinvolvement,withonereportingthat“agenciesarebeginningtoletgoofcertainagendaswithyouthandfamiliesandprocessingwiththefamiliestodecide

25

whattheywanttoworkon”andanotherthat“systems[are]understandingfamiliesbetter,whichcreatesbetterplansandoutcomesbasedonthefamilies`strengthsandculture.Moreindividualizedgoalsandplanscreatebettersuccess.”Some,butnotallCMPs(59%),reportedthattheyhaveastructuredorformalon‐goingprocesstogatheradditionalevidenceofeffectivenessinfamilyinvolvement.TheseCMPsmostcommonlydescribedthecollectionoffamilysatisfactionsurveys(69%),followedbytrackingfamilyparticipationinISSTmeetings(63%),andtrackingfamilyrepresentativeparticipationinIOGmeetings(38%).Someofthelearningsfromtheseeffortsarehighlightedabove.Further,manyCMPsprovidedanecdotalevidenceoftheeffectivenessoftheireffortsinthisareaviastoriesofhowfamilyengagementleddirectlytoimprovementsinservicedeliveryandpositiveoutcomesforchildrenandfamilies.Someofthesestoriesaresharedonthefollowingpage.

Localevaluationoffamilyinvolvementefforts

Threesiteshighlightedinformationobtainedviaeffortstoevaluatetheirfamilyinvolvementefforts:

“Thevastmajorityoffamiliesthathavecompletedthesurveyfeelthattheirvoiceswereheard,theirconcernsandneedsconsidered,andtheirchildren’sneedsmetthroughtheCMP.”

“WehaverealizedthatwhenmorefamilymembersareinvolvedintheISSTprocess,theyaremoreengagedintheprocess,thetargetedyouthismoresupportedandconfidentinthemeeting,andtheyaremorelikelytocontinuetoengageinfollowup[ISST]conferences.”

“Family/youthpostservicesurveysindicatefamiliesfeeltheywereanequalpartoftheprocess,thattheprocesswasrespectful,andthattheirperspectiveswereprioritized.”

26

v. Challengestofamilyinvolvement

Whiletheinitiativehasseennotablegrowthinthedegreeoffamilyinvolvementoverall,CMPsalsoidentifiedanumberofbarriersbelievedtolimiteffectiveengagementoffamilies.AsillustratedinTable9below,themostfrequentlycitedbarrierwasidentifyingfamiliesappropriateforIOGparticipation(64%),followedbyalackoffamilyknowledgeorexperienceaboutCMPprocessesandconfidentialityconcerns(39%;seeTable9).

Table9.CMP‐reportedbarrierstofamilyinvolvement

IssuesidentifyingappropriatefamiliesforIOGparticipation 18 (64%)Confidentialityissues 11 (39%)Lackofknowledge/experienceoffamilyoryouthrecruitedaboutCMPprocesses 11 (39%)Timeconstraints/scheduling 7 (25%)Lackofcommitmentamongrecruitedyouthandfamilies 6 (21%)Lackoffundingtocompensatefamilyandyouthmembers 6 (21%)Geography/distance 3 (11%)Personnelturnover 1 (4%)Otherbarriers 7 (25%)

BarrierDescription(N=29)

NumberofCMPsReporting(%)

Note:Totalssumtogreaterthan29CMPsand100%becauseCMPscouldselectupto3barriers.

CMPsdescribedavarietyofapproachestoaddressthesebarriers.Selectstrategiesarelistedbelowandillustratethewiderangeofmethodsbeingusedtofosterfamilyinvolvementstatewide:

Caseexamplesofeffectivefamilyinvolvement“The[ISST]metwiththechildandfamilytodiscusswayswecouldsupportthechildtomaintainaneducationalsetting,inthedelinquencysystemandathome.Thefamilyrequesteddaytreatmentservices,coaching(mentoring)servicesandinhometherapy.[TheISST]fundedallthreeservicesandcontinuedtomeetwiththefamilyonamonthlybasistoaddressanyconcernsandhelpthefamilyaccomplishtheiridentifiedgoals.Thechildisnowonareducedschoolscheduleandisgettinghomeboundservicesthroughtheschooldistrict.ThefamilyandchildhavesuccessfullycompletedtheirinhomefamilytherapyprogramandtheyouthhasnotgottenanynewchargessinceJanuaryofthisyear.DHShasclosedtheircaseastherearenolongeranybeyondcontrolofparentissues.”OneCMP‐servedfamily“mentionedthattheyappreciatedthefactwelistenedtothemandhelpedthembrainstormideasversustellingthemwhattheyshoulddo.Using[theISST],theFacilitatorwasabletocoordinatewiththeschooltoletthemknowthatthedaughterwasonnewmedication,thefatherreceivedcounselingthroughourmentalhealthpartner,andthefamilyworkedthroughthe[ISST]process.Schoolreportsthedaughterisdoingmuchbetterinschool;anddadisstableonhismedicationandcontinueswithcounseling,andtheparentscontinuetoco‐parentthechildren.”

27

Recruitment/retention.SomeCMPsdescribedstrategiesfocusedontherecruitmentoffamilymembersandrepresentativeswithappropriateknowledgeofandexperiencewiththeCMP,includingoutreachtograduatesfromlocalprograms,aswellasprovidingfinancialcompensationtofamilyrepresentativesinrecognitionoftheircontributions.

Traininganddevelopment.CMPsalsodescribedeffortstosecureandprovidefinancialsupporttoenablefamilyrepresentativestoattendtrainingsdeemedrelevanttotheirrolesandresponsibilitieswithintheIOG.

Planning.Finally,someCMPsdescribedstrategiesintendedtoformalizetheirfocusonfamilyinvolvement,includingthecreationofaYouthAdvisoryCouncil,whichinformsIOGprocessesanddecision‐making,aswellasthedevelopmentofformalplanningofeffortstoincreasefamilyinvolvement.

Insummary,the31countiesparticipatingintheCMPinitiativehaveaddressedthislegislativegoaltopromotegreateruniformityinthesystemsservingColoradofamilies,children,andyouth,byimplementingspecifiedorganizationalstructures,suchasIOGsandISSTs,andprocessessuchasfamilyinvolvement,necessarytoimplementasystemofcollaborativemanagement.AlthoughlegislationhassetsomecommonstandardsforCMPs,therehasbeenconsiderableleewayforlocalsitestoadapttheirstructuresandprocesses.TheevaluationrevealedthatIOGsacrossthestatesharesimilarprioritizedgoals,butsetupdiversestructuresandactivitiestoachievegoals(e.g.,intermsofthemembershipandoperationsoftheirIOGs,ISSTs,andstaffing).Overall,sitesperceivetheirlocalcollaborativestructuresandrelationshipstobestrongandeffectivelyresultinginimprovementsintheirservicedeliverysystemsandcapacitytoaddresscommunityneeds.Inparticular,CMPsdescribedsignificantprogresstowardincreasedandmeaningfulfamilyinvolvementatalllevels.AlthoughCMPscontinuetoexperiencesomebarriers,theyreportedactiveeffortstocontinuetolearnfromandbuildupontheirsuccesses,aswellasspecificstrategiesintendedtoaddressbarriersencounteredtodate.

28

LegislativeGoalII:Reduceduplicationandeliminatefragmentationofservicesprovidedtochildrenorfamilieswhowouldbenefitfromintegratedmulti‐agencyservices.ThesecondlegislativegoaloftheCollaborativeManagementProgramistointegratemulti‐agencyservicesforfamiliesandchildren,suchthattheduplicationandfragmentationinservicedeliveryisreducedoreliminated.Serviceduplicationreferstosituationswherefamiliesreceivethesameservicesfrommultipleagencies.Fragmentationreflectstheconditionofuncoordinated,conflicting,orunrelatedserviceprovisionwhentheseservicesarenotmanagedorcoordinatedacrossagencies.Bothissuescanbeaddressed,atleastinpart,throughtheprovisionofinteragencyplanningandservicemanagementefforts.Sucheffortscanresultinthealignmentofrequirementsforfamilies,easingofmulti‐agencynavigation,andthecreationofcostefficienciesbyeliminatingunnecessaryduplicationofservices.CMPsreportimplementingavarietyofstrategiesdesignedtoimpactduplicationandfragmentationofservicedelivery.Theseinclude:a)coordinationofclientconsentsandassessments;b)integratedserviceplanning;c)arrangingpaymentforandprovisionofservices;andd)informationsharingacrossagencies.Activitiesineachoftheseareasareoutlinedbelow,followedbyadiscussionoftheirperceivedeffectiveness,andrelatedbarriers.DataaboutthestrategiesbywhichCMPshaveworkedtoaddressbothduplicationandfragmentationweredrawnfrommid‐yearandannualreportsandsurveydataoncollaborativeeffectiveness.

A. Coordinationofclientconsentsandassessment

Projectcoordinatorswereaskedwhethertheyusecommonprocessesanddatacollectionformstoobtainclientconsentsandassessmentinformation.Amajority(74%)ofCMPsuseacommonconsentformthatissharedacrossallagencies,meaningthattheindividualorfamilyneedsignonlyoneconsentform.Mostoften,thiscommonreleaseoccursattheISSTandenablesthesharingofclient‐levelinformationacrossallagenciesinvolvedinagivencase.However,three(3)CMPsindicatedthatnotallpartneragenciesareincludedassomeagenciesrequireadditionalconsentfromyouthorfamilies,andotherselectnottousethecommonform.Two(2)CMPsindicatedthat,whiletheydohaveacommonconsent,individualyouthorfamiliesareallowedtospecifywhichagenciesmayandmaynothaveaccesstotheirinformationortorevoketheirconsentatanytime.AbouthalfofCMPs(14)reportedhavingacommonclientassessmentformandprocessthatissharedacrossagencies.Whilethespecificformvaries,themostcommon

29

CMPimpactonserviceplanninganddelivery

OneCMPreportedthat“initiatingtheCMP…hasshiftedtheperspectiveofpartneringorganizationstothatofamorecollaborativefocusonprovidingthoseservices.AswedevelopandfullyimplementourISSTprocedures,therewillbeamoreevidentimpactontheactualdeliveryofdirectservices.”

assessmentsarethoseusedwithinthejuvenilejusticesystem[e.g.,ColoradoJuvenileRiskAssessment(CJRA),MassachusettsYouthScreeningInstrument(MAYSI)].Someoftheseassessmentsareextensive(e.g.,CJRA)andhavebeenwidelyadoptedorarerequiredbymultipleprogramsinColorado(e.g.,SB94,diversionprograms);thus,sharingassessmentdatareducesduplicationinproviderandfamilytimeandeffort.OneCMPdescribedtheexpectedrelationshipbetweenitscommonclientassessmentandeffortstoreduceduplicationandfragmentationinthefollowingway:[allclientsservedbytheISSTprocess]“engageinacomprehensiveassessmentprocess.Theinformationcollected…isdocumentedbytheISST’scoordinatorandsharedwithallagencieswhoparticipateontheISSTinordertohelpavoidduplicationofservicesandensurethecontinuityofprovidedservices.”

B. Integrationofservicedeliveryplanning

Anothercomponentofcoordinatedserviceplanninganddeliveryisthedevelopmentofanintegratedserviceplan.ThelegislationrequiresISSTstodevelopservicedeliveryplansforeachstaffedyouthand/orfamily.Whilethestructureoftheseplansisnotdictated,thelargergoalsoftheinitiative(i.e.,tofosterintegratedandappropriateservicedelivery,toreduceduplicationandeliminatefragmentationofservices)wouldsuggestthatasingle,individualizedplanbedeveloped.CMPswereaskedtodescribethedegreetowhichtheirISSTplanspossessthesetwokeycharacteristicsandreportedthefollowing:

AllserviceplansdevelopedduringISSTprocessesareindividualizedatsomelevel,withthemajority(78%)indicatingthatplansareindividualizedtothefamilybeingserved,and22%indicatingthatplansareindividualizedtotheyouthbeingserved.

Additionally,nearlyall(92.5%)ofCMPsreportedthatISSTserviceplansareintegrated.However,onlyhalfofthese(13)indicatedthatasingular,integratedserviceplanisdevelopedandsharedamongproviders.Somesitesdescribedthatpartneringagenciesagreeuponservicecomponentsyetcreatemultiple(i.e.,agency‐specific)plans.Whiletheseplansmayreflectsomeintegrationofservices,thepresenceofasingle,agreed‐uponplanmaysignaltofamiliesagreatercommitmentamongallagenciestotruecoordinationandsharedresponsibilityofservices.

C. CoordinationofpaymentforandprovisionofservicestoCMPyouth

CMPsdescribedstrategiesbywhichresponsibilityforcostsand/ordeliveryofservicestoyouthandfamiliesaresharedamongpartneragencies.Over80%ofCMPssharethecosts

30

ofservicedeliveryonacase‐by‐casebasis,withindividualagenciesdesignatedtopayforparticularcomponentsofeachindividualserviceplan.Veryfew(7%)CMPsreportedhavingashared,generalfund,fromwhichservicesarepaid.CMPsalsodescribedthefrequencywithwhichpartneragenciesareexpresslydesignatedtoeitherpayforand/ordeliverservicestotheyouthorfamiliestheyserve.Nearlyallsitesindicateddesignatingaspecificagencywitheachoftheseresponsibilitieseitheroftenoralways(seeTable10below).Processessuchastheseservetoclarifyagencies’rolesinservicedeliveryandmaybeviewedasindicatorsofcollaborationandmoreefficientservicedelivery.

Table10.CMPdesignationofagencyserviceresponsibilities

Always 10 (37%) 19 (70%)Often 15 (56%) 7 (26%)Sometimes 1 (4%) 1 (4%)Rarely 1 (4%) 0 (0%)Never 0 (0%) 0 (0%)

NumberofCMPs(%)(N=27)

Frequency Agency(ies)aredesignatedtopay

forservices

Agency(ies)aredesignatedto

provideservices

CMPsalsotargetconflictingmandatesortreatmentrequirementsacrosspartneragenciesinordertoreduceduplicationandfragmentationintheirlocalsystems.All29CMPsreportedsomereductioninthisareaandoverhalfoftheCMPs(15)indicatedreducingduplicationandfragmentation“alot,”througheffortssuchasclarifyingandsimplifyingpoliciesandreducingconflictingrequirements.AnexamplereflectedinaCMPMOUstatesthat“Whenanexisting[ISST]caseexistsandplacementdecisionisrequired,thediscussionanddecisionwilloccurwithinthe[ISST]meetingandwillnotalsogothroughthe[establishedlocalDHSplacementteam]process."Thistypeofagreementstreamlinesprocessesanddecision‐makingtobetterservefamiliesinthecounty.

D. Informationsharingpractices

Thecreationofsystemsthatfacilitatethesharingofclientdataandrelatedinformationacrossagenciesrepresentsanotherstrategythatsupportsinter‐agencycoordinationandserviceintegration.Allowingstafffrommultiplesystemstoaccessacommonsetofclient‐leveldatacreatesopportunitiestoidentifyandaddressinstancesofserviceduplicationand/orfragmentation.SomeCMPsreportedhavingestablishedsuchsystemsbothforthe

31

purposesofsharinginformationgenerally,andforsharinginformationspecificallyaboutindividualyouthorfamiliesservedbythecollaborative.EachofthestrategieslistedinTable11(below)representamethodCMPsusetoenhancemoresystematicaccesstodataaboutcollaborativeactivitiesandservicesprovidedtoclients.Table11.CMPinformationsharingstrategies

InformationSharingStrategy(n=28)

Formaldatasharingagreementinplace. 13 (46%)Cross‐agencyaccesstodataforservicedelivery. 9 (32%)Cross‐agencyaccesstodataformeasurement/evaluation. 6 (21%)

NumberofCMPs

Reporting(%)

Theexistenceofaformaldatasharingagreementindicatesthat,tosomedegree,CMPpartnershavebeenabletoarticulateandformallydocumenttheirexpectationsofoneanotherwithrespecttothehandlingandsharingofdataaboutthecollaborativeanditsclients.Whilethirteen(13)CMPsindicatedhavingformaldatasharingagreementsinplace,thedegreeofspecificity,detailandbreadthoftheseagreementsvariessignificantlyacrossthesesites.Inmostinstances,itappearsthattheagreementsreferencedbyprojectsareembeddedinotherdocuments,suchasreleasesofinformationorCMPMemorandaofUnderstandingwiththestate,thataddressarangeofissuesbeyonddatasharing.Thesematerialsdonottypicallyaddresscross‐agencydatasharingwiththedegreeofdetailappropriateforsharingsensitiveclient‐levelinformationacrossCMPpartners.Asshowninthetableabove,nine(9)CMPsindicatedthattheyprovidecross‐agencyaccesstodataforpurposesofservicedelivery.Forexample,oneCMPreportedthat“sharinginformationacrossagencieshasbeeninvaluable.Weoftenaredealingwiththesamefamiliesandhavenoideathattheyareaskingforservicesfromotheragencies.”Again,thestructureandformalityoftheseprocessesvaries.Ingeneral,theseprocessesrelyonexistingdatabases,maintainedeitherbyalocalagencypartnerorastate‐levelagency.Mostfrequently,cross‐agencydataaccessisprovidedforspecificindividualsassociatedwiththeCMP(e.g.,theCMPCoordinatororISSTstaff)andforselectdatasystems,mostoftenTrails.Further,mostCMPsdescribedcross‐agencyaccessbetweentwo,ratherthanall,partneragencies.

32

CMPsalsodescribedthesystemsanddatatheyusetosupportprogramevaluationandmeasurement.Theseeffortsandrelateddataarelikelytohelpsupporttheassessmentofperformanceandtheidentificationofopportunitiesforimprovement.Onlyaboutone‐fifthofCMPs(6)reportedprocessesthatsupportcross‐agencyinformationsharingforthispurpose.However,on‐goingworkbytheSSCandinfrastructuredevelopedaspartofthestatewideevaluationhavecreatedsignificantopportunitiestoexpandthetrackingandsharingofdatatomeasureCMPperformance,asdiscussedfurtherinthedescriptionofthestatewidemeasurementplan(page55).Finally,nearlyallCMPs(27)reportedinformalmethodsofdataandinformationsharing,suchastelephoneoremailcommunication.Whilethesemethodsplayanimportantroleincoordinatedserviceplanninganddelivery,theymaybeinsufficienttosupportallCMPdataandinformationsharingneeds,particularlygiventhecomplexandfast‐pacednatureofCMPwork.

Localhighlight:CrossoverYouthPracticeModelOneCMPisimplementingtheGeorgetownUniversityCenterforJuvenileJusticeReformCrossoverYouthPracticeModel,whichitcitesasakeyapproachinitseffortstoreduceduplicationandfragmentation.Specifically,theCMPreportedthat“theCrossoverYouthPracticeModelisusedtoidentifyyouthinvolvedintheChildWelfaresystemassoonastheyarearrestedandscreenedbySB94.MoreformalizedjointplanningisconductedbytheJuvenileJusticesystemandtheChildWelfaresystem.DiversionalsoaddressesCrossovercasesandworkstonotduplicateexpectations.TheCourthasbeenusingthe…ISSTtoassistinthedevelopmentofmanagementplanswhenyoutharedeemedincompetent.OneISSThasincreasedthenumberoftruancyyouthstaffedinordertoreduceunnecessaryreferralsto[theDepartmentofHumanServices]whenservicescanbeprovidedbypartneragenciesorbarrierscanbeproblem‐solvedcreativelywiththefamilyandassistanceprovided.”PertheCenterwebsite,theCrossoverYouthPracticeModelbringstogetherresearchandpracticelearningaboutstrategiestoserveyouthengagedinboththechildwelfareandjuvenilejusticesystems.ConsistenttheCMPprinciples,themodelemphasizes“thefollowingpractices:thecreationofaprocessforidentifyingcrossoveryouthatthepointofcrossingover,ensuringthatworkersareexchanginginformationinatimelymanner,includingfamiliesinalldecision‐makingaspectsofthecase…andmaximizingtheservicesutilizedbyeachsystemtopreventcrossoverfromoccurring.”(http://cjjr.georgetown.edu/pm/practicemodel.html)

33

E. EffectivenessofCMPsinreducingduplicationandeliminatingfragmentation

TheCollaborativeSuccessSurvey(seelinktoadditionalinformationinAppendixB)includedmultipleitemstoassesstheeffectivenessofCMPeffortstoreducebothduplicationandfragmentation.SurveydataindicatethatIOGmembersperceivetheirCMPcollaborativetohaveledtoreductionsinduplicationandfragmentationacrossallmeasurementareas,asshowninFigure4.IOGmembersassignedanaverageratingof4.5orhigheroneachitem,indicatinghighperceivedsuccessinstreamliningandintegratingservices,andprovidingmorecomprehensiveservicesthataredeliveredbyfewerprofessionals,thusreducingduplication.

Figure4.Effectivenessofeffortstoreduceduplicationandfragmentation

1 2 3 4 5 6

Fewerprofessionalsservingonefamily

Morecomprehensiveserviceplans

Increasedcontinuityofservices

Greaterintegrationofservices

Streamlinedserviceprovision

Reducedduplicationofservices

Note:Itemsareratedona6pointscale(1=stronglydisagree,2=disagree,3=disagreemorethanagree,4=agreemorethandisagree,5=agree,6=stronglyagree).

F. Challengesassociatedwithduplicationandfragmentation

Despiteprogresstowardreducingserviceduplicationandfragmentation,manyCMPsreportedsignificantbarriersthatimpedeadvancementinthisarea.First,someCMPsexpressedfrustrationaboutthelackofconnectionbetweendifferentstate‐leveldatabases,suchasTrailsandEclipse.Becauseclient‐levelrecordscannoteasilybelinkedacrossthesedatabases,CMPandpartneragencystaffmaystruggletoobtaincompleteinformationaboutyouthbeingservedacrosssystems.CMPsindicatedthatestablishingconnectionsacrossthesedatabasescouldenablethemtomoreeffectivelyidentifyyouthappropriateforCMPservices,aswellasassessinstancesinwhichintegratedplanningmightbebeneficial.CMPsalsociteddifficultiesaccessingdatainspecificsystems(e.g.,thosemaintainedby

34

schoolsandprivateproviders)andensuringaccesstoexistingdataforindividualpersonnel(e.g.,enablingISSTfacilitatorstoaccessTrailsdata).Second,manyCMPsdescribedbarriersaroundinformationsharingandconfidentiality,specificallynotingchallengesindevelopingacommonreleaseofinformationthatmeetsallfederalandstatelawsaswellasthepoliciesofindividualpartneragencies.Finally,someCMPsindicatedthattheeconomicdownturnhaslimitedtheresourcesavailabletosupportinformationsharingandresultedinstaffturnoverinpartneragencies.WhereassignificantprogresshadbeenmadebyCMPstoreduceduplicationandfragmentationofservicesacrosstheirlocalsystems,furtheropportunitiesexisttotargetthislegislativegoal.Currentstrategiesincludedevelopmentofsharedformsandintegratedservicedeliveryplans,informationanddatasharingpractices,andjointly‐determineddivisionofresponsibilitiesforpaymentandserviceprovision.Securingsharedaccesstokeydatasourcesacrossagenciesremainsanimportantareaforcontinuedworkatthelocalaswellasstatelevelinordertoensureintegratedplanningcanoccur.Inaddition,itwillbeimportantmovingforwardforallCMPstodevelopmulti‐agencyconsentforms,ifthesearenotalreadyinplace,aspartofbroadservicemanagementeffortsaswellastoensurethatclient‐levelinformationcanbesharedacrossCMPpartners.Finally,developmentofasingular,integratedserviceplanthatcanbesharedamongprovidersisanotherareaforcontinuedexplorationinordertotargetreductionsinduplicationandfragmentationofservicesformulti‐systemfamilies.

35

CMPeffectonservicesforyouthandfamilies

“Historically,smallruralcommunitieshavehadtorelyoncollaborationtomakelimitedfundinggoalongway.Byformalizingtheprocessandaddingfunding,ithasallowedmanymoreservicestostayinplacethatwouldhavebeeneliminatedduetotheeconomicclimate.Theseservicesareabsolutelycriticalinlowsocio‐economicareaswhereneedyfamiliesarealargeportionofthepopulation.”TheCMPhasledto“afocusonEBPs[Evidence‐basedPrograms]asasystemandstartingtocreateacommonlanguageonwhatweareaboutandwhatwewanttoaccomplish.”

LegislativeGoalIII:Increasethequality,appropriateness,andeffectivenessofservicesdeliveredtochildrenorfamilieswhowouldbenefitfromintegratedmulti‐agencyservices.Thethirdlegislativegoaloftheinitiativeistoimproveservicesprovidedtofamiliesengagedwithmultipleagencies.Inresponse,CMPshavedevelopedandimplementedanumberofstrategiesthataredesignedtoimpactservicequality,appropriateness,andeffectivenessformulti‐systemfamilies.Thissectionexaminessomespecificapproaches,includingtheuseofevidence‐basedorevidence‐informedmodels,andstrategiestoidentifyexistingservicegaps.Weconcludethesectionwithadiscussionoftheevidenceofperceivedeffectivenessinthisarea.DataandinformationpresentedthroughoutthissectionweredrawnfromCMPannualreportsandsurveydataoncollaborativeeffectivenessfromthe2010‐2011fiscalyear.

A. Servicequality

Yearoneevaluationfindingsindicatedthatstakeholders,byandlarge,believethatlocalCMPshadimprovedthequalityanddeliveryoflocallyprovidedservices.Inthesecondyear,CMPscontinuedtomakeprogressinthisareawith17citingimprovementsinservicequalityasaprimarysuccessthisyear.Specifically,IOGshaveundertakeneffortstoestablishorrefinetheirISSTmodels,introducenewprograms,focuseffortstowardidentifiedtargetpopulations,ensureearlycasecontact,andprovideongoingmonitoringandfollow‐upwithfamilies.WithintheISSTstructure,CMPshaveaddressedservicequalitythroughtheimplementationofevidence‐based,orevidence‐informed(i.e.,afederalagencyisreviewingtheevidencetodetermineiftheprogramcanbedesignatedasevidence‐based)servicemodels.YearoneevaluationresultsindicatedthatHighFidelityWraparoundviiiandTeam/FamilyDecisionMakingixweretwocommonlyimplementedmodels.Thisyear,CMPswereaskedtoprovideadditionalinformationabouttheirchosenevidence‐basedorevidenced‐informedmodels.

36

Communityneedsandtheuseofevidence‐basedservices

“AsaresultoftheCMPtherehasbeenaformalvenueforIOGmemberagenciestomeet,discussissuesandmakecollaborativeplanstoaddressissuesrelatedtochildren,youthandfamilieswhoarehighriskofincreasedinvolvementwiththeIOGmemberagencies.ThiscoordinationhasalsoledtoadedicationtocollaborationthatextendsbeyondtheIOGandintotheotherarenasthatouragenciesintersect.Therehavebeenmoreprotocolsestablishedthatencouragecollaborationandmulti‐agencyassessment.Thecollaborativeapproachhasalsoledtoassessingcommunityneedsanddevelopingprogrammingandservicestomeettheneedsofchildren,youth,andfamiliesthatisevidence‐basedorevidenceinformed.”

TheWraparoundmodelcontinuestobeoneofthemostcommonlyimplementedevidence‐basedISSTapproaches(reportedby20CMPs)andisdemonstratedeffectiveinintegratingservicedeliveryandenhancingquality.Table12reflectsthenumberandproportionofCMPsreportingtheuseofaWraparoundmodel,includingwhethertheyimplementedHighFidelityWraparound;whethertheirWraparoundmodelisofferedtoalloronlyselectCMPyouthorfamilies;andwhethertheycollectevaluativeinformationreflectingthedegreeofimplementationfidelity.

Table12.CMPuseofthewraparoundmodelinservicedelivery

WraparoundPractices

CMPsUsingWraparound(n=27) 20 (74%)CMPsUsingHighFidelityWraparound(n=20) 7 (35%)CMPsOfferingWraparoundtoAllChildren(n=20) 14 (70%)CMPsCollectingDataonWraparoundImplementationQuality(n=20) 7 (35%)

NumberofCMPs(%)

Asnotedabove,themajority(74%)ofCMPsreportedusingaWraparoundmodel,withaboutathird(35%)ofthoseusingthemorehighlystructuredHighFidelityWraparoundmodel.Themajority(70%)ofCMPsofferingWraparounddosoforallyouthorfamiliesserved.Ingeneral,thesix(6)CMPsnotofferingWraparoundtoallyouthorfamiliesindicatedthattheyselectyouthorfamiliesforWraparoundbasedonthecomplexityofthecase,withmorecomplexcasesreceivingpriority.WhilewidespreaduseofaWraparoundmodelreflectspositivelyonthequalityofserviceprovision,itisnoteworthythatonlyaboutone‐third(35%)ofthesecountiestrackdataonthequalityoftheirimplementationefforts(i.e.,fidelity).Additionally,nineteen(19)CMPsreportedtheuseofotherevidence‐informedprogramsandpractices,andtwo(2)reportedthatuseofsuchprogramsisapriorityforthefuture.WhentheydonotserveasthemainISSTmodel,evidence‐basedorevidence‐informedprogramsarefrequentlyofferedthroughoneoftheIOGpartneragenciestofamilieswho

37

haveparticipatedintheISSTprocess(i.e.,asasuggestedinterventionemergingfromtheintegratedplan).Consistentwithfindingsfromlastyear,CMPshighlightedtheuseofnationallyrecognizedpreventiveinterventionssuchasMulti‐SystemicTherapyxorFunctionalFamilyTherapyxi,whileothersimplementprevention‐focusedprogramssuchasLifeSkillsTrainingxiiortheIncredibleYearscurriculum.xiii

B. Appropriatenessofservices

Appropriatenessofservicesreferstothefitbetweentheservicesprovidedandtheindividualfamilybeingserved.Servicescanfailtobeappropriatewhentheydonotconsiderthefullrangeoffamilycircumstancesoraccountforfamilyinteractionswithotheragenciesandpotentiallycompetingdemands.Meaningfuleffortsintheareasoffamilyengagement,treatmentmatching,andservicecoordinationhavebeenshowninresearchtoincreaseratesoftreatmentcompletionandtoimproveoutcomes.xivAsdescribedintheyearoneevaluationreport,theadoptionofISSTsisacentralstrategyforpromotingappropriatenessofservicedeliverywithinaCMP,particularlywhentheISSTsuccessfullyengagesfamiliesininterventionplanningandserviceselectionefforts.Thisyear,CMPswereaskedtofurtherdescribeeffortstoensuretheappropriatenessofservicesfortheyouthandfamiliestheyserve.Table13belowpresentsalistofkeystrategiesbywhichserviceappropriatenessmightbeassessedandenhanced,alongwiththenumberandproportionofCMPsreportingeach.

Table13.CMPstrategiestoenhanceappropriatenessofservices

Strategy

IdentifiedserviceareathatiscurrentlynotbeingaddressedthroughinformalneedsassessmentwithCMPpartners(n=27)

24 (89%)

IdentifiedserviceareathatiscurrentlynotbeingaddressedthroughformalneedsassessmentwithCMPpartners(n=27)

16 (59%)

Implementedanewprogramormodelthatspecificallytargetsthepopulationandserviceneed/gapthatwasidentifiedbyIOG(n=27)

20 (74%)

Implementedorenhancedservicestobemoreculturallyappropriate/culturallycompetent(n=27)

11 (41%)

Other(n=25) 5 (20%)

NumberofCMPs(%)

38

Asreflectedabove,thevastmajority(24;89%)indicatedusinginformalneedsassessments,suchasdiscussionsofexistingservicegaps,asameanstoensuringserviceappropriateness.Thenextmostcommonstrategy,reportedby20CMPs,wastheimplementationofnewprogramsormodelsthatareintendedtofillanidentifiedservicegap.SpecificprogramsvaryamongCMPsbutincludedthecreationofanewISST,anewmentoringprogram,enhancedcasemanagement,andayoungparentsupportgroup.Adifferentareaoffocusreflectedeffortstoimprovetheculturalfitofservicestoidentifiedfamilies.ElevenCMPsreportedworkingtowardimprovementsinthisareaandexamplestrategiesincludedutilizationoftranslationservices,supportoflocalgroupsfocusedonculturalcompetency,andtheprovisionofculturalcompetencytrainingstostaff,eachofwhichwasreportedby3CMPs.

C. Effectivenessofservices

TheCollaborativeSuccessSurvey,describedearlier(seeAppendixBforfurtherinformation),includedaseriesofitemsrelatedtoservicequalityincludingcommitmenttotheuseofevidence‐basedorevidence‐informedmodels,improvementsinservicequalitywithinindividualpartneragencies,andeaseofaccesstoservices.AsshowninFigure5,respondentsreportedthatCMPimplementationhashadapositiveeffectontheoverallqualityofservicesprovidedtofamilies,assigninganaverageratingofgreaterthan4.5onasix‐pointscaleoneachoftheseitems.

Caseexamplesofeffortstoenhanceappropriatenessofservices

OneCMPreportedtherecentcompletionof“atwo‐yearreviewofthecontinuumofservices,includingquantitativeandqualitativeassessments,andacomprehensiveandinclusivecollaborativeprocess.Thisresultedinthedevelopmentofanewcontinuumofservicesmodelthatwillbeimplementedoverthenext2‐3years.”“[TheCMPuses]athree‐tieredmodelwhereeachtierrepresentsthelevelofreferredclient/familyriskandthenmatchthattotheappropriatemodel,levelofservice,andfundingallocation.Theideaistoimprovetheappropriatenessofservicesandtonotfloodlowriskfamilieswithtoomanyservicesandensureweareusingourincentivefundinginthemostappropriatemanner.”

39

Figure5.Effectivenessofeffortstoenhanceservicequalityandappropriateness

1 2 3 4 5 6

Improvedqualityofservices

Committedtoevidence‐based/informedmodels

Easierforfamiliestoaccessservices

Committedtoa"nowrongdoor"approach

Note:Itemsareratedona6pointscale(1=stronglydisagree,2=disagree,3=disagreemorethanagree,4=agreemorethandisagree,5=agree,6=stronglyagree).CMPscontinuetoimplementstrategiesfocusedontheidentificationandimplementationofservicesmeetinghighstandardsofquality,whilealsoensuringthatthoseservicesarewellmatchedtothestrengthsandneedsoftheyouthandfamiliesbeingserved.Additionally,theSSCandCMPportalprovideon‐goingopportunitiesforCMPstolearnaboutevidence‐basedprogramsbeingimplementedthroughoutthestate,aswellasthestrategiesothercommunitiesareusingtoensurethequality,appropriateness,andeffectivenessoftheirservices.Nonetheless,itwillbevaluableforCMPstakeholderstoexploreotherservicesandprogramsthatmaybewell‐alignedwithCMPgoals.Tothatend,evaluationactivitiesinyearthreewillincludecontinuedeffortstoidentifybestpracticesrelatedtobothcollaborationandservicedelivery,drawingupontheextantresearchliterature,aswellasCMPpracticetodate.

40

Cost‐sharing:Useofblendedandbraidedfunds

“[OurCMP]continuestoincreasethefrequencyofblended/braidedfundingforthepurchaseofspecificallyidentifiedservicesforyouth/familieswhichhasincreasedeffectivenessoftheintegratedplansforclients,aswellasincreasedthestrategicuseofmultiplefundingstreams,andreducingfragmentationofservicesbetweenagencies.Ithasalsoelevatedsomediscussionsandinformalagreementsaroundwhopaysforwhatandwhenasitrelatestoasystemicapproachratherthanagency‐specific.Thereisincreasedcooperationonprojectsbetweenpartneragencieswhenitcomestothepursuitorinvolvementingrants,whichhasalsosupportedmorecross‐systemtrainingofstaffincertainareas.”

LegislativeGoalIV:Encouragecostsharingamongserviceprovidersthatleadstocost‐reductionfortheservicesprovidedtochildrenandfamiliesinthechildwelfaresystem,includingthefostercaresystem,inthestateofColoradoThecollaborativemanagementapproachassumesthatdecreasesinserviceduplicationcombinedwitheffortstopreventdeeperandmorecostlyinvolvementinvarioussocialservicesystemswillleadtobetterfamilyoutcomesandnetcost‐savingsovertime.Infact,researchdemonstratesthatcostsaredrivenhigherbypoorlycoordinateddeliveryefforts.xvThedevelopmentofinteragencyprocessesthatsuccessfullyintegrateservicedeliveryshouldreducetheneedfor,andthecostsof,duplicatedcare.ThissectionoutlinesCMPs’approachestocost‐andresource‐sharing,andreportedperceptionsofsuccess.WefirstexploretheextenttowhichsitesallocateandshareresourcestosupportCMPoperationsandadministrativeneeds;payforexistingandnewservices;andensuresustainabilityofprograms.Wethenexaminereportedcostsavingsandre‐investmentoffunds.Thesectionconcludeswithadiscussionofthechallengesrelatedtocost‐benefitanalysisandtheinfrastructurenecessarytospecifycostmodelsacrosstheCMPinitiative.Datapresentedinthissectionwerederivedfrom2010‐2011fiscalyearannualreports.

A. Cost‐sharingapproaches

ThemanagementoffundsandthedevelopmentofeffectiveprocessestoshareoperationalandservicedeliverycostsamongagencypartnersarecoreactivitiesofIOGs.Althoughonepartneragencytypicallyservesasthefiscalagent(thisisthelocalDHSin69%ofCMPs),allocationdecisionsaremostoftenmadecollaboratively,bytheIOG(72%)oranIOGsubcommittee(24%).Specificmethodsofcost‐sharingaredescribedbelowandsummarizedbelowinTable14.

41

i.SharingcostsofCMPoperations

SitesreportedthatIOGpartners“frequently”or“always”jointlypaidforstaffingandadministrativecosts(25and20CMPs,respectively).ThisoftenisachievedthroughpoolingfundstosupportsalariesofIOGcoordinatorsorprogrampersonnel,andcontributingin‐kindresourcessuchasofficespaceandtrainings.Partneragenciesalsofrequentlysharefundinginordertodevelopnewprograms,thusexpandingservicecapacity.Table14.CostsharingamongCMPpartners

Always 12 (46%) 8 (30%) 3 (11%)Frequently 13 (50%) 12 (44%) 11 (41%)Sometimes 1 (4%) 6 (22%) 10 (37%)Rarely 0 (0%) 1 (4%) 1 (4%)Never 0 (0%) 0 (0%) 2 (7%)

FrequencyStaffingCosts

(n=26)

AdministrativeCosts(n=27)

CostsofNewPrograms(n=27)

ExtenttoWhichCMPPartnersShareCostsNumberofCMPs(%)

ii.Sharingcostsofservices

CMPsemphasizedthatcost‐andresource‐sharingisanintegralpartoftheISSTprocess.Asnotedearlier,themajorityofCMPsindicatedthatduringISSTmeetings,agenciesjointlyagreeonwhowillpayforservicesproposedintheinterventionplan.Agenciesmayagreetosharethecostofanexpensiveservice,ornegotiatetohaveeachagencypayforseparateservices.Asacomplementtothesecost‐sharingprocesses,someCMPsallocateportionsoftheiryearlyincentivefundingtopayforinterventionservices.Theseventeen(17)CMPswithavailabledatareportedspendingnearly$714,000onservicesprovideddirectlytochildrenorfamilies,whichrepresented29%oftheir2010‐11fiscalyearincentivefunds.

iii.Ensuringsustainabilityofservices

ManyCMPsareworkingtoexpandfundingandpromoteprogramsustainability.Toachievethis,IOGpartneragenciesoftensharegrantwritingresourcesinpursuitofexternalgrantopportunities.Inthe2010‐2011fiscalyear,10ofthe29CMPs(34%)reportedreceivingcompetitivefundsfromfederalagencies,stateagenciesandblockgrants,andprivatefoundations(seeAppendixCfordetailedinformation).Together,theseeffortsgarneredatleast$1,084,776inadditionalfunding.3CMPsoftenaresuccessfulintheseendeavorsbecauseoftheirstrongcollaborativerelationships.Theexistingpartnershipsareattractivetofundersbecausethefoundationneededforeffectiveimplementationis

3 This total is underestimated due to missing data on total amounts received.

42

alreadypresent.TheinfrastructurebuiltbytheCMPinitiativeisrecognizedasbeingakeymechanismtoimplementprogrammingacrossColorado.Forexample,CMPcommunitiesareafocusofastatewideplanninginitiativetoensurethatcountiesembracetheSystemsofCareapproachtoservingchildrenandfamilieswithseriousemotionaldisturbances.

B. Costreductions,potentialcostsavings,andreinvestmentoffunds

Anunderlyingassumptionoftheinitiativeisthatcollaborativeeffortswillleadtocostreductionsandthuscostsavings,whichmaybere‐investedinlocalCMPactivities.However,precisemeasurementofthesecostcomponentsisdifficult,asitrequiresacalculatedestimateofthepotentialcostsforagivenindividualorfamilyhadCMPeffortsnotbeenprovided.Whileoutcomesmaybeimproved,itisdifficulttodeterminewhetherthesewereachievedatlowercost.Moreoverthelackofstandardizedoutcomesandmeasurementprotocolsacrossprojects,asevidencedinthefirsttwoyearsoftheevaluation,preventstheassignmentofpotentialsavingsestimates.

Inthe2010‐2011fiscalyear,onlythreeCMPsreportedhavingaformalprocesstomeasureactualcostsavings.Twelve(12)providedroughestimatesofsavingsbasedupontheirperceptionsoftheextenttowhichservicesmayhaveledtoreductionsinthenumberofchildren,andtheamountoftimespentinout‐of‐homeplacements,includingfostercare,therapeuticresidentialandpsychiatriccare,andcommitmentanddetention.ThesetwelveCMPsreportedatotalof$5.9millionofsavingsthatwerereinvestedinCMPefforts.

CMPsdescribedothersourcesofperceivedcostreductions,butwereunabletoquantifytheassociatedcostsavings.Theseinclude:

Fewerfamiliesenteringthechildwelfaresystem Reductionsintruancyrates(leadingtopresumedlowercostsineducation,child

welfare,andjuvenilejusticeprograms,andincreasedrevenuesforschools) Reductionsinthelengthofmentalhealthtreatmentandinpatientstays Improvementsincoordinationandaccesstoneededcareandfundingsourcessuch

asMedicaidandCHP+ Successfulpreventionofcostlyservicesthroughtargetedprograms,including

pregnancyprevention.

43

ArecommendationfromtheyearoneevaluationwastoworktowardmethodstotrackcostsofCMPservicesmoresystematically.Inthecurrentyear,CMPsbasedestimatesofanISSTstaffingontimespentbyprovidersinanaveragemeeting.Amongthe21CMPsthatrespondedtothisitem,theaveragecostwas$323,witharangeof$110to$800.Thelargerangeofestimatedservicecostsistobeexpected,giventhediversityofISSTmodelsandservices.Thatis,someCMPsstructuretheirISSTssuchthatbrief,frequentmeetingswithsmallernumbersofparticipatingprovidersareheld,whileothersmayholdonelonginitialISSTplanningmeetingwithalargernumbersofproviders.Somesitesindicatedthatitwasdifficultorunworkabletocalculateameaningfulaveragecost,giventhattheirserviceplanningmeetingsareadaptedtotheneedofeachfamily.

i.ReinvestmentoffundsTotheextentthatCMPsdoexperiencecost‐savings,thelegislationrequiresthemtore‐investthosefundsintoCMP‐relatedoperations.Twenty‐one(84%)ofthe25CMPsreportingthesedatainthe2010‐2011fiscalyeardescribedtheiractualorplannedreinvestmentoffunds.Asreported,fundswerefrequentlyreinvestedintheprovisionofadditionalsupporttoexistingornewservices(16CMPs),tosupportcoordinatorsalaries(4CMPs),andtobuildneededinfrastructure(e.g.,databasedevelopment;3CMPs).Otherlessfrequentlymentionedusesincludedcreating“emergency”accountstoassistfamiliesinpayingfordirectservicesorimmediateneedsandpayingforexternallocalevaluationservicestoassessperformanceonkeyoutcomes.

C. Challengesrelatedtomeasuringcostsandbenefitsattheinitiativelevel

Testingthecost‐savingshypothesisassumedinlegislationiscomplicatedgiventherangeanddiversityofexistingprojects,processesandoutcomes.Below,weoutlinekeydifficultiesassociatedwithquantifyingcostsandbenefitsforCMPefforts.Wethendescribe

Localhighlight:MeasurementofcostreductionsOneCMPestimatescostsavingsusingapre‐andpost‐servicedesign.Theycollectdataretrospectivelyonactualcostsofservicesincurredbyfamiliesduringtheone‐yearperiodpriortoenrollmentintheirISST.BaselinecostsincludeanyDHSCoreServices,mentalhealthtreatmentcoveredbyMedicaid,DYCdetention,andanydaytreatmentcosts.ThetotalbaselinecostsarecomparedwiththeactualcostofimplementingtheirWraparoundmodelservices.ThisCMPreportedestimatedsavingsof$236,000thisyear.ThismethodisnoteworthyasitreliesuponactualcostsamongfamiliesservedbytheCMP;thus,totalsavings,whilestillanestimate,couldbereasonablyattributedtoCMPservicedelivery.

44

thesupportsneededtomovetowardthespecificationofoneormorecostmodelsfortheinitiativeinthefuture.Initssimplestform,cost‐benefitanalysissumsthevalueofthebenefitsaccruingfromasetofactionsandthensubtractsfromthis,thesumofthecostsassociatedwiththoseactions.Thus,inordertoconductacost‐benefitanalysis,itisnecessarytoassignmonetaryfigurestobothsidesofthisbasicequation:costsandbenefits.ThisischallengingfortheCMPinitiativeforavarietyofreasons.First,itisdifficulttoassignmonetaryvaluesonthecostsideoftheequationasmostCMPeffortsdonothavesufficientlyclearprogrammodelsand/orlacktherequiredlevelofprecisionneededtospecifycosts.Thatis,modelshavenotbecomestandardenoughorimplementedwithsufficientfidelitytoenableaccuratecostassignmenttoefforts.Becauseitisalsocriticaltoassignastandardcostregimenacrossmultiplesites,thisvariabilityproblemiscompoundedwhenseveralCMPsareconsideredforanalysis.Thebenefitssideissomewhatmorecomplicated.ItisassumedthatCMPeffortsresultinoutcomesthatareachievedatlowercostbecausewithouttheCMP,serviceswouldhavebeenmoreexpensiveduetoserviceduplication,fragmentation,andaprolongedneedforcostlyservices.Inordertoassignmonetaryvaluestothesebenefits,onemusthypothesizewhatwouldhavehappenedwithoutCMPservices;thatis,whatthelikelytrajectoryofnegativeoutcomesfromagivensetofproblemsmighthavebeenwithoutCMPinterventionandwhattheassociatedcostswouldhavebeen.IfthetotaloftheseestimatedcostsexceedsthetotalcostofCMPcollaborativeprocessesplusprovidedservices,thentheCMPwouldbeassumedtohavehadanetfinancialbenefit.However,specificationofthesevarioushypotheticalcostsisverydifficultbecausewedonothavereliablemethodsofpredictinglong‐termchildandfamilyoutcomes.Shouldtheinitiativeseektodevelopacost‐benefitmodelinthefuture,manyofthebarriersabovewouldneedtobeaddressedthroughthefollowing:

greateradoptionofapreciseandstandardprogrammodel increasedimplementationfidelity agreementoncorrespondingoutcomes

However,beforemovinginthisdirection,theinitiativewouldneedtoconsidersomeimportanttrade‐offs.Specifically,theCMPeffortplacesahighvalueonlocaldecisionmakingandthediversityofapproachesisoneofitsstrengths.Whilethepromiseofarigorouscost‐benefitapproachmayseemliketheidealevaluationmodel,implementing

45

requiredchangeswouldnecessitategreaterstandardizationacrosssites,andmaycomeattheexpenseoflocalinvestmentandsupportfortheCMP.TheplantouniformlymeasureaselectsetofservicedeliveryandoutcomeindicatorsacrossallCMPsinthe2011‐2012fiscalyear(mentionedaboveanddiscussedindetailinthefollowingsection)movestheinitiativeastepclosertomeasuringcostsandbenefits.Withcontinuedevaluationeffortsinthisarea,itishopedthatovertime,therewillbeanopportunitytospecifyacost‐benefitmodelandconductrelatedanalyses.Insummary,theevaluationrevealedthatlocalpartneragencieshavedevelopedclearprocessestosharecostsofoperationsandservices.Inmanycases,localCMPshaveeffectivelyleveragedtheircollaborativeeffortstoobtainadditionalfundingtosustainand/orgrowtheirprogramsandservices.However,itisdifficulttoascertaintheextenttowhichCMPstructuresandprocesseshaveledtocostreductionsandcostsavings,giventhedifficultiesinmeasurement.Onesitewashighlightedfortheirmethodofaccountingforpre‐andpost‐interventionservicecostsindeterminingcostsavings.However,thelocalresourcesneededtocollectretrospectiveservicedatamaybeprohibitiveforCMPsservinglargenumbersofchildrenandfamilies.Thus,whilerepresentingapromisingstrategytomorepreciselyassesscostsavingsatthelocallevel,thismethodmaynotbefeasibletoapplyacrossallCMPs.Animportantnextstepfortheinitiativewillbetodevelopfeasiblemethodstomorespecificallyassesscosts(e.g.,costtemplates).Inaddition,evaluationactivitiesinyearthreewillexplorethedevelopmentofuniformmethodsofmeasuringcostsavingsassociatedwithspecificCMPprocessesand,totheextentpossible,specificoutcomes.

46

LegislativeGoalV:Leadtobetteroutcomesandcost‐reductionfortheservicesprovidedtochildrenandfamiliesinthechildwelfaresystem,includingthefostercaresystem,inthestateofColorado.ThefifthandfinalgoaloftheCMPinitiativeistoimproveoutcomesforchildrenandfamilieswhorequireservicesfrommultipleagencies.Itassumesthatinteragencysystemimprovementsresultinginmoreefficientandeffectiveservicedeliveryalsoshouldleadtomoresuccessfulandpermanentoutcomesforchildrenandfamilies.ThissectionfirstdescribesthecomplexnatureofCMPservicepopulationdefinitionsandreportsestimatesofthenumberofchildreneligiblefor,andservedby,CMPs.Wethenexplorewhattheevaluationrevealedaboutdefiningandmeasuringoutcomes,anddiscusscurrentlimitationsinassessingthestatewideimpactoftheCMP.WepresentfindingsfromlocalCMPperformancemeasuresduringthe2010‐2011fiscalyearand,whereappropriate,discusstheseinlightoffindingsfromlastyear.Thissectionconcludeswithadetailedoutlineoftheupdatedevaluationplanwhichhelpstoaddresssomeoftheidentifiedlimitations.Datasourcesforthissectionincludemid‐yearandannualreports.

A. DefiningCMPtargetpopulations

Currently,itisdifficulttoobtainprecisenumbersofindividualsandfamilieswhoareeligiblefor,andservedbyCMPs,bothwithinandacrosssites.Accordingto24‐1.9‐102(2)(c)CRS,eachMemorandumofUnderstandingissupposedtoincludeafunctionaldefinitionof“childrenandfamilieswhowouldbenefitfromintegratedmulti‐agencyservices."However,thereisnostandarddefinitionofeligibilityforCMPserviceefforts.Asaresult,CMPshavespecifiedvaryingdefinitionsoftheireligibleandservedpopulations,whichrangeinbothbreadthandspecificity.Examplesinclude:allchildrenwithinthecounty;allchildrenwithopencasesintheircounty’schildwelfaresystem;allchildreneligibleforservicesinanyoftheirpartneragencies;allchildrenreceivingservicesinatleasttwopartneragencies;andallchildren“atrisk”forinvolvementwithpartneragencies.Additionally,methodsusedtoidentify,countandmanageinformationonyouthandfamiliesalsovary.FewCMPs,forexample,havedatamanagementsystemsthattrackindividualsorfamiliesservedthroughtheprogram.Further,manyCMPsrelyoncounty‐,district‐,orstate‐leveldatabasesorpartneragencydatasystemstoobtaincounts;however,thesecountsoftenincludelargerpopulationsthanthoseservedthroughlocalCMPefforts.Finally,someCMPscountindividuals,otherscountfamiliesandsomecountbothastheirunitsofmeasurement,andcountsmaybeunduplicated(eachindividualorfamilycountedonce)orduplicated(eachindividualorfamilyrepresentedmorethanonce)

47

inthetotal.Thevariationinhowindividualsandfamiliesaredefinedandcountedatthelocallevelmakesitdifficulttodeterminetheextenttowhichlocalprojectsareservingthetargetpopulationasoutlinedinthelegislation(e.g.,childrenandfamiliesinvolvedinmultiplesystemsofcare).Takentogether,thepresentcountsofeligibleandservedpopulationsdiscussedbelowshouldbeinterpretedwithcautionandarelikelyoverestimated.

i. EstimatesofCMPeligibleandscreenedpopulation

Inthe2010‐2011fiscalyear,therewasanestimated70,000individualsconsideredeligibleforCMPservicesandapproximatelyhalf(30,000)ofthatpopulationwasreportedlyscreenedforservicesacrossallCMPs.CMPsestimatedthefollowingnumbersoftotalindividualsserved,calculatedatdifferentlevelsofservicedelivery(allincludeduplicatedcases):

19,600individualswhoparticipatedinlocalservicesoreffortsthatweredirectlyassociatedwiththeirCMP.TheseeffortsreflecttotalsfromallCMPsandcouldincludeservicespaidbyCMPincentivefunds,servicesdonatedin‐kindorprovidedbypartneringagencies,allthoseservedbyISSTs,andanylarge‐scalepreventionorinterventioneffortsthatweresupportedbytheirIOG.

7,000individualswhoreceivedservicesthroughthelocalISSTprocess.Thesereflecttheindividualswhoparticipatedinamulti‐agencyproviderstaffingmeetingsacrossallCMPs.

5,300individualswhoreceivedservicesthatwerefunded,atleastinpart,byCMPincentivefunds.Thistotalnumberwascalculatedacross19CMPswhereindividualsparticipatedinprogramswhichtheIOGfinanciallysupportedwithincentivefunds.FourCMPsindicatedthatnoindividualsreceivedservicessupportedbyincentivefunding(i.e.,partneragenciesprovidedin‐kindservices),andanotherfourhadnotyetreceivedincentivefunding.

B. DefiningCMPoutcomes

Eachyear,localIOGpartnersdescribetheirplantoassessprogressonkeyoutcomesintheirMOUs.Theyspecifycomponentsoftheirmeasurementplan,whicharedefinedbelow:

Targetedoutcomedomain.LocalCMPsarerequiredtoassessperformanceinthefouroutcomedomains(i.e.,childwelfare,juvenilejustice,education,andhealth/mentalhealth).Sitesmayalsoassessperformanceinotherdomainssuchasdomesticviolence.

48

Targetedoutcomes.CMPsindicatewhatgeneraloutcomestheyaremeasuringwithinthesedomains(e.g.,reducetruancy,increasestabilityofout‐of‐homeplacements).

Performanceindicators/measuresandgoals.Foreachoutcome,sitesdefinetheindicator(specificmeasure)thattheyemploytoassessprogressontheoutcomeandstatetheirannualtargetgoal(e.g.,increasetherateofCMP‐servedyouthwhoremainintheirhomesafterreceivingCMPservicesby2%).

Atthebeginningoftheinitiative,theSteeringCommitteespecifiedalistofapproximately20outcomesreflectingkeygoalsoftheCMP(e.g.,reduceout‐of‐homeplacements,reducerecidivism).Overthepastsixyears,however,thenumberandvarietyofoutcomesandrelatedperformanceindicatorsassociatedwiththeseoutcomesexpandedgreatly.Aspartoftheformativeevaluation,outcomesandperformanceindicatorswereexaminedeachyeartoassesstheextentofstandardizationacrossCMPs,aswellassimilaritiesinmeasurementanddatasources.ItwasdeterminedinyearoneoftheevaluationthatCMPsreportedonover150differentperformanceindicatorsandgoals.Inthesecondyear,thereviewrevealedthefollowing:

Numberofperformanceindicators/goals.Onaverage,eachCMPassessed7goals,althoughafewCMPsmeasuredtwentyormore.

Goalsbydomain.The28CMPswithatleastoneyearofdataproposed198performancegoals,distributedalmostevenlyinchildwelfare(49),juvenilejustice(49),education(49),andhealth/mentalhealthdomains(42).Ninegoalswereinotherareas,includingdomesticviolence.

Processversuspopulation‐levelindicators.Two‐thirds(134)ofstatedgoalsinvolvedpopulation‐levelindicators(e.g.,assessingchildandfamilyoutcomes),while64involvedprocessindicators(e.g.,improvementsinservicedelivery).

Definitionsofperformanceindicators.CMPsvarygreatlyintheirdefinedindicators.Despitesomecommonoutcomeareas,localCMPsmeasureverydifferentaspectsoftheseoutcomes,whichprecludedanyaggregateanalysisofeffects.

Populationlevel.Performanceismeasuredatdifferentpopulationlevels(i.e.,rateforentirecountyversusyouthservedthroughtheISST).

Datasources.CommondatasourcesincludetheCDHSTrailsdatabase,Eclipse/ICONstatejudicialdatabase,localschooldistrictdatabases,andDBHandColoradoClientAssessmentRecords(seeAppendixDforafurtherdescriptionofdatasources).

Thesesummaryfindingsshowthatingeneral,localsitestendtotracktheirperformanceinmanyoutcomes,utilizingmultipleindicatorswithinoutcomes.Thisisexpectedtosome

49

degree,giventhatCMPsemployavarietyofservicemodelsthataretargetedtolocalcommunityneeds,whicharediverse.Awide‐rangingmeasurementapproachallowsCMPstodemonstrateachievementsonseveralfronts.However,itmayalsocomeatacost,iflocalsites’attentionsandresourcesaredividedintermsofworkingontoomanyfrontsoroutcomesatonce.ThemoreCMPsarefocusedonasmallsetofcentraltargetgoalsanddedicatingresourcestodevelopingstrongindicatorsofthesegoals,theeasieritistodemonstrateaclearandpositiveimpactofefforts.Inaddition,with198differentperformanceindicators,allwithvaryingdefinitionsandmethodsofmeasurement,itiscurrentlyimpossibletoaggregateindicatorstoassessimpactacrossmultipleCMPsoratthestatelevel.However,tofacilitatedevelopmentofasummaryofachievementsacrosstheCMPinitiative,performanceindicatorswerecategorizedintothetenmostfrequentlyassessed(common)outcomes.Theseoutcomes,examplesofrelatedindicators,andthenumberofCMPsselectingtheseinyearsoneandtwoarepresentedinTable15onthefollowingpage.

50

Table15.Thetenmostcommonlyassessedoutcomes

Domain CommonOutcome ExampleIndicatorsNumberofCMPsassessingoutcomeFY09‐10 FY10‐11

ChildWelfare

Enhancestabilityofout‐of‐homeplacements

Reductionsinthenumberofmovesandlengthoftimethatchildreninout‐of‐homecaresettingsexperience

6 10

Preventout‐of‐homeplacement/increasereunification

Reductionsintherateofout‐of‐homeplacement,numberofchildrenservedthroughtheCMPwhosuccessfullyremainintheirhomes,reductionsintheaveragelengthofstayinfostercare

13 13

JuvenileJustice

Successfulcompletionofprobation

Increasingpositivedischarges,decreasingprobationrevocations,andincreasingthenumberofyouthnotviolatingcourtorders

17 16

Loworreducedusageofcommitment/detentionfacilities

MaintainingalowrateofAverageDailyPlacement(ADP)indetentionandlowratesoflong‐termplacement

5 7

Reduceormaintainlowratesofre‐offense/recidivism

Reducingthenumberofyouththatreoffendorareadjudicateddelinquent,orincreasingthenumberofyouthnotincurringnewchargeswhilereceivingCMPservices

6 8

Education

Increaseattendance/reducetruancy

Increasingyouthattendanceanddecreasingtruancyrates

18 25

Increaseorimprovestudentachievement

Improvementsinreadingscores,grades,andstandardachievementscores

7 4

Health/MentalHealth

Improvedleveloffunctioninganddecreaseinproblemseverity

Improvementsintheleveloffunctioningorproblemseverityofindividualsreceivingmentalhealthservices

12 14

Increasepreventionandtreatmentforsubstanceuse/abuse

Improvementinnumberofyouthremainingintreatmentprogramsforaspecificperiodoftime

5 1

Decreaseormaintainlowratesofhospitalization/inpatientservices

Lowerratesofinpatientcareandtherapeuticresidentialchildcarefosterplacementsandfewercasesofadmittingyouthintohigherlevelsofcare

3 5

i. Overallperformance

Wefirstexaminedthepercentofallperformancegoalsmeasuredacrosstheinitiative.CMPsreportedmeeting78%ofthe198totalperformancegoals.AsshowninFigure6below,successinmeetinggoalswashighinthreeofthefourdomains,withlesssuccessintheeducationdomain.

51

Figure6.Percentageoftotalperformancegoalsmet,bydomain

Note:N=198

ii. Performanceintencommonoutcomeareas

Inordertoexaminesummaryachievementsinspecificoutcomeareas,wedetailperformanceonthe103indicatorsreflectingthetencommonoutcomes,among26CMPsthathaveparticipatedintheinitiativeforatleasttwoyearsthusenablingassessmentofprogressfromFY2009‐10toFY2010‐11(seeTable16below).

Table16.Percentofperformancegoalsmet

Enhancestabilityofout‐of‐homeplacements 88%Preventout‐of‐homeplacement/increasereunification

80%

Successfulcompletionofprobation 69%Loworreducedusageofcommitment/detentionfacilities

86%

Reduceormaintainlowratesofre‐offense/recidivism

88%

Increaseattendance/reducetruancy 44%Increaseorimprovestudentachievement 60%Improvedleveloffunctioninganddecreaseinproblemseverity

60%

Increasepreventionandtreatmentforsubstanceuse/abuse*

100%

Decreaseormaintainlowratesofhospitalization/inpatientservices

57%

Health/MentalHealth

%GoalsMetFY2010‐11

Domain CommonOutcome

ChildWelfare

JuvenileJustice

Education

*Note:OnlyoneperformancemeasurewasassessedinthisoutcomeareainFY10‐11.

52

Improvingyouthoutcomes

“Themostsignificantchangeinoursystemhasbeentheeffectivecollaborationofpartneragencies.Thetruebeneficiariesofthishasbeenouryouth.Largelyduetothepartnership,wehavereducedthenumberofyouthcommittedtotheDivisionofYouthCorrections,we'vebeenabletoreducethenumberofyouthonprobation,placedinmentalhealthhospitalsorTRCCFlevelplacements.Theseyoutharebeingmaintainedintheirowncommunitiesandwiththeirrespectivefamilies.”

Ingeneral,themajorityoftheperformancegoalstargetingoutcomesinthechildwelfareandjuvenilejusticedomainsweremet,withmorevariableperformanceonoutcomesineducationandhealth/mentalhealthdomains.Afewinterestingdifferencesinproposedmeasurementandperformancebetweenthetwoevaluationyearswerenoted:

Preventout‐of‐homeplacements/increasereunification.Thenumberofproposedgoalsforthisoutcomegrewfrom15goalsinthe2009‐2010fiscalyearto21goalsinthecurrent(2010‐2011)year,suggestingthatthismaybeanareaofgrowinginterestandfocusamonglocalCMPs.

Successfulcompletionorterminationofprobation.CMPsmayhavetargetedadditionaleffortsinthisareaoverthepastyear,asalargershareofthesegoalswasmetthisyearthanlast(69%and44%,respectively).However,thesignificanceofthisshiftisuncertain,aslocalCMPsmayhavesetgoalswithlowerexpectationsinresponsetolastyear’spoorerperformance.

Increaseattendance/reducetruancy.Moreperformancegoals(25)weremeasuredinthisareacomparedtolastyear(18),suggestingthattheremaybegrowinginterestand/orneedtoprovidemulti‐agencyintegratedservicesinthisarea.

Increasepreventionandtreatmentforsubstanceuse/abuse.OnlyoneCMPmeasuredagoalinthisareathisyear,ascomparedtothefiveCMPsthatdidsointhepreviousyear.Thisdifferencemayhaveresultedfromchallengeswithdatacollection,particularlyinrelationtotimelycalculationofoutcomesforannualreporting.

Insummary,CMPeffortsresultedinreportedimprovementsinlocally‐definedperformanceindicatorsduringthefiscalyear.CMPsmetapproximatelythree‐quartersoftheirproposedgoals,thoughsuccessratesvariedsomewhatbyoutcome.AlthoughCMPsreporteddatareflectingtencommonoutcomes,thetargetpopulationsandspecificperformanceindicatorsforthoseoutcomesweredefinedandmeasureddifferentlyacrossCMPs.Thus,datacouldnotbemeaningfullyanalyzedacrosssitesbecausedirectcomparisonisneitherpossiblenorappropriate.Asdiscussedabove,issuesrelatedtooutcomemeasurementandanalysisledtothecreationofanOutcomesWorkgroup,discussedbelow.

53

C. Planformeasurementofstandardoutcomesinevaluationyear3

ThenewstatewideevaluationmeasurementplanwascollaborativelydevelopedbytheOutcomesWorkgroup,usingtheprocessoutlinedintheevaluationmethodssection(page9)andfurtherdetailedinacomprehensivereport(seeAppendixBforalinktothefullreport).ThemeasurementplanwasadoptedbytheCMPStateSteeringCommitteeinMarchof2011andimplementationbeganinJuly2011.TheplancentersonthecollectionandanalysisofaselectsetofindicatorsmeasureduniformlyacrossCMPsandaggregatedatthestatelevel.TheseindicatorsassesscoreCMPprocessesandoutcomes,includingthenumberofchildrenserved,theextentofserviceintegrationacrossagencies,andimprovementsinthefunctioningofchildrenandfamiliesservedbytheCMP.Additionalinformationaboutthespecificindicators,datacollectionprocesses,selectedoutcomes,anddataanalysisassociatedwiththisplanispresentedbelow.

i. Processmeasuresandmethodofdatacollection

ProcessindicatorswereselectedtoobtainanaccuratecountanddescriptionoftheCMPtargetpopulationandtotrackprogressoncoreservicecomponents.Theseprocessindicatorsare:

Numberofchildren/familiesreferredtoISSTs Numberandtypeofagenciesorsystemsthat:

o Familiesareinvolvedinatthetimeofenrollmento Participateinthefamily’sserviceplanningo ProvideservicestofamiliesasaresultoftheISST

Numberofcaseswherefamilymembersparticipatedinserviceplanning Numberofcasesinwhichanintegratedserviceplanwasdeveloped Numberandtypeofoutcomes,amongaselectionofoutcomesinthechildwelfare

andjuvenilejusticedomains,thatarealignedwithISST‐servedcasesAtwopagedatacollectionformtocollectitemsneededtomeasuretheabovelistedindicatorswascreated.Theformalsoincludesclientidentifyinganddemographicinformationandstateidentifiernumbersfromtwostatewidedatasystems(Trails,ICON/Eclipse).Tosupportlocal‐andstate‐leveldatatrackingandanalysis,anon‐lineclient‐leveldatabasesupportedbyEffortstoOutcomes(ETO)©softwarewasdevelopedspecificallyfortheinitiative.ETOwasselectedforanumberofreasons:itisawell‐establishedsocialservicedatatrackingsystem;itisflexiblesothatadditionaldataitemscanbeeasilyintegrated;andcustomreportscanbecreatedsothatlocalCMPscantrackandreportontheirserviceinformationthroughouttheyear.EachCMPhasestablishedprocessestocollectthedataon

54

childrenandfamiliesparticipatingininitialISSTmeetings.MostsiteswillenterthesedataintotheETOsystem;theotherswilltrackthesedataitemsintheirlocaldatabaseandprovideOMNIwiththedatafile.OMNIisprovidingtrainingandtechnicalassistanceforthesedatacollectionsystemsandwillbeconductingqualitycontrolchecks.

ii. Outcomesmeasuresandmethodofdatacollection

TheOutcomesWorkgroupidentifiedasetofindicatorsthatreflectcoreexpectedoutcomesoftheCMPinitiativeinthechildwelfareandjuvenilejusticedomains.Indicatorsinthesetwodomainswereselectedfortheinitialrolloutofthenewplan,giventhattheyarecurrentlycollectedthroughexistingstatewidedatabases(TrailsandICON/Eclipse).AlloftheseindicatorswillbemeasuredamongyouthwhoparticipateinISSTs.Theselectedsetofindicatorsis:

Number(rate)withnewopeninvolvementsinTrailsafterCMPservicesbegan Number(rate)withnosubstantiatedabusefindingafterCMPservicesbegan Numberofmovesexperiencedwheninout‐of‐homeplacement Number(rate)dischargedtoapermanenthome Number(rate)whosuccessfullycompleteprobationand/orparole Number(rate)ofrevocationsbytechnicalviolationswherecaseresultedin

unsuccessfultermination Number(rate)whorecidivate,resultinginunsuccessfulcompletionofprobation

and/orparoleCMPsindicatewhichoftheabove‐listedoutcomesalignwitheachcase(i.e.,whichreflectinterventiongoalsorrepresentpotentialoutcomesrelevanttothechildbeingserved).ThisinformationwillbeusedtoidentifysubsetsofCMP‐servedchildrentoincludeintheanalysisofeachoutcome.Intheupcomingyear,inconjunctionwiththeSSCandEC,acollaborativeprocesswillbedevelopedtocreateastandardmeasurementapproachforindicatorsintheeducationandhealth/mentalhealthdomains.

iii. Proposedanalysis

OMNIevaluatorshaveestablishedprocesseswithresearchstaffinthestatechildwelfareandjudicialagenciestoconductaggregatedanalysesoftheselectedoutcomes.Usingthestateidentifiernumbersandclientidentifyinginformation,researchstaffattheseagencieswillmatchclientinformationwithoutcomedataextractedfromTrailsandICON/Eclipsesystems.Theywillthencalculateoutcomes,aggregatedatlocalandstatelevels,and

55

providetheseresultstoOMNI.OMNIwillthenanalyzetheprocessdatacollectedthroughETOorextractedfromexistinglocaldatabases.AnalysisofprocessdatawillresultinanaccountingofthetotalnumberofchildrenservedthroughISSTs,arrayofreferralsources,extentandpatternsofmulti‐systeminvolvement,familiesinvolvedinserviceplanning,andthetypesofoutcomesISSTservicesareattemptingtoachieve.OMNIwillsharerelevantresultswitheachlocalCMPandinthestatewideevaluationreport.Thestatewidemeasurementplanisonecomponentofamulti‐methodapproachtoevaluatingtheCMPinitiativeandreportingonmandatedlegislativegoals,asrequiredbythelegislation.Otherdatacollectionactivities,includingmid‐yearandend‐of‐yearreportingandadministrationofsurveys,willcontinueinordertoinformtheevaluationinotherkeyareasandtocomplywithotherlegislativerequirements.Withtheestablishmentofthestatewidedatamanagementsystemandclient‐trackingprocesses,thereistheopportunitymovingforwardfortheinitiativetoidentifyadditionalrelevantstandardoutcomestotrackovertime.OutcomesthataretargetsofstatewideeffortscouldbeassessedamongCMP‐servedyouthinordertoexploretheimpactofthecollaborativeapproach.Forexample,recommendationsinanOctober2011AnnieE.CaseyFoundationreportsuggestthattheColoradoDepartmentofHumanServicesfocuseffortsonspecificoutcomes,includinguseofcongregatecare,family‐basedplacements,andre‐abuserates.TheseoutcomesalignwithCMPtargetsandgoals,andcouldbeintegratedintothestatewidemeasurementplan.

56

YearthreeevaluationdirectionsInthefirstyearofevaluation,quantitativeandqualitativeresearchmethodswereemployedtogatherabaselineunderstandingofCMPcorecomponents,structures,andprocesses(i.e.,phaseIformativeevaluation).Theresultingfindingsinformedaseriesofrecommendationsthatwereinitiatedinyeartwo.Principally,thesecenteredaroundtwoareas:1)identifystandarddatacollectionopportunitiesacrossCMPsandestablishmeasurementsystemstosupporttheseefforts;and2)continuetoinformCMPbestpracticesandprogrammodelsthroughongoingprocessandformativeevaluationefforts.EvaluationactivitiesplannedforthethirdyearareintendedtoincreasecapacityandenhancedataqualityinwaysthatwillallowforanexaminationofCMPoutcomes(i.e.,phaseIIIsummativeevaluation).Theshiftawayfromformativeevaluationandinfrastructurebuildingtoasummativedesignwillrequirelocaladjustmentsindatacollectionpracticestoensureconsistentmeasurementstatewide.Thismaybechallengingtosomeprojects,andOMNI,CDHS,andtheEvaluationSubcommitteearecommittedtosupportingCMPsinmakingthesechanges.ThefollowinginformationoutlinesworkthatwilloccurinthethirdyearoftheevaluationtosupportmorerigorousmeasurementofCMPoutcomes.A. Continuetodevelopandimplementsystemstocollectandanalyzeuniform

measuresofCMPoutcomes,outputs,andprocesses1. Continueeffortstofocusoutcomemeasurementonasmallsetofwelldefined,uniformly

measuredoutcomesacrossCMPsonCMP‐targetedpopulations.Thenewstatewidemeasurementsystemimplementedforthe2011‐2012fiscalyearfacilitatestheanalysisofstandardoutcomesinthechildwelfareandjuvenilejusticedomains.ThesedataaremeasuredonchildrenandfamiliesservedbylocalISSTs,enablingustodrawadirectlinkbetweenprogressonoutcomesovertimeandCMPprograms.Inthe2011‐2012fiscalyear,mechanismstomeasurestandardoutcomesintheeducationandhealth/mentalhealthdomainwillbecollaborativelydeveloped,andimplementedinfiscalyear2012‐2013.

2. Continuetodevelopstandardoutputandprocessmeasuresandmeasurementapproachestosupportimproveddatacollectionandanalysisofprogressonlegislativegoals,atthestatewideandlocallevels.Thesetofoutputandprocessindicatorsmeasuredinthenewstatewidemeasurementsystemdescribedinthesectionaboveinformeffortstoassessreductionsinduplicationandfragmentationandtheextentoffamilyinvolvementin

57

serviceplanning.Inaddition,theevaluationwillcontinuetoassesscollaborativeeffectiveness,throughstakeholdersurveysandannualreporting.Intheupcomingyear,OMNIwillworkwithlocalCMPstofurtherrefineanddevelopadditionalmeasurementstrategiesintheseareas,aswellasimprovementsinquality,effectiveness,andappropriatenessofservices.Specifically,theon‐linestatewidemeasurementdatacollectionsystemprovidesastructureandmechanismbywhichlocalCMPscoulddevelopadditionaluniformmeasuresofCMPprocesses.Thesecouldincludethetrackingoffamilyengagementandsatisfactionwithservices,typesofinterventionservices,andotheraspectsofservicedelivery.

3. Developstandardmeasuresofcostandcost‐savingstobeimplementedacrossCMPs.

Formativeevaluationeffortshaverevealedsignificantdifficultiesinmeasuringlocalcosts,costreductions,andpotentialcostsavings.Utilizingtheinformationgathered,aswellasevidenceandexamplesfromthefield,OMNI’sevaluationteamwillworktoidentifycommonCMPcomponentsforwhichcostscanbeassigned,anddevelopasetofstandardizedmeasurestoassesscostsandcostreductionsassociatedwithCMPservicedelivery.MeasureswillberefinedincollaborationwiththeECandCMPcoordinatorsandthenimplementedacrossCMPsinfutureyears.

4. MonitorandtracktrendsinstatewidemeasuresofkeyCMPoutcomes.OMNI,inpartnershipwiththeEC,willdevelopanon‐lineplatformtodisplayrelevantdatainordertobringawarenesstoCMPinitiativeeffortsamongleadersandcommunitymembersthroughoutthestate.Thison‐linedatadisplaywillenablemonitoringoftrendsovertimeinthefourCMPoutcomedomains(childwelfare,juvenilejustice,education,andhealth/mentalhealth)atthelocal‐andstate‐level.

B. ContinuetodefineandsupportdisseminationofinformationoneffectiveIOG,

ISST,andcollaborativepractices1. SummarizeresearchonbestpracticesrelatedtocoreCMPcomponents.OMNIevaluators

areleadingeffortstoreviewresearchfindingsinthesystemsliteraturewhichwillbesharedwithlocalCMPsandstakeholdersintwopublicationstobedevelopedinthe2011‐2012fiscalyear.Theseincludethefollowing:

“SystemsFacilitationGuide.”Thisreportwillsummarizetheextantresearchliteratureonsystems‐levelcollaborationandthecorecomponentscontributingtoCMPstructuresandprocesses(e.g.,familyinvolvement,integratedserviceplanning).Thegoalistoidentifybestpractices.

58

Briefevaluationandresearchreportsonkeytopics.Intheupcomingyear,theevaluationteamwillcreatequarterlyevaluationnewslettersexploringtopicsrelevanttostateandlocalCMPstakeholders,suchasmeasurementofcostsavings.

2. SupporttheCMP’sbroadereffortstodefineanddisseminatebestpracticesinlocalCMPimplementationandoperations.OMNIstaffwillparticipateinanumberofeffortsrelatedtodisseminatingbestpractices.Theseincludethefollowing:

CMPHandbookdevelopment.OMNIwillcontributetoeffortsbeingledbyCMPcoordinatorstodevelopaCMPHandbook,whichwillincludekeyinformationtoorientnewCMPsandnewstakeholderstotheinitiative,theestablishinglegislation,SSCpoliciesandprocedures,localevaluationandmeasurementprocesses,andcollectivelearningsregardingeffectiveimplementationofIOGandISSTstructuresandprocesses.

CMPinformationsharingactivities.OMNIwillcontinuetosupportCMPcommunitynetworkingactivities,throughmaintainingtheCMPSharePointportal(anon‐lineinformationsharingtool)andattendingand/orfacilitatinglocally‐initiatedregionalmeetingsfocusedonCMPimplementationtopics.

Familyinvolvementtechnicalassistance.OMNIwillcontinuetosupporttheFamilyVoiceandChoiceCommittee’seffortstoexamineevaluationfindingstoidentifytechnicalassistanceneedsrelatedtofamilyinvolvementacrosstheinitiative.

Incentivefunding.OMNIwillfacilitatetheSSC’sdiscussionsregardingthedevelopmentofnewmethodstodetermineperformancebasedincentivefundingallocations.

Movingforward,theevaluationwillfocusoncontinueddevelopmentofmeasurementinfrastructuretocreateasolidfoundationforsummativeevaluationofCMPeffortsstatewide.OMNIwillcontinuetogroundtheevaluationinaparticipatoryapproach,workinginclosecollaborationwiththeCMPStateSteeringCommitteeandtheEvaluationSub‐committee,inanefforttoensurethatresultsarewellalignedwiththeneedsandinterestsoftheinitiativeanditsstakeholders.

59

Endnotes iColoradoRevisedStatute,Title24,Article1.9.(2010).Retrievedfromhttp://www.michie.com/colorado/lpext.dll?f=templates&fn=main‐h.htm&cp.iiCaliforniaDepartmentofEducation.(2007).Handbookondevelopingandevaluatinginteragencycollaborationinearlychildhoodspecialeducationprograms.RetrievedJune1st,2010fromhttp://www.cde.ca.gov/sp/se/fp/documents/eciacolbrtn.pdf.iiiU.S.DepartmentofHealthandHumanServices.(2010).Guidingprinciplesofsystemsofcare.RetrievedJune1st,2010fromhttp://www.childwelfare.gov/pubs/soc/socc.cfm.ivEmshoff,J.G.,Darnell,A.J.,Darnell,D.A.,Erickson,S.W.,Schneider,S.&Hudgins,R.(2007).Systemschangeasanoutcomeandaprocessintheworkofcommunitycollaborativesforhealth.AmericanJournalofCommunityPsychology,39,255‐267.vHicks,D.,Larson,C.,Nelson,C.,Olds,D.,&Johnston,E.(2008).Theinfluenceofcollaborationonprogramoutcomes:TheColoradoNurseFamilyPartnership.EvaluationReview,32,453‐477.viIbid.viiChrislip,D.D.,Larson,C.E.(1994).Collaborativeleadership:Howcitizensandcivicleaderscanmakeadifference.SanFrancisco:Jossey‐Bass.viiiBrunsEJ,SatherA,StambaughL.(2008).Nationaltrendsinimplementingwraparound:ResultsfromtheStateWraparoundSurvey,2007.In:Bruns,E.J.&Walker,J.S.,(Eds.),Resourceguidetowraparound.Portland,OR:NationalWraparoundInitiative,ResearchandTrainingCenterforFamilySupportandChildren’sMentalHealth.ixAnnieE.CaseyFoundation.(2002).Teamdecisionmaking:Involvingthefamilyandcommunityinchildwelfaredecisions.FamilytoFamilyToolsforRebuildingFosterCare.RetrievedNovember1st,2009fromhttp://www.aecf.org/upload/pdffiles/familytofamily/f2f_tdm_sept_02.pdf.xHenggeler,S.W.,Schoenwald,S.K.,Borduin,C.M.,Rowland,M.D.,&Cunningham,P.B.(2009).Multi‐systemictherapyforantisocialbehaviorinchildrenandadolescents.NewYork:Guildford.xiAlexander,J.,Pugh,C.,Parsons,B.,&Sexton,T.(2000).FunctionalFamilyTherapy.InD.S.Elliott(Ed.),BlueprintsforViolencePrevention(Vol.3).Boulder,CO:VenturePublishing.xiiBotvin,G.J.,Griffin,K.W.,Paul,E.,&Macaulay,A.P.(2003).PreventingtobaccoandalcoholuseamongelementaryschoolstudentsthroughLifeSkillsTraining.JournalofChildandAdolescentSubstanceAbuse,12,1‐18.xiiiWebster‐Stratton,C.&Reid,J.(2010).TheIncredibleYears:Evidence‐basedparentingandchildprogramsforfamiliesinvolvedinthechildwelfaresystem.NewYork:JohnWiley&Sons.

60

xivAltman,J.C.(2005).Engagementinchildren,youth,andfamilyservices:Currentresearchandbestpractices.InG.Mallon&P.Hess(Eds.),ChildWelfareforthetwenty‐firstcentury:Ahandbookofpractices,policiesandprograms(pp.72‐86).NewYork:Columbia.xvGoerge,R.M.,Smithgall,C.,Seshadri,R.,&Ballard,P.(2010).IllinoisFamiliesandTheirUseofMultipleServiceSystems.Chicago:ChapinHallattheUniversityofChicago.

61

Appendices

62

AppendixA.StructureofstatemanagementoftheCMP

StateSteeringCommitteeProvidesprogramoversight,advancessystemreformgoalsofCMPlegislation,composedofrepresentativesoforganizationsofmandatorysignatories,familyadvocacy,andparticipatingcounties.

EvaluationSubcommitteeGuidesstatewideevaluation,promoteseffectiveevaluationpracticesatthecountylevel,andconsultswithOMNIInstitute

Communications/MarketingSubcommitteeGathersanddisseminatesinformationaboutcollaborativeactivitiestostateandlocalstakeholders,facilitatesinformationsharing

FamilyVoiceandChoiceSubcommitteeCo‐ledbyalocalCMPcoordinatorandstatefamily‐drivenorganization,thiscommitteegathersandsharesinformationaboutpromisingstrategiestoengagefamiliesattheIOG,ISST,andservice‐deliverylevels.

SustainabilitySubcommitteeSupportssustainabilitythroughpoliticaladvocacy,strategicplanning,grantwriting,andcultivationoforganizationalpartnerships.

DHSProgramDirectorProvidesguidanceandoversight,reviewscounty‐levelmemorandaofunderstandingandannualreports,allocatesfundsandincentivefunds

63

AppendixB:Additionalreportsandproducts

BelowaredescriptionsofselectadditionalreportsandproductsdevelopedduringthecourseoftheCMPstatewideevaluationtodate.ThesedocumentscanbeaccessedusingthefollowinglinktotheCMPinitiativeportal(notethatrecentreportsmaystillbeunderfinalreviewandthusmaynotbepublishedonthissiteyet):http://collaboration.omni.org/sites/1451/SitePages/Home.aspxReportspublishedin2010

Report Date Description

CollaborativeManagementProgram:Year1EvaluationProgressReport

February2010

ThisreportdescribesOMNIInstitute’squalitativeandquantitativeapproachtostatewideevaluationforthefirstyearofevaluation.Thereportdiscussesinitialformativefindings,addressingtopicssuchasbarrierstoCMPimplementation,resourceallocationconflicts,IOGperformanceissues,andobservedsuccessoflocalCMPimplementation.

StateManagementSurveyFindings November2010

InFall2010,atotalof29StateSteeringCommitteemembersandIOGCoordinatorsrespondedtotheStateManagementSurvey,anonlinesurveydesignedtoelicitfeedbackregardingCMPmanagement,progress,andareasforfurtherdevelopment.ThisreportpresentstheresultsoftheStateManagementSurvey,whichincludesitemsthatassessprogressandfunctioningoftheStateManagementOffice,theStateSteeringCommittee,theFamilyVoice&ChoiceSubcommittee,theEvaluationSubcommittee,andtheoverallHB1451initiative.

CollaborationandOverallSuccessSurvey:InitialDataSummary

December2010

ThisreportprovidesasummaryofinitialfindingsfromtwosurveysadministeredtoIOGmembersin22CMPsinSpring2010.ThesurveysassessedIOGmembers’perceptionsoftheirCMP’scollaborativeprocesses,aswellastheperceivedsuccessoftheircollaborativeeffortsonseveralgoalsofthestatelegislation.Thefindingsprovideaninitialassessmentofcollaborativeprocessesandoutcomes,animportantpartofdemonstratingthebenefitoftheCMPapproach.

SummaryofSelectChildWelfareDataIndicatorsbyCounty

June2010

Thisreportpresentscounty‐level“snapshots”ofperformanceonsixchildwelfaredataindicatorsforthetimeperiodof2003to2009(representedinaveragesacrossyears),asdefinedbytheU.S.DepartmentofHealthandHumanServicesandreportedtotheColoradoDepartmentofHumanServicesthroughtheTrailsdatabase.Thisreportalsoprovidesgeneralbackgroundinformationontheavailablecounty‐levelstatewidedatarelatedtoCMPefforts.

64

Report Date Description

EmergingEvidenceSupportingCollaborativeManagementPrograms

July2010

OMNIInstitutepreparedabriefdocumentinSummer2010detailingevidencethatsupportsthesuccessofcommunitycollaborativestomeetlegislationgoals.ThedocumentincludesasummaryoftheSystemsofCare(SOC)philosophyandhighlightstheHighFidelityWraparoundapproachasonepossiblewaytoimplementcollaborativecarepracticestoimprovechildandfamilyoutcomes.

CollaborativeProcessesandEmergingBestPractices

August2010

Thisreportsummarizestheresultsofin‐depthinterviewsconductedwithasampleofCMPstakeholders(localIOGmembers)andresponsestoopen‐endedquestionsfromasurveyonthequalitiesofcollaborationefforts.Thereportprovidesadiscussionofthemajorthemesthatreflectfindings,alongwithabriefpresentationofpotentialbestpractices.

CollaborativeManagementProgram:Year1StatewideEvaluationFindings

October2010

Thisreportsummarizesfindingsandlearningsemergingfromevaluationeffortsoverthefirstyearofthestatewideevaluation.ThereportdetailsOMNIInstitute’sworktodateonidentifyingcommonanduniquecomponentsofCMPserviceandsystemsdeliverymodelsandimplementation;measuringthequalityofcollaborativestructures,relationships,andprocessesaswellastheperceivedeffectivenessofthecollaborativeapproach;buildinginfrastructureandprocessestoassessprogressonlegislativegoalsinastandardwayacrossCMPs;identifyinghowCMPsaremeasuringlocalprogramimpactsbyidentifyingeachCMP’stargetedpopulationsandchild/familyoutcomes,dataindicators,andperformancegoals;andassessingthecapacitytoassesscross‐CMPimpactsonkeypopulation‐leveloutcomesandprovidingrecommendationsandguidanceinestablishinginfrastructuretomovetowardsstatewideoutcomeanalyses.

65

Reportspublishedin2011

Report Date Description

CollaborativeManagementProgram:Year2EvaluationProgressReport

February2011

BuildinguponfindingsfromYear1,statewideevaluationeffortsinYear2involvedfocusingmeasurementonasmallsetofwell‐defined,uniformlymeasuredoutcomes.Thisreportsummarizestheresultsandrecommendationsthatledtothisfocus,resultinginthedevelopmentofanOutcomesWorkgroup;describestheOutcomesWorkgroupprocessandprogress;presentsthemeasurementplanthatemergedfromthisprocess;anddetailstheproposednextstepsintheprocess.

StateMeasurementPlan:BriefSummaryforCMPCoordinatorsandIOGs

March2011

OMNIdevelopedabriefsummaryofthestatemeasurementplan(seepreviousentry)fordisseminationamongCMPcoordinatorsandfortheirIOGs.Thepurposeofthissummarywastodetailtherationale,objectives,dataindicators,datacollectionstrategies,andnextstepsfortheimplementationofthestatemeasurementplan.ThisdocumentwasintendedtopresentkeycontentabouttheOutcomesWorkgroup’seffortsandtheresultingstatemeasurementplaninaformatappropriateforwidedistribution.

StateMeasurementPlan:PowerPointPresentationforCoordinators

April2011

Inadditiontothebriefsummaryjustdescribed,thispresentationwasdevelopedasatooltoassistCMPcoordinatorsintheireffortstodescribethestatemeasurementplantoIOGmembersanddiscusslocalCMProlesandresponsibilities.Likethetwoprecedingresources,thepresentationprovidesasummaryoftheOutcomesWorkgroup’sefforts,thestatemeasurementplanthatemergedfromtheWorkgroup’splanningprocess,andnextstepsforimplementingthestatemeasurementplan.

CurrentPracticesinthePreventionofChildAbuseandNeglect

April2011

Thisbriefreportprovidesanoverviewofevidence‐basedprogramsandinterventionsmostwidelyrecognizedintheareaofchildabuseprevention.Alsoincludedarediscussionsaboutthetheoreticalframeworksthatprovidethebasisfortheseevidence‐basedpractices,aswellaslinkstomoreinformationaboutadditionalpromisingprograms.

CollaborationandOverallSuccessSurvey:SummaryTable

June2011

ThisbriefdocumentprovidesasummarytableofaggregateCollaborationandOverallSuccesssurveyresultsfromsurveyre‐administrationinSpring2011(seeCollaborationandOverallSuccessSurvey:InitialDataSummary–December2010abovefordetailsregardingthesurvey).Dataresultsforanumberofsurveydomainswerereportedbyfiscalyear,bycoordinatorstatus,andbypopulationdensitycharacteristics.

66

Report Date Description

CollaborationandOverallSuccessSurvey:County‐LevelSurveyResults

July2011Thesereportsprovidedescriptiveinformationregardingeachcounty’sIOGmembers’generalperceptionsoftheirCMPstructure,collaborativeprocesses,andemergingoutcomes.TheresultsinthesereportsreflectCMPsscoresfortwofiscalyears(2009‐10and2010‐11).

FamilyInvolvementSurveyReport August2011

InSummer2011,theFamilyVoiceandChoiceCommitteecollecteddatarelatedtoCMPfamilyinvolvementinordertounderstandfamilyadvocacyactivityasimplementedamongCMPcounties.Thisreporthighlightspreliminaryfindingsfromdatacollectionefforts,includinganoverviewofcurrentfamilyadvocacyactivities,aswellasadiscussionforresourcedevelopmentandtrainingopportunitiesforCMPcounties.

67

AppendixC.SourcesofadditionalfundingreportedbyCMPs

FundingType

SourceofFunds DescriptionNumberofCMPsreporting

TotalAmountsReceived

Federal

JuvenileAccountabilityBlockGrant

Fundsprovidedasblockgrantstostatesforprogramspromotinggreateraccountabilityinthejuvenilejusticesystem

1 unreported

BureauofJuvenileAssistance

Fundsprovidedforprogramsaimingtoimprovecriminaljusticesystemsandservicesforcrimevictims

1 unreported

SupportSystemsforRuralHomelessYouth

FundsawardedtoColoradotosupportprogrammingforhomelessyouthandfamilies

1 $15,000

BasicCenterProgram

GrantfromFamilyandYouthServicesBureautosupportorganizationsandsheltersthatserveandprotectrunaways,homeless,andstreetyouth

1 $15,000

PromotingSafeandStableFamilies

FundsdistributedbyUSHSStostatestograntservicesthataddressfamilysupport,familypreservation,reunification,andadoption

2 $73,500

OfficeofJuvenileJusticeandDelinquencyPrevention

Fundsforjuvenilejusticeprogramming

1 unreported

State

DivisionofCriminalJusticeJuvenileAssistanceGrants(JAG)

Competitivefundingfordevelopmentofprogramsthatpreventorreducecrimeanddelinquencyusingcollaborativeevidence‐basedandpromisingpractices

6 $415,000*

TonyGrampsasYouthServicesGrant

Competitivefundingtolocalorganizationsthattargetyouthandtheirfamilieswithprogramsdesignedtoreduceyouthcrimeandviolence

1 unreported

StatewideStrategicUseFund

FundsdistributedforprogramsservingColorado'smostvulnerablefamilies

2 $382,489

DivisionofBehavioralHealthPreventionBlockGrant

Fundsforpreventionprogramming 1 $98,787

Privatefoundation/

other

UnitedWay Fundsforsocialservices 1 $81,000

CenterforJuvenileJusticeReformTechnicalAssistanceGrant

Providestechnicalassistanceforimplementinganevidence‐basedcollaborativejuvenilejusticepracticemodel

1 unreported

ElPomarFoundationBuildaGenerationGrant

Fundsforsocialprograms 2 $4,000

68

AppendixD.FrequentlyuseddatasourcesformeasurementofCMPperformancegoals

Domain DataSource Description

ChildWelfare

Trails

Trailsisastatewideautomatedcasemanagementsystemdetailingclientinformationacrossyouthcorrectionsandchildwelfarepopulations.TrailsiscurrentlyusedbyvariousdivisionsoftheColoradoDepartmentofHumanServices.

JuvenileJustice

Eclipse/ICON

ICON/Eclipseisastatewideautomatedcasemanagementsystemforthedistrictandcountycourts,andisusedbytrialcourtsandprobation.ICON/EclipselinkstotheColoradoIntegratedCriminalJusticeInformationSystem(CICJIS)whichisusedbytheDivisionofYouthCorrectionsandotherlawenforcementagenciestotrackoffenses.

Trails Seeabove

Education

Localschooldistrictrecords

Districtsmaintainlocaldatabasesthattrackabsences,tardiness,andotherstudent‐leveldata

Web‐basedrecordsmanagementprograms(PROSTAR,GoEdu.comandInfiniteCampus)

Onlinedatatrackingsystemstorecordattendance,grades,studentinformation;withschool,teacher,parent,andyouthlevelaccess

NWEADistrictTesting Standardizedmeasuresofacademicprogress(MAP)testsimplementedindistrict‐widetesting

Health/MentalHealth

CCAR/DMH

CDHS’sColoradoClientAssessmentRecords(CCAR)isamulti‐dimensionalchecklistandleveloffunctioningmeasurethatisconsistentlycollectedatintakeanddischargeinseveralmentalhealthsettingsacrossthestate,includinginpatientandsubstanceuseprograms.

DACODS/DBH

DACODSisaclientleveldatacollectioninstrumentusedbytheDivisionofBehavioralHealth(DBH)totracksubstanceusetreatmentandprogramimplementationvariables

CHIPS/CMHCprogramCentennialMentalHealthCenter’sinformationsystemtrackingclient‐levelmentalhealthservicerelateddata