intermediary organizations and education innovation · center for public research and leadership 6...
TRANSCRIPT
Intermediary Organizations and Education Innovation
FrameworksandToolsforUnderstandingandEvaluatingIntermediaryOrganizationsandTheirRoleinK-12EducationInnovation
April2017
CenterforPublicResearchandLeadership
Center for Public Research and Leadership
TheCenterforPublicResearchandLeadership(CPRL)isapartnershipoftopU.S.graduateandprofessionalschools.CPRLbringstogetherbusiness,education,law,andpolicystudentstostudyandprovidehigh-qualityresearch,strategicplanning,andevaluationsupporttopublic-andsocial-sectororganizationsseekingtransformationalchangeinK-12education.
Center for Public Research and Leadership
3
CONTENTS
Foreward. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
Acknowledgements. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
Introduction:TheCaseforIntermediaryOrganizations. . . . . . . . . . . . . . . . 6
Part1:DescribingtheRoleofanIntermediaryinDiffusingEducationInnovation. . . . . 10
Part2:AssessingtheRoleofanIntermediaryinDiffusingEducationInnovation. . . . . 14
Closing. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
References. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
AdditionalResources:MeasurementToolkitforIntermediaryOrganizations. . . . . . . 30
AdditionalResources:PhasedRubricsforAssessingIntermediaryOrganizations. . . . . . .
86
Center for Public Research and Leadership
4
FOREWARD In2014,theBill&MelindaGatesFoundationbeganinvestinginpersonalizedlearning—amodelofinstructionsthatishighlylearner-centered—asastrategyforachievingsystemicchangeinK-12education.SincethattimetheFoundationhasinvestedinavarietyofprojectsrangingfromschoolanddistrictredesignstotechnologydevelopmenttoresearchandevaluationallaimedatdiffusing,orspreading,innovativepersonalizedlearningpracticesand,inturn,improvingstudentoutcomesacrossthenation.TheseinvestmentsincludefundingseveralregionalintermediaryorganizationstoactaslocalhubsforinnovationandsystemtransformationthroughtheNextGenerationLearningChallenges(NGLC)RegionalFundsforBreakthroughSchoolsinitiative.
NGLC,anon-profitassociationthatworkstocatalyzeandacceleratethebroadadoptionofeffectiveandinnovativeeducationpractices,launchedtheRegionalFundsforBreakthroughSchoolsinitiativewithadditionalfundingfromtheEliandEdytheBroadFoundationandtheMichaelandSusanDellFoundation.TheRegionalFundinvestmentsaremeanttosupporteducationentrepreneurs,igniteinnovation,encouragecooperationandalignmentwithinregions,andexpandinterestinpersonalized,learner-centeredinstructionalmodels.In2014,NGLCinvestedinsixRegionalPartners:
• CityBridgeEducation(Washington,D.C.)• ColoradoEducationInitiative• LEAPInnovations(Chicago)• NewEnglandSecondarySchoolConsortium• NewSchoolsforNewOrleans• RogersFamilyFoundation(Oakland)
BMGF’sinvestmentinNGLCandtheRegionalPartnersoffersanumberofstrategicbenefitstotheFoundation.ThenetworkhasthepotentialtoacceleratethespeedandreachoftheFoundation’sgrantmaking;itcanofferenhancedlocalexpertiseanddeep,stablesupporttopartners;anditprovidesBMGF,NGLC,andtheRegionalPartnerswithanopportunitytopoollearningfromacrossadiversesetofcontexts.However,workinginnetworkedintermediarystructuresalsoposeschallenges,oneofwhichisestablishingmonitoringandevaluationmechanismsthatareresponsivetothecomplexityofanintermediary’srole.Thiscomplexitystemsfromthefactthatintermediariesworkatmultiple,interrelatedlevelstoimpactchangeandanintermediary’sstrategymustcontinuallyadjusttochangesinanyoneoftheselayers.
TheCenterforPublicResearchLeadership(CPRL)soughttoaddressthischallengebyengagingasubsetoftheNGLCRegionalPartnersinthedevelopmentofframeworkstodescribeandevaluatethevariousstrategiestheyemployastheyworktospreadinnovativeinstructionalpracticesacrosstheirregionsand,inthelongrun,improvestudentoutcomes.
Center for Public Research and Leadership
5
ACKNOWLEDGEMENTS ThroughgeneroussupportprovidedbytheBill&MelindaGatesFoundation,CPRLspentthelastyearresearchingtheroleintermediaryorganizationsplayinthediffusionofinnovativeideasandpractices.Alongtheway,wecollaboratedwithinspiringandforward-thinkingorganizationsthatareworkingtoprovidestudentswithrigorous,personalizedlearningexperiences.TheseorganizationincludedNextGenerationLearningChallenges(NLGC)andfourofNGLC’sRegionalFundPartners—CityBridgeEducation,ColoradoEducationInitiative,NewEnglandSecondarySchoolConsortium,andNewSchoolsforNewOrleans.Wearedeeplyappreciativeoftheirengagementinthisproject.
Intotal,wespokewithover30individualsfromacrossthesefiveorganizations,andwearethankfultoeachpersonfortheinsightsshared.Inparticular,wearegratefulforthetimeandenergydevotedbyMargretAngell,CarolineHill,andArthurMcKeefromCityBridgeFoundation;HeatherChikoore,ChristinaJean,EmilyLove,andSamOlsonfromColoradoEducationInitiative;LaurenHinthorneandMarkKostinfromNewEnglandSecondarySchoolConsortium;IndrinaKanthfromNewSchoolsforNewOrleans;andAndyCalkinsandSarahLuchsfromNGLC.TheseindividualsspenthourswithCPRLbothinpersonandonthephonedescribingtheirvisionsforeducation,thestrategiesthatguidetheirday-to-dayoperations,andthelessonstheyhavelearnedalongtheway.
WealsowanttothankLedyardMcFaddenfromSchoolWorksandAmyNowellfromLEAPInnovationsforsharingtheirexpertiseduringourNovember2016conveningatColumbiaUniversity.
Intheend,theworkwouldnothavebeenpossiblewithouttheknowledgewegleanedfromeachoftheseindividuals.
Center for Public Research and Leadership
6
INTRODUCTION
Intermediary Organizations Atthemostbasiclevel,anintermediaryorganizationisago-betweenormediator.However,thespecificrolesplayedbyintermediariesarediverse(Blanketal.,2003;Szanton,2003;Wynn,2000).Someintermediariesactasoversightorganizationsthatsetstandardsandmonitorprogress.Othersprovideservicesandresourcesmuchlikeanexternalconsultantortechnicalassistanceprovider.Stillothersfocusonfosteringconnectionsbetweenindividualsororganizationsinordertofacilitatecommunicationandcollaborationlikeanetworkhuborbackboneorganizationtypicallydoes.Finally,manyintermediariesplaytheroleofagrantmakinginstitutionandprovideindividualsandorganizationswithfundingforspecificinitiatives.Toensureacommonlanguage,thisreportdefinesintermediaryorganizationsasindependentlyoperatingentitiesthatworkbetweenmultipleactorstofacilitatecommunicationandcollaboration;buildcapacityandknowledge;andovertimebringaboutchangeintheactors,theiractivities,andtheresultstheyachieve.
Intermediaryorganizationsareoftenpartiallyorcompletelysupportedbylargerphilanthropicfoundationsorgovernmentgrantmakingagencies.Suchstructuresofferbenefitstoboththelargerfunderandtotheindividualororganizationalactorssupportedbytheintermediary.Thesebenefitsarelistedbelow(Szanton,2003;GEO2013,GEO2014).Ingeneral,intermediaryorganizationsallowfunderstoremainagileandleanwhilestillensuringtheirintendedbeneficiariesreceivedhigh-quality,supporttailoredtotheiruniquecontextsandneeds.
“Intermediariesareorganizationsthatoccupythespaceinbetweenatleasttwootherparties…Intermediaryorganizationsoperateindependentlyofthesetwopartiesandprovidedistinctvaluebeyond
whatthepartiesalonewouldbeabletodeveloportoamassbythemselves.Atthesametime,intermediaryorganizationsdependonthosepartiestoperformtheiressentialfunctions.”
—MeredithHonig,“TheNewMiddleManagement:IntermediaryOrganizationsinEducationPolicyImplementation”
------
“Intermediariescomeintobeingtohelpachievemoreefficient,effectiverelationshipsbetweenlayers…Matureintermediariesessentiallytakeonmanyofthefunctionsofsystems—standardssetting,
qualityassurance,training,advocacy,funddevelopment,datacollection.”
—KarenPittman,“TheImportanceofIntermediaryOrganizationstoImplementingCommunityInitiatives”
BenefitstoFunders• Increasedspeedandreachofgrantmaking• Reducedoverhead• Politicalprotections• Easeofprogramexit• Increasedoversightandsupportfor
beneficiaries• Increasedcredibilitywithpractitioners• Increaseddiversificationandmore
opportunitiesforlearning
BenefitstoPartners• Freshperspectives• Specializedskillsandknowledge• Contextualexperienceandexpertise• Durabilityofsupport• Increasedaccesstoresourcesandtechnical
assistanceproviders• Increasedaccesstoprofessionallearning
communities
Center for Public Research and Leadership
7
Anotherbenefitofworkingthroughintermediaryorganizationsisthenetworkstructuretheycreate.Thisstructuredevelopsasintermediariesbuildconnectionswithmultiplepartners.AreportfromtheU.S.DepartmentofHealthandHumanServicesdescribestwotypesofintermediariesthatresultfromdifferentdecisionsabouthowtoengagewithpartners—distributiveandconvening(2008).Adistributiveintermediarydisseminatesservicesandresourcestoapartnerthroughaone-to-onerelationship.Theresultisahighlycentralized,hub-and-spokenetworkstructure.Aconveningintermediarydisseminatesservicesandresourcestopartnersdirectlyaswell,butalsoworkstoconnectpartnerstooneanotherformingamoredecentralizednetwork.
Thenetworkstructureallowsintermediariestooperateatmultiplelevelssimultaneously.Atthepartnerlevelanintermediarymayengageindirectfunding,technicalassistance,coaching,orothercapacity-buildingactivities.Atthenetworklevel,intermediariesrecruitnewpartners,facilitatecollaboration,andhostconvenings.Inadditiontothesetwolevelsintermediariesmayalsoworkatathirdlevel—thesystemlevel.Atthislevelanintermediarytargetsthepolitical,economic,orsocialconditionssurroundingandimpactingitsnetwork.
Itisanintermediary’sabilitytosuccessfullyworkateachoftheselevelsthatmakesitwellsuitedtoscalingnewideasandpractices.InDiffusionofInnovation,EverettRogersdefinesdiffusionastheprocessbywhichaninnovationiscommunicatedthroughcertainchannelsovertimeamongmembersofasocialsystem(1962).Thisdefinitionhighlightsthethreelayersatwhichanintermediaryoperates:members,communicationschannels,andthesocialsystem.Bystrengthenandaligningtheselayers,anintermediarycandirectlysupportdiffusion.AfourthpieceinEverett’sdefinition—theinnovation—isalsoanelementthatintermediariescanimpactthroughknowledge-managementactivities.However,thisreportcombinesknowledge-managementactivitieswithpartnercapacitybuilding.
Report Overview Thisreportseekstoclarifytheroleofintermediaryorganizationsinthediffusionofinnovativeeducationpracticesaswellasbuildthecapacitytomonitorandevaluatethisrole.Inordertoachievethesegoalsthereportpresentstwointerrelatedframeworks—theDiffusionLeversandtheDiffusionIntermediaryEvaluationFramework.Theformerisadescriptiveframeworktoassistinunderstandinganddifferentiatingbetweenintermediaries’strategies.Thelatterisamonitoringandevaluationframeworkforassessinganintermediary’sreadinessforandsuccessatimplementingitsstrategy,aswellastheimpactthisstrategyhas.Thereportisorganizedintotwoprimarysectionstocorrespondwiththeseframeworks.
Part 1: Describing the Role of an Intermediary in Diffusing Education Innovation
Part1focusesontheDiffusionLevers,whichprovideacommonlanguagefordescribingthestrategiesofintermediaryorganizations.ThesectiondiscussesthreeLevers—CapacityBuilding,NetworkWeaving,and
Center for Public Research and Leadership
8
SystemCultivating—eachofwhichiscommonlyemployedbyanintermediaryinanefforttospreadnewideasandpractices.
Part 2: Assessing the Role of an Intermediary in Diffusing Education Innovation
Part2presentstheDiffusionIntermediaryEvaluationFramework.Thisframeworkoutlinesathree-phaseapproachtoevaluatingintermediaryorganizationsovertime.Eachphaseconsistsofdimensionsforevaluationaswellasspecificcriteria,guidingquestions,andsampleindicators.TheDiffusionLeversareincorporatedintotheEvaluationFrameworktoensurethatitreflectsthespecificrolesintermediariestypicallyplayandtheshort-termoutcomestheycanbeexpectedtoachieve.
Additional Resources
Inadditiontotheprimaryreportsectionsoutlinedabove,thisreportprovidesadditionalresourcestofurthersupportusersinunderstandingandevaluatingintermediaryorganizations.Theseinclude:
• ToolkitsforeachofthethreeDiffusionLeversthatcontainthecriteria,guidingquestions,andsampleindicatorsforevaluationaswellasasmallselectionofdatacollectiontools.
• EvaluationrubricsalignedtothephasesoftheDiffusionIntermediaryEvaluationFramework.
Intended Users CPRLhopesthatthisreportwillbeusefultoanumberofdifferentusersincluding:
v Funderscurrentlyworkingwith,orlookingtoworkwith,intermediaryorganizationsandseekingfurtherinsightintotherolessuchorganizationscanplay,aswellashowtoevaluatetheorganizations’potentialforinfluence,interimprogress,andlong-termimpact.
v Existingintermediaryorganizationsalreadyworkingwithanetworkofpartnerstocatalyzeandaccelerateinnovationineducation,andwhoarelookingforclarityaroundhowtoevaluatetheirrole.
v Non-profits,consortia,foundations,orothergroupsconsideringbecomingintermediariesandsearchingforadditionalinsightintothedevelopmentoftheirvisionandstrategyfordiffusion.
v Individualsandorganizationsconsideringapartnershipwithanintermediaryorganizationandlookingtounderstandtheexperiencesandbenefitstheycanexpect.
Research Methods Abroadrangeofexistingliteratureonthediversestrategiesandgoalsofintermediaryorganizations,aswellastheworkofasampleofcurrentNGLCRegionalPartners,informedthisproject.
Collaboration with NGLC Regional Partners
FourofNGLC’scurrentRegionalPartners—CityBridgeFoundation,ColoradoEducationInitiative,NewEnglandSecondarySchoolConsortium,andNewSchoolsforNewOrleans—participatedinthisprojectalongwithNGLCitself.Theiruniqueandhighlydevelopedstrategiesprovidedthebackboneforthisreport.PriortoworkingwiththeRegionalPartners,CPRLrevieweddocumentsdiscussingtheworkofeachintermediary.CPRLthenconductedinterviewswithmultiplestaffmembersfromeachRegionalPartnerandwithNGLCprogramstaff.Followingthispreliminaryresearch,CPRLengagedtheRegionalPartnersandNGLCinaone-dayconveningatColumbiaUniversityfocusedondocumentingtheorganizations’uniquestrategies.Finally,thegroupconvenedagainlateintheprojecttodiscusstwohigh-priorityareasforevaluation—networkevaluationandevaluatingchangesininstructionalpractice.ThissecondeventwasfurthersupportedbyLEAPInnovationsandSchoolWorks.
Center for Public Research and Leadership
9
Review of Related Literature
InadditiontocollaboratingwiththeRegionalPartners,CPRLreviewedvariousstrandsofexistingresearch.Topicsreviewedincludesystems,networks,anddiffusiontheory;networkevaluation;collectiveimpact;scaling;andpriorresearchontheuseofintermediaryorganizationstosupportimprovedsocialoutcomes.Intotal,CPRLconsultedover100books,articles,andwebpagesthroughoutthecourseoftheproject.
Center for Public Research and Leadership
10
PART
1 DESCRIBING THE ROLE OF AN INTERMEDIARY IN DIFFUSING EDUCATION INNOVATION
Introduction to the Diffusion Levers TheDiffusionLeversprovideasimplesetoflanguageandconceptsfordescribingthemechanismsintermediaryorganizationsusetospreadnewideasandpractices.ThethreeLevers—CapacityBuilding,NetworkWeaving,andSystemCultivating—highlightthethreelevelsatwhichanintermediarycanwork.TheCapacityBuildingLeverfocusesontheindividualpartnersandtheirknowledgeandresources.Thesepartnersmaybepeopleororganizations.TheNetworkWeavingLeverfocusesonallpartnerstogetherasanetworkandthecommunicationandcollaborationoccurringbetweenthem.TheSystemCultivatingLeverfocusesonthelargerecosysteminwhichthenetworkoperatesincludingthesupportiveorrestrictivesocial,economic,andpoliticalconditionspresentintheecosystem.
Eachlevercanbefurtherspecifiedintoactionsthatintermediaryorganizationstakestoapplytheleveraswellasshort-termoutcomesthatresultfromitsuse.However,theleversthemselvesarenotmutuallyexclusive,andtheapplicationofoneoftenfurtherenablestheapplicationoftheothers.Forexample,thedevelopmentofanextensivenetworkofcloselyconnectedpartnersislikelytofacilitatethegenerationandsharingofnewknowledgewhilealsoincreasingthevisibilityoftheintermediary’smissionanditscampaigningcapacity.Finally,
Center for Public Research and Leadership
11
whileintermediariesoftenemployallthreeleverssimultaneously,thedegreetowhicheachisusedandthespecificactionstakenarelikelytodifferbasedonanintermediary’svisionandstrategy,localcontext,andorganizationalcapacity.Together,thethreeLeversdrivetowardthespreadofnewideasandpracticesatboththeindividualandorganizationallevel,and,overtime,thesechangesinpracticedrivetowardimprovedsocialoutcomes.Eachleverisdescribedinmoredetailinthepagesthatfollow.
Capacity Building CapacityBuildingentailsprovidingindividualsandorganizationswiththeknowledgeandresourcesneededtoincreaseunderstandingandimproveperceptionsofinnovation.
Individualsandorganizationspartnerwithintermediaryorganizationsinordertoachievemorethantheycouldachieveontheirown.Thesepartnersrecognizethatintermediaryorganizationsofferaddedcapacityintheformofincreasedfinancialresources,knowledgeofeffectivepractices,toolsforandsupportinimplementingthesepractices,andmore.Commonactionstakenbyintermediaryorganizationtobuildpartnercapacityincludeknowledgemanagement—theprocessofcapturing,synthesizing,organizing,anddisseminatingknowledge—trainingpartnersthroughgroupworkshopsorindividualcoaching,monitoringpartnerprocesses,andbrokeringresourcessuchasfundingandtechnicalassistancesupport.
Theseactionsresultinpartners’increasedreadinesstochangetheirbehaviorsandpractices.Theideaof“readiness”isinformedbycapacitybuildingandchangemanagementliterature,aswellastheinnovation-decisionprocess,whichalldescribevariousstagesindividualsandorganizationsgothroughwhendeterminingwhethertheywillchange(ConnollyandYork,2002;Rogers,2003;KirkpatrickandKirkpatrick,2005).Takentogether,thisbodyofresearchdescribesthatapartnermust(1)buildunderstandingofthenewbehaviorsorpractices,(2)developapositiveperceptionofthem,(3)establishtheintentiontochange,(4)implementthechange,and(5)internalizeorinstitutionalizethechange.Theshort-termoutcomesoftheCapacityBuildingLeverarethefirsttwostagesofthisprocess—increasedunderstandingandimprovedperceptions.Overtime,theseoutcomeswilldrivetowardchangesinpracticeandideallytowardimprovedsocialoutcomes.Apositiverelationshipbetweenchangesinpracticeandimprovedoutcomescreatesavirtuouscycledrivingfurtheradoptionofthenewpracticesacrossthefieldatlarge(Meadows,2008).
Center for Public Research and Leadership
12
Network Weaving Connectingindividualsandorganizationstobuildstrongcommunicationschannelsandahealthycommunitythroughwhichinnovationcanspread.
Anintermediaryorganizationcannotbeseparatedfromitsnetworkofpartners,and,asaresult,approachingthedesignandfacilitationofthisnetworkwithintentionalityisakeypartofanintermediary’srole.IntheirbookConnectingtoChangetheWorld,PeterPlastrik,MadelineTaylor,andJohnClevelandidentifyeightdesignissuespertainingtonetworks:
1. Purpose:Network’sreasonforbeing.2. Membership:Eligibilityandparticipationrequirementsformembersaswellasdesiredsizeand
compositionofmembership.3. ValueProposition:Benefitsmemberswillreceive.4. Coordination,Facilitation,andCommunication:Hownetworkmembersworkwitheachother.5. Resources:Network’sfundingmodel.6. Governance:Decision-makingparticipantsandprocesses.7. Assessment:Monitoringandevaluationstrategy.8. OperatingPrinciples:Rulesthatguideculture.
Makingthesedesigndecisionsisakeypartofanintermediary’sNetworkWeavingrole.AdditionNetworkWeavingactionsincluderecruitingmembersintothenetwork,facilitatingongoingcommunicationandcollaborationbetweenthesemembers,andconveningthenetworkface-to-faceorvirtually.
Keyshort-termoutcomesthatresultfromNetworkWeavingactionsincludemeasuresofnetworkformsuchasthesizeandcompositionofthenetwork,thenumberofconnectionsthatexist,andstrengthorpurposeoftheseconnections,aswellasmeasuresofnetworkfunctionssuchasthestabilityofthenetwork’sinfrastructureandoperationsandthedegreetowhichcertainconditionsareinplace,suchastrustandaccountability(Bonbright&Khangram,2010;Plastrik,Taylor,&Cleveland,2014;Taylor,Whatley,&Coffman,2015).Whileresearchindicatesthatdecentralizedandopennetworkstypicalsupportinnovationmoreeffectivelythancentralized,closednetworks,thereisnoonebestdesign(Plastrik,Taylor,&Cleveland,2014).Asaresult,thespecifictargetsfortheseoutcomesshouldaligntotheintermediary’sinitialnetworkdesign.
Center for Public Research and Leadership
13
System Cultivating Supportingchangestopolitical,organizational,economic,andsocialconditionstoreducebarrierstoandincreasesupportsforinnovation.
Inordertosupportthespreadofnewideasandpractices,itisalsocrucialforanintermediaryorganizationtoensurethesocial,economic,andpoliticalconditionssurroundingitsnetworkarealignedandsupportive.Anintermediarymaydothisthroughavarietyofactions,suchaspolicyanalysisandpoliticaladvocacy,campaigningandcoalitionbuilding,andreformingorganizationalsystemsandstructures.Inaddition,becausethespecificactionsanintermediarytakeswillbecontingentuponthecurrentandevolvingconditionsinplacewithinthelocalcontextwhereitoperates,itiscrucialforanintermediarytoassesstheseconditionsonaregularbasisthroughactivitieslikesystemmapping.
Short-termoutcomesrelatedtoSystemCultivatingincludeincreasedalignmentofpoliciesandpracticestotheinnovationaswellasexpandedpublicengagement.Morespecifically,SystemCultivatingeffortsarelikelytoleadtothereplacementofformallegislationorinformalorganizationalpracticesthatpreventinnovationwithpoliciesandpracticestheenableorrequireinnovation.Inaddition,theseeffortsarelikelytoleadtotheeliminationofpublicmisconceptionsaboutorproteststonewpractices,andtheremovaloffundingobstaclesthathinderimplementation.
ProofPoints:AnImpactofSuccessfulIntermediaryOrganizations
Center for Public Research and Leadership
14
PART
2
ASSESSING THE ROLE OF AN INTERMEDIARY IN DIFFUSING EDUCATION INNOVATION Introduction Whiletheprevioussectionofthisreportoutlinedadescriptiveframeworkforunderstandingthetypesofstrategies,orDiffusionLevers,usedbyintermediaryorganizationsworkingtodiffuseeducationinnovation,thissectionoutlinesanevaluativeframeworkforassessinganintermediary’sreadinessforandsuccessatemployingthesestrategies.Asdescribedinfurtherdetailtotheright,theDiffusionIntermediary’sMonitoringandEvaluationFramework(“Framework”)addressesthecomplexityofevaluatingtherolesofanintermediarythatarecausedbythediversityofstrategiesusedandthetendencyforthesestrategiestochangeovertime.TomaintainalignmentwiththeDiffusionLevers,thisframeworkwasinformedbyliteratureontheevaluationofcapacitybuilding,networks,andsystemchange.Additionalresearchrelatedtoorganizationalcapacity,readinessassessments,andscalingeducationreformalsocontributedtothedevelopmentofthisframeworkandensuresthatitisinclusiveofallstagesofanintermediary’swork.AfulllistofreferencesisinAppendixA.
Framework Organization
TheFramework(Figure1)isorganizedintothreesuggestedassessmentphases:
PhaseI:PotentialforInfluence-Assessingthelikelihoodofanintermediaryimpactingthediffusionofinnovationwithinitslocaleducationcontext.
PhaseII:InterimProgress-Assessinganintermediary’sstrategyimplementationandshort-termoutcomes.
PhaseIII:ImpactsontheField-Assessinganintermediary’sinterimimpactonthediffusionofinnovativeaswellasthelong-termimpactonsocialoutcomes.
Eachphaseconsistsoftwoorthreedimensions,andeachdimensionisfurtherspecifiedintoanumberofcriteriawithguidingquestionsandsampleindicators.
Responding to Diversity and Change
Thediversityofintermediaryorganizations’rolesacrossvariouscontextsandchangestotheserolesovertimerequiresanapproachtomonitoringandevaluationthatisflexibleandfocusesonprocessaswellasoutcomes.Theevaluationframeworkandprocesslaidoutinthisreportseekstodirectlyrespondtotheseneedsasoutlinedbelow.
The Challenges The Solutions
DiversityofStrategy
Intermediaryorganizationsare
diverse.Theyengageinavarietyoftypesof
activitiesinresponsetotheiruniquecontexts
andstrategies.
AFlexibleFramework
à AlignmenttoDiffusionLeversframeworkincreasesapplicabilityacrossdiverseintermediaries.
à Guidingquestionsaregeneralizabletodifferentcontextsandintermediaries.
à Sampleindicatorscanbecustomized.
ChangesinStrategy
Intermediaryorganizationsare
engagedinchangingcomplexsystems,andtheirstrategieschangeinresponsetoshiftsincontextandinterim
results.
AFocusonProcess
à Threephasesassessintermediary’sinitialpotential,interimprogress,andimpact.
à Includesprocesscriteriaaswellasoutcomecriteria.
à Includescontinuouslearningcriteriatoassessdocumentationoflessonslearnedandmidcoursecorrections.
Center for Public Research and Leadership
15
DiffusionInterm
ediaryM
onito
ringan
dEvalua
ting
Fram
ework
Center for Public Research and Leadership
Phase I: Potential for Influence PotentialforInfluenceassessesthelikelihoodofanintermediaryimpactingthediffusionofnewideasandpracticesacrossitscontext.Thislikelihoodshouldbeassessedattheoutsetofanintermediary’swork.Itcouldbedoneaspartofaninternalself-assessmentbeforeanewintermediarydecidestoformorwhenanexistingintermediaryconsidersmovingintoanewregion.Itcouldalsobepartofanexternalassessmentcompletedbyapotentialfunder.Thephaseconsistsofthreedimensionsforevaluation:
• ReadinessofLocalContext-Theregion’ssupportandneedforinnovativeeducationreformideasandpractices,asindicatedbythepolitical,social,andeconomicconditions.
• Vision&Strategy-Theintermediaryorganization’stheoryofaction—theinnovativeideasorpracticesitseekstospreadandthestrategyitwillusetodoso.
• OrganizationalCapacity-Theresourcesandinfrastructuretheintermediarybringstobearinimplementingitsstrategy.
ThesethreedimensionswereinformedbyProfessorMarkMoore’stheoriespertainingtostrategyinthepublicsectorandspecificallythestrategictriangle(Moore,1995).Thissimpleconceptualframeworkdrawsattentiontothreeissuesthatpublicsectororganizations,includingthetypesofintermediaryorganizationsdiscussedhere,mustconsiderwhenplanning.Theseincludepublicvalueorwhattheorganizationseekstoimpactwithinthesector,legitimacyandsupportortheexternalresources,structures,andconditionstheorganizationwillrelyon,andoperationalcapabilitiesortheinternalresources,structures,andconditionstheorganizationcanemploy.Mooreviewsthesethreeissuesashighlyinterdependent,withchangesinonedirectlyimpactingtheothertwo.AswithMoore’striangle,thethreedimensionsofPotentialforInfluenceareinterdependent.Asaresult,eachshouldbeconsideredinrelationtotheothersduringbothplanningandevaluation.Inotherwords,anintermediary’spotentialshouldbeviewedinrelationtothecurrentconditionintheregionwhereitoperatesandthedegreetowhichtheintermediaryhastheinfrastructureandoperationsitwillneedgivenitslong-termvisionandthestrategyitproposesforachievingthisvisionwithintheregion.
Thefollowtablesoutlinethethreedimensionsinmoredetail.Eachdimensionisfurtherspecifiedincriteriaforevaluationaswellasguidingquestionsandsampleindicators.
1.1 Readiness of Local Context
To what extent does the local context demonstrate both a need for and openness to the intermediary’s vision?
Criteria GuidingQuestions SampleIndicators
SupportforInnovation
Towhatextentisthelocalcontextsupportiveofthe
changesandworkproposedbytheintermediary?
• School-baseddecision-makingpolicies• Policies,orpolicywaivers,thatsupportnon-traditionaluse
oftime,talents,andspacewithinschools• Availabilityofinnovation-focusedfundingfromstateor
localgovernmentorfromprivatephilanthropies• Educationinnovationnon-profitsandincubators• Educationconferencesorprofessionalgatheringsfocused
oninnovativeideasandpractices• Presenceofpractitioners,researchers,funders,and
ReadinessofLocalContext
OrganizationalCapacity
Vision&Strategy
Center for Public Research and Leadership
17
1.2 Vision and Strategy
To what extent does the intermediary have a logical and viable vision and strategy that is likely to transform instructional practice and improve student outcomes?
technicalassistanceproviderswhoareinterestedinworkingwiththeintermediary
• Mediacoveragethatdemonstratespositiveattitudestowardseducationreformandinnovationgenerallyorthenewideasandpracticesthattheintermediaryseekstodiffuse
FieldNeed
Towhatextentisthelocaleducationsystem
demonstratinganeedforimprovement?
Towhatextentdoesthelocaleducationsystemneed
additionalsupportofthekindanintermediaryprovides?
• Presenceofindividualsororganizationslikelytoparticipate
• Presenceofpersistentorgrowingracialand/oreconomicachievementgaps
• Persistentlylowachievementcomparedtothenationalaverage
• Localachievementhasstagnatedordeclinedoverthepastfiveyears
• Dearthofsupportorganizationsworkingtowardeducationreformorinnovation
• Lackofdiversityintypesofsupportsprovidedbyschoolsupportorganizations
Description GuidingQuestions SampleIndicators
TransformativeVision
Towhatextentdoestheintermediary’svisionfor
changebreakfromtraditionaleducationalpracticesandsystems?
• Visionincludesnewanduniqueusesoftimeandspace• Visionincludesthecreativeintegrationoftechnology• Visionincludesatypicalstaffingallocationsandrole
definitions• Visionredefinescoresocialinteractionsbetweenteachers
andstudentsthatarecurrentlyinplaceinthelocalcontext• Visionisbasedonunderlyingassumptionsabouthow
studentslearnandwhatgoodinstructionmeansthatbreakfromthenormforthelocalcontext
ViabilityofStudentImpact
Howrobustistheorganization’sbasis(inevidenceorinlogic)forpredictingthatitsvisionwillpositivelyimpact
students?
• Linksbetweentheideasandpracticestheintermediaryseekstospreadandpositivestudentoutcomesaredocumentedandsupportedbyrigorousacademicresearch
• Relationshipsbetweentheideasandpracticestheintermediaryseekstospreadandpositivestudentoutcomesarebasedonalogical,detailedhypothesis
• Evidenceofanintermediary’spastworkpositivelyimpactingstudentoutcomesthatcannotbeeasilydiscreditedorexplainedbyotherchangesinthelocalcontext
Center for Public Research and Leadership
18
1.3 Organizational Capacity
To what extent does the intermediary have the resources and infrastructure required to enact its strategy?
Description GuidingQuestions SampleIndicators
InternalResources
Towhatextentdoestheintermediaryhave
sufficientfinancialandhumanresourcestosupport
itsstrategy?
• Sufficientfundsavailabletocompleteworkoutlinedinstrategy
• StaffexpertiseinkeystrategicareassuchasK-12pedagogy,adultlearning,changemanagement,knowledgemanagement,researchandevaluation,etc.
• Sufficientnumberofstaff• Suitabletechnologyinfrastructuretosupportgoals,
especiallyaroundpartnercommunicationandknowledgemanagement
• Sufficientaccesstospace
Connectedness
Towhatextentdoestheintermediaryhave
relationshipswithfield(e.g.,K-12education)andsystem
partners(e.g.,policymakers,funders,technicalassistanceproviders)across
thelocalcontext?
• Regularcommunicationwithfunders,technicalassistanceproviders,policymakers,practitioners,researchers,etc.
• Instancesofpriorcollaborationwithfunders,technicalassistanceproviders,policymakers,practitioners,researchers,etc.
InformedStrategy
Towhatextentdoestheintermediary’sstrategyincorporatethekey
DiffusionLeversofcapacitybuilding,networkweaving,andsystemcultivating?
• Strategyincludesactivitiesaimedatincreasingpartnerknowledge,skills,andresources,suchasgrantmaking,1:1coaching,knowledgedissemination,etc.
• Strategyincludesactivitiesintendedtobuildandcontinuallystrengthenconnectionsbetweenpartners,suchasconveningpartners,managingcommunicationssystems,facilitatingcollaborativeproject,etc.
• Strategyincludesactivitiesintendedtodecreasesystemicbarriersand/orincreasesystemicsupports,suchasdraftinglegislature,identifyingandbuildingchampionsforthework,rigoroussystemmapping,etc.
OperationalAlignmenttoStrategy
Towhatextentaretheintermediary’sday-to-dayactionsandprogramslinkedtoitsstrategy?
• Intermediarycanmapallelementsofprogrammingbacktothevariousprongsofitsstrategy
• Staffcanarticulatehowday-to-daytasksaligntospecificprogrammaticelements
Center for Public Research and Leadership
19
LearningOrientation
Towhatextentdoestheintermediaryregularlyreviewandreflectonitssuccessesandfailures,changesininternal
organizationalcapacity,andexternalcontextual
conditionsinordertolearnandmakemidcourse
corrections?
• Staffisopenandhonestaboutsuccessesandfailures• Staffhavearticulatedgoalsfortheirownimprovement• Comprehensiveandintegratedperformancemanagement
systemisusedthatconsidersbothprocessandoutcomes• Benchmarking(internalorexternal)isintegratedandused
regularly• Qualitativeandquantitativedataalignedtobothleading
andlaggingindicatorsarereviewedatregularintervals• Demonstratedpracticeofmakingadjustmentsand
improvementsbasedonqualitativeandquantitativedata• Third-partyexpertsbroughtintoassistinreviewing
progress,assessingneed,anddefiningapathforward
Leadership
Towhatextentdoestheintermediary’s
leader/leadershipteamdemonstratecharacteristics
thatallowthemtoeffectivelyandefficientlymanagetheorganization?
• Leadershiparticulatesaclearvisionforwhatispossibleinthefutureandhowtheintermediary’sstrategywillachievethisvision
• Leadershiptakesstepstomotivate,inspire,andrallyothersbehindtheintermediary’svision
• Leadershippromotestheimportanceoflearningandimprovementthroughwordsandactions
• Leadershippromotescreativityandrisktakingthroughwordsandactions
• Staffarticulatefeelingsofsupportandempowermentbyleadership
• Staffarticulateconfidenceinleadershipknowledge,skills,anddispositions
FieldInfluence
TowhatextentistheorganizationconsideredaninfluentialleaderintheK-12
educationfield?
• Reports,tools,orotherresourcescreatedbytheintermediaryarewidelyusedacrossthefield
• Intermediaryhaspositiveandwidenamerecognitionwithinthelocalcontextitseekstoimpact
• IntermediaryhaspositiveandwidenamerecognitionwithintheK-12educationsector
• Intermediaryispresentatkeysector-specificconferencesorotherprofessionalevents
Phase II: Interim Progress InterimProgressassessesanintermediary’simplementationofitsstrategyandtheresultsthatstemfromit.Thephaseincludesthreedimensionsforevaluation:
• ImplementationofStrategy-Theextentandqualityofanintermediary’sactivitiesandthedegreetowhichthesemeettheintermediary’stargets.
• Short-termOutcomes-Theearlychangestopartner,network,andsystemcapacitiesandconditionsstemmingfromactivitiescompleted.
Center for Public Research and Leadership
20
• ContinuousLearning-Anintermediary’sowninternallearningandimprovementprocessesandoutcomes.
TheInterimProgressdimensionsandcriteriashouldbeusedtoassessanintermediaryassoonasitbeginsrecruitingandprovidingresourcesandsupporttopartnersandatregularinternalsafterthattime.ThesixcriteriaintheShort-TermOutcomesdimensionbelowaredirectlyalignedtotheDiffusionLeversdiscussedintheprevioussectionofthisreport—CapacityBuilding,NetworkWeaving,andSystemCultivating.WhileitislikelythatmostintermediaryorganizationswillemployallthreeoftheDiffusionLevers,andasaresultshouldevaluateforallsixoutcomes,asmallnumberofintermediariesmayonlyemployselectLevers.Theywouldthenonlyexpecttoseeselectcriteria.
Eachdimensionisfurtherspecifiedincriteriaforevaluationaswellasguidingquestionsandsampleindicators.
2. 1 Implementation of Strategy
To what extent and with what level of quality is the intermediary implementing its strategy?
Criteria GuidingQuestions SampleIndicators
ExtentofImplementation
Howmanyresourcesand/orhowmuchsupportdidtheintermediaryprovideover
theperiod?
• Numberofreportswritten• Hoursoftrainingprovided• Numberofcoachingsessionscompleted• Numberofemailcommunicationssent• Numbertimespartnerswereconvened• Numberofcollaborativeprojectsengagedinbypartners
QualityofImplementation
Towhatextentweretheresourcesandsupportsof
highquality?
• Satisfactionratesgatheredfrompartnersorfieldactors• Statementsofsatisfactiongatheredfrompartnersorfield
actors• Participantreturnratesacrossmulti-dayworkshopsor
events• Clickratesforresourcesdisseminatedthroughemailor
website• Shareratesforresourcesdisseminatedthroughemailor
website• Alignmentbetweeninternalorexternalqualitycriteria
andactualresourceorsupportsprovided(e.g.,betweenqualitycriteriaforpolicybriefandactualpolicybriefsdrafted)
2.2 Short-term Outcomes
To what extent has the intermediary achieved desired short-term outcomes expected of a Diffusion Intermediary, such as increased partner capacity, a strong network, and improved system conditions?
Criteria GuidingQuestions SampleIndicators
PartnerUnderstanding
Towhatextentareindividualpartnersexpandingtheir
understandingofinnovation?
• Number/percentofpartnerswhoreportattainingnewknowledgeorskills
• Number/percentofpartnerswhoreporthavingexisting
Center for Public Research and Leadership
21
knowledgeorskillsreinforcedorchallenged• Extenttowhichpartnerscancorrectlyrecallknowledgeor
demonstrateskills• Extenttowhichpartnerscanspecifysimilaritiesand
differencesbetweenpriorandnewknowledgeandskills
PartnerPerceptions
Towhatextentareindividualpartnersdevelopingpositiveviewstowardtheinnovation?
• Number/percentofpartnerswhoareconfidentintheirabilitytousenewknowledgeandskills
• Number/percentofpartnerswhoreportnewknowledgeandskillschangedorreinforcedtheirviews,opinions,andbeliefsabouteducation
• Extenttowhichpartnershaveformedpositiveperceptionsofthenewknowledgeandskills
• Number/percentofpartnerswhointendtouseinformationandknowledgegainedfromKMoutput
NetworkForm
Whatindividualsororganizationsmakeupthe
network?
Howcantheconnectionsbetweennetworkmembers
becharacterized?
• Numberofmembers• Geographicdistributionofmembers• Sectorsrepresentedacrossmembership• Typesofmembersrepresented(i.e.,individualsversus
organizations)• Numberofconnectionsforagivenmember(degree)• Distancebetweenonememberandanother(closeness)• Numberoftimesamemberoccupiestheshortestpath
betweentwootherelementsorclusters(betweenness)• Densityofconnectionsaroundcentralhubs(networkcore)• Densityofconnectionaroundtheoutsideofthenetwork
(periphery)• Extentofcentralizationaroundasinglehub• Amountofinformationflowingthroughconnections• Typeofinformationflowingthroughconnections
NetworkFunction
Towhatextentdoesthenetworkhavetheresources,structures,andconditionsinplaceneededtooperateeffectivelyandefficiently?
• Amountoffinancialandmaterialresourcesavailablefornetworkoperations
• Efficiencyofcommunicationstructures• Effectivenessofdecision-makingstructures• Presenceofsupportiveconditions(e.g.,trust,
accountability,alignment)• Qualityofproductscreatedbythenetwork(research,
tools,etc.)
SystemPoliciesandPractices
Towhatextentaretheformalandinformalpoliciesandpracticesthatimpact
partnersenablingordemandinginnovation?
• Legislationthatdemandsinnovation• Humanresourcessystemsandstructuresthatenable
innovation• Accountabilitysystemsandstructuresthatenable
innovation• Professionalstandardsthataligntoinnovation• Organization-widestandardoperatingproceduresthat
Center for Public Research and Leadership
22
aligntoinnovation
SystemEngagementandPublic
Will
Towhatextentisthepublicdemandingandsupporting
change?
• Frequencyofpositivemediacoverage• Sizeandfrequencyofpublicgatheringstodemonstrate
supportforinnovation• Dollarsofpublicandprivatefundingdistributedtothe
regionforinnovation• Fundingandinfrastructureconditionsthatenable
innovation
2.3 Continuous Learning
To what extent is the intermediary gathering insights from its successes and failures and proactively responding to these insights?
Criteria GuidingQuestions SampleIndicators
UnderstandingofProgress
Towhatextenthastheintermediaryidentifiedareas
ofsuccessandfailure?
Whatlessonslearnedhastheintermediarypulledfromitssuccessesandfailuresthatcouldbenefititsownworkor
thelargerfield?
• Qualitativeand/orquantitativeprocessdatathatrevealtheextenttowhichtheintermediaryimplementeditsstrategyasintended
• Qualitativeand/orquantitativeoutcomesdatathatrevealtheextenttowhichthisintermediaryachieveddesiredoutcomes
• Documentedsuccessesandfailuresrevealedbyprocessandoutcomedata
• Department-orprogram-widefamiliaritywiththesesuccessesandfailures
• Documentedlessonslearnedthatlinkbacktosuccessesandfailures
• Department-orprogram-widefamiliaritywithlessonslearned
MidcourseCorrections
Towhatextentistheintermediaryadjustingitsstrategyororganizational
capacityasaresultoflessonslearned?
• Identifiablechangestoarticulatedstrategybasedonlessonslearned
• Observablechangestoenactedstrategybasedonlessonslearned
• Identifiablechangestointermediary’sinfrastructureoroperationsbasedonlessonslearned
• Department-orprogram-wideunderstandingofchangesandhowtheyconnecttolessonslearned
Phase III: Impacts on the Field ImpactsontheFieldassessesanintermediary’sultimategoals—thediffusionofinnovationacrossitsregionandimprovementstosocialoutcomesresultingfromdiffusion.AsthisframeworkisspecifictoanintermediaryintheK-12educationfield,thedimensions,criteria,andindicatorslistedcontainfieldspecificlanguage.However,theycanbeeasilymodifiedtoalignwithotherfields,suchashealthcareorjuvenilejustice.PhaseIIIcontainstwodimensions:
Center for Public Research and Leadership
23
• DiffusionofInnovation-Thedegreetowhichchangestonewandinnovativepracticesarescaledacrosstheregion.
• ImprovedStudentOutcomes-Theincreasedknowledgeandskillsofstudentsresultingfromchangestopractice.
ThecriteriausedtoassesstheDiffusiondimensionwereinformedbyresearchpertainingtoboththediffusionofinnovationandthescalingofeducationreform.Morespecifically,thedimensionsofSpread,Depth,Ownership,andSustainabilitywereinformedbyProfessorCynthiaCoburn’smultifaceteddefinitionoftheconceptofscale(Coburn,2003).Thisdefinitionlooksbeyondquantitativemeasuresofgrowthtoricherqualitativemeasuresofthequalityandembeddednessofchange.
Finally,thedimensionsofImprovedStudentOutcomesdescribedherewereinformedbythecurrentbestthinkingregardingtheknowledgeandskillsstudentsneedtobesuccessfulincollege,career,andlife.Indoingso,thedimensionsseektoestablishaholisticstudentoutcomesdefinition.However,thelanguageusedisgenericandsowilllikelyneedtobecustomizedtospecificregionalandeducationalcontexts.Eachdimensionisfurtherspecifiedincriteriaforevaluationaswellasguidingquestionsandsampleindicators.
Diffusion of Innovation
To what extent has the intermediary been able to diffuse the innovative practices it has championed broadly throughout the region?
Criteria GuidingQuestions SampleIndicators
Depth
Towhatextentarechangestoinstructionalpracticedeep
andconsequential,thusreflectingnewvaluesand
beliefs?
• Extentofchangestoteachers’underlyingbeliefsabouthowstudentslearn
• Extentofchangestoteachers’underlyingbeliefsaboutthenatureofcontent
• Extentofchangestoteachers’underlyingbeliefsaboutthedefinitionofhigh-qualityinstruction
• Extentofchangestostudentandteacherrolesintheclassroom
• Extentofchangestopatternsofteacherandstudenttalks• Extentofchangestohowteachersandstudentstreatone
another
Spread
Towhatextentarechangestoinstructionalpractice
expandingoutwardstomoreandmoreclassrooms,
schools,and/ordistricts?
Towhatextentarechangestoinstructionalpracticeexpandinginwardsand
influencingclassroom,school,anddistrictpoliciesandoperatingprocedures?
• Number/percentofindividualsororganizationschangingpractices
• Degreetowhichnewpracticesareinfluencingdecision-makingattheindividualororganizationallevel
• Specificchangestooperatingpoliciesandproceduresnotinitiallyrequiredbycompletedtobettersupportinnovation
• Degreetowhichteachersdrawonnewpedagogicalknowledge,values,andbeliefsinspacesnotinitiallytargetedbyreform
• Degreetowhichpedagogicalprinciplesofthereformareembeddedindistrictpolicyandprocedures
Center for Public Research and Leadership
24
Ownership
Towhatextentisauthorityforthereformbeingtakenonbythedistricts,schools,and
teachers?
• Presenceofstructuresandmechanismsforongoingteacherlearningaboutinnovation
• Presenceofestablishedstrategiesforcontinuingtofundinnovationactivities
• Degreetowhichleadershiphastakenresponsibilityforcontinuingtospreadinnovation
• Useofreform-centeredideasofstructuresinschoolordistrictdecision-making
Sustainability
Towhatextentdochangestopracticeremaininplaceafter
externalsupportsarenolongerpresent?
• Number/percentofteacherswhocontinueusinginnovativepracticesafterexternalsupportsareremoved
• Number/percentofteacherswhobeginusinginnovativepracticesafterexternalsupportsareremoved
Equity
Towhatextentarechangestopracticeoccurringequally
acrossdifferentsocioeconomicgroups?
• Gapindepthofdiffusionacrossdifferentgeographicareasandsocioeconomicgroups
• Gapinspreadofdiffusionacrossdifferentgeographicareasandsocioeconomicgroups
• Gapinownershipofdiffusionacrossdifferentgeographicareasandsocioeconomicgroups
• Gapinequityofdiffusionacrossdifferentgeographicareasandsocioeconomicgroups
Improved Student Outcomes
To what extent are social outcomes improving in parallel with the changes in individual and organizational practice?
Criteria GuidingQuestions SampleIndicators
Knowledge
Towhatextentarestudentsdevelopingincreased
understandingofimportantcontentknowledge?
• Rateofmasteryoncontentknowledgeassessments• Percentgrowthinmasteryoncontentknowledge
assessments• Extentofcontentknowledgeunderstanding
demonstratedthoughperformancetasks
CognitiveandMetacognitive
Skills
Towhatextentarestudentsdevelopingkeymental
processingskillsaswellastheabilitytomonitorandassess
theuseoftheseskills?
• Regularitywithwhichkeyskillsaredemonstrated• Number/percentofstudentsdemonstratingkeyskills• Extenttowhichkeyskillsaredemonstratedthrough
performancetasks,behaviors,ordiscussions
HabitsandDispositions
Towhatextentarestudentsdevelopingthequalitiesand
mindsetsneededtobesuccessfulincollege,career,
andlife?
• Regularitywithwhichkeyhabitsanddispositionsaredemonstrated
• Number/percentofstudentsdemonstratingkeyhabitsanddispositions
• Extenttowhichhabitsanddispositionsaredemonstratedthroughperformancetasks,behaviors,ordiscussions
Center for Public Research and Leadership
25
CLOSING Thesuccessfuldiffusionofnewpracticesisapersistentchallengewithinournation’sschoolsandschoolsystems.Educationalunderachievementandinequityarewickedproblemsthataredeeplyembeddedinthewaysinstitutionsandsocietyoperatepreventingeasysolutions.However,practitioners,funders,researchers,andotherseagertoseemeaningfulchangeinstudentopportunitiesandoutcomesareturningtoneworganizationalstructuresandstrategiestomovetheneedle.OneofthesestrategiesistheuseofnetworkedintermediarystructuresliketheNGLCRegionalFundsforBreakthroughSchoolsinitiative.
Thesestructureshavethepotentialtoprovidecustomizedsupportacrossadiverserangeofuniquelocalcontexts,whilealsofacilitatingcross-contextsharingandlearning.Tocapitalizeonthispotential,theintermediaryorganizationsthatoperateascrucialhubswithinthenetworkmustoftenworkatmultiplelevelssimultaneously.Theysupportandbuildthecapacityofindividualandorganizationalpartners,facilitatecommunicationandcollaborationacrosstheirpartnernetworksasawhole,andfostertheeconomic,social,andpoliticalconditionsthenetworkneedstoinnovate.
CPRLbeganitsworkwitharecognitionoftheimportantroleintermediaryorganizationsplayintransformingeducationandwiththedesiretoclarifythisroleandbuildgreatercapacitytomonitorandevaluateit.Thisreport,andtheaccompanyingtools,reflectwhatwehavecometounderstandabouthowintermediariesimpactchangewiththehopethatitwillsupportcurrentandfutureintermediaries,funders,andpartnerorganizationinbuildingsuccessfulnetworkedintermediarystructures.
Center for Public Research and Leadership
26
REFERENCES
Capacity Building
Connolly,P.&York,P.(2002).Evaluatingcapacity-buildingeffortsfornonprofitorganizations.ODPractitioner.34(4):33-39.
Dalkir,K.(2011).Introductiontoknowledgemanagement.Knowledgemanagementintheoryandpractice.MITPress:Cambridge,MA.
Kirkpatrick,D.&Kirkpatrick,J.(2005).Thefourlevel’biggestchallenge.Transferringlearningtobehavior:Usingthefourlevelstoimproveperformance.Berret-KoehlerPublishers:Oakland,CA.
Ohkubo,S.,Sullivan,T.M.,&Harlan,S.V.(2013).Guidetomonitoringandevaluatingknowledgemanagementinglobalhealthprograms.USAID.Retrievedfromhttps://www.k4health.org/sites/default/files/km-monitoring-and-eval-guide_0.pdf.
Diffusion and Scaling
Ashley,S.R.(2009).Innovationdiffusion:Implicationsforevaluation.InJ.M.Ottoson&P.Hawe(Eds.),Knowledgeutilization,diffusion,implementation,transfer,andtranslation:Implicationsforevaluation.NewDirectionsforEvaluation,124,35–45.
Coburn,C.(2003).Rethinkingscale:Movingbeyondnumberstodeepandlastingchange.EducationalResearcher,32(6):3-12.
Rogers,E.(1962).Diffusionofinnovation.TheFreePress:NewYork,NY.
Intermediaries
Anthony,E.K.&Austin,M.J.(2008)Theroleofanintermediaryorganizationinpromotingresearchinschoolsofsocialwork:ThecaseoftheBayAreaSocialServicesConsortium.SocialWorkResearch,32(4),287-293.
Blank,M.J.,Brand,B.,Deich,S.,Kazis,R.,Politz,B.,&Trippe,S.(2003).Localintermediaryorganizations:Connectingthedotsforchildren,youth,andfamilies.Retrievedfromhttp://www.jff.org/publications/local-intermediary-organizations-connecting-dots-children-youth-and-families.
Cooper,A.&Shewchuk,S.(2015).Knowledgebrokersineducation:Howintermediaryorganizationsarebridgingthegapbetweenresearch,policyandpracticeinternationally.EducationPolicyAnalysisArchives,23(118).
Fisched,D.J.(2005).Workforceintermediaries:Poweringregionaleconomiesinthenewcentury.Retrievedfromhttps://eric.ed.gov/?id=ED485943.
Foley,E.(2010).ApproachesofBill&MelindaGatesFoundation-fundedintermediaryorganizationstostructuringandsupportingsmallhighschoolsinNewYorkCity.Retrievedfromhttp://files.eric.ed.gov/fulltext/ED510236.pdf.
FSG.(2013).Backboneeffectiveness:27indicators.Retrievedfromhttp://www.collaborationforimpact.com/wp-content/uploads/2014/01/Backbone_Effectiveness_Indicators.pdf.
GatewayCenterforGiving.(2012).MeasuringtheworkofintermediariesintheSt.Louisregion.Retrievedfromhttp://www.centerforgiving.org/Portals/0/Documents/Measuring%20the%20Work%20of%20Intermediaries%20Paper%20FINAL.pdf.
Center for Public Research and Leadership
27
GatewayCenterforGiving.(2011).UnderstandingtheintermediaryinfrastructureinSt.Louis:Whotheyare,whattheydo,andcommonmisconceptions.Retrievedfromhttp://www.centerforgiving.org/portals/0/intermediaries%20report_final_v3.pdf.
GrantmakersforEffectiveOrganizations.(2013).Smarterrelationships,betterresults:Makingthemostofgrantmakers’workwithintermediaries.Retrievedfromhttp://www.geofunders.org/resource-library/all/record/a066000000AjtYcAAJ.
GrantmakersforEffectiveOrganizations.(2014).Whydofundersworkwithintermediaries?Retrievedfromhttp://www.geofunders.org/resource-library/all/record/a066000000H2hN3AAJ.
Honig,M.I.(2004).Thenewmiddlemanagement:Intermediaryorganizationsineducationpolicyimplementation.EducationalEvaluationandPolicyAnalysis,26(1),65-87.
Johnson,E.,Rothstein,F.andGajdosik,J.(2004),Theintermediaryroleinyouthworkerprofessionaldevelopment:Successesandchallenges.NewDirectionsforYouthDevelopment,104:51–64.
Lopez,E.M.,Kreider,H.&Coffman,J.(2005).Intermediaryorganizationsascapacitybuildersinfamilyeducationalinvolvement.UrbanEducation,40(1),78-105.
Lubienski,C.,Scott,J.,&DeBray,E.(2011).Theriseofintermediaryorganizationsinknowledgeproduction,advocacy,andeducationalpolicy.TeachersCollegeRecord,22.
Piha,S.&Pittman,K.Theimportantroleofintermediariesincollectiveimpactwork.TheForumforYouthInvestment.Retrievedfromhttps://collectiveimpactforum.org/sites/default/files/The%20Important%20Role%20of%20Intermediaries%20in%20Collective%20Impact%20Work.pdf.
Turner,S.,Merchant,K.,Kania,J.&Martin,E.(2012)Understandingthevalueofbackboneorganizations.StanfordSocialInnovationReview.http://www.fsg.org/publications/understanding-value-backbone-organizations-collective-impact.
U.S.DepartmentofHealthandHumanServices.(2008).Breakthroughperformance:10emergentpracticesofleadingintermediaries.Retrievedfromhttp://www.daremightythings.com/pdf/media_center/breakthrough_guide.pdf.
Scott,G.(2007).Fundercollaboratives:Aphilanthropicstrategyforsupportingworkforceintermediaries.Retrievedfromhttp://www.jff.org/sites/default/files/publications/FunderCollab.pdf.
Szanton,P.L.(2003).Towardmoreeffectiveuseofintermediaries.NewYork:FoundationCenter.Retrievedfromfoundationcenter.org/gainknowledge/practicematters/.
Wynn,J.R.(2000).Theroleoflocalintermediariesintheyouthdevelopmentfield.Retrievedfromhttps://www.chapinhall.org/sites/default/files/Intermediary-disc-paper.pdf
Network Weaving
Barabasi,A.(2002).Linked:Thenewscienceofnetworks.Cambridge,MA:PerseusPublishing.
Baran,P.(1964).Ondistributedcommunications:Introductiontodistributedcommunicationsnetworks.TheRANDCorporation.Retrievedfromhttp://www.rand.org/content/dam/rand/pubs/research_memoranda/2006/RM3420.pdf.
Bonbright,D.&Khagram,S.(2010).Innovationsforscalingimpactandkeystoneaccountability:Nextgenerationnetworkevaluation.Retrievedfromhttp://www.sdc-learningandnetworkingblog.admin.ch/uploads/2010/07/ME-of-Networks-2010.pdf.
Bryk,A.S.,Gomez,L.,Grunow,A.,&LeMahieu,P.(2015)Learningtoimprove.Cambridge,MA:HarvardEducationPress.
Center for Public Research and Leadership
28
CenterforAdvancementofCollaborativeStrategiesinHealth.(N.d.)Partnershipself-assessmenttool.Retrievedfromhttp://www.lmgforhealth.org/sites/default/files/Center_for_the_Advancement_of_Collaborative_Strategies_in_Health_%28CACSH%29_Parternship_Self-Assessments.pdf.
Creech,H.(2001).Measuringwhileyoumanage:Planning,monitoringandevaluatingknowledgenetworks.Winnipeg,Manitoba:InternationalInstituteforSustainableDevelopment.Retrievedfromhttps://www.iisd.org/pdf/2001/networks_evaluation.pdf.
Creech,H.(2001).Formfollowsfunction.Winnipeg,Manitoba:InternationalInstituteforSustainableDevelopment.Retrievedfromhttps://www.iisd.org/pdf/2001/networks_structure.pdf.
Creech,H.&Ramji,A.,(2004)Knowledgenetworks:Guidelinesforassessment.Winnipeg,Manitoba:InternationalInstituteforSustainableDevelopment.Retrievedfromhttps://www.iisd.org/pdf/2004/networks_guidelines_for_assessment.pdf
Fowler,C.H.&Christakis,N.A.(2009).Connected:Thesurprisingpowerofoursocialnetworksandhowtheyshapeourlives.NewYork,NY:Little,BrownandCompany.
Holley,J.(2012).Networkweaverhandbook.Athens,OH:NetworkWeaversPublishing.
Lowell,S.(2006).Buildingthefieldofdreams:Socialnetworksasasourceofsector-levelcapacityintheafter-schoolworld.Retrievedfromhttps://barrfdn-prod.s3.amazonaws.com/documents/14/attachments/Building_the_Field_of_Dreams.pdf?1421961228.
Malinsky,E.,&Lubelsky,Chad.(2011)Networkevaluation:Cultivatinghealthynetworksforsocialchange.Retrievedfrom:http://socialinnovation.ca/networkevaluation.
NetworkImpact&CenterforEvaluationInnovation.(2014).Thestateofnetworkevaluation:Aguide.Retrievedfromhttp://www.networkimpact.org/the-state-of-network-evaluation-a-guide/.
NetworkImpact&CenterforEvaluationInnovation.(2009).Networkhealthscorecard.Retrievedfromhttps://www.networkimpact.org/downloads/NH_Scorecard.pdf.
Plastrik,P.&Taylor,M.(2006).Netgains:Ahandbookfornetworkbuildersseekingsocialchange.Retrievedfromhttps://networkimpact.org/downloads/NetGainsHandbookVersion1.pdf.
Plastik,P.,Taylor,M.,&Cleveland,J.(2014).Connectingtochangetheworld:Harnessingthepowerofnetworksforsocialimpact.Washington,DC:IslandPress.
Scearce,D.(N.d.).Networkeffectivenessdiagnosticanddevelopmenttool.Retrievedfromhttp://www.workingwikily.net/network_diagnostic.pdf.
Scearce,D.(2011).Catalyzingnetworksforsocialchange:Afunder'sguide.Washington,D.C.:GrantmakersforEffectiveOrganizations.
Scearce,D.,Kasper,G.,McLeodGrant,H.(2010).Workingwikily.StanfordSocialInnovationReview.Retrievedfromhttps://ssir.org/articles/entry/working_wikily.
Taylor,M.,Whatley,A.,&Coffman,J.(2015).Networkevaluationinpractice:Approachesandapplications.TheFoundationReview,7(2).
Kania,J.&Kramer,M.(2011).Collectiveimpact.StanfordSocialInnovationReview.Retrievedfromhttp://www.fsg.org/publications/collective-impact.
Parkhurst,M.&Preskill,H.(2014).Learninginaction:Evaluatingcollectiveimpact.StanfordSocialInnovationReview.Retrievedfromhttps://ssir.org/articles/entry/learning_in_action_evaluating_collective_impact.
Preskill,H.,Parkhust,M.,&SplanskyJuster,J.(2014).Guidetoevaluatingcollectiveimpact.Retrievedfromhttp://www.fsg.org/publications/guide-evaluating-collective-impact.
Center for Public Research and Leadership
29
Systems Thinking and Change
TheBridgespanGroup.(2009).Thestrongfieldframework:Aguideandtoolkitforfundersandnonprofitscommittedtolarge-scaleimpact.Retrievedfromhttps://irvine-dot-org.s3.amazonaws.com/documents/64/attachments/strongfieldframework.pdf?1412656138
Meadows,D.(2008).Thinkinginsystems:Aprimer.WhiteRiverJunction,VT:ChelseaGreenPublishing.
Preskill,H.,Gopal,S.,Mack,K.,&Cook,J.(2015).Evaluatingcomplexity:Propositionsforimprovingpractice.FSG.Retrievedfromhttp://www.issuelab.org/resource/evaluating_complexity_propositions_for_improving_practice
Stroh,D.(2015).Systemsthinkingforsocialchange:Apracticalguidetosolvingcomplexproblems,avoidingunintendedconsequences,andachievinglastingresults.WhiteRiverJunction,VT:ChelseaGreenPublishing.
Latham,N.(2014).Apracticalguidetoevaluatingsystemschangeinahumanservicessystemcontext.LearningforAction.Retrievedfromhttps://drive.google.com/file/d/0BxI97tx1dL4Zb19GOEE0Y1Q3SGM/view?pref=2&pli=1
Hargreaves,M.(2010).Evaluatingsystemchange:Aplanningguide.MathematicaPolicyResearch,Inc.Retrievedfromhttps://drive.google.com/file/d/0BxI97tx1dL4ZRVYtODJld0drX28/view?pref=2&pli=1.
Coffman,J.(2009).Auser’sguidetoadvocacyevaluationplanning.HarvardFamilyResearchProject.Retrievedfromhttp://www.hfrp.org/.
Coffman,J.&Reed,E.Uniquemethodsinadvocacyevaluation.Retrievedfromhttp://www.pointk.org/resources/files/Unique_Methods_Brief.pdf.
HarvardFamilyResearchProject(2007).Advocacyandpolicychange.TheEvaluationExchange.13(1):1-32.
Forti,M.(2012).Measuringadvocacy-Yeswecan!StanfordSocialInnovationReview.Retrievedfromhttps://drive.google.com/file/d/0BxI97tx1dL4ZbU5zLWFVX0hzb0E/view.
Reisman,J.,Gienapp,A.,&Stachowiak,S.(2007).Aguidetomeasuringadvocacyandpolicy.TheAnnieE.CaseyFoundation.Retrievedfromhttps://drive.google.com/file/d/0BxI97tx1dL4ZX09HTktUYld4MjQ/view.
ADDITIONALRESOURCES
Measurement Toolkit for Intermediary Organizations AppendixtoIntermediaryOrganizationsandEducationInnovation
CenterforPublicResearchandLeadership
C E N T E R F O R P U B L I C R E S E A R C H A N D L E A D E R S H I P
31
Diffusion Levers Evaluation Toolkit: Capacity Building
TheDiffusionIntermediaryEvaluationFrameworkincludestwocriteriaforassessingshort-termCapacityBuildingoutcomes:PartnerUnderstandingandPartnerPerceptions.Thesecriteriaarebasedonareviewoftheliteraturelistedattheendofthissection.Thisreviewidentifiedacommonsetofstepsindividualsmovethroughastheydecidewhetherornottoadoptanewpracticeincludingbuildingawarenessandunderstanding,formingpositiveperceptions,implementing,andfinallyinternalizingthenewpractices.Thefirsttwostepsinformedthecriterialistedhere.LatterstepsarepartofPhase3intheFramework.
Evaluative Criteria, Questions, and Sample Indicators
Criteria GuidingQuestions SampleIndicators
PartnerUnderstanding
Towhatextentareindividualpartnersexpandingtheirunderstandingof
innovation?
• Number/percentofpartnerswhoreportattainingnewknowledgeorskills
• Number/percentofpartnerswhoreporthavingexistingknowledgeorskillsreinforcedorchallenged
• Extenttowhichpartnerscancorrectlyrecallknowledgeordemonstrateskills
• Extenttowhichpartnerscanspecifysimilaritiesanddifferencesbetweenpriorandnewknowledgeandskills
PartnerPerceptions
Towhatextentareindividualpartners
developingpositiveviewstowardtheinnovation?
• Number/percentofpartnerswhoareconfidentintheirabilitytousenewknowledgeandskills
• Number/percentofpartnerswhoreportnewknowledgeandskillschangedorreinforcedtheirviews,opinions,andbeliefsabouteducation
• Extenttowhichpartnershaveformedpositiveperceptionsofthenewknowledgeandskills
• Number/percentofpartnerswhointendtouseinformationandknowledgegainedfromKMoutput
Tools
SampleSurveyQuestionstoAssessPerceptionsofNewInstructionalPracticesCenterforPublicResearchandLeadershipThesesamplequestionscanbecustomizedtocreateasurveyinstrumentforassessingpartnerperceptionsbeforeorafteracapacity-buildingevent.Thequestionswereinformedbydiffusiontheory,specificallythePerceivedAttributesofInnovationframework(Rogers,1962).
PartnerUnderstanding
ü PartnerPerception
SampleSurveyQuestionstoAssessUnderstandingandIntentiontoActCenterforPublicResearchandLeadership
C E N T E R F O R P U B L I C R E S E A R C H A N D L E A D E R S H I P
32
Thesesamplequestionscaninformthecreationofasurveyinstrumenttoassesspartnerlearningandintentiontoactfollowingacapacity-buildingevent. ü Partner
Understanding
ü PartnerPerception
Resources Connolly,P.&York,P.(2002).Evaluatingcapacity-buildingeffortsfornonprofitorganizations.ODPractitioner.34(4):33-
39.
Dalkir,K.(2011).Introductiontoknowledgemanagement.Knowledgemanagementintheoryandpractice.MITPress:Cambridge,MA.
Kirkpatrick,D.&Kirkpatrick,J.(2005).Thefourlevel’biggestchallenge.Transferringlearningtobehavior:Usingthefourlevelstoimproveperformance.Berret-KoehlerPublishers:Oakland,CA.
McCombs,B.L.&Whisler,J.TheLearner-centeredclassroomandschool.(1997).Wiley&Sons,Inc.SanFrancisco:CA.
Ohkubo,S.,Sullivan,T.M.,&Harlan,S.V.(2013).Guidetomonitoringandevaluatingknowledgemanagementinglobalhealthprograms.USAID.Retrievedfromhttps://www.k4health.org/sites/default/files/km-monitoring-and-eval-guide_0.pdf.
C E N T E R F O R P U B L I C R E S E A R C H A N D L E A D E R S H I P
33
PERCEPTIONSOFNEWINSTRUCTIONALPRACTICESInformedbyDiffusionofInnovation,EverettRoger’s(1962)
RelativeAdvantage StronglyAgree Agree Disagree Strongly
Disagree
RelativeAdvantage
1. [Thenewinstructionalpractices]willmakemyjobeasier.
2. IwillbemoreeffectiveasateacherifIimplement[thenewinstructionalpractices].
3. Myexistinginstructionalpracticesbuildstudentknowledgeandskillsjustaswellas[thenewinstructionalpractices]will.
4. Idon’tthink[thenewinstructionalpractices]willtransformmystudents’learningenoughtobeworthtrying.
Compatibility
5. [Thenewinstructionalpractices]alignwellwiththewaymyschoolsystemoperates.
6. [Thenewinstructionalpractices]donotfitwellwiththewayIteach.
7. Ibelievethat[thenewinstructionalpractices]aretherightwaytoteacherstudents.
8. Mypeerswillnotsupport[thenewinstructionalpractices].
Thesurveyitemsbelowcanbecustomizedandusedtoassesstheextenttowhichindividualsaredevelopingpositiveperceptionsofanewpracticeorsetofpractices.Theitemsareorganizedintofivecategories:
• RelativeAdvantage-Thedegreetowhichaninnovationisperceivedasbeingbetterthantheideaitsupersedes.
• Compatibility-Thedegreetowhichaninnovationisperceivedasconsistentwiththevalues,pastexperiences,andneedsofpotentialadopters.
• Complexity-Thedegreetowhichaninnovationisperceivedasrelativelydifficulttounderstandanduse.• Trialability-Thedegreetowhichaninnovationmaybeexperimentedwithonalimitedbasis.• Observability-Thedegreetowhichtheresultsofaninnovationarevisibletoothers.
C E N T E R F O R P U B L I C R E S E A R C H A N D L E A D E R S H I P
34
ResponseKeyQuestions1,2,5,7,10,13,16,19Stronglyagree=0Agree=1Disagree=2StronglyDisagree=3
Questions3,4,6,8,9,11,12,14,15,17,18,20Stronglyagree=3Agree=2Disagree=1StronglyDisagree=0
RelativeAdvantage StronglyAgree
Agree Disagree StronglyDisagree
Complexity
9. IhavetroubleimaginghowIcouldpossiblyimplement[thenewinstructionalpractices].
10. Overall,Ibelieve[thenewinstructionalpractices]wouldbeeasyformycolleaguestoimplement.
11. Iwouldstruggletoexplain[thenewinstructionalpractices]tootherpeopleinmyschoolsystem.
12. Gettingtheresourcesandsupportsneededfor[thenewinstructionalpractices]wouldbecomplicatedandtimeconsuming.
Trialability
13. Icouldgobacktomyclassroomtomorrowandeasilytrysomeof[thenewinstructionalpractices].
14. Experimentingwith[thenewinstructionalpractices]insmallwayswouldn’treallybepossible.It’sallornothing.
15. Itwouldbechallengingtotestout[thenewinstructionalpractices]withoutgettingadditionalsupportfrommyschoolsystem.
16. Iseepiecesof[thenewinstructionalpractices]thatIcoulddemonstrateformypeerswhenIgetbacktowork.
Observability
17. Ihavetroublevisualizing[thenewinstructionalpractices].
18. WhenIgointoclassroomswhereteachersareimplementing[thenewinstructionalpractices]Ihavetroubleidentifyingwhatisdifferent.
19. Classroomsusing[thenewinstructionalpractices]lookverydifferentfrommostotherclassroomsI’veseen.
20. IwishIcouldsee[thenewinstructionalpractices]inactionbutit’shardtofindhigh-qualityexamples.
C E N T E R F O R P U B L I C R E S E A R C H A N D L E A D E R S H I P
35
SAMPLEPARTNERCAPACITY-BUILDINGQUESTIONS
Learning
Towhatdegreedoyoufeelyouhaveimprovedknowledgeorskillsinthefollowingareas?
Pleasecheckoneboxtotherightofeacharea.
NoImprovement
MinimalImprovement
ModerateImprovement
LargeImprovement
1.[Knowledgeorskillcategory]
2.[Knowledgeorskillcategory]
3.[Knowledgeorskillcategory]
4.[Knowledgeorskillcategory]
5.[Knowledgeorskillcategory]
6.Howwouldyoucharacterizetheoverallimprovementinyourknowledge/skillsasaresultofthisevent?
� NoImprovement� MinimalImprovement� ModerateImprovement� LargeImprovement
7.Howlikelyisitthatyouwillbeabletoapplytheknowledge/skillslearnedinthiseventwithoutsupportwhenyoureturntoyourschool?
� VeryUnlikely� SomewhatUnlikely� SomewhatLikely� VeryLikely
8.Pleasedescribeanyfactorsthatmayinterferewithyourapplyingtheknowledgeandskillsfromtheeventwhenyoureturntoyourschool.
Thesurveyitemsbelowcanbecustomizedandusedtoassesstheextenttowhichpartnersdevelopednewunderstandingsandpositiveperceptionsbecauseofcapacity-buildingactivities.
C E N T E R F O R P U B L I C R E S E A R C H A N D L E A D E R S H I P
36
PerceptionandIntention
Forthefollowingstatementsindicatepossibleactionsyoumighttakeafterthisevent.Indicatetheextenttowhicheachstatementistrueorfalseforyou.9.Ihaveidentified1-2specificstrategiesasaresultofthiseventthatIwillfocusonintegratingintomypracticewithinthenextmonth.
� DefinitelyFalse� ProbablyFalse� ProbablyTrue� DefinitelyTrue
10.Inthenextmonth,IwillshareatleastsomeofwhatIlearnedinthiseventwithacolleague.
� DefinitelyFalse� ProbablyFalse� ProbablyTrue� DefinitelyTrue
11.Inthenextmonth,IwillseekoutmoreinformationrelatedtowhatIlearnedtoday.
� DefinitelyFalse� ProbablyFalse� ProbablyTrue� DefinitelyTrue
12.Whataspectsoftheeventweremostusefultoyou?
C E N T E R F O R P U B L I C R E S E A R C H A N D L E A D E R S H I P
37
Diffusion Levers Evaluation Toolkit: Network Weaving
TheDiffusionIntermediaryEvaluationFrameworkincludestwocriteriaforassessingshort-termNetworkWeaving
outcomes:NetworkFormandNetworkFunction.Thesecriteriaarebasedonascanofthenetworkevaluationliteraturelistedattheendofthissection.Thescanidentifiedthreeprimaryareasforevaluationspecifictonetworks.These
includethetwoareaslistedhere—form(i.e.structure,shape,connectivity)andfunction(i.e.health,vibrancy,
operations)—aswellasnetworkimpacts.ThethirdareawasincorporatedintoPhase3oftheFrameworkbecauseit
resultsfromacombinationofallthreeDiffusionLevers.
Evaluative Criteria, Questions, and Sample Indicators
Criteria GuidingQuestions SampleIndicators
NetworkForm
Whatindividualsororganizationsmakeup
thenetwork?
Howcantheconnectionsbetweennetwork
membersbecharacterized?
• Totalnumberofmembers
• Geographicdistributionofmembers
• Sectorsrepresentedbymembers
• Typesofmembers(i.e.individualsversusorganizations)
• Numberofconnectionsforagivenmember(degree)
• Distancebetweenonememberandanother(closeness)
• Frequencywithwhichamemberoccupiesshortestpathbetweentwo
othermembers(betweenness)
• Densityofconnectionsaroundnetworkcore
• Densityofconnectionsaroundperiphery
• Extentofcentralizationaroundasinglehub
• Amountofinformationflowingthroughconnections
• Typeofinformationflowingthroughconnections
NetworkFunction
Towhatextentdoesthenetworkhaveresources,structures,andconditions
neededtooperateeffectivelyandefficiently?
• Financialresourceavailablefornetworkoperations
• Materialresourceavailablefornetworkoperations
• Efficiencyofcommunicationstructures
• Effectivenessofdecision-makingstructures
• Peeraccountabilityacrossthenetwork
• Trustbetweenmembers
• Balanceofparticipationacrossmembers
• Extentofalignmentbetweenmembers
• Workproducedasaresultofcollaboration
C E N T E R F O R P U B L I C R E S E A R C H A N D L E A D E R S H I P
38
Tools
KumuKumuInc.Awed-basednetwork-andsystem-mappingplatformwithasocialnetwork
analysisoptionthatallowsfornetworkdatatobemanuallyenteredoruploaded
throughExcelorGoogleSheets.Kumuallowsausertocalculatevariousnetwork
metrics(degree,closeness,betweeness,etc.)aswellascodetheresultingmap
usingdifferentcolors,gradations,andlinetypes.
ü NetworkForm
NetworkFunction
MapYourNetworkbyHandNetworkWeaverHandbookBasicdirectionsformappingnetworksbyhand.Thisprocessissuitableforsmall
networksandcanbeengagedincollaborativelyasanetworkweavingactivity.
ü NetworkForm
NetworkFunction
Network Design and Assessment Scales TheCenterforPublicResearchandLeadershipAplanningandreflectiontoolthatidentifiestencharacteristicsnetworksmay
varyinrelationtoalongwithscalesandguidingquestionsforself-assessment.
ThistoolwasinspiredbytheMonitorInstitute’sresearchonnetworkdesign.
ü NetworkForm
NetworkFunction
Network Effectiveness Diagnostic and Development Tool MonitorInstituteAflexibletoolforassessingandstrengtheninganetwork’sfunction.Itidentifies
variouscharacteristicsandalignedattributesthataredesirableforanetworkto
have.Thetoolalsoprovidesdifferentcharacteristicsfornetworkswithclear
boundariesandmembers(“boundednetworks”)andnetworkswithambiguous
boundariesandmembers(“unboundednetworks”).
NetworkForm
ü NetworkFunction
Network Health Score Card NetworkImpactandCauseCommunicationsA22-questionsurveythatassessesnetworkhealth,orfunction,infourareas(1)
networkpurpose,(2)networkperformance,(3)networkoperations,and(4)
networkcapacity.Eachquestionaskstherespondenttoratethedegreetowhich
theyagreewithaquestionona5-pointscale.
NetworkForm
ü NetworkFunction
PartnerSelf-AssessmentToolCenterforAdvancementofCollaborativeStrategiesinHealth
Avalidatedsurveycontaining67questionspertainingtocollaborationspread
acrosseleventopics:synergy,leadership,efficiency,administrationand
management,non-financialresources,financialandothercapitalresources,
decisionmaking,benefitsofparticipation,drawbacksofparticipation,benefits
anddrawbacksofparticipatinginthepartnership,andsatisfactionwith
participation.
NetworkForm
ü NetworkFunction
C E N T E R F O R P U B L I C R E S E A R C H A N D L E A D E R S H I P
39
Sample Network Form Questions TheCenterforPublicResearchandLeadershipSamplequestionstocustomizeanduseingatheringinformationontheformof
anetwork.Thedatacollectedcanbevisualizedthroughasocialnetworkanalysis
orbecompiledandreportedoutthroughmoretraditionaltablesandgraphs.
ü NetworkForm
NetworkFunction
Sample Interview Questions for Network Members TheCenterforPublicResearchandLeadershipQuestionsthatcanbecustomizedandusedininterviewsorfocusgroupswith
NetworkMembers.Thesequestionswereinformedbyvarioussourcesandcover
boththeformandthefunctionofthenetwork.
ü NetworkForm
ü NetworkFunction
Resources CenterforAdvancementofCollaborativeStrategiesinHealth.(N.d.)Partnershipself-assessmenttool.Retrievedfrom
http://www.lmgforhealth.org/sites/default/files/Center_for_the_Advancement_of_Collaborative_Strategies
_in_Health_%28CACSH%29_Parternship_Self-Assessments.pdf.
Creech,H.&Ramji,A.,(2004)Knowledgenetworks:Guidelinesforassessment.Winnipeg,Manitoba:International
InstituteforSustainableDevelopment.
Creech,H.(2001).Measuringwhileyoumanage:Planning,monitoringandevaluatingknowledgenetworks.Winnipeg,
Manitoba:InternationalInstituteforSustainableDevelopment.
Creech,H.(2001).Formfollowsfunction.Winnipeg,Manitoba:InternationalInstituteforSustainableDevelopment.
Holley,J.(2012).Networkweaverhandbook.Athens,OH:NetworkWeaversPublishing.
Malinsky,E.,&Lubelsky,Chad.(2011)Networkevaluation:Cultivatinghealthynetworksforsocialchange.Retrievedfrom:http://socialinnovation.ca/networkevaluation.
InnovationsforScalingImpactandKeystoneAccountability:NextGenerationNetworkEvaluation(DavidBonbright&
SanjeevKhagram,2010)
MonitorInstiture.Engage:HowFundersCanSupportandLeverageNetworkforSocialImpact.Retrievedfromhttp://engage.rockefellerfoundation.org/what-could-a-network-help-me-achieve/what-network-design-would-
be-the-most-useful/
NetworkImpact&CenterforEvaluationInnovation.(2014).Thestateofnetworkevaluation:Aguide.Retrievedfrom
http://www.networkimpact.org/the-state-of-network-evaluation-a-guide/.
NetworkImpact&CenterforEvaluationInnovation.(2009).Networkhealthscorecard.Retrievedfrom
https://www.networkimpact.org/downloads/NH_Scorecard.pdf.
Plastik,P.,Taylor,M.,&Cleveland,J.(2014).Connectingtochangetheworld:Harnessingthepowerofnetworksforsocialimpact.Washington,DC:IslandPress.
Plastrik,P.&Taylor,M.(2006).Netgains:Ahandbookfornetworkbuildersseekingsocialchange.Retrievedfrom
https://networkimpact.org/downloads/NetGainsHandbookVersion1.pdf.
Scearce,D.(N.d.).Networkeffectivenessdiagnosticanddevelopmenttool.Retrievedfromhttp://www.workingwikily.
net/network_diagnostic.pdf.
Taylor,M.,Whatley,A.,&Coffman,J.(2015).Networkevaluationinpractice:Approachesandapplications.TheFoundationReview,7(2).
C E N T E R F O R P U B L I C R E S E A R C H A N D L E A D E R S H I P
40
MAPYOURNETWORKBYHANDSource:NetworkWeaverHandbook,JuneHolley,2012
Steps
1. Haveeveryoneinthegroupdrawacircleoraddamovablestickertorepresentanodeforoneselfandlabelit.
Thendrawnodesforothersinthenetworkyouareworkingwithclosely.
2. Drawlinesbetweenyouandtheothersusingthekeybellow.Thendrawlinesbetweenanyofthepeopleinthe
networkwhoknoweachother.ThisistheCore.
3. Nextdrawnodesforotherindividualsandorganizationsthatyouareworkingwithontheproject,butless
frequently,anddrawlinestotheindividualsinyourprojectnetworkwhohavearelationshipwiththat
individual.ThisisthePeriphery.
4. Addotherindividualsorgroupstheyareconnectedto(whomightaddvaluetoyournetwork)anddrawlines
connectingthem.ThisinyourPotentialNetwork.
5. Aroundtheoutsideedge,putnodesforindividualsorgroupsyouarenotworkingwithinthisproject,butwho
couldaddvaluetotheprojectiftheywereinvolved.Thesemightbepeoplewithexpertise,peoplefromother
communitieswhohavebeeninnovatinginwaysthatwouldbeofvaluetoyournetwork,orpeoplewhoare
oftenleftoutofprojects.ThisisalsoyouPotentialNetwork.
Insmallnetworks,itisfeasibletomapthenetworkbyhandonalargeposter,awall,oranonlinemind-mapping
platformlikeMindMeister.Todothissuccessfully,itishelpfultohaveallormostnetworkmemberspresent.
RelationshipKey
Havecollaboratedonaproject(color1)
Gotothemfororofferadvice,information,resources,andexpertise
(color2)
Knowthepersonbuthaven’tcollaboratedorexchangesresourcesyet
(color3)
C E N T E R F O R P U B L I C R E S E A R C H A N D L E A D E R S H I P
41
NETWORKDESIGNSCALESANDQUESTIONS
InformedbyWhatNetworkDesignWouldbetheMostUseful?,TheMonitorInstituteand
NetworkEvaluation:CultivatingHealthyNetworksforSocialChange,EliMalinskyandChadLubelsky,2011
PARTA:NETWORKDESIGN
Completethescalesandquestionsbelowbasedonthevisionforthenetwork6monthsfromtoday.
Size
� � � � � � � �
Hundreds Handful
Howmanymembersdoweaspiretohave?
Whatisourreasonforthisdesignchoice?
SectorRepresentation
� � � � � � � �
SameSector MixedSector
Whatsectorsdowehopearerepresented?
Whatisourreasonforthisdesignchoice?
Networkscantakevariousforms.UsethescalesandalignedquestionsinPartAofthissurveytoclarifyand
documentnetworkdesignchoices.UsethescalesandalignedquestionsinPartBtoassessprogress.
C E N T E R F O R P U B L I C R E S E A R C H A N D L E A D E R S H I P
42
Geography
� � � � � � � �
City-Based Global
Whatgeographicareadoweintendtocover?
Whatisourreasonforthisdesignchoice?
MemberType
� � � � � � � �
Systems Individuals
Whattypeofmembersdoweseektohave?
Whatisourreasonforthisdesignchoice?
Orientation
� � � � � � � �
Action Learning
Whatorientationdoweplantotake?
Whatisourreasonforthisdesignchoice?
C E N T E R F O R P U B L I C R E S E A R C H A N D L E A D E R S H I P
43
Centralization
� � � � � � � �
Centralized Decentralized
Whatdegreeofcentralizationdoweplantoestablish?
Whatisourreasonforthisdesignchoice?
GoalAlignment
� � � � � � � � �
Maximum Minimum
Whatlevelofgoalalignmentshouldweseekacrossmembers?
Whatisourreasonforthisdesignchoice?
Entry
� � � � � � � �
EntirelyClosed EntirelyOpen
Whatlevelofopennesswillwehavetonewmembers?
Whatisourreasonforthisdesignchoice?
C E N T E R F O R P U B L I C R E S E A R C H A N D L E A D E R S H I P
44
Leadership
� � � � � � � �
Distributed Top-down
Whatwillourapproachtoleadershipbe?
Whatisourreasonforthisdesignchoice?
Focus
� � � � � � � �
Established Evolving
Whatconstancyoffocusdoweaspiretohave?
Whatisourreasonforthisdesignchoice?
ValueProposition
� � � � � � � �
Individual Collective
Whatdegreeofdifferenceinmembervaluepropositionsdowewant?
Whatisourreasonforthisdesignchoice?
C E N T E R F O R P U B L I C R E S E A R C H A N D L E A D E R S H I P
45
NETWORKPROGRESSSCALESANDQUESTIONS
PARTB:NETWORKPROGRESS
Completethescalesandquestionsbelowbasedonthenetworkscurrentstatus.
Size
� � � � � � � �
Hundreds Handful
Howmanymembersdowehave?
Howdoweknowthis?Whatevidencedowehave?
SectorRepresentation
� � � � � � � �
SameSector MixedSector
Whatsectorsarerepresented?
Howdoweknowthis?Whatevidencedowehave?
Networkscantakevariousforms.UsethescalesandalignedquestionsinPartAofthissurveytoclarifyand
documentnetworkdesignchoices.UsethescalesandalignedquestionsinPartBtoassessprogressagainstinitial
goals.
C E N T E R F O R P U B L I C R E S E A R C H A N D L E A D E R S H I P
46
Geography
� � � � � � � �
City-Based Global
Whatgeographicareadowecover?
Howdoweknowthis?Whatevidencedowehave?
MemberType
� � � � � � � �
Systems Individuals
Whattypeofmembersdowehave?
Howdoweknowthis?Whatevidencedowehave?
Orientation
� � � � � � � �
Action Learning
Whatorientationdowetake?
Howdoweknowthis?Whatevidencedowehave?
C E N T E R F O R P U B L I C R E S E A R C H A N D L E A D E R S H I P
47
Centralization
� � � � � � � �
Centralized Decentralized
Whatdegreeofcentralizationwasestablished?
Howdoweknowthis?Whatevidencedowehave?
GoalAlignment
� � � � � � � �
Maximum Minimum
Whatlevelofgoalalignmentexistsacrossmembers?
Howdoweknowthis?Whatevidencedowehave?
Entry
� � � � � � � �
EntirelyClosed EntirelyOpen
Whatlevelofopennessdowehavetonewmembers?
Howdoweknowthis?Whatevidencedowehave?
C E N T E R F O R P U B L I C R E S E A R C H A N D L E A D E R S H I P
48
Leadership
� � � � � � � �
Distributed Top-down
Whathasourapproachtoleadershipbeen?
Howdoweknowthis?Whatevidencedowehave?
Focus
� � � � � � � �
Established Evolving
Whatconstancyoffocusdidwehave?
Howdoweknowthis?Whatevidencedowehave?
ValueProposition
� � � � � � � �
Individual Collective
Whatdegreeofdifferenceinmembervaluepropositionsdowehave?
Howdoweknowthis?Whatevidencedowehave?
OVERVIEW
This is a tool for assessing the health of your network, and exploringactions to take to develop or strengthen it. This tool is intended for useby individuals working within or through social change networks.
Instructions:
1. Begin by identifying your network as either:
1. BOUNDED: a network with clear boundaries.The participants are known.
1. UNBOUNDED: a network with fuzzy boundaries.The participants are not all known.
2. Rate your network (high, medium, low) against attributes within eight areas ofnetwork health
3. Step back and jot down notes on your network’s performance in each area ofhealth. Note whether or not this is a priority area for strengthening. Dependingon where your network is at in its lifecycle, different attributes may be atdifferent levels of priority
4. Elicit multiple perspectives on your network’s health. Ask leaders from acrossyour network to take the diagnostic. Compare and aggregate results
5. Next, link your priority areas with actions for strengthening networks. Theactions are by no means prescriptive and do not correlate directly to theattributes within each area of network health. They are meant to spur yourthinking about the range of specific steps you might take to strengthen yournetwork
Sources: This tool was created with inputs from multiple sources – most significantlyresearch done by Monitor Institute for Packard Foundation grantees in 2008-09, and thework of the following network experts: Beth Kanter, June Holley, Marty Kearns, Pete Plastrikand Madeleine Taylor, Clay Shirky, and Jane Wei-Skillern.
Created by the Monitor Institute, www.monitorinstitute.com
Please direct queries about this tool to [email protected]
This work is licensed under the Creative Commons Share Alike 3.0Unported License.
Characteristic Desired Attributes HIGH MED LOW
Notes on OverallPerformance Potential Actions to Strengthen the Network
Value
Clearly articulated give and get forparticipants
• Engage network participants in framing network purpose and goals
• Clearly articulate value the network aspires to deliver to participants
• Regularly test the network value with participants and refine / update asneeded
• Ensure that the network is accountable to the community it seeks to serveDelivers value/outcomes to participants
Network value propositions are aligned andevolve with participant demand
Participation
Participation includes the necessary diversityknowledge, skills and capacity to achievepurpose
• Map the network --to determine who is in the network and how connectedthey are, and identify new participants and strategies for engaging them
• Determine network boundaries – who is in and who is out. Determine howporous these boundaries should be
• Welcome and orient new participants, develop a standard process fordoing so
• Hire a network weaver to bring the right participants into the mix andincrease connectivity throughout the network
• Create workspaces that invite community building and participation—online and in-person
• Identify ‘open triangles’ and close them – identify two people who youknow and who would benefit from knowing each other and introducethem
• Encourage small collaborative projects among just two or three networkparticipants
• Codify a code of conduct, share it broadly, and live by it
New participants can quickly becomeproductive within the network
High voluntary engagement in the network
Participants have a formal or informal codeof conduct and high level of trust with oneanother
Participants regularly interact andcollaborate with one another without goingthrough a central hub
Form
Network has a concept of its structure, how itsuits its purpose, and how it might evolve(e.g. from hub and spoke to multi-hubstructure)
• Map the network in order to visualize structure, diagnose strengths andweaknesses, and identify strategies for growing the network
• Facilitate an open strategic conversation that encourages participationfrom across the network; solicit the ‘wisdom of the crowd’
• Grow the number of people on the periphery of the network and createopportunities for their fresh ideas to flow into the network
• Create an innovation fund – a dedicated resource for cutting edge workthat creates a mandate for risk-taking
Balance of top-down and bottom-upstrategies for doing the work of the network
Network spaces invite self-organized action
1
Characteristic Desired Attributes HIGH MED LOW
Notes on OverallPerformance Potential Actions to Strengthen the Network
Leadership
Leaders inspire and help participantsrecognize and work towards common goals
• Identify individuals with strong group process skills and aptitude for IT-enabled collaboration to take on more responsibility
• Develop a system for diversifying and refreshing leadership
• Get out of the way – target opportunities for network participants toconnect and collaborate directly
• Bridge difference. Connect people and ideas that normally don’t gotogether
Leaders seek out opportunities to highlightand bridge difference in service of networkgoals
Leadership is shared. Responsibility andcontrol is spread throughout the network
Leadership is refreshed and renewed toreflect the network as it evolves
Governance
Governance is reflective of diverseconstituencies in the network andtransparent
• Formalize governance system with an eye to identifying opportunities toshare decision-making power
• Create mechanisms for voices from the periphery to influence decision-making
Governance is formalized in a group,committee or board (not a single person)
Governing body rotates its members overtime
Connection
Network is resilient. If some highly connectedparticipants leave, the network remainsstrong
• Align communication tools with what members are comfortable using orcan be trained to use. Don’t assume the network should adopt the latestadvances
• Follow the 1/10/100 rule :1% create content, 10% comment on it, and100% view content
• Allocate time and budget for designing, facilitating and maintaining onlinenetwork communications
• Look to young people to guide your use of social media
• Design shared spaces –online forums and in-person common spaces thatencourage interactions
• Seek out lower cost opportunities for connecting network participants in-person – e.g. host a reception at a commonly attended conference
Ample shared space, online and in-person,allowing participants to easily connect
Network use of social media tools andstrategies are appropriate given participantskills and habits
Network use of social media tools andstrategies are a good fit for types ofinteractions needed to meet the purpose
2
Characteristic Desired Attributes HIGH MED LOW
Notes on OverallPerformance Potential Actions to Strengthen the Network
Capacity
Participants know where resources are in thenetwork—knowledge , skills, and capacity
• Create systems to help participants find and share relevant expertise
• Broadcast basic needs to the network and tap excess capacity to fill them
• Surface the talent in the network. Don’t assume external expertise needsto be brought in to address network needsNetwork can identify and prioritize filling
knowledge, skills and capacity gaps
Effective model for financial sustainability inplace
Learning &Adaptation
Network gathers feedback, and captureslearning as stories
• Create mechanisms for regular gathering of feedback from networkparticipants
• Invite key network participants to develop a shared set of metrics
• Create a network map that will serve as a baseline measurement. Map thenetwork again in 1-2 years, or once there has been enough time for changein the network. Compare the two maps to assess change in the system
• Create a mechanism – with dedicated resources – for ongoing capture oflearning and stories throughout the network
Network has an agreed upon desired impactand a common set of metrics to measure thatimpact
Network regularly measures, evaluates, andreflects on its impact to refine its goals andactivities
3
Characteristic Desired Attributes HIGH MED LOW
Notes on OverallPerformance Potential Actions to Strengthen the Network
Value
Clearly articulated give and get forparticipants
• Engage network participants in framing network purpose and goals
• Clearly articulate value the network aspires to deliver to participants
• Regularly test the network value with participants and refine / update asneeded
• Ensure that the network is accountable to the community it seeks to serveDelivers value/outcomes to participants
Network value propositions are aligned andevolve with participant demand
Participation
Participation includes the necessary diversityknowledge, skills and capacity to achievepurpose
• Map the network --to determine who is in the network and how connectedthey are, and identify new members and strategies for engaging them
• Determine network boundaries – who is in and who is out. Determine howporous these boundaries should be
• Welcome and orient new participants; develop a standard process fordoing so
• Hire a network weaver to bring the right participants into the mix andincrease connectivity throughout the network
• Create workspaces that invite community building and participation—online and in-person
• Identify ‘open triangles’ and close them – identify two people who youknow and who would benefit from knowing each other and introducethem
• Encourage small collaborative projects among just two or three networkparticipants
• Codify a code of conduct, share it broadly, and live by it
High voluntary engagement in the networkby participants
High voluntary engagement in the network
Participants have a formal or informal codeof conduct and trust one another
Participants regularly interact andcollaborate with one another without goingthrough a central hub
Form
Network has a concept of its structure, how itsuits its purpose, and how it might evolve
• Map the network in order to visualize structure, diagnose strengths andweaknesses, and identify strategies for growing the network
• Facilitate an open strategic conversation that encourages participationfrom across the network; solicit the ‘wisdom of the crowd’
• Grow the number of people on the periphery of the network and createopportunities for their fresh ideas to flow into the network
• Create an innovation fund – a dedicated resource for cutting edge workthat creates a mandate for risk-taking
Balance of top-down and bottom-upstrategies for doing the work of the network
Network spaces invite self-organized action
1
Characteristic Desired Attributes HIGH MED LOW
Notes on OverallPerformance Potential Actions to Strengthen the Network
Leadership
Leaders inspire and help participantsrecognize and work towards common goals
• Identify individuals with strong group process skills and aptitude for IT-enabled collaboration to take on more responsibility
• Develop a system for diversifying and refreshing leadership
• Get out of the way – target opportunities for network participants toconnect and collaborate directly
• Bridge difference. Connect people and ideas that normally don’t gotogether
Leaders seek out opportunities to highlightand bridge difference in service of networkgoals
Leadership is shared. Responsibility andcontrol are pushed out to the network
Leadership is refreshed and renewed toreflect the network as it evolves
Governance
Governance is reflective of diverseconstituencies within the network andtransparent
• Formalize governance system with an eye to identifying opportunities toshare decision-making power
• Create mechanisms for voices from the periphery to influence decision-making
Connection
Network is resilient. Connectivity is strongenough throughout that if some highlyconnected participants leave, the networkremains strong
• Align communication tools with what participants are comfortable using orcan be trained to use. Don’t assume the network should adopt the latestadvances
• Follow the 1/10/100 rule :1% create content, 10% comment on it, and100% view content
• Allocate time and budget for designing, facilitating and maintaining onlinenetwork communications
• Look to young people to guide your use of social media
• Design shared spaces –online forums and in-person common spaces thatencourage interactions
• Seek out lower cost opportunities for connecting network participants in-person – e.g. host a reception at a commonly attended conference
Ample well-designed space, online and/or in-person, allowing participants to easilyconnect
Network use of social media supportsobjectives in external communications plan
Network use of social media is embraced andunderstood by network leaders andparticipants
2
Characteristic Desired Attributes HIGH MED LOW
Notes on OverallPerformance Potential Actions to Strengthen the Network
Capacity
Network can identify and prioritize fillingknowledge, skills and capacity gaps
• Create systems to help participants find and share relevant expertise
• Broadcast basic needs to the network and tap excess capacity to fill them
• Surface the talent in the network. Don’t assume external expertise needsto be brought in to address network needsEffective model for financial sustainability in
place
Learning &Adaptation
Network gathers feedback, and captureslearning as stories
• Create mechanisms for regular gathering of feedback from networkparticipants
• Invite key network participants to develop a shared set of metrics
• Create a network map that will serve as a baseline measurement. Map thenetwork again in 1-2 years, or once there has been enough time for changein the network. Compare the two maps to assess change in the system
• Create a mechanism – with dedicated resources – for ongoing capture oflearning and stories throughout the network
Network has a clearly articulated desiredimpact and a set of metrics to measure thatimpact
Network regularly measures, evaluates, andreflects on its impact to refine its goals andactivities
3
o o o o o
o o o o o
o o o o o
o o o o o
o o o o o
o o o o o
o o o o o
o o o o o
o o o o o
o o o o o
o o o o o
o o o o o
o o o o o
o o o o o
o o o o o
o o o o o
o o o o o
o o o o o
o o o o o
o o o o o
o o o o o
o o o o o
Questionnaire
Instructions This questionnaire asks questions about different aspects of your partnership. It will take about 15 minutes to complete. The questionnaire allows you to express your opinions and provide information about your experiences anonymously. DO NOT WRITE YOUR NAME ANYWHERE ON THE QUESTIONNAIRE and your name will not be attached in any way to the responses you give. By answering the questions, you will help your partnership learn about its strengths and weaknesses and about steps that your partnership can take in order to improve the collaboration process. The answers that people in your partnership give will be used to generate a report for your partnership. Only the people in your partnership will have access to this report. There are no right or wrong answers to the questions. Thoughtful and honest responses will give your partnership the most valuable information. Please answer every question, and please check only one answer per question. To complete the questionnaire:
• Please use a BLUE or BLACK ink pen.
• Be sure to read all the answer choices before marking your answer. • Answer each question by placing a legible check mark or “X” in the box to the left of your
answer, like this:
[ √ ] Extremely well OR [ X ] Extremely well
• Please return the completed questionnaire in a manner that protects your anonymity, as instructed by your coordinator.
CENTER FOR THE ADVANCEMENT OF COLLABORATIVE STRATEGIES IN HEALTH 2
Synergy Please think about the people and organizations that are participants in your partnership.
a. By working together, how well are these partners able to identify new and creative ways to solve problems?
[ ] Extremely well [ ] Very well [ ] Somewhat well [ ] Not so well [ ] Not well at all
b. By working together, how well are these partners able to include the views and priorities
of the people affected by the partnership’s work?
[ ] Extremely well [ ] Very well [ ] Somewhat well [ ] Not so well [ ] Not well at all
c. By working together, how well are these partners able to develop goals that are widely understood and supported among partners?
[ ] Extremely well [ ] Very well [ ] Somewhat well [ ] Not so well [ ] Not well at all
d. By working together, how well are these partners able to identify how different services and programs in the community relate to the problems the partnership is trying to address?
[ ] Extremely well [ ] Very well [ ] Somewhat well [ ] Not so well [ ] Not well at all
e. By working together, how well are these partners able to respond to the needs and
problems of the community?
[ ] Extremely well [ ] Very well [ ] Somewhat well [ ] Not so well [ ] Not well at all
CENTER FOR THE ADVANCEMENT OF COLLABORATIVE STRATEGIES IN HEALTH 3
f. By working together, how well are these partners able to implement strategies that are most likely to work in the community?
[ ] Extremely well [ ] Very well [ ] Somewhat well [ ] Not so well [ ] Not well at all
g. By working together, how well are these partners able to obtain support from individuals and organizations in the community that can either block the partnership’s plans or help move them forward?
[ ] Extremely well [ ] Very well [ ] Somewhat well [ ] Not so well [ ] Not well at all
h. By working together, how well are these partners able to carry out comprehensive
activities that connect multiple services, programs, or systems?
[ ] Extremely well [ ] Very well [ ] Somewhat well [ ] Not so well [ ] Not well at all
i. By working together, how well are these partners able to clearly communicate to people in the community how the partnership’s actions will address problems that are important to them?
[ ] Extremely well [ ] Very well [ ] Somewhat well [ ] Not so well [ ] Not well at all
CENTER FOR THE ADVANCEMENT OF COLLABORATIVE STRATEGIES IN HEALTH 4
Leadership Please think about all of the people who provide either formal or informal leadership in this partnership. Please rate the total effectiveness of your partnership’s leadership in each of the following areas:
a. Taking responsibility for the partnership
[ ] Excellent [ ] Very good [ ] Good [ ] Fair [ ] Poor [ ] Don’t know
b. Inspiring or motivating people involved in the partnership
[ ] Excellent [ ] Very good [ ] Good [ ] Fair [ ] Poor [ ] Don’t know
c. Empowering people involved in the partnership
[ ] Excellent [ ] Very good [ ] Good [ ] Fair [ ] Poor [ ] Don’t know
d. Communicating the vision of the partnership
[ ] Excellent [ ] Very good [ ] Good [ ] Fair [ ] Poor [ ] Don’t know
e. Working to develop a common language within the partnership
[ ] Excellent [ ] Very good [ ] Good [ ] Fair [ ] Poor [ ] Don’t know
CENTER FOR THE ADVANCEMENT OF COLLABORATIVE STRATEGIES IN HEALTH 5
Please rate the total effectiveness of your partnership’s leadership in:
f. Fostering respect, trust, inclusiveness, and openness in the partnership [ ] Excellent [ ] Very good [ ] Good [ ] Fair [ ] Poor [ ] Don’t know
g. Creating an environment where differences of opinion can be voiced
[ ] Excellent [ ] Very good [ ] Good [ ] Fair [ ] Poor [ ] Don’t know
h. Resolving conflict among partners
[ ] Excellent [ ] Very good [ ] Good [ ] Fair [ ] Poor [ ] Don’t know
i. Combining the perspectives, resources, and skills of partners
[ ] Excellent [ ] Very good [ ] Good [ ] Fair [ ] Poor [ ] Don’t know
j. Helping the partnership be creative and look at things differently
[ ] Excellent [ ] Very good [ ] Good [ ] Fair [ ] Poor [ ] Don’t know
CENTER FOR THE ADVANCEMENT OF COLLABORATIVE STRATEGIES IN HEALTH 6
Please rate the total effectiveness of your partnership’s leadership in: k. Recruiting diverse people and organizations into the partnership
[ ] Excellent [ ] Very good [ ] Good [ ] Fair [ ] Poor [ ] Don’t know
Efficiency
1. Please choose the statement that best describes how well your partnership uses the partners’ financial resources.
[ ] The partnership makes excellent use of partners’ financial resources. [ ] The partnership makes very good use of partners’ financial resources. [ ] The partnership makes good use of partners’ financial resources. [ ] The partnership makes fair use of partners’ financial resources. [ ] The partnership makes poor use of partners’ financial resources.
2. Please choose the statement that best describes how well your partnership uses the partners’ in-kind resources (e.g., skills, expertise, information, data, connections, influence, space, equipment, goods).
[ ] The partnership makes excellent use of partners’ in-kind resources. [ ] The partnership makes very good use of partners’ in-kind resources. [ ] The partnership makes good use of partners’ in-kind resources. [ ] The partnership makes fair use of partners’ in-kind resources. [ ] The partnership makes poor use of partners’ in-kind resources.
3. Please choose the statement that best describes how well your partnership uses the
partners’ time.
[ ] The partnership makes excellent use of partners’ time. [ ] The partnership makes very good use of partners’ time. [ ] The partnership makes good use of partners’ time. [ ] The partnership makes fair use of partners’ time. [ ] The partnership makes poor use of partners’ time.
CENTER FOR THE ADVANCEMENT OF COLLABORATIVE STRATEGIES IN HEALTH 7
Administration and Management We would like you to think about the administrative and management activities in your partnership. Please rate the effectiveness of your partnership in carrying out each of the following activities:
a. Coordinating communication among partners
[ ] Excellent [ ] Very good [ ] Good [ ] Fair [ ] Poor [ ] Don’t know
b. Coordinating communication with people and organizations outside the partnership [ ] Excellent [ ] Very good [ ] Good [ ] Fair [ ] Poor [ ] Don’t know
c. Organizing partnership activities, including meetings and projects [ ] Excellent [ ] Very good [ ] Good [ ] Fair [ ] Poor [ ] Don’t know
d. Applying for and managing grants and funds
[ ] Excellent [ ] Very good [ ] Good [ ] Fair [ ] Poor [ ] Don’t know
e. Preparing materials that inform partners and help them make timely decisions [ ] Excellent [ ] Very good [ ] Good [ ] Fair [ ] Poor [ ] Don’t know
CENTER FOR THE ADVANCEMENT OF COLLABORATIVE STRATEGIES IN HEALTH 8
Please rate the effectiveness of your partnership in:
f. Performing secretarial duties [ ] Excellent [ ] Very good [ ] Good [ ] Fair [ ] Poor [ ] Don’t know
g. Providing orientation to new partners as they join the partnership
[ ] Excellent [ ] Very good [ ] Good [ ] Fair [ ] Poor [ ] Don’t know
h. Evaluating the progress and impact of the partnership
[ ] Excellent [ ] Very good [ ] Good [ ] Fair [ ] Poor [ ] Don’t know
i. Minimizing the barriers to participation in the partnership’s meetings and activities
(e.g., by holding them at convenient places and times, and by providing transportation and childcare)
[ ] Excellent [ ] Very good [ ] Good [ ] Fair [ ] Poor [ ] Don’t know
CENTER FOR THE ADVANCEMENT OF COLLABORATIVE STRATEGIES IN HEALTH 9
Non-financial Resources A partnership needs non-financial resources in order to work effectively and achieve its goals. For each of the following types of resources, to what extent does your partnership have what it needs to work effectively?
a. Skills and expertise (e.g., leadership, administration, evaluation, law, public policy, cultural competency, training, community organizing) [ ] All of what it needs [ ] Most of what it needs [ ] Some of what it needs [ ] Almost none of what it needs [ ] None of what it needs [ ] Don’t know
b. Data and information (e.g., statistical data, information about community perceptions, values, resources, and politics) [ ] All of what it needs [ ] Most of what it needs [ ] Some of what it needs [ ] Almost none of what it needs [ ] None of what it needs [ ] Don’t know
c. Connections to target populations
[ ] All of what it needs [ ] Most of what it needs [ ] Some of what it needs [ ] Almost none of what it needs [ ] None of what it needs [ ] Don’t know
d. Connections to political decision-makers, government agencies, other
organizations/groups [ ] All of what it needs [ ] Most of what it needs [ ] Some of what it needs [ ] Almost none of what it needs [ ] None of what it needs [ ] Don’t know
CENTER FOR THE ADVANCEMENT OF COLLABORATIVE STRATEGIES IN HEALTH 10
For each of the following types of resources, to what extent does your partnership have what it needs to work effectively?
e. Legitimacy and credibility
[ ] All of what it needs [ ] Most of what it needs [ ] Some of what it needs [ ] Almost none of what it needs [ ] None of what it needs [ ] Don’t know
f. Influence and ability to bring people together for meetings and activities
[ ] All of what it needs [ ] Most of what it needs [ ] Some of what it needs [ ] Almost none of what it needs [ ] None of what it needs [ ] Don’t know
Financial and Other Capital Resources
A partnership also needs financial and other capital resources in order to work effectively and achieve its goals. For each of the following types of resources, to what extent does your partnership have what it needs to work effectively?
a. Money [ ] All of what it needs [ ] Most of what it needs [ ] Some of what it needs [ ] Almost none of what it needs [ ] None of what it needs [ ] Don’t know
b. Space
[ ] All of what it needs [ ] Most of what it needs [ ] Some of what it needs [ ] Almost none of what it needs [ ] None of what it needs [ ] Don’t know
CENTER FOR THE ADVANCEMENT OF COLLABORATIVE STRATEGIES IN HEALTH 11
For the following type of resources, to what extent does your partnership have what it needs to work effectively?
c. Equipment and goods [ ] All of what it needs [ ] Most of what it needs [ ] Some of what it needs [ ] Almost none of what it needs [ ] None of what it needs [ ] Don’t know
Decision Making
a. How comfortable are you with the way decisions are made in the partnership? [ ] Extremely comfortable [ ] Very comfortable [ ] Somewhat comfortable [ ] A little comfortable [ ] Not at all comfortable
b. How often do you support the decisions made by the partnership?
[ ] All of the time [ ] Most of the time [ ] Some of the time [ ] Almost none of the time [ ] None of the time
c. How often do you feel that you have been left out of the decision making process?
[ ] All of the time [ ] Most of the time [ ] Some of the time [ ] Almost none of the time [ ] None of the time
CENTER FOR THE ADVANCEMENT OF COLLABORATIVE STRATEGIES IN HEALTH 12
Benefits of Participation For each of the following benefits, please indicate whether you have or have not received the benefit as a result of participating in the partnership.
a. Enhanced ability to address an important issue [ ] Yes [ ] No
b. Development of new skills [ ] Yes [ ] No
c. Heightened public profile
[ ] Yes [ ] No
d. Increased utilization of my expertise or services
[ ] Yes [ ] No
e. Acquisition of useful knowledge about services, programs, or people in the
community [ ] Yes [ ] No
f. Enhanced ability to affect public policy
[ ] Yes [ ] No
g. Development of valuable relationships
[ ] Yes [ ] No
h. Enhanced ability to meet the needs of my constituency or clients
[ ] Yes [ ] No
i. Ability to have a greater impact than I could have on my own
[ ] Yes [ ] No
CENTER FOR THE ADVANCEMENT OF COLLABORATIVE STRATEGIES IN HEALTH 13
As a result of your participation in the partnership, have you experienced the following benefits:
j. Ability to make a contribution to the community [ ] Yes [ ] No
k. Acquisition of additional financial support
[ ] Yes [ ] No
Drawbacks of Participation For each of the following drawbacks, please indicate whether or not you have or have not experienced the drawback as a result of participating in this partnership.
a. Diversion of time and resources away from other priorities or obligations [ ] Yes [ ] No
b. Insufficient influence in partnership activities
[ ] Yes [ ] No
c. Viewed negatively due to association with other partners or the partnership
[ ] Yes [ ] No
d. Frustration or aggravation
[ ] Yes [ ] No
e. Insufficient credit given to me for contributing to the accomplishments of the
partnership [ ] Yes [ ] No
f. Conflict between my job and the partnership’s work
[ ] Yes [ ] No
CENTER FOR THE ADVANCEMENT OF COLLABORATIVE STRATEGIES IN HEALTH 14
Comparing Benefits and Drawbacks So far, how have the benefits of participating in this partnership compared to the drawbacks? [ ] Benefits greatly exceed the drawbacks [ ] Benefits exceed the drawbacks [ ] Benefits and drawbacks are about equal [ ] Drawbacks exceed the benefits [ ] Drawbacks greatly exceed the benefits Satisfaction with Participation
a. How satisfied are you with the way the people and organizations in the partnership work together? [ ] Completely satisfied [ ] Mostly satisfied [ ] Somewhat satisfied [ ] A little satisfied [ ] Not at all satisfied
b. How satisfied are you with your influence in the partnership?
[ ] Completely satisfied [ ] Mostly satisfied [ ] Somewhat satisfied [ ] A little satisfied [ ] Not at all satisfied
c. How satisfied are you with your role in the partnership?
[ ] Completely satisfied [ ] Mostly satisfied [ ] Somewhat satisfied [ ] A little satisfied [ ] Not at all satisfied
d. How satisfied are you with the partnership’s plans for achieving its goals?
[ ] Completely satisfied [ ] Mostly satisfied [ ] Somewhat satisfied [ ] A little satisfied [ ] Not at all satisfied
CENTER FOR THE ADVANCEMENT OF COLLABORATIVE STRATEGIES IN HEALTH 15
e. How satisfied are you with the way the partnership is implementing its plans? [ ] Completely satisfied [ ] Mostly satisfied [ ] Somewhat satisfied [ ] A little satisfied [ ] Not at all satisfied
C E N T E R F O R P U B L I C R E S E A R C H A N D L E A D E R S H I P
73
AdditionalResources:MeasurementToolkitforIntermediaryOrganizations. . . . . . .
SAMPLENETWORKFORMQUESTIONS
Who You Are Your Name ___________________________________________________ Your Organization or School (if applicable) _______________________________________________ Today’s Date (MM/DD/YY): _________________ When did you join the network (MM/YY)? __________________ Which of the following groups do you primarily represent? Please choose only one.
☐ Individual School � Regional Education Service Agency
☐ School District ��Technical Assistance Provider
☐ Charter Management Organization ��Funder
☐ State Level Agency� ��Researcher�
Who You Communicate With For each person, check the box that best describes how often you communicated in the last six months (e.g. in writing, over the phone, face-to-face, or in meetings). If you don’t know the person, check the box marked “Don’t Know Person.” If it is you check the box labeled “It’s me.”
Person Not in Last 6 Months
Once in Last 6 Months
Multiple Time in Last 6 Months
Monthly Multiple Times a Months
Weekly Multiple Times a Week
Don’t Know Person
It’s me
Person A ___ ___ ___ ___ ___ ___ ___ ___ ___
Person B ___ ___ ___ ___ ___ ___ ___ ___ ___
… ___ ___ ___ ___ ___ ___ ___ ___ ___
Who You Learn From For each person, check the box that best describes how often they have provided you with information you used to do your work in the last six months (e.g. new idea, a report, contact
Use the following questions as a starting point for designing a network survey to assess the membership and structure (i.e. form, structure, or connectivity) of your network.
C E N T E R F O R P U B L I C R E S E A R C H A N D L E A D E R S H I P
74
information, etc.). If you don’t know the person, check the box marked “Don’t Know Person.” If it is you check the box labeled “It’s me.”
Person Not in Last 6 Month
Once in Last 6 Month
Multiple Time in Last 6 Months
Monthly Multiple Times a Month
Weekly Multiple Times a Week
Don’t Know Person
It’s me
Person A ___ ___ ___ ___ ___ ___ ___ ___ ___
Person B ___ ___ ___ ___ ___ ___ ___ ___ ___
… ___ ___ ___ ___ ___ ___ ___ ___ ___
Who You Collaborate With For each person, check the box that best describes how often you have worked together to do work in the last six months (e.g. plan an event, write a report, conduct an observation). If you don’t know the person, check the box marked “Don’t Know Person.” If it is you check the box labeled “It’s me.”*
Person Not in Last 6 Month
Once in Last 6 Month
Multiple Time in Last 6 Months
Monthly Multiple Times a Month
Weekly Multiple Times a Week
Don’t Know Person
It’s me
Person A ___ ___ ___ ___ ___ ___ ___ ___ ___
Person B ___ ___ ___ ___ ___ ___ ___ ___ ___
… ___ ___ ___ ___ ___ ___ ___ ___ ___
C E N T E R F O R P U B L I C R E S E A R C H A N D L E A D E R S H I P
75
Who You Seek Out For each person, indicate the primary benefit you currently receive from their membership in the network. If you don’t know the person, check the box marked “Don’t Know Person.” If it is you check the box labeled “It’s me.”
Person
Information that helps me act and
capitalize on opportunities
Access to funding or material resources
Access to key decision makers
Problem-solving interactions that push me thinking
Personal support including the
ability to brag or vent
Motivation and a reminder of the
importance of out work
Person A ___ ___ ___ ___ ___ ___
Person B ___ ___ ___ ___ ___ ___
… ___ ___ ___ ___ ___ ___
Who is Missing In the cells below, list up to 12 people you feel are missing from the above lists.
1. 2. 3.
4. 5. 6.
7. 8. 9.
10. 11. 12.
THANK YOU!
C E N T E R F O R P U B L I C R E S E A R C H A N D L E A D E R S H I P
76
SAMPLEQUESTIONSFORNETWORKMEMBERS
Membership
1. Approximately how many individuals or organizations are in the network?
2. Who makes up the network? What sectors or roles are represented? • Practitioners? • Policy makers? • Funders? • Researchers
3. In your view, is this membership size and diversity sufficient to accomplish network goals?
• Why or why not? Structure
4. Who do you interact with most within the network? • What types of interactions do you have?
o Information or resources sharing? o Problem-solving support or thought partnering? o Collaboration on projects?
5. Who do you interact with least within the network?
• Why are your interactions with these individuals or groups limited?
6. Do you interact directly with other network members or are interactions typically facilitated and overseen by network leadership? • How do you interact and communicate? • How do leaders facilitate interactions and communications?
7. Which, if any, network members stand out because of their abundance of connections with
other members? • Who are these members connected with? • What interactions do these members have with others?
8. What, if any, connections across the network could be improved? • Are there any members who seem disengaged? • Are there connections that are fading?
The following questions can be used in interviews and focus groups with network members to assess network outcomes. Questions are organized according to the evaluative criteria of membership, structure, operations, and benefits. Questions should be selected and modified to fit a network’s context, design, and purpose.
C E N T E R F O R P U B L I C R E S E A R C H A N D L E A D E R S H I P
77
Operations 9. What is working well about the network’s operations?
• Communication systems? • Meetings and events? • Resource sharing? • Leadership? • Collaboration?
10. What is not work well about the network’s operations and should be improved or ended?
• Communication systems? • Meetings and events? • Resource sharing? • Leadership? • Collaboration?
11. Do you believe your participation in this network is positively impacting your day-to-day work? Why or why not? • Is there timely and sufficient alignment between your work and topics explored in the
network? • Have there been any times when your participation in the network has felt burdensome or
disconnected from your interests and needs?
12. Does the network seem to have sufficient resources? • To your knowledge, does the network seem to be financial stable? • Does the network have access to facilities? • Does the network have technology resources and sharing platforms to support its work? • Is the allocation of staff or network members sufficient enough to support network
activities?
13. How effectively is the network led? • Are leaders organized? • Do leaders communicate proactively and transparently? • Are leaders knowledgeable and credible?
Benefits
14. When you first joined, what did you hope to gain by participating in the network? � • Knowledge? • Resources? • Connections? • Support?
15. Is there anyone or anything you hoped to influence through participating in the network?
• Policies? • Organizational structures? • Public perceptions? • Mindsets? • Behaviors?
C E N T E R F O R P U B L I C R E S E A R C H A N D L E A D E R S H I P
78
16. When you first joined, what did you want to contribute to the network? � • Knowledge? • Resources? • Thought-partnership?
17. In your view what has the network accomplished or produced that stands out?
• Has the network had any breakthrough moments? • Has the network developed new knowledge, resources, or tools? • Has the network effectively addressed barriers to its work or developed new supports for its
work?
18. What do you think has changed about your work as a result of your participation in the network? • Changes to your knowledge, mindsets, or behaviors?
19. To what extent have you been able to influence the people and things you hoped to influence?
• Policies? Structures? Public perceptions? Mindsets? Behaviors?
20. How have you been able to contribute to the network? • Knowledge? Resources? Thought-partnership?
21. Do your contributions meet your expectation?
• Why do you think this is?
22. Do the network’s results meet your expectations? a. Why do you think this is?
C E N T E R F O R P U B L I C R E S E A R C H A N D L E A D E R S H I P
79
Diffusion Levers Evaluation Toolkit: System Cultivating
TheDiffusionIntermediaryEvaluationFrameworkincludestwocriteriaforassessingshort-termoutcomesstemmingfromSystemCultivatingactivities:SystemPolicies&PracticesandSystemEngagement&PublicWill.Thesecriteriaarebasedonascanofthesystemthinkingandsystemchangeliteraturelistedattheendofthissection.
Evaluative Criteria, Questions, and Sample Indicators
Criteria GuidingQuestions SampleIndicators
SystemPolicies&Practices
Towhatextentarethe
formalandinformal
policiesandpracticesthat
impactpartnersenabling
ordemandinginnovation?
• Legislationthatdemandsinnovation• Humanresourcessystemsandstructuresthatenableinnovation• Accountabilitysystemsandstructuresthatenableinnovation• Professionalstandardsthataligntoinnovation• Organization-widestandardoperatingproceduresthatalignto
innovation
SystemEngagement&PublicWill
Towhatextentisthe
publicdemandingand
supportingchange?
• Frequencyofpositivemediacoverage• Sizeandfrequencyofpublicgatheringstodemonstratesupportfor
innovation• Dollarsofpublicandprivatefundingdistributedtotheregionfor
innovation• Fundingandinfrastructureconditionsthatenableinnovation
Tools
BellwetherSurveySparkPolicyInstitute
Abellwethersurveyisacommonmethodfortrackingpoliticalwill.Itentailstalkingwitha“bellwether”—aninfluentialandpoliticallyinformedindividual—todeterminehowlikelyitisthatapoliticalissuewillbeactedonbasedonhowkeydecisionmakersarethinkingandtalkingabouttheissueandwhereitsitsinthepoliticalagenda.Asimilarprocesscanbeusedtoassesswillwithinthepublic,funders,researchers,orotherkeysystemactors.
ü SystemPolicies&Practices
üSystemEngagement&PublicWill
Coalition Building Self-Assessment AGuidetoMeasuringAdvocacyandPolicy
Thisself-assessmentcanbecustomizedtoanintermediary’stargetedsystem-levelstakeholdergroups.Itallowsanintermediaryorganizationtodeterminetheextenttowhichitmetitsgoalsaroundbuildingchampions,orindividualswhovocallysupporttheintermediary’svisionandstrategyanddevotetimeandresourcestoengagingothersinit.
SystemPolicies&Practices
üSystemEngagement&PublicWill
MediaTrackingandAnalysisOverviewCenterforPublicResearchandLeadership
Thisdocumentprovidesabriefoverviewofmediatracking.Mediatracking SystemPolicies&Practices
C E N T E R F O R P U B L I C R E S E A R C H A N D L E A D E R S H I P
80
monitorshowmediacoveragerelatedtoakeytopicevolvesovertime.Theprocessinvolvesusinganews-trackingservicelikeLexisNexistosystematicallygathercontentandthenreviewingthiscontentforpatternsinframing,frequency,lengths,etc.
üSystemEngagement&PublicWill
PolicymakerRatingTemplateSparkPolicyInstitute
Policymakerratingisasystematicprocessofgaugingpolicymakersupportforspecificissues.Itinvolvespolicyadvocatesrating(1)apolicymaker’slevelofsupportforanissue,(2)thepolicymakerlevelofinfluencesonthepolicy,and(3)advocateslevelofconfidenceintheaccuracyofthefirsttworating.
ü SystemPolicies&Practices
SystemEngagement&PublicWill
Resources BridgespanGroup.(2009).Thestrongfieldframework:Aguideandtoolkitforfundersandnonprofitscommittedto
large-scaleimpact.Retrievedfromhttps://irvine-dot-org.s3.amazonaws.com/documents/64/attachments/strongfieldframework.pdf?1412656138
Coffman,J.(2009).Auser’sguidetoadvocacyevaluationplanning.HarvardFamilyResearchProject.Retrievedfromhttp://www.hfrp.org/.
Coffman,J.&Reed,E.Uniquemethodsinadvocacyevaluation.Retrievedfromhttp://www.pointk.org/resources/files/Unique_Methods_Brief.pdf.
Forti,M.(2012).Measuringadvocacy-Yeswecan!StanfordSocialInnovationReview.Retrievedfromhttps://drive.google.com/file/d/0BxI97tx1dL4ZbU5zLWFVX0hzb0E/view.
Hargreaves,M.(2010).Evaluatingsystemchange:Aplanningguide.MathematicaPolicyResearch,Inc.Retrievedfromhttps://drive.google.com/file/d/0BxI97tx1dL4ZRVYtODJld0drX28/view?pref=2&pli=1.
HarvardFamilyResearchProject(2007).Advocacyandpolicychange.TheEvaluationExchange.13(1):1-32.
Latham,N.(2014).Apracticalguidetoevaluatingsystemschangeinahumanservicessystemcontext.LearningforAction.Retrievedfromhttps://drive.google.com/file/d/0BxI97tx1dL4Zb19GOEE0Y1Q3SGM/view?pref=2&pli=1
Meadows,D.(2008).Thinkinginsystems:Aprimer.WhiteRiverJunction,VT:ChelseaGreenPublishing.
Preskill,H.,Gopal,S.,Mack,K.,&Cook,J.(2015).Evaluatingcomplexity:Propositionsforimprovingpractice.FSG.Retrievedfromhttp://www.issuelab.org/resource/evaluating_complexity_propositions_for_improving_practice
Reisman,J.,Gienapp,A.,&Stachowiak,S.(2007).Aguidetomeasuringadvocacyandpolicy.TheAnnieE.CaseyFoundation.Retrievedfromhttps://drive.google.com/file/d/0BxI97tx1dL4ZX09HTktUYld4MjQ/view.
Stroh,D.(2015).Systemsthinkingforsocialchange:Apracticalguidetosolvingcomplexproblems,avoidingunintended
consequences,andachievinglastingresults.WhiteRiverJunction,VT:ChelseaGreenPublishing.
C E N T E R F O R P U B L I C R E S E A R C H A N D L E A D E R S H I P
81
BELLWETHERSURVEYQUESTIONSSources:SparkPolicyInstitute
IntroductionThankyoufortakingthetimetotalktome.LetmestartbygivingyouabriefoverviewofwhyIwantedtotalktoyoutoday.Iamgatheringinformationfromkeypeopleofinfluenceaboutthepolicylandscapeandprioritiesinourstate.TodayI’mgoingtoaskyouaboutourstate’spolicylandscapeingeneral,andthensomemoredetailedquestionsaboutspecificpolicies.Weareinterestedinyouropinionsandreactionstoourquestions;ofcoursetherearenorightorwronganswers.
Questions1. Currently,whatthreeissuesdoyouthinkareatthetopofthe[state/federal/local]policyagenda?�2. Howfamiliarareyouwith[thepolicyofinterest]?�3. Whatindividuals,constituencies,orgroupsdoyouseeasthemainadvocatesfor[thepolicy]?Whodoyouseeas
themainopponents?�4. Consideringthecurrenteducational,social,andpoliticalcontext,doyouthink[thepolicy]shouldbeadoptednowor
inthenearfuture?�5. Lookingahead,howlikelydoyouthinkitisthat[thepolicy]willbeadoptedinthenext5years?�6. Currently,whatindividuals,constituencies,orgroupsdoyouseeatthemainadvocatesfor(yourpolicyissue)?Who
doyouseeasthemainopponents?7. If[thepolicy]isadopted,whatissuesdoyouthinkthestateneedstobemostconcernedaboutrelatedtoits
implementation?�
ClosingThankyoufortakingtimetoanswermyquestionsonpolicyissuesandhealthcarepolicyspecifically.Youranswerswillbesummarizedwithresponsesfromotherleadersinourstate.Yourindividualresponseswillnotbesharedwithanyoneoutsideourevaluationteam.
Abellwethersurveyisacommonmethodfortrackingpoliticalwill.Itentailstalkingwitha“bellwether”—aninfluentialandpoliticallyinformedindividual—aboutthepoliticalenvironmentandthestandingofspecificpolicies.Thefollowingquestionscanbecustomizedandusedtocompletebellwethersurveys.
C E N T E R F O R P U B L I C R E S E A R C H A N D L E A D E R S H I P
82
COALITIONBUILDINGSELF-ASSESSMENTModifiedfromAGuidetoMeasuringAdvocacyandPolicy,TheAnnieE.CaseyFoundation(2007)
Powerfulfriends
aboundandare
willingandfrequent
spokespersonsforour
missionusing
political,social,or
economiccapitalto
gainsupportfrom
others
Somenewchampions
havebeenidentified
andaretakingsome
publicstepsto
supportourmission,
butitisnotgenerally
atthetopoftheir
agendas
Limitedchampions
withothersnot
seeinganyrelevance
ofourmissiontotheir
workorviewingusas
apossiblecompeting
demandonresources
Substantialprogress
hasbeenmadein
thelast6months
(checkforyes)
[StakeholderGroup-E.g.schoolsuperintendents,policymakers,funders]
1 2 3 4 5 *
Evidence:
[StakeholderGroup-E.g.schoolsuperintendents,policymakers,funders]
1 2 3 4 5 *
Evidence:
[StakeholderGroup-E.g.schoolsuperintendents,policymakers,funders]
Thisself-assessmentcanbecustomizedtoanintermediary’stargetedsystem-levelstakeholdergroups.Foreachofthestakeholdergroups,markanumberfrom1-5toindicatetheextenttowhichpowerfulconnectionshavebeenestablishedwithindividualsorgroupsmakingupthecategoryofstakeholders.Ifdesired,alsocheckthoseareaswheretheintermediaryhasmadesubstantialprogressinthelastsixmonths.
C E N T E R F O R P U B L I C R E S E A R C H A N D L E A D E R S H I P
83
1 2 3 4 5 *
Evidence:
[StakeholderGroup-E.g.schoolsuperintendents,policymakers,funders]
1 2 3 4 5 *
Evidence:
C E N T E R F O R P U B L I C R E S E A R C H A N D L E A D E R S H I P
84
MEDIATRACKINGANDANALYSISOVERVIEW
Tracking
Useanews-trackingservicelikeLexisNexusorGoogleNewsAlertssystematicallygathercontent.Carefullyselectthekeywordstouseinthesearchaswellasthetypesofpublicationsthesearchwillpullfrom.Theseshouldremainconsistentthroughouttheprocesses.
Title Author Source Date Length TopicIsthemessageone
wewantedtoconvey?
Istheinformationaccurate?
Whoismentioned
inthearticle?
Analysis
Atregularintervalsreviewthedatabaseusingthequestionsbelowassampleprompts.
• Whatchangesinthefrequencyofrelevantmediacoveredhaveweseenoverthelast6months?o Iscoveredincreasing?Decreasing?Stable?o Whereisfrequencychanging?Whereisitstable?
• Whatchangesinthecontentofrelevantmediacoveredhaveweseenoverthelast6months?o Towhatextentiscoveragealignedwithourorganizationalvisionandstrategy?o Towhatextentiscoveragegainingnuanceandcredibility?o Towhatextentiscoveragehighlightingkeyplayersinthefield?
• Whatrelationshipbetweenfrequencyandcontenthaveweseenoverthelast6months?• Whyareweseeingthesepatterns?• Whatstepscanwetaketodisruptoracceleratethesepatterns?
Thisdocumentprovidesabriefoverviewofmediatracking.Mediatrackingmonitorshowmediacoveragerelatedtoakeytopicevolvesovertime.Theprocessinvolvesusinganews-trackingservicelikeLexisNexistosystematicallygathercontentandthenreviewingthiscontentforpatternsinframing,frequency,lengths,etc.
C E N T E R F O R P U B L I C R E S E A R C H A N D L E A D E R S H I P
85
POLICYMAKERRATINGTEMPLATESource:SparkPolicyInstitute
Issue (Onesentenceorshortphrasethatclearlyarticulatesyourissue)PolicymakerName RaterName
Scale Rating DefinitionSupport(Supportfortheissue) �Notatall
supportiveorinopposition.
Noevidencethispersonhastakenaction,spokenabout,orotherwisedirectlysupportedthispolicyissue.OR,evidencethispersonopposestheissue.
�Notverysupportive
Thispersonhasverballyexpressedsomesupport,butprimarilyinone-on-oneconversationsandsmallgroupmeetings.
�Supportive
Thispersondemonstratessupportthroughactionssuchas:voting,speakinginpublic,quotedinthemedia,encouragingotherstosupporttheissue,helpingnegotiation/supportbills.
�Extremelysupportive
Thispersonisknownasachampionfortheissue,playsaleadershiproleinadvancingtheissue,andconsistentlymakestheissueapriorityontheiragenda.
Influence(extenttowhichpolicymakermeetsthefollowingcriteria)
• Majoritypartymember• Relevantcontentexpertise• Seniority/experience• Reputation/respect• Keycommitteemember• Formalleadershipposition
�Notatallinfluential Meetsnocriteriaoronecriteria.
�Somewhatinfluential Meetsatleasttwocriteria.
�Influential Meetingsthreeorfourcriteria.
�Extremelyinfluential Meetingsfiveorsixcriteria.
Confidence(yourconfidencelevelinyourrating)
�Notveryconfident
Ratingsbasedonthirdhandinformation.Notverifiable.
�Somewhatconfident
Ratingsbasedonconsistentinformationfromoneormoresource,butnot100%verifiable.
�Confident Ratingsbasedondirectcontactwiththepolicymakerorinformationfromahighlytrusted,verifiablesource.
Thissampleversioncanbeeasilyadaptedformanytopics.Primarily,youwillwanttoupdateittomatchyoursettinginthefollowingways:
1. Defineyourissueandputitinthefirstrow.2. Makesurethatthedefinitionsofsupportareappropriatetothesettingsofyourpolicymakers(e.g.ifthey
arenotlegislators,youmaynotwanttokeeplegislationasevidenceoftheirsupport);3. Updatethecriteriaforinfluencetomatchyourpolicymakers’setting.
ADDITIONALRESOURCES
Phrased Rubrics for Assessing Intermediary Organizations AppendixtoIntermediaryOrganizationsandEducationInnovation
CenterforPublicResearchandLeadership
C E N T E R F O R P U B L I C R E S E A R C H A N D L E A D E R S H I P
AlignedtotheDiffusionIntermediaryEvaluationFramework
87
Phase I: Potential for Influence
ReadinessofLocalContext
CRITERIA WEAKEVIDENCE DEVELOPINGEVIDENCE SUFFICIENTEVIDENCE EXTENSIVEEVIDENCE
SupportforInnovation
Towhatextentisthelocalcontextopentoandsupportiveofthechangesandworkproposedbytheintermediary?
• Theexistingpolicyenvironmentisprohibitiveoftheideasandpracticestheintermediaryseekstoachieve,andfew,ifany,explicitavenuestoalterthesepoliciesexist.
• Thelocalityhasnofinancial,
informational,orpeersupportavailableforexperimentationorentrepreneurialismwithintheeducationsector.
• Thepublicisrisk-averse,
hasdeepmisconceptionsorproteststotheneweducationalideasandpracticesbeingproposed,and/orhasahistoryofprotestingeducationalreformandinnovation.
• Theexistingpolicyenvironmentispermissiveoftheideasandpracticestheintermediaryseekstoachieveandsomeisolatedpoliticalavenuesexisttosupportinnovation.
• Thelocalityhassome
formalorinformalsupportforexperimentationorentrepreneurialismwithintheeducationsectoravailableintheformofsomefunding,peertopeercollaborationstructures,and/orinformationalresources.
• Thepublicislargely
disengagedfromdiscussionsofeducationinnovationanddemonstrateneitherprotestnorsupport.
• Theexistingpolicyenvironmentpromotestheideasandpracticestheintermediaryseekstoachieveandpoliticalavenuesexisttosupportinnovation.
• Thelocalityhasvarious
formalorinformalsupportsforexperimentationorentrepreneurialismavailablewithintheeducationsectorintheformoffunding,peertopeercollaborationstructures,and/orinformationalresources.
• Thepublicdemonstratessomeengagementwithandsupportforeducationinnovation.
• Theexistingpolicyenvironmentalreadyrequirestheideasandpracticestheintermediaryseekstoachieveinpartorintotal.
• Thelocalityhasextensive
formalorinformalsupportforexperimentationorentrepreneurialismwithintheeducationsectoravailableintheformoffunding,peertopeercollaborationstructures,and/orinformationalresources.
• Thepublicdemonstrates
deepengagementwithandademandforeducationinnovation.
C E N T E R F O R P U B L I C R E S E A R C H A N D L E A D E R S H I P
AlignedtotheDiffusionIntermediaryEvaluationFramework
88
FieldNeedTowhatextentisthelocal
educationsystemdemonstratinganeedfor
improvement?
Towhatextentdoesthelocaleducationsystem
needadditionalsupportofthekindanintermediary
canprovide?
• Theeducationsystemdemonstratesnocurrentneedforimprovement—studentachievementratesarewellabovethenationalaverageandequalacrossdifferentracial,ethnic,andsocioeconomicgroups.
• Theeducationsectorwithinthelocalcontexthasnoneedforfurthersupportofthekindtheintermediaryprovides.
• Theeducationsystemdemonstratesaslightneedforimprovement—studentachievementratesareaverageatbestand/orgapsexistacrossdifferentracial,ethnic,andsocioeconomicgroupssimilartothoseseennationally.
• Theeducationsectorwithinthelocalcontextalreadyhasextensivesupportssimilartothoseofferedbytheintermediary.
• Theeducationsystemdemonstratesamoderateneedforimprovement—studentachievementratesarebelownationalaveragesand/orgapsexistacrossdifferentracial,ethnic,andsocioeconomicgroupsthatarehigherthantothoseseennationally.
• Theeducationsectorwithinthelocalcontexthasfewsupportssimilartothoseofferedbytheintermediary.
• Theeducationsystemdemonstratesanurgentneedforimprovement—studentachievementratestrailfarbehindnationalaveragesand/orgapsexistacrossdifferentracial,ethnic,andsocioeconomicgroupsthatareamongthehighestinthenation.
• Theeducationsectorwithinthelocalcontexthasnosupportssimilartothoseofferedbytheintermediary.
Vision&StrategyCRITERIA WEAKEVIDENCE DEVELOPINGEVIDENCE SUFFICIENTEVIDENCE EXTENSIVEEVIDENCE
TransformativeVisionTowhatextentdoestheintermediary’svisionfor
changebreakfromtraditionaleducationalpracticesandsystems?
• Theintermediary’sdesiredimpactisunstated,incoherent,and/ordoesnotincludemeaningfullydefininginstructionortheinteractionsthatoccurbetweenstudentandteacher.
• Theintermediary’sdesiredimpactisslightlyunclearand/oronlyminimallyredefinesinstructionortheinteractionsthatoccurbetweenstudentandteacher.
• Theintermediary’sdesiredimpactisforthemostpartclearandredefinesinstructionortheinteractionsthatoccurbetweenstudentandteacherinnotableways.
• Theintermediary’sdesiredimpactisclearandsubstantiallyredefinesinstructionortheinteractionsthatoccurbetweenstudentandteacher.
ViabilityofStudentImpact
Howrobustistheorganization’sbasis(in
evidenceand/orlogic)forpredictingthatitsvisionwillpositivelyimpact
students?
• Thereisnoevidenceoforlogicalexplanationforarelationshipbetweenthenewpracticesproposedbytheintermediaryandimprovedlevelsofstudentsuccess.
• Thereisearlyorscatteredevidenceofarelationshipbetweenthenewpracticesproposedbytheintermediaryandimprovedlevelsofstudentsuccessorthereisanearlyhypothesisforwhyimprovedoutcomescanbeexpected.
• Thereissubstantial,thoughnotconclusive,evidenceofarelationshipbetweenthenewpracticesproposedbytheintermediaryandimprovedlevelsofstudentsuccessorthereisacredibleresearch-basedhypothesisforwhyimprovedoutcomescanbeexpected.
• Thereisconclusiveevidenceofarelationshipbetweenthenewpracticesproposedbytheintermediaryandimprovedlevelsofstudentsuccessorthereisaresearch-backedhypothesisforwhyimprovedoutcomescanbeexpected.
C E N T E R F O R P U B L I C R E S E A R C H A N D L E A D E R S H I P
AlignedtotheDiffusionIntermediaryEvaluationFramework
89
InformedStrategyTowhatextentdoestheintermediary’sstrategyincorporatethekeyDiffusionLeversofCapacityBuilding,
NetworkWeaving,andSystemCultivating?
• ThestrategydoesnotincludeactivitiesthatemploythekeyDiffusionLevers(i.e.,CapacityBuilding,NetworkWeaving,andSystemCultivating).
• ThestrategyincludesfewactivitiesthatemploythekeyDiffusionLevers(i.e.,CapacityBuilding,NetworkWeaving,andSystemCultivating)oractivitiestodemonstratelittleplanningregardingwhichLeverstoemployandtowhatdegree.
• ThestrategyincludesseveralactivitiesthatemploythekeyDiffusionLevers(i.e.,CapacityBuilding,NetworkWeaving,andSystemCultivating)andreflectssomeconsiderationofthehowtheLeverswillintersect.
• ThestrategyincludesanarrayofactivitiesthatemployallthekeyDiffusionLevers(i.e.,CapacityBuilding,NetworkWeaving,andSystemCultivating)andreflectscarefulconsiderationofhowtheLeverswillintersect.
OperationalAlignmenttoStrategy
Towhatextentaretheintermediary’sday-to-dayactionsandprogramslinkedtoitsstrategy?
• Coreprogramsandserviceslackalignmenttotheintermediary’sarticulatedstrategyand/orstaffmember’sday-to-daytasksdonotsupportcoreprogramsandservices.
• Someprogramsandservicesmapbacktotheintermediary’sarticulatedstrategywhileothersdonot,and/orsomeday-to-daystafftasksaligntotheseprogramsandserviceswhileothersdonot.
• Mostprogramsandservicesmapbacktotheorganization’sarticulatedstrategy,andday-to-daystafftaskslargelyaligntotheseprogramsandservices,minimizingwastedtimeandresources.
• Thereistightalignmentbetweenprogramsandservices,andtheorganization’sarticulatedstrategyandday-to-daystafftasksaligntotheseprogramsandservices,eliminatingwastedtimeandresources.
OrganizationalCapacityCRITERIA WEAKEVIDENCE DEVELOPINGEVIDENCE SUFFICIENTEVIDENCE EXTENSIVEEVIDENCE
InternalResourcesTowhatextentdoesthe
intermediaryhavesufficientfinancialandhumanresourcestosupportitsstrategy?
• Theintermediarycurrentlydoesnothavesufficientresourcestoimplementtheactivitiesthatmakeupitsarticulatedstrategyandhasnoplanforattainingtheseresources.
• Theintermediaryisapproachingsufficientresourcestoimplementtheactivitiesthatmakeupitsarticulatedstrategyorhasacoherentplanforattainingtheseresources.
• Theintermediarycurrentlyhassufficientresourcestoimplementtheactivitiesthatmakeupitsarticulatedstrategy.
• Theintermediarycurrentlyhasmorethansufficientresourcestoimplementtheactivitiesthatmakeupitsarticulatedstrategy.
ConnectednessTowhatextentdoesthe
intermediaryhaverelationshipswithfield
(e.g.,K-12education)andsystempartners(e.g.,policymakesr,funders,technicalassistance
providers)acrossthelocalcontext?
• Theintermediaryhasnotbuiltrelationshipswithkeyfieldorsystempartnersandisisolated.
• Theintermediaryhasbuiltrelationshipswithsomekeyfieldand/orsystempartnersandisbecomingpartofanetworkoflike-mindedorganizations.
• Theintermediaryhasbuiltrelationshipswithmanykeyfieldand/orsystempartnersandispartofanetworkoflike-mindedorganizations.
• Theintermediaryhasbuiltrelationshipswithanextensivenumberofkeyfieldand/orsystempartnersandisawellknowplayerinanetworkoflike-mindedorganizations.
C E N T E R F O R P U B L I C R E S E A R C H A N D L E A D E R S H I P
AlignedtotheDiffusionIntermediaryEvaluationFramework
90
LearningOrientationTowhatextentdoestheintermediaryregularlyreviewandreflectonitssuccessesandfailures,changesininternal
organizationalcapacity,andexternalcontextualconditionsinorderto
learnandmakemidcoursecorrections?
• Theintermediaryisnotpreparedtocollectandadapttoevidenceofitsownsuccessesorfailures,oradapttochangesinorganizationalcapacityorcontextualconditions.
• Theintermediaryissomewhatpreparedtocollectandadapttoevidenceofitsownsuccessesorfailures,oradapttochangesinorganizationalcapacityorcontextualconditions.
• Theintermediaryismostlypreparedtocollectandadapttoevidenceofitsownsuccessesorfailures,oradapttochangesinorganizationalcapacityorcontextualconditions.
• Theintermediaryisfullypreparedtocollectandadapttoevidenceofitsownsuccessesorfailures,oradapttochangesinorganizationalcapacityorcontextualconditions.
LeadershipTowhatextentdoestheintermediary’sleadershipexhibitcharacteristics(e.g.,vision,strategicplanning,inspiration,
creativity,supportofstaff,alearningstance)thatallowthemtoeffectivelyandefficientlymanage
theorganization?
• Thereisnoevidenceoftheintermediary’sleadershiphavingthecharacteristicsneededtoefficientlyandeffectivelymanagetheorganization.
• Thereissomeevidenceoftheintermediary’sleadershiphavingthecharacteristicsneededtoefficientlyandeffectivelymanagetheorganization.
• Thereissomeevidenceoftheintermediary’sleadershiphavingthecharacteristicsneededtoefficientlyandeffectivelymanagetheorganization.
• Thereisextensiveevidenceoftheintermediary’sleadershiphavingthecharacteristicsneededtoefficientlyandeffectivelymanagetheorganization.
FieldInfluenceTowhatextentisthe
organizationconsideredaninfluentialleaderinthe
K-12educationfield?
• TheintermediaryisnotyetwellregardingintheK-12educationfield,anditsworkdoesnotyetserveasamodelforthefield.
TheintermediaryissomewhatwellregardedintheK-12educationfield,anditsworkservesasamodelforsomeinthefield.
TheintermediaryiswellregardedintheK-12educationfield,anditsworkservesasamodelformanyinthefield.
TheintermediaryisextremelywellregardedintheK-12educationfield,anditsworkconsistentlyservesasamodelforthefield.
C E N T E R F O R P U B L I C R E S E A R C H A N D L E A D E R S H I P
AlignedtotheDiffusionIntermediaryEvaluationFramework
91
Phase II: Interim Progress
2.1ImplementationofStrategyCRITERIA WEAKEVIDENCE DEVELOPINGEVIDENCE SUFFICIENTEVIDENCE EXTENSIVEEVIDENCE
ExtentofActionsTakenHowmanyresources
and/orhowmuchsupportdidtheintermediary
provideovertheperiod?
• Theintermediaryhasnotmetitstargetsfortheamountofresourcesandsupportsitintendedtoprovideovertheperiod.
• Theintermediaryisapproachingitstargetsfortheamountofresourcesandsupportsitintendedtoprovideovertheperiod.
• Theintermediaryachieveditstargetsfortheamountofresourcesandsupportsitintendedtoprovideovertheperiod.
• Theintermediaryexceededtargetsfortheamountofresourcesandsupportsitintendedtoprovideovertheperiod.
QualityofActionsTakenTowhatextentweretheresourcesandsupportsof
highquality?
• Theintermediary’sresourcesandsupportsdidnotmeetstandardsofquality.
• Theintermediary’sresourcesandsupportsareapproachingstandardsofquality.
• Theintermediary’sresourcesandsupportsmetstandardsofquality.
• Theintermediary’sresourcesandsupportsexceededstandardsofquality.
2.2Short-termOutcomesCRITERIA WEAKEVIDENCE DEVELOPINGEVIDENCE SUFFICIENTEVIDENCE EXTENSIVEEVIDENCE
PartnerUnderstandingTowhatextentareindividualpartnersexpandingtheirunderstandingof
innovation?
• Theintermediary’sindividualpartnershavenotyetincreasedtheirunderstandingofkeyknowledgeandskillsrelatedtotheinnovation.
• Theintermediary’sindividualpartnershavebeguntoincreasetheirunderstandingofkeyknowledgeandskillsrelatedtotheinnovation.
• Theintermediary’sindividualpartnershavesubstantiallyincreasedtheirunderstandingofkeyknowledgeandskillsrelatedtotheinnovation.
• Theintermediary’sindividualpartnershavegainedextensiveexpertiseinkeyknowledgeandskillsrelatedtotheinnovation.
PartnerPerceptionsTowhatextentareindividualpartners
developingpositiveviewstowardtheinnovation?
• Theintermediary’sindividualpartnershavenotyetimprovedtheirviewstowardtheinnovation.
• Theintermediary’sindividualpartnershavebeguntoimprovetheirviewstowardtheinnovation.
• Theintermediary’sindividualpartnershavesubstantiallyimprovedtheirviewstowardtheinnovation.
• Theintermediary’sindividualpartnershavebecomeengagedandactiveproponentsoftheinnovation.
C E N T E R F O R P U B L I C R E S E A R C H A N D L E A D E R S H I P
AlignedtotheDiffusionIntermediaryEvaluationFramework
92
NetworkFormWhatindividualsor
organizationsmakeupthenetwork?
Howcantheconnections
betweennetworkmembersbecharacterized?
• Thenetworkhasnotreacheditstargetspertainingtomembershipsizeandcomposition.
• Therearefewifany
connectionsbetweennetworkmemberssofarandconnectionsthatdoexistarenotproducingaddedvalueformembersorforthefieldatlarge.
• Thenetworkhasnearlyreacheditstargetspertainingtomembershipsizeandcomposition.
• Therearesome
connectionsbetweennetworkmembersandtheseconnectionsarebeginningtoproduceaddedvalueformembersorforthefieldatlarge.
• Thenetworkhasreacheditstargetspertainingtomembershipsizeandcomposition.
• Therearesubstantial
connectionsbetweennetworkmembersandtheseconnectionsareproducingaddedvalueformanynetworkmembersorforthefieldatlarge.
• Thenetworkhassurpassedittargetspertainingtomembershipsizeandcomposition.
• Therearesubstantial
connectionsbetweennetworkmembersandtheseconnectionsareproducingaddedvalueforallornearlyallnetworkmembersaswellasforthefieldatlarge.
NetworkFunctionTowhatextentdoesthe
networkhavetheresources,structures,andconditionsinplaceneededtooperateeffectivelyand
efficiently?
• Thenetworkdoesnotyethavetheinfrastructure,operationalsystems,orconditionsneededtofunctioneffectivelyandefficiently.
• Thenetworkhassomeoftheinfrastructure,operationalsystems,andconditionsneededtofunctioneffectivelyandefficiently.
• Thenetworkhasmostoftheinfrastructure,operationalsystems,andconditionsneededtofunctioneffectivelyandefficiently.
• Thenetworkhastheinfrastructure,operationalsystems,andconditionsneededtofunctioneffectivelyandefficiently.
SystemPoliciesandPractices
Towhatextentaretheformalandinformal
policiesandpracticesthatimpactpartnersenablingordemandinginnovation?
• Formalandinformalpoliciesandpracticesthatimpactpartnersareprohibitiveofinnovation.
• Formalandinformalpoliciesandpracticesthatimpactpartnersallowforbutdonotpromoteinnovation.
• Formalandinformalpoliciesandpracticesthatimpactpartnerspromoteinnovation.
• Formalandinformalpoliciesandpracticesthatimpactpartnersrequireinnovation.
SystemEngagementandPublicWill
Towhatextentisthepublicdemandingandsupportingchange?
• Thepublicisrisk-averseandactivelyprotestsinnovation.
• Thepublicislargelydisengagedfrominnovation,butdoesnotposeanaddedbarriertothework.
• Thepublicisbeginningtocallforandsupport,eitherverballyorfinancially,innovation.
• Thepublicisdemandinginnovationandisactivelysupportingitverballyorfinancially.
C E N T E R F O R P U B L I C R E S E A R C H A N D L E A D E R S H I P
AlignedtotheDiffusionIntermediaryEvaluationFramework
93
2.3ContinuousLearningCRITERIA WEAKEVIDENCE DEVELOPINGEVIDENCE SUFFICIENTEVIDENCE EXTENSIVEEVIDENCE
UnderstandingofProgress
Towhatextenthastheintermediaryidentifiedareasofsuccessand
failure?
Whatlessonslearnedhastheintermediarypulledfromitssuccessesand
failuresthatcouldbenefititsownworkorthelarger
field?
• Theintermediaryisnotcollectingevidenceofitsownsuccessorfailure.
• Thegranteehasnotpulledhelpfullessonslearnedfromitsprogressthatitcanrespondtointernallyorthatactorsinthefieldcanlearnfrom.
• Theintermediaryiscollectinglimitedevidenceofitsownsuccessorfailure.
• Thegranteehaspulledsomehelpfullessonslearnedfromitsprogressthatitcanrespondtointernallyorthatactorsinthefieldcanlearnfrom.
• Theintermediaryiscollectingavarietyofevidenceofitsownsuccessorfailure.
• Thegranteehaspulledvarioushelpfullessonslearnedfromitsprogressthatitcanrespondtointernallyorthatactorsinthefieldcanlearnfrom.
• Theintermediaryiscollectingcomprehensiveevidenceofitsownsuccessorfailure.
• Thegranteehaspulledan
extensivenumberofhelpfullessonslearnedfromitsprogressthatitcanrespondtointernallyorthatactorsinthefieldcanlearnfrom.
MidcourseCorrectionsTowhatextentisthe
intermediaryadjustingitsstrategyororganizationalcapacityasaresultof
lessonslearned?
• Theintermediaryhasnotadapteditsvisionandstrategyororganizationalcapacityasaresultoflessonslearned.
• Theintermediaryhasmadesomeinitialchangestoitsvisionandstrategyororganizationalcapacityasaresultoflessonslearnedbuthasnotfullyrespondedtotheselessons.
• Theintermediaryhasmadesubstantialchangestoitsvisionandstrategyororganizationalcapacityasaresultoflessonslearnedbuthasnotfullyrespondedtotheselessons.
• Theintermediaryhasmadesubstantialchangestoitsvisionandstrategyororganizationalcapacityasaresultoflessonslearneddemonstratingafullandcompleteresponsetotheselessons.
C E N T E R F O R P U B L I C R E S E A R C H A N D L E A D E R S H I P
AlignedtotheDiffusionIntermediaryEvaluationFramework
94
Phase III: Impacts on the Field
3.1DiffusionofInnovationCRITERIA WEAKEVIDENCE DEVELOPINGEVIDENCE SUFFICIENTEVIDENCE EXTENSIVEEVIDENCEDepths
Towhatextentarechangestopracticedeepandconsequential,thusreflectingnewvaluesand
beliefs?
• Changestoinstructionreflectnoshiftsineducators’viewsandbeliefsabouthowstudentslearn,theroleofteacherandstudent,andsubjectcontent.
• Changestoinstructionarebeginningtoreflectsomefundamentalshiftsineducators’viewsandbeliefsabouthowstudentslearn,theroleofteacherandstudent,andsubjectcontentthatsomewhatalignstoinnovation.
• Changestoinstructionreflectsubstantialshiftsineducators’viewsandbeliefsabouthowstudentslearn,theroleofteacherandstudent,andsubjectcontentthatmostlyalignstoinnovation.
• Changestoinstructionreflectacompleteshiftineducators’viewsandbeliefsabouthowstudentslearn,theroleofteacherandstudent,andsubjectcontentthatalignstoinnovation.
SpreadTowhatextentare
changestoinstructionalpracticeexpanding
outwardstomoreandmoreclassrooms,schools,
and/ordistricts?
Towhatextentarechangestoinstructional
practiceexpandinginwardsandinfluencingclassroom,school,andpoliciesandoperating
procedures?
• Innovationhasnotyetspreadoutwardtoevenasmallnumberofclassrooms,schools,and/ordistrictsinthetargetedregion.
• Innovationhasnotyet
spreadinwardstochangeclassroom,school,ordistrictstandardoperatingproceduresorareasnotoriginallytargetedforchange.
• Innovationisbeginningtospreadoutwardtoanumberofclassrooms,schools,and/ordistrictsinthetargetedregion.
• Innovationisbeginningtospreadinwardstochangeclassroom,school,ordistrictstandardoperatingproceduresorareasnotoriginallytargetedforchange.
• Innovationhasspreadoutwardtoasubstantialnumberofclassrooms,schools,and/ordistrictsinthetargetedregion.
• Innovationhasspreadinwardstosubstantiallychangeclassroom,school,ordistrictstandardoperatingproceduresorareasnotoriginallytargetedforchange.
• Innovationhasspreadoutwardtothevastmajorityofclassrooms,schools,and/ordistrictsinthetargetedregion.
• Innovationhasspreadinwardsandcompletelychangedclassroom,school,ordistrictstandardoperatingproceduresorareasnotoriginallytargetedforchange.
OwnershipTowhatextentis
authorityforthereformbeingtakenonbythedistricts,schools,and
teachers?
• Teachers,schools,ordistrictshavenotyettakenonauthorityformanagingandexpandinginnovationandexternalpartiesremainthedrivingforce.
• Teachers,schools,ordistrictsarebeginningtotakeonauthorityformanagingandexpandinginnovationbutexternalpartiesremainadrivingforce.
• Teachers,schools,ordistrictshavetakenonsubstantialauthorityformanagingandexpandinginnovationandexternalpartiesarenolongerthedrivingforce.
• Teachers,schools,ordistrictshavetakenoncompleteauthorityformanagingandexpandinginnovationandexternalpartiesarenolongeradrivingforce.
C E N T E R F O R P U B L I C R E S E A R C H A N D L E A D E R S H I P
AlignedtotheDiffusionIntermediaryEvaluationFramework
95
SustainabilityTowhatextentdochangestopracticeremaininplaceafter
externalsupportsarenolongerpresent?
• Innovationhasnotremainedinplacesincedepartureofexternalsupports;instructionalmodelandstandardoperatingproceduresareshiftingbacktotheirpreviousstate.
• Innovationhasremainedsomewhatinplacesincedepartureofexternalsupports;instructionalmodelandstandardoperatingprocedureshavenotcompletelyshiftedbacktotheirpreviousstate.
• Innovationhasmostlyremainedinplacesincedepartureofexternalsupports;instructionalmodelandstandardoperatingprocedureshaveonlyslightlyshiftedbacktotheirpreviousstate.
• Innovationhasremainedfullyinplacesincedepartureofexternalsupports;instructionalmodelsandstandardoperatingprocedureshavenotshiftedbacktotheirpreviousstate.
EquityTowhatextentarechangestopractice
occurringequallyacrossdifferentsocioeconomic
groups?
• Innovationisisolatedtosmallpocketsoftheregion,resultinginunequalopportunitiesforstudentsalignedtosocioeconomicstatus.
• Innovationismostlyisolatedtosmallpocketsoftheregion,resultinginunequalopportunitiesforstudentsalignedtosocioeconomicstatus.
• Innovationismostlybalancedacrosstheregion,resultinginlargelyequalopportunitiesforstudentsalignedtosocioeconomicstatus.
• Innovationisbalancedacrosstheregion,resultinginequalopportunitiesforstudentsalignedtosocioeconomicstatus.
3.2ImprovedStudentOutcomesCRITERIA WEAKEVIDENCE DEVELOPINGEVIDENCE SUFFICIENTEVIDENCE EXTENSIVEEVIDENCEKnowledge
Towhatextentarestudentsdeveloping
increasedunderstandingofimportantcontent
knowledge?
• Fewifanystudentshaveincreasedtheirmasteryofcontentknowledge.
• Somestudentshaveincreasedtheirmasteryofcontentknowledge.
• Moststudentshaveincreasedtheirmasteryofcontentknowledge.
• All,ornearlyall,studentshaveincreasedtheirmasteryofcontentknowledge.
CognitiveandMetacognitiveSkillsTowhatextentare
studentsdevelopingkeymentalprocessingskillsas
wellastheabilitytomonitorandassesstheuseoftheseskills?
• Fewifanystudentshaveexhibitingincreasedcognitiveandmetacognitiveskills.
• Somestudentshaveexhibitingincreasedcognitiveandmetacognitiveskills.
• Moststudentshaveexhibitingincreasedcognitiveandmetacognitiveskills.
• All,ornearlyall,studentshaveexhibitingincreasedcognitiveandmetacognitiveskills.
MindsetsandDispositions
Towhatextentarestudentsdevelopingthequalitiesandmindsets
neededtobesuccessfulincollege,career,andlife?
• Fewifanystudentshaveexhibitingimprovedmindsetsanddispositionsneededtobesuccessfulincollege,career,andlife.
• Somestudentshaveexhibitingimprovedmindsetsanddispositionsneededtobesuccessfulincollege,career,andlife.
• Moststudentshaveexhibitingimprovedmindsetsanddispositionsneededtobesuccessfulincollege,career,andlife.
• All,ornearlyall,studentshaveexhibitingimprovedmindsetsanddispositionsneededtobesuccessfulincollege,career,andlife.