a roadmap for policy change - jhu center for applied ......the roadmap, there's information...
TRANSCRIPT
1
Titlepage
2
11
12
13
131
132
133
134
135
136
137
138
14
15
16
17
18
TableofContentsTitlepage
IntroductionandpurposeoftheRoadmap
UsingtheRoadmap
Gatherinformationabouttheproblem
Strategicallyengagekeypartners
Reviewtheevidenceandcollectdetails
Decideonapolicytoimplementbasedonlocalfactors
Establishaframeworktomonitorprogress
Assessdataaccessibilityqualityandsecurity
Designimplementationandevaluation
Shareyourstoryandsuccess
Workingwitharangeofevidence
Thankyou
References
Resources
AboutGovEx
IntroductionRemembertheDAREprogramWhatbeganasapartnershipbetweentheLAPDandLApublicschoolstohelpkidssaynotodrugsbecameapoliticallypopularcampaignthatwasadoptedbyschooldistrictsaroundthecountryMeanwhilethroughthe1990sand2000smorethan30rigorousevaluationsfoundthattheprogramdidnotwork-withonestudyevenfindingthatitactuallyincreasedyouthdruguseThreedecadeslaterthefederalgovernmentisstilltalkingaboutrevivingthispopularbutdetrimentalprogram
Thechallengesthatcitiesface
ThestoryofDARErevealssomeoftheotherchallengesfacinggovernmentsinimplementingevidence-basedpoliciesPresentedwithconstraintsontimeandpressuretoactpolicymakersmayrelyonemotionsandgutfeelingsasashortcuttodecisionmakingItcanbehardtorejectthefundingandpoliticalcapitalthatoftencomeswithaprogramthathasthekindofmomentumthatDAREhadEvenifelectedofficialsadmitthatapolicyisproblematicoratleastnolongereffectiveittakesleadershiptoputenergyandresourcesintochangingcourseAndeventhencityleadershavetomakethecasetoresidentsforwhyachangeisnecessaryespeciallyaftertheyhaveinvestedtimeandpublicdollarsintoapreviousapproach
LocalgovernmentsareincreasinglyonthefrontlinesofaddressinglargescaleproblemswithglobalimplicationssuchasclimatechangeimmigrationandeconomicinequalityAlthoughresearchersarevalidatingdiagnosticmodelsandevidence-basedinterventionstohelpsolvethoseproblemslocalgovernmentstaffhavelimitedtimeandcapacitytofindevaluateandimplementthesesolutionsSearchesbycitystaffmayleadtopromisingexamplesinothercitieslinkstoscholarlyarticleswebsitesofnonprofitsthatworkonthisissuejournalarticlesandrelatednewsarticlesRatherthanweighingandevaluatingthisinformationcitiesmaychoosetotryonepolicysolutionthatseemsfeasibleWithoutcarefulmonitoringorevaluationthoughthistrialanderrorapproachcanleadtoanenvironmentinwhichcitiesfundserviceprovidersbasedonexistingrelationshipsandavoidrepurposingfundingfromprogramsthatareineffectiveforfearofrepercussionssuchaspeoplelosingtheirjobsFurtherconversationsaboutwhatothercitiesaredoingandtheavailableliteraturesometimeshappenonanadhocbasiscitystaffdonotoftenfeelpreparedtoconductasystematicreviewWhencitiesdomakechangestheyaresometimesincrementalandattheprogramlevelinsteadofaddressingmultiplefactorsatonceatasystemlevelAttemptingtoaddressaproblemthatstoobroadsuchasblightcanbe
IntroductionandpurposeoftheRoadmap
3
dauntingWithoutadeepunderstandingoftherootcausesoftheproblemitcanbedifficulttoidentifythebestevidencetouseFinallyitcanbedifficulttoknowhowtoimplementaparticularpolicyinyourcityandwhetheritwillhavethedesiredeffect
WhentheUnifiedGovernmentofKansasCityKSandWyandotteCounty(UG)beganworkingonitsblightproblemtheUGconvenedaresearchteamtolookforevidencefromothercitiesUsingsearchtermslikeblightlandbankabandonedpropertiesandvacancythatledtheUGtolearnaboutthesimilareffortsinthecitiesofNewOrleansBaltimoreMobileKansasCityMOandcitiesinOhioandMemphisTheteamalsoremainedopentohearingideasfrominternalstaffwholearnedaboutpracticesfromothercitiesthroughtheirownprofessionalorganizationsanduniversityaffiliationsthattheyhadnotbeenempoweredtotestorimplementpreviouslySincethetopicofblightrangesfrompropertymaintenancetosafetyissuestheCityresearchteamhadtodoalotofreadinganddebatingoverconflictingconceptsfortacklingawiderangeofissueswithinthatspectrumThisinvolvedaskingquestionsaboutwhetherthepolicywillworkinKansasCityandwhatchangeswouldneedtobemadeforittobesuccessfulWhenresearchinglandbankstheCitylearnedabouttheCenterforCommunityProgressanonprofitthathelpscommunitiesaddressvacantabandonedanddeterioratedpropertiesConversationswiththeCenterforCommunityProgressledtoexploringtheideaofavacantpropertyregistryUponconductingfurtherresearchUGstaffcameacrossinformationonacourtrulingfromChicagothatcertainvacantbuildingsdonotneedtofollowtheCityofChicagorsquosvacantbuildingordinanceThisdirectedthesearchtounderstandifothercitieshadexamplesofwhethervacantpropertyregistriesareworkingTodeterminewhetherthismightworkinthecontextofKansasCitystaffbeganvettingideastoseeiftherewereanyaspectsthatconflictedwithCityordinancesTheUGalsoexaminedthefeasibilityofimplementingthenecessarytechnologyandfeasibilityfromtheperspectivesofpropertyownersandUGstaffThisinitialresearchandanalysisphasetookaboutsixmonthsandwasledbytheGeneralServicesDirectoraVISTAinternandtheresearchteamIndevelopingthisRoadmaptheCitymentionedthatitwouldbehelpfultohaveaccesstocasestudieswithdetailedinformationonthepolicysolutionsthatcitieshavetakenandthespecificsofhowtheyhavebeenimplemented
PurposeoftheRoadmap
ThisRoadmapismeanttoempowercitiestotakeownershipofthepolicychangeprocesstoframeproblemsinawaythatcreatesademandforevidenceandtoformlong-termrelationshipswithpoliticalpolicypractitionerandresearcheralliesThisworkisnecessaryResearchfromthefieldofimplementationsciencesuggestssuccessfulimplementationofevidenceintopracticehappenswhencitiestakethetimetoidentifytherightpeopletoworkwithchoosethebestevidencefromavarietyofsourcesandconsiderthebestwaytoadapt
IntroductionandpurposeoftheRoadmap
4
thatevidencetotheirowncontextThroughourinterviewswithcitystaffweobservedthatframingtheproblemclearlyformingadvisorygroupsorsomeothercoalitionandidentifyingandimplementinganappropriatesolutionarenecessaryingredientstosuccessfulpolicychange
ThroughthisRoadmapGovExhopestofacilitateanewwaytohelpAmericancitiesrethinkthehowtheysolveproblemsandaddressthemostsalientpolicychallengesinurbangovernance
IntroductionandpurposeoftheRoadmap
5
UsingtheRoadmap
WhoshouldusethisRoadmap
ThisRoadmapisforanystaffmemberincitygovernmentwhoischargedwithsolvingaproblemorimplementingapolicyAlthoughthestepscanbeusedtochangeexistingprogramsorimplementnewprogramsthefocusoftheRoadmapisonlong-termsustainablepolicychangeWeusethetermpolicytomeanlawsregulationsfundingdecisionsandinterventions
CitystaffwhoarereadingthisguidemaybecomingfromdifferentstartingpointsItisimportanttobehonestaboutwherethatisForinstanceyoumaybelookingfornewsolutionstolongstandingseeminglyintractableproblemssuchasanannualincreaseingunviolenceormaybeyouarefacingnewerchallengesliketheopioidepidemicOrperhapsyouarelookingforadata-drivenwaytoaddressissuesrelatedtoday-to-dayservicedeliverysuchaspermittingorbikelanesThestepsareapplicablewhetheryouareworkingonalargescaleissueoralocalmunicipalissue
YoucanusethisguidewhetheryouareadecisionmakerorfrontlineworkerThroughouttheRoadmaptheresinformationaboutwhenandhowtoengageotherpeopleCollaborationisakeythemethroughthisRoadmapThegoodnewsisyoudonothavetodothisworkalone
TheapproachandthetoolscontainedintheRoadmapshouldberelevantwhetheryouareinastrongmayororcouncil-managerformofgovernmentElevatingthisprocesstothelevelofMayorCityManagerorCityCouncilmaytaketimeandmultiplestepsbutwehopeyouwillfindwaystoincorporatedataandevidenceintoyourday-to-daywork
WhatarethestepsoftheRoadmap
ThechaptersinthisRoadmaprepresentthehigh-levelstepsinvolvedinpolicychangeYouwillnoticethatthesectionsrefertooneanotherandbuildonthepreviousstepsTheRoadmapismeanttobeiterativeyouwillrevisitsomesteps
TherootsoftheproblemsthatcitiesareworkingtosolvearecomplexandmaybethelegacyofhistoricalfederalpolicyorsweepingsocialtrendsAlthoughthisRoadmapdoesnotexplicitlyincludestepsonreviewingthishistoryweencouragereaderstodevelopadeepunderstandingoftheprobleminyourowncitywhileconsideringcurrentstateandfederalcontext
UsingtheRoadmap
6
IninterviewswithGovExonthepolicychangeprocessintheircitiespeoplewasthewordmostcommonlyusedbyintervieweeswhendescribingthevariousstepstoimplementanewpolicyorprogramYouwillnoticethatthisisathemethroughoutthisRoadmapThefirstsectionbeginswithidentifyingtherelevantpeopleThechapteronidentifyingrolesforkeypartnersservesasafoundationforthefollowingstepsThediscussionofevidenceencouragesreaderstoconsiderwaysinwhichpeoplecanbuildanduseevidencetodesignstrongprogramsandinitiativeschangehumanbehaviorassessimpactensureeffectiveuseofpublicresourcesandimprovelives
Meaningfulapplicationofevidencebypolicymakersdoesnothappenwithone-timesharingofcuttingedgeevidenceRathermeaningfulinfluencerequiresalong-termstrategytoformalliancesanddevelopknowledgeofpolicymakersandpolicymakingInadditiontoworkingwiththerightpeopleandunderstandingevidenceandcontextwehopethatRoadmapuserswillformapeernetworkthatsharesnotjustwhichpolicysolutionsexistbutwhatimplementationlookslikeAsonecitywespoketoindevelopingthisRoadmapputitalotofourcitysproblemshavebeensolvedbyotherswejustneedtofigureouthowtoimplementthesolutions
TheexamplesthroughouttheRoadmapareillustrativeofgovernmentsthathaveeffectivelyimplementedtherelevantstepsinthisRoadmapbutpleasenotethatnocitydoesallofthisperfectlyTogetherthestepsinthisRoadmaparemeanttohelpcitiesframetheirproblemscreateademandforevidenceandbuildsustainablecoalitionsformeaningfulpolicychange
UsingtheRoadmap
7
GatherinformationabouttheproblemInyourearliestphaseitiscriticaltodevelopacomprehensiveunderstandingoftheproblemYouwillwanttolearnaboutthepeoplewhoareaffectedbytheproblemyouaretryingtosolveandthosewhocanimpacttheproblemFromthereyouwillgetinsightintotherootcauseoftheproblemtheworkthathasbeentriedinthepastandtheeffectivenessofthoseeffortsTheworkfromthischapterwillserveasthefoundationforactivitiesinsubsequentchaptersYoumayrevisitthesestepsasyouidentifyindicatorsandplanyourimplementationandanalysis
BesuretodedicatetimeandresourcestothisstepItcanbetemptingtoskipaheadtolookingforsolutionsortoassumethatyouhaveanunderstandingoftheproblemasaresultofapublicengagementprocess
TheCityofLewisvilleTXconductedaCitywidepublicengagementprocesstodevelopitsLewisville2025StrategicPlanOneoftheBigMovesoutlinedintheplanwasExtendingtheGreenorexpandinggreenspaceandconnectingrecreationareaswithneighborhoodsTheactionstepstoachievethisgoalincludedcleaningupcreeksandremovingdebristomakethemmoreaccessibleThecleanupswentsmoothlyinseveralneighborhoodsbuttheCitymetwithoppositionfromoneneighborhoodthatpreferredthenaturalhabitatandexpressedconcernabouttheenvironmentalimpactofthecleanupsAccordingtoLewisvilleiftheCityhadmappedallofthestakeholderstodeterminewhoneededtobeatthetableandwhowasmissingfocusinginformationgatheringontheneighborhoodlevelithaveavoidedthisproblem
Startwiththewhothepeople
Establishaprojectteam
BeginbynamingtheprojectteamforthisprocessincludingaprojectleadTheprojectleadshouldbesomeoneinCitygovernmentwhoisrespectedcangalvanizeotherkeypartnersandhasthetimetoleadthisworkThisissomeonewhomayhavetohavedifficultconversationswithCityleadershipdepartmentsandotherfundedpartnersItisnotnecessaryfortheprojectleadtobeapolicyexpertItismoreimportantforthemtohavetherightrelationshipsandpositionwithintheorganizationTheprojectteamshouldincludeindividualswithdiverseperspectivesfrommultipledepartmentsThereshouldbesomeonewithpolicyexperienceontheprojectteamtogroundtheconversationinrealisticactionHavingateammemberwhocancoordinatewithresearchersonrigorousanalysisandtranslatethatworkforothersiskeyAdditionallybesuretoincludefrontlinestaffworkersTheywillhavevaluableinsightbasedontheirinteractionswithresidentsandpolicysolutions
Gatherinformationabouttheproblem
8
havetomakesensetothepeoplewhowillbechargedwithimplementingthemBuildingastrongdiverseprojectteamisimportanttoensurethattheworkdoesnotliveanddiebyoneperson
Identifykeypartners
NextidentifythepeoplewhotheprojectteamcanworkwithontheissueyouarefacingincludingpeoplewithlivedexperienceintheproblemyourcityistryingtosolvedepartmentstaffandsubjectmatterexpertsoutsideofcitygovernmentThisistheconceptbehindbuildwithnotforamovementledbyLaurenellenMcCannthatemphasizescommunityexpertiseandleadershipMappingoutkeyplayersandtheirrolesiscriticaldotoearlyintheinformationgatheringphasesincecoalitionbuildingissuchanintegralpartofthepolicychangeprocessThisprocessmightconsistofstakeholdermapping(describedbelow)communityresourcemappingandculturalmappingInyourmappingremembertoconsiderpotentialalliesandthosewhomaypresentroadblocksalongtheway
PolicymakingisnottopdownorlinearInsomecasescitiesmayhavetosharedecision-makingpowerwithotheractorsForinstanceinmanycitiesthelocalContinuumofCare(CoC)istheleadentityonhomelessnessservicesCoCsapplyforandstewardfederalfundstoaddresshomelessnessandcoordinateservicesamonglocalprovidersAlthoughcitygovernmentsdohavearoleinCoCstheyarenotalwaystheleadagencychargedwithdistributingfundsandthereforetheirroleinhomelessnesspolicymaybelimitedCountygovernmentsoftenhavehealthdepartmentswhilecitiesdonotInthosecasesthebalanceofpowerfavorscountygovernment
TheCityofDentonTXwasrecentlyfacedwithanissueofresidentcomplaintsaboutanewdocklessbikesharethatarrivedintownandisviewedbysomeasexpensivelitterInresponseDentonsCityCouncilwhichwasalreadydiscussingpotentialrulesforbikesharecompanieselevatedthisissueandchargedcitystaffwithidentifyingasolutionPresentlytheCityisreviewingordinancesfromnearbycitiestodeterminehowtheyaresolvingthisproblemandwhatsolutionsmaybefeasibleinDentonThemappingexamplebelowrepresentslocalactorsinDentonwithastakeinthisissuewhohavemultipleperspectivesandlevelsofpower
Gatherinformationabouttheproblem
9
UsethistemplatedevelopedbytheCentreforPublicImpact(CPI)tobrainstormwhoyoumightreachouttoasyouembarkontheroadtopolicychangeRemembertothinkabouthoweachpersonorgrouprelatestotheproblem
BegintodefinethewhatortheprobleminyourCity
TherestofthissectionwillhelpyourefinetheproblemstatementFornowgathersomekeypartnerstoclearlyarticulatewhychangeisneededUsethetemplatebelowdevelopedbyDenverPeakAcademytocapturetheproblemstatementYouwillbuildonthislater
ProblemStatementndashWhyChangeisNeeded(1-2Sentences)
ThroughourinterviewswithcitystaffweheardthatwhendiscussingmunicipalchallengesitcanbetemptingtojumptoasolutionbeforetakingthetimetoreallyunderstandtherootcauseoftheproblemThismakessensesincecityleadersandstaffareresponsiblefordeliveringservicesandbeingresponsivetoresidentswhileaddressingnewproblemsthatariseinthecity
TherootsoftheproblemsthatcitiesareworkingtosolvearecomplexandmaybethelegacyofhistoricalfederalpolicyorsweepingsocialtrendsTohelptheproblemseemlessdauntingitcanhelptobreakthemdownintotherootcausetoidentifyamanageablecomponentoftheproblemUsingafishboneexerciseorfivewhystechniquecanalsohelp
Gatherinformationabouttheproblem
10
tofindtherootcauseoftheproblemandavoidsimplyaddressingsymptomsoftheproblemandrepeatingthemistakesofhistoryToolssuchasjourneymapscanhelpyouunderstandtheexperienceofthosewhoareaffectedbytheproblem
TheCentreforPublicImpactwithsupportfromBloombergPhilanthropiesdevelopedanexercisetosupportcitiesinidentifyingtherightsizedproblemAccordingtoCPIarightsizedproblemisonethatislargeenoughtopotentiallyapplytoothercitiesbutsmallenoughtosolvewithinareasonabletimeframe
AtthebeginningofMayorBerkesadministrationintheCityofChattanoogaTNveteransexperiencinghomelessnesswereaskingtheCityforhelpinfindinghousingTalkingtothispopulationrevealedthatveteranshadtroubleidentifyingaffordablehousingwhichmotivatedtheAdministrationtoworkonasolutionTheproblemofveteransexperiencinghomelessnesswastherightsizebecauseitfocusedononesegmentofpeopleexperiencinghomelessness-veterans-andtheirspecificchallengeinidentifyingaffordablehousing
Developacomprehensiveunderstandingofhowyourcitycurrentlyaddressestheissue
IntheChattanoogaexamplementionedaboveCitystaffdiscoveredthatveteranswerehavingtroublenavigatingtheCitysvariousaffordablehousingprogramsandgettingaccesstothehousingandservicesthattheyneededAtthesametimenewlyelectedMayorBerkewasexploringcurrentandfutureopportunitiestoservethispopulationRatherthancontinuetousefederalfundingforemergencysheltersMayorBerkedirectedcitystafftoprioritize
Gatherinformationabouttheproblem
11
HousingFirstwhichvaluesconnectingpeoplewithpermanenthousingovertransitionalhousingandsupportiveserviceafterspeakingwithrepresentativesattheUSDepartmentofHousingandUrbanDevelopment(HUD)
ThisexaminationofwhattheCityiscurrentlyfundingandthesourcesofthatfundingwasaturningpointinChattanoogaUnderstandingthisproblemfromasystemsperspective-includingpoliciesandfundingstreamsatmultiplelevelsofgovernmentsocialnetworksthroughwhichinformationisoftenspreadtechnologythebuiltenvironmentofmarket-rateandpublichousing-helpedthenewMayoranalyzeexistingeffortsandleadstaffandotheractorsinthesamedirection
Begintogatherthedata
TotrulycapturethesuiteofservicesthatyourCityisprovidingaskdepartmentstafftolisttheprogramsrelatedtoyourproblemstatementthatyourcityrunsorfundsWhenlistingtheprogramscapturedetailssuchas
programnamethenameofthedatasetthatisrelevanttoeachprogramadescriptionofthedatasetthesourceofthedatasetifthedataisinternalorexternalwhetherthisdataisalreadybeingcollected
ThisinformationwillformthebasisofyourinventoryofadministrativedataordatasetsthatarekeptbygovernmentsforoperationalpurposesThismayincludecallcenterrecords(311911)taxdatapropertyassessmentsinspectioninformationsocialservicerecordsetcAtthispointlistallrelevantadministrativedatasetsYouwillprioritizewhichdatasetstogathermoreinformationonlaterwhenyouaredevelopingindicators
Asampletemplatethatcanbeusedtocapturethisinformationisbelow
ProblemStatement____________________________________________
Gatherinformationabouttheproblem
12
ProgramName
DatasetTitle
DatasetDescription DataSource
Departmentorexternalpartner
DoestheCityalreadycollectthisdata
Whatisthenameoftheprogram
Human-readablenameofthedataset
Shouldbeunderstandablebynon-technicalusers
Whatistheoriginalsystemapplicationfilethathousesit
Departmentorexternalpartnerthatmaintainsthedata
Isthisdataalreadycollected
TogatherthisinformationconsiderassigningonepointpersontosurveyorinterviewcitystaffSpeakingwithfrontlineworkersstaffwhoworkwithfederalgrantsdepartmentheadsandeveryoneinbetweenwillensureadeepunderstandingoftherelevantprogramsanddatasetsDuringthecourseoftheseconversationsbesuretoaskcitystaffwhichorganizationstheyworkwithexternallyandcapturethatinformationaswelleveniftheyarenotrecipientsofcityfundingInadditiontoconversationswithexpertsincitygovernmentyourcitysbudgetdocumentsareanothergreatsourceofinformation
Collectdataandobservetrends
Nexttrytogainanunderstandingofwhytheproblemstillexistsifithasbeensomethingyouhaveworkedonforawhileorwhyitisemergingnowifitisnew
AnalyzingadministrativedatacandemonstratepasteffectivenessofinitiativesrelatedtoyourproblemstatementUsingthedatasetinventorythatyoubeganintheprevioussteptalktodepartmentstaffaboutwhichdatasetsaremostrelevanttotheproblemandgatherafewyearsworthofhistoricaldataBegintolookfortrendsinthisdatadiscussingpotentialreasonsforthosetrendswithrelevantstaffCanthetrendsbeexplainedbyseasonalitychangesinfundingstaffingfluctuationsorsomeotherreasonBasedonthehistoricaldataanalysiscreateabaselineagainstwhichfuturechangescanbeobserved
OtherdatasetscanhelpprovideadditionalcontextForinstanceifthenumberofpeopleexperiencinghomelessnessinyourcityisincreasingitmaybehelpfultotakealookathousingaffordabilityIfyoudonothavethedatayouneedconsideraproxymeasureAsanexamplehousingisoftenconsideredaffordableifitcosts30orlessofahouseholdsincomeButifhouseholdincomedataisnotavailableatthecitylevelAreaMedianIncome(AMI)canbeahelpfulproxy
Gatherinformationabouttheproblem
13
QualitativedatacanalsohelpanswerquestionsaroundtheeffectivenessofpreviouseffortsTrytoobtainavarietyofperspectives--bothmainstreamandoutliers--fromnetworksbeyondyourowntoprovideyouwiththegreatestvantagepointforevaluatingpotentialimpactUnderstandinghowthoseeffortswerereceivedbydifferentpartsofyourcommunityandiftheremaybeelementsofprevioussystemstopreservewillbecrucialtothesuccessofanypolicychangeThisinformationmaybeobtainedthroughfocusgroupsinterviewsandsurveysFormoreinformationonhowtousethesequalitativeresearchtoolspleaseseetheseresourcesdevelopedbytheAgencyforHealthCareandResearchandUrbanInstitute
FinallytakealookatnationaldatatoseewhereelsethisproblemmightexistThisresearchmaybehelpfulinlaterstepswhenyouarelookingforpromisingpoliciesinothercities
Puttingitalltogether
UsethetemplatebelowdevelopedbyDenversPeakAcademytocaptureyourrefinedrightsizedproblemstatementandcurrentandfuturestatesWorkingthroughthisexercisewithpartnersishelpfultogainbuy-inTheresponseswillbeimportantwhengettingotherpartnersonboard
WhyChangeIsNeeded
Yourright-sizedproblem
CurrentState
Analysisoftrendsindata
FutureState
Ideallywhataretheoutcomesyouhopetoachieve
Checklist
Thekeystepstogatheringinformationarebelow
Atthispointfocusontheproblemratherthanjumpingtosolutions
Listentopeoplewhoareaskingforhelpandtoadvocateswhoareraisingissues
Beasspecificaspossibleinidentifyingtherootcauseandhoninginonthepartoftheproblemthatyourcitycanaddress
Identifytrendsbasedonmultiplesourcesofdataincludinghistoricaladministrativedataproxymeasuresqualitativedataandnationalmeasures
Gatherinformationabouttheproblem
14
Captureyourproblemstatementtheresultsofyouranalysisanddesiredoutcomes
Gatherinformationabouttheproblem
15
StrategicallyengagekeypartnersPolicytheoryresearchoftenfocusesonincreasingpolicymakersaccesstoevidenceasawaytoavoidtakingshortcutslikemakingdecisionsbasedonemotionImplementationsciencesuggeststhatincreasingaccesstoevidenceisnotenoughandthatpolicychangeshouldbeexaminedinthewidercontextofdebatecoalitionformingandpersuasionMeaningfulpolicychangerequiresldquoalong-termstrategytoformalliancesanddevelopknowledgeofpolicymakersandpolicymakingratherthanusingevidenceinpolicymakingonanadhocorone-offbasis
UnderstandingnetworksandsystemsisimportanttodevelopsustainablepolicychangeThissectionismeanttohelpreaderstakethestepstobuildengagementstrategiesforpartnersthatrepresentdifferentsystemsandnetworksEngagementshouldbeanongoingprocessofconnectingpeopletooneanotherandtoanissueEngagementshouldnotbeinnameonly-wehaveallexperiencedengagementactivitiesthatgivepeopleaspecificroletoplaythatleadstoaneatpredictableoutcomeAccordingtotheBuildWithmovementundervaluingandunderinvestinginengagementstrategiescanleadtoadisconnectofcertainindividualsorgroupsandmistrustofgovernmentWhendevelopingengagementactivitiesrememberthathowpeopleengageandwhethertheyengagehastodowithhowyoustructureopportunitiesforengagement
Conveneacoreadvisorygroup
AnadvisorygroupcomprisedofpeoplefrommultiplesectorscanbehelpfulasyouconsiderthelocalsocialenvironmentalandpoliticalcontextinwhichyouintendtointroducepolicychangeRevisityourstakeholdermapanddeterminewhowouldbebesttoserveonthisgroupTomakesureyouareconsideringtheproblemsandsolutionsthroughanequitylensincludepeoplewithlivedexperiencewhohaveexpertiseintheeffectivenessofsystemswhetherthosesystemsfunctionasintendedandhowtheymayimpacthumanbehaviorYoumayalsowishtoincludesomeonewithexperienceworkingwithdataandanalysis-thispersoncanhelpdeterminethetypeofdatathatmaybereadilyavailableandusefultocollectandcanalsohelpplanandshapeyouranalysistofigureoutwhichprogramsareeffectiveBesuretoinvitefront-lineworkersaswell-itscriticalthatwhateversolutionscomeoutofthisprocessaremeaningfultothosewhowillberesponsibleforimplementingthem
BringingpeopletogethercanbechallengingbutthereareafewwaystosetandmanageexpectationsFirstmakesurethatthegroupisawareoftheproblemstatementcurrentstateandfuturestate-thiscanhelpstarteveryoneoffonthesamepageNextspend
Strategicallyengagekeypartners
16
sometimegettingtoknoweachotherrsquosworktolearnabouteveryonesstrengthsThissharedunderstandingcanhelpasyouclarifyrolesandresponsibilitiesofeachpartnerinthegroup--thisisimportanttoavoidfinger-pointinginthefutureandtoensurethatnoneofthetasksslipthroughthecracksInsomecasestherearemanyactorswithdifferentresponsibilitiesasinthecaseofBaltimoreCitys38-memberYouthFundTaskForcewhichischairedbynonprofitandcorporatepartnersandincludesrepresentativesfromthenonprofitandfaithcommunitiesstudentsthebusinesssectorlocaluniversitiesCityCouncilandtheMayorsOffice
OncerolesandresponsibilitiesareclearyoumayconsiderdevelopingacharterguidelinesortermsofreferencetosetexpectationsforthegroupBesuretodistinguishthisfromotherboardsandcommissionsthatyourcitymayhaveNextyoumaywishtoestablisharegularongoingmeetingscheduleBesuretosendagendasandmeetingmaterialsaheadoftimeandclarifytheobjectiveofeachmeeting
InSeattletheCitydevelopeditsHousingAffordabilityandLivabilityAgenda(HALA)torespondtorisinghousingcostsandhomelessnessThisisatrendinotherwestcoastcitiesaswellwhichmaybeexplainedbyaninfluxofnewresidentswhoareattractedbytheemergenceoftechjobstheattractionoftheoutdoorlifestyleortherecentpassageofprogressivepolicieslikelegalizedgaymarriagea$15minimumwageandlegalizationofmarijuanaSeattlebroughttogetheramulti-stakeholderadvisorygroupcomprisedoflocalhousingexpertsrentersandhomeownersdevelopersandgovernmentofficialstoshareinformationabouttheirneedsanddeveloprecommendationsbasedontheseneedsTheinterestsandmotivationsofthisgroupwerevariedandittookabouttenmonthstofindcommongroundTheseriesofrecommendationsthattheadvisorygroupcreatedismeanttoserveresidentswithaspectrumofneeds
Buildaprocessforcommunityparticipation
Ofcoursenoteveryonecanbeincludedinanadvisorygroupandadvisorygroupmembersmaynotnecessarilyrepresentthevariedperspectivesoftheirpeers-forinstancenoteveryhomeownerhasthesamesetofneedsTogatherinputfromawiderswathofthepopulationitcanbehelpfultobuildaprocesstoinvitethepublicatlargetosharetheirideasCitiesdothisinavarietyofwaysfromonlineconversationsandsurveystoin-personmeetingsRememberthatbeingintentionalaboutgatheringtheviewpointsofpeopleismoreimportantthanusingthelatesttechnologyandtools
TheinsightgeneratedbycommunityparticipationexercisescanbeincludedinyourreviewofevidencewhichisdiscussedinasubsequentchapterInWestVirginiaastatethathasbeenhithardbytheopioidepidemicstatehealthofficialsdraftedanopioidresponseplanthatwasinformedbypublicengagementaswellaslocalandregionalsubjectmatter
Strategicallyengagekeypartners
17
expertsThepublicwasinvitedtopresentcommentsonideasfordecreasingoverdoseearlyinterventiontreatmentandrecoveryPublichealthandmedicalfacultyfromWestVirginiaUniversityMarshallUniversityandJohnsHopkinsUniversityprovidedresponsestothecommentsTheresultwasanOpioidResponsePlanthathealthcommissionerRahulGuptapresentedtoGovernorJimJustice
AsamplespectrumtoplanyourowncommunityparticipationprocesscreatedbyLaurenellenMcCannisbelow
SourceLaurenellenMcCann
Community-basedparticipatoryresearchisanotherwaytoworkwiththecommunitytoimproveoutcomesCBPRmeansthecommunityisinvolvedinallaspectsoftheresearchprocesswhichisofteniterativeInCBPRtraditionalresearchersandcommunitymembersshareknowledgeaswellasresourcesandcredit
CBPRInActionInSanFranciscoanenvironmentaljusticenonprofitteamedupwiththeCitysDepartmentofPublicHealthandanoutsideevaluatortounderstandtherootcausesofobesityandfoodinsecurityinlow-incomeneighborhoodsandidentifysolutionsThroughresearchmethodssuchasinterviewingresidentsandstoreownersdiagrammingstoreshelvesGISmappinganeconomicanalysisandreviewofsimilarpoliciesinnearbycitiestheteamobservedthatthestoresthatresidentshasveaccesstowerestockedwithlesshealthyfoodsandthatresidentswantedmorehealthyfoodandlessalcoholandtobaccoThisprojectresultedinlocalandstatelegislationtoencouragestorestostockfreshproduce
Strategicallyengagekeypartners
18
Developrelationshipswithsubjectmatterexperts
OnewaytokeepsubjectmatterexpertsengagedistoinvolvetheminthecommunityparticipationprocessasdescribedaboveIntheWestVirginiaexamplesubjectmatterexpertsrespondedtoandsynthesizedthepubliccommentsOtheropportunitiesaredescribedbelow
AdvisoryAcitydepartmentheadMayororstaffmembermaydevelopaone-on-oneadvisoryrelationshipwithuniversityfacultymemberoracollaborativebetweentheCityandUniversitytoaddressaparticularproblemAgreatplacetostartistolookforrelevantpracticecenterswithinyourlocaluniversityandreachingouttosubjectmatterexpertsInadditiontotheirexpertiseontherelevantpolicytopicheorshemayalsobeabletoidentifyotherexpertsandevenprovidesupportonhiringtherighttalenttoleadcertainprogramsAnadvisoryrelationshipwithauniversitysubjectmatterexpertcanalsohelpwhencitiesareconsideringbringinginnewprogramsTheseexpertsareoftenfamiliarwithprogramsthatarebasedonnationalmodelsandcanprovideinsightintoaprogramsoriginandlineageeffectivenessinotherplacesandthepotentialforimplementationinyourcityAcademicscanalsohelpidentifyindicatorsbasedongovernmentindicatorsorliteratureFinallyrelationshipsbetweenpolicymakersandresearchersprovidethemostbroadlyeffectiveroutetopullingresearchintopolicy
ResearchandevaluationAcademicexpertsmaysupportcitiesinevaluation-indeedevaluatingcityinitiativesisatraditionalmodelofcityuniversityrelationshipsSharingtheinformationyoucollectedintheGatherInformationphasecanhelpresearchersunderstandyourworkandgoalswhichinturnwillstrengthenthesupportthattheycanprovideDiscussingdetailsaboutyourinterventionandevaluationplanatthebeginningoftheprocessmeansthataresearchpartnercanhelpwiththeevaluationdesignanddatacollectionForinstancetheWilsonSheehanLabforEconomicOpportunitieswalkscitiesthroughconsideringthefollowingquestionsbeforepartneringonanevaluation
1 Isthereaclearpolicyprogrammodelwithobjectivescreeningcriteria
2 Isitpossibletoevaluatewhathappensbothwiththepolicyprogramandintheabsenceofthepolicyprogram
3 Arethereoneormoremeasurableoutcomesofinterest
4 Arethereenoughpeopleforrigorousquantitativeanalysis
5 Whatdatawillbenecessaryforthestudyandwhataretheprocessesforaccessingthesedatasource(s)
6 Isthereacompellingreasontoevaluate
Strategicallyengagekeypartners
19
InadditiontoassessingreadinesstoconductanevaluationresearcherscanhelpyouhoneinonthepurposeoftheevaluationTheWorldBankdescribesseveralpotentialusesofevaluationwhicharedefinedhereInbriefevaluationcanhelpdecisionmakers
makeresourceallocationdecisionsrethinkrootcausesofproblemsidentifynewproblemsdecideonabestalternativedeterminewhetherprogramsorpoliciesarebeingimplementedwithfidelitydeterminewhetherprogramsorpoliciesarehavingthedesiredimpact
Researcherscanalsoprovidesupportinsharingtheresultswiththescientificcommunitythroughpapersandconferenceswhichisvaluableincreatingapeernetworkofevidence-basedcitypractitioners
TrainingandtechnicalassistanceUniversitypartnersespeciallythoseatpracticecenterscanbehelpfulindevelopingorupdatingtrainingcurriculumtrainingthetraineranddeliveringtrainingtocitystaff
Strategicallyengagekeypartners
20
ReviewtheevidenceandcollectdetailsOnceyouhaveathoroughunderstandingoftheproblemdataandactorsyouwillneedtoconductyourownresearchonavailableevidenceandlearnaboutthelandscapeofpolicyoptionsYourgoalwillbetodevelopanaccessiblelistofinformationaboutpoliciesthathavehelpedsolvesimilarproblemsinotherplacesThischapterwillguideyouthroughthestepsofpreparingyourreviewersunderstandinglevelsofevidenceandcollectingthedetailsonevidencethatyouwillneedfordiscussionwithdecisionmakersinyourcity
Selectandprepareyourreviewers
AssignapersonorteamtoreviewrelevantinterventionsthathavedemonstratedsuccessinotherjurisdictionsTheirgoalwillbetoproduceareviewofalloftherelevantprogramspoliciesorinitiativesthatyourteamcouldconsiderimplementingTheywillbeconductingaliteraturereviewsupplementedbyinterviewswithexpertsandpeerswhohaveinformationaboutpotentialprograms
SincethepeopleyouassigntothistaskareprovidingthebaselineofpotentialoptionstoimplementitisimportantthatyouchoosepeopleyouareconfidentwillbethoughtfulandthoroughintheirworkItisalsocriticalthattheyareawareoftheimportanceofsourcecredibilityEnsureyourreviewershavefamiliarizedthemselveswithrulesforcriticallyanalyzinginformationsourcesbeforetheygetstartedIftheyareincludingnewsarticlesaspartoftheirsearchtheyshouldalsobeawareofmethodsforevaluatingnewscredibility
AskyourreviewerstodevelopaplanandtimelinetocompletethereviewThisstartswithsolidifyingagoverningresearchquestionOnewaytoknowyouarecreatingadetailedresearchquestionistousetheacronymPICOremindingyoutoincludereferencetothepopulationintervention(ifyouhaveoneinmind)comparisonandoutcomeelementsinyourresearchquestionForexampleifyouknowthatyouhaveapopulationofteenagerswhoarehomelessyouhavelearnedfromyourstakeholdersthatmanyoftheseteenagersidentifyasLGBTandyouhaveheardsomethingabouttheapproachofcreatingdrop-incentersfordeliveringsupportyoucouldask
ForLGBTyouthfacinghomelessness(population)howeffectivearedrop-incenters(intervention)relativetostandardsocialservices(comparison)atsettlingtheminpermanenthousing(outcome)
Ifyoudonothaveaninterventioninmindyoucanleaveouttheinterventionandcomparisonandsimplyask
Reviewtheevidenceandcollectdetails
21
HowcanwehelpLGBTyouthfacinghomelessness(population)obtainpermanenthousing(outcome)
OnceyourreviewershavetheirresearchquestionsusethequestionstocreatealistofkeywordstheycanuseintheirliteraturesearchThiswillhelpkeeptheircollectionofarticlesreasonablyfocusedHavethemdescribewhattheirsearchprocesswillbeincludinglistingthedatabasestheywillreviewandtheirplanforstoringtheinformationtheywillcollect(Herearesomeadditionalsuggestionsforstructuringareviewplan)
WhatyourreviewerswillbedoingisalightweightversionofarapidreviewTotheextentthattheywouldliketodevelopamethodologyforspecificsearchprotocolsincludingcriticalappraisalprocessestheycancertainlygoaheadanddoso
Usedifferentlevelsofevidenceforthemostcompletepicture
AsdiscussedintheWorkingwitharangeofevidencechaptertheNestaStandardsofEvidenceframework--reproducedbelow--ishelpfultounderstandthelevelofconfidencethataprogramorinterventionishavingapositiveimpactThisapproachletsyouputinterventionsthatyouhaveheardaboutlocallybutwhichhavenotbeenstudiedtogetherwiththosewithademonstratedcausaleffectThisishelpfulsinceanumberofpoliciesprogramsandpracticesthatyouwillwanttoconsidermaynotyethavemuchofaresearchrecord
Reviewtheevidenceandcollectdetails
22
NearlyanypieceofinformationaboutaninterventioncanbeclassifiedwithinthestandardsofevidenceForexampleifsomeonehasacrediblehypothesisabouthowaninterventioncouldproducebenefitthistypeofevidencewouldbeclassifiedasLevel1ThistypeofevidencecouldincludeinterventionsinothercitiesthatyoureadaboutinthenewsorhearaboutataconferenceForinstancewhenUSDAallowedschoolsandschooldistrictstoprovidefreebreakfastandlunchtoallstudentsthroughtheCommunityEligibilityProvision(CEP)manydistrictsimplementedthisoptionevenbeforetherewasevidenceofeffectivenessThiswouldfitintoLevel1sinceitbuildsonthelogicthatprovidingaccesstomealswilldecreasefoodinsecurity
YoumayalsofindpiecesofinformationthatqualifyasLevel2evidenceForinstanceasmoreschooldistrictsimplementedCEPprincipalstestifiedaboutpositivechangestheyobservedinstudentssuchasamorepositivedemeanorthatcanleadtobetteracademicperformanceDataalsoshowedthatthestudentsdescribedwerereceivingmoremealsthanbeforetheprogramsimplementationHoweveritwastoosoontobuildevidencetoshowacausalrelationshipbetweentheincreaseinfreemealsandincreasedacademicperformance
ldquoLower-levelrdquoevidencecanconsistofnuancedbutvaluableelementsthatrelatetoaninterventionrsquospotentialfitwithacommunityQualitativeinformationsuchastestimonyfrompeoplewhohaveexperiencedaninterventionandthosewhohavefacilitatedorimplementeditwillalwaysbeusefultoconsiderandprovidesdetailedlocalcontext
Experimentalstudies--describedstartingatLevel3withcontrolledexperimentsandgoingthroughLevel5withfullyfleshedoutprotocolsandprocessesforreplication--stilloccupythehigherlevelsofthisdescriptionofevidenceThisprioritizationshowsthevalueofrigorousstudyforevaluatingtheimpactofaparticularpolicyorprogramratherthanotherfactorsthatcouldbeinfluencingoutcomesofinterestRememberthatmultipletypesofevidencecancomplementrigorousimpactevaluationsandgivecityleadersadditionalinsightsandbroaderperspective
Accessinformation
Encourageyourreviewerstobegintheirsearchforpromisingpracticesingovernmentandnon-profitclearinghousesandinacademicpolicyjournalsWorkpublishedinthesesourceshasalreadybeensubjectedtoahighlevelofprofessionalscrutinyandgenerallycontainsagreatdealofdetailwhichmeansthatyourreviewersaremorelikelytobeabletopullouttheinformationyouneedtomakeagooddecision
Clearinghouses
Reviewtheevidenceandcollectdetails
23
Herearesomeoftheclearinghouses(collectionsofprogramevaluations)forevidence-basedpoliciesandprograms
ResultsFirstClearinghouseDatabaseThePewCharitableTrustsCollectstogethereightotherclearinghousestoprovideinformationoninterventionsinthefollowingcategoriesadultcriminaljusticeeducationchildwelfarehealthjuvenilejusticementalhealthsubstanceabuseandyouthdevelopmentIndividualclearinghousesincludedinResultsFirstare
BlueprintsforHealthyYouthDevelopment-foryouthdevelopmentandviolenceprevention
CaliforniaEvidence-BasedClearinghouse-forchildwelfareinterventions
TheLauraandJohnArnoldFoundationrsquosSocialProgramsThatWork-forsocialservicesprograms
TheUSDepartmentofEducationrsquosWhatWorksClearinghouse-foreducationinterventions
TheUSDepartmentofHealthandHumanServicesrsquoResearch-TestedInterventionPrograms(RTIPs)-forcancercontrolinterventions
TheUSDepartmentofHealthandHumanServicesrsquoTeenPregnancyPreventionEvidenceReview(TPPEvidenceReview)-forsexualriskbehaviorandsexualhealthoutcomes
TheUSDepartmentofJusticersquosCrimeSolutionsgov-forcriminaljusticejuvenilejusticeandcrimevictimservices
TheUSSubstanceAbuseandMentalHealthServicesAdministrationrsquosNationalRegistryofEvidence-basedProgramsandPractices(NREPP)-substanceuseandmentalhealthinterventions
TheUniversityofWisconsinPopulationHealthInstituteandRobertWoodJohnsonFoundationrsquosCountyHealthRankingsandRoadmapsWhatWorksforHealth(WhatWorksforHealth)-healthbehaviorsclinicalcaresocialandeconomicfactorsandthephysicalenvironment
UKWhatWorksCentresTheBritishgovernmenthascharteredanetworkofsubject-specificorganizationswhichcollectsynthesizeandreportoutevidenceontheeffectivenessofpolicyprogramsandpractices
NationalInstituteforHealthandCareExcellence(NICE)-healthandsocialcare
Reviewtheevidenceandcollectdetails
24
SuttonTrustEducationalEndowmentFoundation-educationalachievement
CollegeofPolicingWhatWorksCentreforCrimeReduction-crimereduction
EarlyInterventionFoundation-earlyintervention
WhatWorksCentreforLocalEconomicGrowth-localeconomicgrowth
CentreforAgeingBetter-qualityoflifeforolderpeople
WhatWorksCentreforWellbeing-wellbeing
WilsonSheehanLabforEconomicOpportunities(LEO)UniversityofNotreDameProvideslinkstotheirevaluationsofinterventionsincriminaljusticeself-sufficiencyeducationhealthandhousingandhomelessness
ClearinghouseforLaborEvaluationandResearchUSDepartmentofLaborProvidesresearchonworkforceandlabor-relatedpoliciesandprograms
J-PALEvaluationsAbdulLatifJameelPovertyActionLabStudiesonpoverty-reducinginterventionsfromaroundtheworld
CampbellSystematicReviewsCampbellCollaborationSystematicreviewofpolicyinterventionsbyscholarlycoordinatinggroupsinsevenissueareascrimeandjusticeeducationinternationaldevelopmentmethodssocialwelfareknowledgetranslationandimplementationanddisability
Benefit-CostResultsWashingtonStateInstituteforPublicPolicyAWashingtonState-charteredprogramtoestablishthelikelihoodofrelativecostsandbenefitsforprogramsacrossthefollowingareasjuvenilejusticeadultcriminaljusticechildwelfarepre-Kto12educationchildrensmentalhealthhealthcaresubstanceusedisordersadultmentalhealthpublichealthandPreventionworkforcedevelopmentandhighereducation
EffectiveInterventionsCentersforDiseaseControlandPreventionRepositoryforevidence-informedHIVpreventioncareinterventions(withstepsonhowtoimplement)
Journalsandindexes
OnceyouhavereviewedrelevantinformationaccessiblethroughclearinghousesitisworthtakingalookatindividualstudiespublishedasscholarlyarticlesYourreviewerswilldothisbysearchingarticleindexesandacademicjournalsUnfortunatelyanumberofusefulacademicjournalsandindexingplatformsmaynotbegenerallyaccessibletoyourreviewersifyoudonothaveasubscriptiontothemToaddressthisconsiderpartneringwithalocal
Reviewtheevidenceandcollectdetails
25
collegeoruniversitytoprovideyouwithalistofarticlesfromthefollowingsources(Thisisalsoanexcellentopportunitytoseekresearchsupportfromanyacademicstakeholdersyouidentifiedpreviously)
Indexes
GoogleScholarincludesmostpeer-reviewedonlineacademicjournalsandbooksconferencepapersthesesanddissertationspreprintsabstractstechnicalreportsandotherscholarlyliteratureincludingcourtopinionsandpatentsItallowsyoutospecifyapublicationrangeinyoursearchandprovideslinkstolikelysourcesforalllistedarticles
PolicyFileIndexisauniqueresourceforUSpublicpolicyresearchinthatitgrantsusersaccesstotimelyupdatedinformationfromover350publicpolicythinktanksnongovernmentalorganizationsresearchinstitutesuniversitycentersadvocacygroupsandotherentities
PolicyArchiveworkstocollectanddisseminatesummariesandfulltextsvideosreportsbriefsandmultimediamaterialofthinktankuniversitygovernmentandfoundation-fundedpolicyresearch
Anothersourceforhigh-qualityresearchisthereportsoftheCongressionalResearchServiceYoucanfindtheirreportsaggregatedhere
Specificjournals
Oftenthemostusefulacademicjournalsforunearthingpolicyorprogramreviewswillbeineachspecificsubjectarea--thatisineducationjournalsforeducationpoliciesYoucanuseageneralindexlikeGoogleScholartobegintolearnwhichjournalsaremostoftenusefulforyoursearchHoweveryoucanalsosearchthefollowingtwogeneralistpolicypublicationsforissuesandarticlesrelatedtoyourtopic
PublicAdministrationReviewJournalofPolicyAnalysisandManagement
Newssourcessubjectmatterexpertsandinterviews
AlthoughitisvaluabletoknowwhathasbeenexploredthroughascholarlyresearchframeworkitisalsousefultolearnwhatkindofinterventionsarebeingtriedinthefieldrightnowandhowpeoplefeelaboutthesetrialsSearchsubscriptionnewsaggregatorslikeLexisNexisorfreenewsaggregatorslikeGoogleNewsorYahooNewsforyourissueareaplusthewordpolicyorprogramtoseewhatkindsofinterventionsarebeingcoveredinthenews
Reviewtheevidenceandcollectdetails
26
DonotforgettocontacttheadvisorygroupmemberswhowerechosenfortheirsubjectmatterexpertiseInadditiontoprovidinggoodmaterialthroughinterviewsthemselvestheymayhaveideasaboutothersubjectmatterexpertsyoushouldcontacttoaskaboutpotentialinterventions
Peercitiesandnetworks
InadditiontoreadingabouttheworkofothercitiesitisidealifyouareabletospeakwithpeoplewhoareinvolvedwithcurrentimplementationsofinterestinginterventionsInyourcityyoulikelyhaverelationshipswithyourneighborsbutitcanbehelpfultolookoutsideofyourregionforsolutionsaswellTofindcounterpartsinothercitieswhoareworkingonaprogramyouwouldliketolearnmoreaboutyoucancontactstaffinintermediaryorganizations(likeGovExoryourprofessionalassociations)whomightbeinapositiontoputyouintouchimmediatelyorwhomightseeanothergoodrouteforsettingupaconversationHereisalistofnetworksthatmaybehelpful
InternationalCityCountyManagementAssociation(ICMA)NationalAssociationofCounties(NACO)NationalLeagueofCitiesUSConferenceofMayorsWhatWorksCities
Collectthedetails
Onceyourreviewershavefoundevidenceonpossiblepoliciesorprogramstoimplementtheyshouldcapturekeydetailsthatwillhelpdecisionmakersunderstandthepotentialeffectivenessandapplicabilityofeachpotentialintervention
Foreachsourcethathasprovidedinformationyouwanttoconveyusethetemplatebelowtocaptureasmanyofthefollowingdetailsaspossible(Youdonotneedtocompletethisforsourcesthatprovideduplicativeinformationorareotherwisenotthatuseful)
Programorpolicyname (egldquoHousingFirstrdquoorldquoblightedpropertiesregistryrdquo)
Sourcetype (egjournalarticleinterviewsubjectmatterexpertstudyassociationcommunityparticipationprocess)
Sourcedetails (egarticletitleandpublicationnameandtitleofinterviewee)
Evaluationmethodology
(egrandomizedevaluationcomparativeevaluationoftwodifferentgroupsbeforeafteranalysis)
Evidencelevel (egLevel12345)
Programorpolicy
Reviewtheevidenceandcollectdetails
27
description
Programorpolicygoal (1sentencecapturingtheprincipalaimoftheintervention)
Existingimplementations
(WherehastheinterventionbeentriedListasmanysitesasyouareawareof)
Characteristicsofimplementationsites
(ForeachimplementationsiteofferbasicgeographicIDmdashcityandstateThenyouwillwanttocollectthedetailswhicharerelevanttoyourdecisionmakersrsquoconsiderationsaboutwhatmakesalocationldquosimilartordquoorldquodifferentfromrdquoyoursituationYouwillatleastwanttoincludeforeachimplementationsitePopulationnumberPopulationdensity(ruralurbansuburban)Socio-economiccharacteristicsBeforestartingaskyourdecisionmakersaboutanyparticularcharacteristicsofprogramimplementationsitestheywouldwanttoknowbeforeseekingtoreplicate)
Characteristicsofimplementationpartners
(Anyavailabledetailsaboutorganizationsinstitutionsorpeoplewhowereinvolvedwithimplementingtheintervention-egindependentchildcarecenterpublicelementaryschoolcitypublichealthnurses)
Characteristicsoftargetpopulationeligibleprogramparticipants
(Anyavailabledetailsaboutpeoplewhoarethebeneficiariesoftheprogramorotherwiseincludedoraffectedbytheintervention-egnumberofprogramparticipantscategoriesofpropertyownersaffected)
Policyorprogramduration
(HowlonghastheinterventionbeeninplaceineachimplementationsiteHasitendedandifsowhy)
Whatbenefitscamefromeachimplementation
(Anyobservedpositiveeffectsoftheinterventionincludingeffectsize)
Statisticalestimatesofinterventioneffectiveness
(Anyavailablestatisticalestimatesoftheinterventionrsquoseffects)
Whatdrawbackswereobserved
(Anyavailabledetailsaboutunintendedorunavoidablenegativeeffectsoftheintervention)
Programorpolicycosts
(Anyavailabledetailsaboutprogramcostsperimplementation)
Reviewtheevidenceandcollectdetails
28
Reviewtheevidenceandcollectdetails
29
DecideonapolicytoimplementbasedonlocalfactorsTogetthecriticalbuy-infromleadershiptoimplementanewpolicyorprogramtheprojectleadshouldscheduleameetingwithdecisionmakerstopresenttheevidencesurroundingthepolicyoptionscapturedinthetemplateDependingonyourinsightintoyourcityleadershipswillingnesstomakechangeyoumaywishtosharetheresearchyouhaveconductedontherootcausesoftheproblemandsomeofyouranalysisontheimpactofcurrenteffortstodate
Considertheapplicabilityofanapproach
AsdiscussedinthepreviouschapterpolicymakersoftenlookforevidenceofeffectiveprogramsinothercitiesbutoftenthequestionarisesofhowgeneralizableaprogramisfromonecitytoanotherTheAbdulLatifJameelPovertyActionLab(J-PAL)suggeststhatinreviewingevidencefromanotherlocationitishelpfultoidentifythespecificmechanismthatwasused(deliveryofprogramorserviceinformationsharingetc)identifythelocalconditionsthatwereinplaceobservethebehaviorchangethatresultedfromthemechanismandfinallytounderstandlocalcontextandconditions(legalorcostconstraintsculturalnormsetc)Thissetofpresentationelementscanbedrawnfromthelistofdetailscollectedintheresearchtemplate
EducationpolicymakersinChicagoILusedfindingsfromprogramsinKenyaandIndiainwhichchildrenwhohadfallenbehindtheirpeersweregiveneducationalsupporttargetedtotheirleveloflearningThemechanismusedwastargetedteachingandthereweresimilarconditionspresentinallthreecountriesAlthoughsomestudentshadfallenbehindinthecurriculumteacherswereincentivizedtoteachgrade-levelmaterialsConsiderationsofhowthelocalcontextinChicagomightaffectimplementationofatargetedteachingprogramincludedunderstandinghowtheneighborhoodsthatthestudentswerefrommayhaveaffectedtheireducationaloutcomesandthehighcostofteachinginsmallgroupsInChicagoMatchEducationprovidedtwo-on-onetutoringwhichmatchedstudentswithwell-educatedindividualswhohavenothadformalteachertrainingforalower-costmodelTheresultswerepositiveStudentswhoweretutoredperformedbetterinmathandweremorelikelytopasshighschoolclasses
ThegeneralizabilityframeworkproposedbyJ-PALisbelowUsethisasatemplateasyouconsiderpotentialpolicysolutions
Decideonapolicytoimplementbasedonlocalfactors
30
SourceTheGeneralizabilityPuzzle
OnceyouhavenarrowedthepolicyoptionsdiscussthequestionsbelowforeachoptionsyouareconsideringThisstepiscriticalasunderstandinglocalcontextissoimportantinsuccessfulpolicyimplementation
CommunityContextQuestionnaire
Localcontext
DoesthisoptionmakesenseinourcommunityIsthisaonesizefitsallapproachIsanyparticulargroupabsentfromthisapproachIfsoisthatok
Youranswers
Effectiveness
WhatwouldweexpecttoobserveifthepilotisaseffectivehereasinCityX
Youranswers
Timing
DoesthetimingmakesenseIsthistherighttimetoimplementthispolicychange
Decideonapolicytoimplementbasedonlocalfactors
31
WherearethepotentialroadblocksthatcouldaffectthescheduleIsthereanywaytomitigatethisHowlongwouldittaketoseeanimpact
Youranswers
Implementation
DowehavethestaffcapacitytoimplementthisArethereadditionalpartnersthatyoucanbringtothetabletosupportimplementationDoesthismakesensegivenourformofgovernment
Youranswers
Authority
DoesourcityhavetheauthoritytoenactthisIfsowhatistheappropriatemechanismAretherelegalbarriersArethereanyexistingpreemptionissuesfromthestateorfederalgovernmentCanthosebeaddressedthroughlegislationadvocacyIstherepoliticalsupportoroppositiontothis
Youranswers
Cost
HowmuchwillthiscostannuallyHowwillwepayforitCanthepolicybeimplementedwithexistingfunding
IfsoisthisfundingreliableHowwouldthispolicyoptionaffectourcitysbudgetintheshortandlongtermWhatotherfundingoptionscanweconsiderIsthereanalternativethatwouldcostlessandgiveussimilarresultsWillanyconstituencybeadverselyaffectediffundingforprogramsthatimpactthemisshiftedtoanotherprogrampolicyorinitiative
Youranswers
Decideonapolicytoimplementbasedonlocalfactors
32
EstablishaframeworktomonitorprogressNowthatyouhavereviewedtheevidenceconsideredlocalcontextandchosenyourpolicyitistimetodeterminehowyourcitywillmonitorprogressofthisnewapproachCitystaffandleadersneedtoestablishtheinfrastructureforongoingcontinuousdecisionmakingbasedonwhetherthepolicychangeisleadingtothedesiredoutcomesforresidentsandwhetherthechangeismeaningfulandrationaltofrontlineworkersSettingarbitrarygoalsatthelocallevelthatconflictwithgoalsatotherlevelsisconfusingtotheindividualsresponsibleforachievingthegoalsGoalsthataresettoohightoachievecanbedemoralizingtostaff
ThissectionincludesstepstosetupavenueinwhichCitystaffandleadershiproutinelyvisitkeyindicatorstodeterminewhatisorisnotworkingandtomakedecisionsinreal-timeFormorereadingonperformancemanagementpleasevisittheGovExPerformanceManagementGettingStartedGuide
InperformancemanagementtermssuchasindicatormeasureandmetricmaybeusedinterchangeablyInthisguidewewilluseindicatorInyourworkthemostimportantthingistobeconsistentinwhichevertermyouuseThisvideomaybehelpfulinunderstandingthisconceptandcommunicatingittoothers
Establishthefundamentalsandgatheryourteam
FirstdeterminewhetherthereisanestablishedvenueorregularmeetingscheduleduringwhichyoucandiscussprogressIfyourcityhasanexistingperformancemanagementstructureonarelatedtopicaddingadiscussionoftheprogressofpolicychangetoyouragendacanbeagreatideaIfthiswillbeastandalonemeetingdecidehowfrequentlytheprojectteamanddecisionmakerswillmeetandputthosemeetingsonthecalendarfortheentirelengthoftheproject
DeterminewhowillstaffandattendroutinemeetingsonCityprogressThesemeetingsmayincludetheMayorandorCityManagerdepartmentheadstaffwhoareimplementingthepolicychangeandanalysts
Identifyindicators
ItiscriticalthatindicatorsbeanimportantcityandcommunitypriorityaswellassomethingtheCityplaysamajorroleinaddressingTheremaybeinstancesinwhichtheCitywillneedtoreachouttoorcollaboratewithexternalpartnerstogetamorecompletepictureofprogressontheoverallpolicychange
Establishaframeworktomonitorprogress
33
Anoutcomeindicatordescribesthedesiredeffectofthetargetedpopulation
ExampleBellevuesfamilyfriendlyneighborhoodsindicatormeasuresthepercentofresidentswhobelievetheyliveinaneighborhoodthatsupportsfamiliesparticularlythosewithchildren
WithyourkeypartnersrevisityourproblemstatementcurrentstateandfuturestateexerciseandbegintoidentifytheoutcomeindicatorsthatwillillustratesuccessFornowchooseasingletop-levelkeyperformanceindicator(KPI)ItiscriticalthattheKPI-thepieceyouwillmeasure-beapriorityforthecommunityandsomethingtheCityplaysamajorroleinaddressing
Outputindicatorsmeasuretheprocessofprogramorservicedelivery
ExampleSanDiegoMayorKevinFaulconersgoaltorepair1000milesofstreetby2020
ReviewingtheadministrativedatasetinventorybegintoparedowntherelevantprogramsanddatasetsthatsupportyourKPIWhatarethemeasuresofeffectivenessforeachprogramTheseareyouroutputindicators
Thediagrambelowillustratestherelationshipbetweenoutputandoutcomeindicators
ThinkaboutimplementationindicatorsasmeasuringstepsintheprogramsrolloutthatarenecessarytoobtaindesiredchangesImplementationoutcomescanfocusontheinterventionstaffdeliveringtheinterventionanorganizationorsystemandcanaddressfeasibilityadaptabilitysustainabilityandcostTwokeyquestionstoconsiderwhendevelopingimplementationindicatorsare
1 Istheinterventionacceptableandappropriatetothecommunityand
2 IsitfeasibletoimplementinterventionXatthistimewiththeseresources
Establishaframeworktomonitorprogress
34
Amoreextensivelistwithdefinitionsofpotentialimplementationoutcomesislinkedhere
UsethetablebelowtochoosethedesiredimplementationoutcomesforyourpilotandtoplanyourmeasurementtechniqueThisshouldincludelevelofanalysis(individualproviderorconsumerorganizationorsetting)yourmeasurementtechnique(surveysobservationinterviewfocusgroupsadministrativedataanalysis)andthephasewhenyouwillcollectthisinformation(earlymidlate)Theremaybemorethanonelevelofanalysismeasurementtechniqueandphaseofdatacollection
Implementationoutcome
Definitionofoutcome
Levelofanalysisandmeasurement
Yourmeasurementtechnique
Phaseofpilotwhenyouwillcollectdata
Usethespacebelowtocaptureyourindicatorwishlist
Indicatorwishlist
Outcomes_____________________________________________________________________________
Outputs________________________________________________________________________________
Implementationoutcomes______________________________________________________________
Setapublichigh-levelgoal
Thehigh-levelgoaliscriticalinestablishingdirectiontogalvanizethepeoplewhowillbedoingtheworkWhendiscussingthegoalusetheMayorsbullypulpitLookforkeymilestonesinCitygovernmenttosharethegoalstatementsuchasthebeginningofaMayoraladministrationaStateoftheCityspeechorotherpublicremarks
GoalsshouldbeldquoSMARTrdquo
Specificaspossible(innamingthepopulationwhowillbebetterofftheactionthecitywilltake)
Measurable
Achievable
Establishaframeworktomonitorprogress
35
Results-focused
Time-bound
ThegoalstatementshouldincludeyouroutcomeindicatorlikeChattanoogasgoalldquoWewillendchronicveteranhomelessnessbyDecember2016rdquoAdiagramtohepyoubuildaSMARTgoalaroundyouroutcomeindicatorisbelow
Component Description Yourturn
Specific WhowillbebetteroffWhatactionthecitywilltake
Measurable Howwillyoumeasuresuccess
Achievable Isthegoalattainable
Results-focused Whatistheexpectedresult
Time-bound When
Setrealisticambitioustargetsandtimelinesforeachindicator
ToestablisharealisticandambitioustargetforyourgoalstatementusetheadministrativedatacollectedforyourchosenKPItoreviewactualCityperformanceinthemostrecentyearcomparedwiththethreeorfourmostrecentyearsworthofdataBegintocalculatesummarystatistics(egtotalminimumaveragemedianmaximum)toidentifytrendsCreatecumulativeandmonthlychartstoidentifytrendsandanomaliesBasedonyouranalysisofhistoricaldataignoretheanomaliesandassumetheproductivitytrendswillfollowthesameoverallpatternfrommonth-to-monthFinallyapplyanincreasedproductivityfactortotheperformanceeachmonthtoforecastfutureprogressBasedonthisanalysischooseatargetthatinspiresimprovementwhileaccountingforreality
Manycitiesthataremorematureintheirperformancemanagementpracticesrecognizetheimportanceoftakingacommunity-levelapproachtoimprovingoutcomesforresidentsThesecitiesarecreatinggoaltaxonomiesthatincludemeasuresrelatedtopublicsafetyhousingandevenrecreationCitiesdriveprogressonthesegoalsbyimplementingstrategiesthatengagemultipledepartmentsandcollaboratewithserviceprovidersinthecommunity
EventuallyyouwilldevelopgoalstatementsfortheoutputsinordertocapturetheefforttoachievetheoutcomecontainedinyourgoalstatementYoursetofindicatorsshouldbecomposedofshort-andinterim-termsupportivemeasureswithaccuratetargetsandrelevanttimelinesAnexamplegoaltaxonomyfromtheCityofLasVegassResultsVegasprogramisbelowUsethistaxonomyasamodelforyourownsuiteofindicatorsandgoalstatements
Establishaframeworktomonitorprogress
36
Yourturn
Thetemplatebelowismeanttohelpyoucraftagoalstatementforeachsupportingmeasure
Goalstatement
Departments
Outcome
Activities
KPI
Target
Timeframe
Conductanddiscussrelevantanalysis
ThrougharoutineseriesofmeetingscontinuallymonitorprogressofKPIssupportingindicatorsandanyotherdatacollectedsuchasqualitativedatatoidentifytrendsthatwillindicatewhetherthechosenpolicyishavingthedesiredimpactThesedatapointsshoulddrivethediscussionduringthesemeetingspointingtoopportunitiesforfurtherinquiryinto
Establishaframeworktomonitorprogress
37
whatisdrivingtheobservedtrendsAnalystsshouldbeencouragedandtrainedtoaskquestionsofthedepartmentsproducingthedataandconductsitevisitswhenpossibletodeepentheirunderstandingofthedataPresentationoftheanalysisshouldfocusononeideaatatimetokeeptheconversationfocusedonkeypointsandspecificopportunitiesforimprovementYoumayconsiderdevelopingapublicfacingdashboardtosharethisprogressmorebroadly
AgeneralcomparisonofthestatusquodatawithwhatwasobservedduringthepilotcanprovidedecisionmakerswithaquickunderstandingofthedifferencesbetweentheoldandnewapproachesalthoughthiswillnotshowcausalityOftenthesecomparisonswillincludeoperationalmetricssuchasdaysordollarssavedbutrememberthattheimpactonpeoplewillbemostimportantIfyouareworkingwithanevaluationpartnertheywillsupportthisstep
Usethespacebelowtosummarizetheresultsofyouranalysis
MeasureTargetvalue(seesectiononsettingtargetsabove)
Pilotvalue(whatdidyouobserveduringthepilot)
Historicalaverage(ifavailable)
Results(WhoisbetteroffIsanyonenegativelyimpacted1-2sentencesreassessingyourgoalstatement)
DocumentyourprocessandprogressincludingfeedbackfromtheparticipantsimplementationpartnersandtheprojectteamDidanythingunexpectedoccurduringthepilotArethereinternalandexternalfactorsthatmayaffectyouroutcomesThisiswheretheimplementationoutcomescanbecanhelpfulIftheinterventiondidnotgoasplannedconsiderwhetheritisbecausetheinterventionwasnotacceptableorfeasible
UsethetemplatebelowcreatedbyDenversPeakAcademytocapturethisinformationIfthe306090daytimelinedoesnotmakesenseforyourcityfeelfreetocustomizeit
CurrentState
FutureState
30Day
60Day
90day
Whatwentwell(+)
Whatdidnotgowellshouldbechanged(-)
IfaquestionordiscussionpointarisesthatcannotbeansweredinthemeetinganassignedstaffmembershouldcapturethequestionduringthemeetingAfterwardthefollowupitemsshouldbesenttorelevantpartnersandthenextmeetingshouldbeginwithadiscussionof
Establishaframeworktomonitorprogress
38
thoseitems
Takeaction
DiscussionsaboutwhetheryourindicatorsaremovingintherightdirectionandwhatmightbedrivingtrendsmayleadtoadiscoverythattherearenotenoughresourcesallocatedtothispolicyorissueTheremaybelimitedfundingforstaffprogrammingortechnologythatisdrivingthedatainthewrongdirectionIfCityleadershipisintheroomthisisagreattimetomakethecaseforadditionalfundingduringthebudgetseasonBycollectinganalyzinganddiscussingdataonthispolicyroutinelyyouarebuildingevidencetomakethecasefortherightleveloffundingtoreachtheestablishedtargets
Establishaframeworktomonitorprogress
39
Assessdataaccessibilityqualityandsecurity
Determineaccessibilityandqualityofdata
OnceyouhavecompiledyourwishlistofindicatorsitistimetodeterminetheaccessibilityandqualityofthedataTopreventcomplicationyoumaywishtoprioritizerelevantindicatorsforwhichyouarealreadycollectingdataAvoidthetemptationofadoptingatechnologysolutiontocollectnewdataatthispointsuchasaphotosurveytoolorotherappFornowyouaretryingtodeterminewhichdatasetsarerelevantaccessibleandhighqualityBesuretoworkwithyourkeyinternalpartnerswhileyouareidentifyingyourmeasuresDepartmentstaffwhoworkonprogramsarefamiliarwiththeindicatorsthatwillaccuratelysupporttheoutcomesanddatabaseadministratorswillbevaluablewhendiscussingaccessibilityandqualityofthedata
Abrieflistofconsiderationsfortheaccessibilityofyourdataisbelow
ArewecurrentlycollectingthisdatasetorwillwebecollectingitforthefirsttimeHowfarbackdoestherecordgo(timehorizon)WouldweneednewtechnologytocollectoranalyzethedataCanthedatabecollectedmanuallyorcanitbeautomatedHowfrequentlyisthedataproduced(continuouslyhourlydailyweeklymonthlyquarterlyyearlyetc)IsthedataavailablefortheentireCitypopulationorasubsetofthepopulationHowgranularisthedata(individuallevelCensusblockneighborhoodcitywideetc)DoesthedatacontainsensitiveinformationIsthereaproxymeasurethatwecanusefornowIsthedatamachinereadable
AsnotedbytheWilsonSheehanLabforEconomicOpportunities(LEO)attheUniversityofNotreDameleveragingadministrativedatacansignificantlyreducethecostsandtimelinesofrigorousevaluationCitiescanestablishaninternaldatasharingagreementacrossdepartmentsincludinglanguagethatenablesadministrativedatatobeusedforresearchpurposesCitiescanalsoidentifyexternaladministrativedata-holderssuchasacountyschooldistrictpolicedepartmenthospitaloranotherstatethatmayhaveoutcomedataimportantformeasuringprogramimpactCitiescanleverageexistingrelationshipsandorworkwithresearchinstitutionstodevelopstreamlineddatasharingprocessesandagreementswiththeseexternalentities
Assessdataaccessibilityqualityandsecurity
40
Evaluatethequalityofyourdata
QualityisasimportantasaccessibilityToevaluatedataqualityyoumaywishtocreatearubricliketheonebelow
Value ValueDescription
3 ExcellentDataQualityThedataiscompleteaccurateandupdatedregularly
2 AdequateDataQualityThedataissomewhatcompleteaccurateandupdatedfairlyregularly
1 PoorDataQualityThedataisnotcompleteaccurateorupdatedregularly
ThisisagreattimetobuildonthedatainventorythatyoustartedinthegatherinformationpageSuggestedadditionalfieldstoaddtoyourdatainventorytemplatearebelow
Program DatasetTitle
DatasetDescription DataSource
Departmentorexternalpartner
DoestheCityalreadycollectthisdata
Whatisthenameoftheprogram
Human-readablenameofthedataset
Shouldbeunderstandablebynon-technicalusers
Whatistheoriginalsystemapplicationfilethathousesit
Departmentorexternalpartnerthatmaintainsthedata
Isthisdataalreadycollected
Updatefrequency-ThefileformatforthedatasetTypicallycsvxlsxshapefiletxtetc
Format-Howoftenthedatasetisupdated
Timerange-Theperiodoftimecoveredinthisparticulardataset
Datause-Howisthisdatasetcommonlyused
Datausers-Whotypicallyusesthisdataset
Dataownerstitle-Titleofthedataowner
Dataownersposition-DataOwnersjobtitle
Dataownersemailaddress-DataOwnersemailaddressWhomanagesthedataandorisresponsibleforgrantingpermissiontoaccessthedataWhounderstandswhatthedatasetincludesandcananswerquestionsaboutit
Dataqualityconcerns-Describetheconcernsyoumayhaveaboutthequalityofthedataset
Assessdataaccessibilityqualityandsecurity
41
Division-DivisionwherethedataismaintainedIfmorethanonedivisionisresponsiblelisttheprimaryownerofthedataset
Sensitivedatacomments-DescribetheissuesthatwouldpreventthecityfromsharingthedatapubliclyCommonsensitivityconcernsincludeprivacyviolationssecurityissuesorhighcostorstaffdemandstopostthedata
Publishingstatus-Thistheplacetoputanycommentsregardingthecurrentstatusofthedataset
Publishedlink-Thisisthelinktothepublisheddatasetontheopendataportalorcitywebsite
Reviewdatasecurityandprivacyprotectionprotocols
OftenapolicychangealsoinvolvesachangeinthedatathatisbeingcollectedhowthatdataisbeinghandledwhohasaccesstoitandahostofotheradjustmentsAsyoutakestockofthedataassetsthatareavailableforyourinitiativeitisvitaltoalsothinkthroughwhatwillbedonetoprotectthatdataandtheprivacyofthepeopleitrepresentsAbigpartofgooddatasecurityinvolvesstrongdatagovernancepracticesthatdefineandidentifysensitivedataandconsciouslyworktobalancethevalueandriskofstoringsuchsensitivedataOnewayofevaluatingyourdatacomesfromcaselawrelatedtotheUSFreedomofInformationActwhichwehavetranslatedtothechecklistbelow
1 Woulddisclosureresultinasubstantialinvasionofprivacy2 Haveweconsideredtheextentorvalueofthepublicinterestandthepurposeorobject
oftheindividualsseekingdisclosure3 Istheinformationavailablefromothersources4 Wastheinformationgivenwithanexpectationofconfidentiality5 Isitpossibletomouldreliefsoastolimittheinvasionofindividualprivacy
TheseriskscouldbemitigatedusingavarietyofstrategiesfromaggregatingdatatoobfuscatingindividualstoredactingfieldsthatmayincludePersonallyIdentifiableInformation(PII)orre-evaluatingyourdatacollectionmethodstofindalternativewaysofgettingtheinformationyouneedThesestepsshouldbecodifiedandmanagedbyadatagovernancecommitteeifpossibleForthesensitivedatathatisdeemedessentialcreateaplanaroundpotentialdatabreachesinordertoquicklyandconfidentlyrespondtoemergencysituations
ProactivelypreparingforthesesituationsisoftenthebeststrategyfordatasecuritybutdonotforgetthetechnicalsideInvolveyourITadministratorsinconversationsaboutdatastorageandaccessensuringthattheywillbeprotectedbyenterprise-gradecybersecuritysystems
Assessdataaccessibilityqualityandsecurity
42
Assessdataaccessibilityqualityandsecurity
43
Designimplementationandevaluation
Designtheimplementation
NowthatyouhaveyouraccountabilityplanandkeyindicatorsitistimetogetspecificaboutyourplanforimplementationThetemplatebelowwillhelpyouthinkthroughthespecificmechanismsofimplementingpolicychange
UnderstandingorganizationsandsystemsiscriticaltounderstandingandmakingdecisionsaboutwhereandhowinterventionsshouldbeappliedApolicyimplementedinasinglesettingmayshowpromisingresultsbutimplementingmultiplepoliciesandprogramsatthesametimewithdifferentactorsresponsibleforeachaspectcanhelpensuresustainablebehaviorchange
InSomervilleMAShapeUpSomerville(SUS)isacitywidepublichealthprogramthatisimplementedacrossmultiplelevelsofinfluenceandsystemsthroughinterventionsbeforeduringandafterschooltoincludechangestobuiltenvironmentsuchasbikelanesregionalmasstransportationandphysicaleducationequipmentinschoolsandgymstoincreasephysicalactivitywellnesspoliciesandschoolgardensnutritioneducationduringandafterschoolandhealthyrestaurantprogramstosupporthealthyfoodandhealthyeatinghabits
Usethetemplatebelowtoplanyourimplementation
Policyimplementationplanningtemplate
ListthespecificdesiredoutcomesInthepreviousstepyouworkedwithdepartmentstafftoestablishspecificmeasurableoutcomesandinterimindicatorsPrioritizetheindicatorsthatwilldemonstrateprogressUsethespacebelowtocapturethedesiredoutcomesofthepolicychange
ExampleShapeupSomervillebeganwiththegoalofpreventingobesityin1stthrough3rdgraders
Yourturn
____________________________________________________________________________
____________________________________________________________________________
Designimplementationandevaluation
44
SpecifythetargetpopulationAreyoudesigninganinterventionthattargetsonlycertainsitesAreyouplanningonimplementingaprogramorinitiativeforaspecifictargetaudienceClearlyspecifythepopulationyouplantotarget
IfyouplantoconductarigorousimpactevaluationasnotedbytheWilsonSheehanLabforEconomicOpportunitiesitisimportanttoestablishconcreteobjectiveeligibilitycriteriaandensurethatthesecriteriaareappliedconsistentlybyallstaffduringscreeningandintakeItisalsovitaltoidentifytheindividualswhoareeligibleforaparticularprogramorserviceevenifeveryoneeligiblecannotbeservedbyexistingresourcesThishelpscitiesidentifyanyunmetneedsandguidefutureresourceallocationwhilealsocreatingthepossibilityofarigorousimpactevaluationbyidentifyingapotentialcomparisongroupFinallywhenconductingscreeningandintakeitisimperativetocollectandmaintainbasicinformation(egnamedateofbirthgender)aboutallprogram-eligibleindividualsnotjustthosewhoparticipateintheprogram
ExampleIntheShapeUpSomervilleexampletheprogrambeganbytargeting1stthrough3rdgraders
Yourturn
____________________________________________________________________________
____________________________________________________________________________
DefinethelocationWherewillthepolicychangetakeplaceWillimplementationtakeplaceatonesiteormultiplesites
ExampleShapeUpSomervilleinterventionsweredeliveredinschools
Yourturn
____________________________________________________________________________
____________________________________________________________________________
ClarifyimplementationpartnersandarticulatetheirrolesWhatarethekeyorganizationsinstitutionsorpeopleresponsibleforimplementationTheprojectteamshouldbeasclearandspecificaspossibleinprovidingadescriptionoftheconcreteactivitiesthatimplementationpartnersaremeanttocarryoutPracticeprofilesmaybeahelpfultooltohelpdefinewhatisrequiredoftheactorswhoareresponsibleforimplementingtheintervention
Designimplementationandevaluation
45
ExampleInSomervillemayoralleadershipandengagementofmultiplecityagencies(publichealthprimarycareschoolsCambridgeHealthAllianceetc)streamlinedandconnectedthevariousShapeUpSomervilleinitiativesandgrantswhichhelpedtokeepthetalentandenergyfocusedonhealthyeatingandactivelivingforallsectorsofSomervilleSUSpartnersultimatelyincludedschoolsnonprofitsCityandstateagenciesandnationalfoundations
Yourturn
____________________________________________________________________________
____________________________________________________________________________
DescribetheinterventionWhatarethestepsactivitiesorprocessesRememberthatthemechanismthespecificstepsorprocessthatleadtobehaviorchangeiskeytounderstandingtheeffectivenessofthepolicychangeandsharingtheresultswithothers
ExampleGoalsofShapeUpSomervilleweretoinfluenceeverypartofanearlyelementaryschoolchildrsquosdayndashbeforeduringandafter-schooltopromotehealthyeatingphysicalactivityandtheirweightstatusInterventionincludedfoodserviceenhancementstoimprovebreakfastandlunchandstaffprofessionaldevelopmentwalktoschoolactivitiesSUSclassroomcurriculumandprofessionaldevelopmentschoolwellnesspolicydevelopmentSUSafter-schoolcurriculumandprofessionaldevelopmentoutreachandeducationtohomethroughmaterialsforumseventsldquoSUSapprovedrdquorestaurantscommunityoutreachandcapacitybuildingthroughpolicydevelopmenttrainingsmediaplacements
Yourturn
____________________________________________________________________________
____________________________________________________________________________
ClarifytimelineandmilestonesfordeliveryandevaluationHowfrequentlywilltheservicesinterventionbedeliveredHowlongwilltheprogramrunWhatisthetimetoexpectedimpacts(eg12monthsafterprogramcompletion)
Designimplementationandevaluation
46
ExampleShapeUpSomervilleledbyTuftsUniversityFriedmanSchoolofNutritionScienceandPolicyconductedaquasi-experimental(nonrandomized)interventionstudyamong1stthrough3rdgradestudentsinSomerville(intervention)andtwocomparisoncommunitiesfor9-monthsduringoneschoolyear(fall2002ndashspring2003)ThestudyshowedpositiveassociationoftheinterventiononchangeinBMISUSreleasedreportssummarizingevaluationsprogresstodateplansforadditionalprojectsandimpactstenandfifteenyearsintotheinitiative
Yourturn
____________________________________________________________________________
____________________________________________________________________________
PlanyouranalysisTheprevioussectionaddressedestablishinganaccountabilityframeworktomonitorprogressandimplementationDeterminewhichindicatorswillbediscussthroughthatprocesswhowillsharethedataandanalysisandhowfeedbackfromthoseconversationswillbeincorporatedbytheprojectteam
ExampleTheMayorrsquosofficeimplementedadata-drivendecisionmakingapproachcalledSomerStatwhichwassynergisticwiththeapproachestakenbytheresearchandevaluationteamsdeployedforthespecificgrants
Yourturn
____________________________________________________________________________
____________________________________________________________________________
PlanforsustainabilityIfthepolicyandprogramchangeissuccessfulhowwilltheprojectteamsustainandexpandtoadditionalsitesorpopulations
ExampleAlthoughmostofthepartnershipsforgedintheearlyyearsweremaintainednewgrantsandcollaborationsexpandedattentiontoolderpublicschoolstudentsactivetransportationandtheGreenLinesubwayextensionthebuiltenvironmentfarmerrsquosmarketeffortssuchasldquoGrowninSomervillerdquowithUnionMainStreeteconomicdevelopmentandhealthdisparitiesaswellasdatacollectionmonitoringandsurveillance
Yourturn
Designimplementationandevaluation
47
____________________________________________________________________________
____________________________________________________________________________
Pilotplanningtemplate
IfyouplantoconductapilotfirstdeterminethepurposeofthepilotDoyouwanttoobservewhetherthepolicychangeresultsinthedesiredbehaviorchangeAreyouconcernedaboutfeasibilityscalabilityorcostDoyouwanttotestwhetheryourdesiredresultscanbeachievedthroughthispolicyapproachClarifythegoalofthepilotandthequestionsyouhopetoanswerThiswillhelpyouidentifythekeyindicatorsthatyouwillusetomeasurewhetherthepilotissuccessfulBeforebeginningestablishabaselineofcurrentperformanceandsetyourtargetsaccordinglybasedonwhatyouhopetoaccomplishthroughthepilot
NextestablishthepilotplanforyourprojectteamincludingmilestonesforteamcheckinstodeterminewhethertomoveaheadorchangecourseThisisseparatefromthemeetingsthatwilloccuraspartoftheaccountabilityframeworkUsethespacebelowtocreateyourtimelineanddevelopyourmeetingschedule
Task Personresponsible Startdate Checkindate Deliverable
Planyourevaluation
WhileyouaredesigningtheimplementationitisalsotimetobegintodevelopyourplanforanalysisAsnotedbytheWilsonSheehanLabforEconomicOpportunitiesdecisionsaboutprogramsetup-includingoutreachscreeningintakedatatrackingandenrollment-candeterminewhetherornotarigorousimpactevaluationwillbepossibleAsdiscussedcityleadersmayfindithelpfultoincludeevaluationpartnersinearlyconversationsaboutprogramsetupDuringpreviousstagesyoumayhaveworkedwitharesearchertoplanyourproposedpolicychangeYoushouldbeclearaboutthepurposeoftheevaluationanddevelopanevaluationplanBelowareadditionalconsiderations
Causationvstrendanalysis
IntheEstablishaframeworktomonitorprogressstagetherewasguidanceontrackingtrendsovertimeItisimportanttonoteherethatmonitoringdataisdifferentfromanalyzingacausaleffectTrendanalysiswillillustratethatchangesareoccurringandmayleadtoquestionsthatcouldbeansweredbyyourevaluationbuttrendanalysisalonedoesnotprovideevidenceaboutwhatisdrivingthosechangesYoumayseechangesinyour
Designimplementationandevaluation
48
outcomeindicatorsbasedonvariablesoutsideoftheCityscontrolForinstanceachangeinfederaltradepolicycanhaveanimpactemploymentratesinyourCityoutsideofyourCitysnewjobtrainingandplacementprogram
ToaddresscausalityyoumustcomparetheoutcomesofindividualswhoreceivedtheinterventionwiththosewhodidnotWithoutacomparisongroupofsimilarindividualswhowerenotexposedtotheinterventionitisimpossibletoknowwhichfactorsareresponsiblefortheoutcomesRandomizedControlTrials(RCTs)inwhichindividualsarerandomlyassignedtotreatmentandcontrolgroupsarethegoldstandardinprovingcausalityIfyoucandemonstratethatoutcomesimprovedforthosewhoreceivedtheinterventionanddidnotimproveatallorimprovedlessforacomparisongroupyoucanbesomewhatcertainthattheinterventionwasthecause
Examplesofothernon-causalimpactevaluationsthatarerigorousandreliableincludecertainquasi-experimentalstudiesusingcomparisongroupdesignssmartinterruptedtimeseriesanalysesthatmodeltrendsovertimeinwaysthatallowestimationofinterventioneffectswelldesignedpre-posttestanalysesneedsassessmentsimplementationstudiesetcFormoreinformationondifferenttypesofevaluationsseetheSageResearchMethodswebsite
Lookfordifferencesingroups
BuildintimetoalsoexaminethepilotdatatoseeifgroupsareexperiencingdifferentresultsReviewingperformancebasedonoutcomemeasuresforapopulationinaggregatemayhidethesubtlewaysthatsomepeoplemaybeaffecteddifferentlysoitisnecessarytocarefullyanalyzethedatafordisparateoutcomesForinstancetheMovingtoOpportunity(MTO)studythatexaminedtheimpactofhousingvouchersonfamilieswholivedinhigh-povertyneighborhoodsfoundthatchildrenwhomovedbeforetheageof13hadbettereducationandhigherincomesasadultsHoweveradultsdidnotseeimprovededucationandemploymentoutcomes
Disaggregatingdataintomorespecificcategoriesofraceandethnicitycanhelpunderstandtrendsanddisparitiesinamoregranularwaywhichcanhelpdecisionmakerstargetpoliciesandinterventionsinamoreequitableway
EarlierintheRoadmapweworkedtoidentifystakeholdersandwhomightbepositivelyandnegativelyimpactedbyachangeinpolicyUsetheresultsofthatexercisetoinvolvethecommunityininterpretingdataanalysisandtogatherqualitativefeedbackfromthecommunitytocallattentiontoissuesForinstanceanincreaseinyouthdetentionmaybeexplainedbyinefficienciesorlackofaccesstotransportation
Designimplementationandevaluation
49
Designimplementationandevaluation
50
Shareyourstoryandsuccess
Sharetheresults
Storytelling
AnintegralstepinyourstrategyissharingtheresultsofyourpilotorpolicychangewithyourcommunityThisincludesdevelopingmessagingaroundtheworkofyourorganizationandcreatingaplantodisseminateyourstoriesWhytellstoriesaboutyourwork
Informandorunderstandasubjectorsituation
Giveneededcontextofdataorbitsofinformation
Relatablestoriesarememorableforreadersviewers
ReadersorviewerswillseetheinformationasmorecompellingwhenthereisademonstratedandrelatablechangeorimpactonthemselvesortheirfriendsneighborsandcolleaguesStoriesareaboutpeopleandgovernmentisfundamentallyabouthelpingpeopleSotellthestoriesofthepeopleyouandyourcitycolleagueshelped
Keepthesetipsinmindasyoudevelopyourstorywithyourteam
WhatwereyourfindingsandwhyshouldanyonecareTellyouraudiencehowtheproblemwassolvedWhathastheimpactbeenonresidentsandcitystafftodateWhatwasthesuccessandweretheremultiplesuccessesalongthewayIfyouhavepreliminaryresultsordatashowingimprovementtosharethisisthetimetodoit
WhowasinvolvedandimpactedReviewthelistofstakeholdersHowwereeachofthemimpactedbytheprogramHumanizeyourstoryandimpactbyaddingadditionalvoicesThesevoicescanbeoneofthefollowing
Apersonorfamilywhowaspositivelyaffectedbythechangesuccess
Aninvolvedthird-partydescribingboththeproblemandthesolution(ieanorganizationnonprofitprovidersbusinesstradeassociationsportsteametc)whoispartofthecommunitythatthegovernmentworkedwithonthechangesuccess
Someonefromthegovernmentitselfwhocanspeakaboutthechangesuccess
TellmoreaboutyourcityDescribeyourcityincludingpopulationdemographicsindustriesetcWhatareitschallengesWhatisitknownforWhatmakesituniqueHowisitsimilarordifferentfromothercitiesintheUS
Shareyourstoryandsuccess
51
DescribeyourcityrsquoschallengeWriteoutthechallengeyourcityisfacing(asdefinedinyourproblemstatement)HowdoesitaffectresidentsHowdoesthisproblemaffecttheworkofcitystaffmembers
AddthedetailsProvidedetailsonyourappliedresearchstrategyTalkaboutwhatyourcityhasdonenthepasttoresolvetheissueDiscusshowyourstrategyhaschangedwhenyoubegantoadoptthisapproachWasmoneysavedorinvested
Usethisstorymapexercisetoorganizeyourthoughtsandguaranteethatyouincludealloftheelementsofatraditionalstory
AfteryoudevelopyourmessagingcreateacommunicationsmarketingandoutreachstrategydesignedtospreadyourmessageYourstrategyshouldwalkyouthroughthefollowingkeyactivities
1 IdentifyandprofileyouraudienceWhencreatinganddisseminatingyourmessageitisalwaysgoodtothinkaboutyouraudienceWhoexactlywouldbeinterestedinwhatyouhavetosayThegeneralpopulationofthecityPeoplewhoworkinnonprofitsBusinessownersItmattersbecausenoteveryonewholivesinacitymaybethetargetoftheinformationBusinessownersmaybemoreinterestedindatashowingeffectsonthebusinessdistrictthanothersandsoonWewillcoverthislaterbutthisstepwillalsohelpyoutothinkofthemosteffectivewaytoreachyouraudience
2 Defineyourcalltoaction(ifyouhaveone)SpendsometimeconsideringwhyyouareinterestedintellingyourstoryDoyousimplywanttoupdatethepubliconyourprogressorcelebrateyoursuccessesAreyouinterestedinyouraudiencechangingitsbehaviororsupportingaspecificinitiativebeinglaunchedbyyourorganizationIfyouwanttohaveyouraudienceactorbehaveinacertainwaybespecificandincorporateitinyourmessaging
Shareyourstoryandsuccess
52
3 SelectperformancemeasuresThinkabouthowyouwoulddetermineifyourcampaignissuccessfulandselectappropriateperformancemeasuresYouwillrevisitthesemeasuresattheendofyourcampaigncommunicationsplanimplementationtodeterminewhetheryouneedtomodifyyourcommunicationsandoutreachstrategyforthefuture
4 MethodstocommunicateyourstoryTherearemanywaystocommunicatesuccessofyourprojectHerearesomesuggestionsonreachingaudiencesItisimportanttodeterminewhichmethodsarethemosteffectiveforyourdesiredaudiencetoreceiveinformationDoyourresearch(PewResearchCenterhasexcellentstudiesonthereachofmediadigitalmediaandmarketing)ForexampleifseniorsareyourtargetpopulationandtheyarelessinclinedtouseInstagram(thatistrue)youmaywanttoconsiderFacebookorthenightlylocalnewsbothofwhichareusedmoreoftenbyseniorsBelowisalistofsomemethodsandtoolsyoucanuse
i Pressreleasestolocalnewspapersandtvstations
ii ShortsmartphonevideosAgreatwaytoengageanaudienceisthroughvideoAskonequestionofsomeonewhoispartofthestoryMaybethereisasocialworkerwhocandiscusshungerinyourcityThesevideoscanbereusedonmultipleplatforms
iii Long-formwriting
iv TellingindividualstoriesthroughaseriesofphotosandshortnarrativesforpublicationonFacebookorInstagram
v Op-edsopinionpiecesplacedinthelocalpaper
vi Emailupdatestoyourlist
vii Connectingwithappropriateorganizationsthatareinvestedinthetopicandworkwiththecommunityandaskthemtodisseminateinformationtotheirmembers
viii EmployingldquoWordofMouthrdquo(WOM)tacticsbycreatingtownhallsorfreeeventsforresidentstoattendintheirownneighborhoods
5 DeterminethetypeofcollateralUsuallycampaignsthattargetresidentsrequireseveralavenuestoreachthemPeoplearenotalwaysinthesameplaceorspaceHavingakeenunderstandingofyouraudiencewillassistyouwithyourchoicesonwhatneedstobedevelopedSometimesthatmeansimagesforsocialmediaaonepageflierthatgivespeopleinformationonattendinganeventoranin-depthbrochure
Shareyourstoryandsuccess
53
6 ImplementyourstrategyKeepinmindstrategiescanbefluidSometimessomethingmaynotturnoutasexpectedevenifduringplanningsessionsitwasldquosupposedtordquoItisokYoucananalyzewhathappenedduringadebriefsessionandonceyoursurveysandothermeasurementshavebeenfinalized
Visualizations
Withtherightvisualizationyoucanreviewandshareinformationinsightsanalysisperformanceandtrendsallofwhichisvaluabletoacross-sectionofthepublicButwhichvisualizationisrightforyouTochoosetherighttypeofchartgraphdeterminethetypeofquestionyouwanttoanswerSomecommoncharttypesarebelowandmoreinformationcanbefoundhere
Exploreadditionalanalyticaltechniques
Duringthegoal-settingphaseyouestablishedoutputoutcomeandimplementationmeasuresBasedontheresultsofyourpilotyoumayneedtoadjustyourmeasuresWiththepilotdatainhandyoumayhavesomeideasofhowfutureanalysescouldbedifferentandpotentiallyimprovethequalityofyourprogramTakesometimetobrainstormotherrelevantdatasetsthatyoudidnothavethetimetopursueearlierincludingmessydepartmentaldatathatneedsmoreattentionorsourcesofpublicdatafromthestateandfederalgovernmentnonprofitsandevenprivatecompaniesthatcouldsupplementyourunderstandingofanissueUsingmachinelearningtechniquesyoumightbeabletobetter
Shareyourstoryandsuccess
54
predictdemandforhomelessservicesusingNOAAweatherforecastspreemptwatermainbreaksusingsoilstratigraphyorinvestineconomicdevelopmentbymonitoringrealestatelistingdata
Advancedanalyticaltechniquescanalsohelpyourimplementationrunmoreeffectivelybyintegratingdata-informedtargetinganddecisiontoolsintotheprocessByautomatingroutineworkflowsandsurfacingsubtletrendsthesetoolscanhelpyouiterateinamoreprecisemannerandprovidekeystrategicguidancetofront-lineemployeesthatwouldotherwisebeunavailableThesesortsofdata-science-enhancedprocessescanbeclassifiedasoneofsixprojecttypesasexploredfurtherbytheNewOrleansLANOLAlyticsteamFormoreguidanceonscopingasimilarprojecttheGovExadvancedanalyticsscopingguidegivesyouastep-by-stepoverviewofhowtobesuccessful
Usethespacebelowtolistyourideasforpossibleadvancedanalyticsprojectsbytype
Typeofproject Ideas
Findingtheneedleinthehaystack
Prioritizingforimpact
Earlywarningtools
Betterquickerdecisions
Optimizingresourceallocation
Experimentingforwhatworks
Keepthefeedbackloopgoing
GatherinformationonpublicsentimentaboutpolicychangethroughfocusgroupssurveysandpublicmeetingsledbyadvisorygroupmembersNotewhethernewinformationisavailablethatcanhelpanalyzeorcontextualizeyourprogress(egfederalorstatedatareports)Sharetheresultsofyourpilotandthepolicychangeprocesswithothercitiesandresearchers
Shareyourstoryandsuccess
55
WorkingwitharangeofevidencePresentlythereisalotofattentionandinvestmentinevidence-basedpolicymakingincludingthefederalbipartisanCommissiononEvidence-BasedPolicymakingthePewCharitableTrustsreportsandtoolkitsonHowStatesEngageinEvidence-BasedPolicymakingandtheLauraandJohnArnoldFoundationsupportedEvidence-BasedPolicymakingCollaborativeManyoftheseinitiativesfocusonrigorousevaluation--meaningrandomizedcontrolledtrials(RCT)orotherexperimentalmodels--asthebestkindofevidencetouseHoweverthisbodyofevidenceisofteninsufficientforthespecifickindsofquestionsthatgovernmentscanhavewhenmakingadecisionaboutpolicyInthisguideweproposeanapproachtosupportgovernmentsinincorporatingadditionaltypesofevidenceintotheirdecisionmaking
OnewaytounderstandarangeofevidenceistouseNestarsquosStandardsofEvidencetoframeanddefineevidenceTheBritishnonprofitNestaworkstoinjectpositivechangeintothewayUKgovernmentpoliciesandprogramsfunctionOneoftheirkeyareasofworkhasbeendevelopingWhatWorksCentresorganizationsthatcollectcommissionsynthesizeanddisseminateevidenceaboutinterventioneffectivenessacrossarangeofsocialimpactareasNestadevelopedthefollowingStandardsofEvidencetoprovideausefulwaytodocumenthowdifferentkindsofinformationdemonstratedifferentlevelsofevidence
Workingwitharangeofevidence
56
Theimportanceofpotentiallybiasedinformation
Onechallengepresentedbyanevidenceclassificationthatacknowledgesthevalueofnon-scientificformsofevidenceistofigureouthowtobalanceallofthepiecestogetherwhenyouusethemTounderstandhowtodothiswehavetothinkabouthowtopositionpotentiallybiasedinformationindecisionmaking
WhenwelookatinformationwhichhasnotbeenrigorouslyevaluatedweacceptthatthisinformationmightbebiasedThewholereasonweconductrigorousevaluationistoremovebiasfromourobservationsWeconductstructuredtrialstoproduceresultsthatareconsistentandgeneralizableIfafindingisunbiasedthisaccuracymdashthislackofbiasmdashcanbeprovedbecauseanyindependenttestercanreproduceitwhentheyfollowanidenticalprocess
YoucanlookatpotentiallybiasedinformationasjustinformationthatisearlieroninitspathtorefinementreplicationandverifiedaccuracyApieceofinformationbasedonanyoneindividualrsquosobservationsislikelytobebiasedbecauseitiscomingfromjustonesourceBiasisreducedbygatheringtogetherlotsofindependentsourcesofinformationandfiguringoutitsmostcommonattributes
MaybethebiasedinformationmakesaclaimthatistoolargemdashitsayssomethingistrueunderallconditionswhenitisreallyonlytrueundercertainconditionsorthataneffectofaninterventionisbiggerthanitstrueaverageorthataneffectlastsovertimewhenitdoesnotPerhapschangingtheclaimtobemorespecificmdashcriticallyreviewingtheinformationtoseeifspecificconditionsmustbemettoachievetheoutcomesmdashwouldmaketheclaimmoreaccurate
IfyoutestedyourclaimfounditdidnotholdupinsomecaseschangedittesteditagainuntilyourclaimrepeatedlyheldupthenyoucoulddemonstratethattheclaimwasnolongerbiasedbutaccurateThisprocessoftestingimprovingtestingandvalidatingiswhatwegetfromstructuredscientificstudy
Withindividualpiecesofinformation--crediblehypothesescasestudiesexpertopinion--youcannotassumethatkindofaccuracyAtthesametimetheseearlyobservationsarefoundationaltoanyreplicablescientificobservationsWewillnevergetanywhereifwedonotstartwiththesepiecesofinformationeveniftheymightbebiased
Combininglevelsofevidencetogetthemostdetailedinformationforyourcity
ItisespeciallyimportanttoacceptdifferentformsofevidencewhenyoufacequestionsthathavenotyetbeentestedbyindependentstudiesSomeofthemostimportantquestionsrelatetothingsthatareveryspecifictoyoursituationYoufaceaparticularcombinationof
Workingwitharangeofevidence
57
issuesactorstrendsandresourcesthatisultimatelydifferentfromeveryotherplaceintheworld
ThisisimportantbecauseaspolicypractitionersandnotscientistsyoucaremoreaboutyourspecificcasethanahypotheticalaveragegeneralcaseYoucarelessaboutthetotallikelyrangeofimpactsfromapolicythanhowthissingleimplementationwillworkYouwantinformationaboutpoliciesthatwillprovideyouwiththebestpossiblepredictionabouthowtheywillworkinyourcontext
ByputtingdifferentkindsofevidenceintothesameframeworkNestarsquosStandardsofEvidenceimplythatwhenyoucannotgetrigorouslytestedexperimentallyverifiedinformationitisappropriatetoworkwithwhatwedohaveThisisrelevantforthemunicipalimplementationcontextbecausenostudypulledfromaliteraturereviewwillaccountforallofthevariablesyoufaceinyourowncontext
Intheabsenceofhavingastudythatpredictshowaninterventionworksinyourspecificcontextyouwillneednon-scientificinformationtotrytofillinthosegapsYouwillneedtogatherstoriesfromresidentsstaffmembersandotherstakeholdersthatsitonLevels1or2ofthestandardsofevidence
Someofthekindsofinformationyouwillprobablygetmoreeasilyfrompeoplethanstudiesinclude
howpeopleincitieslikeyoursreactedtoaninterventioninpublicforumshowpeoplewhoimplementedaninterventioninacitylikeyoursfiguredouthowtogetitpassedhowthecitycommunicatedpubliclyabouttheapparentbenefitsanddrawbacksoftheinterventionthepoliticalacceptabilityoftheintervention
Youcanremainsimultaneouslyawarethattheinformationyouaregatheringthroughindividualstoriesisbiased--notgeneralizableinformedbyspecificperspectives--whilealsorecognizingthathavingsomeinformationaboutsomethingimportantisbetterthannoinformationEvidencefromthefirsttwoevidencelevelsareessentialbuildingblocksofknowledgeandperfectlyusefulwhenyoucannotgetdetailedunbiasedknowledge
Youhavetostartsomewhereandyoushouldalwaysstartwithevidenceyouhaveaboutyourveryspecificconditionsthatyouknowareimportantforsuccess
Workingwitharangeofevidence
58
Thankyou
Interviewswithcities
GovExheldsemi-structuredinterviewswithsixstaffmembersinfourcitiestoestablishtheinitialstepsoftheRoadmapWeaskedrespondentstotalkaboutaninitiativeorpolicythattheyhavebeenresponsibleforimplementingintheircityRespondentsfromthreecitiesspokeabouttheireffortstoaddresshomelessnessandrespondentsfromonecityspokeaboutblightAfterrespondentsidentifiedtheproblemtheyspokeabouthowthecityprioritizedthisissuethevariousstepsinvolvedandbarrierspresent
GovExconductedtextanalysisoftheseinterviewswhichrevealedthatpeoplewasthemostcommonlyusedtermduringtheseconversationswhendiscussingthestepstochangeaprogramorpolicyThisisathemethroughoutthisRoadmapThefirstsectionbeginswithidentifyingtherelevantpeopleTheconversationsaroundevidencearefocusedonhumanbehaviorratherthanmethodologyorstrengthofevidence
Thankyou
59
OncewehadadraftoftheRoadmapwefollowedupwithcitiesonanindividualbasistolearnmoreaboutwhatwouldbehelpfulWearegratefultothefollowingcitystafffortheinterviewsandindividualconversations
MichaelBaskinChiefPolicyOfficerChattanoogaTN
MelissaBridgesPerformanceandInnovationCoordinatorLittleRockAR
VictoriaCarreoacutenAdministrativeOfficerLasVegasNV
TyroneGrandstrandCommunityDevelopmentPlannerFargoND
TibLaughlinDirectorofGeneralServicesKansasCityKS
TimMorelandDirectorofPerformanceManagementandOpenDataChattanoogaTN
CharlesRosendahlManagementAnalystDentonTX
Thankyou
60
MelissaSiebenAssistantCountyAdministratorKansasCityKS
GinaThompsonDirectorofStrategicServicesLewisvilleTX
JHUFacultycollaboration
GovExalsoconvenedthecitycontactsandfacultypartnersinaseriesofAdvisoryGroupmeetingsThesemeetingsservedtoconnectcitiesandfacultytounderstandoneanothersperspective
StefanBaralBloombergSchoolofPublicHealth
AshleyBernerJHUSchoolofEducation
MelissaDavey-RothwellBloombergSchoolofPublicHealth
JonLinksBloombergSchoolofPublicHealth
TamarMendelsonBloombergSchoolofPublicHealth
KristinMmariBloombergSchoolofPublicHealth
KeshiaPollackPorterBloombergSchoolofPublicHealth
JoshSharfsteinBloombergSchoolofPublicHealth
LizStuartBloombergSchoolofPublicHealth
LindsayThompsonCareyBusinessSchool
DanielWebsterBloombergSchoolofPublicHealth
Partnersindataandevidence-basedpolicy
Wealsospokewithpartnerswhoareexpertsinthefieldofevidence-basedpolicywhosharedtheirknowledgeandresourcesofcurrentandfuturethinkinginevidence-informedthinking
WendyBarrenoAssociateDirectorWilsonSheehanLabforEconomicOpportunitiesUniversityofNotreDame
TiffanyDavenportDirectorofDataPartnershipsBaltimoresPromise
TomiHiersDirectorBaltimoreCivicSiteAnnieECaseyFoundation
ChrisKingsleySeniorAssociateAnnieECaseyFoundation
KathrynOliverAssociateProfessorLondonSchoolofHygieneandTropicalMedicine
Thankyou
61
KristenWatkinsSeniorPolicyAssociateJ-PAL
GovExParticipants
GovExAnalystMiriamMcKinneyanalyzedtheinterviewtranscriptsandconductedtextanalysisSeniorImplementationAdvisorEmilyShawwillintegrateRoadmapactivitieswithadditionalservicedeliveryactivitiesatGovEx
Thankyou
62
ReferencesAbrahamKGampHaskinsR(2017)Thepromiseofevidence-basedpolicymakingReportofthecommissiononevidence-basedpolicymakingWashingtonDCCommissiononEvidence-basedPolicymakingBigmove2Extendingthegreen(nd)Retrievedfromhttpswwwcityoflewisvillecomdoing-businesslewisville-2025big-move-2-extending-the-greenBatesMAampGlennersterR(2017)ThegeneralizabilitypuzzleRetrievedfromhttpsssirorgarticlesentrythe_generalizability_puzzleBuildwithnotfor(2018January)RetrievedfromhttpsbuildwithnotforwordpresscomCairneyPampOliverK(2017)Evidence-basedpolicymakingisnotlikeevidence-basedmedicinesohowfarshouldyougotobridgethedividebetweenevidenceandpolicyHealthResearchPolicyandSystemsdoi101186s12961-017-0192-xCairneyPOliverKampWellsteadA(2016)TobridgethedividebetweenevidenceandpolicyReduceambiguityasmuchasuncertaintyPublicAdministrationReview76(3)399-402Retrievedfromhttpdxdoiorg101111puar12555Centreforpublicimpact(nd)RetrievedfromhttpswwwcentreforpublicimpactorgCernaL(2013)ThenatureofpolicychangeandimplementationAreviewofdifferenttheoreticalapproachesOrganisationforEconomicCo-OperationandDevelopmentRetrievedfromhttpswwwoecdorgeduceriTheNatureofPolicyChangeandImplementationpdfColoradoSprings(nd)Retrievedfromhttpscoloradospringsgovcommunicationspagewhat-expect-when-you-speakupContinuumofcare(coc)program(nd)RetrievedfromhttpswwwhudexchangeinfoprogramscocCouncilpresidentkicksoff$12millionyouthfund(nd)Retrievedfromhttpwwwbaltimorecitycouncilcomcontentcouncil-president-kicks-12-million-youth-fundDurlakJAampDuPreEP(2008)ImplementationmattersAreviewofresearchontheinfluenceofimplementationonprogramoutcomesandthefactorsaffectingimplementationAmericanJournalofCommunityPsychology41327-350Retrievedfromhttpswwwncbinlmnihgovpubmed18322790Fishbonediagram(ishikawa)(nd)Retrievedfromhttpasqorglearn-about-qualitycause-analysis-toolsoverviewfishbonehtmlFivewhysGettingtotherootofaproblemquickly(nd)RetrievedfromhttpswwwmindtoolscompagesarticlenewTMC_5Whtm
References
63
FligsteinNampMcAdamD(2011)TowardageneraltheoryofstrategicactionfieldsSociologicalTheory29(1)1-26RetrievedfromhttpsociologyberkeleyedusitesdefaultfilesfacultyfligsteinFligstein20McAdam20Soc20Theory20PaperpdHeinkelP(2018January12)CitymullspermitsforbikesharingcompaniesRetrievedfromhttpwwwdentonrccomnewsdenton20180112city-mulls-permits-bike-sharing-companiesHowisareamedianincomecalculated(2012October)Retrievedfromhttpswwwhudexchangeinfofaqs843how-is-area-median-income-calculatedLampkinLMampHatryHP(2003)KeystepsinoutcomemanagementUrbanInstituteRetrievedfromhttpswwwurbanorgresearchpublicationkey-steps-outcome-managementMarshDampSmithM(2000)UnderstandingpolicynetworksTowardsadialecticalapproachPoliticalStudies484-21Retrievedfromhttpjournalssagepubcomdoiabs1011111467-924800247McCannL(nd)RetrievedfromhttplaurenellencomMoultonSampSandfortJR(2016)ThestrategicactionfieldframeworkforpolicyimplementationresearchPolicyStudiesJournal45(1)Retrievedfromhttponlinelibrarywileycomstore101111psj12147assetpsj12147pdfv=1ampt=jdkhhcp4amps=e184c625e1855a6b1615ccb99ee27b1c0b3212d8PaynterB(2018January1)What100citiesarelearningfromeachotherbysharingtheirdataRetrievedfromhttpswwwfastcompanycom40518004what-100-cities-are-learning-from-each-other-by-sharing-their-dataPeakacademy(nd)Retrievedfromhttpswwwdenvergovorgcontentdenvergovenmayors-officeprograms-initiativespeak-performancepeak-academyhtmlPinnockHBarwickMCarpenterCREldridgeSGrandesGGriffithsCJRycroft-MaloneJMeissnerPMurrayEPatelASheikhAampTaylorSJC(2017)Standardsforreportingimplementationstudies(stari)ExplanationandelaborationdocumentBMJOpenRetrievedfromhttpbmjopenbmjcomcontentbmjopen74e013318fullpdfProctorESilmereHRaghavanRHovmandPAaronsGBungerAGriffeyRampHensleyM(2011)OutcomesforimplementationresearchConceptualdistinctionsmeasurementchallengesandresearchagendaAdministrationandPolicyinMentalHealth38(2)65-76doi101007s10488-010-0319-7TandonSD(2018)ProgressincommunityhealthpartnershipsResearcheducationandactionRetrievedfromhttpswwwpressjhuedujournalsprogress-community-health-partnerships-research-education-and-actionProvanKGampKenisP(2007)ModesofnetworkgovernanceStructuremanagementandeffectivenessJournalofPublicAdministrationResearchandTheory
References
64
18(2)229-252Retrievedfromhttpsacademicoupcomjpartarticle182229935895SabatierPA(1988)Anadvocacycoalitionframeworkofpolicychangeandtheroleofpolicy-orientedlearningthereinPolicySciences21(3)129-168Retrievedfromhttpwwwjstororgstable4532139SandstromAampCarlssonL(2008)TheperformanceofpolicynetworksTherelationbetweennetworkstructureandnetworkperformancePolicyStudiesJournal36497-524Retrievedfromhttponlinelibrarywileycomdoi101111j1541-0072200800281xabstractSchwartzMampWilsonE(nd)WhocanaffordtoliveinahomeAlookatdatafromthe2006americancommunitysurveyRetrievedfromhttpswwwcensusgovhousingcensuspublicationswho-can-affordpdfShearerJCAbelsonAKouyateBLavisJNampWaltG(2016)WhydopolicieschangeInstitutionsinterestsideasandnetworksinthreecasesofpolicyreformHealthPolicyandPlanning31(9)1200-1211Retrievedfromhttpsdoiorg101093heapolczw052ThelessonsofadministrativedataHigh-profilepolicy-relevantresearchpoweredbyadministrativedata(nd)Retrievedfromhttpswwwpovertyactionlaborgsitesdefaultfilesresources20161114-The-Lessons-of-Administrative-DatapdfToolsandotherneedsforqualitativeandquantitativemethods(2015February)AgencyforHealthcareResearchandQualityRetrievedfromhttpwwwahrqgovprofessionalsquality-patient-safetytalkingqualityassesstoolshtml
WeaverRCampBagchi-SenS(2013)SpatialanalysisofurbandeclineThegeographyofblightAppliedGeography4061-70
References
65
UniversityPartners
Name Title Affiliation Area(s)ofexpertise
Contactinformation
Examplesofworkingwithnon-academicdecisionmakers
WendyPBarreno
AssociateDirector
WilsonSheehanLabforEconomicOpportunitiesUniversityofNotreDame
LEOsfocusareasincludehealtheducationhousingchildwelfareeconomicopportunitycriminaljustice
wbarrenonduduorleondedu
MelissaDavey-RothwellPhD
AssociateScientist
JohnsHopkinsBloombergSchoolofPublicHealth
Health mdavey1jhuedu410-614-5854
BaltimoreCityHealthDepartmentLeadersMarylandDepartmentofHealthAddictionRecoveryServicesinDCExecutiveDirectorsofcommunityorganizationsacrosstheUSthatfocusonHIVSTIsubstanceabuseandimplementationscience
ElizabethStuart
AssociateDeanforEducationandProfessorDepartmentsofMentalHealthBiostatisticsandHealthPolicyand
JohnsHopkinsBloombergSchoolofPublicHealth
Educationsocialpolicystatisticalmethods
estuartjhuedu
Resources
66
Management
Design-ThinkingResources
EngagementGuideforCivicTech
Journeymap
PerformanceManagementResources
DenverPeakAcademy
Proxymeasuresguide
Talkingaboutperformancewithoutcomplicatingit(video)
SettingPerformanceTargetsguide
Datavisualizationguide
Performancemanagementgettingstartedguide
Exercises
Fishboneexercise
5Whys
AdministrativeData
TheLessonsofAdministrativeData(JPAL)
EvaluationResources
Quantitativeandqualitativemethods(USDepartmentofHealthandHumanServicesAgencyforHealthcareResearchandQuality)
KeyStepsinOutcomeManagement(UrbanInstitute)
RCT-YES
Sageresearchmethods
BestPracticesforMixedMethodsResearchintheHealthSciences
ACCORDChecklist
Pilotandimplementationplanning
Benchmarkingexamplesandproxymeasures
Seattleopendataplaybook
Resources
67
Datasciencescopingguide
Resources
68
AboutGovExGovExhelpsgovernmentsusedataandevidencetoimproveresidentsrsquolivesWesupportgovernmentsinbuildingtheirinternalcapacitytooperatesustainableanalyticspracticeswhichimprovecommunityoutcomesinfieldsfrompublicsafetytoeconomicdevelopmentGovExhasaproventrackrecordofworkinginconcertwithpartnerstoprovidetechnicalassistanceandtrainingatscale
SinceourlaunchinApril2015GovExhas
Trainedover2000governmentemployeesinpersonandonlineintopicsincludingdatamanagementperformancemanagementopendatacommunityengagementandanalytics
Providedtechnicalassistancetomorethan100localgovernmentsacrosstheglobeimprovingtheiruseofdatatoachieveefficienciesinservicedeliveryandbetteroutcomesforresidents
Assessedthecapacityofmorethan130governmentalorganizationstodeterminehowleadersstaffandexternalstakeholdersmanageandusedata
Localgovernmentsareincreasinglyonthefrontlinesofaddressinglargescaleproblemswithglobalimplicationssuchasclimatechangepublicsafetyandeconomicinequality
Whileevidence-basedinterventionstohelpsolvethoseproblemsareavailablelocalgovernmentstaffhavelimitedtimeandcapacitytofindandimplementresearch-basedsolutionsAsaresultcitiescontinuetoworkthewaytheyalwayshave--makingincrementalimprovementstoprogramsbutnotalwayscreatingchangeatasystem-level
ThroughresourcedevelopmentandcollaborationwithJHUfacultyGovExissupportingcityleadersinusingdataandevidencetomakedecisionsandcreatinglastingrelationshipswithresearchers
TheRoadmapwasdevelopedbyGovExaspartofitspartnershipwithBloombergPhilanthropiesWhatWorksCitiesinitiative
AboutGovEx
69
- Title page
- Introduction and purpose of the Roadmap
- Using the Roadmap
-
- Gather information about the problem
- Strategically engage key partners
- Review the evidence and collect details
- Decide on a policy to implement based on local factors
- Establish a framework to monitor progress
- Assess data accessibility quality and security
- Design implementation and evaluation
- Share your story and success
-
- Working with a range of evidence
- Thank you
- References
- Resources
- About GovEx
-
Titlepage
2
11
12
13
131
132
133
134
135
136
137
138
14
15
16
17
18
TableofContentsTitlepage
IntroductionandpurposeoftheRoadmap
UsingtheRoadmap
Gatherinformationabouttheproblem
Strategicallyengagekeypartners
Reviewtheevidenceandcollectdetails
Decideonapolicytoimplementbasedonlocalfactors
Establishaframeworktomonitorprogress
Assessdataaccessibilityqualityandsecurity
Designimplementationandevaluation
Shareyourstoryandsuccess
Workingwitharangeofevidence
Thankyou
References
Resources
AboutGovEx
IntroductionRemembertheDAREprogramWhatbeganasapartnershipbetweentheLAPDandLApublicschoolstohelpkidssaynotodrugsbecameapoliticallypopularcampaignthatwasadoptedbyschooldistrictsaroundthecountryMeanwhilethroughthe1990sand2000smorethan30rigorousevaluationsfoundthattheprogramdidnotwork-withonestudyevenfindingthatitactuallyincreasedyouthdruguseThreedecadeslaterthefederalgovernmentisstilltalkingaboutrevivingthispopularbutdetrimentalprogram
Thechallengesthatcitiesface
ThestoryofDARErevealssomeoftheotherchallengesfacinggovernmentsinimplementingevidence-basedpoliciesPresentedwithconstraintsontimeandpressuretoactpolicymakersmayrelyonemotionsandgutfeelingsasashortcuttodecisionmakingItcanbehardtorejectthefundingandpoliticalcapitalthatoftencomeswithaprogramthathasthekindofmomentumthatDAREhadEvenifelectedofficialsadmitthatapolicyisproblematicoratleastnolongereffectiveittakesleadershiptoputenergyandresourcesintochangingcourseAndeventhencityleadershavetomakethecasetoresidentsforwhyachangeisnecessaryespeciallyaftertheyhaveinvestedtimeandpublicdollarsintoapreviousapproach
LocalgovernmentsareincreasinglyonthefrontlinesofaddressinglargescaleproblemswithglobalimplicationssuchasclimatechangeimmigrationandeconomicinequalityAlthoughresearchersarevalidatingdiagnosticmodelsandevidence-basedinterventionstohelpsolvethoseproblemslocalgovernmentstaffhavelimitedtimeandcapacitytofindevaluateandimplementthesesolutionsSearchesbycitystaffmayleadtopromisingexamplesinothercitieslinkstoscholarlyarticleswebsitesofnonprofitsthatworkonthisissuejournalarticlesandrelatednewsarticlesRatherthanweighingandevaluatingthisinformationcitiesmaychoosetotryonepolicysolutionthatseemsfeasibleWithoutcarefulmonitoringorevaluationthoughthistrialanderrorapproachcanleadtoanenvironmentinwhichcitiesfundserviceprovidersbasedonexistingrelationshipsandavoidrepurposingfundingfromprogramsthatareineffectiveforfearofrepercussionssuchaspeoplelosingtheirjobsFurtherconversationsaboutwhatothercitiesaredoingandtheavailableliteraturesometimeshappenonanadhocbasiscitystaffdonotoftenfeelpreparedtoconductasystematicreviewWhencitiesdomakechangestheyaresometimesincrementalandattheprogramlevelinsteadofaddressingmultiplefactorsatonceatasystemlevelAttemptingtoaddressaproblemthatstoobroadsuchasblightcanbe
IntroductionandpurposeoftheRoadmap
3
dauntingWithoutadeepunderstandingoftherootcausesoftheproblemitcanbedifficulttoidentifythebestevidencetouseFinallyitcanbedifficulttoknowhowtoimplementaparticularpolicyinyourcityandwhetheritwillhavethedesiredeffect
WhentheUnifiedGovernmentofKansasCityKSandWyandotteCounty(UG)beganworkingonitsblightproblemtheUGconvenedaresearchteamtolookforevidencefromothercitiesUsingsearchtermslikeblightlandbankabandonedpropertiesandvacancythatledtheUGtolearnaboutthesimilareffortsinthecitiesofNewOrleansBaltimoreMobileKansasCityMOandcitiesinOhioandMemphisTheteamalsoremainedopentohearingideasfrominternalstaffwholearnedaboutpracticesfromothercitiesthroughtheirownprofessionalorganizationsanduniversityaffiliationsthattheyhadnotbeenempoweredtotestorimplementpreviouslySincethetopicofblightrangesfrompropertymaintenancetosafetyissuestheCityresearchteamhadtodoalotofreadinganddebatingoverconflictingconceptsfortacklingawiderangeofissueswithinthatspectrumThisinvolvedaskingquestionsaboutwhetherthepolicywillworkinKansasCityandwhatchangeswouldneedtobemadeforittobesuccessfulWhenresearchinglandbankstheCitylearnedabouttheCenterforCommunityProgressanonprofitthathelpscommunitiesaddressvacantabandonedanddeterioratedpropertiesConversationswiththeCenterforCommunityProgressledtoexploringtheideaofavacantpropertyregistryUponconductingfurtherresearchUGstaffcameacrossinformationonacourtrulingfromChicagothatcertainvacantbuildingsdonotneedtofollowtheCityofChicagorsquosvacantbuildingordinanceThisdirectedthesearchtounderstandifothercitieshadexamplesofwhethervacantpropertyregistriesareworkingTodeterminewhetherthismightworkinthecontextofKansasCitystaffbeganvettingideastoseeiftherewereanyaspectsthatconflictedwithCityordinancesTheUGalsoexaminedthefeasibilityofimplementingthenecessarytechnologyandfeasibilityfromtheperspectivesofpropertyownersandUGstaffThisinitialresearchandanalysisphasetookaboutsixmonthsandwasledbytheGeneralServicesDirectoraVISTAinternandtheresearchteamIndevelopingthisRoadmaptheCitymentionedthatitwouldbehelpfultohaveaccesstocasestudieswithdetailedinformationonthepolicysolutionsthatcitieshavetakenandthespecificsofhowtheyhavebeenimplemented
PurposeoftheRoadmap
ThisRoadmapismeanttoempowercitiestotakeownershipofthepolicychangeprocesstoframeproblemsinawaythatcreatesademandforevidenceandtoformlong-termrelationshipswithpoliticalpolicypractitionerandresearcheralliesThisworkisnecessaryResearchfromthefieldofimplementationsciencesuggestssuccessfulimplementationofevidenceintopracticehappenswhencitiestakethetimetoidentifytherightpeopletoworkwithchoosethebestevidencefromavarietyofsourcesandconsiderthebestwaytoadapt
IntroductionandpurposeoftheRoadmap
4
thatevidencetotheirowncontextThroughourinterviewswithcitystaffweobservedthatframingtheproblemclearlyformingadvisorygroupsorsomeothercoalitionandidentifyingandimplementinganappropriatesolutionarenecessaryingredientstosuccessfulpolicychange
ThroughthisRoadmapGovExhopestofacilitateanewwaytohelpAmericancitiesrethinkthehowtheysolveproblemsandaddressthemostsalientpolicychallengesinurbangovernance
IntroductionandpurposeoftheRoadmap
5
UsingtheRoadmap
WhoshouldusethisRoadmap
ThisRoadmapisforanystaffmemberincitygovernmentwhoischargedwithsolvingaproblemorimplementingapolicyAlthoughthestepscanbeusedtochangeexistingprogramsorimplementnewprogramsthefocusoftheRoadmapisonlong-termsustainablepolicychangeWeusethetermpolicytomeanlawsregulationsfundingdecisionsandinterventions
CitystaffwhoarereadingthisguidemaybecomingfromdifferentstartingpointsItisimportanttobehonestaboutwherethatisForinstanceyoumaybelookingfornewsolutionstolongstandingseeminglyintractableproblemssuchasanannualincreaseingunviolenceormaybeyouarefacingnewerchallengesliketheopioidepidemicOrperhapsyouarelookingforadata-drivenwaytoaddressissuesrelatedtoday-to-dayservicedeliverysuchaspermittingorbikelanesThestepsareapplicablewhetheryouareworkingonalargescaleissueoralocalmunicipalissue
YoucanusethisguidewhetheryouareadecisionmakerorfrontlineworkerThroughouttheRoadmaptheresinformationaboutwhenandhowtoengageotherpeopleCollaborationisakeythemethroughthisRoadmapThegoodnewsisyoudonothavetodothisworkalone
TheapproachandthetoolscontainedintheRoadmapshouldberelevantwhetheryouareinastrongmayororcouncil-managerformofgovernmentElevatingthisprocesstothelevelofMayorCityManagerorCityCouncilmaytaketimeandmultiplestepsbutwehopeyouwillfindwaystoincorporatedataandevidenceintoyourday-to-daywork
WhatarethestepsoftheRoadmap
ThechaptersinthisRoadmaprepresentthehigh-levelstepsinvolvedinpolicychangeYouwillnoticethatthesectionsrefertooneanotherandbuildonthepreviousstepsTheRoadmapismeanttobeiterativeyouwillrevisitsomesteps
TherootsoftheproblemsthatcitiesareworkingtosolvearecomplexandmaybethelegacyofhistoricalfederalpolicyorsweepingsocialtrendsAlthoughthisRoadmapdoesnotexplicitlyincludestepsonreviewingthishistoryweencouragereaderstodevelopadeepunderstandingoftheprobleminyourowncitywhileconsideringcurrentstateandfederalcontext
UsingtheRoadmap
6
IninterviewswithGovExonthepolicychangeprocessintheircitiespeoplewasthewordmostcommonlyusedbyintervieweeswhendescribingthevariousstepstoimplementanewpolicyorprogramYouwillnoticethatthisisathemethroughoutthisRoadmapThefirstsectionbeginswithidentifyingtherelevantpeopleThechapteronidentifyingrolesforkeypartnersservesasafoundationforthefollowingstepsThediscussionofevidenceencouragesreaderstoconsiderwaysinwhichpeoplecanbuildanduseevidencetodesignstrongprogramsandinitiativeschangehumanbehaviorassessimpactensureeffectiveuseofpublicresourcesandimprovelives
Meaningfulapplicationofevidencebypolicymakersdoesnothappenwithone-timesharingofcuttingedgeevidenceRathermeaningfulinfluencerequiresalong-termstrategytoformalliancesanddevelopknowledgeofpolicymakersandpolicymakingInadditiontoworkingwiththerightpeopleandunderstandingevidenceandcontextwehopethatRoadmapuserswillformapeernetworkthatsharesnotjustwhichpolicysolutionsexistbutwhatimplementationlookslikeAsonecitywespoketoindevelopingthisRoadmapputitalotofourcitysproblemshavebeensolvedbyotherswejustneedtofigureouthowtoimplementthesolutions
TheexamplesthroughouttheRoadmapareillustrativeofgovernmentsthathaveeffectivelyimplementedtherelevantstepsinthisRoadmapbutpleasenotethatnocitydoesallofthisperfectlyTogetherthestepsinthisRoadmaparemeanttohelpcitiesframetheirproblemscreateademandforevidenceandbuildsustainablecoalitionsformeaningfulpolicychange
UsingtheRoadmap
7
GatherinformationabouttheproblemInyourearliestphaseitiscriticaltodevelopacomprehensiveunderstandingoftheproblemYouwillwanttolearnaboutthepeoplewhoareaffectedbytheproblemyouaretryingtosolveandthosewhocanimpacttheproblemFromthereyouwillgetinsightintotherootcauseoftheproblemtheworkthathasbeentriedinthepastandtheeffectivenessofthoseeffortsTheworkfromthischapterwillserveasthefoundationforactivitiesinsubsequentchaptersYoumayrevisitthesestepsasyouidentifyindicatorsandplanyourimplementationandanalysis
BesuretodedicatetimeandresourcestothisstepItcanbetemptingtoskipaheadtolookingforsolutionsortoassumethatyouhaveanunderstandingoftheproblemasaresultofapublicengagementprocess
TheCityofLewisvilleTXconductedaCitywidepublicengagementprocesstodevelopitsLewisville2025StrategicPlanOneoftheBigMovesoutlinedintheplanwasExtendingtheGreenorexpandinggreenspaceandconnectingrecreationareaswithneighborhoodsTheactionstepstoachievethisgoalincludedcleaningupcreeksandremovingdebristomakethemmoreaccessibleThecleanupswentsmoothlyinseveralneighborhoodsbuttheCitymetwithoppositionfromoneneighborhoodthatpreferredthenaturalhabitatandexpressedconcernabouttheenvironmentalimpactofthecleanupsAccordingtoLewisvilleiftheCityhadmappedallofthestakeholderstodeterminewhoneededtobeatthetableandwhowasmissingfocusinginformationgatheringontheneighborhoodlevelithaveavoidedthisproblem
Startwiththewhothepeople
Establishaprojectteam
BeginbynamingtheprojectteamforthisprocessincludingaprojectleadTheprojectleadshouldbesomeoneinCitygovernmentwhoisrespectedcangalvanizeotherkeypartnersandhasthetimetoleadthisworkThisissomeonewhomayhavetohavedifficultconversationswithCityleadershipdepartmentsandotherfundedpartnersItisnotnecessaryfortheprojectleadtobeapolicyexpertItismoreimportantforthemtohavetherightrelationshipsandpositionwithintheorganizationTheprojectteamshouldincludeindividualswithdiverseperspectivesfrommultipledepartmentsThereshouldbesomeonewithpolicyexperienceontheprojectteamtogroundtheconversationinrealisticactionHavingateammemberwhocancoordinatewithresearchersonrigorousanalysisandtranslatethatworkforothersiskeyAdditionallybesuretoincludefrontlinestaffworkersTheywillhavevaluableinsightbasedontheirinteractionswithresidentsandpolicysolutions
Gatherinformationabouttheproblem
8
havetomakesensetothepeoplewhowillbechargedwithimplementingthemBuildingastrongdiverseprojectteamisimportanttoensurethattheworkdoesnotliveanddiebyoneperson
Identifykeypartners
NextidentifythepeoplewhotheprojectteamcanworkwithontheissueyouarefacingincludingpeoplewithlivedexperienceintheproblemyourcityistryingtosolvedepartmentstaffandsubjectmatterexpertsoutsideofcitygovernmentThisistheconceptbehindbuildwithnotforamovementledbyLaurenellenMcCannthatemphasizescommunityexpertiseandleadershipMappingoutkeyplayersandtheirrolesiscriticaldotoearlyintheinformationgatheringphasesincecoalitionbuildingissuchanintegralpartofthepolicychangeprocessThisprocessmightconsistofstakeholdermapping(describedbelow)communityresourcemappingandculturalmappingInyourmappingremembertoconsiderpotentialalliesandthosewhomaypresentroadblocksalongtheway
PolicymakingisnottopdownorlinearInsomecasescitiesmayhavetosharedecision-makingpowerwithotheractorsForinstanceinmanycitiesthelocalContinuumofCare(CoC)istheleadentityonhomelessnessservicesCoCsapplyforandstewardfederalfundstoaddresshomelessnessandcoordinateservicesamonglocalprovidersAlthoughcitygovernmentsdohavearoleinCoCstheyarenotalwaystheleadagencychargedwithdistributingfundsandthereforetheirroleinhomelessnesspolicymaybelimitedCountygovernmentsoftenhavehealthdepartmentswhilecitiesdonotInthosecasesthebalanceofpowerfavorscountygovernment
TheCityofDentonTXwasrecentlyfacedwithanissueofresidentcomplaintsaboutanewdocklessbikesharethatarrivedintownandisviewedbysomeasexpensivelitterInresponseDentonsCityCouncilwhichwasalreadydiscussingpotentialrulesforbikesharecompanieselevatedthisissueandchargedcitystaffwithidentifyingasolutionPresentlytheCityisreviewingordinancesfromnearbycitiestodeterminehowtheyaresolvingthisproblemandwhatsolutionsmaybefeasibleinDentonThemappingexamplebelowrepresentslocalactorsinDentonwithastakeinthisissuewhohavemultipleperspectivesandlevelsofpower
Gatherinformationabouttheproblem
9
UsethistemplatedevelopedbytheCentreforPublicImpact(CPI)tobrainstormwhoyoumightreachouttoasyouembarkontheroadtopolicychangeRemembertothinkabouthoweachpersonorgrouprelatestotheproblem
BegintodefinethewhatortheprobleminyourCity
TherestofthissectionwillhelpyourefinetheproblemstatementFornowgathersomekeypartnerstoclearlyarticulatewhychangeisneededUsethetemplatebelowdevelopedbyDenverPeakAcademytocapturetheproblemstatementYouwillbuildonthislater
ProblemStatementndashWhyChangeisNeeded(1-2Sentences)
ThroughourinterviewswithcitystaffweheardthatwhendiscussingmunicipalchallengesitcanbetemptingtojumptoasolutionbeforetakingthetimetoreallyunderstandtherootcauseoftheproblemThismakessensesincecityleadersandstaffareresponsiblefordeliveringservicesandbeingresponsivetoresidentswhileaddressingnewproblemsthatariseinthecity
TherootsoftheproblemsthatcitiesareworkingtosolvearecomplexandmaybethelegacyofhistoricalfederalpolicyorsweepingsocialtrendsTohelptheproblemseemlessdauntingitcanhelptobreakthemdownintotherootcausetoidentifyamanageablecomponentoftheproblemUsingafishboneexerciseorfivewhystechniquecanalsohelp
Gatherinformationabouttheproblem
10
tofindtherootcauseoftheproblemandavoidsimplyaddressingsymptomsoftheproblemandrepeatingthemistakesofhistoryToolssuchasjourneymapscanhelpyouunderstandtheexperienceofthosewhoareaffectedbytheproblem
TheCentreforPublicImpactwithsupportfromBloombergPhilanthropiesdevelopedanexercisetosupportcitiesinidentifyingtherightsizedproblemAccordingtoCPIarightsizedproblemisonethatislargeenoughtopotentiallyapplytoothercitiesbutsmallenoughtosolvewithinareasonabletimeframe
AtthebeginningofMayorBerkesadministrationintheCityofChattanoogaTNveteransexperiencinghomelessnesswereaskingtheCityforhelpinfindinghousingTalkingtothispopulationrevealedthatveteranshadtroubleidentifyingaffordablehousingwhichmotivatedtheAdministrationtoworkonasolutionTheproblemofveteransexperiencinghomelessnesswastherightsizebecauseitfocusedononesegmentofpeopleexperiencinghomelessness-veterans-andtheirspecificchallengeinidentifyingaffordablehousing
Developacomprehensiveunderstandingofhowyourcitycurrentlyaddressestheissue
IntheChattanoogaexamplementionedaboveCitystaffdiscoveredthatveteranswerehavingtroublenavigatingtheCitysvariousaffordablehousingprogramsandgettingaccesstothehousingandservicesthattheyneededAtthesametimenewlyelectedMayorBerkewasexploringcurrentandfutureopportunitiestoservethispopulationRatherthancontinuetousefederalfundingforemergencysheltersMayorBerkedirectedcitystafftoprioritize
Gatherinformationabouttheproblem
11
HousingFirstwhichvaluesconnectingpeoplewithpermanenthousingovertransitionalhousingandsupportiveserviceafterspeakingwithrepresentativesattheUSDepartmentofHousingandUrbanDevelopment(HUD)
ThisexaminationofwhattheCityiscurrentlyfundingandthesourcesofthatfundingwasaturningpointinChattanoogaUnderstandingthisproblemfromasystemsperspective-includingpoliciesandfundingstreamsatmultiplelevelsofgovernmentsocialnetworksthroughwhichinformationisoftenspreadtechnologythebuiltenvironmentofmarket-rateandpublichousing-helpedthenewMayoranalyzeexistingeffortsandleadstaffandotheractorsinthesamedirection
Begintogatherthedata
TotrulycapturethesuiteofservicesthatyourCityisprovidingaskdepartmentstafftolisttheprogramsrelatedtoyourproblemstatementthatyourcityrunsorfundsWhenlistingtheprogramscapturedetailssuchas
programnamethenameofthedatasetthatisrelevanttoeachprogramadescriptionofthedatasetthesourceofthedatasetifthedataisinternalorexternalwhetherthisdataisalreadybeingcollected
ThisinformationwillformthebasisofyourinventoryofadministrativedataordatasetsthatarekeptbygovernmentsforoperationalpurposesThismayincludecallcenterrecords(311911)taxdatapropertyassessmentsinspectioninformationsocialservicerecordsetcAtthispointlistallrelevantadministrativedatasetsYouwillprioritizewhichdatasetstogathermoreinformationonlaterwhenyouaredevelopingindicators
Asampletemplatethatcanbeusedtocapturethisinformationisbelow
ProblemStatement____________________________________________
Gatherinformationabouttheproblem
12
ProgramName
DatasetTitle
DatasetDescription DataSource
Departmentorexternalpartner
DoestheCityalreadycollectthisdata
Whatisthenameoftheprogram
Human-readablenameofthedataset
Shouldbeunderstandablebynon-technicalusers
Whatistheoriginalsystemapplicationfilethathousesit
Departmentorexternalpartnerthatmaintainsthedata
Isthisdataalreadycollected
TogatherthisinformationconsiderassigningonepointpersontosurveyorinterviewcitystaffSpeakingwithfrontlineworkersstaffwhoworkwithfederalgrantsdepartmentheadsandeveryoneinbetweenwillensureadeepunderstandingoftherelevantprogramsanddatasetsDuringthecourseoftheseconversationsbesuretoaskcitystaffwhichorganizationstheyworkwithexternallyandcapturethatinformationaswelleveniftheyarenotrecipientsofcityfundingInadditiontoconversationswithexpertsincitygovernmentyourcitysbudgetdocumentsareanothergreatsourceofinformation
Collectdataandobservetrends
Nexttrytogainanunderstandingofwhytheproblemstillexistsifithasbeensomethingyouhaveworkedonforawhileorwhyitisemergingnowifitisnew
AnalyzingadministrativedatacandemonstratepasteffectivenessofinitiativesrelatedtoyourproblemstatementUsingthedatasetinventorythatyoubeganintheprevioussteptalktodepartmentstaffaboutwhichdatasetsaremostrelevanttotheproblemandgatherafewyearsworthofhistoricaldataBegintolookfortrendsinthisdatadiscussingpotentialreasonsforthosetrendswithrelevantstaffCanthetrendsbeexplainedbyseasonalitychangesinfundingstaffingfluctuationsorsomeotherreasonBasedonthehistoricaldataanalysiscreateabaselineagainstwhichfuturechangescanbeobserved
OtherdatasetscanhelpprovideadditionalcontextForinstanceifthenumberofpeopleexperiencinghomelessnessinyourcityisincreasingitmaybehelpfultotakealookathousingaffordabilityIfyoudonothavethedatayouneedconsideraproxymeasureAsanexamplehousingisoftenconsideredaffordableifitcosts30orlessofahouseholdsincomeButifhouseholdincomedataisnotavailableatthecitylevelAreaMedianIncome(AMI)canbeahelpfulproxy
Gatherinformationabouttheproblem
13
QualitativedatacanalsohelpanswerquestionsaroundtheeffectivenessofpreviouseffortsTrytoobtainavarietyofperspectives--bothmainstreamandoutliers--fromnetworksbeyondyourowntoprovideyouwiththegreatestvantagepointforevaluatingpotentialimpactUnderstandinghowthoseeffortswerereceivedbydifferentpartsofyourcommunityandiftheremaybeelementsofprevioussystemstopreservewillbecrucialtothesuccessofanypolicychangeThisinformationmaybeobtainedthroughfocusgroupsinterviewsandsurveysFormoreinformationonhowtousethesequalitativeresearchtoolspleaseseetheseresourcesdevelopedbytheAgencyforHealthCareandResearchandUrbanInstitute
FinallytakealookatnationaldatatoseewhereelsethisproblemmightexistThisresearchmaybehelpfulinlaterstepswhenyouarelookingforpromisingpoliciesinothercities
Puttingitalltogether
UsethetemplatebelowdevelopedbyDenversPeakAcademytocaptureyourrefinedrightsizedproblemstatementandcurrentandfuturestatesWorkingthroughthisexercisewithpartnersishelpfultogainbuy-inTheresponseswillbeimportantwhengettingotherpartnersonboard
WhyChangeIsNeeded
Yourright-sizedproblem
CurrentState
Analysisoftrendsindata
FutureState
Ideallywhataretheoutcomesyouhopetoachieve
Checklist
Thekeystepstogatheringinformationarebelow
Atthispointfocusontheproblemratherthanjumpingtosolutions
Listentopeoplewhoareaskingforhelpandtoadvocateswhoareraisingissues
Beasspecificaspossibleinidentifyingtherootcauseandhoninginonthepartoftheproblemthatyourcitycanaddress
Identifytrendsbasedonmultiplesourcesofdataincludinghistoricaladministrativedataproxymeasuresqualitativedataandnationalmeasures
Gatherinformationabouttheproblem
14
Captureyourproblemstatementtheresultsofyouranalysisanddesiredoutcomes
Gatherinformationabouttheproblem
15
StrategicallyengagekeypartnersPolicytheoryresearchoftenfocusesonincreasingpolicymakersaccesstoevidenceasawaytoavoidtakingshortcutslikemakingdecisionsbasedonemotionImplementationsciencesuggeststhatincreasingaccesstoevidenceisnotenoughandthatpolicychangeshouldbeexaminedinthewidercontextofdebatecoalitionformingandpersuasionMeaningfulpolicychangerequiresldquoalong-termstrategytoformalliancesanddevelopknowledgeofpolicymakersandpolicymakingratherthanusingevidenceinpolicymakingonanadhocorone-offbasis
UnderstandingnetworksandsystemsisimportanttodevelopsustainablepolicychangeThissectionismeanttohelpreaderstakethestepstobuildengagementstrategiesforpartnersthatrepresentdifferentsystemsandnetworksEngagementshouldbeanongoingprocessofconnectingpeopletooneanotherandtoanissueEngagementshouldnotbeinnameonly-wehaveallexperiencedengagementactivitiesthatgivepeopleaspecificroletoplaythatleadstoaneatpredictableoutcomeAccordingtotheBuildWithmovementundervaluingandunderinvestinginengagementstrategiescanleadtoadisconnectofcertainindividualsorgroupsandmistrustofgovernmentWhendevelopingengagementactivitiesrememberthathowpeopleengageandwhethertheyengagehastodowithhowyoustructureopportunitiesforengagement
Conveneacoreadvisorygroup
AnadvisorygroupcomprisedofpeoplefrommultiplesectorscanbehelpfulasyouconsiderthelocalsocialenvironmentalandpoliticalcontextinwhichyouintendtointroducepolicychangeRevisityourstakeholdermapanddeterminewhowouldbebesttoserveonthisgroupTomakesureyouareconsideringtheproblemsandsolutionsthroughanequitylensincludepeoplewithlivedexperiencewhohaveexpertiseintheeffectivenessofsystemswhetherthosesystemsfunctionasintendedandhowtheymayimpacthumanbehaviorYoumayalsowishtoincludesomeonewithexperienceworkingwithdataandanalysis-thispersoncanhelpdeterminethetypeofdatathatmaybereadilyavailableandusefultocollectandcanalsohelpplanandshapeyouranalysistofigureoutwhichprogramsareeffectiveBesuretoinvitefront-lineworkersaswell-itscriticalthatwhateversolutionscomeoutofthisprocessaremeaningfultothosewhowillberesponsibleforimplementingthem
BringingpeopletogethercanbechallengingbutthereareafewwaystosetandmanageexpectationsFirstmakesurethatthegroupisawareoftheproblemstatementcurrentstateandfuturestate-thiscanhelpstarteveryoneoffonthesamepageNextspend
Strategicallyengagekeypartners
16
sometimegettingtoknoweachotherrsquosworktolearnabouteveryonesstrengthsThissharedunderstandingcanhelpasyouclarifyrolesandresponsibilitiesofeachpartnerinthegroup--thisisimportanttoavoidfinger-pointinginthefutureandtoensurethatnoneofthetasksslipthroughthecracksInsomecasestherearemanyactorswithdifferentresponsibilitiesasinthecaseofBaltimoreCitys38-memberYouthFundTaskForcewhichischairedbynonprofitandcorporatepartnersandincludesrepresentativesfromthenonprofitandfaithcommunitiesstudentsthebusinesssectorlocaluniversitiesCityCouncilandtheMayorsOffice
OncerolesandresponsibilitiesareclearyoumayconsiderdevelopingacharterguidelinesortermsofreferencetosetexpectationsforthegroupBesuretodistinguishthisfromotherboardsandcommissionsthatyourcitymayhaveNextyoumaywishtoestablisharegularongoingmeetingscheduleBesuretosendagendasandmeetingmaterialsaheadoftimeandclarifytheobjectiveofeachmeeting
InSeattletheCitydevelopeditsHousingAffordabilityandLivabilityAgenda(HALA)torespondtorisinghousingcostsandhomelessnessThisisatrendinotherwestcoastcitiesaswellwhichmaybeexplainedbyaninfluxofnewresidentswhoareattractedbytheemergenceoftechjobstheattractionoftheoutdoorlifestyleortherecentpassageofprogressivepolicieslikelegalizedgaymarriagea$15minimumwageandlegalizationofmarijuanaSeattlebroughttogetheramulti-stakeholderadvisorygroupcomprisedoflocalhousingexpertsrentersandhomeownersdevelopersandgovernmentofficialstoshareinformationabouttheirneedsanddeveloprecommendationsbasedontheseneedsTheinterestsandmotivationsofthisgroupwerevariedandittookabouttenmonthstofindcommongroundTheseriesofrecommendationsthattheadvisorygroupcreatedismeanttoserveresidentswithaspectrumofneeds
Buildaprocessforcommunityparticipation
Ofcoursenoteveryonecanbeincludedinanadvisorygroupandadvisorygroupmembersmaynotnecessarilyrepresentthevariedperspectivesoftheirpeers-forinstancenoteveryhomeownerhasthesamesetofneedsTogatherinputfromawiderswathofthepopulationitcanbehelpfultobuildaprocesstoinvitethepublicatlargetosharetheirideasCitiesdothisinavarietyofwaysfromonlineconversationsandsurveystoin-personmeetingsRememberthatbeingintentionalaboutgatheringtheviewpointsofpeopleismoreimportantthanusingthelatesttechnologyandtools
TheinsightgeneratedbycommunityparticipationexercisescanbeincludedinyourreviewofevidencewhichisdiscussedinasubsequentchapterInWestVirginiaastatethathasbeenhithardbytheopioidepidemicstatehealthofficialsdraftedanopioidresponseplanthatwasinformedbypublicengagementaswellaslocalandregionalsubjectmatter
Strategicallyengagekeypartners
17
expertsThepublicwasinvitedtopresentcommentsonideasfordecreasingoverdoseearlyinterventiontreatmentandrecoveryPublichealthandmedicalfacultyfromWestVirginiaUniversityMarshallUniversityandJohnsHopkinsUniversityprovidedresponsestothecommentsTheresultwasanOpioidResponsePlanthathealthcommissionerRahulGuptapresentedtoGovernorJimJustice
AsamplespectrumtoplanyourowncommunityparticipationprocesscreatedbyLaurenellenMcCannisbelow
SourceLaurenellenMcCann
Community-basedparticipatoryresearchisanotherwaytoworkwiththecommunitytoimproveoutcomesCBPRmeansthecommunityisinvolvedinallaspectsoftheresearchprocesswhichisofteniterativeInCBPRtraditionalresearchersandcommunitymembersshareknowledgeaswellasresourcesandcredit
CBPRInActionInSanFranciscoanenvironmentaljusticenonprofitteamedupwiththeCitysDepartmentofPublicHealthandanoutsideevaluatortounderstandtherootcausesofobesityandfoodinsecurityinlow-incomeneighborhoodsandidentifysolutionsThroughresearchmethodssuchasinterviewingresidentsandstoreownersdiagrammingstoreshelvesGISmappinganeconomicanalysisandreviewofsimilarpoliciesinnearbycitiestheteamobservedthatthestoresthatresidentshasveaccesstowerestockedwithlesshealthyfoodsandthatresidentswantedmorehealthyfoodandlessalcoholandtobaccoThisprojectresultedinlocalandstatelegislationtoencouragestorestostockfreshproduce
Strategicallyengagekeypartners
18
Developrelationshipswithsubjectmatterexperts
OnewaytokeepsubjectmatterexpertsengagedistoinvolvetheminthecommunityparticipationprocessasdescribedaboveIntheWestVirginiaexamplesubjectmatterexpertsrespondedtoandsynthesizedthepubliccommentsOtheropportunitiesaredescribedbelow
AdvisoryAcitydepartmentheadMayororstaffmembermaydevelopaone-on-oneadvisoryrelationshipwithuniversityfacultymemberoracollaborativebetweentheCityandUniversitytoaddressaparticularproblemAgreatplacetostartistolookforrelevantpracticecenterswithinyourlocaluniversityandreachingouttosubjectmatterexpertsInadditiontotheirexpertiseontherelevantpolicytopicheorshemayalsobeabletoidentifyotherexpertsandevenprovidesupportonhiringtherighttalenttoleadcertainprogramsAnadvisoryrelationshipwithauniversitysubjectmatterexpertcanalsohelpwhencitiesareconsideringbringinginnewprogramsTheseexpertsareoftenfamiliarwithprogramsthatarebasedonnationalmodelsandcanprovideinsightintoaprogramsoriginandlineageeffectivenessinotherplacesandthepotentialforimplementationinyourcityAcademicscanalsohelpidentifyindicatorsbasedongovernmentindicatorsorliteratureFinallyrelationshipsbetweenpolicymakersandresearchersprovidethemostbroadlyeffectiveroutetopullingresearchintopolicy
ResearchandevaluationAcademicexpertsmaysupportcitiesinevaluation-indeedevaluatingcityinitiativesisatraditionalmodelofcityuniversityrelationshipsSharingtheinformationyoucollectedintheGatherInformationphasecanhelpresearchersunderstandyourworkandgoalswhichinturnwillstrengthenthesupportthattheycanprovideDiscussingdetailsaboutyourinterventionandevaluationplanatthebeginningoftheprocessmeansthataresearchpartnercanhelpwiththeevaluationdesignanddatacollectionForinstancetheWilsonSheehanLabforEconomicOpportunitieswalkscitiesthroughconsideringthefollowingquestionsbeforepartneringonanevaluation
1 Isthereaclearpolicyprogrammodelwithobjectivescreeningcriteria
2 Isitpossibletoevaluatewhathappensbothwiththepolicyprogramandintheabsenceofthepolicyprogram
3 Arethereoneormoremeasurableoutcomesofinterest
4 Arethereenoughpeopleforrigorousquantitativeanalysis
5 Whatdatawillbenecessaryforthestudyandwhataretheprocessesforaccessingthesedatasource(s)
6 Isthereacompellingreasontoevaluate
Strategicallyengagekeypartners
19
InadditiontoassessingreadinesstoconductanevaluationresearcherscanhelpyouhoneinonthepurposeoftheevaluationTheWorldBankdescribesseveralpotentialusesofevaluationwhicharedefinedhereInbriefevaluationcanhelpdecisionmakers
makeresourceallocationdecisionsrethinkrootcausesofproblemsidentifynewproblemsdecideonabestalternativedeterminewhetherprogramsorpoliciesarebeingimplementedwithfidelitydeterminewhetherprogramsorpoliciesarehavingthedesiredimpact
Researcherscanalsoprovidesupportinsharingtheresultswiththescientificcommunitythroughpapersandconferenceswhichisvaluableincreatingapeernetworkofevidence-basedcitypractitioners
TrainingandtechnicalassistanceUniversitypartnersespeciallythoseatpracticecenterscanbehelpfulindevelopingorupdatingtrainingcurriculumtrainingthetraineranddeliveringtrainingtocitystaff
Strategicallyengagekeypartners
20
ReviewtheevidenceandcollectdetailsOnceyouhaveathoroughunderstandingoftheproblemdataandactorsyouwillneedtoconductyourownresearchonavailableevidenceandlearnaboutthelandscapeofpolicyoptionsYourgoalwillbetodevelopanaccessiblelistofinformationaboutpoliciesthathavehelpedsolvesimilarproblemsinotherplacesThischapterwillguideyouthroughthestepsofpreparingyourreviewersunderstandinglevelsofevidenceandcollectingthedetailsonevidencethatyouwillneedfordiscussionwithdecisionmakersinyourcity
Selectandprepareyourreviewers
AssignapersonorteamtoreviewrelevantinterventionsthathavedemonstratedsuccessinotherjurisdictionsTheirgoalwillbetoproduceareviewofalloftherelevantprogramspoliciesorinitiativesthatyourteamcouldconsiderimplementingTheywillbeconductingaliteraturereviewsupplementedbyinterviewswithexpertsandpeerswhohaveinformationaboutpotentialprograms
SincethepeopleyouassigntothistaskareprovidingthebaselineofpotentialoptionstoimplementitisimportantthatyouchoosepeopleyouareconfidentwillbethoughtfulandthoroughintheirworkItisalsocriticalthattheyareawareoftheimportanceofsourcecredibilityEnsureyourreviewershavefamiliarizedthemselveswithrulesforcriticallyanalyzinginformationsourcesbeforetheygetstartedIftheyareincludingnewsarticlesaspartoftheirsearchtheyshouldalsobeawareofmethodsforevaluatingnewscredibility
AskyourreviewerstodevelopaplanandtimelinetocompletethereviewThisstartswithsolidifyingagoverningresearchquestionOnewaytoknowyouarecreatingadetailedresearchquestionistousetheacronymPICOremindingyoutoincludereferencetothepopulationintervention(ifyouhaveoneinmind)comparisonandoutcomeelementsinyourresearchquestionForexampleifyouknowthatyouhaveapopulationofteenagerswhoarehomelessyouhavelearnedfromyourstakeholdersthatmanyoftheseteenagersidentifyasLGBTandyouhaveheardsomethingabouttheapproachofcreatingdrop-incentersfordeliveringsupportyoucouldask
ForLGBTyouthfacinghomelessness(population)howeffectivearedrop-incenters(intervention)relativetostandardsocialservices(comparison)atsettlingtheminpermanenthousing(outcome)
Ifyoudonothaveaninterventioninmindyoucanleaveouttheinterventionandcomparisonandsimplyask
Reviewtheevidenceandcollectdetails
21
HowcanwehelpLGBTyouthfacinghomelessness(population)obtainpermanenthousing(outcome)
OnceyourreviewershavetheirresearchquestionsusethequestionstocreatealistofkeywordstheycanuseintheirliteraturesearchThiswillhelpkeeptheircollectionofarticlesreasonablyfocusedHavethemdescribewhattheirsearchprocesswillbeincludinglistingthedatabasestheywillreviewandtheirplanforstoringtheinformationtheywillcollect(Herearesomeadditionalsuggestionsforstructuringareviewplan)
WhatyourreviewerswillbedoingisalightweightversionofarapidreviewTotheextentthattheywouldliketodevelopamethodologyforspecificsearchprotocolsincludingcriticalappraisalprocessestheycancertainlygoaheadanddoso
Usedifferentlevelsofevidenceforthemostcompletepicture
AsdiscussedintheWorkingwitharangeofevidencechaptertheNestaStandardsofEvidenceframework--reproducedbelow--ishelpfultounderstandthelevelofconfidencethataprogramorinterventionishavingapositiveimpactThisapproachletsyouputinterventionsthatyouhaveheardaboutlocallybutwhichhavenotbeenstudiedtogetherwiththosewithademonstratedcausaleffectThisishelpfulsinceanumberofpoliciesprogramsandpracticesthatyouwillwanttoconsidermaynotyethavemuchofaresearchrecord
Reviewtheevidenceandcollectdetails
22
NearlyanypieceofinformationaboutaninterventioncanbeclassifiedwithinthestandardsofevidenceForexampleifsomeonehasacrediblehypothesisabouthowaninterventioncouldproducebenefitthistypeofevidencewouldbeclassifiedasLevel1ThistypeofevidencecouldincludeinterventionsinothercitiesthatyoureadaboutinthenewsorhearaboutataconferenceForinstancewhenUSDAallowedschoolsandschooldistrictstoprovidefreebreakfastandlunchtoallstudentsthroughtheCommunityEligibilityProvision(CEP)manydistrictsimplementedthisoptionevenbeforetherewasevidenceofeffectivenessThiswouldfitintoLevel1sinceitbuildsonthelogicthatprovidingaccesstomealswilldecreasefoodinsecurity
YoumayalsofindpiecesofinformationthatqualifyasLevel2evidenceForinstanceasmoreschooldistrictsimplementedCEPprincipalstestifiedaboutpositivechangestheyobservedinstudentssuchasamorepositivedemeanorthatcanleadtobetteracademicperformanceDataalsoshowedthatthestudentsdescribedwerereceivingmoremealsthanbeforetheprogramsimplementationHoweveritwastoosoontobuildevidencetoshowacausalrelationshipbetweentheincreaseinfreemealsandincreasedacademicperformance
ldquoLower-levelrdquoevidencecanconsistofnuancedbutvaluableelementsthatrelatetoaninterventionrsquospotentialfitwithacommunityQualitativeinformationsuchastestimonyfrompeoplewhohaveexperiencedaninterventionandthosewhohavefacilitatedorimplementeditwillalwaysbeusefultoconsiderandprovidesdetailedlocalcontext
Experimentalstudies--describedstartingatLevel3withcontrolledexperimentsandgoingthroughLevel5withfullyfleshedoutprotocolsandprocessesforreplication--stilloccupythehigherlevelsofthisdescriptionofevidenceThisprioritizationshowsthevalueofrigorousstudyforevaluatingtheimpactofaparticularpolicyorprogramratherthanotherfactorsthatcouldbeinfluencingoutcomesofinterestRememberthatmultipletypesofevidencecancomplementrigorousimpactevaluationsandgivecityleadersadditionalinsightsandbroaderperspective
Accessinformation
Encourageyourreviewerstobegintheirsearchforpromisingpracticesingovernmentandnon-profitclearinghousesandinacademicpolicyjournalsWorkpublishedinthesesourceshasalreadybeensubjectedtoahighlevelofprofessionalscrutinyandgenerallycontainsagreatdealofdetailwhichmeansthatyourreviewersaremorelikelytobeabletopullouttheinformationyouneedtomakeagooddecision
Clearinghouses
Reviewtheevidenceandcollectdetails
23
Herearesomeoftheclearinghouses(collectionsofprogramevaluations)forevidence-basedpoliciesandprograms
ResultsFirstClearinghouseDatabaseThePewCharitableTrustsCollectstogethereightotherclearinghousestoprovideinformationoninterventionsinthefollowingcategoriesadultcriminaljusticeeducationchildwelfarehealthjuvenilejusticementalhealthsubstanceabuseandyouthdevelopmentIndividualclearinghousesincludedinResultsFirstare
BlueprintsforHealthyYouthDevelopment-foryouthdevelopmentandviolenceprevention
CaliforniaEvidence-BasedClearinghouse-forchildwelfareinterventions
TheLauraandJohnArnoldFoundationrsquosSocialProgramsThatWork-forsocialservicesprograms
TheUSDepartmentofEducationrsquosWhatWorksClearinghouse-foreducationinterventions
TheUSDepartmentofHealthandHumanServicesrsquoResearch-TestedInterventionPrograms(RTIPs)-forcancercontrolinterventions
TheUSDepartmentofHealthandHumanServicesrsquoTeenPregnancyPreventionEvidenceReview(TPPEvidenceReview)-forsexualriskbehaviorandsexualhealthoutcomes
TheUSDepartmentofJusticersquosCrimeSolutionsgov-forcriminaljusticejuvenilejusticeandcrimevictimservices
TheUSSubstanceAbuseandMentalHealthServicesAdministrationrsquosNationalRegistryofEvidence-basedProgramsandPractices(NREPP)-substanceuseandmentalhealthinterventions
TheUniversityofWisconsinPopulationHealthInstituteandRobertWoodJohnsonFoundationrsquosCountyHealthRankingsandRoadmapsWhatWorksforHealth(WhatWorksforHealth)-healthbehaviorsclinicalcaresocialandeconomicfactorsandthephysicalenvironment
UKWhatWorksCentresTheBritishgovernmenthascharteredanetworkofsubject-specificorganizationswhichcollectsynthesizeandreportoutevidenceontheeffectivenessofpolicyprogramsandpractices
NationalInstituteforHealthandCareExcellence(NICE)-healthandsocialcare
Reviewtheevidenceandcollectdetails
24
SuttonTrustEducationalEndowmentFoundation-educationalachievement
CollegeofPolicingWhatWorksCentreforCrimeReduction-crimereduction
EarlyInterventionFoundation-earlyintervention
WhatWorksCentreforLocalEconomicGrowth-localeconomicgrowth
CentreforAgeingBetter-qualityoflifeforolderpeople
WhatWorksCentreforWellbeing-wellbeing
WilsonSheehanLabforEconomicOpportunities(LEO)UniversityofNotreDameProvideslinkstotheirevaluationsofinterventionsincriminaljusticeself-sufficiencyeducationhealthandhousingandhomelessness
ClearinghouseforLaborEvaluationandResearchUSDepartmentofLaborProvidesresearchonworkforceandlabor-relatedpoliciesandprograms
J-PALEvaluationsAbdulLatifJameelPovertyActionLabStudiesonpoverty-reducinginterventionsfromaroundtheworld
CampbellSystematicReviewsCampbellCollaborationSystematicreviewofpolicyinterventionsbyscholarlycoordinatinggroupsinsevenissueareascrimeandjusticeeducationinternationaldevelopmentmethodssocialwelfareknowledgetranslationandimplementationanddisability
Benefit-CostResultsWashingtonStateInstituteforPublicPolicyAWashingtonState-charteredprogramtoestablishthelikelihoodofrelativecostsandbenefitsforprogramsacrossthefollowingareasjuvenilejusticeadultcriminaljusticechildwelfarepre-Kto12educationchildrensmentalhealthhealthcaresubstanceusedisordersadultmentalhealthpublichealthandPreventionworkforcedevelopmentandhighereducation
EffectiveInterventionsCentersforDiseaseControlandPreventionRepositoryforevidence-informedHIVpreventioncareinterventions(withstepsonhowtoimplement)
Journalsandindexes
OnceyouhavereviewedrelevantinformationaccessiblethroughclearinghousesitisworthtakingalookatindividualstudiespublishedasscholarlyarticlesYourreviewerswilldothisbysearchingarticleindexesandacademicjournalsUnfortunatelyanumberofusefulacademicjournalsandindexingplatformsmaynotbegenerallyaccessibletoyourreviewersifyoudonothaveasubscriptiontothemToaddressthisconsiderpartneringwithalocal
Reviewtheevidenceandcollectdetails
25
collegeoruniversitytoprovideyouwithalistofarticlesfromthefollowingsources(Thisisalsoanexcellentopportunitytoseekresearchsupportfromanyacademicstakeholdersyouidentifiedpreviously)
Indexes
GoogleScholarincludesmostpeer-reviewedonlineacademicjournalsandbooksconferencepapersthesesanddissertationspreprintsabstractstechnicalreportsandotherscholarlyliteratureincludingcourtopinionsandpatentsItallowsyoutospecifyapublicationrangeinyoursearchandprovideslinkstolikelysourcesforalllistedarticles
PolicyFileIndexisauniqueresourceforUSpublicpolicyresearchinthatitgrantsusersaccesstotimelyupdatedinformationfromover350publicpolicythinktanksnongovernmentalorganizationsresearchinstitutesuniversitycentersadvocacygroupsandotherentities
PolicyArchiveworkstocollectanddisseminatesummariesandfulltextsvideosreportsbriefsandmultimediamaterialofthinktankuniversitygovernmentandfoundation-fundedpolicyresearch
Anothersourceforhigh-qualityresearchisthereportsoftheCongressionalResearchServiceYoucanfindtheirreportsaggregatedhere
Specificjournals
Oftenthemostusefulacademicjournalsforunearthingpolicyorprogramreviewswillbeineachspecificsubjectarea--thatisineducationjournalsforeducationpoliciesYoucanuseageneralindexlikeGoogleScholartobegintolearnwhichjournalsaremostoftenusefulforyoursearchHoweveryoucanalsosearchthefollowingtwogeneralistpolicypublicationsforissuesandarticlesrelatedtoyourtopic
PublicAdministrationReviewJournalofPolicyAnalysisandManagement
Newssourcessubjectmatterexpertsandinterviews
AlthoughitisvaluabletoknowwhathasbeenexploredthroughascholarlyresearchframeworkitisalsousefultolearnwhatkindofinterventionsarebeingtriedinthefieldrightnowandhowpeoplefeelaboutthesetrialsSearchsubscriptionnewsaggregatorslikeLexisNexisorfreenewsaggregatorslikeGoogleNewsorYahooNewsforyourissueareaplusthewordpolicyorprogramtoseewhatkindsofinterventionsarebeingcoveredinthenews
Reviewtheevidenceandcollectdetails
26
DonotforgettocontacttheadvisorygroupmemberswhowerechosenfortheirsubjectmatterexpertiseInadditiontoprovidinggoodmaterialthroughinterviewsthemselvestheymayhaveideasaboutothersubjectmatterexpertsyoushouldcontacttoaskaboutpotentialinterventions
Peercitiesandnetworks
InadditiontoreadingabouttheworkofothercitiesitisidealifyouareabletospeakwithpeoplewhoareinvolvedwithcurrentimplementationsofinterestinginterventionsInyourcityyoulikelyhaverelationshipswithyourneighborsbutitcanbehelpfultolookoutsideofyourregionforsolutionsaswellTofindcounterpartsinothercitieswhoareworkingonaprogramyouwouldliketolearnmoreaboutyoucancontactstaffinintermediaryorganizations(likeGovExoryourprofessionalassociations)whomightbeinapositiontoputyouintouchimmediatelyorwhomightseeanothergoodrouteforsettingupaconversationHereisalistofnetworksthatmaybehelpful
InternationalCityCountyManagementAssociation(ICMA)NationalAssociationofCounties(NACO)NationalLeagueofCitiesUSConferenceofMayorsWhatWorksCities
Collectthedetails
Onceyourreviewershavefoundevidenceonpossiblepoliciesorprogramstoimplementtheyshouldcapturekeydetailsthatwillhelpdecisionmakersunderstandthepotentialeffectivenessandapplicabilityofeachpotentialintervention
Foreachsourcethathasprovidedinformationyouwanttoconveyusethetemplatebelowtocaptureasmanyofthefollowingdetailsaspossible(Youdonotneedtocompletethisforsourcesthatprovideduplicativeinformationorareotherwisenotthatuseful)
Programorpolicyname (egldquoHousingFirstrdquoorldquoblightedpropertiesregistryrdquo)
Sourcetype (egjournalarticleinterviewsubjectmatterexpertstudyassociationcommunityparticipationprocess)
Sourcedetails (egarticletitleandpublicationnameandtitleofinterviewee)
Evaluationmethodology
(egrandomizedevaluationcomparativeevaluationoftwodifferentgroupsbeforeafteranalysis)
Evidencelevel (egLevel12345)
Programorpolicy
Reviewtheevidenceandcollectdetails
27
description
Programorpolicygoal (1sentencecapturingtheprincipalaimoftheintervention)
Existingimplementations
(WherehastheinterventionbeentriedListasmanysitesasyouareawareof)
Characteristicsofimplementationsites
(ForeachimplementationsiteofferbasicgeographicIDmdashcityandstateThenyouwillwanttocollectthedetailswhicharerelevanttoyourdecisionmakersrsquoconsiderationsaboutwhatmakesalocationldquosimilartordquoorldquodifferentfromrdquoyoursituationYouwillatleastwanttoincludeforeachimplementationsitePopulationnumberPopulationdensity(ruralurbansuburban)Socio-economiccharacteristicsBeforestartingaskyourdecisionmakersaboutanyparticularcharacteristicsofprogramimplementationsitestheywouldwanttoknowbeforeseekingtoreplicate)
Characteristicsofimplementationpartners
(Anyavailabledetailsaboutorganizationsinstitutionsorpeoplewhowereinvolvedwithimplementingtheintervention-egindependentchildcarecenterpublicelementaryschoolcitypublichealthnurses)
Characteristicsoftargetpopulationeligibleprogramparticipants
(Anyavailabledetailsaboutpeoplewhoarethebeneficiariesoftheprogramorotherwiseincludedoraffectedbytheintervention-egnumberofprogramparticipantscategoriesofpropertyownersaffected)
Policyorprogramduration
(HowlonghastheinterventionbeeninplaceineachimplementationsiteHasitendedandifsowhy)
Whatbenefitscamefromeachimplementation
(Anyobservedpositiveeffectsoftheinterventionincludingeffectsize)
Statisticalestimatesofinterventioneffectiveness
(Anyavailablestatisticalestimatesoftheinterventionrsquoseffects)
Whatdrawbackswereobserved
(Anyavailabledetailsaboutunintendedorunavoidablenegativeeffectsoftheintervention)
Programorpolicycosts
(Anyavailabledetailsaboutprogramcostsperimplementation)
Reviewtheevidenceandcollectdetails
28
Reviewtheevidenceandcollectdetails
29
DecideonapolicytoimplementbasedonlocalfactorsTogetthecriticalbuy-infromleadershiptoimplementanewpolicyorprogramtheprojectleadshouldscheduleameetingwithdecisionmakerstopresenttheevidencesurroundingthepolicyoptionscapturedinthetemplateDependingonyourinsightintoyourcityleadershipswillingnesstomakechangeyoumaywishtosharetheresearchyouhaveconductedontherootcausesoftheproblemandsomeofyouranalysisontheimpactofcurrenteffortstodate
Considertheapplicabilityofanapproach
AsdiscussedinthepreviouschapterpolicymakersoftenlookforevidenceofeffectiveprogramsinothercitiesbutoftenthequestionarisesofhowgeneralizableaprogramisfromonecitytoanotherTheAbdulLatifJameelPovertyActionLab(J-PAL)suggeststhatinreviewingevidencefromanotherlocationitishelpfultoidentifythespecificmechanismthatwasused(deliveryofprogramorserviceinformationsharingetc)identifythelocalconditionsthatwereinplaceobservethebehaviorchangethatresultedfromthemechanismandfinallytounderstandlocalcontextandconditions(legalorcostconstraintsculturalnormsetc)Thissetofpresentationelementscanbedrawnfromthelistofdetailscollectedintheresearchtemplate
EducationpolicymakersinChicagoILusedfindingsfromprogramsinKenyaandIndiainwhichchildrenwhohadfallenbehindtheirpeersweregiveneducationalsupporttargetedtotheirleveloflearningThemechanismusedwastargetedteachingandthereweresimilarconditionspresentinallthreecountriesAlthoughsomestudentshadfallenbehindinthecurriculumteacherswereincentivizedtoteachgrade-levelmaterialsConsiderationsofhowthelocalcontextinChicagomightaffectimplementationofatargetedteachingprogramincludedunderstandinghowtheneighborhoodsthatthestudentswerefrommayhaveaffectedtheireducationaloutcomesandthehighcostofteachinginsmallgroupsInChicagoMatchEducationprovidedtwo-on-onetutoringwhichmatchedstudentswithwell-educatedindividualswhohavenothadformalteachertrainingforalower-costmodelTheresultswerepositiveStudentswhoweretutoredperformedbetterinmathandweremorelikelytopasshighschoolclasses
ThegeneralizabilityframeworkproposedbyJ-PALisbelowUsethisasatemplateasyouconsiderpotentialpolicysolutions
Decideonapolicytoimplementbasedonlocalfactors
30
SourceTheGeneralizabilityPuzzle
OnceyouhavenarrowedthepolicyoptionsdiscussthequestionsbelowforeachoptionsyouareconsideringThisstepiscriticalasunderstandinglocalcontextissoimportantinsuccessfulpolicyimplementation
CommunityContextQuestionnaire
Localcontext
DoesthisoptionmakesenseinourcommunityIsthisaonesizefitsallapproachIsanyparticulargroupabsentfromthisapproachIfsoisthatok
Youranswers
Effectiveness
WhatwouldweexpecttoobserveifthepilotisaseffectivehereasinCityX
Youranswers
Timing
DoesthetimingmakesenseIsthistherighttimetoimplementthispolicychange
Decideonapolicytoimplementbasedonlocalfactors
31
WherearethepotentialroadblocksthatcouldaffectthescheduleIsthereanywaytomitigatethisHowlongwouldittaketoseeanimpact
Youranswers
Implementation
DowehavethestaffcapacitytoimplementthisArethereadditionalpartnersthatyoucanbringtothetabletosupportimplementationDoesthismakesensegivenourformofgovernment
Youranswers
Authority
DoesourcityhavetheauthoritytoenactthisIfsowhatistheappropriatemechanismAretherelegalbarriersArethereanyexistingpreemptionissuesfromthestateorfederalgovernmentCanthosebeaddressedthroughlegislationadvocacyIstherepoliticalsupportoroppositiontothis
Youranswers
Cost
HowmuchwillthiscostannuallyHowwillwepayforitCanthepolicybeimplementedwithexistingfunding
IfsoisthisfundingreliableHowwouldthispolicyoptionaffectourcitysbudgetintheshortandlongtermWhatotherfundingoptionscanweconsiderIsthereanalternativethatwouldcostlessandgiveussimilarresultsWillanyconstituencybeadverselyaffectediffundingforprogramsthatimpactthemisshiftedtoanotherprogrampolicyorinitiative
Youranswers
Decideonapolicytoimplementbasedonlocalfactors
32
EstablishaframeworktomonitorprogressNowthatyouhavereviewedtheevidenceconsideredlocalcontextandchosenyourpolicyitistimetodeterminehowyourcitywillmonitorprogressofthisnewapproachCitystaffandleadersneedtoestablishtheinfrastructureforongoingcontinuousdecisionmakingbasedonwhetherthepolicychangeisleadingtothedesiredoutcomesforresidentsandwhetherthechangeismeaningfulandrationaltofrontlineworkersSettingarbitrarygoalsatthelocallevelthatconflictwithgoalsatotherlevelsisconfusingtotheindividualsresponsibleforachievingthegoalsGoalsthataresettoohightoachievecanbedemoralizingtostaff
ThissectionincludesstepstosetupavenueinwhichCitystaffandleadershiproutinelyvisitkeyindicatorstodeterminewhatisorisnotworkingandtomakedecisionsinreal-timeFormorereadingonperformancemanagementpleasevisittheGovExPerformanceManagementGettingStartedGuide
InperformancemanagementtermssuchasindicatormeasureandmetricmaybeusedinterchangeablyInthisguidewewilluseindicatorInyourworkthemostimportantthingistobeconsistentinwhichevertermyouuseThisvideomaybehelpfulinunderstandingthisconceptandcommunicatingittoothers
Establishthefundamentalsandgatheryourteam
FirstdeterminewhetherthereisanestablishedvenueorregularmeetingscheduleduringwhichyoucandiscussprogressIfyourcityhasanexistingperformancemanagementstructureonarelatedtopicaddingadiscussionoftheprogressofpolicychangetoyouragendacanbeagreatideaIfthiswillbeastandalonemeetingdecidehowfrequentlytheprojectteamanddecisionmakerswillmeetandputthosemeetingsonthecalendarfortheentirelengthoftheproject
DeterminewhowillstaffandattendroutinemeetingsonCityprogressThesemeetingsmayincludetheMayorandorCityManagerdepartmentheadstaffwhoareimplementingthepolicychangeandanalysts
Identifyindicators
ItiscriticalthatindicatorsbeanimportantcityandcommunitypriorityaswellassomethingtheCityplaysamajorroleinaddressingTheremaybeinstancesinwhichtheCitywillneedtoreachouttoorcollaboratewithexternalpartnerstogetamorecompletepictureofprogressontheoverallpolicychange
Establishaframeworktomonitorprogress
33
Anoutcomeindicatordescribesthedesiredeffectofthetargetedpopulation
ExampleBellevuesfamilyfriendlyneighborhoodsindicatormeasuresthepercentofresidentswhobelievetheyliveinaneighborhoodthatsupportsfamiliesparticularlythosewithchildren
WithyourkeypartnersrevisityourproblemstatementcurrentstateandfuturestateexerciseandbegintoidentifytheoutcomeindicatorsthatwillillustratesuccessFornowchooseasingletop-levelkeyperformanceindicator(KPI)ItiscriticalthattheKPI-thepieceyouwillmeasure-beapriorityforthecommunityandsomethingtheCityplaysamajorroleinaddressing
Outputindicatorsmeasuretheprocessofprogramorservicedelivery
ExampleSanDiegoMayorKevinFaulconersgoaltorepair1000milesofstreetby2020
ReviewingtheadministrativedatasetinventorybegintoparedowntherelevantprogramsanddatasetsthatsupportyourKPIWhatarethemeasuresofeffectivenessforeachprogramTheseareyouroutputindicators
Thediagrambelowillustratestherelationshipbetweenoutputandoutcomeindicators
ThinkaboutimplementationindicatorsasmeasuringstepsintheprogramsrolloutthatarenecessarytoobtaindesiredchangesImplementationoutcomescanfocusontheinterventionstaffdeliveringtheinterventionanorganizationorsystemandcanaddressfeasibilityadaptabilitysustainabilityandcostTwokeyquestionstoconsiderwhendevelopingimplementationindicatorsare
1 Istheinterventionacceptableandappropriatetothecommunityand
2 IsitfeasibletoimplementinterventionXatthistimewiththeseresources
Establishaframeworktomonitorprogress
34
Amoreextensivelistwithdefinitionsofpotentialimplementationoutcomesislinkedhere
UsethetablebelowtochoosethedesiredimplementationoutcomesforyourpilotandtoplanyourmeasurementtechniqueThisshouldincludelevelofanalysis(individualproviderorconsumerorganizationorsetting)yourmeasurementtechnique(surveysobservationinterviewfocusgroupsadministrativedataanalysis)andthephasewhenyouwillcollectthisinformation(earlymidlate)Theremaybemorethanonelevelofanalysismeasurementtechniqueandphaseofdatacollection
Implementationoutcome
Definitionofoutcome
Levelofanalysisandmeasurement
Yourmeasurementtechnique
Phaseofpilotwhenyouwillcollectdata
Usethespacebelowtocaptureyourindicatorwishlist
Indicatorwishlist
Outcomes_____________________________________________________________________________
Outputs________________________________________________________________________________
Implementationoutcomes______________________________________________________________
Setapublichigh-levelgoal
Thehigh-levelgoaliscriticalinestablishingdirectiontogalvanizethepeoplewhowillbedoingtheworkWhendiscussingthegoalusetheMayorsbullypulpitLookforkeymilestonesinCitygovernmenttosharethegoalstatementsuchasthebeginningofaMayoraladministrationaStateoftheCityspeechorotherpublicremarks
GoalsshouldbeldquoSMARTrdquo
Specificaspossible(innamingthepopulationwhowillbebetterofftheactionthecitywilltake)
Measurable
Achievable
Establishaframeworktomonitorprogress
35
Results-focused
Time-bound
ThegoalstatementshouldincludeyouroutcomeindicatorlikeChattanoogasgoalldquoWewillendchronicveteranhomelessnessbyDecember2016rdquoAdiagramtohepyoubuildaSMARTgoalaroundyouroutcomeindicatorisbelow
Component Description Yourturn
Specific WhowillbebetteroffWhatactionthecitywilltake
Measurable Howwillyoumeasuresuccess
Achievable Isthegoalattainable
Results-focused Whatistheexpectedresult
Time-bound When
Setrealisticambitioustargetsandtimelinesforeachindicator
ToestablisharealisticandambitioustargetforyourgoalstatementusetheadministrativedatacollectedforyourchosenKPItoreviewactualCityperformanceinthemostrecentyearcomparedwiththethreeorfourmostrecentyearsworthofdataBegintocalculatesummarystatistics(egtotalminimumaveragemedianmaximum)toidentifytrendsCreatecumulativeandmonthlychartstoidentifytrendsandanomaliesBasedonyouranalysisofhistoricaldataignoretheanomaliesandassumetheproductivitytrendswillfollowthesameoverallpatternfrommonth-to-monthFinallyapplyanincreasedproductivityfactortotheperformanceeachmonthtoforecastfutureprogressBasedonthisanalysischooseatargetthatinspiresimprovementwhileaccountingforreality
Manycitiesthataremorematureintheirperformancemanagementpracticesrecognizetheimportanceoftakingacommunity-levelapproachtoimprovingoutcomesforresidentsThesecitiesarecreatinggoaltaxonomiesthatincludemeasuresrelatedtopublicsafetyhousingandevenrecreationCitiesdriveprogressonthesegoalsbyimplementingstrategiesthatengagemultipledepartmentsandcollaboratewithserviceprovidersinthecommunity
EventuallyyouwilldevelopgoalstatementsfortheoutputsinordertocapturetheefforttoachievetheoutcomecontainedinyourgoalstatementYoursetofindicatorsshouldbecomposedofshort-andinterim-termsupportivemeasureswithaccuratetargetsandrelevanttimelinesAnexamplegoaltaxonomyfromtheCityofLasVegassResultsVegasprogramisbelowUsethistaxonomyasamodelforyourownsuiteofindicatorsandgoalstatements
Establishaframeworktomonitorprogress
36
Yourturn
Thetemplatebelowismeanttohelpyoucraftagoalstatementforeachsupportingmeasure
Goalstatement
Departments
Outcome
Activities
KPI
Target
Timeframe
Conductanddiscussrelevantanalysis
ThrougharoutineseriesofmeetingscontinuallymonitorprogressofKPIssupportingindicatorsandanyotherdatacollectedsuchasqualitativedatatoidentifytrendsthatwillindicatewhetherthechosenpolicyishavingthedesiredimpactThesedatapointsshoulddrivethediscussionduringthesemeetingspointingtoopportunitiesforfurtherinquiryinto
Establishaframeworktomonitorprogress
37
whatisdrivingtheobservedtrendsAnalystsshouldbeencouragedandtrainedtoaskquestionsofthedepartmentsproducingthedataandconductsitevisitswhenpossibletodeepentheirunderstandingofthedataPresentationoftheanalysisshouldfocusononeideaatatimetokeeptheconversationfocusedonkeypointsandspecificopportunitiesforimprovementYoumayconsiderdevelopingapublicfacingdashboardtosharethisprogressmorebroadly
AgeneralcomparisonofthestatusquodatawithwhatwasobservedduringthepilotcanprovidedecisionmakerswithaquickunderstandingofthedifferencesbetweentheoldandnewapproachesalthoughthiswillnotshowcausalityOftenthesecomparisonswillincludeoperationalmetricssuchasdaysordollarssavedbutrememberthattheimpactonpeoplewillbemostimportantIfyouareworkingwithanevaluationpartnertheywillsupportthisstep
Usethespacebelowtosummarizetheresultsofyouranalysis
MeasureTargetvalue(seesectiononsettingtargetsabove)
Pilotvalue(whatdidyouobserveduringthepilot)
Historicalaverage(ifavailable)
Results(WhoisbetteroffIsanyonenegativelyimpacted1-2sentencesreassessingyourgoalstatement)
DocumentyourprocessandprogressincludingfeedbackfromtheparticipantsimplementationpartnersandtheprojectteamDidanythingunexpectedoccurduringthepilotArethereinternalandexternalfactorsthatmayaffectyouroutcomesThisiswheretheimplementationoutcomescanbecanhelpfulIftheinterventiondidnotgoasplannedconsiderwhetheritisbecausetheinterventionwasnotacceptableorfeasible
UsethetemplatebelowcreatedbyDenversPeakAcademytocapturethisinformationIfthe306090daytimelinedoesnotmakesenseforyourcityfeelfreetocustomizeit
CurrentState
FutureState
30Day
60Day
90day
Whatwentwell(+)
Whatdidnotgowellshouldbechanged(-)
IfaquestionordiscussionpointarisesthatcannotbeansweredinthemeetinganassignedstaffmembershouldcapturethequestionduringthemeetingAfterwardthefollowupitemsshouldbesenttorelevantpartnersandthenextmeetingshouldbeginwithadiscussionof
Establishaframeworktomonitorprogress
38
thoseitems
Takeaction
DiscussionsaboutwhetheryourindicatorsaremovingintherightdirectionandwhatmightbedrivingtrendsmayleadtoadiscoverythattherearenotenoughresourcesallocatedtothispolicyorissueTheremaybelimitedfundingforstaffprogrammingortechnologythatisdrivingthedatainthewrongdirectionIfCityleadershipisintheroomthisisagreattimetomakethecaseforadditionalfundingduringthebudgetseasonBycollectinganalyzinganddiscussingdataonthispolicyroutinelyyouarebuildingevidencetomakethecasefortherightleveloffundingtoreachtheestablishedtargets
Establishaframeworktomonitorprogress
39
Assessdataaccessibilityqualityandsecurity
Determineaccessibilityandqualityofdata
OnceyouhavecompiledyourwishlistofindicatorsitistimetodeterminetheaccessibilityandqualityofthedataTopreventcomplicationyoumaywishtoprioritizerelevantindicatorsforwhichyouarealreadycollectingdataAvoidthetemptationofadoptingatechnologysolutiontocollectnewdataatthispointsuchasaphotosurveytoolorotherappFornowyouaretryingtodeterminewhichdatasetsarerelevantaccessibleandhighqualityBesuretoworkwithyourkeyinternalpartnerswhileyouareidentifyingyourmeasuresDepartmentstaffwhoworkonprogramsarefamiliarwiththeindicatorsthatwillaccuratelysupporttheoutcomesanddatabaseadministratorswillbevaluablewhendiscussingaccessibilityandqualityofthedata
Abrieflistofconsiderationsfortheaccessibilityofyourdataisbelow
ArewecurrentlycollectingthisdatasetorwillwebecollectingitforthefirsttimeHowfarbackdoestherecordgo(timehorizon)WouldweneednewtechnologytocollectoranalyzethedataCanthedatabecollectedmanuallyorcanitbeautomatedHowfrequentlyisthedataproduced(continuouslyhourlydailyweeklymonthlyquarterlyyearlyetc)IsthedataavailablefortheentireCitypopulationorasubsetofthepopulationHowgranularisthedata(individuallevelCensusblockneighborhoodcitywideetc)DoesthedatacontainsensitiveinformationIsthereaproxymeasurethatwecanusefornowIsthedatamachinereadable
AsnotedbytheWilsonSheehanLabforEconomicOpportunities(LEO)attheUniversityofNotreDameleveragingadministrativedatacansignificantlyreducethecostsandtimelinesofrigorousevaluationCitiescanestablishaninternaldatasharingagreementacrossdepartmentsincludinglanguagethatenablesadministrativedatatobeusedforresearchpurposesCitiescanalsoidentifyexternaladministrativedata-holderssuchasacountyschooldistrictpolicedepartmenthospitaloranotherstatethatmayhaveoutcomedataimportantformeasuringprogramimpactCitiescanleverageexistingrelationshipsandorworkwithresearchinstitutionstodevelopstreamlineddatasharingprocessesandagreementswiththeseexternalentities
Assessdataaccessibilityqualityandsecurity
40
Evaluatethequalityofyourdata
QualityisasimportantasaccessibilityToevaluatedataqualityyoumaywishtocreatearubricliketheonebelow
Value ValueDescription
3 ExcellentDataQualityThedataiscompleteaccurateandupdatedregularly
2 AdequateDataQualityThedataissomewhatcompleteaccurateandupdatedfairlyregularly
1 PoorDataQualityThedataisnotcompleteaccurateorupdatedregularly
ThisisagreattimetobuildonthedatainventorythatyoustartedinthegatherinformationpageSuggestedadditionalfieldstoaddtoyourdatainventorytemplatearebelow
Program DatasetTitle
DatasetDescription DataSource
Departmentorexternalpartner
DoestheCityalreadycollectthisdata
Whatisthenameoftheprogram
Human-readablenameofthedataset
Shouldbeunderstandablebynon-technicalusers
Whatistheoriginalsystemapplicationfilethathousesit
Departmentorexternalpartnerthatmaintainsthedata
Isthisdataalreadycollected
Updatefrequency-ThefileformatforthedatasetTypicallycsvxlsxshapefiletxtetc
Format-Howoftenthedatasetisupdated
Timerange-Theperiodoftimecoveredinthisparticulardataset
Datause-Howisthisdatasetcommonlyused
Datausers-Whotypicallyusesthisdataset
Dataownerstitle-Titleofthedataowner
Dataownersposition-DataOwnersjobtitle
Dataownersemailaddress-DataOwnersemailaddressWhomanagesthedataandorisresponsibleforgrantingpermissiontoaccessthedataWhounderstandswhatthedatasetincludesandcananswerquestionsaboutit
Dataqualityconcerns-Describetheconcernsyoumayhaveaboutthequalityofthedataset
Assessdataaccessibilityqualityandsecurity
41
Division-DivisionwherethedataismaintainedIfmorethanonedivisionisresponsiblelisttheprimaryownerofthedataset
Sensitivedatacomments-DescribetheissuesthatwouldpreventthecityfromsharingthedatapubliclyCommonsensitivityconcernsincludeprivacyviolationssecurityissuesorhighcostorstaffdemandstopostthedata
Publishingstatus-Thistheplacetoputanycommentsregardingthecurrentstatusofthedataset
Publishedlink-Thisisthelinktothepublisheddatasetontheopendataportalorcitywebsite
Reviewdatasecurityandprivacyprotectionprotocols
OftenapolicychangealsoinvolvesachangeinthedatathatisbeingcollectedhowthatdataisbeinghandledwhohasaccesstoitandahostofotheradjustmentsAsyoutakestockofthedataassetsthatareavailableforyourinitiativeitisvitaltoalsothinkthroughwhatwillbedonetoprotectthatdataandtheprivacyofthepeopleitrepresentsAbigpartofgooddatasecurityinvolvesstrongdatagovernancepracticesthatdefineandidentifysensitivedataandconsciouslyworktobalancethevalueandriskofstoringsuchsensitivedataOnewayofevaluatingyourdatacomesfromcaselawrelatedtotheUSFreedomofInformationActwhichwehavetranslatedtothechecklistbelow
1 Woulddisclosureresultinasubstantialinvasionofprivacy2 Haveweconsideredtheextentorvalueofthepublicinterestandthepurposeorobject
oftheindividualsseekingdisclosure3 Istheinformationavailablefromothersources4 Wastheinformationgivenwithanexpectationofconfidentiality5 Isitpossibletomouldreliefsoastolimittheinvasionofindividualprivacy
TheseriskscouldbemitigatedusingavarietyofstrategiesfromaggregatingdatatoobfuscatingindividualstoredactingfieldsthatmayincludePersonallyIdentifiableInformation(PII)orre-evaluatingyourdatacollectionmethodstofindalternativewaysofgettingtheinformationyouneedThesestepsshouldbecodifiedandmanagedbyadatagovernancecommitteeifpossibleForthesensitivedatathatisdeemedessentialcreateaplanaroundpotentialdatabreachesinordertoquicklyandconfidentlyrespondtoemergencysituations
ProactivelypreparingforthesesituationsisoftenthebeststrategyfordatasecuritybutdonotforgetthetechnicalsideInvolveyourITadministratorsinconversationsaboutdatastorageandaccessensuringthattheywillbeprotectedbyenterprise-gradecybersecuritysystems
Assessdataaccessibilityqualityandsecurity
42
Assessdataaccessibilityqualityandsecurity
43
Designimplementationandevaluation
Designtheimplementation
NowthatyouhaveyouraccountabilityplanandkeyindicatorsitistimetogetspecificaboutyourplanforimplementationThetemplatebelowwillhelpyouthinkthroughthespecificmechanismsofimplementingpolicychange
UnderstandingorganizationsandsystemsiscriticaltounderstandingandmakingdecisionsaboutwhereandhowinterventionsshouldbeappliedApolicyimplementedinasinglesettingmayshowpromisingresultsbutimplementingmultiplepoliciesandprogramsatthesametimewithdifferentactorsresponsibleforeachaspectcanhelpensuresustainablebehaviorchange
InSomervilleMAShapeUpSomerville(SUS)isacitywidepublichealthprogramthatisimplementedacrossmultiplelevelsofinfluenceandsystemsthroughinterventionsbeforeduringandafterschooltoincludechangestobuiltenvironmentsuchasbikelanesregionalmasstransportationandphysicaleducationequipmentinschoolsandgymstoincreasephysicalactivitywellnesspoliciesandschoolgardensnutritioneducationduringandafterschoolandhealthyrestaurantprogramstosupporthealthyfoodandhealthyeatinghabits
Usethetemplatebelowtoplanyourimplementation
Policyimplementationplanningtemplate
ListthespecificdesiredoutcomesInthepreviousstepyouworkedwithdepartmentstafftoestablishspecificmeasurableoutcomesandinterimindicatorsPrioritizetheindicatorsthatwilldemonstrateprogressUsethespacebelowtocapturethedesiredoutcomesofthepolicychange
ExampleShapeupSomervillebeganwiththegoalofpreventingobesityin1stthrough3rdgraders
Yourturn
____________________________________________________________________________
____________________________________________________________________________
Designimplementationandevaluation
44
SpecifythetargetpopulationAreyoudesigninganinterventionthattargetsonlycertainsitesAreyouplanningonimplementingaprogramorinitiativeforaspecifictargetaudienceClearlyspecifythepopulationyouplantotarget
IfyouplantoconductarigorousimpactevaluationasnotedbytheWilsonSheehanLabforEconomicOpportunitiesitisimportanttoestablishconcreteobjectiveeligibilitycriteriaandensurethatthesecriteriaareappliedconsistentlybyallstaffduringscreeningandintakeItisalsovitaltoidentifytheindividualswhoareeligibleforaparticularprogramorserviceevenifeveryoneeligiblecannotbeservedbyexistingresourcesThishelpscitiesidentifyanyunmetneedsandguidefutureresourceallocationwhilealsocreatingthepossibilityofarigorousimpactevaluationbyidentifyingapotentialcomparisongroupFinallywhenconductingscreeningandintakeitisimperativetocollectandmaintainbasicinformation(egnamedateofbirthgender)aboutallprogram-eligibleindividualsnotjustthosewhoparticipateintheprogram
ExampleIntheShapeUpSomervilleexampletheprogrambeganbytargeting1stthrough3rdgraders
Yourturn
____________________________________________________________________________
____________________________________________________________________________
DefinethelocationWherewillthepolicychangetakeplaceWillimplementationtakeplaceatonesiteormultiplesites
ExampleShapeUpSomervilleinterventionsweredeliveredinschools
Yourturn
____________________________________________________________________________
____________________________________________________________________________
ClarifyimplementationpartnersandarticulatetheirrolesWhatarethekeyorganizationsinstitutionsorpeopleresponsibleforimplementationTheprojectteamshouldbeasclearandspecificaspossibleinprovidingadescriptionoftheconcreteactivitiesthatimplementationpartnersaremeanttocarryoutPracticeprofilesmaybeahelpfultooltohelpdefinewhatisrequiredoftheactorswhoareresponsibleforimplementingtheintervention
Designimplementationandevaluation
45
ExampleInSomervillemayoralleadershipandengagementofmultiplecityagencies(publichealthprimarycareschoolsCambridgeHealthAllianceetc)streamlinedandconnectedthevariousShapeUpSomervilleinitiativesandgrantswhichhelpedtokeepthetalentandenergyfocusedonhealthyeatingandactivelivingforallsectorsofSomervilleSUSpartnersultimatelyincludedschoolsnonprofitsCityandstateagenciesandnationalfoundations
Yourturn
____________________________________________________________________________
____________________________________________________________________________
DescribetheinterventionWhatarethestepsactivitiesorprocessesRememberthatthemechanismthespecificstepsorprocessthatleadtobehaviorchangeiskeytounderstandingtheeffectivenessofthepolicychangeandsharingtheresultswithothers
ExampleGoalsofShapeUpSomervilleweretoinfluenceeverypartofanearlyelementaryschoolchildrsquosdayndashbeforeduringandafter-schooltopromotehealthyeatingphysicalactivityandtheirweightstatusInterventionincludedfoodserviceenhancementstoimprovebreakfastandlunchandstaffprofessionaldevelopmentwalktoschoolactivitiesSUSclassroomcurriculumandprofessionaldevelopmentschoolwellnesspolicydevelopmentSUSafter-schoolcurriculumandprofessionaldevelopmentoutreachandeducationtohomethroughmaterialsforumseventsldquoSUSapprovedrdquorestaurantscommunityoutreachandcapacitybuildingthroughpolicydevelopmenttrainingsmediaplacements
Yourturn
____________________________________________________________________________
____________________________________________________________________________
ClarifytimelineandmilestonesfordeliveryandevaluationHowfrequentlywilltheservicesinterventionbedeliveredHowlongwilltheprogramrunWhatisthetimetoexpectedimpacts(eg12monthsafterprogramcompletion)
Designimplementationandevaluation
46
ExampleShapeUpSomervilleledbyTuftsUniversityFriedmanSchoolofNutritionScienceandPolicyconductedaquasi-experimental(nonrandomized)interventionstudyamong1stthrough3rdgradestudentsinSomerville(intervention)andtwocomparisoncommunitiesfor9-monthsduringoneschoolyear(fall2002ndashspring2003)ThestudyshowedpositiveassociationoftheinterventiononchangeinBMISUSreleasedreportssummarizingevaluationsprogresstodateplansforadditionalprojectsandimpactstenandfifteenyearsintotheinitiative
Yourturn
____________________________________________________________________________
____________________________________________________________________________
PlanyouranalysisTheprevioussectionaddressedestablishinganaccountabilityframeworktomonitorprogressandimplementationDeterminewhichindicatorswillbediscussthroughthatprocesswhowillsharethedataandanalysisandhowfeedbackfromthoseconversationswillbeincorporatedbytheprojectteam
ExampleTheMayorrsquosofficeimplementedadata-drivendecisionmakingapproachcalledSomerStatwhichwassynergisticwiththeapproachestakenbytheresearchandevaluationteamsdeployedforthespecificgrants
Yourturn
____________________________________________________________________________
____________________________________________________________________________
PlanforsustainabilityIfthepolicyandprogramchangeissuccessfulhowwilltheprojectteamsustainandexpandtoadditionalsitesorpopulations
ExampleAlthoughmostofthepartnershipsforgedintheearlyyearsweremaintainednewgrantsandcollaborationsexpandedattentiontoolderpublicschoolstudentsactivetransportationandtheGreenLinesubwayextensionthebuiltenvironmentfarmerrsquosmarketeffortssuchasldquoGrowninSomervillerdquowithUnionMainStreeteconomicdevelopmentandhealthdisparitiesaswellasdatacollectionmonitoringandsurveillance
Yourturn
Designimplementationandevaluation
47
____________________________________________________________________________
____________________________________________________________________________
Pilotplanningtemplate
IfyouplantoconductapilotfirstdeterminethepurposeofthepilotDoyouwanttoobservewhetherthepolicychangeresultsinthedesiredbehaviorchangeAreyouconcernedaboutfeasibilityscalabilityorcostDoyouwanttotestwhetheryourdesiredresultscanbeachievedthroughthispolicyapproachClarifythegoalofthepilotandthequestionsyouhopetoanswerThiswillhelpyouidentifythekeyindicatorsthatyouwillusetomeasurewhetherthepilotissuccessfulBeforebeginningestablishabaselineofcurrentperformanceandsetyourtargetsaccordinglybasedonwhatyouhopetoaccomplishthroughthepilot
NextestablishthepilotplanforyourprojectteamincludingmilestonesforteamcheckinstodeterminewhethertomoveaheadorchangecourseThisisseparatefromthemeetingsthatwilloccuraspartoftheaccountabilityframeworkUsethespacebelowtocreateyourtimelineanddevelopyourmeetingschedule
Task Personresponsible Startdate Checkindate Deliverable
Planyourevaluation
WhileyouaredesigningtheimplementationitisalsotimetobegintodevelopyourplanforanalysisAsnotedbytheWilsonSheehanLabforEconomicOpportunitiesdecisionsaboutprogramsetup-includingoutreachscreeningintakedatatrackingandenrollment-candeterminewhetherornotarigorousimpactevaluationwillbepossibleAsdiscussedcityleadersmayfindithelpfultoincludeevaluationpartnersinearlyconversationsaboutprogramsetupDuringpreviousstagesyoumayhaveworkedwitharesearchertoplanyourproposedpolicychangeYoushouldbeclearaboutthepurposeoftheevaluationanddevelopanevaluationplanBelowareadditionalconsiderations
Causationvstrendanalysis
IntheEstablishaframeworktomonitorprogressstagetherewasguidanceontrackingtrendsovertimeItisimportanttonoteherethatmonitoringdataisdifferentfromanalyzingacausaleffectTrendanalysiswillillustratethatchangesareoccurringandmayleadtoquestionsthatcouldbeansweredbyyourevaluationbuttrendanalysisalonedoesnotprovideevidenceaboutwhatisdrivingthosechangesYoumayseechangesinyour
Designimplementationandevaluation
48
outcomeindicatorsbasedonvariablesoutsideoftheCityscontrolForinstanceachangeinfederaltradepolicycanhaveanimpactemploymentratesinyourCityoutsideofyourCitysnewjobtrainingandplacementprogram
ToaddresscausalityyoumustcomparetheoutcomesofindividualswhoreceivedtheinterventionwiththosewhodidnotWithoutacomparisongroupofsimilarindividualswhowerenotexposedtotheinterventionitisimpossibletoknowwhichfactorsareresponsiblefortheoutcomesRandomizedControlTrials(RCTs)inwhichindividualsarerandomlyassignedtotreatmentandcontrolgroupsarethegoldstandardinprovingcausalityIfyoucandemonstratethatoutcomesimprovedforthosewhoreceivedtheinterventionanddidnotimproveatallorimprovedlessforacomparisongroupyoucanbesomewhatcertainthattheinterventionwasthecause
Examplesofothernon-causalimpactevaluationsthatarerigorousandreliableincludecertainquasi-experimentalstudiesusingcomparisongroupdesignssmartinterruptedtimeseriesanalysesthatmodeltrendsovertimeinwaysthatallowestimationofinterventioneffectswelldesignedpre-posttestanalysesneedsassessmentsimplementationstudiesetcFormoreinformationondifferenttypesofevaluationsseetheSageResearchMethodswebsite
Lookfordifferencesingroups
BuildintimetoalsoexaminethepilotdatatoseeifgroupsareexperiencingdifferentresultsReviewingperformancebasedonoutcomemeasuresforapopulationinaggregatemayhidethesubtlewaysthatsomepeoplemaybeaffecteddifferentlysoitisnecessarytocarefullyanalyzethedatafordisparateoutcomesForinstancetheMovingtoOpportunity(MTO)studythatexaminedtheimpactofhousingvouchersonfamilieswholivedinhigh-povertyneighborhoodsfoundthatchildrenwhomovedbeforetheageof13hadbettereducationandhigherincomesasadultsHoweveradultsdidnotseeimprovededucationandemploymentoutcomes
Disaggregatingdataintomorespecificcategoriesofraceandethnicitycanhelpunderstandtrendsanddisparitiesinamoregranularwaywhichcanhelpdecisionmakerstargetpoliciesandinterventionsinamoreequitableway
EarlierintheRoadmapweworkedtoidentifystakeholdersandwhomightbepositivelyandnegativelyimpactedbyachangeinpolicyUsetheresultsofthatexercisetoinvolvethecommunityininterpretingdataanalysisandtogatherqualitativefeedbackfromthecommunitytocallattentiontoissuesForinstanceanincreaseinyouthdetentionmaybeexplainedbyinefficienciesorlackofaccesstotransportation
Designimplementationandevaluation
49
Designimplementationandevaluation
50
Shareyourstoryandsuccess
Sharetheresults
Storytelling
AnintegralstepinyourstrategyissharingtheresultsofyourpilotorpolicychangewithyourcommunityThisincludesdevelopingmessagingaroundtheworkofyourorganizationandcreatingaplantodisseminateyourstoriesWhytellstoriesaboutyourwork
Informandorunderstandasubjectorsituation
Giveneededcontextofdataorbitsofinformation
Relatablestoriesarememorableforreadersviewers
ReadersorviewerswillseetheinformationasmorecompellingwhenthereisademonstratedandrelatablechangeorimpactonthemselvesortheirfriendsneighborsandcolleaguesStoriesareaboutpeopleandgovernmentisfundamentallyabouthelpingpeopleSotellthestoriesofthepeopleyouandyourcitycolleagueshelped
Keepthesetipsinmindasyoudevelopyourstorywithyourteam
WhatwereyourfindingsandwhyshouldanyonecareTellyouraudiencehowtheproblemwassolvedWhathastheimpactbeenonresidentsandcitystafftodateWhatwasthesuccessandweretheremultiplesuccessesalongthewayIfyouhavepreliminaryresultsordatashowingimprovementtosharethisisthetimetodoit
WhowasinvolvedandimpactedReviewthelistofstakeholdersHowwereeachofthemimpactedbytheprogramHumanizeyourstoryandimpactbyaddingadditionalvoicesThesevoicescanbeoneofthefollowing
Apersonorfamilywhowaspositivelyaffectedbythechangesuccess
Aninvolvedthird-partydescribingboththeproblemandthesolution(ieanorganizationnonprofitprovidersbusinesstradeassociationsportsteametc)whoispartofthecommunitythatthegovernmentworkedwithonthechangesuccess
Someonefromthegovernmentitselfwhocanspeakaboutthechangesuccess
TellmoreaboutyourcityDescribeyourcityincludingpopulationdemographicsindustriesetcWhatareitschallengesWhatisitknownforWhatmakesituniqueHowisitsimilarordifferentfromothercitiesintheUS
Shareyourstoryandsuccess
51
DescribeyourcityrsquoschallengeWriteoutthechallengeyourcityisfacing(asdefinedinyourproblemstatement)HowdoesitaffectresidentsHowdoesthisproblemaffecttheworkofcitystaffmembers
AddthedetailsProvidedetailsonyourappliedresearchstrategyTalkaboutwhatyourcityhasdonenthepasttoresolvetheissueDiscusshowyourstrategyhaschangedwhenyoubegantoadoptthisapproachWasmoneysavedorinvested
Usethisstorymapexercisetoorganizeyourthoughtsandguaranteethatyouincludealloftheelementsofatraditionalstory
AfteryoudevelopyourmessagingcreateacommunicationsmarketingandoutreachstrategydesignedtospreadyourmessageYourstrategyshouldwalkyouthroughthefollowingkeyactivities
1 IdentifyandprofileyouraudienceWhencreatinganddisseminatingyourmessageitisalwaysgoodtothinkaboutyouraudienceWhoexactlywouldbeinterestedinwhatyouhavetosayThegeneralpopulationofthecityPeoplewhoworkinnonprofitsBusinessownersItmattersbecausenoteveryonewholivesinacitymaybethetargetoftheinformationBusinessownersmaybemoreinterestedindatashowingeffectsonthebusinessdistrictthanothersandsoonWewillcoverthislaterbutthisstepwillalsohelpyoutothinkofthemosteffectivewaytoreachyouraudience
2 Defineyourcalltoaction(ifyouhaveone)SpendsometimeconsideringwhyyouareinterestedintellingyourstoryDoyousimplywanttoupdatethepubliconyourprogressorcelebrateyoursuccessesAreyouinterestedinyouraudiencechangingitsbehaviororsupportingaspecificinitiativebeinglaunchedbyyourorganizationIfyouwanttohaveyouraudienceactorbehaveinacertainwaybespecificandincorporateitinyourmessaging
Shareyourstoryandsuccess
52
3 SelectperformancemeasuresThinkabouthowyouwoulddetermineifyourcampaignissuccessfulandselectappropriateperformancemeasuresYouwillrevisitthesemeasuresattheendofyourcampaigncommunicationsplanimplementationtodeterminewhetheryouneedtomodifyyourcommunicationsandoutreachstrategyforthefuture
4 MethodstocommunicateyourstoryTherearemanywaystocommunicatesuccessofyourprojectHerearesomesuggestionsonreachingaudiencesItisimportanttodeterminewhichmethodsarethemosteffectiveforyourdesiredaudiencetoreceiveinformationDoyourresearch(PewResearchCenterhasexcellentstudiesonthereachofmediadigitalmediaandmarketing)ForexampleifseniorsareyourtargetpopulationandtheyarelessinclinedtouseInstagram(thatistrue)youmaywanttoconsiderFacebookorthenightlylocalnewsbothofwhichareusedmoreoftenbyseniorsBelowisalistofsomemethodsandtoolsyoucanuse
i Pressreleasestolocalnewspapersandtvstations
ii ShortsmartphonevideosAgreatwaytoengageanaudienceisthroughvideoAskonequestionofsomeonewhoispartofthestoryMaybethereisasocialworkerwhocandiscusshungerinyourcityThesevideoscanbereusedonmultipleplatforms
iii Long-formwriting
iv TellingindividualstoriesthroughaseriesofphotosandshortnarrativesforpublicationonFacebookorInstagram
v Op-edsopinionpiecesplacedinthelocalpaper
vi Emailupdatestoyourlist
vii Connectingwithappropriateorganizationsthatareinvestedinthetopicandworkwiththecommunityandaskthemtodisseminateinformationtotheirmembers
viii EmployingldquoWordofMouthrdquo(WOM)tacticsbycreatingtownhallsorfreeeventsforresidentstoattendintheirownneighborhoods
5 DeterminethetypeofcollateralUsuallycampaignsthattargetresidentsrequireseveralavenuestoreachthemPeoplearenotalwaysinthesameplaceorspaceHavingakeenunderstandingofyouraudiencewillassistyouwithyourchoicesonwhatneedstobedevelopedSometimesthatmeansimagesforsocialmediaaonepageflierthatgivespeopleinformationonattendinganeventoranin-depthbrochure
Shareyourstoryandsuccess
53
6 ImplementyourstrategyKeepinmindstrategiescanbefluidSometimessomethingmaynotturnoutasexpectedevenifduringplanningsessionsitwasldquosupposedtordquoItisokYoucananalyzewhathappenedduringadebriefsessionandonceyoursurveysandothermeasurementshavebeenfinalized
Visualizations
Withtherightvisualizationyoucanreviewandshareinformationinsightsanalysisperformanceandtrendsallofwhichisvaluabletoacross-sectionofthepublicButwhichvisualizationisrightforyouTochoosetherighttypeofchartgraphdeterminethetypeofquestionyouwanttoanswerSomecommoncharttypesarebelowandmoreinformationcanbefoundhere
Exploreadditionalanalyticaltechniques
Duringthegoal-settingphaseyouestablishedoutputoutcomeandimplementationmeasuresBasedontheresultsofyourpilotyoumayneedtoadjustyourmeasuresWiththepilotdatainhandyoumayhavesomeideasofhowfutureanalysescouldbedifferentandpotentiallyimprovethequalityofyourprogramTakesometimetobrainstormotherrelevantdatasetsthatyoudidnothavethetimetopursueearlierincludingmessydepartmentaldatathatneedsmoreattentionorsourcesofpublicdatafromthestateandfederalgovernmentnonprofitsandevenprivatecompaniesthatcouldsupplementyourunderstandingofanissueUsingmachinelearningtechniquesyoumightbeabletobetter
Shareyourstoryandsuccess
54
predictdemandforhomelessservicesusingNOAAweatherforecastspreemptwatermainbreaksusingsoilstratigraphyorinvestineconomicdevelopmentbymonitoringrealestatelistingdata
Advancedanalyticaltechniquescanalsohelpyourimplementationrunmoreeffectivelybyintegratingdata-informedtargetinganddecisiontoolsintotheprocessByautomatingroutineworkflowsandsurfacingsubtletrendsthesetoolscanhelpyouiterateinamoreprecisemannerandprovidekeystrategicguidancetofront-lineemployeesthatwouldotherwisebeunavailableThesesortsofdata-science-enhancedprocessescanbeclassifiedasoneofsixprojecttypesasexploredfurtherbytheNewOrleansLANOLAlyticsteamFormoreguidanceonscopingasimilarprojecttheGovExadvancedanalyticsscopingguidegivesyouastep-by-stepoverviewofhowtobesuccessful
Usethespacebelowtolistyourideasforpossibleadvancedanalyticsprojectsbytype
Typeofproject Ideas
Findingtheneedleinthehaystack
Prioritizingforimpact
Earlywarningtools
Betterquickerdecisions
Optimizingresourceallocation
Experimentingforwhatworks
Keepthefeedbackloopgoing
GatherinformationonpublicsentimentaboutpolicychangethroughfocusgroupssurveysandpublicmeetingsledbyadvisorygroupmembersNotewhethernewinformationisavailablethatcanhelpanalyzeorcontextualizeyourprogress(egfederalorstatedatareports)Sharetheresultsofyourpilotandthepolicychangeprocesswithothercitiesandresearchers
Shareyourstoryandsuccess
55
WorkingwitharangeofevidencePresentlythereisalotofattentionandinvestmentinevidence-basedpolicymakingincludingthefederalbipartisanCommissiononEvidence-BasedPolicymakingthePewCharitableTrustsreportsandtoolkitsonHowStatesEngageinEvidence-BasedPolicymakingandtheLauraandJohnArnoldFoundationsupportedEvidence-BasedPolicymakingCollaborativeManyoftheseinitiativesfocusonrigorousevaluation--meaningrandomizedcontrolledtrials(RCT)orotherexperimentalmodels--asthebestkindofevidencetouseHoweverthisbodyofevidenceisofteninsufficientforthespecifickindsofquestionsthatgovernmentscanhavewhenmakingadecisionaboutpolicyInthisguideweproposeanapproachtosupportgovernmentsinincorporatingadditionaltypesofevidenceintotheirdecisionmaking
OnewaytounderstandarangeofevidenceistouseNestarsquosStandardsofEvidencetoframeanddefineevidenceTheBritishnonprofitNestaworkstoinjectpositivechangeintothewayUKgovernmentpoliciesandprogramsfunctionOneoftheirkeyareasofworkhasbeendevelopingWhatWorksCentresorganizationsthatcollectcommissionsynthesizeanddisseminateevidenceaboutinterventioneffectivenessacrossarangeofsocialimpactareasNestadevelopedthefollowingStandardsofEvidencetoprovideausefulwaytodocumenthowdifferentkindsofinformationdemonstratedifferentlevelsofevidence
Workingwitharangeofevidence
56
Theimportanceofpotentiallybiasedinformation
Onechallengepresentedbyanevidenceclassificationthatacknowledgesthevalueofnon-scientificformsofevidenceistofigureouthowtobalanceallofthepiecestogetherwhenyouusethemTounderstandhowtodothiswehavetothinkabouthowtopositionpotentiallybiasedinformationindecisionmaking
WhenwelookatinformationwhichhasnotbeenrigorouslyevaluatedweacceptthatthisinformationmightbebiasedThewholereasonweconductrigorousevaluationistoremovebiasfromourobservationsWeconductstructuredtrialstoproduceresultsthatareconsistentandgeneralizableIfafindingisunbiasedthisaccuracymdashthislackofbiasmdashcanbeprovedbecauseanyindependenttestercanreproduceitwhentheyfollowanidenticalprocess
YoucanlookatpotentiallybiasedinformationasjustinformationthatisearlieroninitspathtorefinementreplicationandverifiedaccuracyApieceofinformationbasedonanyoneindividualrsquosobservationsislikelytobebiasedbecauseitiscomingfromjustonesourceBiasisreducedbygatheringtogetherlotsofindependentsourcesofinformationandfiguringoutitsmostcommonattributes
MaybethebiasedinformationmakesaclaimthatistoolargemdashitsayssomethingistrueunderallconditionswhenitisreallyonlytrueundercertainconditionsorthataneffectofaninterventionisbiggerthanitstrueaverageorthataneffectlastsovertimewhenitdoesnotPerhapschangingtheclaimtobemorespecificmdashcriticallyreviewingtheinformationtoseeifspecificconditionsmustbemettoachievetheoutcomesmdashwouldmaketheclaimmoreaccurate
IfyoutestedyourclaimfounditdidnotholdupinsomecaseschangedittesteditagainuntilyourclaimrepeatedlyheldupthenyoucoulddemonstratethattheclaimwasnolongerbiasedbutaccurateThisprocessoftestingimprovingtestingandvalidatingiswhatwegetfromstructuredscientificstudy
Withindividualpiecesofinformation--crediblehypothesescasestudiesexpertopinion--youcannotassumethatkindofaccuracyAtthesametimetheseearlyobservationsarefoundationaltoanyreplicablescientificobservationsWewillnevergetanywhereifwedonotstartwiththesepiecesofinformationeveniftheymightbebiased
Combininglevelsofevidencetogetthemostdetailedinformationforyourcity
ItisespeciallyimportanttoacceptdifferentformsofevidencewhenyoufacequestionsthathavenotyetbeentestedbyindependentstudiesSomeofthemostimportantquestionsrelatetothingsthatareveryspecifictoyoursituationYoufaceaparticularcombinationof
Workingwitharangeofevidence
57
issuesactorstrendsandresourcesthatisultimatelydifferentfromeveryotherplaceintheworld
ThisisimportantbecauseaspolicypractitionersandnotscientistsyoucaremoreaboutyourspecificcasethanahypotheticalaveragegeneralcaseYoucarelessaboutthetotallikelyrangeofimpactsfromapolicythanhowthissingleimplementationwillworkYouwantinformationaboutpoliciesthatwillprovideyouwiththebestpossiblepredictionabouthowtheywillworkinyourcontext
ByputtingdifferentkindsofevidenceintothesameframeworkNestarsquosStandardsofEvidenceimplythatwhenyoucannotgetrigorouslytestedexperimentallyverifiedinformationitisappropriatetoworkwithwhatwedohaveThisisrelevantforthemunicipalimplementationcontextbecausenostudypulledfromaliteraturereviewwillaccountforallofthevariablesyoufaceinyourowncontext
Intheabsenceofhavingastudythatpredictshowaninterventionworksinyourspecificcontextyouwillneednon-scientificinformationtotrytofillinthosegapsYouwillneedtogatherstoriesfromresidentsstaffmembersandotherstakeholdersthatsitonLevels1or2ofthestandardsofevidence
Someofthekindsofinformationyouwillprobablygetmoreeasilyfrompeoplethanstudiesinclude
howpeopleincitieslikeyoursreactedtoaninterventioninpublicforumshowpeoplewhoimplementedaninterventioninacitylikeyoursfiguredouthowtogetitpassedhowthecitycommunicatedpubliclyabouttheapparentbenefitsanddrawbacksoftheinterventionthepoliticalacceptabilityoftheintervention
Youcanremainsimultaneouslyawarethattheinformationyouaregatheringthroughindividualstoriesisbiased--notgeneralizableinformedbyspecificperspectives--whilealsorecognizingthathavingsomeinformationaboutsomethingimportantisbetterthannoinformationEvidencefromthefirsttwoevidencelevelsareessentialbuildingblocksofknowledgeandperfectlyusefulwhenyoucannotgetdetailedunbiasedknowledge
Youhavetostartsomewhereandyoushouldalwaysstartwithevidenceyouhaveaboutyourveryspecificconditionsthatyouknowareimportantforsuccess
Workingwitharangeofevidence
58
Thankyou
Interviewswithcities
GovExheldsemi-structuredinterviewswithsixstaffmembersinfourcitiestoestablishtheinitialstepsoftheRoadmapWeaskedrespondentstotalkaboutaninitiativeorpolicythattheyhavebeenresponsibleforimplementingintheircityRespondentsfromthreecitiesspokeabouttheireffortstoaddresshomelessnessandrespondentsfromonecityspokeaboutblightAfterrespondentsidentifiedtheproblemtheyspokeabouthowthecityprioritizedthisissuethevariousstepsinvolvedandbarrierspresent
GovExconductedtextanalysisoftheseinterviewswhichrevealedthatpeoplewasthemostcommonlyusedtermduringtheseconversationswhendiscussingthestepstochangeaprogramorpolicyThisisathemethroughoutthisRoadmapThefirstsectionbeginswithidentifyingtherelevantpeopleTheconversationsaroundevidencearefocusedonhumanbehaviorratherthanmethodologyorstrengthofevidence
Thankyou
59
OncewehadadraftoftheRoadmapwefollowedupwithcitiesonanindividualbasistolearnmoreaboutwhatwouldbehelpfulWearegratefultothefollowingcitystafffortheinterviewsandindividualconversations
MichaelBaskinChiefPolicyOfficerChattanoogaTN
MelissaBridgesPerformanceandInnovationCoordinatorLittleRockAR
VictoriaCarreoacutenAdministrativeOfficerLasVegasNV
TyroneGrandstrandCommunityDevelopmentPlannerFargoND
TibLaughlinDirectorofGeneralServicesKansasCityKS
TimMorelandDirectorofPerformanceManagementandOpenDataChattanoogaTN
CharlesRosendahlManagementAnalystDentonTX
Thankyou
60
MelissaSiebenAssistantCountyAdministratorKansasCityKS
GinaThompsonDirectorofStrategicServicesLewisvilleTX
JHUFacultycollaboration
GovExalsoconvenedthecitycontactsandfacultypartnersinaseriesofAdvisoryGroupmeetingsThesemeetingsservedtoconnectcitiesandfacultytounderstandoneanothersperspective
StefanBaralBloombergSchoolofPublicHealth
AshleyBernerJHUSchoolofEducation
MelissaDavey-RothwellBloombergSchoolofPublicHealth
JonLinksBloombergSchoolofPublicHealth
TamarMendelsonBloombergSchoolofPublicHealth
KristinMmariBloombergSchoolofPublicHealth
KeshiaPollackPorterBloombergSchoolofPublicHealth
JoshSharfsteinBloombergSchoolofPublicHealth
LizStuartBloombergSchoolofPublicHealth
LindsayThompsonCareyBusinessSchool
DanielWebsterBloombergSchoolofPublicHealth
Partnersindataandevidence-basedpolicy
Wealsospokewithpartnerswhoareexpertsinthefieldofevidence-basedpolicywhosharedtheirknowledgeandresourcesofcurrentandfuturethinkinginevidence-informedthinking
WendyBarrenoAssociateDirectorWilsonSheehanLabforEconomicOpportunitiesUniversityofNotreDame
TiffanyDavenportDirectorofDataPartnershipsBaltimoresPromise
TomiHiersDirectorBaltimoreCivicSiteAnnieECaseyFoundation
ChrisKingsleySeniorAssociateAnnieECaseyFoundation
KathrynOliverAssociateProfessorLondonSchoolofHygieneandTropicalMedicine
Thankyou
61
KristenWatkinsSeniorPolicyAssociateJ-PAL
GovExParticipants
GovExAnalystMiriamMcKinneyanalyzedtheinterviewtranscriptsandconductedtextanalysisSeniorImplementationAdvisorEmilyShawwillintegrateRoadmapactivitieswithadditionalservicedeliveryactivitiesatGovEx
Thankyou
62
ReferencesAbrahamKGampHaskinsR(2017)Thepromiseofevidence-basedpolicymakingReportofthecommissiononevidence-basedpolicymakingWashingtonDCCommissiononEvidence-basedPolicymakingBigmove2Extendingthegreen(nd)Retrievedfromhttpswwwcityoflewisvillecomdoing-businesslewisville-2025big-move-2-extending-the-greenBatesMAampGlennersterR(2017)ThegeneralizabilitypuzzleRetrievedfromhttpsssirorgarticlesentrythe_generalizability_puzzleBuildwithnotfor(2018January)RetrievedfromhttpsbuildwithnotforwordpresscomCairneyPampOliverK(2017)Evidence-basedpolicymakingisnotlikeevidence-basedmedicinesohowfarshouldyougotobridgethedividebetweenevidenceandpolicyHealthResearchPolicyandSystemsdoi101186s12961-017-0192-xCairneyPOliverKampWellsteadA(2016)TobridgethedividebetweenevidenceandpolicyReduceambiguityasmuchasuncertaintyPublicAdministrationReview76(3)399-402Retrievedfromhttpdxdoiorg101111puar12555Centreforpublicimpact(nd)RetrievedfromhttpswwwcentreforpublicimpactorgCernaL(2013)ThenatureofpolicychangeandimplementationAreviewofdifferenttheoreticalapproachesOrganisationforEconomicCo-OperationandDevelopmentRetrievedfromhttpswwwoecdorgeduceriTheNatureofPolicyChangeandImplementationpdfColoradoSprings(nd)Retrievedfromhttpscoloradospringsgovcommunicationspagewhat-expect-when-you-speakupContinuumofcare(coc)program(nd)RetrievedfromhttpswwwhudexchangeinfoprogramscocCouncilpresidentkicksoff$12millionyouthfund(nd)Retrievedfromhttpwwwbaltimorecitycouncilcomcontentcouncil-president-kicks-12-million-youth-fundDurlakJAampDuPreEP(2008)ImplementationmattersAreviewofresearchontheinfluenceofimplementationonprogramoutcomesandthefactorsaffectingimplementationAmericanJournalofCommunityPsychology41327-350Retrievedfromhttpswwwncbinlmnihgovpubmed18322790Fishbonediagram(ishikawa)(nd)Retrievedfromhttpasqorglearn-about-qualitycause-analysis-toolsoverviewfishbonehtmlFivewhysGettingtotherootofaproblemquickly(nd)RetrievedfromhttpswwwmindtoolscompagesarticlenewTMC_5Whtm
References
63
FligsteinNampMcAdamD(2011)TowardageneraltheoryofstrategicactionfieldsSociologicalTheory29(1)1-26RetrievedfromhttpsociologyberkeleyedusitesdefaultfilesfacultyfligsteinFligstein20McAdam20Soc20Theory20PaperpdHeinkelP(2018January12)CitymullspermitsforbikesharingcompaniesRetrievedfromhttpwwwdentonrccomnewsdenton20180112city-mulls-permits-bike-sharing-companiesHowisareamedianincomecalculated(2012October)Retrievedfromhttpswwwhudexchangeinfofaqs843how-is-area-median-income-calculatedLampkinLMampHatryHP(2003)KeystepsinoutcomemanagementUrbanInstituteRetrievedfromhttpswwwurbanorgresearchpublicationkey-steps-outcome-managementMarshDampSmithM(2000)UnderstandingpolicynetworksTowardsadialecticalapproachPoliticalStudies484-21Retrievedfromhttpjournalssagepubcomdoiabs1011111467-924800247McCannL(nd)RetrievedfromhttplaurenellencomMoultonSampSandfortJR(2016)ThestrategicactionfieldframeworkforpolicyimplementationresearchPolicyStudiesJournal45(1)Retrievedfromhttponlinelibrarywileycomstore101111psj12147assetpsj12147pdfv=1ampt=jdkhhcp4amps=e184c625e1855a6b1615ccb99ee27b1c0b3212d8PaynterB(2018January1)What100citiesarelearningfromeachotherbysharingtheirdataRetrievedfromhttpswwwfastcompanycom40518004what-100-cities-are-learning-from-each-other-by-sharing-their-dataPeakacademy(nd)Retrievedfromhttpswwwdenvergovorgcontentdenvergovenmayors-officeprograms-initiativespeak-performancepeak-academyhtmlPinnockHBarwickMCarpenterCREldridgeSGrandesGGriffithsCJRycroft-MaloneJMeissnerPMurrayEPatelASheikhAampTaylorSJC(2017)Standardsforreportingimplementationstudies(stari)ExplanationandelaborationdocumentBMJOpenRetrievedfromhttpbmjopenbmjcomcontentbmjopen74e013318fullpdfProctorESilmereHRaghavanRHovmandPAaronsGBungerAGriffeyRampHensleyM(2011)OutcomesforimplementationresearchConceptualdistinctionsmeasurementchallengesandresearchagendaAdministrationandPolicyinMentalHealth38(2)65-76doi101007s10488-010-0319-7TandonSD(2018)ProgressincommunityhealthpartnershipsResearcheducationandactionRetrievedfromhttpswwwpressjhuedujournalsprogress-community-health-partnerships-research-education-and-actionProvanKGampKenisP(2007)ModesofnetworkgovernanceStructuremanagementandeffectivenessJournalofPublicAdministrationResearchandTheory
References
64
18(2)229-252Retrievedfromhttpsacademicoupcomjpartarticle182229935895SabatierPA(1988)Anadvocacycoalitionframeworkofpolicychangeandtheroleofpolicy-orientedlearningthereinPolicySciences21(3)129-168Retrievedfromhttpwwwjstororgstable4532139SandstromAampCarlssonL(2008)TheperformanceofpolicynetworksTherelationbetweennetworkstructureandnetworkperformancePolicyStudiesJournal36497-524Retrievedfromhttponlinelibrarywileycomdoi101111j1541-0072200800281xabstractSchwartzMampWilsonE(nd)WhocanaffordtoliveinahomeAlookatdatafromthe2006americancommunitysurveyRetrievedfromhttpswwwcensusgovhousingcensuspublicationswho-can-affordpdfShearerJCAbelsonAKouyateBLavisJNampWaltG(2016)WhydopolicieschangeInstitutionsinterestsideasandnetworksinthreecasesofpolicyreformHealthPolicyandPlanning31(9)1200-1211Retrievedfromhttpsdoiorg101093heapolczw052ThelessonsofadministrativedataHigh-profilepolicy-relevantresearchpoweredbyadministrativedata(nd)Retrievedfromhttpswwwpovertyactionlaborgsitesdefaultfilesresources20161114-The-Lessons-of-Administrative-DatapdfToolsandotherneedsforqualitativeandquantitativemethods(2015February)AgencyforHealthcareResearchandQualityRetrievedfromhttpwwwahrqgovprofessionalsquality-patient-safetytalkingqualityassesstoolshtml
WeaverRCampBagchi-SenS(2013)SpatialanalysisofurbandeclineThegeographyofblightAppliedGeography4061-70
References
65
UniversityPartners
Name Title Affiliation Area(s)ofexpertise
Contactinformation
Examplesofworkingwithnon-academicdecisionmakers
WendyPBarreno
AssociateDirector
WilsonSheehanLabforEconomicOpportunitiesUniversityofNotreDame
LEOsfocusareasincludehealtheducationhousingchildwelfareeconomicopportunitycriminaljustice
wbarrenonduduorleondedu
MelissaDavey-RothwellPhD
AssociateScientist
JohnsHopkinsBloombergSchoolofPublicHealth
Health mdavey1jhuedu410-614-5854
BaltimoreCityHealthDepartmentLeadersMarylandDepartmentofHealthAddictionRecoveryServicesinDCExecutiveDirectorsofcommunityorganizationsacrosstheUSthatfocusonHIVSTIsubstanceabuseandimplementationscience
ElizabethStuart
AssociateDeanforEducationandProfessorDepartmentsofMentalHealthBiostatisticsandHealthPolicyand
JohnsHopkinsBloombergSchoolofPublicHealth
Educationsocialpolicystatisticalmethods
estuartjhuedu
Resources
66
Management
Design-ThinkingResources
EngagementGuideforCivicTech
Journeymap
PerformanceManagementResources
DenverPeakAcademy
Proxymeasuresguide
Talkingaboutperformancewithoutcomplicatingit(video)
SettingPerformanceTargetsguide
Datavisualizationguide
Performancemanagementgettingstartedguide
Exercises
Fishboneexercise
5Whys
AdministrativeData
TheLessonsofAdministrativeData(JPAL)
EvaluationResources
Quantitativeandqualitativemethods(USDepartmentofHealthandHumanServicesAgencyforHealthcareResearchandQuality)
KeyStepsinOutcomeManagement(UrbanInstitute)
RCT-YES
Sageresearchmethods
BestPracticesforMixedMethodsResearchintheHealthSciences
ACCORDChecklist
Pilotandimplementationplanning
Benchmarkingexamplesandproxymeasures
Seattleopendataplaybook
Resources
67
Datasciencescopingguide
Resources
68
AboutGovExGovExhelpsgovernmentsusedataandevidencetoimproveresidentsrsquolivesWesupportgovernmentsinbuildingtheirinternalcapacitytooperatesustainableanalyticspracticeswhichimprovecommunityoutcomesinfieldsfrompublicsafetytoeconomicdevelopmentGovExhasaproventrackrecordofworkinginconcertwithpartnerstoprovidetechnicalassistanceandtrainingatscale
SinceourlaunchinApril2015GovExhas
Trainedover2000governmentemployeesinpersonandonlineintopicsincludingdatamanagementperformancemanagementopendatacommunityengagementandanalytics
Providedtechnicalassistancetomorethan100localgovernmentsacrosstheglobeimprovingtheiruseofdatatoachieveefficienciesinservicedeliveryandbetteroutcomesforresidents
Assessedthecapacityofmorethan130governmentalorganizationstodeterminehowleadersstaffandexternalstakeholdersmanageandusedata
Localgovernmentsareincreasinglyonthefrontlinesofaddressinglargescaleproblemswithglobalimplicationssuchasclimatechangepublicsafetyandeconomicinequality
Whileevidence-basedinterventionstohelpsolvethoseproblemsareavailablelocalgovernmentstaffhavelimitedtimeandcapacitytofindandimplementresearch-basedsolutionsAsaresultcitiescontinuetoworkthewaytheyalwayshave--makingincrementalimprovementstoprogramsbutnotalwayscreatingchangeatasystem-level
ThroughresourcedevelopmentandcollaborationwithJHUfacultyGovExissupportingcityleadersinusingdataandevidencetomakedecisionsandcreatinglastingrelationshipswithresearchers
TheRoadmapwasdevelopedbyGovExaspartofitspartnershipwithBloombergPhilanthropiesWhatWorksCitiesinitiative
AboutGovEx
69
- Title page
- Introduction and purpose of the Roadmap
- Using the Roadmap
-
- Gather information about the problem
- Strategically engage key partners
- Review the evidence and collect details
- Decide on a policy to implement based on local factors
- Establish a framework to monitor progress
- Assess data accessibility quality and security
- Design implementation and evaluation
- Share your story and success
-
- Working with a range of evidence
- Thank you
- References
- Resources
- About GovEx
-
IntroductionRemembertheDAREprogramWhatbeganasapartnershipbetweentheLAPDandLApublicschoolstohelpkidssaynotodrugsbecameapoliticallypopularcampaignthatwasadoptedbyschooldistrictsaroundthecountryMeanwhilethroughthe1990sand2000smorethan30rigorousevaluationsfoundthattheprogramdidnotwork-withonestudyevenfindingthatitactuallyincreasedyouthdruguseThreedecadeslaterthefederalgovernmentisstilltalkingaboutrevivingthispopularbutdetrimentalprogram
Thechallengesthatcitiesface
ThestoryofDARErevealssomeoftheotherchallengesfacinggovernmentsinimplementingevidence-basedpoliciesPresentedwithconstraintsontimeandpressuretoactpolicymakersmayrelyonemotionsandgutfeelingsasashortcuttodecisionmakingItcanbehardtorejectthefundingandpoliticalcapitalthatoftencomeswithaprogramthathasthekindofmomentumthatDAREhadEvenifelectedofficialsadmitthatapolicyisproblematicoratleastnolongereffectiveittakesleadershiptoputenergyandresourcesintochangingcourseAndeventhencityleadershavetomakethecasetoresidentsforwhyachangeisnecessaryespeciallyaftertheyhaveinvestedtimeandpublicdollarsintoapreviousapproach
LocalgovernmentsareincreasinglyonthefrontlinesofaddressinglargescaleproblemswithglobalimplicationssuchasclimatechangeimmigrationandeconomicinequalityAlthoughresearchersarevalidatingdiagnosticmodelsandevidence-basedinterventionstohelpsolvethoseproblemslocalgovernmentstaffhavelimitedtimeandcapacitytofindevaluateandimplementthesesolutionsSearchesbycitystaffmayleadtopromisingexamplesinothercitieslinkstoscholarlyarticleswebsitesofnonprofitsthatworkonthisissuejournalarticlesandrelatednewsarticlesRatherthanweighingandevaluatingthisinformationcitiesmaychoosetotryonepolicysolutionthatseemsfeasibleWithoutcarefulmonitoringorevaluationthoughthistrialanderrorapproachcanleadtoanenvironmentinwhichcitiesfundserviceprovidersbasedonexistingrelationshipsandavoidrepurposingfundingfromprogramsthatareineffectiveforfearofrepercussionssuchaspeoplelosingtheirjobsFurtherconversationsaboutwhatothercitiesaredoingandtheavailableliteraturesometimeshappenonanadhocbasiscitystaffdonotoftenfeelpreparedtoconductasystematicreviewWhencitiesdomakechangestheyaresometimesincrementalandattheprogramlevelinsteadofaddressingmultiplefactorsatonceatasystemlevelAttemptingtoaddressaproblemthatstoobroadsuchasblightcanbe
IntroductionandpurposeoftheRoadmap
3
dauntingWithoutadeepunderstandingoftherootcausesoftheproblemitcanbedifficulttoidentifythebestevidencetouseFinallyitcanbedifficulttoknowhowtoimplementaparticularpolicyinyourcityandwhetheritwillhavethedesiredeffect
WhentheUnifiedGovernmentofKansasCityKSandWyandotteCounty(UG)beganworkingonitsblightproblemtheUGconvenedaresearchteamtolookforevidencefromothercitiesUsingsearchtermslikeblightlandbankabandonedpropertiesandvacancythatledtheUGtolearnaboutthesimilareffortsinthecitiesofNewOrleansBaltimoreMobileKansasCityMOandcitiesinOhioandMemphisTheteamalsoremainedopentohearingideasfrominternalstaffwholearnedaboutpracticesfromothercitiesthroughtheirownprofessionalorganizationsanduniversityaffiliationsthattheyhadnotbeenempoweredtotestorimplementpreviouslySincethetopicofblightrangesfrompropertymaintenancetosafetyissuestheCityresearchteamhadtodoalotofreadinganddebatingoverconflictingconceptsfortacklingawiderangeofissueswithinthatspectrumThisinvolvedaskingquestionsaboutwhetherthepolicywillworkinKansasCityandwhatchangeswouldneedtobemadeforittobesuccessfulWhenresearchinglandbankstheCitylearnedabouttheCenterforCommunityProgressanonprofitthathelpscommunitiesaddressvacantabandonedanddeterioratedpropertiesConversationswiththeCenterforCommunityProgressledtoexploringtheideaofavacantpropertyregistryUponconductingfurtherresearchUGstaffcameacrossinformationonacourtrulingfromChicagothatcertainvacantbuildingsdonotneedtofollowtheCityofChicagorsquosvacantbuildingordinanceThisdirectedthesearchtounderstandifothercitieshadexamplesofwhethervacantpropertyregistriesareworkingTodeterminewhetherthismightworkinthecontextofKansasCitystaffbeganvettingideastoseeiftherewereanyaspectsthatconflictedwithCityordinancesTheUGalsoexaminedthefeasibilityofimplementingthenecessarytechnologyandfeasibilityfromtheperspectivesofpropertyownersandUGstaffThisinitialresearchandanalysisphasetookaboutsixmonthsandwasledbytheGeneralServicesDirectoraVISTAinternandtheresearchteamIndevelopingthisRoadmaptheCitymentionedthatitwouldbehelpfultohaveaccesstocasestudieswithdetailedinformationonthepolicysolutionsthatcitieshavetakenandthespecificsofhowtheyhavebeenimplemented
PurposeoftheRoadmap
ThisRoadmapismeanttoempowercitiestotakeownershipofthepolicychangeprocesstoframeproblemsinawaythatcreatesademandforevidenceandtoformlong-termrelationshipswithpoliticalpolicypractitionerandresearcheralliesThisworkisnecessaryResearchfromthefieldofimplementationsciencesuggestssuccessfulimplementationofevidenceintopracticehappenswhencitiestakethetimetoidentifytherightpeopletoworkwithchoosethebestevidencefromavarietyofsourcesandconsiderthebestwaytoadapt
IntroductionandpurposeoftheRoadmap
4
thatevidencetotheirowncontextThroughourinterviewswithcitystaffweobservedthatframingtheproblemclearlyformingadvisorygroupsorsomeothercoalitionandidentifyingandimplementinganappropriatesolutionarenecessaryingredientstosuccessfulpolicychange
ThroughthisRoadmapGovExhopestofacilitateanewwaytohelpAmericancitiesrethinkthehowtheysolveproblemsandaddressthemostsalientpolicychallengesinurbangovernance
IntroductionandpurposeoftheRoadmap
5
UsingtheRoadmap
WhoshouldusethisRoadmap
ThisRoadmapisforanystaffmemberincitygovernmentwhoischargedwithsolvingaproblemorimplementingapolicyAlthoughthestepscanbeusedtochangeexistingprogramsorimplementnewprogramsthefocusoftheRoadmapisonlong-termsustainablepolicychangeWeusethetermpolicytomeanlawsregulationsfundingdecisionsandinterventions
CitystaffwhoarereadingthisguidemaybecomingfromdifferentstartingpointsItisimportanttobehonestaboutwherethatisForinstanceyoumaybelookingfornewsolutionstolongstandingseeminglyintractableproblemssuchasanannualincreaseingunviolenceormaybeyouarefacingnewerchallengesliketheopioidepidemicOrperhapsyouarelookingforadata-drivenwaytoaddressissuesrelatedtoday-to-dayservicedeliverysuchaspermittingorbikelanesThestepsareapplicablewhetheryouareworkingonalargescaleissueoralocalmunicipalissue
YoucanusethisguidewhetheryouareadecisionmakerorfrontlineworkerThroughouttheRoadmaptheresinformationaboutwhenandhowtoengageotherpeopleCollaborationisakeythemethroughthisRoadmapThegoodnewsisyoudonothavetodothisworkalone
TheapproachandthetoolscontainedintheRoadmapshouldberelevantwhetheryouareinastrongmayororcouncil-managerformofgovernmentElevatingthisprocesstothelevelofMayorCityManagerorCityCouncilmaytaketimeandmultiplestepsbutwehopeyouwillfindwaystoincorporatedataandevidenceintoyourday-to-daywork
WhatarethestepsoftheRoadmap
ThechaptersinthisRoadmaprepresentthehigh-levelstepsinvolvedinpolicychangeYouwillnoticethatthesectionsrefertooneanotherandbuildonthepreviousstepsTheRoadmapismeanttobeiterativeyouwillrevisitsomesteps
TherootsoftheproblemsthatcitiesareworkingtosolvearecomplexandmaybethelegacyofhistoricalfederalpolicyorsweepingsocialtrendsAlthoughthisRoadmapdoesnotexplicitlyincludestepsonreviewingthishistoryweencouragereaderstodevelopadeepunderstandingoftheprobleminyourowncitywhileconsideringcurrentstateandfederalcontext
UsingtheRoadmap
6
IninterviewswithGovExonthepolicychangeprocessintheircitiespeoplewasthewordmostcommonlyusedbyintervieweeswhendescribingthevariousstepstoimplementanewpolicyorprogramYouwillnoticethatthisisathemethroughoutthisRoadmapThefirstsectionbeginswithidentifyingtherelevantpeopleThechapteronidentifyingrolesforkeypartnersservesasafoundationforthefollowingstepsThediscussionofevidenceencouragesreaderstoconsiderwaysinwhichpeoplecanbuildanduseevidencetodesignstrongprogramsandinitiativeschangehumanbehaviorassessimpactensureeffectiveuseofpublicresourcesandimprovelives
Meaningfulapplicationofevidencebypolicymakersdoesnothappenwithone-timesharingofcuttingedgeevidenceRathermeaningfulinfluencerequiresalong-termstrategytoformalliancesanddevelopknowledgeofpolicymakersandpolicymakingInadditiontoworkingwiththerightpeopleandunderstandingevidenceandcontextwehopethatRoadmapuserswillformapeernetworkthatsharesnotjustwhichpolicysolutionsexistbutwhatimplementationlookslikeAsonecitywespoketoindevelopingthisRoadmapputitalotofourcitysproblemshavebeensolvedbyotherswejustneedtofigureouthowtoimplementthesolutions
TheexamplesthroughouttheRoadmapareillustrativeofgovernmentsthathaveeffectivelyimplementedtherelevantstepsinthisRoadmapbutpleasenotethatnocitydoesallofthisperfectlyTogetherthestepsinthisRoadmaparemeanttohelpcitiesframetheirproblemscreateademandforevidenceandbuildsustainablecoalitionsformeaningfulpolicychange
UsingtheRoadmap
7
GatherinformationabouttheproblemInyourearliestphaseitiscriticaltodevelopacomprehensiveunderstandingoftheproblemYouwillwanttolearnaboutthepeoplewhoareaffectedbytheproblemyouaretryingtosolveandthosewhocanimpacttheproblemFromthereyouwillgetinsightintotherootcauseoftheproblemtheworkthathasbeentriedinthepastandtheeffectivenessofthoseeffortsTheworkfromthischapterwillserveasthefoundationforactivitiesinsubsequentchaptersYoumayrevisitthesestepsasyouidentifyindicatorsandplanyourimplementationandanalysis
BesuretodedicatetimeandresourcestothisstepItcanbetemptingtoskipaheadtolookingforsolutionsortoassumethatyouhaveanunderstandingoftheproblemasaresultofapublicengagementprocess
TheCityofLewisvilleTXconductedaCitywidepublicengagementprocesstodevelopitsLewisville2025StrategicPlanOneoftheBigMovesoutlinedintheplanwasExtendingtheGreenorexpandinggreenspaceandconnectingrecreationareaswithneighborhoodsTheactionstepstoachievethisgoalincludedcleaningupcreeksandremovingdebristomakethemmoreaccessibleThecleanupswentsmoothlyinseveralneighborhoodsbuttheCitymetwithoppositionfromoneneighborhoodthatpreferredthenaturalhabitatandexpressedconcernabouttheenvironmentalimpactofthecleanupsAccordingtoLewisvilleiftheCityhadmappedallofthestakeholderstodeterminewhoneededtobeatthetableandwhowasmissingfocusinginformationgatheringontheneighborhoodlevelithaveavoidedthisproblem
Startwiththewhothepeople
Establishaprojectteam
BeginbynamingtheprojectteamforthisprocessincludingaprojectleadTheprojectleadshouldbesomeoneinCitygovernmentwhoisrespectedcangalvanizeotherkeypartnersandhasthetimetoleadthisworkThisissomeonewhomayhavetohavedifficultconversationswithCityleadershipdepartmentsandotherfundedpartnersItisnotnecessaryfortheprojectleadtobeapolicyexpertItismoreimportantforthemtohavetherightrelationshipsandpositionwithintheorganizationTheprojectteamshouldincludeindividualswithdiverseperspectivesfrommultipledepartmentsThereshouldbesomeonewithpolicyexperienceontheprojectteamtogroundtheconversationinrealisticactionHavingateammemberwhocancoordinatewithresearchersonrigorousanalysisandtranslatethatworkforothersiskeyAdditionallybesuretoincludefrontlinestaffworkersTheywillhavevaluableinsightbasedontheirinteractionswithresidentsandpolicysolutions
Gatherinformationabouttheproblem
8
havetomakesensetothepeoplewhowillbechargedwithimplementingthemBuildingastrongdiverseprojectteamisimportanttoensurethattheworkdoesnotliveanddiebyoneperson
Identifykeypartners
NextidentifythepeoplewhotheprojectteamcanworkwithontheissueyouarefacingincludingpeoplewithlivedexperienceintheproblemyourcityistryingtosolvedepartmentstaffandsubjectmatterexpertsoutsideofcitygovernmentThisistheconceptbehindbuildwithnotforamovementledbyLaurenellenMcCannthatemphasizescommunityexpertiseandleadershipMappingoutkeyplayersandtheirrolesiscriticaldotoearlyintheinformationgatheringphasesincecoalitionbuildingissuchanintegralpartofthepolicychangeprocessThisprocessmightconsistofstakeholdermapping(describedbelow)communityresourcemappingandculturalmappingInyourmappingremembertoconsiderpotentialalliesandthosewhomaypresentroadblocksalongtheway
PolicymakingisnottopdownorlinearInsomecasescitiesmayhavetosharedecision-makingpowerwithotheractorsForinstanceinmanycitiesthelocalContinuumofCare(CoC)istheleadentityonhomelessnessservicesCoCsapplyforandstewardfederalfundstoaddresshomelessnessandcoordinateservicesamonglocalprovidersAlthoughcitygovernmentsdohavearoleinCoCstheyarenotalwaystheleadagencychargedwithdistributingfundsandthereforetheirroleinhomelessnesspolicymaybelimitedCountygovernmentsoftenhavehealthdepartmentswhilecitiesdonotInthosecasesthebalanceofpowerfavorscountygovernment
TheCityofDentonTXwasrecentlyfacedwithanissueofresidentcomplaintsaboutanewdocklessbikesharethatarrivedintownandisviewedbysomeasexpensivelitterInresponseDentonsCityCouncilwhichwasalreadydiscussingpotentialrulesforbikesharecompanieselevatedthisissueandchargedcitystaffwithidentifyingasolutionPresentlytheCityisreviewingordinancesfromnearbycitiestodeterminehowtheyaresolvingthisproblemandwhatsolutionsmaybefeasibleinDentonThemappingexamplebelowrepresentslocalactorsinDentonwithastakeinthisissuewhohavemultipleperspectivesandlevelsofpower
Gatherinformationabouttheproblem
9
UsethistemplatedevelopedbytheCentreforPublicImpact(CPI)tobrainstormwhoyoumightreachouttoasyouembarkontheroadtopolicychangeRemembertothinkabouthoweachpersonorgrouprelatestotheproblem
BegintodefinethewhatortheprobleminyourCity
TherestofthissectionwillhelpyourefinetheproblemstatementFornowgathersomekeypartnerstoclearlyarticulatewhychangeisneededUsethetemplatebelowdevelopedbyDenverPeakAcademytocapturetheproblemstatementYouwillbuildonthislater
ProblemStatementndashWhyChangeisNeeded(1-2Sentences)
ThroughourinterviewswithcitystaffweheardthatwhendiscussingmunicipalchallengesitcanbetemptingtojumptoasolutionbeforetakingthetimetoreallyunderstandtherootcauseoftheproblemThismakessensesincecityleadersandstaffareresponsiblefordeliveringservicesandbeingresponsivetoresidentswhileaddressingnewproblemsthatariseinthecity
TherootsoftheproblemsthatcitiesareworkingtosolvearecomplexandmaybethelegacyofhistoricalfederalpolicyorsweepingsocialtrendsTohelptheproblemseemlessdauntingitcanhelptobreakthemdownintotherootcausetoidentifyamanageablecomponentoftheproblemUsingafishboneexerciseorfivewhystechniquecanalsohelp
Gatherinformationabouttheproblem
10
tofindtherootcauseoftheproblemandavoidsimplyaddressingsymptomsoftheproblemandrepeatingthemistakesofhistoryToolssuchasjourneymapscanhelpyouunderstandtheexperienceofthosewhoareaffectedbytheproblem
TheCentreforPublicImpactwithsupportfromBloombergPhilanthropiesdevelopedanexercisetosupportcitiesinidentifyingtherightsizedproblemAccordingtoCPIarightsizedproblemisonethatislargeenoughtopotentiallyapplytoothercitiesbutsmallenoughtosolvewithinareasonabletimeframe
AtthebeginningofMayorBerkesadministrationintheCityofChattanoogaTNveteransexperiencinghomelessnesswereaskingtheCityforhelpinfindinghousingTalkingtothispopulationrevealedthatveteranshadtroubleidentifyingaffordablehousingwhichmotivatedtheAdministrationtoworkonasolutionTheproblemofveteransexperiencinghomelessnesswastherightsizebecauseitfocusedononesegmentofpeopleexperiencinghomelessness-veterans-andtheirspecificchallengeinidentifyingaffordablehousing
Developacomprehensiveunderstandingofhowyourcitycurrentlyaddressestheissue
IntheChattanoogaexamplementionedaboveCitystaffdiscoveredthatveteranswerehavingtroublenavigatingtheCitysvariousaffordablehousingprogramsandgettingaccesstothehousingandservicesthattheyneededAtthesametimenewlyelectedMayorBerkewasexploringcurrentandfutureopportunitiestoservethispopulationRatherthancontinuetousefederalfundingforemergencysheltersMayorBerkedirectedcitystafftoprioritize
Gatherinformationabouttheproblem
11
HousingFirstwhichvaluesconnectingpeoplewithpermanenthousingovertransitionalhousingandsupportiveserviceafterspeakingwithrepresentativesattheUSDepartmentofHousingandUrbanDevelopment(HUD)
ThisexaminationofwhattheCityiscurrentlyfundingandthesourcesofthatfundingwasaturningpointinChattanoogaUnderstandingthisproblemfromasystemsperspective-includingpoliciesandfundingstreamsatmultiplelevelsofgovernmentsocialnetworksthroughwhichinformationisoftenspreadtechnologythebuiltenvironmentofmarket-rateandpublichousing-helpedthenewMayoranalyzeexistingeffortsandleadstaffandotheractorsinthesamedirection
Begintogatherthedata
TotrulycapturethesuiteofservicesthatyourCityisprovidingaskdepartmentstafftolisttheprogramsrelatedtoyourproblemstatementthatyourcityrunsorfundsWhenlistingtheprogramscapturedetailssuchas
programnamethenameofthedatasetthatisrelevanttoeachprogramadescriptionofthedatasetthesourceofthedatasetifthedataisinternalorexternalwhetherthisdataisalreadybeingcollected
ThisinformationwillformthebasisofyourinventoryofadministrativedataordatasetsthatarekeptbygovernmentsforoperationalpurposesThismayincludecallcenterrecords(311911)taxdatapropertyassessmentsinspectioninformationsocialservicerecordsetcAtthispointlistallrelevantadministrativedatasetsYouwillprioritizewhichdatasetstogathermoreinformationonlaterwhenyouaredevelopingindicators
Asampletemplatethatcanbeusedtocapturethisinformationisbelow
ProblemStatement____________________________________________
Gatherinformationabouttheproblem
12
ProgramName
DatasetTitle
DatasetDescription DataSource
Departmentorexternalpartner
DoestheCityalreadycollectthisdata
Whatisthenameoftheprogram
Human-readablenameofthedataset
Shouldbeunderstandablebynon-technicalusers
Whatistheoriginalsystemapplicationfilethathousesit
Departmentorexternalpartnerthatmaintainsthedata
Isthisdataalreadycollected
TogatherthisinformationconsiderassigningonepointpersontosurveyorinterviewcitystaffSpeakingwithfrontlineworkersstaffwhoworkwithfederalgrantsdepartmentheadsandeveryoneinbetweenwillensureadeepunderstandingoftherelevantprogramsanddatasetsDuringthecourseoftheseconversationsbesuretoaskcitystaffwhichorganizationstheyworkwithexternallyandcapturethatinformationaswelleveniftheyarenotrecipientsofcityfundingInadditiontoconversationswithexpertsincitygovernmentyourcitysbudgetdocumentsareanothergreatsourceofinformation
Collectdataandobservetrends
Nexttrytogainanunderstandingofwhytheproblemstillexistsifithasbeensomethingyouhaveworkedonforawhileorwhyitisemergingnowifitisnew
AnalyzingadministrativedatacandemonstratepasteffectivenessofinitiativesrelatedtoyourproblemstatementUsingthedatasetinventorythatyoubeganintheprevioussteptalktodepartmentstaffaboutwhichdatasetsaremostrelevanttotheproblemandgatherafewyearsworthofhistoricaldataBegintolookfortrendsinthisdatadiscussingpotentialreasonsforthosetrendswithrelevantstaffCanthetrendsbeexplainedbyseasonalitychangesinfundingstaffingfluctuationsorsomeotherreasonBasedonthehistoricaldataanalysiscreateabaselineagainstwhichfuturechangescanbeobserved
OtherdatasetscanhelpprovideadditionalcontextForinstanceifthenumberofpeopleexperiencinghomelessnessinyourcityisincreasingitmaybehelpfultotakealookathousingaffordabilityIfyoudonothavethedatayouneedconsideraproxymeasureAsanexamplehousingisoftenconsideredaffordableifitcosts30orlessofahouseholdsincomeButifhouseholdincomedataisnotavailableatthecitylevelAreaMedianIncome(AMI)canbeahelpfulproxy
Gatherinformationabouttheproblem
13
QualitativedatacanalsohelpanswerquestionsaroundtheeffectivenessofpreviouseffortsTrytoobtainavarietyofperspectives--bothmainstreamandoutliers--fromnetworksbeyondyourowntoprovideyouwiththegreatestvantagepointforevaluatingpotentialimpactUnderstandinghowthoseeffortswerereceivedbydifferentpartsofyourcommunityandiftheremaybeelementsofprevioussystemstopreservewillbecrucialtothesuccessofanypolicychangeThisinformationmaybeobtainedthroughfocusgroupsinterviewsandsurveysFormoreinformationonhowtousethesequalitativeresearchtoolspleaseseetheseresourcesdevelopedbytheAgencyforHealthCareandResearchandUrbanInstitute
FinallytakealookatnationaldatatoseewhereelsethisproblemmightexistThisresearchmaybehelpfulinlaterstepswhenyouarelookingforpromisingpoliciesinothercities
Puttingitalltogether
UsethetemplatebelowdevelopedbyDenversPeakAcademytocaptureyourrefinedrightsizedproblemstatementandcurrentandfuturestatesWorkingthroughthisexercisewithpartnersishelpfultogainbuy-inTheresponseswillbeimportantwhengettingotherpartnersonboard
WhyChangeIsNeeded
Yourright-sizedproblem
CurrentState
Analysisoftrendsindata
FutureState
Ideallywhataretheoutcomesyouhopetoachieve
Checklist
Thekeystepstogatheringinformationarebelow
Atthispointfocusontheproblemratherthanjumpingtosolutions
Listentopeoplewhoareaskingforhelpandtoadvocateswhoareraisingissues
Beasspecificaspossibleinidentifyingtherootcauseandhoninginonthepartoftheproblemthatyourcitycanaddress
Identifytrendsbasedonmultiplesourcesofdataincludinghistoricaladministrativedataproxymeasuresqualitativedataandnationalmeasures
Gatherinformationabouttheproblem
14
Captureyourproblemstatementtheresultsofyouranalysisanddesiredoutcomes
Gatherinformationabouttheproblem
15
StrategicallyengagekeypartnersPolicytheoryresearchoftenfocusesonincreasingpolicymakersaccesstoevidenceasawaytoavoidtakingshortcutslikemakingdecisionsbasedonemotionImplementationsciencesuggeststhatincreasingaccesstoevidenceisnotenoughandthatpolicychangeshouldbeexaminedinthewidercontextofdebatecoalitionformingandpersuasionMeaningfulpolicychangerequiresldquoalong-termstrategytoformalliancesanddevelopknowledgeofpolicymakersandpolicymakingratherthanusingevidenceinpolicymakingonanadhocorone-offbasis
UnderstandingnetworksandsystemsisimportanttodevelopsustainablepolicychangeThissectionismeanttohelpreaderstakethestepstobuildengagementstrategiesforpartnersthatrepresentdifferentsystemsandnetworksEngagementshouldbeanongoingprocessofconnectingpeopletooneanotherandtoanissueEngagementshouldnotbeinnameonly-wehaveallexperiencedengagementactivitiesthatgivepeopleaspecificroletoplaythatleadstoaneatpredictableoutcomeAccordingtotheBuildWithmovementundervaluingandunderinvestinginengagementstrategiescanleadtoadisconnectofcertainindividualsorgroupsandmistrustofgovernmentWhendevelopingengagementactivitiesrememberthathowpeopleengageandwhethertheyengagehastodowithhowyoustructureopportunitiesforengagement
Conveneacoreadvisorygroup
AnadvisorygroupcomprisedofpeoplefrommultiplesectorscanbehelpfulasyouconsiderthelocalsocialenvironmentalandpoliticalcontextinwhichyouintendtointroducepolicychangeRevisityourstakeholdermapanddeterminewhowouldbebesttoserveonthisgroupTomakesureyouareconsideringtheproblemsandsolutionsthroughanequitylensincludepeoplewithlivedexperiencewhohaveexpertiseintheeffectivenessofsystemswhetherthosesystemsfunctionasintendedandhowtheymayimpacthumanbehaviorYoumayalsowishtoincludesomeonewithexperienceworkingwithdataandanalysis-thispersoncanhelpdeterminethetypeofdatathatmaybereadilyavailableandusefultocollectandcanalsohelpplanandshapeyouranalysistofigureoutwhichprogramsareeffectiveBesuretoinvitefront-lineworkersaswell-itscriticalthatwhateversolutionscomeoutofthisprocessaremeaningfultothosewhowillberesponsibleforimplementingthem
BringingpeopletogethercanbechallengingbutthereareafewwaystosetandmanageexpectationsFirstmakesurethatthegroupisawareoftheproblemstatementcurrentstateandfuturestate-thiscanhelpstarteveryoneoffonthesamepageNextspend
Strategicallyengagekeypartners
16
sometimegettingtoknoweachotherrsquosworktolearnabouteveryonesstrengthsThissharedunderstandingcanhelpasyouclarifyrolesandresponsibilitiesofeachpartnerinthegroup--thisisimportanttoavoidfinger-pointinginthefutureandtoensurethatnoneofthetasksslipthroughthecracksInsomecasestherearemanyactorswithdifferentresponsibilitiesasinthecaseofBaltimoreCitys38-memberYouthFundTaskForcewhichischairedbynonprofitandcorporatepartnersandincludesrepresentativesfromthenonprofitandfaithcommunitiesstudentsthebusinesssectorlocaluniversitiesCityCouncilandtheMayorsOffice
OncerolesandresponsibilitiesareclearyoumayconsiderdevelopingacharterguidelinesortermsofreferencetosetexpectationsforthegroupBesuretodistinguishthisfromotherboardsandcommissionsthatyourcitymayhaveNextyoumaywishtoestablisharegularongoingmeetingscheduleBesuretosendagendasandmeetingmaterialsaheadoftimeandclarifytheobjectiveofeachmeeting
InSeattletheCitydevelopeditsHousingAffordabilityandLivabilityAgenda(HALA)torespondtorisinghousingcostsandhomelessnessThisisatrendinotherwestcoastcitiesaswellwhichmaybeexplainedbyaninfluxofnewresidentswhoareattractedbytheemergenceoftechjobstheattractionoftheoutdoorlifestyleortherecentpassageofprogressivepolicieslikelegalizedgaymarriagea$15minimumwageandlegalizationofmarijuanaSeattlebroughttogetheramulti-stakeholderadvisorygroupcomprisedoflocalhousingexpertsrentersandhomeownersdevelopersandgovernmentofficialstoshareinformationabouttheirneedsanddeveloprecommendationsbasedontheseneedsTheinterestsandmotivationsofthisgroupwerevariedandittookabouttenmonthstofindcommongroundTheseriesofrecommendationsthattheadvisorygroupcreatedismeanttoserveresidentswithaspectrumofneeds
Buildaprocessforcommunityparticipation
Ofcoursenoteveryonecanbeincludedinanadvisorygroupandadvisorygroupmembersmaynotnecessarilyrepresentthevariedperspectivesoftheirpeers-forinstancenoteveryhomeownerhasthesamesetofneedsTogatherinputfromawiderswathofthepopulationitcanbehelpfultobuildaprocesstoinvitethepublicatlargetosharetheirideasCitiesdothisinavarietyofwaysfromonlineconversationsandsurveystoin-personmeetingsRememberthatbeingintentionalaboutgatheringtheviewpointsofpeopleismoreimportantthanusingthelatesttechnologyandtools
TheinsightgeneratedbycommunityparticipationexercisescanbeincludedinyourreviewofevidencewhichisdiscussedinasubsequentchapterInWestVirginiaastatethathasbeenhithardbytheopioidepidemicstatehealthofficialsdraftedanopioidresponseplanthatwasinformedbypublicengagementaswellaslocalandregionalsubjectmatter
Strategicallyengagekeypartners
17
expertsThepublicwasinvitedtopresentcommentsonideasfordecreasingoverdoseearlyinterventiontreatmentandrecoveryPublichealthandmedicalfacultyfromWestVirginiaUniversityMarshallUniversityandJohnsHopkinsUniversityprovidedresponsestothecommentsTheresultwasanOpioidResponsePlanthathealthcommissionerRahulGuptapresentedtoGovernorJimJustice
AsamplespectrumtoplanyourowncommunityparticipationprocesscreatedbyLaurenellenMcCannisbelow
SourceLaurenellenMcCann
Community-basedparticipatoryresearchisanotherwaytoworkwiththecommunitytoimproveoutcomesCBPRmeansthecommunityisinvolvedinallaspectsoftheresearchprocesswhichisofteniterativeInCBPRtraditionalresearchersandcommunitymembersshareknowledgeaswellasresourcesandcredit
CBPRInActionInSanFranciscoanenvironmentaljusticenonprofitteamedupwiththeCitysDepartmentofPublicHealthandanoutsideevaluatortounderstandtherootcausesofobesityandfoodinsecurityinlow-incomeneighborhoodsandidentifysolutionsThroughresearchmethodssuchasinterviewingresidentsandstoreownersdiagrammingstoreshelvesGISmappinganeconomicanalysisandreviewofsimilarpoliciesinnearbycitiestheteamobservedthatthestoresthatresidentshasveaccesstowerestockedwithlesshealthyfoodsandthatresidentswantedmorehealthyfoodandlessalcoholandtobaccoThisprojectresultedinlocalandstatelegislationtoencouragestorestostockfreshproduce
Strategicallyengagekeypartners
18
Developrelationshipswithsubjectmatterexperts
OnewaytokeepsubjectmatterexpertsengagedistoinvolvetheminthecommunityparticipationprocessasdescribedaboveIntheWestVirginiaexamplesubjectmatterexpertsrespondedtoandsynthesizedthepubliccommentsOtheropportunitiesaredescribedbelow
AdvisoryAcitydepartmentheadMayororstaffmembermaydevelopaone-on-oneadvisoryrelationshipwithuniversityfacultymemberoracollaborativebetweentheCityandUniversitytoaddressaparticularproblemAgreatplacetostartistolookforrelevantpracticecenterswithinyourlocaluniversityandreachingouttosubjectmatterexpertsInadditiontotheirexpertiseontherelevantpolicytopicheorshemayalsobeabletoidentifyotherexpertsandevenprovidesupportonhiringtherighttalenttoleadcertainprogramsAnadvisoryrelationshipwithauniversitysubjectmatterexpertcanalsohelpwhencitiesareconsideringbringinginnewprogramsTheseexpertsareoftenfamiliarwithprogramsthatarebasedonnationalmodelsandcanprovideinsightintoaprogramsoriginandlineageeffectivenessinotherplacesandthepotentialforimplementationinyourcityAcademicscanalsohelpidentifyindicatorsbasedongovernmentindicatorsorliteratureFinallyrelationshipsbetweenpolicymakersandresearchersprovidethemostbroadlyeffectiveroutetopullingresearchintopolicy
ResearchandevaluationAcademicexpertsmaysupportcitiesinevaluation-indeedevaluatingcityinitiativesisatraditionalmodelofcityuniversityrelationshipsSharingtheinformationyoucollectedintheGatherInformationphasecanhelpresearchersunderstandyourworkandgoalswhichinturnwillstrengthenthesupportthattheycanprovideDiscussingdetailsaboutyourinterventionandevaluationplanatthebeginningoftheprocessmeansthataresearchpartnercanhelpwiththeevaluationdesignanddatacollectionForinstancetheWilsonSheehanLabforEconomicOpportunitieswalkscitiesthroughconsideringthefollowingquestionsbeforepartneringonanevaluation
1 Isthereaclearpolicyprogrammodelwithobjectivescreeningcriteria
2 Isitpossibletoevaluatewhathappensbothwiththepolicyprogramandintheabsenceofthepolicyprogram
3 Arethereoneormoremeasurableoutcomesofinterest
4 Arethereenoughpeopleforrigorousquantitativeanalysis
5 Whatdatawillbenecessaryforthestudyandwhataretheprocessesforaccessingthesedatasource(s)
6 Isthereacompellingreasontoevaluate
Strategicallyengagekeypartners
19
InadditiontoassessingreadinesstoconductanevaluationresearcherscanhelpyouhoneinonthepurposeoftheevaluationTheWorldBankdescribesseveralpotentialusesofevaluationwhicharedefinedhereInbriefevaluationcanhelpdecisionmakers
makeresourceallocationdecisionsrethinkrootcausesofproblemsidentifynewproblemsdecideonabestalternativedeterminewhetherprogramsorpoliciesarebeingimplementedwithfidelitydeterminewhetherprogramsorpoliciesarehavingthedesiredimpact
Researcherscanalsoprovidesupportinsharingtheresultswiththescientificcommunitythroughpapersandconferenceswhichisvaluableincreatingapeernetworkofevidence-basedcitypractitioners
TrainingandtechnicalassistanceUniversitypartnersespeciallythoseatpracticecenterscanbehelpfulindevelopingorupdatingtrainingcurriculumtrainingthetraineranddeliveringtrainingtocitystaff
Strategicallyengagekeypartners
20
ReviewtheevidenceandcollectdetailsOnceyouhaveathoroughunderstandingoftheproblemdataandactorsyouwillneedtoconductyourownresearchonavailableevidenceandlearnaboutthelandscapeofpolicyoptionsYourgoalwillbetodevelopanaccessiblelistofinformationaboutpoliciesthathavehelpedsolvesimilarproblemsinotherplacesThischapterwillguideyouthroughthestepsofpreparingyourreviewersunderstandinglevelsofevidenceandcollectingthedetailsonevidencethatyouwillneedfordiscussionwithdecisionmakersinyourcity
Selectandprepareyourreviewers
AssignapersonorteamtoreviewrelevantinterventionsthathavedemonstratedsuccessinotherjurisdictionsTheirgoalwillbetoproduceareviewofalloftherelevantprogramspoliciesorinitiativesthatyourteamcouldconsiderimplementingTheywillbeconductingaliteraturereviewsupplementedbyinterviewswithexpertsandpeerswhohaveinformationaboutpotentialprograms
SincethepeopleyouassigntothistaskareprovidingthebaselineofpotentialoptionstoimplementitisimportantthatyouchoosepeopleyouareconfidentwillbethoughtfulandthoroughintheirworkItisalsocriticalthattheyareawareoftheimportanceofsourcecredibilityEnsureyourreviewershavefamiliarizedthemselveswithrulesforcriticallyanalyzinginformationsourcesbeforetheygetstartedIftheyareincludingnewsarticlesaspartoftheirsearchtheyshouldalsobeawareofmethodsforevaluatingnewscredibility
AskyourreviewerstodevelopaplanandtimelinetocompletethereviewThisstartswithsolidifyingagoverningresearchquestionOnewaytoknowyouarecreatingadetailedresearchquestionistousetheacronymPICOremindingyoutoincludereferencetothepopulationintervention(ifyouhaveoneinmind)comparisonandoutcomeelementsinyourresearchquestionForexampleifyouknowthatyouhaveapopulationofteenagerswhoarehomelessyouhavelearnedfromyourstakeholdersthatmanyoftheseteenagersidentifyasLGBTandyouhaveheardsomethingabouttheapproachofcreatingdrop-incentersfordeliveringsupportyoucouldask
ForLGBTyouthfacinghomelessness(population)howeffectivearedrop-incenters(intervention)relativetostandardsocialservices(comparison)atsettlingtheminpermanenthousing(outcome)
Ifyoudonothaveaninterventioninmindyoucanleaveouttheinterventionandcomparisonandsimplyask
Reviewtheevidenceandcollectdetails
21
HowcanwehelpLGBTyouthfacinghomelessness(population)obtainpermanenthousing(outcome)
OnceyourreviewershavetheirresearchquestionsusethequestionstocreatealistofkeywordstheycanuseintheirliteraturesearchThiswillhelpkeeptheircollectionofarticlesreasonablyfocusedHavethemdescribewhattheirsearchprocesswillbeincludinglistingthedatabasestheywillreviewandtheirplanforstoringtheinformationtheywillcollect(Herearesomeadditionalsuggestionsforstructuringareviewplan)
WhatyourreviewerswillbedoingisalightweightversionofarapidreviewTotheextentthattheywouldliketodevelopamethodologyforspecificsearchprotocolsincludingcriticalappraisalprocessestheycancertainlygoaheadanddoso
Usedifferentlevelsofevidenceforthemostcompletepicture
AsdiscussedintheWorkingwitharangeofevidencechaptertheNestaStandardsofEvidenceframework--reproducedbelow--ishelpfultounderstandthelevelofconfidencethataprogramorinterventionishavingapositiveimpactThisapproachletsyouputinterventionsthatyouhaveheardaboutlocallybutwhichhavenotbeenstudiedtogetherwiththosewithademonstratedcausaleffectThisishelpfulsinceanumberofpoliciesprogramsandpracticesthatyouwillwanttoconsidermaynotyethavemuchofaresearchrecord
Reviewtheevidenceandcollectdetails
22
NearlyanypieceofinformationaboutaninterventioncanbeclassifiedwithinthestandardsofevidenceForexampleifsomeonehasacrediblehypothesisabouthowaninterventioncouldproducebenefitthistypeofevidencewouldbeclassifiedasLevel1ThistypeofevidencecouldincludeinterventionsinothercitiesthatyoureadaboutinthenewsorhearaboutataconferenceForinstancewhenUSDAallowedschoolsandschooldistrictstoprovidefreebreakfastandlunchtoallstudentsthroughtheCommunityEligibilityProvision(CEP)manydistrictsimplementedthisoptionevenbeforetherewasevidenceofeffectivenessThiswouldfitintoLevel1sinceitbuildsonthelogicthatprovidingaccesstomealswilldecreasefoodinsecurity
YoumayalsofindpiecesofinformationthatqualifyasLevel2evidenceForinstanceasmoreschooldistrictsimplementedCEPprincipalstestifiedaboutpositivechangestheyobservedinstudentssuchasamorepositivedemeanorthatcanleadtobetteracademicperformanceDataalsoshowedthatthestudentsdescribedwerereceivingmoremealsthanbeforetheprogramsimplementationHoweveritwastoosoontobuildevidencetoshowacausalrelationshipbetweentheincreaseinfreemealsandincreasedacademicperformance
ldquoLower-levelrdquoevidencecanconsistofnuancedbutvaluableelementsthatrelatetoaninterventionrsquospotentialfitwithacommunityQualitativeinformationsuchastestimonyfrompeoplewhohaveexperiencedaninterventionandthosewhohavefacilitatedorimplementeditwillalwaysbeusefultoconsiderandprovidesdetailedlocalcontext
Experimentalstudies--describedstartingatLevel3withcontrolledexperimentsandgoingthroughLevel5withfullyfleshedoutprotocolsandprocessesforreplication--stilloccupythehigherlevelsofthisdescriptionofevidenceThisprioritizationshowsthevalueofrigorousstudyforevaluatingtheimpactofaparticularpolicyorprogramratherthanotherfactorsthatcouldbeinfluencingoutcomesofinterestRememberthatmultipletypesofevidencecancomplementrigorousimpactevaluationsandgivecityleadersadditionalinsightsandbroaderperspective
Accessinformation
Encourageyourreviewerstobegintheirsearchforpromisingpracticesingovernmentandnon-profitclearinghousesandinacademicpolicyjournalsWorkpublishedinthesesourceshasalreadybeensubjectedtoahighlevelofprofessionalscrutinyandgenerallycontainsagreatdealofdetailwhichmeansthatyourreviewersaremorelikelytobeabletopullouttheinformationyouneedtomakeagooddecision
Clearinghouses
Reviewtheevidenceandcollectdetails
23
Herearesomeoftheclearinghouses(collectionsofprogramevaluations)forevidence-basedpoliciesandprograms
ResultsFirstClearinghouseDatabaseThePewCharitableTrustsCollectstogethereightotherclearinghousestoprovideinformationoninterventionsinthefollowingcategoriesadultcriminaljusticeeducationchildwelfarehealthjuvenilejusticementalhealthsubstanceabuseandyouthdevelopmentIndividualclearinghousesincludedinResultsFirstare
BlueprintsforHealthyYouthDevelopment-foryouthdevelopmentandviolenceprevention
CaliforniaEvidence-BasedClearinghouse-forchildwelfareinterventions
TheLauraandJohnArnoldFoundationrsquosSocialProgramsThatWork-forsocialservicesprograms
TheUSDepartmentofEducationrsquosWhatWorksClearinghouse-foreducationinterventions
TheUSDepartmentofHealthandHumanServicesrsquoResearch-TestedInterventionPrograms(RTIPs)-forcancercontrolinterventions
TheUSDepartmentofHealthandHumanServicesrsquoTeenPregnancyPreventionEvidenceReview(TPPEvidenceReview)-forsexualriskbehaviorandsexualhealthoutcomes
TheUSDepartmentofJusticersquosCrimeSolutionsgov-forcriminaljusticejuvenilejusticeandcrimevictimservices
TheUSSubstanceAbuseandMentalHealthServicesAdministrationrsquosNationalRegistryofEvidence-basedProgramsandPractices(NREPP)-substanceuseandmentalhealthinterventions
TheUniversityofWisconsinPopulationHealthInstituteandRobertWoodJohnsonFoundationrsquosCountyHealthRankingsandRoadmapsWhatWorksforHealth(WhatWorksforHealth)-healthbehaviorsclinicalcaresocialandeconomicfactorsandthephysicalenvironment
UKWhatWorksCentresTheBritishgovernmenthascharteredanetworkofsubject-specificorganizationswhichcollectsynthesizeandreportoutevidenceontheeffectivenessofpolicyprogramsandpractices
NationalInstituteforHealthandCareExcellence(NICE)-healthandsocialcare
Reviewtheevidenceandcollectdetails
24
SuttonTrustEducationalEndowmentFoundation-educationalachievement
CollegeofPolicingWhatWorksCentreforCrimeReduction-crimereduction
EarlyInterventionFoundation-earlyintervention
WhatWorksCentreforLocalEconomicGrowth-localeconomicgrowth
CentreforAgeingBetter-qualityoflifeforolderpeople
WhatWorksCentreforWellbeing-wellbeing
WilsonSheehanLabforEconomicOpportunities(LEO)UniversityofNotreDameProvideslinkstotheirevaluationsofinterventionsincriminaljusticeself-sufficiencyeducationhealthandhousingandhomelessness
ClearinghouseforLaborEvaluationandResearchUSDepartmentofLaborProvidesresearchonworkforceandlabor-relatedpoliciesandprograms
J-PALEvaluationsAbdulLatifJameelPovertyActionLabStudiesonpoverty-reducinginterventionsfromaroundtheworld
CampbellSystematicReviewsCampbellCollaborationSystematicreviewofpolicyinterventionsbyscholarlycoordinatinggroupsinsevenissueareascrimeandjusticeeducationinternationaldevelopmentmethodssocialwelfareknowledgetranslationandimplementationanddisability
Benefit-CostResultsWashingtonStateInstituteforPublicPolicyAWashingtonState-charteredprogramtoestablishthelikelihoodofrelativecostsandbenefitsforprogramsacrossthefollowingareasjuvenilejusticeadultcriminaljusticechildwelfarepre-Kto12educationchildrensmentalhealthhealthcaresubstanceusedisordersadultmentalhealthpublichealthandPreventionworkforcedevelopmentandhighereducation
EffectiveInterventionsCentersforDiseaseControlandPreventionRepositoryforevidence-informedHIVpreventioncareinterventions(withstepsonhowtoimplement)
Journalsandindexes
OnceyouhavereviewedrelevantinformationaccessiblethroughclearinghousesitisworthtakingalookatindividualstudiespublishedasscholarlyarticlesYourreviewerswilldothisbysearchingarticleindexesandacademicjournalsUnfortunatelyanumberofusefulacademicjournalsandindexingplatformsmaynotbegenerallyaccessibletoyourreviewersifyoudonothaveasubscriptiontothemToaddressthisconsiderpartneringwithalocal
Reviewtheevidenceandcollectdetails
25
collegeoruniversitytoprovideyouwithalistofarticlesfromthefollowingsources(Thisisalsoanexcellentopportunitytoseekresearchsupportfromanyacademicstakeholdersyouidentifiedpreviously)
Indexes
GoogleScholarincludesmostpeer-reviewedonlineacademicjournalsandbooksconferencepapersthesesanddissertationspreprintsabstractstechnicalreportsandotherscholarlyliteratureincludingcourtopinionsandpatentsItallowsyoutospecifyapublicationrangeinyoursearchandprovideslinkstolikelysourcesforalllistedarticles
PolicyFileIndexisauniqueresourceforUSpublicpolicyresearchinthatitgrantsusersaccesstotimelyupdatedinformationfromover350publicpolicythinktanksnongovernmentalorganizationsresearchinstitutesuniversitycentersadvocacygroupsandotherentities
PolicyArchiveworkstocollectanddisseminatesummariesandfulltextsvideosreportsbriefsandmultimediamaterialofthinktankuniversitygovernmentandfoundation-fundedpolicyresearch
Anothersourceforhigh-qualityresearchisthereportsoftheCongressionalResearchServiceYoucanfindtheirreportsaggregatedhere
Specificjournals
Oftenthemostusefulacademicjournalsforunearthingpolicyorprogramreviewswillbeineachspecificsubjectarea--thatisineducationjournalsforeducationpoliciesYoucanuseageneralindexlikeGoogleScholartobegintolearnwhichjournalsaremostoftenusefulforyoursearchHoweveryoucanalsosearchthefollowingtwogeneralistpolicypublicationsforissuesandarticlesrelatedtoyourtopic
PublicAdministrationReviewJournalofPolicyAnalysisandManagement
Newssourcessubjectmatterexpertsandinterviews
AlthoughitisvaluabletoknowwhathasbeenexploredthroughascholarlyresearchframeworkitisalsousefultolearnwhatkindofinterventionsarebeingtriedinthefieldrightnowandhowpeoplefeelaboutthesetrialsSearchsubscriptionnewsaggregatorslikeLexisNexisorfreenewsaggregatorslikeGoogleNewsorYahooNewsforyourissueareaplusthewordpolicyorprogramtoseewhatkindsofinterventionsarebeingcoveredinthenews
Reviewtheevidenceandcollectdetails
26
DonotforgettocontacttheadvisorygroupmemberswhowerechosenfortheirsubjectmatterexpertiseInadditiontoprovidinggoodmaterialthroughinterviewsthemselvestheymayhaveideasaboutothersubjectmatterexpertsyoushouldcontacttoaskaboutpotentialinterventions
Peercitiesandnetworks
InadditiontoreadingabouttheworkofothercitiesitisidealifyouareabletospeakwithpeoplewhoareinvolvedwithcurrentimplementationsofinterestinginterventionsInyourcityyoulikelyhaverelationshipswithyourneighborsbutitcanbehelpfultolookoutsideofyourregionforsolutionsaswellTofindcounterpartsinothercitieswhoareworkingonaprogramyouwouldliketolearnmoreaboutyoucancontactstaffinintermediaryorganizations(likeGovExoryourprofessionalassociations)whomightbeinapositiontoputyouintouchimmediatelyorwhomightseeanothergoodrouteforsettingupaconversationHereisalistofnetworksthatmaybehelpful
InternationalCityCountyManagementAssociation(ICMA)NationalAssociationofCounties(NACO)NationalLeagueofCitiesUSConferenceofMayorsWhatWorksCities
Collectthedetails
Onceyourreviewershavefoundevidenceonpossiblepoliciesorprogramstoimplementtheyshouldcapturekeydetailsthatwillhelpdecisionmakersunderstandthepotentialeffectivenessandapplicabilityofeachpotentialintervention
Foreachsourcethathasprovidedinformationyouwanttoconveyusethetemplatebelowtocaptureasmanyofthefollowingdetailsaspossible(Youdonotneedtocompletethisforsourcesthatprovideduplicativeinformationorareotherwisenotthatuseful)
Programorpolicyname (egldquoHousingFirstrdquoorldquoblightedpropertiesregistryrdquo)
Sourcetype (egjournalarticleinterviewsubjectmatterexpertstudyassociationcommunityparticipationprocess)
Sourcedetails (egarticletitleandpublicationnameandtitleofinterviewee)
Evaluationmethodology
(egrandomizedevaluationcomparativeevaluationoftwodifferentgroupsbeforeafteranalysis)
Evidencelevel (egLevel12345)
Programorpolicy
Reviewtheevidenceandcollectdetails
27
description
Programorpolicygoal (1sentencecapturingtheprincipalaimoftheintervention)
Existingimplementations
(WherehastheinterventionbeentriedListasmanysitesasyouareawareof)
Characteristicsofimplementationsites
(ForeachimplementationsiteofferbasicgeographicIDmdashcityandstateThenyouwillwanttocollectthedetailswhicharerelevanttoyourdecisionmakersrsquoconsiderationsaboutwhatmakesalocationldquosimilartordquoorldquodifferentfromrdquoyoursituationYouwillatleastwanttoincludeforeachimplementationsitePopulationnumberPopulationdensity(ruralurbansuburban)Socio-economiccharacteristicsBeforestartingaskyourdecisionmakersaboutanyparticularcharacteristicsofprogramimplementationsitestheywouldwanttoknowbeforeseekingtoreplicate)
Characteristicsofimplementationpartners
(Anyavailabledetailsaboutorganizationsinstitutionsorpeoplewhowereinvolvedwithimplementingtheintervention-egindependentchildcarecenterpublicelementaryschoolcitypublichealthnurses)
Characteristicsoftargetpopulationeligibleprogramparticipants
(Anyavailabledetailsaboutpeoplewhoarethebeneficiariesoftheprogramorotherwiseincludedoraffectedbytheintervention-egnumberofprogramparticipantscategoriesofpropertyownersaffected)
Policyorprogramduration
(HowlonghastheinterventionbeeninplaceineachimplementationsiteHasitendedandifsowhy)
Whatbenefitscamefromeachimplementation
(Anyobservedpositiveeffectsoftheinterventionincludingeffectsize)
Statisticalestimatesofinterventioneffectiveness
(Anyavailablestatisticalestimatesoftheinterventionrsquoseffects)
Whatdrawbackswereobserved
(Anyavailabledetailsaboutunintendedorunavoidablenegativeeffectsoftheintervention)
Programorpolicycosts
(Anyavailabledetailsaboutprogramcostsperimplementation)
Reviewtheevidenceandcollectdetails
28
Reviewtheevidenceandcollectdetails
29
DecideonapolicytoimplementbasedonlocalfactorsTogetthecriticalbuy-infromleadershiptoimplementanewpolicyorprogramtheprojectleadshouldscheduleameetingwithdecisionmakerstopresenttheevidencesurroundingthepolicyoptionscapturedinthetemplateDependingonyourinsightintoyourcityleadershipswillingnesstomakechangeyoumaywishtosharetheresearchyouhaveconductedontherootcausesoftheproblemandsomeofyouranalysisontheimpactofcurrenteffortstodate
Considertheapplicabilityofanapproach
AsdiscussedinthepreviouschapterpolicymakersoftenlookforevidenceofeffectiveprogramsinothercitiesbutoftenthequestionarisesofhowgeneralizableaprogramisfromonecitytoanotherTheAbdulLatifJameelPovertyActionLab(J-PAL)suggeststhatinreviewingevidencefromanotherlocationitishelpfultoidentifythespecificmechanismthatwasused(deliveryofprogramorserviceinformationsharingetc)identifythelocalconditionsthatwereinplaceobservethebehaviorchangethatresultedfromthemechanismandfinallytounderstandlocalcontextandconditions(legalorcostconstraintsculturalnormsetc)Thissetofpresentationelementscanbedrawnfromthelistofdetailscollectedintheresearchtemplate
EducationpolicymakersinChicagoILusedfindingsfromprogramsinKenyaandIndiainwhichchildrenwhohadfallenbehindtheirpeersweregiveneducationalsupporttargetedtotheirleveloflearningThemechanismusedwastargetedteachingandthereweresimilarconditionspresentinallthreecountriesAlthoughsomestudentshadfallenbehindinthecurriculumteacherswereincentivizedtoteachgrade-levelmaterialsConsiderationsofhowthelocalcontextinChicagomightaffectimplementationofatargetedteachingprogramincludedunderstandinghowtheneighborhoodsthatthestudentswerefrommayhaveaffectedtheireducationaloutcomesandthehighcostofteachinginsmallgroupsInChicagoMatchEducationprovidedtwo-on-onetutoringwhichmatchedstudentswithwell-educatedindividualswhohavenothadformalteachertrainingforalower-costmodelTheresultswerepositiveStudentswhoweretutoredperformedbetterinmathandweremorelikelytopasshighschoolclasses
ThegeneralizabilityframeworkproposedbyJ-PALisbelowUsethisasatemplateasyouconsiderpotentialpolicysolutions
Decideonapolicytoimplementbasedonlocalfactors
30
SourceTheGeneralizabilityPuzzle
OnceyouhavenarrowedthepolicyoptionsdiscussthequestionsbelowforeachoptionsyouareconsideringThisstepiscriticalasunderstandinglocalcontextissoimportantinsuccessfulpolicyimplementation
CommunityContextQuestionnaire
Localcontext
DoesthisoptionmakesenseinourcommunityIsthisaonesizefitsallapproachIsanyparticulargroupabsentfromthisapproachIfsoisthatok
Youranswers
Effectiveness
WhatwouldweexpecttoobserveifthepilotisaseffectivehereasinCityX
Youranswers
Timing
DoesthetimingmakesenseIsthistherighttimetoimplementthispolicychange
Decideonapolicytoimplementbasedonlocalfactors
31
WherearethepotentialroadblocksthatcouldaffectthescheduleIsthereanywaytomitigatethisHowlongwouldittaketoseeanimpact
Youranswers
Implementation
DowehavethestaffcapacitytoimplementthisArethereadditionalpartnersthatyoucanbringtothetabletosupportimplementationDoesthismakesensegivenourformofgovernment
Youranswers
Authority
DoesourcityhavetheauthoritytoenactthisIfsowhatistheappropriatemechanismAretherelegalbarriersArethereanyexistingpreemptionissuesfromthestateorfederalgovernmentCanthosebeaddressedthroughlegislationadvocacyIstherepoliticalsupportoroppositiontothis
Youranswers
Cost
HowmuchwillthiscostannuallyHowwillwepayforitCanthepolicybeimplementedwithexistingfunding
IfsoisthisfundingreliableHowwouldthispolicyoptionaffectourcitysbudgetintheshortandlongtermWhatotherfundingoptionscanweconsiderIsthereanalternativethatwouldcostlessandgiveussimilarresultsWillanyconstituencybeadverselyaffectediffundingforprogramsthatimpactthemisshiftedtoanotherprogrampolicyorinitiative
Youranswers
Decideonapolicytoimplementbasedonlocalfactors
32
EstablishaframeworktomonitorprogressNowthatyouhavereviewedtheevidenceconsideredlocalcontextandchosenyourpolicyitistimetodeterminehowyourcitywillmonitorprogressofthisnewapproachCitystaffandleadersneedtoestablishtheinfrastructureforongoingcontinuousdecisionmakingbasedonwhetherthepolicychangeisleadingtothedesiredoutcomesforresidentsandwhetherthechangeismeaningfulandrationaltofrontlineworkersSettingarbitrarygoalsatthelocallevelthatconflictwithgoalsatotherlevelsisconfusingtotheindividualsresponsibleforachievingthegoalsGoalsthataresettoohightoachievecanbedemoralizingtostaff
ThissectionincludesstepstosetupavenueinwhichCitystaffandleadershiproutinelyvisitkeyindicatorstodeterminewhatisorisnotworkingandtomakedecisionsinreal-timeFormorereadingonperformancemanagementpleasevisittheGovExPerformanceManagementGettingStartedGuide
InperformancemanagementtermssuchasindicatormeasureandmetricmaybeusedinterchangeablyInthisguidewewilluseindicatorInyourworkthemostimportantthingistobeconsistentinwhichevertermyouuseThisvideomaybehelpfulinunderstandingthisconceptandcommunicatingittoothers
Establishthefundamentalsandgatheryourteam
FirstdeterminewhetherthereisanestablishedvenueorregularmeetingscheduleduringwhichyoucandiscussprogressIfyourcityhasanexistingperformancemanagementstructureonarelatedtopicaddingadiscussionoftheprogressofpolicychangetoyouragendacanbeagreatideaIfthiswillbeastandalonemeetingdecidehowfrequentlytheprojectteamanddecisionmakerswillmeetandputthosemeetingsonthecalendarfortheentirelengthoftheproject
DeterminewhowillstaffandattendroutinemeetingsonCityprogressThesemeetingsmayincludetheMayorandorCityManagerdepartmentheadstaffwhoareimplementingthepolicychangeandanalysts
Identifyindicators
ItiscriticalthatindicatorsbeanimportantcityandcommunitypriorityaswellassomethingtheCityplaysamajorroleinaddressingTheremaybeinstancesinwhichtheCitywillneedtoreachouttoorcollaboratewithexternalpartnerstogetamorecompletepictureofprogressontheoverallpolicychange
Establishaframeworktomonitorprogress
33
Anoutcomeindicatordescribesthedesiredeffectofthetargetedpopulation
ExampleBellevuesfamilyfriendlyneighborhoodsindicatormeasuresthepercentofresidentswhobelievetheyliveinaneighborhoodthatsupportsfamiliesparticularlythosewithchildren
WithyourkeypartnersrevisityourproblemstatementcurrentstateandfuturestateexerciseandbegintoidentifytheoutcomeindicatorsthatwillillustratesuccessFornowchooseasingletop-levelkeyperformanceindicator(KPI)ItiscriticalthattheKPI-thepieceyouwillmeasure-beapriorityforthecommunityandsomethingtheCityplaysamajorroleinaddressing
Outputindicatorsmeasuretheprocessofprogramorservicedelivery
ExampleSanDiegoMayorKevinFaulconersgoaltorepair1000milesofstreetby2020
ReviewingtheadministrativedatasetinventorybegintoparedowntherelevantprogramsanddatasetsthatsupportyourKPIWhatarethemeasuresofeffectivenessforeachprogramTheseareyouroutputindicators
Thediagrambelowillustratestherelationshipbetweenoutputandoutcomeindicators
ThinkaboutimplementationindicatorsasmeasuringstepsintheprogramsrolloutthatarenecessarytoobtaindesiredchangesImplementationoutcomescanfocusontheinterventionstaffdeliveringtheinterventionanorganizationorsystemandcanaddressfeasibilityadaptabilitysustainabilityandcostTwokeyquestionstoconsiderwhendevelopingimplementationindicatorsare
1 Istheinterventionacceptableandappropriatetothecommunityand
2 IsitfeasibletoimplementinterventionXatthistimewiththeseresources
Establishaframeworktomonitorprogress
34
Amoreextensivelistwithdefinitionsofpotentialimplementationoutcomesislinkedhere
UsethetablebelowtochoosethedesiredimplementationoutcomesforyourpilotandtoplanyourmeasurementtechniqueThisshouldincludelevelofanalysis(individualproviderorconsumerorganizationorsetting)yourmeasurementtechnique(surveysobservationinterviewfocusgroupsadministrativedataanalysis)andthephasewhenyouwillcollectthisinformation(earlymidlate)Theremaybemorethanonelevelofanalysismeasurementtechniqueandphaseofdatacollection
Implementationoutcome
Definitionofoutcome
Levelofanalysisandmeasurement
Yourmeasurementtechnique
Phaseofpilotwhenyouwillcollectdata
Usethespacebelowtocaptureyourindicatorwishlist
Indicatorwishlist
Outcomes_____________________________________________________________________________
Outputs________________________________________________________________________________
Implementationoutcomes______________________________________________________________
Setapublichigh-levelgoal
Thehigh-levelgoaliscriticalinestablishingdirectiontogalvanizethepeoplewhowillbedoingtheworkWhendiscussingthegoalusetheMayorsbullypulpitLookforkeymilestonesinCitygovernmenttosharethegoalstatementsuchasthebeginningofaMayoraladministrationaStateoftheCityspeechorotherpublicremarks
GoalsshouldbeldquoSMARTrdquo
Specificaspossible(innamingthepopulationwhowillbebetterofftheactionthecitywilltake)
Measurable
Achievable
Establishaframeworktomonitorprogress
35
Results-focused
Time-bound
ThegoalstatementshouldincludeyouroutcomeindicatorlikeChattanoogasgoalldquoWewillendchronicveteranhomelessnessbyDecember2016rdquoAdiagramtohepyoubuildaSMARTgoalaroundyouroutcomeindicatorisbelow
Component Description Yourturn
Specific WhowillbebetteroffWhatactionthecitywilltake
Measurable Howwillyoumeasuresuccess
Achievable Isthegoalattainable
Results-focused Whatistheexpectedresult
Time-bound When
Setrealisticambitioustargetsandtimelinesforeachindicator
ToestablisharealisticandambitioustargetforyourgoalstatementusetheadministrativedatacollectedforyourchosenKPItoreviewactualCityperformanceinthemostrecentyearcomparedwiththethreeorfourmostrecentyearsworthofdataBegintocalculatesummarystatistics(egtotalminimumaveragemedianmaximum)toidentifytrendsCreatecumulativeandmonthlychartstoidentifytrendsandanomaliesBasedonyouranalysisofhistoricaldataignoretheanomaliesandassumetheproductivitytrendswillfollowthesameoverallpatternfrommonth-to-monthFinallyapplyanincreasedproductivityfactortotheperformanceeachmonthtoforecastfutureprogressBasedonthisanalysischooseatargetthatinspiresimprovementwhileaccountingforreality
Manycitiesthataremorematureintheirperformancemanagementpracticesrecognizetheimportanceoftakingacommunity-levelapproachtoimprovingoutcomesforresidentsThesecitiesarecreatinggoaltaxonomiesthatincludemeasuresrelatedtopublicsafetyhousingandevenrecreationCitiesdriveprogressonthesegoalsbyimplementingstrategiesthatengagemultipledepartmentsandcollaboratewithserviceprovidersinthecommunity
EventuallyyouwilldevelopgoalstatementsfortheoutputsinordertocapturetheefforttoachievetheoutcomecontainedinyourgoalstatementYoursetofindicatorsshouldbecomposedofshort-andinterim-termsupportivemeasureswithaccuratetargetsandrelevanttimelinesAnexamplegoaltaxonomyfromtheCityofLasVegassResultsVegasprogramisbelowUsethistaxonomyasamodelforyourownsuiteofindicatorsandgoalstatements
Establishaframeworktomonitorprogress
36
Yourturn
Thetemplatebelowismeanttohelpyoucraftagoalstatementforeachsupportingmeasure
Goalstatement
Departments
Outcome
Activities
KPI
Target
Timeframe
Conductanddiscussrelevantanalysis
ThrougharoutineseriesofmeetingscontinuallymonitorprogressofKPIssupportingindicatorsandanyotherdatacollectedsuchasqualitativedatatoidentifytrendsthatwillindicatewhetherthechosenpolicyishavingthedesiredimpactThesedatapointsshoulddrivethediscussionduringthesemeetingspointingtoopportunitiesforfurtherinquiryinto
Establishaframeworktomonitorprogress
37
whatisdrivingtheobservedtrendsAnalystsshouldbeencouragedandtrainedtoaskquestionsofthedepartmentsproducingthedataandconductsitevisitswhenpossibletodeepentheirunderstandingofthedataPresentationoftheanalysisshouldfocusononeideaatatimetokeeptheconversationfocusedonkeypointsandspecificopportunitiesforimprovementYoumayconsiderdevelopingapublicfacingdashboardtosharethisprogressmorebroadly
AgeneralcomparisonofthestatusquodatawithwhatwasobservedduringthepilotcanprovidedecisionmakerswithaquickunderstandingofthedifferencesbetweentheoldandnewapproachesalthoughthiswillnotshowcausalityOftenthesecomparisonswillincludeoperationalmetricssuchasdaysordollarssavedbutrememberthattheimpactonpeoplewillbemostimportantIfyouareworkingwithanevaluationpartnertheywillsupportthisstep
Usethespacebelowtosummarizetheresultsofyouranalysis
MeasureTargetvalue(seesectiononsettingtargetsabove)
Pilotvalue(whatdidyouobserveduringthepilot)
Historicalaverage(ifavailable)
Results(WhoisbetteroffIsanyonenegativelyimpacted1-2sentencesreassessingyourgoalstatement)
DocumentyourprocessandprogressincludingfeedbackfromtheparticipantsimplementationpartnersandtheprojectteamDidanythingunexpectedoccurduringthepilotArethereinternalandexternalfactorsthatmayaffectyouroutcomesThisiswheretheimplementationoutcomescanbecanhelpfulIftheinterventiondidnotgoasplannedconsiderwhetheritisbecausetheinterventionwasnotacceptableorfeasible
UsethetemplatebelowcreatedbyDenversPeakAcademytocapturethisinformationIfthe306090daytimelinedoesnotmakesenseforyourcityfeelfreetocustomizeit
CurrentState
FutureState
30Day
60Day
90day
Whatwentwell(+)
Whatdidnotgowellshouldbechanged(-)
IfaquestionordiscussionpointarisesthatcannotbeansweredinthemeetinganassignedstaffmembershouldcapturethequestionduringthemeetingAfterwardthefollowupitemsshouldbesenttorelevantpartnersandthenextmeetingshouldbeginwithadiscussionof
Establishaframeworktomonitorprogress
38
thoseitems
Takeaction
DiscussionsaboutwhetheryourindicatorsaremovingintherightdirectionandwhatmightbedrivingtrendsmayleadtoadiscoverythattherearenotenoughresourcesallocatedtothispolicyorissueTheremaybelimitedfundingforstaffprogrammingortechnologythatisdrivingthedatainthewrongdirectionIfCityleadershipisintheroomthisisagreattimetomakethecaseforadditionalfundingduringthebudgetseasonBycollectinganalyzinganddiscussingdataonthispolicyroutinelyyouarebuildingevidencetomakethecasefortherightleveloffundingtoreachtheestablishedtargets
Establishaframeworktomonitorprogress
39
Assessdataaccessibilityqualityandsecurity
Determineaccessibilityandqualityofdata
OnceyouhavecompiledyourwishlistofindicatorsitistimetodeterminetheaccessibilityandqualityofthedataTopreventcomplicationyoumaywishtoprioritizerelevantindicatorsforwhichyouarealreadycollectingdataAvoidthetemptationofadoptingatechnologysolutiontocollectnewdataatthispointsuchasaphotosurveytoolorotherappFornowyouaretryingtodeterminewhichdatasetsarerelevantaccessibleandhighqualityBesuretoworkwithyourkeyinternalpartnerswhileyouareidentifyingyourmeasuresDepartmentstaffwhoworkonprogramsarefamiliarwiththeindicatorsthatwillaccuratelysupporttheoutcomesanddatabaseadministratorswillbevaluablewhendiscussingaccessibilityandqualityofthedata
Abrieflistofconsiderationsfortheaccessibilityofyourdataisbelow
ArewecurrentlycollectingthisdatasetorwillwebecollectingitforthefirsttimeHowfarbackdoestherecordgo(timehorizon)WouldweneednewtechnologytocollectoranalyzethedataCanthedatabecollectedmanuallyorcanitbeautomatedHowfrequentlyisthedataproduced(continuouslyhourlydailyweeklymonthlyquarterlyyearlyetc)IsthedataavailablefortheentireCitypopulationorasubsetofthepopulationHowgranularisthedata(individuallevelCensusblockneighborhoodcitywideetc)DoesthedatacontainsensitiveinformationIsthereaproxymeasurethatwecanusefornowIsthedatamachinereadable
AsnotedbytheWilsonSheehanLabforEconomicOpportunities(LEO)attheUniversityofNotreDameleveragingadministrativedatacansignificantlyreducethecostsandtimelinesofrigorousevaluationCitiescanestablishaninternaldatasharingagreementacrossdepartmentsincludinglanguagethatenablesadministrativedatatobeusedforresearchpurposesCitiescanalsoidentifyexternaladministrativedata-holderssuchasacountyschooldistrictpolicedepartmenthospitaloranotherstatethatmayhaveoutcomedataimportantformeasuringprogramimpactCitiescanleverageexistingrelationshipsandorworkwithresearchinstitutionstodevelopstreamlineddatasharingprocessesandagreementswiththeseexternalentities
Assessdataaccessibilityqualityandsecurity
40
Evaluatethequalityofyourdata
QualityisasimportantasaccessibilityToevaluatedataqualityyoumaywishtocreatearubricliketheonebelow
Value ValueDescription
3 ExcellentDataQualityThedataiscompleteaccurateandupdatedregularly
2 AdequateDataQualityThedataissomewhatcompleteaccurateandupdatedfairlyregularly
1 PoorDataQualityThedataisnotcompleteaccurateorupdatedregularly
ThisisagreattimetobuildonthedatainventorythatyoustartedinthegatherinformationpageSuggestedadditionalfieldstoaddtoyourdatainventorytemplatearebelow
Program DatasetTitle
DatasetDescription DataSource
Departmentorexternalpartner
DoestheCityalreadycollectthisdata
Whatisthenameoftheprogram
Human-readablenameofthedataset
Shouldbeunderstandablebynon-technicalusers
Whatistheoriginalsystemapplicationfilethathousesit
Departmentorexternalpartnerthatmaintainsthedata
Isthisdataalreadycollected
Updatefrequency-ThefileformatforthedatasetTypicallycsvxlsxshapefiletxtetc
Format-Howoftenthedatasetisupdated
Timerange-Theperiodoftimecoveredinthisparticulardataset
Datause-Howisthisdatasetcommonlyused
Datausers-Whotypicallyusesthisdataset
Dataownerstitle-Titleofthedataowner
Dataownersposition-DataOwnersjobtitle
Dataownersemailaddress-DataOwnersemailaddressWhomanagesthedataandorisresponsibleforgrantingpermissiontoaccessthedataWhounderstandswhatthedatasetincludesandcananswerquestionsaboutit
Dataqualityconcerns-Describetheconcernsyoumayhaveaboutthequalityofthedataset
Assessdataaccessibilityqualityandsecurity
41
Division-DivisionwherethedataismaintainedIfmorethanonedivisionisresponsiblelisttheprimaryownerofthedataset
Sensitivedatacomments-DescribetheissuesthatwouldpreventthecityfromsharingthedatapubliclyCommonsensitivityconcernsincludeprivacyviolationssecurityissuesorhighcostorstaffdemandstopostthedata
Publishingstatus-Thistheplacetoputanycommentsregardingthecurrentstatusofthedataset
Publishedlink-Thisisthelinktothepublisheddatasetontheopendataportalorcitywebsite
Reviewdatasecurityandprivacyprotectionprotocols
OftenapolicychangealsoinvolvesachangeinthedatathatisbeingcollectedhowthatdataisbeinghandledwhohasaccesstoitandahostofotheradjustmentsAsyoutakestockofthedataassetsthatareavailableforyourinitiativeitisvitaltoalsothinkthroughwhatwillbedonetoprotectthatdataandtheprivacyofthepeopleitrepresentsAbigpartofgooddatasecurityinvolvesstrongdatagovernancepracticesthatdefineandidentifysensitivedataandconsciouslyworktobalancethevalueandriskofstoringsuchsensitivedataOnewayofevaluatingyourdatacomesfromcaselawrelatedtotheUSFreedomofInformationActwhichwehavetranslatedtothechecklistbelow
1 Woulddisclosureresultinasubstantialinvasionofprivacy2 Haveweconsideredtheextentorvalueofthepublicinterestandthepurposeorobject
oftheindividualsseekingdisclosure3 Istheinformationavailablefromothersources4 Wastheinformationgivenwithanexpectationofconfidentiality5 Isitpossibletomouldreliefsoastolimittheinvasionofindividualprivacy
TheseriskscouldbemitigatedusingavarietyofstrategiesfromaggregatingdatatoobfuscatingindividualstoredactingfieldsthatmayincludePersonallyIdentifiableInformation(PII)orre-evaluatingyourdatacollectionmethodstofindalternativewaysofgettingtheinformationyouneedThesestepsshouldbecodifiedandmanagedbyadatagovernancecommitteeifpossibleForthesensitivedatathatisdeemedessentialcreateaplanaroundpotentialdatabreachesinordertoquicklyandconfidentlyrespondtoemergencysituations
ProactivelypreparingforthesesituationsisoftenthebeststrategyfordatasecuritybutdonotforgetthetechnicalsideInvolveyourITadministratorsinconversationsaboutdatastorageandaccessensuringthattheywillbeprotectedbyenterprise-gradecybersecuritysystems
Assessdataaccessibilityqualityandsecurity
42
Assessdataaccessibilityqualityandsecurity
43
Designimplementationandevaluation
Designtheimplementation
NowthatyouhaveyouraccountabilityplanandkeyindicatorsitistimetogetspecificaboutyourplanforimplementationThetemplatebelowwillhelpyouthinkthroughthespecificmechanismsofimplementingpolicychange
UnderstandingorganizationsandsystemsiscriticaltounderstandingandmakingdecisionsaboutwhereandhowinterventionsshouldbeappliedApolicyimplementedinasinglesettingmayshowpromisingresultsbutimplementingmultiplepoliciesandprogramsatthesametimewithdifferentactorsresponsibleforeachaspectcanhelpensuresustainablebehaviorchange
InSomervilleMAShapeUpSomerville(SUS)isacitywidepublichealthprogramthatisimplementedacrossmultiplelevelsofinfluenceandsystemsthroughinterventionsbeforeduringandafterschooltoincludechangestobuiltenvironmentsuchasbikelanesregionalmasstransportationandphysicaleducationequipmentinschoolsandgymstoincreasephysicalactivitywellnesspoliciesandschoolgardensnutritioneducationduringandafterschoolandhealthyrestaurantprogramstosupporthealthyfoodandhealthyeatinghabits
Usethetemplatebelowtoplanyourimplementation
Policyimplementationplanningtemplate
ListthespecificdesiredoutcomesInthepreviousstepyouworkedwithdepartmentstafftoestablishspecificmeasurableoutcomesandinterimindicatorsPrioritizetheindicatorsthatwilldemonstrateprogressUsethespacebelowtocapturethedesiredoutcomesofthepolicychange
ExampleShapeupSomervillebeganwiththegoalofpreventingobesityin1stthrough3rdgraders
Yourturn
____________________________________________________________________________
____________________________________________________________________________
Designimplementationandevaluation
44
SpecifythetargetpopulationAreyoudesigninganinterventionthattargetsonlycertainsitesAreyouplanningonimplementingaprogramorinitiativeforaspecifictargetaudienceClearlyspecifythepopulationyouplantotarget
IfyouplantoconductarigorousimpactevaluationasnotedbytheWilsonSheehanLabforEconomicOpportunitiesitisimportanttoestablishconcreteobjectiveeligibilitycriteriaandensurethatthesecriteriaareappliedconsistentlybyallstaffduringscreeningandintakeItisalsovitaltoidentifytheindividualswhoareeligibleforaparticularprogramorserviceevenifeveryoneeligiblecannotbeservedbyexistingresourcesThishelpscitiesidentifyanyunmetneedsandguidefutureresourceallocationwhilealsocreatingthepossibilityofarigorousimpactevaluationbyidentifyingapotentialcomparisongroupFinallywhenconductingscreeningandintakeitisimperativetocollectandmaintainbasicinformation(egnamedateofbirthgender)aboutallprogram-eligibleindividualsnotjustthosewhoparticipateintheprogram
ExampleIntheShapeUpSomervilleexampletheprogrambeganbytargeting1stthrough3rdgraders
Yourturn
____________________________________________________________________________
____________________________________________________________________________
DefinethelocationWherewillthepolicychangetakeplaceWillimplementationtakeplaceatonesiteormultiplesites
ExampleShapeUpSomervilleinterventionsweredeliveredinschools
Yourturn
____________________________________________________________________________
____________________________________________________________________________
ClarifyimplementationpartnersandarticulatetheirrolesWhatarethekeyorganizationsinstitutionsorpeopleresponsibleforimplementationTheprojectteamshouldbeasclearandspecificaspossibleinprovidingadescriptionoftheconcreteactivitiesthatimplementationpartnersaremeanttocarryoutPracticeprofilesmaybeahelpfultooltohelpdefinewhatisrequiredoftheactorswhoareresponsibleforimplementingtheintervention
Designimplementationandevaluation
45
ExampleInSomervillemayoralleadershipandengagementofmultiplecityagencies(publichealthprimarycareschoolsCambridgeHealthAllianceetc)streamlinedandconnectedthevariousShapeUpSomervilleinitiativesandgrantswhichhelpedtokeepthetalentandenergyfocusedonhealthyeatingandactivelivingforallsectorsofSomervilleSUSpartnersultimatelyincludedschoolsnonprofitsCityandstateagenciesandnationalfoundations
Yourturn
____________________________________________________________________________
____________________________________________________________________________
DescribetheinterventionWhatarethestepsactivitiesorprocessesRememberthatthemechanismthespecificstepsorprocessthatleadtobehaviorchangeiskeytounderstandingtheeffectivenessofthepolicychangeandsharingtheresultswithothers
ExampleGoalsofShapeUpSomervilleweretoinfluenceeverypartofanearlyelementaryschoolchildrsquosdayndashbeforeduringandafter-schooltopromotehealthyeatingphysicalactivityandtheirweightstatusInterventionincludedfoodserviceenhancementstoimprovebreakfastandlunchandstaffprofessionaldevelopmentwalktoschoolactivitiesSUSclassroomcurriculumandprofessionaldevelopmentschoolwellnesspolicydevelopmentSUSafter-schoolcurriculumandprofessionaldevelopmentoutreachandeducationtohomethroughmaterialsforumseventsldquoSUSapprovedrdquorestaurantscommunityoutreachandcapacitybuildingthroughpolicydevelopmenttrainingsmediaplacements
Yourturn
____________________________________________________________________________
____________________________________________________________________________
ClarifytimelineandmilestonesfordeliveryandevaluationHowfrequentlywilltheservicesinterventionbedeliveredHowlongwilltheprogramrunWhatisthetimetoexpectedimpacts(eg12monthsafterprogramcompletion)
Designimplementationandevaluation
46
ExampleShapeUpSomervilleledbyTuftsUniversityFriedmanSchoolofNutritionScienceandPolicyconductedaquasi-experimental(nonrandomized)interventionstudyamong1stthrough3rdgradestudentsinSomerville(intervention)andtwocomparisoncommunitiesfor9-monthsduringoneschoolyear(fall2002ndashspring2003)ThestudyshowedpositiveassociationoftheinterventiononchangeinBMISUSreleasedreportssummarizingevaluationsprogresstodateplansforadditionalprojectsandimpactstenandfifteenyearsintotheinitiative
Yourturn
____________________________________________________________________________
____________________________________________________________________________
PlanyouranalysisTheprevioussectionaddressedestablishinganaccountabilityframeworktomonitorprogressandimplementationDeterminewhichindicatorswillbediscussthroughthatprocesswhowillsharethedataandanalysisandhowfeedbackfromthoseconversationswillbeincorporatedbytheprojectteam
ExampleTheMayorrsquosofficeimplementedadata-drivendecisionmakingapproachcalledSomerStatwhichwassynergisticwiththeapproachestakenbytheresearchandevaluationteamsdeployedforthespecificgrants
Yourturn
____________________________________________________________________________
____________________________________________________________________________
PlanforsustainabilityIfthepolicyandprogramchangeissuccessfulhowwilltheprojectteamsustainandexpandtoadditionalsitesorpopulations
ExampleAlthoughmostofthepartnershipsforgedintheearlyyearsweremaintainednewgrantsandcollaborationsexpandedattentiontoolderpublicschoolstudentsactivetransportationandtheGreenLinesubwayextensionthebuiltenvironmentfarmerrsquosmarketeffortssuchasldquoGrowninSomervillerdquowithUnionMainStreeteconomicdevelopmentandhealthdisparitiesaswellasdatacollectionmonitoringandsurveillance
Yourturn
Designimplementationandevaluation
47
____________________________________________________________________________
____________________________________________________________________________
Pilotplanningtemplate
IfyouplantoconductapilotfirstdeterminethepurposeofthepilotDoyouwanttoobservewhetherthepolicychangeresultsinthedesiredbehaviorchangeAreyouconcernedaboutfeasibilityscalabilityorcostDoyouwanttotestwhetheryourdesiredresultscanbeachievedthroughthispolicyapproachClarifythegoalofthepilotandthequestionsyouhopetoanswerThiswillhelpyouidentifythekeyindicatorsthatyouwillusetomeasurewhetherthepilotissuccessfulBeforebeginningestablishabaselineofcurrentperformanceandsetyourtargetsaccordinglybasedonwhatyouhopetoaccomplishthroughthepilot
NextestablishthepilotplanforyourprojectteamincludingmilestonesforteamcheckinstodeterminewhethertomoveaheadorchangecourseThisisseparatefromthemeetingsthatwilloccuraspartoftheaccountabilityframeworkUsethespacebelowtocreateyourtimelineanddevelopyourmeetingschedule
Task Personresponsible Startdate Checkindate Deliverable
Planyourevaluation
WhileyouaredesigningtheimplementationitisalsotimetobegintodevelopyourplanforanalysisAsnotedbytheWilsonSheehanLabforEconomicOpportunitiesdecisionsaboutprogramsetup-includingoutreachscreeningintakedatatrackingandenrollment-candeterminewhetherornotarigorousimpactevaluationwillbepossibleAsdiscussedcityleadersmayfindithelpfultoincludeevaluationpartnersinearlyconversationsaboutprogramsetupDuringpreviousstagesyoumayhaveworkedwitharesearchertoplanyourproposedpolicychangeYoushouldbeclearaboutthepurposeoftheevaluationanddevelopanevaluationplanBelowareadditionalconsiderations
Causationvstrendanalysis
IntheEstablishaframeworktomonitorprogressstagetherewasguidanceontrackingtrendsovertimeItisimportanttonoteherethatmonitoringdataisdifferentfromanalyzingacausaleffectTrendanalysiswillillustratethatchangesareoccurringandmayleadtoquestionsthatcouldbeansweredbyyourevaluationbuttrendanalysisalonedoesnotprovideevidenceaboutwhatisdrivingthosechangesYoumayseechangesinyour
Designimplementationandevaluation
48
outcomeindicatorsbasedonvariablesoutsideoftheCityscontrolForinstanceachangeinfederaltradepolicycanhaveanimpactemploymentratesinyourCityoutsideofyourCitysnewjobtrainingandplacementprogram
ToaddresscausalityyoumustcomparetheoutcomesofindividualswhoreceivedtheinterventionwiththosewhodidnotWithoutacomparisongroupofsimilarindividualswhowerenotexposedtotheinterventionitisimpossibletoknowwhichfactorsareresponsiblefortheoutcomesRandomizedControlTrials(RCTs)inwhichindividualsarerandomlyassignedtotreatmentandcontrolgroupsarethegoldstandardinprovingcausalityIfyoucandemonstratethatoutcomesimprovedforthosewhoreceivedtheinterventionanddidnotimproveatallorimprovedlessforacomparisongroupyoucanbesomewhatcertainthattheinterventionwasthecause
Examplesofothernon-causalimpactevaluationsthatarerigorousandreliableincludecertainquasi-experimentalstudiesusingcomparisongroupdesignssmartinterruptedtimeseriesanalysesthatmodeltrendsovertimeinwaysthatallowestimationofinterventioneffectswelldesignedpre-posttestanalysesneedsassessmentsimplementationstudiesetcFormoreinformationondifferenttypesofevaluationsseetheSageResearchMethodswebsite
Lookfordifferencesingroups
BuildintimetoalsoexaminethepilotdatatoseeifgroupsareexperiencingdifferentresultsReviewingperformancebasedonoutcomemeasuresforapopulationinaggregatemayhidethesubtlewaysthatsomepeoplemaybeaffecteddifferentlysoitisnecessarytocarefullyanalyzethedatafordisparateoutcomesForinstancetheMovingtoOpportunity(MTO)studythatexaminedtheimpactofhousingvouchersonfamilieswholivedinhigh-povertyneighborhoodsfoundthatchildrenwhomovedbeforetheageof13hadbettereducationandhigherincomesasadultsHoweveradultsdidnotseeimprovededucationandemploymentoutcomes
Disaggregatingdataintomorespecificcategoriesofraceandethnicitycanhelpunderstandtrendsanddisparitiesinamoregranularwaywhichcanhelpdecisionmakerstargetpoliciesandinterventionsinamoreequitableway
EarlierintheRoadmapweworkedtoidentifystakeholdersandwhomightbepositivelyandnegativelyimpactedbyachangeinpolicyUsetheresultsofthatexercisetoinvolvethecommunityininterpretingdataanalysisandtogatherqualitativefeedbackfromthecommunitytocallattentiontoissuesForinstanceanincreaseinyouthdetentionmaybeexplainedbyinefficienciesorlackofaccesstotransportation
Designimplementationandevaluation
49
Designimplementationandevaluation
50
Shareyourstoryandsuccess
Sharetheresults
Storytelling
AnintegralstepinyourstrategyissharingtheresultsofyourpilotorpolicychangewithyourcommunityThisincludesdevelopingmessagingaroundtheworkofyourorganizationandcreatingaplantodisseminateyourstoriesWhytellstoriesaboutyourwork
Informandorunderstandasubjectorsituation
Giveneededcontextofdataorbitsofinformation
Relatablestoriesarememorableforreadersviewers
ReadersorviewerswillseetheinformationasmorecompellingwhenthereisademonstratedandrelatablechangeorimpactonthemselvesortheirfriendsneighborsandcolleaguesStoriesareaboutpeopleandgovernmentisfundamentallyabouthelpingpeopleSotellthestoriesofthepeopleyouandyourcitycolleagueshelped
Keepthesetipsinmindasyoudevelopyourstorywithyourteam
WhatwereyourfindingsandwhyshouldanyonecareTellyouraudiencehowtheproblemwassolvedWhathastheimpactbeenonresidentsandcitystafftodateWhatwasthesuccessandweretheremultiplesuccessesalongthewayIfyouhavepreliminaryresultsordatashowingimprovementtosharethisisthetimetodoit
WhowasinvolvedandimpactedReviewthelistofstakeholdersHowwereeachofthemimpactedbytheprogramHumanizeyourstoryandimpactbyaddingadditionalvoicesThesevoicescanbeoneofthefollowing
Apersonorfamilywhowaspositivelyaffectedbythechangesuccess
Aninvolvedthird-partydescribingboththeproblemandthesolution(ieanorganizationnonprofitprovidersbusinesstradeassociationsportsteametc)whoispartofthecommunitythatthegovernmentworkedwithonthechangesuccess
Someonefromthegovernmentitselfwhocanspeakaboutthechangesuccess
TellmoreaboutyourcityDescribeyourcityincludingpopulationdemographicsindustriesetcWhatareitschallengesWhatisitknownforWhatmakesituniqueHowisitsimilarordifferentfromothercitiesintheUS
Shareyourstoryandsuccess
51
DescribeyourcityrsquoschallengeWriteoutthechallengeyourcityisfacing(asdefinedinyourproblemstatement)HowdoesitaffectresidentsHowdoesthisproblemaffecttheworkofcitystaffmembers
AddthedetailsProvidedetailsonyourappliedresearchstrategyTalkaboutwhatyourcityhasdonenthepasttoresolvetheissueDiscusshowyourstrategyhaschangedwhenyoubegantoadoptthisapproachWasmoneysavedorinvested
Usethisstorymapexercisetoorganizeyourthoughtsandguaranteethatyouincludealloftheelementsofatraditionalstory
AfteryoudevelopyourmessagingcreateacommunicationsmarketingandoutreachstrategydesignedtospreadyourmessageYourstrategyshouldwalkyouthroughthefollowingkeyactivities
1 IdentifyandprofileyouraudienceWhencreatinganddisseminatingyourmessageitisalwaysgoodtothinkaboutyouraudienceWhoexactlywouldbeinterestedinwhatyouhavetosayThegeneralpopulationofthecityPeoplewhoworkinnonprofitsBusinessownersItmattersbecausenoteveryonewholivesinacitymaybethetargetoftheinformationBusinessownersmaybemoreinterestedindatashowingeffectsonthebusinessdistrictthanothersandsoonWewillcoverthislaterbutthisstepwillalsohelpyoutothinkofthemosteffectivewaytoreachyouraudience
2 Defineyourcalltoaction(ifyouhaveone)SpendsometimeconsideringwhyyouareinterestedintellingyourstoryDoyousimplywanttoupdatethepubliconyourprogressorcelebrateyoursuccessesAreyouinterestedinyouraudiencechangingitsbehaviororsupportingaspecificinitiativebeinglaunchedbyyourorganizationIfyouwanttohaveyouraudienceactorbehaveinacertainwaybespecificandincorporateitinyourmessaging
Shareyourstoryandsuccess
52
3 SelectperformancemeasuresThinkabouthowyouwoulddetermineifyourcampaignissuccessfulandselectappropriateperformancemeasuresYouwillrevisitthesemeasuresattheendofyourcampaigncommunicationsplanimplementationtodeterminewhetheryouneedtomodifyyourcommunicationsandoutreachstrategyforthefuture
4 MethodstocommunicateyourstoryTherearemanywaystocommunicatesuccessofyourprojectHerearesomesuggestionsonreachingaudiencesItisimportanttodeterminewhichmethodsarethemosteffectiveforyourdesiredaudiencetoreceiveinformationDoyourresearch(PewResearchCenterhasexcellentstudiesonthereachofmediadigitalmediaandmarketing)ForexampleifseniorsareyourtargetpopulationandtheyarelessinclinedtouseInstagram(thatistrue)youmaywanttoconsiderFacebookorthenightlylocalnewsbothofwhichareusedmoreoftenbyseniorsBelowisalistofsomemethodsandtoolsyoucanuse
i Pressreleasestolocalnewspapersandtvstations
ii ShortsmartphonevideosAgreatwaytoengageanaudienceisthroughvideoAskonequestionofsomeonewhoispartofthestoryMaybethereisasocialworkerwhocandiscusshungerinyourcityThesevideoscanbereusedonmultipleplatforms
iii Long-formwriting
iv TellingindividualstoriesthroughaseriesofphotosandshortnarrativesforpublicationonFacebookorInstagram
v Op-edsopinionpiecesplacedinthelocalpaper
vi Emailupdatestoyourlist
vii Connectingwithappropriateorganizationsthatareinvestedinthetopicandworkwiththecommunityandaskthemtodisseminateinformationtotheirmembers
viii EmployingldquoWordofMouthrdquo(WOM)tacticsbycreatingtownhallsorfreeeventsforresidentstoattendintheirownneighborhoods
5 DeterminethetypeofcollateralUsuallycampaignsthattargetresidentsrequireseveralavenuestoreachthemPeoplearenotalwaysinthesameplaceorspaceHavingakeenunderstandingofyouraudiencewillassistyouwithyourchoicesonwhatneedstobedevelopedSometimesthatmeansimagesforsocialmediaaonepageflierthatgivespeopleinformationonattendinganeventoranin-depthbrochure
Shareyourstoryandsuccess
53
6 ImplementyourstrategyKeepinmindstrategiescanbefluidSometimessomethingmaynotturnoutasexpectedevenifduringplanningsessionsitwasldquosupposedtordquoItisokYoucananalyzewhathappenedduringadebriefsessionandonceyoursurveysandothermeasurementshavebeenfinalized
Visualizations
Withtherightvisualizationyoucanreviewandshareinformationinsightsanalysisperformanceandtrendsallofwhichisvaluabletoacross-sectionofthepublicButwhichvisualizationisrightforyouTochoosetherighttypeofchartgraphdeterminethetypeofquestionyouwanttoanswerSomecommoncharttypesarebelowandmoreinformationcanbefoundhere
Exploreadditionalanalyticaltechniques
Duringthegoal-settingphaseyouestablishedoutputoutcomeandimplementationmeasuresBasedontheresultsofyourpilotyoumayneedtoadjustyourmeasuresWiththepilotdatainhandyoumayhavesomeideasofhowfutureanalysescouldbedifferentandpotentiallyimprovethequalityofyourprogramTakesometimetobrainstormotherrelevantdatasetsthatyoudidnothavethetimetopursueearlierincludingmessydepartmentaldatathatneedsmoreattentionorsourcesofpublicdatafromthestateandfederalgovernmentnonprofitsandevenprivatecompaniesthatcouldsupplementyourunderstandingofanissueUsingmachinelearningtechniquesyoumightbeabletobetter
Shareyourstoryandsuccess
54
predictdemandforhomelessservicesusingNOAAweatherforecastspreemptwatermainbreaksusingsoilstratigraphyorinvestineconomicdevelopmentbymonitoringrealestatelistingdata
Advancedanalyticaltechniquescanalsohelpyourimplementationrunmoreeffectivelybyintegratingdata-informedtargetinganddecisiontoolsintotheprocessByautomatingroutineworkflowsandsurfacingsubtletrendsthesetoolscanhelpyouiterateinamoreprecisemannerandprovidekeystrategicguidancetofront-lineemployeesthatwouldotherwisebeunavailableThesesortsofdata-science-enhancedprocessescanbeclassifiedasoneofsixprojecttypesasexploredfurtherbytheNewOrleansLANOLAlyticsteamFormoreguidanceonscopingasimilarprojecttheGovExadvancedanalyticsscopingguidegivesyouastep-by-stepoverviewofhowtobesuccessful
Usethespacebelowtolistyourideasforpossibleadvancedanalyticsprojectsbytype
Typeofproject Ideas
Findingtheneedleinthehaystack
Prioritizingforimpact
Earlywarningtools
Betterquickerdecisions
Optimizingresourceallocation
Experimentingforwhatworks
Keepthefeedbackloopgoing
GatherinformationonpublicsentimentaboutpolicychangethroughfocusgroupssurveysandpublicmeetingsledbyadvisorygroupmembersNotewhethernewinformationisavailablethatcanhelpanalyzeorcontextualizeyourprogress(egfederalorstatedatareports)Sharetheresultsofyourpilotandthepolicychangeprocesswithothercitiesandresearchers
Shareyourstoryandsuccess
55
WorkingwitharangeofevidencePresentlythereisalotofattentionandinvestmentinevidence-basedpolicymakingincludingthefederalbipartisanCommissiononEvidence-BasedPolicymakingthePewCharitableTrustsreportsandtoolkitsonHowStatesEngageinEvidence-BasedPolicymakingandtheLauraandJohnArnoldFoundationsupportedEvidence-BasedPolicymakingCollaborativeManyoftheseinitiativesfocusonrigorousevaluation--meaningrandomizedcontrolledtrials(RCT)orotherexperimentalmodels--asthebestkindofevidencetouseHoweverthisbodyofevidenceisofteninsufficientforthespecifickindsofquestionsthatgovernmentscanhavewhenmakingadecisionaboutpolicyInthisguideweproposeanapproachtosupportgovernmentsinincorporatingadditionaltypesofevidenceintotheirdecisionmaking
OnewaytounderstandarangeofevidenceistouseNestarsquosStandardsofEvidencetoframeanddefineevidenceTheBritishnonprofitNestaworkstoinjectpositivechangeintothewayUKgovernmentpoliciesandprogramsfunctionOneoftheirkeyareasofworkhasbeendevelopingWhatWorksCentresorganizationsthatcollectcommissionsynthesizeanddisseminateevidenceaboutinterventioneffectivenessacrossarangeofsocialimpactareasNestadevelopedthefollowingStandardsofEvidencetoprovideausefulwaytodocumenthowdifferentkindsofinformationdemonstratedifferentlevelsofevidence
Workingwitharangeofevidence
56
Theimportanceofpotentiallybiasedinformation
Onechallengepresentedbyanevidenceclassificationthatacknowledgesthevalueofnon-scientificformsofevidenceistofigureouthowtobalanceallofthepiecestogetherwhenyouusethemTounderstandhowtodothiswehavetothinkabouthowtopositionpotentiallybiasedinformationindecisionmaking
WhenwelookatinformationwhichhasnotbeenrigorouslyevaluatedweacceptthatthisinformationmightbebiasedThewholereasonweconductrigorousevaluationistoremovebiasfromourobservationsWeconductstructuredtrialstoproduceresultsthatareconsistentandgeneralizableIfafindingisunbiasedthisaccuracymdashthislackofbiasmdashcanbeprovedbecauseanyindependenttestercanreproduceitwhentheyfollowanidenticalprocess
YoucanlookatpotentiallybiasedinformationasjustinformationthatisearlieroninitspathtorefinementreplicationandverifiedaccuracyApieceofinformationbasedonanyoneindividualrsquosobservationsislikelytobebiasedbecauseitiscomingfromjustonesourceBiasisreducedbygatheringtogetherlotsofindependentsourcesofinformationandfiguringoutitsmostcommonattributes
MaybethebiasedinformationmakesaclaimthatistoolargemdashitsayssomethingistrueunderallconditionswhenitisreallyonlytrueundercertainconditionsorthataneffectofaninterventionisbiggerthanitstrueaverageorthataneffectlastsovertimewhenitdoesnotPerhapschangingtheclaimtobemorespecificmdashcriticallyreviewingtheinformationtoseeifspecificconditionsmustbemettoachievetheoutcomesmdashwouldmaketheclaimmoreaccurate
IfyoutestedyourclaimfounditdidnotholdupinsomecaseschangedittesteditagainuntilyourclaimrepeatedlyheldupthenyoucoulddemonstratethattheclaimwasnolongerbiasedbutaccurateThisprocessoftestingimprovingtestingandvalidatingiswhatwegetfromstructuredscientificstudy
Withindividualpiecesofinformation--crediblehypothesescasestudiesexpertopinion--youcannotassumethatkindofaccuracyAtthesametimetheseearlyobservationsarefoundationaltoanyreplicablescientificobservationsWewillnevergetanywhereifwedonotstartwiththesepiecesofinformationeveniftheymightbebiased
Combininglevelsofevidencetogetthemostdetailedinformationforyourcity
ItisespeciallyimportanttoacceptdifferentformsofevidencewhenyoufacequestionsthathavenotyetbeentestedbyindependentstudiesSomeofthemostimportantquestionsrelatetothingsthatareveryspecifictoyoursituationYoufaceaparticularcombinationof
Workingwitharangeofevidence
57
issuesactorstrendsandresourcesthatisultimatelydifferentfromeveryotherplaceintheworld
ThisisimportantbecauseaspolicypractitionersandnotscientistsyoucaremoreaboutyourspecificcasethanahypotheticalaveragegeneralcaseYoucarelessaboutthetotallikelyrangeofimpactsfromapolicythanhowthissingleimplementationwillworkYouwantinformationaboutpoliciesthatwillprovideyouwiththebestpossiblepredictionabouthowtheywillworkinyourcontext
ByputtingdifferentkindsofevidenceintothesameframeworkNestarsquosStandardsofEvidenceimplythatwhenyoucannotgetrigorouslytestedexperimentallyverifiedinformationitisappropriatetoworkwithwhatwedohaveThisisrelevantforthemunicipalimplementationcontextbecausenostudypulledfromaliteraturereviewwillaccountforallofthevariablesyoufaceinyourowncontext
Intheabsenceofhavingastudythatpredictshowaninterventionworksinyourspecificcontextyouwillneednon-scientificinformationtotrytofillinthosegapsYouwillneedtogatherstoriesfromresidentsstaffmembersandotherstakeholdersthatsitonLevels1or2ofthestandardsofevidence
Someofthekindsofinformationyouwillprobablygetmoreeasilyfrompeoplethanstudiesinclude
howpeopleincitieslikeyoursreactedtoaninterventioninpublicforumshowpeoplewhoimplementedaninterventioninacitylikeyoursfiguredouthowtogetitpassedhowthecitycommunicatedpubliclyabouttheapparentbenefitsanddrawbacksoftheinterventionthepoliticalacceptabilityoftheintervention
Youcanremainsimultaneouslyawarethattheinformationyouaregatheringthroughindividualstoriesisbiased--notgeneralizableinformedbyspecificperspectives--whilealsorecognizingthathavingsomeinformationaboutsomethingimportantisbetterthannoinformationEvidencefromthefirsttwoevidencelevelsareessentialbuildingblocksofknowledgeandperfectlyusefulwhenyoucannotgetdetailedunbiasedknowledge
Youhavetostartsomewhereandyoushouldalwaysstartwithevidenceyouhaveaboutyourveryspecificconditionsthatyouknowareimportantforsuccess
Workingwitharangeofevidence
58
Thankyou
Interviewswithcities
GovExheldsemi-structuredinterviewswithsixstaffmembersinfourcitiestoestablishtheinitialstepsoftheRoadmapWeaskedrespondentstotalkaboutaninitiativeorpolicythattheyhavebeenresponsibleforimplementingintheircityRespondentsfromthreecitiesspokeabouttheireffortstoaddresshomelessnessandrespondentsfromonecityspokeaboutblightAfterrespondentsidentifiedtheproblemtheyspokeabouthowthecityprioritizedthisissuethevariousstepsinvolvedandbarrierspresent
GovExconductedtextanalysisoftheseinterviewswhichrevealedthatpeoplewasthemostcommonlyusedtermduringtheseconversationswhendiscussingthestepstochangeaprogramorpolicyThisisathemethroughoutthisRoadmapThefirstsectionbeginswithidentifyingtherelevantpeopleTheconversationsaroundevidencearefocusedonhumanbehaviorratherthanmethodologyorstrengthofevidence
Thankyou
59
OncewehadadraftoftheRoadmapwefollowedupwithcitiesonanindividualbasistolearnmoreaboutwhatwouldbehelpfulWearegratefultothefollowingcitystafffortheinterviewsandindividualconversations
MichaelBaskinChiefPolicyOfficerChattanoogaTN
MelissaBridgesPerformanceandInnovationCoordinatorLittleRockAR
VictoriaCarreoacutenAdministrativeOfficerLasVegasNV
TyroneGrandstrandCommunityDevelopmentPlannerFargoND
TibLaughlinDirectorofGeneralServicesKansasCityKS
TimMorelandDirectorofPerformanceManagementandOpenDataChattanoogaTN
CharlesRosendahlManagementAnalystDentonTX
Thankyou
60
MelissaSiebenAssistantCountyAdministratorKansasCityKS
GinaThompsonDirectorofStrategicServicesLewisvilleTX
JHUFacultycollaboration
GovExalsoconvenedthecitycontactsandfacultypartnersinaseriesofAdvisoryGroupmeetingsThesemeetingsservedtoconnectcitiesandfacultytounderstandoneanothersperspective
StefanBaralBloombergSchoolofPublicHealth
AshleyBernerJHUSchoolofEducation
MelissaDavey-RothwellBloombergSchoolofPublicHealth
JonLinksBloombergSchoolofPublicHealth
TamarMendelsonBloombergSchoolofPublicHealth
KristinMmariBloombergSchoolofPublicHealth
KeshiaPollackPorterBloombergSchoolofPublicHealth
JoshSharfsteinBloombergSchoolofPublicHealth
LizStuartBloombergSchoolofPublicHealth
LindsayThompsonCareyBusinessSchool
DanielWebsterBloombergSchoolofPublicHealth
Partnersindataandevidence-basedpolicy
Wealsospokewithpartnerswhoareexpertsinthefieldofevidence-basedpolicywhosharedtheirknowledgeandresourcesofcurrentandfuturethinkinginevidence-informedthinking
WendyBarrenoAssociateDirectorWilsonSheehanLabforEconomicOpportunitiesUniversityofNotreDame
TiffanyDavenportDirectorofDataPartnershipsBaltimoresPromise
TomiHiersDirectorBaltimoreCivicSiteAnnieECaseyFoundation
ChrisKingsleySeniorAssociateAnnieECaseyFoundation
KathrynOliverAssociateProfessorLondonSchoolofHygieneandTropicalMedicine
Thankyou
61
KristenWatkinsSeniorPolicyAssociateJ-PAL
GovExParticipants
GovExAnalystMiriamMcKinneyanalyzedtheinterviewtranscriptsandconductedtextanalysisSeniorImplementationAdvisorEmilyShawwillintegrateRoadmapactivitieswithadditionalservicedeliveryactivitiesatGovEx
Thankyou
62
ReferencesAbrahamKGampHaskinsR(2017)Thepromiseofevidence-basedpolicymakingReportofthecommissiononevidence-basedpolicymakingWashingtonDCCommissiononEvidence-basedPolicymakingBigmove2Extendingthegreen(nd)Retrievedfromhttpswwwcityoflewisvillecomdoing-businesslewisville-2025big-move-2-extending-the-greenBatesMAampGlennersterR(2017)ThegeneralizabilitypuzzleRetrievedfromhttpsssirorgarticlesentrythe_generalizability_puzzleBuildwithnotfor(2018January)RetrievedfromhttpsbuildwithnotforwordpresscomCairneyPampOliverK(2017)Evidence-basedpolicymakingisnotlikeevidence-basedmedicinesohowfarshouldyougotobridgethedividebetweenevidenceandpolicyHealthResearchPolicyandSystemsdoi101186s12961-017-0192-xCairneyPOliverKampWellsteadA(2016)TobridgethedividebetweenevidenceandpolicyReduceambiguityasmuchasuncertaintyPublicAdministrationReview76(3)399-402Retrievedfromhttpdxdoiorg101111puar12555Centreforpublicimpact(nd)RetrievedfromhttpswwwcentreforpublicimpactorgCernaL(2013)ThenatureofpolicychangeandimplementationAreviewofdifferenttheoreticalapproachesOrganisationforEconomicCo-OperationandDevelopmentRetrievedfromhttpswwwoecdorgeduceriTheNatureofPolicyChangeandImplementationpdfColoradoSprings(nd)Retrievedfromhttpscoloradospringsgovcommunicationspagewhat-expect-when-you-speakupContinuumofcare(coc)program(nd)RetrievedfromhttpswwwhudexchangeinfoprogramscocCouncilpresidentkicksoff$12millionyouthfund(nd)Retrievedfromhttpwwwbaltimorecitycouncilcomcontentcouncil-president-kicks-12-million-youth-fundDurlakJAampDuPreEP(2008)ImplementationmattersAreviewofresearchontheinfluenceofimplementationonprogramoutcomesandthefactorsaffectingimplementationAmericanJournalofCommunityPsychology41327-350Retrievedfromhttpswwwncbinlmnihgovpubmed18322790Fishbonediagram(ishikawa)(nd)Retrievedfromhttpasqorglearn-about-qualitycause-analysis-toolsoverviewfishbonehtmlFivewhysGettingtotherootofaproblemquickly(nd)RetrievedfromhttpswwwmindtoolscompagesarticlenewTMC_5Whtm
References
63
FligsteinNampMcAdamD(2011)TowardageneraltheoryofstrategicactionfieldsSociologicalTheory29(1)1-26RetrievedfromhttpsociologyberkeleyedusitesdefaultfilesfacultyfligsteinFligstein20McAdam20Soc20Theory20PaperpdHeinkelP(2018January12)CitymullspermitsforbikesharingcompaniesRetrievedfromhttpwwwdentonrccomnewsdenton20180112city-mulls-permits-bike-sharing-companiesHowisareamedianincomecalculated(2012October)Retrievedfromhttpswwwhudexchangeinfofaqs843how-is-area-median-income-calculatedLampkinLMampHatryHP(2003)KeystepsinoutcomemanagementUrbanInstituteRetrievedfromhttpswwwurbanorgresearchpublicationkey-steps-outcome-managementMarshDampSmithM(2000)UnderstandingpolicynetworksTowardsadialecticalapproachPoliticalStudies484-21Retrievedfromhttpjournalssagepubcomdoiabs1011111467-924800247McCannL(nd)RetrievedfromhttplaurenellencomMoultonSampSandfortJR(2016)ThestrategicactionfieldframeworkforpolicyimplementationresearchPolicyStudiesJournal45(1)Retrievedfromhttponlinelibrarywileycomstore101111psj12147assetpsj12147pdfv=1ampt=jdkhhcp4amps=e184c625e1855a6b1615ccb99ee27b1c0b3212d8PaynterB(2018January1)What100citiesarelearningfromeachotherbysharingtheirdataRetrievedfromhttpswwwfastcompanycom40518004what-100-cities-are-learning-from-each-other-by-sharing-their-dataPeakacademy(nd)Retrievedfromhttpswwwdenvergovorgcontentdenvergovenmayors-officeprograms-initiativespeak-performancepeak-academyhtmlPinnockHBarwickMCarpenterCREldridgeSGrandesGGriffithsCJRycroft-MaloneJMeissnerPMurrayEPatelASheikhAampTaylorSJC(2017)Standardsforreportingimplementationstudies(stari)ExplanationandelaborationdocumentBMJOpenRetrievedfromhttpbmjopenbmjcomcontentbmjopen74e013318fullpdfProctorESilmereHRaghavanRHovmandPAaronsGBungerAGriffeyRampHensleyM(2011)OutcomesforimplementationresearchConceptualdistinctionsmeasurementchallengesandresearchagendaAdministrationandPolicyinMentalHealth38(2)65-76doi101007s10488-010-0319-7TandonSD(2018)ProgressincommunityhealthpartnershipsResearcheducationandactionRetrievedfromhttpswwwpressjhuedujournalsprogress-community-health-partnerships-research-education-and-actionProvanKGampKenisP(2007)ModesofnetworkgovernanceStructuremanagementandeffectivenessJournalofPublicAdministrationResearchandTheory
References
64
18(2)229-252Retrievedfromhttpsacademicoupcomjpartarticle182229935895SabatierPA(1988)Anadvocacycoalitionframeworkofpolicychangeandtheroleofpolicy-orientedlearningthereinPolicySciences21(3)129-168Retrievedfromhttpwwwjstororgstable4532139SandstromAampCarlssonL(2008)TheperformanceofpolicynetworksTherelationbetweennetworkstructureandnetworkperformancePolicyStudiesJournal36497-524Retrievedfromhttponlinelibrarywileycomdoi101111j1541-0072200800281xabstractSchwartzMampWilsonE(nd)WhocanaffordtoliveinahomeAlookatdatafromthe2006americancommunitysurveyRetrievedfromhttpswwwcensusgovhousingcensuspublicationswho-can-affordpdfShearerJCAbelsonAKouyateBLavisJNampWaltG(2016)WhydopolicieschangeInstitutionsinterestsideasandnetworksinthreecasesofpolicyreformHealthPolicyandPlanning31(9)1200-1211Retrievedfromhttpsdoiorg101093heapolczw052ThelessonsofadministrativedataHigh-profilepolicy-relevantresearchpoweredbyadministrativedata(nd)Retrievedfromhttpswwwpovertyactionlaborgsitesdefaultfilesresources20161114-The-Lessons-of-Administrative-DatapdfToolsandotherneedsforqualitativeandquantitativemethods(2015February)AgencyforHealthcareResearchandQualityRetrievedfromhttpwwwahrqgovprofessionalsquality-patient-safetytalkingqualityassesstoolshtml
WeaverRCampBagchi-SenS(2013)SpatialanalysisofurbandeclineThegeographyofblightAppliedGeography4061-70
References
65
UniversityPartners
Name Title Affiliation Area(s)ofexpertise
Contactinformation
Examplesofworkingwithnon-academicdecisionmakers
WendyPBarreno
AssociateDirector
WilsonSheehanLabforEconomicOpportunitiesUniversityofNotreDame
LEOsfocusareasincludehealtheducationhousingchildwelfareeconomicopportunitycriminaljustice
wbarrenonduduorleondedu
MelissaDavey-RothwellPhD
AssociateScientist
JohnsHopkinsBloombergSchoolofPublicHealth
Health mdavey1jhuedu410-614-5854
BaltimoreCityHealthDepartmentLeadersMarylandDepartmentofHealthAddictionRecoveryServicesinDCExecutiveDirectorsofcommunityorganizationsacrosstheUSthatfocusonHIVSTIsubstanceabuseandimplementationscience
ElizabethStuart
AssociateDeanforEducationandProfessorDepartmentsofMentalHealthBiostatisticsandHealthPolicyand
JohnsHopkinsBloombergSchoolofPublicHealth
Educationsocialpolicystatisticalmethods
estuartjhuedu
Resources
66
Management
Design-ThinkingResources
EngagementGuideforCivicTech
Journeymap
PerformanceManagementResources
DenverPeakAcademy
Proxymeasuresguide
Talkingaboutperformancewithoutcomplicatingit(video)
SettingPerformanceTargetsguide
Datavisualizationguide
Performancemanagementgettingstartedguide
Exercises
Fishboneexercise
5Whys
AdministrativeData
TheLessonsofAdministrativeData(JPAL)
EvaluationResources
Quantitativeandqualitativemethods(USDepartmentofHealthandHumanServicesAgencyforHealthcareResearchandQuality)
KeyStepsinOutcomeManagement(UrbanInstitute)
RCT-YES
Sageresearchmethods
BestPracticesforMixedMethodsResearchintheHealthSciences
ACCORDChecklist
Pilotandimplementationplanning
Benchmarkingexamplesandproxymeasures
Seattleopendataplaybook
Resources
67
Datasciencescopingguide
Resources
68
AboutGovExGovExhelpsgovernmentsusedataandevidencetoimproveresidentsrsquolivesWesupportgovernmentsinbuildingtheirinternalcapacitytooperatesustainableanalyticspracticeswhichimprovecommunityoutcomesinfieldsfrompublicsafetytoeconomicdevelopmentGovExhasaproventrackrecordofworkinginconcertwithpartnerstoprovidetechnicalassistanceandtrainingatscale
SinceourlaunchinApril2015GovExhas
Trainedover2000governmentemployeesinpersonandonlineintopicsincludingdatamanagementperformancemanagementopendatacommunityengagementandanalytics
Providedtechnicalassistancetomorethan100localgovernmentsacrosstheglobeimprovingtheiruseofdatatoachieveefficienciesinservicedeliveryandbetteroutcomesforresidents
Assessedthecapacityofmorethan130governmentalorganizationstodeterminehowleadersstaffandexternalstakeholdersmanageandusedata
Localgovernmentsareincreasinglyonthefrontlinesofaddressinglargescaleproblemswithglobalimplicationssuchasclimatechangepublicsafetyandeconomicinequality
Whileevidence-basedinterventionstohelpsolvethoseproblemsareavailablelocalgovernmentstaffhavelimitedtimeandcapacitytofindandimplementresearch-basedsolutionsAsaresultcitiescontinuetoworkthewaytheyalwayshave--makingincrementalimprovementstoprogramsbutnotalwayscreatingchangeatasystem-level
ThroughresourcedevelopmentandcollaborationwithJHUfacultyGovExissupportingcityleadersinusingdataandevidencetomakedecisionsandcreatinglastingrelationshipswithresearchers
TheRoadmapwasdevelopedbyGovExaspartofitspartnershipwithBloombergPhilanthropiesWhatWorksCitiesinitiative
AboutGovEx
69
- Title page
- Introduction and purpose of the Roadmap
- Using the Roadmap
-
- Gather information about the problem
- Strategically engage key partners
- Review the evidence and collect details
- Decide on a policy to implement based on local factors
- Establish a framework to monitor progress
- Assess data accessibility quality and security
- Design implementation and evaluation
- Share your story and success
-
- Working with a range of evidence
- Thank you
- References
- Resources
- About GovEx
-
dauntingWithoutadeepunderstandingoftherootcausesoftheproblemitcanbedifficulttoidentifythebestevidencetouseFinallyitcanbedifficulttoknowhowtoimplementaparticularpolicyinyourcityandwhetheritwillhavethedesiredeffect
WhentheUnifiedGovernmentofKansasCityKSandWyandotteCounty(UG)beganworkingonitsblightproblemtheUGconvenedaresearchteamtolookforevidencefromothercitiesUsingsearchtermslikeblightlandbankabandonedpropertiesandvacancythatledtheUGtolearnaboutthesimilareffortsinthecitiesofNewOrleansBaltimoreMobileKansasCityMOandcitiesinOhioandMemphisTheteamalsoremainedopentohearingideasfrominternalstaffwholearnedaboutpracticesfromothercitiesthroughtheirownprofessionalorganizationsanduniversityaffiliationsthattheyhadnotbeenempoweredtotestorimplementpreviouslySincethetopicofblightrangesfrompropertymaintenancetosafetyissuestheCityresearchteamhadtodoalotofreadinganddebatingoverconflictingconceptsfortacklingawiderangeofissueswithinthatspectrumThisinvolvedaskingquestionsaboutwhetherthepolicywillworkinKansasCityandwhatchangeswouldneedtobemadeforittobesuccessfulWhenresearchinglandbankstheCitylearnedabouttheCenterforCommunityProgressanonprofitthathelpscommunitiesaddressvacantabandonedanddeterioratedpropertiesConversationswiththeCenterforCommunityProgressledtoexploringtheideaofavacantpropertyregistryUponconductingfurtherresearchUGstaffcameacrossinformationonacourtrulingfromChicagothatcertainvacantbuildingsdonotneedtofollowtheCityofChicagorsquosvacantbuildingordinanceThisdirectedthesearchtounderstandifothercitieshadexamplesofwhethervacantpropertyregistriesareworkingTodeterminewhetherthismightworkinthecontextofKansasCitystaffbeganvettingideastoseeiftherewereanyaspectsthatconflictedwithCityordinancesTheUGalsoexaminedthefeasibilityofimplementingthenecessarytechnologyandfeasibilityfromtheperspectivesofpropertyownersandUGstaffThisinitialresearchandanalysisphasetookaboutsixmonthsandwasledbytheGeneralServicesDirectoraVISTAinternandtheresearchteamIndevelopingthisRoadmaptheCitymentionedthatitwouldbehelpfultohaveaccesstocasestudieswithdetailedinformationonthepolicysolutionsthatcitieshavetakenandthespecificsofhowtheyhavebeenimplemented
PurposeoftheRoadmap
ThisRoadmapismeanttoempowercitiestotakeownershipofthepolicychangeprocesstoframeproblemsinawaythatcreatesademandforevidenceandtoformlong-termrelationshipswithpoliticalpolicypractitionerandresearcheralliesThisworkisnecessaryResearchfromthefieldofimplementationsciencesuggestssuccessfulimplementationofevidenceintopracticehappenswhencitiestakethetimetoidentifytherightpeopletoworkwithchoosethebestevidencefromavarietyofsourcesandconsiderthebestwaytoadapt
IntroductionandpurposeoftheRoadmap
4
thatevidencetotheirowncontextThroughourinterviewswithcitystaffweobservedthatframingtheproblemclearlyformingadvisorygroupsorsomeothercoalitionandidentifyingandimplementinganappropriatesolutionarenecessaryingredientstosuccessfulpolicychange
ThroughthisRoadmapGovExhopestofacilitateanewwaytohelpAmericancitiesrethinkthehowtheysolveproblemsandaddressthemostsalientpolicychallengesinurbangovernance
IntroductionandpurposeoftheRoadmap
5
UsingtheRoadmap
WhoshouldusethisRoadmap
ThisRoadmapisforanystaffmemberincitygovernmentwhoischargedwithsolvingaproblemorimplementingapolicyAlthoughthestepscanbeusedtochangeexistingprogramsorimplementnewprogramsthefocusoftheRoadmapisonlong-termsustainablepolicychangeWeusethetermpolicytomeanlawsregulationsfundingdecisionsandinterventions
CitystaffwhoarereadingthisguidemaybecomingfromdifferentstartingpointsItisimportanttobehonestaboutwherethatisForinstanceyoumaybelookingfornewsolutionstolongstandingseeminglyintractableproblemssuchasanannualincreaseingunviolenceormaybeyouarefacingnewerchallengesliketheopioidepidemicOrperhapsyouarelookingforadata-drivenwaytoaddressissuesrelatedtoday-to-dayservicedeliverysuchaspermittingorbikelanesThestepsareapplicablewhetheryouareworkingonalargescaleissueoralocalmunicipalissue
YoucanusethisguidewhetheryouareadecisionmakerorfrontlineworkerThroughouttheRoadmaptheresinformationaboutwhenandhowtoengageotherpeopleCollaborationisakeythemethroughthisRoadmapThegoodnewsisyoudonothavetodothisworkalone
TheapproachandthetoolscontainedintheRoadmapshouldberelevantwhetheryouareinastrongmayororcouncil-managerformofgovernmentElevatingthisprocesstothelevelofMayorCityManagerorCityCouncilmaytaketimeandmultiplestepsbutwehopeyouwillfindwaystoincorporatedataandevidenceintoyourday-to-daywork
WhatarethestepsoftheRoadmap
ThechaptersinthisRoadmaprepresentthehigh-levelstepsinvolvedinpolicychangeYouwillnoticethatthesectionsrefertooneanotherandbuildonthepreviousstepsTheRoadmapismeanttobeiterativeyouwillrevisitsomesteps
TherootsoftheproblemsthatcitiesareworkingtosolvearecomplexandmaybethelegacyofhistoricalfederalpolicyorsweepingsocialtrendsAlthoughthisRoadmapdoesnotexplicitlyincludestepsonreviewingthishistoryweencouragereaderstodevelopadeepunderstandingoftheprobleminyourowncitywhileconsideringcurrentstateandfederalcontext
UsingtheRoadmap
6
IninterviewswithGovExonthepolicychangeprocessintheircitiespeoplewasthewordmostcommonlyusedbyintervieweeswhendescribingthevariousstepstoimplementanewpolicyorprogramYouwillnoticethatthisisathemethroughoutthisRoadmapThefirstsectionbeginswithidentifyingtherelevantpeopleThechapteronidentifyingrolesforkeypartnersservesasafoundationforthefollowingstepsThediscussionofevidenceencouragesreaderstoconsiderwaysinwhichpeoplecanbuildanduseevidencetodesignstrongprogramsandinitiativeschangehumanbehaviorassessimpactensureeffectiveuseofpublicresourcesandimprovelives
Meaningfulapplicationofevidencebypolicymakersdoesnothappenwithone-timesharingofcuttingedgeevidenceRathermeaningfulinfluencerequiresalong-termstrategytoformalliancesanddevelopknowledgeofpolicymakersandpolicymakingInadditiontoworkingwiththerightpeopleandunderstandingevidenceandcontextwehopethatRoadmapuserswillformapeernetworkthatsharesnotjustwhichpolicysolutionsexistbutwhatimplementationlookslikeAsonecitywespoketoindevelopingthisRoadmapputitalotofourcitysproblemshavebeensolvedbyotherswejustneedtofigureouthowtoimplementthesolutions
TheexamplesthroughouttheRoadmapareillustrativeofgovernmentsthathaveeffectivelyimplementedtherelevantstepsinthisRoadmapbutpleasenotethatnocitydoesallofthisperfectlyTogetherthestepsinthisRoadmaparemeanttohelpcitiesframetheirproblemscreateademandforevidenceandbuildsustainablecoalitionsformeaningfulpolicychange
UsingtheRoadmap
7
GatherinformationabouttheproblemInyourearliestphaseitiscriticaltodevelopacomprehensiveunderstandingoftheproblemYouwillwanttolearnaboutthepeoplewhoareaffectedbytheproblemyouaretryingtosolveandthosewhocanimpacttheproblemFromthereyouwillgetinsightintotherootcauseoftheproblemtheworkthathasbeentriedinthepastandtheeffectivenessofthoseeffortsTheworkfromthischapterwillserveasthefoundationforactivitiesinsubsequentchaptersYoumayrevisitthesestepsasyouidentifyindicatorsandplanyourimplementationandanalysis
BesuretodedicatetimeandresourcestothisstepItcanbetemptingtoskipaheadtolookingforsolutionsortoassumethatyouhaveanunderstandingoftheproblemasaresultofapublicengagementprocess
TheCityofLewisvilleTXconductedaCitywidepublicengagementprocesstodevelopitsLewisville2025StrategicPlanOneoftheBigMovesoutlinedintheplanwasExtendingtheGreenorexpandinggreenspaceandconnectingrecreationareaswithneighborhoodsTheactionstepstoachievethisgoalincludedcleaningupcreeksandremovingdebristomakethemmoreaccessibleThecleanupswentsmoothlyinseveralneighborhoodsbuttheCitymetwithoppositionfromoneneighborhoodthatpreferredthenaturalhabitatandexpressedconcernabouttheenvironmentalimpactofthecleanupsAccordingtoLewisvilleiftheCityhadmappedallofthestakeholderstodeterminewhoneededtobeatthetableandwhowasmissingfocusinginformationgatheringontheneighborhoodlevelithaveavoidedthisproblem
Startwiththewhothepeople
Establishaprojectteam
BeginbynamingtheprojectteamforthisprocessincludingaprojectleadTheprojectleadshouldbesomeoneinCitygovernmentwhoisrespectedcangalvanizeotherkeypartnersandhasthetimetoleadthisworkThisissomeonewhomayhavetohavedifficultconversationswithCityleadershipdepartmentsandotherfundedpartnersItisnotnecessaryfortheprojectleadtobeapolicyexpertItismoreimportantforthemtohavetherightrelationshipsandpositionwithintheorganizationTheprojectteamshouldincludeindividualswithdiverseperspectivesfrommultipledepartmentsThereshouldbesomeonewithpolicyexperienceontheprojectteamtogroundtheconversationinrealisticactionHavingateammemberwhocancoordinatewithresearchersonrigorousanalysisandtranslatethatworkforothersiskeyAdditionallybesuretoincludefrontlinestaffworkersTheywillhavevaluableinsightbasedontheirinteractionswithresidentsandpolicysolutions
Gatherinformationabouttheproblem
8
havetomakesensetothepeoplewhowillbechargedwithimplementingthemBuildingastrongdiverseprojectteamisimportanttoensurethattheworkdoesnotliveanddiebyoneperson
Identifykeypartners
NextidentifythepeoplewhotheprojectteamcanworkwithontheissueyouarefacingincludingpeoplewithlivedexperienceintheproblemyourcityistryingtosolvedepartmentstaffandsubjectmatterexpertsoutsideofcitygovernmentThisistheconceptbehindbuildwithnotforamovementledbyLaurenellenMcCannthatemphasizescommunityexpertiseandleadershipMappingoutkeyplayersandtheirrolesiscriticaldotoearlyintheinformationgatheringphasesincecoalitionbuildingissuchanintegralpartofthepolicychangeprocessThisprocessmightconsistofstakeholdermapping(describedbelow)communityresourcemappingandculturalmappingInyourmappingremembertoconsiderpotentialalliesandthosewhomaypresentroadblocksalongtheway
PolicymakingisnottopdownorlinearInsomecasescitiesmayhavetosharedecision-makingpowerwithotheractorsForinstanceinmanycitiesthelocalContinuumofCare(CoC)istheleadentityonhomelessnessservicesCoCsapplyforandstewardfederalfundstoaddresshomelessnessandcoordinateservicesamonglocalprovidersAlthoughcitygovernmentsdohavearoleinCoCstheyarenotalwaystheleadagencychargedwithdistributingfundsandthereforetheirroleinhomelessnesspolicymaybelimitedCountygovernmentsoftenhavehealthdepartmentswhilecitiesdonotInthosecasesthebalanceofpowerfavorscountygovernment
TheCityofDentonTXwasrecentlyfacedwithanissueofresidentcomplaintsaboutanewdocklessbikesharethatarrivedintownandisviewedbysomeasexpensivelitterInresponseDentonsCityCouncilwhichwasalreadydiscussingpotentialrulesforbikesharecompanieselevatedthisissueandchargedcitystaffwithidentifyingasolutionPresentlytheCityisreviewingordinancesfromnearbycitiestodeterminehowtheyaresolvingthisproblemandwhatsolutionsmaybefeasibleinDentonThemappingexamplebelowrepresentslocalactorsinDentonwithastakeinthisissuewhohavemultipleperspectivesandlevelsofpower
Gatherinformationabouttheproblem
9
UsethistemplatedevelopedbytheCentreforPublicImpact(CPI)tobrainstormwhoyoumightreachouttoasyouembarkontheroadtopolicychangeRemembertothinkabouthoweachpersonorgrouprelatestotheproblem
BegintodefinethewhatortheprobleminyourCity
TherestofthissectionwillhelpyourefinetheproblemstatementFornowgathersomekeypartnerstoclearlyarticulatewhychangeisneededUsethetemplatebelowdevelopedbyDenverPeakAcademytocapturetheproblemstatementYouwillbuildonthislater
ProblemStatementndashWhyChangeisNeeded(1-2Sentences)
ThroughourinterviewswithcitystaffweheardthatwhendiscussingmunicipalchallengesitcanbetemptingtojumptoasolutionbeforetakingthetimetoreallyunderstandtherootcauseoftheproblemThismakessensesincecityleadersandstaffareresponsiblefordeliveringservicesandbeingresponsivetoresidentswhileaddressingnewproblemsthatariseinthecity
TherootsoftheproblemsthatcitiesareworkingtosolvearecomplexandmaybethelegacyofhistoricalfederalpolicyorsweepingsocialtrendsTohelptheproblemseemlessdauntingitcanhelptobreakthemdownintotherootcausetoidentifyamanageablecomponentoftheproblemUsingafishboneexerciseorfivewhystechniquecanalsohelp
Gatherinformationabouttheproblem
10
tofindtherootcauseoftheproblemandavoidsimplyaddressingsymptomsoftheproblemandrepeatingthemistakesofhistoryToolssuchasjourneymapscanhelpyouunderstandtheexperienceofthosewhoareaffectedbytheproblem
TheCentreforPublicImpactwithsupportfromBloombergPhilanthropiesdevelopedanexercisetosupportcitiesinidentifyingtherightsizedproblemAccordingtoCPIarightsizedproblemisonethatislargeenoughtopotentiallyapplytoothercitiesbutsmallenoughtosolvewithinareasonabletimeframe
AtthebeginningofMayorBerkesadministrationintheCityofChattanoogaTNveteransexperiencinghomelessnesswereaskingtheCityforhelpinfindinghousingTalkingtothispopulationrevealedthatveteranshadtroubleidentifyingaffordablehousingwhichmotivatedtheAdministrationtoworkonasolutionTheproblemofveteransexperiencinghomelessnesswastherightsizebecauseitfocusedononesegmentofpeopleexperiencinghomelessness-veterans-andtheirspecificchallengeinidentifyingaffordablehousing
Developacomprehensiveunderstandingofhowyourcitycurrentlyaddressestheissue
IntheChattanoogaexamplementionedaboveCitystaffdiscoveredthatveteranswerehavingtroublenavigatingtheCitysvariousaffordablehousingprogramsandgettingaccesstothehousingandservicesthattheyneededAtthesametimenewlyelectedMayorBerkewasexploringcurrentandfutureopportunitiestoservethispopulationRatherthancontinuetousefederalfundingforemergencysheltersMayorBerkedirectedcitystafftoprioritize
Gatherinformationabouttheproblem
11
HousingFirstwhichvaluesconnectingpeoplewithpermanenthousingovertransitionalhousingandsupportiveserviceafterspeakingwithrepresentativesattheUSDepartmentofHousingandUrbanDevelopment(HUD)
ThisexaminationofwhattheCityiscurrentlyfundingandthesourcesofthatfundingwasaturningpointinChattanoogaUnderstandingthisproblemfromasystemsperspective-includingpoliciesandfundingstreamsatmultiplelevelsofgovernmentsocialnetworksthroughwhichinformationisoftenspreadtechnologythebuiltenvironmentofmarket-rateandpublichousing-helpedthenewMayoranalyzeexistingeffortsandleadstaffandotheractorsinthesamedirection
Begintogatherthedata
TotrulycapturethesuiteofservicesthatyourCityisprovidingaskdepartmentstafftolisttheprogramsrelatedtoyourproblemstatementthatyourcityrunsorfundsWhenlistingtheprogramscapturedetailssuchas
programnamethenameofthedatasetthatisrelevanttoeachprogramadescriptionofthedatasetthesourceofthedatasetifthedataisinternalorexternalwhetherthisdataisalreadybeingcollected
ThisinformationwillformthebasisofyourinventoryofadministrativedataordatasetsthatarekeptbygovernmentsforoperationalpurposesThismayincludecallcenterrecords(311911)taxdatapropertyassessmentsinspectioninformationsocialservicerecordsetcAtthispointlistallrelevantadministrativedatasetsYouwillprioritizewhichdatasetstogathermoreinformationonlaterwhenyouaredevelopingindicators
Asampletemplatethatcanbeusedtocapturethisinformationisbelow
ProblemStatement____________________________________________
Gatherinformationabouttheproblem
12
ProgramName
DatasetTitle
DatasetDescription DataSource
Departmentorexternalpartner
DoestheCityalreadycollectthisdata
Whatisthenameoftheprogram
Human-readablenameofthedataset
Shouldbeunderstandablebynon-technicalusers
Whatistheoriginalsystemapplicationfilethathousesit
Departmentorexternalpartnerthatmaintainsthedata
Isthisdataalreadycollected
TogatherthisinformationconsiderassigningonepointpersontosurveyorinterviewcitystaffSpeakingwithfrontlineworkersstaffwhoworkwithfederalgrantsdepartmentheadsandeveryoneinbetweenwillensureadeepunderstandingoftherelevantprogramsanddatasetsDuringthecourseoftheseconversationsbesuretoaskcitystaffwhichorganizationstheyworkwithexternallyandcapturethatinformationaswelleveniftheyarenotrecipientsofcityfundingInadditiontoconversationswithexpertsincitygovernmentyourcitysbudgetdocumentsareanothergreatsourceofinformation
Collectdataandobservetrends
Nexttrytogainanunderstandingofwhytheproblemstillexistsifithasbeensomethingyouhaveworkedonforawhileorwhyitisemergingnowifitisnew
AnalyzingadministrativedatacandemonstratepasteffectivenessofinitiativesrelatedtoyourproblemstatementUsingthedatasetinventorythatyoubeganintheprevioussteptalktodepartmentstaffaboutwhichdatasetsaremostrelevanttotheproblemandgatherafewyearsworthofhistoricaldataBegintolookfortrendsinthisdatadiscussingpotentialreasonsforthosetrendswithrelevantstaffCanthetrendsbeexplainedbyseasonalitychangesinfundingstaffingfluctuationsorsomeotherreasonBasedonthehistoricaldataanalysiscreateabaselineagainstwhichfuturechangescanbeobserved
OtherdatasetscanhelpprovideadditionalcontextForinstanceifthenumberofpeopleexperiencinghomelessnessinyourcityisincreasingitmaybehelpfultotakealookathousingaffordabilityIfyoudonothavethedatayouneedconsideraproxymeasureAsanexamplehousingisoftenconsideredaffordableifitcosts30orlessofahouseholdsincomeButifhouseholdincomedataisnotavailableatthecitylevelAreaMedianIncome(AMI)canbeahelpfulproxy
Gatherinformationabouttheproblem
13
QualitativedatacanalsohelpanswerquestionsaroundtheeffectivenessofpreviouseffortsTrytoobtainavarietyofperspectives--bothmainstreamandoutliers--fromnetworksbeyondyourowntoprovideyouwiththegreatestvantagepointforevaluatingpotentialimpactUnderstandinghowthoseeffortswerereceivedbydifferentpartsofyourcommunityandiftheremaybeelementsofprevioussystemstopreservewillbecrucialtothesuccessofanypolicychangeThisinformationmaybeobtainedthroughfocusgroupsinterviewsandsurveysFormoreinformationonhowtousethesequalitativeresearchtoolspleaseseetheseresourcesdevelopedbytheAgencyforHealthCareandResearchandUrbanInstitute
FinallytakealookatnationaldatatoseewhereelsethisproblemmightexistThisresearchmaybehelpfulinlaterstepswhenyouarelookingforpromisingpoliciesinothercities
Puttingitalltogether
UsethetemplatebelowdevelopedbyDenversPeakAcademytocaptureyourrefinedrightsizedproblemstatementandcurrentandfuturestatesWorkingthroughthisexercisewithpartnersishelpfultogainbuy-inTheresponseswillbeimportantwhengettingotherpartnersonboard
WhyChangeIsNeeded
Yourright-sizedproblem
CurrentState
Analysisoftrendsindata
FutureState
Ideallywhataretheoutcomesyouhopetoachieve
Checklist
Thekeystepstogatheringinformationarebelow
Atthispointfocusontheproblemratherthanjumpingtosolutions
Listentopeoplewhoareaskingforhelpandtoadvocateswhoareraisingissues
Beasspecificaspossibleinidentifyingtherootcauseandhoninginonthepartoftheproblemthatyourcitycanaddress
Identifytrendsbasedonmultiplesourcesofdataincludinghistoricaladministrativedataproxymeasuresqualitativedataandnationalmeasures
Gatherinformationabouttheproblem
14
Captureyourproblemstatementtheresultsofyouranalysisanddesiredoutcomes
Gatherinformationabouttheproblem
15
StrategicallyengagekeypartnersPolicytheoryresearchoftenfocusesonincreasingpolicymakersaccesstoevidenceasawaytoavoidtakingshortcutslikemakingdecisionsbasedonemotionImplementationsciencesuggeststhatincreasingaccesstoevidenceisnotenoughandthatpolicychangeshouldbeexaminedinthewidercontextofdebatecoalitionformingandpersuasionMeaningfulpolicychangerequiresldquoalong-termstrategytoformalliancesanddevelopknowledgeofpolicymakersandpolicymakingratherthanusingevidenceinpolicymakingonanadhocorone-offbasis
UnderstandingnetworksandsystemsisimportanttodevelopsustainablepolicychangeThissectionismeanttohelpreaderstakethestepstobuildengagementstrategiesforpartnersthatrepresentdifferentsystemsandnetworksEngagementshouldbeanongoingprocessofconnectingpeopletooneanotherandtoanissueEngagementshouldnotbeinnameonly-wehaveallexperiencedengagementactivitiesthatgivepeopleaspecificroletoplaythatleadstoaneatpredictableoutcomeAccordingtotheBuildWithmovementundervaluingandunderinvestinginengagementstrategiescanleadtoadisconnectofcertainindividualsorgroupsandmistrustofgovernmentWhendevelopingengagementactivitiesrememberthathowpeopleengageandwhethertheyengagehastodowithhowyoustructureopportunitiesforengagement
Conveneacoreadvisorygroup
AnadvisorygroupcomprisedofpeoplefrommultiplesectorscanbehelpfulasyouconsiderthelocalsocialenvironmentalandpoliticalcontextinwhichyouintendtointroducepolicychangeRevisityourstakeholdermapanddeterminewhowouldbebesttoserveonthisgroupTomakesureyouareconsideringtheproblemsandsolutionsthroughanequitylensincludepeoplewithlivedexperiencewhohaveexpertiseintheeffectivenessofsystemswhetherthosesystemsfunctionasintendedandhowtheymayimpacthumanbehaviorYoumayalsowishtoincludesomeonewithexperienceworkingwithdataandanalysis-thispersoncanhelpdeterminethetypeofdatathatmaybereadilyavailableandusefultocollectandcanalsohelpplanandshapeyouranalysistofigureoutwhichprogramsareeffectiveBesuretoinvitefront-lineworkersaswell-itscriticalthatwhateversolutionscomeoutofthisprocessaremeaningfultothosewhowillberesponsibleforimplementingthem
BringingpeopletogethercanbechallengingbutthereareafewwaystosetandmanageexpectationsFirstmakesurethatthegroupisawareoftheproblemstatementcurrentstateandfuturestate-thiscanhelpstarteveryoneoffonthesamepageNextspend
Strategicallyengagekeypartners
16
sometimegettingtoknoweachotherrsquosworktolearnabouteveryonesstrengthsThissharedunderstandingcanhelpasyouclarifyrolesandresponsibilitiesofeachpartnerinthegroup--thisisimportanttoavoidfinger-pointinginthefutureandtoensurethatnoneofthetasksslipthroughthecracksInsomecasestherearemanyactorswithdifferentresponsibilitiesasinthecaseofBaltimoreCitys38-memberYouthFundTaskForcewhichischairedbynonprofitandcorporatepartnersandincludesrepresentativesfromthenonprofitandfaithcommunitiesstudentsthebusinesssectorlocaluniversitiesCityCouncilandtheMayorsOffice
OncerolesandresponsibilitiesareclearyoumayconsiderdevelopingacharterguidelinesortermsofreferencetosetexpectationsforthegroupBesuretodistinguishthisfromotherboardsandcommissionsthatyourcitymayhaveNextyoumaywishtoestablisharegularongoingmeetingscheduleBesuretosendagendasandmeetingmaterialsaheadoftimeandclarifytheobjectiveofeachmeeting
InSeattletheCitydevelopeditsHousingAffordabilityandLivabilityAgenda(HALA)torespondtorisinghousingcostsandhomelessnessThisisatrendinotherwestcoastcitiesaswellwhichmaybeexplainedbyaninfluxofnewresidentswhoareattractedbytheemergenceoftechjobstheattractionoftheoutdoorlifestyleortherecentpassageofprogressivepolicieslikelegalizedgaymarriagea$15minimumwageandlegalizationofmarijuanaSeattlebroughttogetheramulti-stakeholderadvisorygroupcomprisedoflocalhousingexpertsrentersandhomeownersdevelopersandgovernmentofficialstoshareinformationabouttheirneedsanddeveloprecommendationsbasedontheseneedsTheinterestsandmotivationsofthisgroupwerevariedandittookabouttenmonthstofindcommongroundTheseriesofrecommendationsthattheadvisorygroupcreatedismeanttoserveresidentswithaspectrumofneeds
Buildaprocessforcommunityparticipation
Ofcoursenoteveryonecanbeincludedinanadvisorygroupandadvisorygroupmembersmaynotnecessarilyrepresentthevariedperspectivesoftheirpeers-forinstancenoteveryhomeownerhasthesamesetofneedsTogatherinputfromawiderswathofthepopulationitcanbehelpfultobuildaprocesstoinvitethepublicatlargetosharetheirideasCitiesdothisinavarietyofwaysfromonlineconversationsandsurveystoin-personmeetingsRememberthatbeingintentionalaboutgatheringtheviewpointsofpeopleismoreimportantthanusingthelatesttechnologyandtools
TheinsightgeneratedbycommunityparticipationexercisescanbeincludedinyourreviewofevidencewhichisdiscussedinasubsequentchapterInWestVirginiaastatethathasbeenhithardbytheopioidepidemicstatehealthofficialsdraftedanopioidresponseplanthatwasinformedbypublicengagementaswellaslocalandregionalsubjectmatter
Strategicallyengagekeypartners
17
expertsThepublicwasinvitedtopresentcommentsonideasfordecreasingoverdoseearlyinterventiontreatmentandrecoveryPublichealthandmedicalfacultyfromWestVirginiaUniversityMarshallUniversityandJohnsHopkinsUniversityprovidedresponsestothecommentsTheresultwasanOpioidResponsePlanthathealthcommissionerRahulGuptapresentedtoGovernorJimJustice
AsamplespectrumtoplanyourowncommunityparticipationprocesscreatedbyLaurenellenMcCannisbelow
SourceLaurenellenMcCann
Community-basedparticipatoryresearchisanotherwaytoworkwiththecommunitytoimproveoutcomesCBPRmeansthecommunityisinvolvedinallaspectsoftheresearchprocesswhichisofteniterativeInCBPRtraditionalresearchersandcommunitymembersshareknowledgeaswellasresourcesandcredit
CBPRInActionInSanFranciscoanenvironmentaljusticenonprofitteamedupwiththeCitysDepartmentofPublicHealthandanoutsideevaluatortounderstandtherootcausesofobesityandfoodinsecurityinlow-incomeneighborhoodsandidentifysolutionsThroughresearchmethodssuchasinterviewingresidentsandstoreownersdiagrammingstoreshelvesGISmappinganeconomicanalysisandreviewofsimilarpoliciesinnearbycitiestheteamobservedthatthestoresthatresidentshasveaccesstowerestockedwithlesshealthyfoodsandthatresidentswantedmorehealthyfoodandlessalcoholandtobaccoThisprojectresultedinlocalandstatelegislationtoencouragestorestostockfreshproduce
Strategicallyengagekeypartners
18
Developrelationshipswithsubjectmatterexperts
OnewaytokeepsubjectmatterexpertsengagedistoinvolvetheminthecommunityparticipationprocessasdescribedaboveIntheWestVirginiaexamplesubjectmatterexpertsrespondedtoandsynthesizedthepubliccommentsOtheropportunitiesaredescribedbelow
AdvisoryAcitydepartmentheadMayororstaffmembermaydevelopaone-on-oneadvisoryrelationshipwithuniversityfacultymemberoracollaborativebetweentheCityandUniversitytoaddressaparticularproblemAgreatplacetostartistolookforrelevantpracticecenterswithinyourlocaluniversityandreachingouttosubjectmatterexpertsInadditiontotheirexpertiseontherelevantpolicytopicheorshemayalsobeabletoidentifyotherexpertsandevenprovidesupportonhiringtherighttalenttoleadcertainprogramsAnadvisoryrelationshipwithauniversitysubjectmatterexpertcanalsohelpwhencitiesareconsideringbringinginnewprogramsTheseexpertsareoftenfamiliarwithprogramsthatarebasedonnationalmodelsandcanprovideinsightintoaprogramsoriginandlineageeffectivenessinotherplacesandthepotentialforimplementationinyourcityAcademicscanalsohelpidentifyindicatorsbasedongovernmentindicatorsorliteratureFinallyrelationshipsbetweenpolicymakersandresearchersprovidethemostbroadlyeffectiveroutetopullingresearchintopolicy
ResearchandevaluationAcademicexpertsmaysupportcitiesinevaluation-indeedevaluatingcityinitiativesisatraditionalmodelofcityuniversityrelationshipsSharingtheinformationyoucollectedintheGatherInformationphasecanhelpresearchersunderstandyourworkandgoalswhichinturnwillstrengthenthesupportthattheycanprovideDiscussingdetailsaboutyourinterventionandevaluationplanatthebeginningoftheprocessmeansthataresearchpartnercanhelpwiththeevaluationdesignanddatacollectionForinstancetheWilsonSheehanLabforEconomicOpportunitieswalkscitiesthroughconsideringthefollowingquestionsbeforepartneringonanevaluation
1 Isthereaclearpolicyprogrammodelwithobjectivescreeningcriteria
2 Isitpossibletoevaluatewhathappensbothwiththepolicyprogramandintheabsenceofthepolicyprogram
3 Arethereoneormoremeasurableoutcomesofinterest
4 Arethereenoughpeopleforrigorousquantitativeanalysis
5 Whatdatawillbenecessaryforthestudyandwhataretheprocessesforaccessingthesedatasource(s)
6 Isthereacompellingreasontoevaluate
Strategicallyengagekeypartners
19
InadditiontoassessingreadinesstoconductanevaluationresearcherscanhelpyouhoneinonthepurposeoftheevaluationTheWorldBankdescribesseveralpotentialusesofevaluationwhicharedefinedhereInbriefevaluationcanhelpdecisionmakers
makeresourceallocationdecisionsrethinkrootcausesofproblemsidentifynewproblemsdecideonabestalternativedeterminewhetherprogramsorpoliciesarebeingimplementedwithfidelitydeterminewhetherprogramsorpoliciesarehavingthedesiredimpact
Researcherscanalsoprovidesupportinsharingtheresultswiththescientificcommunitythroughpapersandconferenceswhichisvaluableincreatingapeernetworkofevidence-basedcitypractitioners
TrainingandtechnicalassistanceUniversitypartnersespeciallythoseatpracticecenterscanbehelpfulindevelopingorupdatingtrainingcurriculumtrainingthetraineranddeliveringtrainingtocitystaff
Strategicallyengagekeypartners
20
ReviewtheevidenceandcollectdetailsOnceyouhaveathoroughunderstandingoftheproblemdataandactorsyouwillneedtoconductyourownresearchonavailableevidenceandlearnaboutthelandscapeofpolicyoptionsYourgoalwillbetodevelopanaccessiblelistofinformationaboutpoliciesthathavehelpedsolvesimilarproblemsinotherplacesThischapterwillguideyouthroughthestepsofpreparingyourreviewersunderstandinglevelsofevidenceandcollectingthedetailsonevidencethatyouwillneedfordiscussionwithdecisionmakersinyourcity
Selectandprepareyourreviewers
AssignapersonorteamtoreviewrelevantinterventionsthathavedemonstratedsuccessinotherjurisdictionsTheirgoalwillbetoproduceareviewofalloftherelevantprogramspoliciesorinitiativesthatyourteamcouldconsiderimplementingTheywillbeconductingaliteraturereviewsupplementedbyinterviewswithexpertsandpeerswhohaveinformationaboutpotentialprograms
SincethepeopleyouassigntothistaskareprovidingthebaselineofpotentialoptionstoimplementitisimportantthatyouchoosepeopleyouareconfidentwillbethoughtfulandthoroughintheirworkItisalsocriticalthattheyareawareoftheimportanceofsourcecredibilityEnsureyourreviewershavefamiliarizedthemselveswithrulesforcriticallyanalyzinginformationsourcesbeforetheygetstartedIftheyareincludingnewsarticlesaspartoftheirsearchtheyshouldalsobeawareofmethodsforevaluatingnewscredibility
AskyourreviewerstodevelopaplanandtimelinetocompletethereviewThisstartswithsolidifyingagoverningresearchquestionOnewaytoknowyouarecreatingadetailedresearchquestionistousetheacronymPICOremindingyoutoincludereferencetothepopulationintervention(ifyouhaveoneinmind)comparisonandoutcomeelementsinyourresearchquestionForexampleifyouknowthatyouhaveapopulationofteenagerswhoarehomelessyouhavelearnedfromyourstakeholdersthatmanyoftheseteenagersidentifyasLGBTandyouhaveheardsomethingabouttheapproachofcreatingdrop-incentersfordeliveringsupportyoucouldask
ForLGBTyouthfacinghomelessness(population)howeffectivearedrop-incenters(intervention)relativetostandardsocialservices(comparison)atsettlingtheminpermanenthousing(outcome)
Ifyoudonothaveaninterventioninmindyoucanleaveouttheinterventionandcomparisonandsimplyask
Reviewtheevidenceandcollectdetails
21
HowcanwehelpLGBTyouthfacinghomelessness(population)obtainpermanenthousing(outcome)
OnceyourreviewershavetheirresearchquestionsusethequestionstocreatealistofkeywordstheycanuseintheirliteraturesearchThiswillhelpkeeptheircollectionofarticlesreasonablyfocusedHavethemdescribewhattheirsearchprocesswillbeincludinglistingthedatabasestheywillreviewandtheirplanforstoringtheinformationtheywillcollect(Herearesomeadditionalsuggestionsforstructuringareviewplan)
WhatyourreviewerswillbedoingisalightweightversionofarapidreviewTotheextentthattheywouldliketodevelopamethodologyforspecificsearchprotocolsincludingcriticalappraisalprocessestheycancertainlygoaheadanddoso
Usedifferentlevelsofevidenceforthemostcompletepicture
AsdiscussedintheWorkingwitharangeofevidencechaptertheNestaStandardsofEvidenceframework--reproducedbelow--ishelpfultounderstandthelevelofconfidencethataprogramorinterventionishavingapositiveimpactThisapproachletsyouputinterventionsthatyouhaveheardaboutlocallybutwhichhavenotbeenstudiedtogetherwiththosewithademonstratedcausaleffectThisishelpfulsinceanumberofpoliciesprogramsandpracticesthatyouwillwanttoconsidermaynotyethavemuchofaresearchrecord
Reviewtheevidenceandcollectdetails
22
NearlyanypieceofinformationaboutaninterventioncanbeclassifiedwithinthestandardsofevidenceForexampleifsomeonehasacrediblehypothesisabouthowaninterventioncouldproducebenefitthistypeofevidencewouldbeclassifiedasLevel1ThistypeofevidencecouldincludeinterventionsinothercitiesthatyoureadaboutinthenewsorhearaboutataconferenceForinstancewhenUSDAallowedschoolsandschooldistrictstoprovidefreebreakfastandlunchtoallstudentsthroughtheCommunityEligibilityProvision(CEP)manydistrictsimplementedthisoptionevenbeforetherewasevidenceofeffectivenessThiswouldfitintoLevel1sinceitbuildsonthelogicthatprovidingaccesstomealswilldecreasefoodinsecurity
YoumayalsofindpiecesofinformationthatqualifyasLevel2evidenceForinstanceasmoreschooldistrictsimplementedCEPprincipalstestifiedaboutpositivechangestheyobservedinstudentssuchasamorepositivedemeanorthatcanleadtobetteracademicperformanceDataalsoshowedthatthestudentsdescribedwerereceivingmoremealsthanbeforetheprogramsimplementationHoweveritwastoosoontobuildevidencetoshowacausalrelationshipbetweentheincreaseinfreemealsandincreasedacademicperformance
ldquoLower-levelrdquoevidencecanconsistofnuancedbutvaluableelementsthatrelatetoaninterventionrsquospotentialfitwithacommunityQualitativeinformationsuchastestimonyfrompeoplewhohaveexperiencedaninterventionandthosewhohavefacilitatedorimplementeditwillalwaysbeusefultoconsiderandprovidesdetailedlocalcontext
Experimentalstudies--describedstartingatLevel3withcontrolledexperimentsandgoingthroughLevel5withfullyfleshedoutprotocolsandprocessesforreplication--stilloccupythehigherlevelsofthisdescriptionofevidenceThisprioritizationshowsthevalueofrigorousstudyforevaluatingtheimpactofaparticularpolicyorprogramratherthanotherfactorsthatcouldbeinfluencingoutcomesofinterestRememberthatmultipletypesofevidencecancomplementrigorousimpactevaluationsandgivecityleadersadditionalinsightsandbroaderperspective
Accessinformation
Encourageyourreviewerstobegintheirsearchforpromisingpracticesingovernmentandnon-profitclearinghousesandinacademicpolicyjournalsWorkpublishedinthesesourceshasalreadybeensubjectedtoahighlevelofprofessionalscrutinyandgenerallycontainsagreatdealofdetailwhichmeansthatyourreviewersaremorelikelytobeabletopullouttheinformationyouneedtomakeagooddecision
Clearinghouses
Reviewtheevidenceandcollectdetails
23
Herearesomeoftheclearinghouses(collectionsofprogramevaluations)forevidence-basedpoliciesandprograms
ResultsFirstClearinghouseDatabaseThePewCharitableTrustsCollectstogethereightotherclearinghousestoprovideinformationoninterventionsinthefollowingcategoriesadultcriminaljusticeeducationchildwelfarehealthjuvenilejusticementalhealthsubstanceabuseandyouthdevelopmentIndividualclearinghousesincludedinResultsFirstare
BlueprintsforHealthyYouthDevelopment-foryouthdevelopmentandviolenceprevention
CaliforniaEvidence-BasedClearinghouse-forchildwelfareinterventions
TheLauraandJohnArnoldFoundationrsquosSocialProgramsThatWork-forsocialservicesprograms
TheUSDepartmentofEducationrsquosWhatWorksClearinghouse-foreducationinterventions
TheUSDepartmentofHealthandHumanServicesrsquoResearch-TestedInterventionPrograms(RTIPs)-forcancercontrolinterventions
TheUSDepartmentofHealthandHumanServicesrsquoTeenPregnancyPreventionEvidenceReview(TPPEvidenceReview)-forsexualriskbehaviorandsexualhealthoutcomes
TheUSDepartmentofJusticersquosCrimeSolutionsgov-forcriminaljusticejuvenilejusticeandcrimevictimservices
TheUSSubstanceAbuseandMentalHealthServicesAdministrationrsquosNationalRegistryofEvidence-basedProgramsandPractices(NREPP)-substanceuseandmentalhealthinterventions
TheUniversityofWisconsinPopulationHealthInstituteandRobertWoodJohnsonFoundationrsquosCountyHealthRankingsandRoadmapsWhatWorksforHealth(WhatWorksforHealth)-healthbehaviorsclinicalcaresocialandeconomicfactorsandthephysicalenvironment
UKWhatWorksCentresTheBritishgovernmenthascharteredanetworkofsubject-specificorganizationswhichcollectsynthesizeandreportoutevidenceontheeffectivenessofpolicyprogramsandpractices
NationalInstituteforHealthandCareExcellence(NICE)-healthandsocialcare
Reviewtheevidenceandcollectdetails
24
SuttonTrustEducationalEndowmentFoundation-educationalachievement
CollegeofPolicingWhatWorksCentreforCrimeReduction-crimereduction
EarlyInterventionFoundation-earlyintervention
WhatWorksCentreforLocalEconomicGrowth-localeconomicgrowth
CentreforAgeingBetter-qualityoflifeforolderpeople
WhatWorksCentreforWellbeing-wellbeing
WilsonSheehanLabforEconomicOpportunities(LEO)UniversityofNotreDameProvideslinkstotheirevaluationsofinterventionsincriminaljusticeself-sufficiencyeducationhealthandhousingandhomelessness
ClearinghouseforLaborEvaluationandResearchUSDepartmentofLaborProvidesresearchonworkforceandlabor-relatedpoliciesandprograms
J-PALEvaluationsAbdulLatifJameelPovertyActionLabStudiesonpoverty-reducinginterventionsfromaroundtheworld
CampbellSystematicReviewsCampbellCollaborationSystematicreviewofpolicyinterventionsbyscholarlycoordinatinggroupsinsevenissueareascrimeandjusticeeducationinternationaldevelopmentmethodssocialwelfareknowledgetranslationandimplementationanddisability
Benefit-CostResultsWashingtonStateInstituteforPublicPolicyAWashingtonState-charteredprogramtoestablishthelikelihoodofrelativecostsandbenefitsforprogramsacrossthefollowingareasjuvenilejusticeadultcriminaljusticechildwelfarepre-Kto12educationchildrensmentalhealthhealthcaresubstanceusedisordersadultmentalhealthpublichealthandPreventionworkforcedevelopmentandhighereducation
EffectiveInterventionsCentersforDiseaseControlandPreventionRepositoryforevidence-informedHIVpreventioncareinterventions(withstepsonhowtoimplement)
Journalsandindexes
OnceyouhavereviewedrelevantinformationaccessiblethroughclearinghousesitisworthtakingalookatindividualstudiespublishedasscholarlyarticlesYourreviewerswilldothisbysearchingarticleindexesandacademicjournalsUnfortunatelyanumberofusefulacademicjournalsandindexingplatformsmaynotbegenerallyaccessibletoyourreviewersifyoudonothaveasubscriptiontothemToaddressthisconsiderpartneringwithalocal
Reviewtheevidenceandcollectdetails
25
collegeoruniversitytoprovideyouwithalistofarticlesfromthefollowingsources(Thisisalsoanexcellentopportunitytoseekresearchsupportfromanyacademicstakeholdersyouidentifiedpreviously)
Indexes
GoogleScholarincludesmostpeer-reviewedonlineacademicjournalsandbooksconferencepapersthesesanddissertationspreprintsabstractstechnicalreportsandotherscholarlyliteratureincludingcourtopinionsandpatentsItallowsyoutospecifyapublicationrangeinyoursearchandprovideslinkstolikelysourcesforalllistedarticles
PolicyFileIndexisauniqueresourceforUSpublicpolicyresearchinthatitgrantsusersaccesstotimelyupdatedinformationfromover350publicpolicythinktanksnongovernmentalorganizationsresearchinstitutesuniversitycentersadvocacygroupsandotherentities
PolicyArchiveworkstocollectanddisseminatesummariesandfulltextsvideosreportsbriefsandmultimediamaterialofthinktankuniversitygovernmentandfoundation-fundedpolicyresearch
Anothersourceforhigh-qualityresearchisthereportsoftheCongressionalResearchServiceYoucanfindtheirreportsaggregatedhere
Specificjournals
Oftenthemostusefulacademicjournalsforunearthingpolicyorprogramreviewswillbeineachspecificsubjectarea--thatisineducationjournalsforeducationpoliciesYoucanuseageneralindexlikeGoogleScholartobegintolearnwhichjournalsaremostoftenusefulforyoursearchHoweveryoucanalsosearchthefollowingtwogeneralistpolicypublicationsforissuesandarticlesrelatedtoyourtopic
PublicAdministrationReviewJournalofPolicyAnalysisandManagement
Newssourcessubjectmatterexpertsandinterviews
AlthoughitisvaluabletoknowwhathasbeenexploredthroughascholarlyresearchframeworkitisalsousefultolearnwhatkindofinterventionsarebeingtriedinthefieldrightnowandhowpeoplefeelaboutthesetrialsSearchsubscriptionnewsaggregatorslikeLexisNexisorfreenewsaggregatorslikeGoogleNewsorYahooNewsforyourissueareaplusthewordpolicyorprogramtoseewhatkindsofinterventionsarebeingcoveredinthenews
Reviewtheevidenceandcollectdetails
26
DonotforgettocontacttheadvisorygroupmemberswhowerechosenfortheirsubjectmatterexpertiseInadditiontoprovidinggoodmaterialthroughinterviewsthemselvestheymayhaveideasaboutothersubjectmatterexpertsyoushouldcontacttoaskaboutpotentialinterventions
Peercitiesandnetworks
InadditiontoreadingabouttheworkofothercitiesitisidealifyouareabletospeakwithpeoplewhoareinvolvedwithcurrentimplementationsofinterestinginterventionsInyourcityyoulikelyhaverelationshipswithyourneighborsbutitcanbehelpfultolookoutsideofyourregionforsolutionsaswellTofindcounterpartsinothercitieswhoareworkingonaprogramyouwouldliketolearnmoreaboutyoucancontactstaffinintermediaryorganizations(likeGovExoryourprofessionalassociations)whomightbeinapositiontoputyouintouchimmediatelyorwhomightseeanothergoodrouteforsettingupaconversationHereisalistofnetworksthatmaybehelpful
InternationalCityCountyManagementAssociation(ICMA)NationalAssociationofCounties(NACO)NationalLeagueofCitiesUSConferenceofMayorsWhatWorksCities
Collectthedetails
Onceyourreviewershavefoundevidenceonpossiblepoliciesorprogramstoimplementtheyshouldcapturekeydetailsthatwillhelpdecisionmakersunderstandthepotentialeffectivenessandapplicabilityofeachpotentialintervention
Foreachsourcethathasprovidedinformationyouwanttoconveyusethetemplatebelowtocaptureasmanyofthefollowingdetailsaspossible(Youdonotneedtocompletethisforsourcesthatprovideduplicativeinformationorareotherwisenotthatuseful)
Programorpolicyname (egldquoHousingFirstrdquoorldquoblightedpropertiesregistryrdquo)
Sourcetype (egjournalarticleinterviewsubjectmatterexpertstudyassociationcommunityparticipationprocess)
Sourcedetails (egarticletitleandpublicationnameandtitleofinterviewee)
Evaluationmethodology
(egrandomizedevaluationcomparativeevaluationoftwodifferentgroupsbeforeafteranalysis)
Evidencelevel (egLevel12345)
Programorpolicy
Reviewtheevidenceandcollectdetails
27
description
Programorpolicygoal (1sentencecapturingtheprincipalaimoftheintervention)
Existingimplementations
(WherehastheinterventionbeentriedListasmanysitesasyouareawareof)
Characteristicsofimplementationsites
(ForeachimplementationsiteofferbasicgeographicIDmdashcityandstateThenyouwillwanttocollectthedetailswhicharerelevanttoyourdecisionmakersrsquoconsiderationsaboutwhatmakesalocationldquosimilartordquoorldquodifferentfromrdquoyoursituationYouwillatleastwanttoincludeforeachimplementationsitePopulationnumberPopulationdensity(ruralurbansuburban)Socio-economiccharacteristicsBeforestartingaskyourdecisionmakersaboutanyparticularcharacteristicsofprogramimplementationsitestheywouldwanttoknowbeforeseekingtoreplicate)
Characteristicsofimplementationpartners
(Anyavailabledetailsaboutorganizationsinstitutionsorpeoplewhowereinvolvedwithimplementingtheintervention-egindependentchildcarecenterpublicelementaryschoolcitypublichealthnurses)
Characteristicsoftargetpopulationeligibleprogramparticipants
(Anyavailabledetailsaboutpeoplewhoarethebeneficiariesoftheprogramorotherwiseincludedoraffectedbytheintervention-egnumberofprogramparticipantscategoriesofpropertyownersaffected)
Policyorprogramduration
(HowlonghastheinterventionbeeninplaceineachimplementationsiteHasitendedandifsowhy)
Whatbenefitscamefromeachimplementation
(Anyobservedpositiveeffectsoftheinterventionincludingeffectsize)
Statisticalestimatesofinterventioneffectiveness
(Anyavailablestatisticalestimatesoftheinterventionrsquoseffects)
Whatdrawbackswereobserved
(Anyavailabledetailsaboutunintendedorunavoidablenegativeeffectsoftheintervention)
Programorpolicycosts
(Anyavailabledetailsaboutprogramcostsperimplementation)
Reviewtheevidenceandcollectdetails
28
Reviewtheevidenceandcollectdetails
29
DecideonapolicytoimplementbasedonlocalfactorsTogetthecriticalbuy-infromleadershiptoimplementanewpolicyorprogramtheprojectleadshouldscheduleameetingwithdecisionmakerstopresenttheevidencesurroundingthepolicyoptionscapturedinthetemplateDependingonyourinsightintoyourcityleadershipswillingnesstomakechangeyoumaywishtosharetheresearchyouhaveconductedontherootcausesoftheproblemandsomeofyouranalysisontheimpactofcurrenteffortstodate
Considertheapplicabilityofanapproach
AsdiscussedinthepreviouschapterpolicymakersoftenlookforevidenceofeffectiveprogramsinothercitiesbutoftenthequestionarisesofhowgeneralizableaprogramisfromonecitytoanotherTheAbdulLatifJameelPovertyActionLab(J-PAL)suggeststhatinreviewingevidencefromanotherlocationitishelpfultoidentifythespecificmechanismthatwasused(deliveryofprogramorserviceinformationsharingetc)identifythelocalconditionsthatwereinplaceobservethebehaviorchangethatresultedfromthemechanismandfinallytounderstandlocalcontextandconditions(legalorcostconstraintsculturalnormsetc)Thissetofpresentationelementscanbedrawnfromthelistofdetailscollectedintheresearchtemplate
EducationpolicymakersinChicagoILusedfindingsfromprogramsinKenyaandIndiainwhichchildrenwhohadfallenbehindtheirpeersweregiveneducationalsupporttargetedtotheirleveloflearningThemechanismusedwastargetedteachingandthereweresimilarconditionspresentinallthreecountriesAlthoughsomestudentshadfallenbehindinthecurriculumteacherswereincentivizedtoteachgrade-levelmaterialsConsiderationsofhowthelocalcontextinChicagomightaffectimplementationofatargetedteachingprogramincludedunderstandinghowtheneighborhoodsthatthestudentswerefrommayhaveaffectedtheireducationaloutcomesandthehighcostofteachinginsmallgroupsInChicagoMatchEducationprovidedtwo-on-onetutoringwhichmatchedstudentswithwell-educatedindividualswhohavenothadformalteachertrainingforalower-costmodelTheresultswerepositiveStudentswhoweretutoredperformedbetterinmathandweremorelikelytopasshighschoolclasses
ThegeneralizabilityframeworkproposedbyJ-PALisbelowUsethisasatemplateasyouconsiderpotentialpolicysolutions
Decideonapolicytoimplementbasedonlocalfactors
30
SourceTheGeneralizabilityPuzzle
OnceyouhavenarrowedthepolicyoptionsdiscussthequestionsbelowforeachoptionsyouareconsideringThisstepiscriticalasunderstandinglocalcontextissoimportantinsuccessfulpolicyimplementation
CommunityContextQuestionnaire
Localcontext
DoesthisoptionmakesenseinourcommunityIsthisaonesizefitsallapproachIsanyparticulargroupabsentfromthisapproachIfsoisthatok
Youranswers
Effectiveness
WhatwouldweexpecttoobserveifthepilotisaseffectivehereasinCityX
Youranswers
Timing
DoesthetimingmakesenseIsthistherighttimetoimplementthispolicychange
Decideonapolicytoimplementbasedonlocalfactors
31
WherearethepotentialroadblocksthatcouldaffectthescheduleIsthereanywaytomitigatethisHowlongwouldittaketoseeanimpact
Youranswers
Implementation
DowehavethestaffcapacitytoimplementthisArethereadditionalpartnersthatyoucanbringtothetabletosupportimplementationDoesthismakesensegivenourformofgovernment
Youranswers
Authority
DoesourcityhavetheauthoritytoenactthisIfsowhatistheappropriatemechanismAretherelegalbarriersArethereanyexistingpreemptionissuesfromthestateorfederalgovernmentCanthosebeaddressedthroughlegislationadvocacyIstherepoliticalsupportoroppositiontothis
Youranswers
Cost
HowmuchwillthiscostannuallyHowwillwepayforitCanthepolicybeimplementedwithexistingfunding
IfsoisthisfundingreliableHowwouldthispolicyoptionaffectourcitysbudgetintheshortandlongtermWhatotherfundingoptionscanweconsiderIsthereanalternativethatwouldcostlessandgiveussimilarresultsWillanyconstituencybeadverselyaffectediffundingforprogramsthatimpactthemisshiftedtoanotherprogrampolicyorinitiative
Youranswers
Decideonapolicytoimplementbasedonlocalfactors
32
EstablishaframeworktomonitorprogressNowthatyouhavereviewedtheevidenceconsideredlocalcontextandchosenyourpolicyitistimetodeterminehowyourcitywillmonitorprogressofthisnewapproachCitystaffandleadersneedtoestablishtheinfrastructureforongoingcontinuousdecisionmakingbasedonwhetherthepolicychangeisleadingtothedesiredoutcomesforresidentsandwhetherthechangeismeaningfulandrationaltofrontlineworkersSettingarbitrarygoalsatthelocallevelthatconflictwithgoalsatotherlevelsisconfusingtotheindividualsresponsibleforachievingthegoalsGoalsthataresettoohightoachievecanbedemoralizingtostaff
ThissectionincludesstepstosetupavenueinwhichCitystaffandleadershiproutinelyvisitkeyindicatorstodeterminewhatisorisnotworkingandtomakedecisionsinreal-timeFormorereadingonperformancemanagementpleasevisittheGovExPerformanceManagementGettingStartedGuide
InperformancemanagementtermssuchasindicatormeasureandmetricmaybeusedinterchangeablyInthisguidewewilluseindicatorInyourworkthemostimportantthingistobeconsistentinwhichevertermyouuseThisvideomaybehelpfulinunderstandingthisconceptandcommunicatingittoothers
Establishthefundamentalsandgatheryourteam
FirstdeterminewhetherthereisanestablishedvenueorregularmeetingscheduleduringwhichyoucandiscussprogressIfyourcityhasanexistingperformancemanagementstructureonarelatedtopicaddingadiscussionoftheprogressofpolicychangetoyouragendacanbeagreatideaIfthiswillbeastandalonemeetingdecidehowfrequentlytheprojectteamanddecisionmakerswillmeetandputthosemeetingsonthecalendarfortheentirelengthoftheproject
DeterminewhowillstaffandattendroutinemeetingsonCityprogressThesemeetingsmayincludetheMayorandorCityManagerdepartmentheadstaffwhoareimplementingthepolicychangeandanalysts
Identifyindicators
ItiscriticalthatindicatorsbeanimportantcityandcommunitypriorityaswellassomethingtheCityplaysamajorroleinaddressingTheremaybeinstancesinwhichtheCitywillneedtoreachouttoorcollaboratewithexternalpartnerstogetamorecompletepictureofprogressontheoverallpolicychange
Establishaframeworktomonitorprogress
33
Anoutcomeindicatordescribesthedesiredeffectofthetargetedpopulation
ExampleBellevuesfamilyfriendlyneighborhoodsindicatormeasuresthepercentofresidentswhobelievetheyliveinaneighborhoodthatsupportsfamiliesparticularlythosewithchildren
WithyourkeypartnersrevisityourproblemstatementcurrentstateandfuturestateexerciseandbegintoidentifytheoutcomeindicatorsthatwillillustratesuccessFornowchooseasingletop-levelkeyperformanceindicator(KPI)ItiscriticalthattheKPI-thepieceyouwillmeasure-beapriorityforthecommunityandsomethingtheCityplaysamajorroleinaddressing
Outputindicatorsmeasuretheprocessofprogramorservicedelivery
ExampleSanDiegoMayorKevinFaulconersgoaltorepair1000milesofstreetby2020
ReviewingtheadministrativedatasetinventorybegintoparedowntherelevantprogramsanddatasetsthatsupportyourKPIWhatarethemeasuresofeffectivenessforeachprogramTheseareyouroutputindicators
Thediagrambelowillustratestherelationshipbetweenoutputandoutcomeindicators
ThinkaboutimplementationindicatorsasmeasuringstepsintheprogramsrolloutthatarenecessarytoobtaindesiredchangesImplementationoutcomescanfocusontheinterventionstaffdeliveringtheinterventionanorganizationorsystemandcanaddressfeasibilityadaptabilitysustainabilityandcostTwokeyquestionstoconsiderwhendevelopingimplementationindicatorsare
1 Istheinterventionacceptableandappropriatetothecommunityand
2 IsitfeasibletoimplementinterventionXatthistimewiththeseresources
Establishaframeworktomonitorprogress
34
Amoreextensivelistwithdefinitionsofpotentialimplementationoutcomesislinkedhere
UsethetablebelowtochoosethedesiredimplementationoutcomesforyourpilotandtoplanyourmeasurementtechniqueThisshouldincludelevelofanalysis(individualproviderorconsumerorganizationorsetting)yourmeasurementtechnique(surveysobservationinterviewfocusgroupsadministrativedataanalysis)andthephasewhenyouwillcollectthisinformation(earlymidlate)Theremaybemorethanonelevelofanalysismeasurementtechniqueandphaseofdatacollection
Implementationoutcome
Definitionofoutcome
Levelofanalysisandmeasurement
Yourmeasurementtechnique
Phaseofpilotwhenyouwillcollectdata
Usethespacebelowtocaptureyourindicatorwishlist
Indicatorwishlist
Outcomes_____________________________________________________________________________
Outputs________________________________________________________________________________
Implementationoutcomes______________________________________________________________
Setapublichigh-levelgoal
Thehigh-levelgoaliscriticalinestablishingdirectiontogalvanizethepeoplewhowillbedoingtheworkWhendiscussingthegoalusetheMayorsbullypulpitLookforkeymilestonesinCitygovernmenttosharethegoalstatementsuchasthebeginningofaMayoraladministrationaStateoftheCityspeechorotherpublicremarks
GoalsshouldbeldquoSMARTrdquo
Specificaspossible(innamingthepopulationwhowillbebetterofftheactionthecitywilltake)
Measurable
Achievable
Establishaframeworktomonitorprogress
35
Results-focused
Time-bound
ThegoalstatementshouldincludeyouroutcomeindicatorlikeChattanoogasgoalldquoWewillendchronicveteranhomelessnessbyDecember2016rdquoAdiagramtohepyoubuildaSMARTgoalaroundyouroutcomeindicatorisbelow
Component Description Yourturn
Specific WhowillbebetteroffWhatactionthecitywilltake
Measurable Howwillyoumeasuresuccess
Achievable Isthegoalattainable
Results-focused Whatistheexpectedresult
Time-bound When
Setrealisticambitioustargetsandtimelinesforeachindicator
ToestablisharealisticandambitioustargetforyourgoalstatementusetheadministrativedatacollectedforyourchosenKPItoreviewactualCityperformanceinthemostrecentyearcomparedwiththethreeorfourmostrecentyearsworthofdataBegintocalculatesummarystatistics(egtotalminimumaveragemedianmaximum)toidentifytrendsCreatecumulativeandmonthlychartstoidentifytrendsandanomaliesBasedonyouranalysisofhistoricaldataignoretheanomaliesandassumetheproductivitytrendswillfollowthesameoverallpatternfrommonth-to-monthFinallyapplyanincreasedproductivityfactortotheperformanceeachmonthtoforecastfutureprogressBasedonthisanalysischooseatargetthatinspiresimprovementwhileaccountingforreality
Manycitiesthataremorematureintheirperformancemanagementpracticesrecognizetheimportanceoftakingacommunity-levelapproachtoimprovingoutcomesforresidentsThesecitiesarecreatinggoaltaxonomiesthatincludemeasuresrelatedtopublicsafetyhousingandevenrecreationCitiesdriveprogressonthesegoalsbyimplementingstrategiesthatengagemultipledepartmentsandcollaboratewithserviceprovidersinthecommunity
EventuallyyouwilldevelopgoalstatementsfortheoutputsinordertocapturetheefforttoachievetheoutcomecontainedinyourgoalstatementYoursetofindicatorsshouldbecomposedofshort-andinterim-termsupportivemeasureswithaccuratetargetsandrelevanttimelinesAnexamplegoaltaxonomyfromtheCityofLasVegassResultsVegasprogramisbelowUsethistaxonomyasamodelforyourownsuiteofindicatorsandgoalstatements
Establishaframeworktomonitorprogress
36
Yourturn
Thetemplatebelowismeanttohelpyoucraftagoalstatementforeachsupportingmeasure
Goalstatement
Departments
Outcome
Activities
KPI
Target
Timeframe
Conductanddiscussrelevantanalysis
ThrougharoutineseriesofmeetingscontinuallymonitorprogressofKPIssupportingindicatorsandanyotherdatacollectedsuchasqualitativedatatoidentifytrendsthatwillindicatewhetherthechosenpolicyishavingthedesiredimpactThesedatapointsshoulddrivethediscussionduringthesemeetingspointingtoopportunitiesforfurtherinquiryinto
Establishaframeworktomonitorprogress
37
whatisdrivingtheobservedtrendsAnalystsshouldbeencouragedandtrainedtoaskquestionsofthedepartmentsproducingthedataandconductsitevisitswhenpossibletodeepentheirunderstandingofthedataPresentationoftheanalysisshouldfocusononeideaatatimetokeeptheconversationfocusedonkeypointsandspecificopportunitiesforimprovementYoumayconsiderdevelopingapublicfacingdashboardtosharethisprogressmorebroadly
AgeneralcomparisonofthestatusquodatawithwhatwasobservedduringthepilotcanprovidedecisionmakerswithaquickunderstandingofthedifferencesbetweentheoldandnewapproachesalthoughthiswillnotshowcausalityOftenthesecomparisonswillincludeoperationalmetricssuchasdaysordollarssavedbutrememberthattheimpactonpeoplewillbemostimportantIfyouareworkingwithanevaluationpartnertheywillsupportthisstep
Usethespacebelowtosummarizetheresultsofyouranalysis
MeasureTargetvalue(seesectiononsettingtargetsabove)
Pilotvalue(whatdidyouobserveduringthepilot)
Historicalaverage(ifavailable)
Results(WhoisbetteroffIsanyonenegativelyimpacted1-2sentencesreassessingyourgoalstatement)
DocumentyourprocessandprogressincludingfeedbackfromtheparticipantsimplementationpartnersandtheprojectteamDidanythingunexpectedoccurduringthepilotArethereinternalandexternalfactorsthatmayaffectyouroutcomesThisiswheretheimplementationoutcomescanbecanhelpfulIftheinterventiondidnotgoasplannedconsiderwhetheritisbecausetheinterventionwasnotacceptableorfeasible
UsethetemplatebelowcreatedbyDenversPeakAcademytocapturethisinformationIfthe306090daytimelinedoesnotmakesenseforyourcityfeelfreetocustomizeit
CurrentState
FutureState
30Day
60Day
90day
Whatwentwell(+)
Whatdidnotgowellshouldbechanged(-)
IfaquestionordiscussionpointarisesthatcannotbeansweredinthemeetinganassignedstaffmembershouldcapturethequestionduringthemeetingAfterwardthefollowupitemsshouldbesenttorelevantpartnersandthenextmeetingshouldbeginwithadiscussionof
Establishaframeworktomonitorprogress
38
thoseitems
Takeaction
DiscussionsaboutwhetheryourindicatorsaremovingintherightdirectionandwhatmightbedrivingtrendsmayleadtoadiscoverythattherearenotenoughresourcesallocatedtothispolicyorissueTheremaybelimitedfundingforstaffprogrammingortechnologythatisdrivingthedatainthewrongdirectionIfCityleadershipisintheroomthisisagreattimetomakethecaseforadditionalfundingduringthebudgetseasonBycollectinganalyzinganddiscussingdataonthispolicyroutinelyyouarebuildingevidencetomakethecasefortherightleveloffundingtoreachtheestablishedtargets
Establishaframeworktomonitorprogress
39
Assessdataaccessibilityqualityandsecurity
Determineaccessibilityandqualityofdata
OnceyouhavecompiledyourwishlistofindicatorsitistimetodeterminetheaccessibilityandqualityofthedataTopreventcomplicationyoumaywishtoprioritizerelevantindicatorsforwhichyouarealreadycollectingdataAvoidthetemptationofadoptingatechnologysolutiontocollectnewdataatthispointsuchasaphotosurveytoolorotherappFornowyouaretryingtodeterminewhichdatasetsarerelevantaccessibleandhighqualityBesuretoworkwithyourkeyinternalpartnerswhileyouareidentifyingyourmeasuresDepartmentstaffwhoworkonprogramsarefamiliarwiththeindicatorsthatwillaccuratelysupporttheoutcomesanddatabaseadministratorswillbevaluablewhendiscussingaccessibilityandqualityofthedata
Abrieflistofconsiderationsfortheaccessibilityofyourdataisbelow
ArewecurrentlycollectingthisdatasetorwillwebecollectingitforthefirsttimeHowfarbackdoestherecordgo(timehorizon)WouldweneednewtechnologytocollectoranalyzethedataCanthedatabecollectedmanuallyorcanitbeautomatedHowfrequentlyisthedataproduced(continuouslyhourlydailyweeklymonthlyquarterlyyearlyetc)IsthedataavailablefortheentireCitypopulationorasubsetofthepopulationHowgranularisthedata(individuallevelCensusblockneighborhoodcitywideetc)DoesthedatacontainsensitiveinformationIsthereaproxymeasurethatwecanusefornowIsthedatamachinereadable
AsnotedbytheWilsonSheehanLabforEconomicOpportunities(LEO)attheUniversityofNotreDameleveragingadministrativedatacansignificantlyreducethecostsandtimelinesofrigorousevaluationCitiescanestablishaninternaldatasharingagreementacrossdepartmentsincludinglanguagethatenablesadministrativedatatobeusedforresearchpurposesCitiescanalsoidentifyexternaladministrativedata-holderssuchasacountyschooldistrictpolicedepartmenthospitaloranotherstatethatmayhaveoutcomedataimportantformeasuringprogramimpactCitiescanleverageexistingrelationshipsandorworkwithresearchinstitutionstodevelopstreamlineddatasharingprocessesandagreementswiththeseexternalentities
Assessdataaccessibilityqualityandsecurity
40
Evaluatethequalityofyourdata
QualityisasimportantasaccessibilityToevaluatedataqualityyoumaywishtocreatearubricliketheonebelow
Value ValueDescription
3 ExcellentDataQualityThedataiscompleteaccurateandupdatedregularly
2 AdequateDataQualityThedataissomewhatcompleteaccurateandupdatedfairlyregularly
1 PoorDataQualityThedataisnotcompleteaccurateorupdatedregularly
ThisisagreattimetobuildonthedatainventorythatyoustartedinthegatherinformationpageSuggestedadditionalfieldstoaddtoyourdatainventorytemplatearebelow
Program DatasetTitle
DatasetDescription DataSource
Departmentorexternalpartner
DoestheCityalreadycollectthisdata
Whatisthenameoftheprogram
Human-readablenameofthedataset
Shouldbeunderstandablebynon-technicalusers
Whatistheoriginalsystemapplicationfilethathousesit
Departmentorexternalpartnerthatmaintainsthedata
Isthisdataalreadycollected
Updatefrequency-ThefileformatforthedatasetTypicallycsvxlsxshapefiletxtetc
Format-Howoftenthedatasetisupdated
Timerange-Theperiodoftimecoveredinthisparticulardataset
Datause-Howisthisdatasetcommonlyused
Datausers-Whotypicallyusesthisdataset
Dataownerstitle-Titleofthedataowner
Dataownersposition-DataOwnersjobtitle
Dataownersemailaddress-DataOwnersemailaddressWhomanagesthedataandorisresponsibleforgrantingpermissiontoaccessthedataWhounderstandswhatthedatasetincludesandcananswerquestionsaboutit
Dataqualityconcerns-Describetheconcernsyoumayhaveaboutthequalityofthedataset
Assessdataaccessibilityqualityandsecurity
41
Division-DivisionwherethedataismaintainedIfmorethanonedivisionisresponsiblelisttheprimaryownerofthedataset
Sensitivedatacomments-DescribetheissuesthatwouldpreventthecityfromsharingthedatapubliclyCommonsensitivityconcernsincludeprivacyviolationssecurityissuesorhighcostorstaffdemandstopostthedata
Publishingstatus-Thistheplacetoputanycommentsregardingthecurrentstatusofthedataset
Publishedlink-Thisisthelinktothepublisheddatasetontheopendataportalorcitywebsite
Reviewdatasecurityandprivacyprotectionprotocols
OftenapolicychangealsoinvolvesachangeinthedatathatisbeingcollectedhowthatdataisbeinghandledwhohasaccesstoitandahostofotheradjustmentsAsyoutakestockofthedataassetsthatareavailableforyourinitiativeitisvitaltoalsothinkthroughwhatwillbedonetoprotectthatdataandtheprivacyofthepeopleitrepresentsAbigpartofgooddatasecurityinvolvesstrongdatagovernancepracticesthatdefineandidentifysensitivedataandconsciouslyworktobalancethevalueandriskofstoringsuchsensitivedataOnewayofevaluatingyourdatacomesfromcaselawrelatedtotheUSFreedomofInformationActwhichwehavetranslatedtothechecklistbelow
1 Woulddisclosureresultinasubstantialinvasionofprivacy2 Haveweconsideredtheextentorvalueofthepublicinterestandthepurposeorobject
oftheindividualsseekingdisclosure3 Istheinformationavailablefromothersources4 Wastheinformationgivenwithanexpectationofconfidentiality5 Isitpossibletomouldreliefsoastolimittheinvasionofindividualprivacy
TheseriskscouldbemitigatedusingavarietyofstrategiesfromaggregatingdatatoobfuscatingindividualstoredactingfieldsthatmayincludePersonallyIdentifiableInformation(PII)orre-evaluatingyourdatacollectionmethodstofindalternativewaysofgettingtheinformationyouneedThesestepsshouldbecodifiedandmanagedbyadatagovernancecommitteeifpossibleForthesensitivedatathatisdeemedessentialcreateaplanaroundpotentialdatabreachesinordertoquicklyandconfidentlyrespondtoemergencysituations
ProactivelypreparingforthesesituationsisoftenthebeststrategyfordatasecuritybutdonotforgetthetechnicalsideInvolveyourITadministratorsinconversationsaboutdatastorageandaccessensuringthattheywillbeprotectedbyenterprise-gradecybersecuritysystems
Assessdataaccessibilityqualityandsecurity
42
Assessdataaccessibilityqualityandsecurity
43
Designimplementationandevaluation
Designtheimplementation
NowthatyouhaveyouraccountabilityplanandkeyindicatorsitistimetogetspecificaboutyourplanforimplementationThetemplatebelowwillhelpyouthinkthroughthespecificmechanismsofimplementingpolicychange
UnderstandingorganizationsandsystemsiscriticaltounderstandingandmakingdecisionsaboutwhereandhowinterventionsshouldbeappliedApolicyimplementedinasinglesettingmayshowpromisingresultsbutimplementingmultiplepoliciesandprogramsatthesametimewithdifferentactorsresponsibleforeachaspectcanhelpensuresustainablebehaviorchange
InSomervilleMAShapeUpSomerville(SUS)isacitywidepublichealthprogramthatisimplementedacrossmultiplelevelsofinfluenceandsystemsthroughinterventionsbeforeduringandafterschooltoincludechangestobuiltenvironmentsuchasbikelanesregionalmasstransportationandphysicaleducationequipmentinschoolsandgymstoincreasephysicalactivitywellnesspoliciesandschoolgardensnutritioneducationduringandafterschoolandhealthyrestaurantprogramstosupporthealthyfoodandhealthyeatinghabits
Usethetemplatebelowtoplanyourimplementation
Policyimplementationplanningtemplate
ListthespecificdesiredoutcomesInthepreviousstepyouworkedwithdepartmentstafftoestablishspecificmeasurableoutcomesandinterimindicatorsPrioritizetheindicatorsthatwilldemonstrateprogressUsethespacebelowtocapturethedesiredoutcomesofthepolicychange
ExampleShapeupSomervillebeganwiththegoalofpreventingobesityin1stthrough3rdgraders
Yourturn
____________________________________________________________________________
____________________________________________________________________________
Designimplementationandevaluation
44
SpecifythetargetpopulationAreyoudesigninganinterventionthattargetsonlycertainsitesAreyouplanningonimplementingaprogramorinitiativeforaspecifictargetaudienceClearlyspecifythepopulationyouplantotarget
IfyouplantoconductarigorousimpactevaluationasnotedbytheWilsonSheehanLabforEconomicOpportunitiesitisimportanttoestablishconcreteobjectiveeligibilitycriteriaandensurethatthesecriteriaareappliedconsistentlybyallstaffduringscreeningandintakeItisalsovitaltoidentifytheindividualswhoareeligibleforaparticularprogramorserviceevenifeveryoneeligiblecannotbeservedbyexistingresourcesThishelpscitiesidentifyanyunmetneedsandguidefutureresourceallocationwhilealsocreatingthepossibilityofarigorousimpactevaluationbyidentifyingapotentialcomparisongroupFinallywhenconductingscreeningandintakeitisimperativetocollectandmaintainbasicinformation(egnamedateofbirthgender)aboutallprogram-eligibleindividualsnotjustthosewhoparticipateintheprogram
ExampleIntheShapeUpSomervilleexampletheprogrambeganbytargeting1stthrough3rdgraders
Yourturn
____________________________________________________________________________
____________________________________________________________________________
DefinethelocationWherewillthepolicychangetakeplaceWillimplementationtakeplaceatonesiteormultiplesites
ExampleShapeUpSomervilleinterventionsweredeliveredinschools
Yourturn
____________________________________________________________________________
____________________________________________________________________________
ClarifyimplementationpartnersandarticulatetheirrolesWhatarethekeyorganizationsinstitutionsorpeopleresponsibleforimplementationTheprojectteamshouldbeasclearandspecificaspossibleinprovidingadescriptionoftheconcreteactivitiesthatimplementationpartnersaremeanttocarryoutPracticeprofilesmaybeahelpfultooltohelpdefinewhatisrequiredoftheactorswhoareresponsibleforimplementingtheintervention
Designimplementationandevaluation
45
ExampleInSomervillemayoralleadershipandengagementofmultiplecityagencies(publichealthprimarycareschoolsCambridgeHealthAllianceetc)streamlinedandconnectedthevariousShapeUpSomervilleinitiativesandgrantswhichhelpedtokeepthetalentandenergyfocusedonhealthyeatingandactivelivingforallsectorsofSomervilleSUSpartnersultimatelyincludedschoolsnonprofitsCityandstateagenciesandnationalfoundations
Yourturn
____________________________________________________________________________
____________________________________________________________________________
DescribetheinterventionWhatarethestepsactivitiesorprocessesRememberthatthemechanismthespecificstepsorprocessthatleadtobehaviorchangeiskeytounderstandingtheeffectivenessofthepolicychangeandsharingtheresultswithothers
ExampleGoalsofShapeUpSomervilleweretoinfluenceeverypartofanearlyelementaryschoolchildrsquosdayndashbeforeduringandafter-schooltopromotehealthyeatingphysicalactivityandtheirweightstatusInterventionincludedfoodserviceenhancementstoimprovebreakfastandlunchandstaffprofessionaldevelopmentwalktoschoolactivitiesSUSclassroomcurriculumandprofessionaldevelopmentschoolwellnesspolicydevelopmentSUSafter-schoolcurriculumandprofessionaldevelopmentoutreachandeducationtohomethroughmaterialsforumseventsldquoSUSapprovedrdquorestaurantscommunityoutreachandcapacitybuildingthroughpolicydevelopmenttrainingsmediaplacements
Yourturn
____________________________________________________________________________
____________________________________________________________________________
ClarifytimelineandmilestonesfordeliveryandevaluationHowfrequentlywilltheservicesinterventionbedeliveredHowlongwilltheprogramrunWhatisthetimetoexpectedimpacts(eg12monthsafterprogramcompletion)
Designimplementationandevaluation
46
ExampleShapeUpSomervilleledbyTuftsUniversityFriedmanSchoolofNutritionScienceandPolicyconductedaquasi-experimental(nonrandomized)interventionstudyamong1stthrough3rdgradestudentsinSomerville(intervention)andtwocomparisoncommunitiesfor9-monthsduringoneschoolyear(fall2002ndashspring2003)ThestudyshowedpositiveassociationoftheinterventiononchangeinBMISUSreleasedreportssummarizingevaluationsprogresstodateplansforadditionalprojectsandimpactstenandfifteenyearsintotheinitiative
Yourturn
____________________________________________________________________________
____________________________________________________________________________
PlanyouranalysisTheprevioussectionaddressedestablishinganaccountabilityframeworktomonitorprogressandimplementationDeterminewhichindicatorswillbediscussthroughthatprocesswhowillsharethedataandanalysisandhowfeedbackfromthoseconversationswillbeincorporatedbytheprojectteam
ExampleTheMayorrsquosofficeimplementedadata-drivendecisionmakingapproachcalledSomerStatwhichwassynergisticwiththeapproachestakenbytheresearchandevaluationteamsdeployedforthespecificgrants
Yourturn
____________________________________________________________________________
____________________________________________________________________________
PlanforsustainabilityIfthepolicyandprogramchangeissuccessfulhowwilltheprojectteamsustainandexpandtoadditionalsitesorpopulations
ExampleAlthoughmostofthepartnershipsforgedintheearlyyearsweremaintainednewgrantsandcollaborationsexpandedattentiontoolderpublicschoolstudentsactivetransportationandtheGreenLinesubwayextensionthebuiltenvironmentfarmerrsquosmarketeffortssuchasldquoGrowninSomervillerdquowithUnionMainStreeteconomicdevelopmentandhealthdisparitiesaswellasdatacollectionmonitoringandsurveillance
Yourturn
Designimplementationandevaluation
47
____________________________________________________________________________
____________________________________________________________________________
Pilotplanningtemplate
IfyouplantoconductapilotfirstdeterminethepurposeofthepilotDoyouwanttoobservewhetherthepolicychangeresultsinthedesiredbehaviorchangeAreyouconcernedaboutfeasibilityscalabilityorcostDoyouwanttotestwhetheryourdesiredresultscanbeachievedthroughthispolicyapproachClarifythegoalofthepilotandthequestionsyouhopetoanswerThiswillhelpyouidentifythekeyindicatorsthatyouwillusetomeasurewhetherthepilotissuccessfulBeforebeginningestablishabaselineofcurrentperformanceandsetyourtargetsaccordinglybasedonwhatyouhopetoaccomplishthroughthepilot
NextestablishthepilotplanforyourprojectteamincludingmilestonesforteamcheckinstodeterminewhethertomoveaheadorchangecourseThisisseparatefromthemeetingsthatwilloccuraspartoftheaccountabilityframeworkUsethespacebelowtocreateyourtimelineanddevelopyourmeetingschedule
Task Personresponsible Startdate Checkindate Deliverable
Planyourevaluation
WhileyouaredesigningtheimplementationitisalsotimetobegintodevelopyourplanforanalysisAsnotedbytheWilsonSheehanLabforEconomicOpportunitiesdecisionsaboutprogramsetup-includingoutreachscreeningintakedatatrackingandenrollment-candeterminewhetherornotarigorousimpactevaluationwillbepossibleAsdiscussedcityleadersmayfindithelpfultoincludeevaluationpartnersinearlyconversationsaboutprogramsetupDuringpreviousstagesyoumayhaveworkedwitharesearchertoplanyourproposedpolicychangeYoushouldbeclearaboutthepurposeoftheevaluationanddevelopanevaluationplanBelowareadditionalconsiderations
Causationvstrendanalysis
IntheEstablishaframeworktomonitorprogressstagetherewasguidanceontrackingtrendsovertimeItisimportanttonoteherethatmonitoringdataisdifferentfromanalyzingacausaleffectTrendanalysiswillillustratethatchangesareoccurringandmayleadtoquestionsthatcouldbeansweredbyyourevaluationbuttrendanalysisalonedoesnotprovideevidenceaboutwhatisdrivingthosechangesYoumayseechangesinyour
Designimplementationandevaluation
48
outcomeindicatorsbasedonvariablesoutsideoftheCityscontrolForinstanceachangeinfederaltradepolicycanhaveanimpactemploymentratesinyourCityoutsideofyourCitysnewjobtrainingandplacementprogram
ToaddresscausalityyoumustcomparetheoutcomesofindividualswhoreceivedtheinterventionwiththosewhodidnotWithoutacomparisongroupofsimilarindividualswhowerenotexposedtotheinterventionitisimpossibletoknowwhichfactorsareresponsiblefortheoutcomesRandomizedControlTrials(RCTs)inwhichindividualsarerandomlyassignedtotreatmentandcontrolgroupsarethegoldstandardinprovingcausalityIfyoucandemonstratethatoutcomesimprovedforthosewhoreceivedtheinterventionanddidnotimproveatallorimprovedlessforacomparisongroupyoucanbesomewhatcertainthattheinterventionwasthecause
Examplesofothernon-causalimpactevaluationsthatarerigorousandreliableincludecertainquasi-experimentalstudiesusingcomparisongroupdesignssmartinterruptedtimeseriesanalysesthatmodeltrendsovertimeinwaysthatallowestimationofinterventioneffectswelldesignedpre-posttestanalysesneedsassessmentsimplementationstudiesetcFormoreinformationondifferenttypesofevaluationsseetheSageResearchMethodswebsite
Lookfordifferencesingroups
BuildintimetoalsoexaminethepilotdatatoseeifgroupsareexperiencingdifferentresultsReviewingperformancebasedonoutcomemeasuresforapopulationinaggregatemayhidethesubtlewaysthatsomepeoplemaybeaffecteddifferentlysoitisnecessarytocarefullyanalyzethedatafordisparateoutcomesForinstancetheMovingtoOpportunity(MTO)studythatexaminedtheimpactofhousingvouchersonfamilieswholivedinhigh-povertyneighborhoodsfoundthatchildrenwhomovedbeforetheageof13hadbettereducationandhigherincomesasadultsHoweveradultsdidnotseeimprovededucationandemploymentoutcomes
Disaggregatingdataintomorespecificcategoriesofraceandethnicitycanhelpunderstandtrendsanddisparitiesinamoregranularwaywhichcanhelpdecisionmakerstargetpoliciesandinterventionsinamoreequitableway
EarlierintheRoadmapweworkedtoidentifystakeholdersandwhomightbepositivelyandnegativelyimpactedbyachangeinpolicyUsetheresultsofthatexercisetoinvolvethecommunityininterpretingdataanalysisandtogatherqualitativefeedbackfromthecommunitytocallattentiontoissuesForinstanceanincreaseinyouthdetentionmaybeexplainedbyinefficienciesorlackofaccesstotransportation
Designimplementationandevaluation
49
Designimplementationandevaluation
50
Shareyourstoryandsuccess
Sharetheresults
Storytelling
AnintegralstepinyourstrategyissharingtheresultsofyourpilotorpolicychangewithyourcommunityThisincludesdevelopingmessagingaroundtheworkofyourorganizationandcreatingaplantodisseminateyourstoriesWhytellstoriesaboutyourwork
Informandorunderstandasubjectorsituation
Giveneededcontextofdataorbitsofinformation
Relatablestoriesarememorableforreadersviewers
ReadersorviewerswillseetheinformationasmorecompellingwhenthereisademonstratedandrelatablechangeorimpactonthemselvesortheirfriendsneighborsandcolleaguesStoriesareaboutpeopleandgovernmentisfundamentallyabouthelpingpeopleSotellthestoriesofthepeopleyouandyourcitycolleagueshelped
Keepthesetipsinmindasyoudevelopyourstorywithyourteam
WhatwereyourfindingsandwhyshouldanyonecareTellyouraudiencehowtheproblemwassolvedWhathastheimpactbeenonresidentsandcitystafftodateWhatwasthesuccessandweretheremultiplesuccessesalongthewayIfyouhavepreliminaryresultsordatashowingimprovementtosharethisisthetimetodoit
WhowasinvolvedandimpactedReviewthelistofstakeholdersHowwereeachofthemimpactedbytheprogramHumanizeyourstoryandimpactbyaddingadditionalvoicesThesevoicescanbeoneofthefollowing
Apersonorfamilywhowaspositivelyaffectedbythechangesuccess
Aninvolvedthird-partydescribingboththeproblemandthesolution(ieanorganizationnonprofitprovidersbusinesstradeassociationsportsteametc)whoispartofthecommunitythatthegovernmentworkedwithonthechangesuccess
Someonefromthegovernmentitselfwhocanspeakaboutthechangesuccess
TellmoreaboutyourcityDescribeyourcityincludingpopulationdemographicsindustriesetcWhatareitschallengesWhatisitknownforWhatmakesituniqueHowisitsimilarordifferentfromothercitiesintheUS
Shareyourstoryandsuccess
51
DescribeyourcityrsquoschallengeWriteoutthechallengeyourcityisfacing(asdefinedinyourproblemstatement)HowdoesitaffectresidentsHowdoesthisproblemaffecttheworkofcitystaffmembers
AddthedetailsProvidedetailsonyourappliedresearchstrategyTalkaboutwhatyourcityhasdonenthepasttoresolvetheissueDiscusshowyourstrategyhaschangedwhenyoubegantoadoptthisapproachWasmoneysavedorinvested
Usethisstorymapexercisetoorganizeyourthoughtsandguaranteethatyouincludealloftheelementsofatraditionalstory
AfteryoudevelopyourmessagingcreateacommunicationsmarketingandoutreachstrategydesignedtospreadyourmessageYourstrategyshouldwalkyouthroughthefollowingkeyactivities
1 IdentifyandprofileyouraudienceWhencreatinganddisseminatingyourmessageitisalwaysgoodtothinkaboutyouraudienceWhoexactlywouldbeinterestedinwhatyouhavetosayThegeneralpopulationofthecityPeoplewhoworkinnonprofitsBusinessownersItmattersbecausenoteveryonewholivesinacitymaybethetargetoftheinformationBusinessownersmaybemoreinterestedindatashowingeffectsonthebusinessdistrictthanothersandsoonWewillcoverthislaterbutthisstepwillalsohelpyoutothinkofthemosteffectivewaytoreachyouraudience
2 Defineyourcalltoaction(ifyouhaveone)SpendsometimeconsideringwhyyouareinterestedintellingyourstoryDoyousimplywanttoupdatethepubliconyourprogressorcelebrateyoursuccessesAreyouinterestedinyouraudiencechangingitsbehaviororsupportingaspecificinitiativebeinglaunchedbyyourorganizationIfyouwanttohaveyouraudienceactorbehaveinacertainwaybespecificandincorporateitinyourmessaging
Shareyourstoryandsuccess
52
3 SelectperformancemeasuresThinkabouthowyouwoulddetermineifyourcampaignissuccessfulandselectappropriateperformancemeasuresYouwillrevisitthesemeasuresattheendofyourcampaigncommunicationsplanimplementationtodeterminewhetheryouneedtomodifyyourcommunicationsandoutreachstrategyforthefuture
4 MethodstocommunicateyourstoryTherearemanywaystocommunicatesuccessofyourprojectHerearesomesuggestionsonreachingaudiencesItisimportanttodeterminewhichmethodsarethemosteffectiveforyourdesiredaudiencetoreceiveinformationDoyourresearch(PewResearchCenterhasexcellentstudiesonthereachofmediadigitalmediaandmarketing)ForexampleifseniorsareyourtargetpopulationandtheyarelessinclinedtouseInstagram(thatistrue)youmaywanttoconsiderFacebookorthenightlylocalnewsbothofwhichareusedmoreoftenbyseniorsBelowisalistofsomemethodsandtoolsyoucanuse
i Pressreleasestolocalnewspapersandtvstations
ii ShortsmartphonevideosAgreatwaytoengageanaudienceisthroughvideoAskonequestionofsomeonewhoispartofthestoryMaybethereisasocialworkerwhocandiscusshungerinyourcityThesevideoscanbereusedonmultipleplatforms
iii Long-formwriting
iv TellingindividualstoriesthroughaseriesofphotosandshortnarrativesforpublicationonFacebookorInstagram
v Op-edsopinionpiecesplacedinthelocalpaper
vi Emailupdatestoyourlist
vii Connectingwithappropriateorganizationsthatareinvestedinthetopicandworkwiththecommunityandaskthemtodisseminateinformationtotheirmembers
viii EmployingldquoWordofMouthrdquo(WOM)tacticsbycreatingtownhallsorfreeeventsforresidentstoattendintheirownneighborhoods
5 DeterminethetypeofcollateralUsuallycampaignsthattargetresidentsrequireseveralavenuestoreachthemPeoplearenotalwaysinthesameplaceorspaceHavingakeenunderstandingofyouraudiencewillassistyouwithyourchoicesonwhatneedstobedevelopedSometimesthatmeansimagesforsocialmediaaonepageflierthatgivespeopleinformationonattendinganeventoranin-depthbrochure
Shareyourstoryandsuccess
53
6 ImplementyourstrategyKeepinmindstrategiescanbefluidSometimessomethingmaynotturnoutasexpectedevenifduringplanningsessionsitwasldquosupposedtordquoItisokYoucananalyzewhathappenedduringadebriefsessionandonceyoursurveysandothermeasurementshavebeenfinalized
Visualizations
Withtherightvisualizationyoucanreviewandshareinformationinsightsanalysisperformanceandtrendsallofwhichisvaluabletoacross-sectionofthepublicButwhichvisualizationisrightforyouTochoosetherighttypeofchartgraphdeterminethetypeofquestionyouwanttoanswerSomecommoncharttypesarebelowandmoreinformationcanbefoundhere
Exploreadditionalanalyticaltechniques
Duringthegoal-settingphaseyouestablishedoutputoutcomeandimplementationmeasuresBasedontheresultsofyourpilotyoumayneedtoadjustyourmeasuresWiththepilotdatainhandyoumayhavesomeideasofhowfutureanalysescouldbedifferentandpotentiallyimprovethequalityofyourprogramTakesometimetobrainstormotherrelevantdatasetsthatyoudidnothavethetimetopursueearlierincludingmessydepartmentaldatathatneedsmoreattentionorsourcesofpublicdatafromthestateandfederalgovernmentnonprofitsandevenprivatecompaniesthatcouldsupplementyourunderstandingofanissueUsingmachinelearningtechniquesyoumightbeabletobetter
Shareyourstoryandsuccess
54
predictdemandforhomelessservicesusingNOAAweatherforecastspreemptwatermainbreaksusingsoilstratigraphyorinvestineconomicdevelopmentbymonitoringrealestatelistingdata
Advancedanalyticaltechniquescanalsohelpyourimplementationrunmoreeffectivelybyintegratingdata-informedtargetinganddecisiontoolsintotheprocessByautomatingroutineworkflowsandsurfacingsubtletrendsthesetoolscanhelpyouiterateinamoreprecisemannerandprovidekeystrategicguidancetofront-lineemployeesthatwouldotherwisebeunavailableThesesortsofdata-science-enhancedprocessescanbeclassifiedasoneofsixprojecttypesasexploredfurtherbytheNewOrleansLANOLAlyticsteamFormoreguidanceonscopingasimilarprojecttheGovExadvancedanalyticsscopingguidegivesyouastep-by-stepoverviewofhowtobesuccessful
Usethespacebelowtolistyourideasforpossibleadvancedanalyticsprojectsbytype
Typeofproject Ideas
Findingtheneedleinthehaystack
Prioritizingforimpact
Earlywarningtools
Betterquickerdecisions
Optimizingresourceallocation
Experimentingforwhatworks
Keepthefeedbackloopgoing
GatherinformationonpublicsentimentaboutpolicychangethroughfocusgroupssurveysandpublicmeetingsledbyadvisorygroupmembersNotewhethernewinformationisavailablethatcanhelpanalyzeorcontextualizeyourprogress(egfederalorstatedatareports)Sharetheresultsofyourpilotandthepolicychangeprocesswithothercitiesandresearchers
Shareyourstoryandsuccess
55
WorkingwitharangeofevidencePresentlythereisalotofattentionandinvestmentinevidence-basedpolicymakingincludingthefederalbipartisanCommissiononEvidence-BasedPolicymakingthePewCharitableTrustsreportsandtoolkitsonHowStatesEngageinEvidence-BasedPolicymakingandtheLauraandJohnArnoldFoundationsupportedEvidence-BasedPolicymakingCollaborativeManyoftheseinitiativesfocusonrigorousevaluation--meaningrandomizedcontrolledtrials(RCT)orotherexperimentalmodels--asthebestkindofevidencetouseHoweverthisbodyofevidenceisofteninsufficientforthespecifickindsofquestionsthatgovernmentscanhavewhenmakingadecisionaboutpolicyInthisguideweproposeanapproachtosupportgovernmentsinincorporatingadditionaltypesofevidenceintotheirdecisionmaking
OnewaytounderstandarangeofevidenceistouseNestarsquosStandardsofEvidencetoframeanddefineevidenceTheBritishnonprofitNestaworkstoinjectpositivechangeintothewayUKgovernmentpoliciesandprogramsfunctionOneoftheirkeyareasofworkhasbeendevelopingWhatWorksCentresorganizationsthatcollectcommissionsynthesizeanddisseminateevidenceaboutinterventioneffectivenessacrossarangeofsocialimpactareasNestadevelopedthefollowingStandardsofEvidencetoprovideausefulwaytodocumenthowdifferentkindsofinformationdemonstratedifferentlevelsofevidence
Workingwitharangeofevidence
56
Theimportanceofpotentiallybiasedinformation
Onechallengepresentedbyanevidenceclassificationthatacknowledgesthevalueofnon-scientificformsofevidenceistofigureouthowtobalanceallofthepiecestogetherwhenyouusethemTounderstandhowtodothiswehavetothinkabouthowtopositionpotentiallybiasedinformationindecisionmaking
WhenwelookatinformationwhichhasnotbeenrigorouslyevaluatedweacceptthatthisinformationmightbebiasedThewholereasonweconductrigorousevaluationistoremovebiasfromourobservationsWeconductstructuredtrialstoproduceresultsthatareconsistentandgeneralizableIfafindingisunbiasedthisaccuracymdashthislackofbiasmdashcanbeprovedbecauseanyindependenttestercanreproduceitwhentheyfollowanidenticalprocess
YoucanlookatpotentiallybiasedinformationasjustinformationthatisearlieroninitspathtorefinementreplicationandverifiedaccuracyApieceofinformationbasedonanyoneindividualrsquosobservationsislikelytobebiasedbecauseitiscomingfromjustonesourceBiasisreducedbygatheringtogetherlotsofindependentsourcesofinformationandfiguringoutitsmostcommonattributes
MaybethebiasedinformationmakesaclaimthatistoolargemdashitsayssomethingistrueunderallconditionswhenitisreallyonlytrueundercertainconditionsorthataneffectofaninterventionisbiggerthanitstrueaverageorthataneffectlastsovertimewhenitdoesnotPerhapschangingtheclaimtobemorespecificmdashcriticallyreviewingtheinformationtoseeifspecificconditionsmustbemettoachievetheoutcomesmdashwouldmaketheclaimmoreaccurate
IfyoutestedyourclaimfounditdidnotholdupinsomecaseschangedittesteditagainuntilyourclaimrepeatedlyheldupthenyoucoulddemonstratethattheclaimwasnolongerbiasedbutaccurateThisprocessoftestingimprovingtestingandvalidatingiswhatwegetfromstructuredscientificstudy
Withindividualpiecesofinformation--crediblehypothesescasestudiesexpertopinion--youcannotassumethatkindofaccuracyAtthesametimetheseearlyobservationsarefoundationaltoanyreplicablescientificobservationsWewillnevergetanywhereifwedonotstartwiththesepiecesofinformationeveniftheymightbebiased
Combininglevelsofevidencetogetthemostdetailedinformationforyourcity
ItisespeciallyimportanttoacceptdifferentformsofevidencewhenyoufacequestionsthathavenotyetbeentestedbyindependentstudiesSomeofthemostimportantquestionsrelatetothingsthatareveryspecifictoyoursituationYoufaceaparticularcombinationof
Workingwitharangeofevidence
57
issuesactorstrendsandresourcesthatisultimatelydifferentfromeveryotherplaceintheworld
ThisisimportantbecauseaspolicypractitionersandnotscientistsyoucaremoreaboutyourspecificcasethanahypotheticalaveragegeneralcaseYoucarelessaboutthetotallikelyrangeofimpactsfromapolicythanhowthissingleimplementationwillworkYouwantinformationaboutpoliciesthatwillprovideyouwiththebestpossiblepredictionabouthowtheywillworkinyourcontext
ByputtingdifferentkindsofevidenceintothesameframeworkNestarsquosStandardsofEvidenceimplythatwhenyoucannotgetrigorouslytestedexperimentallyverifiedinformationitisappropriatetoworkwithwhatwedohaveThisisrelevantforthemunicipalimplementationcontextbecausenostudypulledfromaliteraturereviewwillaccountforallofthevariablesyoufaceinyourowncontext
Intheabsenceofhavingastudythatpredictshowaninterventionworksinyourspecificcontextyouwillneednon-scientificinformationtotrytofillinthosegapsYouwillneedtogatherstoriesfromresidentsstaffmembersandotherstakeholdersthatsitonLevels1or2ofthestandardsofevidence
Someofthekindsofinformationyouwillprobablygetmoreeasilyfrompeoplethanstudiesinclude
howpeopleincitieslikeyoursreactedtoaninterventioninpublicforumshowpeoplewhoimplementedaninterventioninacitylikeyoursfiguredouthowtogetitpassedhowthecitycommunicatedpubliclyabouttheapparentbenefitsanddrawbacksoftheinterventionthepoliticalacceptabilityoftheintervention
Youcanremainsimultaneouslyawarethattheinformationyouaregatheringthroughindividualstoriesisbiased--notgeneralizableinformedbyspecificperspectives--whilealsorecognizingthathavingsomeinformationaboutsomethingimportantisbetterthannoinformationEvidencefromthefirsttwoevidencelevelsareessentialbuildingblocksofknowledgeandperfectlyusefulwhenyoucannotgetdetailedunbiasedknowledge
Youhavetostartsomewhereandyoushouldalwaysstartwithevidenceyouhaveaboutyourveryspecificconditionsthatyouknowareimportantforsuccess
Workingwitharangeofevidence
58
Thankyou
Interviewswithcities
GovExheldsemi-structuredinterviewswithsixstaffmembersinfourcitiestoestablishtheinitialstepsoftheRoadmapWeaskedrespondentstotalkaboutaninitiativeorpolicythattheyhavebeenresponsibleforimplementingintheircityRespondentsfromthreecitiesspokeabouttheireffortstoaddresshomelessnessandrespondentsfromonecityspokeaboutblightAfterrespondentsidentifiedtheproblemtheyspokeabouthowthecityprioritizedthisissuethevariousstepsinvolvedandbarrierspresent
GovExconductedtextanalysisoftheseinterviewswhichrevealedthatpeoplewasthemostcommonlyusedtermduringtheseconversationswhendiscussingthestepstochangeaprogramorpolicyThisisathemethroughoutthisRoadmapThefirstsectionbeginswithidentifyingtherelevantpeopleTheconversationsaroundevidencearefocusedonhumanbehaviorratherthanmethodologyorstrengthofevidence
Thankyou
59
OncewehadadraftoftheRoadmapwefollowedupwithcitiesonanindividualbasistolearnmoreaboutwhatwouldbehelpfulWearegratefultothefollowingcitystafffortheinterviewsandindividualconversations
MichaelBaskinChiefPolicyOfficerChattanoogaTN
MelissaBridgesPerformanceandInnovationCoordinatorLittleRockAR
VictoriaCarreoacutenAdministrativeOfficerLasVegasNV
TyroneGrandstrandCommunityDevelopmentPlannerFargoND
TibLaughlinDirectorofGeneralServicesKansasCityKS
TimMorelandDirectorofPerformanceManagementandOpenDataChattanoogaTN
CharlesRosendahlManagementAnalystDentonTX
Thankyou
60
MelissaSiebenAssistantCountyAdministratorKansasCityKS
GinaThompsonDirectorofStrategicServicesLewisvilleTX
JHUFacultycollaboration
GovExalsoconvenedthecitycontactsandfacultypartnersinaseriesofAdvisoryGroupmeetingsThesemeetingsservedtoconnectcitiesandfacultytounderstandoneanothersperspective
StefanBaralBloombergSchoolofPublicHealth
AshleyBernerJHUSchoolofEducation
MelissaDavey-RothwellBloombergSchoolofPublicHealth
JonLinksBloombergSchoolofPublicHealth
TamarMendelsonBloombergSchoolofPublicHealth
KristinMmariBloombergSchoolofPublicHealth
KeshiaPollackPorterBloombergSchoolofPublicHealth
JoshSharfsteinBloombergSchoolofPublicHealth
LizStuartBloombergSchoolofPublicHealth
LindsayThompsonCareyBusinessSchool
DanielWebsterBloombergSchoolofPublicHealth
Partnersindataandevidence-basedpolicy
Wealsospokewithpartnerswhoareexpertsinthefieldofevidence-basedpolicywhosharedtheirknowledgeandresourcesofcurrentandfuturethinkinginevidence-informedthinking
WendyBarrenoAssociateDirectorWilsonSheehanLabforEconomicOpportunitiesUniversityofNotreDame
TiffanyDavenportDirectorofDataPartnershipsBaltimoresPromise
TomiHiersDirectorBaltimoreCivicSiteAnnieECaseyFoundation
ChrisKingsleySeniorAssociateAnnieECaseyFoundation
KathrynOliverAssociateProfessorLondonSchoolofHygieneandTropicalMedicine
Thankyou
61
KristenWatkinsSeniorPolicyAssociateJ-PAL
GovExParticipants
GovExAnalystMiriamMcKinneyanalyzedtheinterviewtranscriptsandconductedtextanalysisSeniorImplementationAdvisorEmilyShawwillintegrateRoadmapactivitieswithadditionalservicedeliveryactivitiesatGovEx
Thankyou
62
ReferencesAbrahamKGampHaskinsR(2017)Thepromiseofevidence-basedpolicymakingReportofthecommissiononevidence-basedpolicymakingWashingtonDCCommissiononEvidence-basedPolicymakingBigmove2Extendingthegreen(nd)Retrievedfromhttpswwwcityoflewisvillecomdoing-businesslewisville-2025big-move-2-extending-the-greenBatesMAampGlennersterR(2017)ThegeneralizabilitypuzzleRetrievedfromhttpsssirorgarticlesentrythe_generalizability_puzzleBuildwithnotfor(2018January)RetrievedfromhttpsbuildwithnotforwordpresscomCairneyPampOliverK(2017)Evidence-basedpolicymakingisnotlikeevidence-basedmedicinesohowfarshouldyougotobridgethedividebetweenevidenceandpolicyHealthResearchPolicyandSystemsdoi101186s12961-017-0192-xCairneyPOliverKampWellsteadA(2016)TobridgethedividebetweenevidenceandpolicyReduceambiguityasmuchasuncertaintyPublicAdministrationReview76(3)399-402Retrievedfromhttpdxdoiorg101111puar12555Centreforpublicimpact(nd)RetrievedfromhttpswwwcentreforpublicimpactorgCernaL(2013)ThenatureofpolicychangeandimplementationAreviewofdifferenttheoreticalapproachesOrganisationforEconomicCo-OperationandDevelopmentRetrievedfromhttpswwwoecdorgeduceriTheNatureofPolicyChangeandImplementationpdfColoradoSprings(nd)Retrievedfromhttpscoloradospringsgovcommunicationspagewhat-expect-when-you-speakupContinuumofcare(coc)program(nd)RetrievedfromhttpswwwhudexchangeinfoprogramscocCouncilpresidentkicksoff$12millionyouthfund(nd)Retrievedfromhttpwwwbaltimorecitycouncilcomcontentcouncil-president-kicks-12-million-youth-fundDurlakJAampDuPreEP(2008)ImplementationmattersAreviewofresearchontheinfluenceofimplementationonprogramoutcomesandthefactorsaffectingimplementationAmericanJournalofCommunityPsychology41327-350Retrievedfromhttpswwwncbinlmnihgovpubmed18322790Fishbonediagram(ishikawa)(nd)Retrievedfromhttpasqorglearn-about-qualitycause-analysis-toolsoverviewfishbonehtmlFivewhysGettingtotherootofaproblemquickly(nd)RetrievedfromhttpswwwmindtoolscompagesarticlenewTMC_5Whtm
References
63
FligsteinNampMcAdamD(2011)TowardageneraltheoryofstrategicactionfieldsSociologicalTheory29(1)1-26RetrievedfromhttpsociologyberkeleyedusitesdefaultfilesfacultyfligsteinFligstein20McAdam20Soc20Theory20PaperpdHeinkelP(2018January12)CitymullspermitsforbikesharingcompaniesRetrievedfromhttpwwwdentonrccomnewsdenton20180112city-mulls-permits-bike-sharing-companiesHowisareamedianincomecalculated(2012October)Retrievedfromhttpswwwhudexchangeinfofaqs843how-is-area-median-income-calculatedLampkinLMampHatryHP(2003)KeystepsinoutcomemanagementUrbanInstituteRetrievedfromhttpswwwurbanorgresearchpublicationkey-steps-outcome-managementMarshDampSmithM(2000)UnderstandingpolicynetworksTowardsadialecticalapproachPoliticalStudies484-21Retrievedfromhttpjournalssagepubcomdoiabs1011111467-924800247McCannL(nd)RetrievedfromhttplaurenellencomMoultonSampSandfortJR(2016)ThestrategicactionfieldframeworkforpolicyimplementationresearchPolicyStudiesJournal45(1)Retrievedfromhttponlinelibrarywileycomstore101111psj12147assetpsj12147pdfv=1ampt=jdkhhcp4amps=e184c625e1855a6b1615ccb99ee27b1c0b3212d8PaynterB(2018January1)What100citiesarelearningfromeachotherbysharingtheirdataRetrievedfromhttpswwwfastcompanycom40518004what-100-cities-are-learning-from-each-other-by-sharing-their-dataPeakacademy(nd)Retrievedfromhttpswwwdenvergovorgcontentdenvergovenmayors-officeprograms-initiativespeak-performancepeak-academyhtmlPinnockHBarwickMCarpenterCREldridgeSGrandesGGriffithsCJRycroft-MaloneJMeissnerPMurrayEPatelASheikhAampTaylorSJC(2017)Standardsforreportingimplementationstudies(stari)ExplanationandelaborationdocumentBMJOpenRetrievedfromhttpbmjopenbmjcomcontentbmjopen74e013318fullpdfProctorESilmereHRaghavanRHovmandPAaronsGBungerAGriffeyRampHensleyM(2011)OutcomesforimplementationresearchConceptualdistinctionsmeasurementchallengesandresearchagendaAdministrationandPolicyinMentalHealth38(2)65-76doi101007s10488-010-0319-7TandonSD(2018)ProgressincommunityhealthpartnershipsResearcheducationandactionRetrievedfromhttpswwwpressjhuedujournalsprogress-community-health-partnerships-research-education-and-actionProvanKGampKenisP(2007)ModesofnetworkgovernanceStructuremanagementandeffectivenessJournalofPublicAdministrationResearchandTheory
References
64
18(2)229-252Retrievedfromhttpsacademicoupcomjpartarticle182229935895SabatierPA(1988)Anadvocacycoalitionframeworkofpolicychangeandtheroleofpolicy-orientedlearningthereinPolicySciences21(3)129-168Retrievedfromhttpwwwjstororgstable4532139SandstromAampCarlssonL(2008)TheperformanceofpolicynetworksTherelationbetweennetworkstructureandnetworkperformancePolicyStudiesJournal36497-524Retrievedfromhttponlinelibrarywileycomdoi101111j1541-0072200800281xabstractSchwartzMampWilsonE(nd)WhocanaffordtoliveinahomeAlookatdatafromthe2006americancommunitysurveyRetrievedfromhttpswwwcensusgovhousingcensuspublicationswho-can-affordpdfShearerJCAbelsonAKouyateBLavisJNampWaltG(2016)WhydopolicieschangeInstitutionsinterestsideasandnetworksinthreecasesofpolicyreformHealthPolicyandPlanning31(9)1200-1211Retrievedfromhttpsdoiorg101093heapolczw052ThelessonsofadministrativedataHigh-profilepolicy-relevantresearchpoweredbyadministrativedata(nd)Retrievedfromhttpswwwpovertyactionlaborgsitesdefaultfilesresources20161114-The-Lessons-of-Administrative-DatapdfToolsandotherneedsforqualitativeandquantitativemethods(2015February)AgencyforHealthcareResearchandQualityRetrievedfromhttpwwwahrqgovprofessionalsquality-patient-safetytalkingqualityassesstoolshtml
WeaverRCampBagchi-SenS(2013)SpatialanalysisofurbandeclineThegeographyofblightAppliedGeography4061-70
References
65
UniversityPartners
Name Title Affiliation Area(s)ofexpertise
Contactinformation
Examplesofworkingwithnon-academicdecisionmakers
WendyPBarreno
AssociateDirector
WilsonSheehanLabforEconomicOpportunitiesUniversityofNotreDame
LEOsfocusareasincludehealtheducationhousingchildwelfareeconomicopportunitycriminaljustice
wbarrenonduduorleondedu
MelissaDavey-RothwellPhD
AssociateScientist
JohnsHopkinsBloombergSchoolofPublicHealth
Health mdavey1jhuedu410-614-5854
BaltimoreCityHealthDepartmentLeadersMarylandDepartmentofHealthAddictionRecoveryServicesinDCExecutiveDirectorsofcommunityorganizationsacrosstheUSthatfocusonHIVSTIsubstanceabuseandimplementationscience
ElizabethStuart
AssociateDeanforEducationandProfessorDepartmentsofMentalHealthBiostatisticsandHealthPolicyand
JohnsHopkinsBloombergSchoolofPublicHealth
Educationsocialpolicystatisticalmethods
estuartjhuedu
Resources
66
Management
Design-ThinkingResources
EngagementGuideforCivicTech
Journeymap
PerformanceManagementResources
DenverPeakAcademy
Proxymeasuresguide
Talkingaboutperformancewithoutcomplicatingit(video)
SettingPerformanceTargetsguide
Datavisualizationguide
Performancemanagementgettingstartedguide
Exercises
Fishboneexercise
5Whys
AdministrativeData
TheLessonsofAdministrativeData(JPAL)
EvaluationResources
Quantitativeandqualitativemethods(USDepartmentofHealthandHumanServicesAgencyforHealthcareResearchandQuality)
KeyStepsinOutcomeManagement(UrbanInstitute)
RCT-YES
Sageresearchmethods
BestPracticesforMixedMethodsResearchintheHealthSciences
ACCORDChecklist
Pilotandimplementationplanning
Benchmarkingexamplesandproxymeasures
Seattleopendataplaybook
Resources
67
Datasciencescopingguide
Resources
68
AboutGovExGovExhelpsgovernmentsusedataandevidencetoimproveresidentsrsquolivesWesupportgovernmentsinbuildingtheirinternalcapacitytooperatesustainableanalyticspracticeswhichimprovecommunityoutcomesinfieldsfrompublicsafetytoeconomicdevelopmentGovExhasaproventrackrecordofworkinginconcertwithpartnerstoprovidetechnicalassistanceandtrainingatscale
SinceourlaunchinApril2015GovExhas
Trainedover2000governmentemployeesinpersonandonlineintopicsincludingdatamanagementperformancemanagementopendatacommunityengagementandanalytics
Providedtechnicalassistancetomorethan100localgovernmentsacrosstheglobeimprovingtheiruseofdatatoachieveefficienciesinservicedeliveryandbetteroutcomesforresidents
Assessedthecapacityofmorethan130governmentalorganizationstodeterminehowleadersstaffandexternalstakeholdersmanageandusedata
Localgovernmentsareincreasinglyonthefrontlinesofaddressinglargescaleproblemswithglobalimplicationssuchasclimatechangepublicsafetyandeconomicinequality
Whileevidence-basedinterventionstohelpsolvethoseproblemsareavailablelocalgovernmentstaffhavelimitedtimeandcapacitytofindandimplementresearch-basedsolutionsAsaresultcitiescontinuetoworkthewaytheyalwayshave--makingincrementalimprovementstoprogramsbutnotalwayscreatingchangeatasystem-level
ThroughresourcedevelopmentandcollaborationwithJHUfacultyGovExissupportingcityleadersinusingdataandevidencetomakedecisionsandcreatinglastingrelationshipswithresearchers
TheRoadmapwasdevelopedbyGovExaspartofitspartnershipwithBloombergPhilanthropiesWhatWorksCitiesinitiative
AboutGovEx
69
- Title page
- Introduction and purpose of the Roadmap
- Using the Roadmap
-
- Gather information about the problem
- Strategically engage key partners
- Review the evidence and collect details
- Decide on a policy to implement based on local factors
- Establish a framework to monitor progress
- Assess data accessibility quality and security
- Design implementation and evaluation
- Share your story and success
-
- Working with a range of evidence
- Thank you
- References
- Resources
- About GovEx
-
thatevidencetotheirowncontextThroughourinterviewswithcitystaffweobservedthatframingtheproblemclearlyformingadvisorygroupsorsomeothercoalitionandidentifyingandimplementinganappropriatesolutionarenecessaryingredientstosuccessfulpolicychange
ThroughthisRoadmapGovExhopestofacilitateanewwaytohelpAmericancitiesrethinkthehowtheysolveproblemsandaddressthemostsalientpolicychallengesinurbangovernance
IntroductionandpurposeoftheRoadmap
5
UsingtheRoadmap
WhoshouldusethisRoadmap
ThisRoadmapisforanystaffmemberincitygovernmentwhoischargedwithsolvingaproblemorimplementingapolicyAlthoughthestepscanbeusedtochangeexistingprogramsorimplementnewprogramsthefocusoftheRoadmapisonlong-termsustainablepolicychangeWeusethetermpolicytomeanlawsregulationsfundingdecisionsandinterventions
CitystaffwhoarereadingthisguidemaybecomingfromdifferentstartingpointsItisimportanttobehonestaboutwherethatisForinstanceyoumaybelookingfornewsolutionstolongstandingseeminglyintractableproblemssuchasanannualincreaseingunviolenceormaybeyouarefacingnewerchallengesliketheopioidepidemicOrperhapsyouarelookingforadata-drivenwaytoaddressissuesrelatedtoday-to-dayservicedeliverysuchaspermittingorbikelanesThestepsareapplicablewhetheryouareworkingonalargescaleissueoralocalmunicipalissue
YoucanusethisguidewhetheryouareadecisionmakerorfrontlineworkerThroughouttheRoadmaptheresinformationaboutwhenandhowtoengageotherpeopleCollaborationisakeythemethroughthisRoadmapThegoodnewsisyoudonothavetodothisworkalone
TheapproachandthetoolscontainedintheRoadmapshouldberelevantwhetheryouareinastrongmayororcouncil-managerformofgovernmentElevatingthisprocesstothelevelofMayorCityManagerorCityCouncilmaytaketimeandmultiplestepsbutwehopeyouwillfindwaystoincorporatedataandevidenceintoyourday-to-daywork
WhatarethestepsoftheRoadmap
ThechaptersinthisRoadmaprepresentthehigh-levelstepsinvolvedinpolicychangeYouwillnoticethatthesectionsrefertooneanotherandbuildonthepreviousstepsTheRoadmapismeanttobeiterativeyouwillrevisitsomesteps
TherootsoftheproblemsthatcitiesareworkingtosolvearecomplexandmaybethelegacyofhistoricalfederalpolicyorsweepingsocialtrendsAlthoughthisRoadmapdoesnotexplicitlyincludestepsonreviewingthishistoryweencouragereaderstodevelopadeepunderstandingoftheprobleminyourowncitywhileconsideringcurrentstateandfederalcontext
UsingtheRoadmap
6
IninterviewswithGovExonthepolicychangeprocessintheircitiespeoplewasthewordmostcommonlyusedbyintervieweeswhendescribingthevariousstepstoimplementanewpolicyorprogramYouwillnoticethatthisisathemethroughoutthisRoadmapThefirstsectionbeginswithidentifyingtherelevantpeopleThechapteronidentifyingrolesforkeypartnersservesasafoundationforthefollowingstepsThediscussionofevidenceencouragesreaderstoconsiderwaysinwhichpeoplecanbuildanduseevidencetodesignstrongprogramsandinitiativeschangehumanbehaviorassessimpactensureeffectiveuseofpublicresourcesandimprovelives
Meaningfulapplicationofevidencebypolicymakersdoesnothappenwithone-timesharingofcuttingedgeevidenceRathermeaningfulinfluencerequiresalong-termstrategytoformalliancesanddevelopknowledgeofpolicymakersandpolicymakingInadditiontoworkingwiththerightpeopleandunderstandingevidenceandcontextwehopethatRoadmapuserswillformapeernetworkthatsharesnotjustwhichpolicysolutionsexistbutwhatimplementationlookslikeAsonecitywespoketoindevelopingthisRoadmapputitalotofourcitysproblemshavebeensolvedbyotherswejustneedtofigureouthowtoimplementthesolutions
TheexamplesthroughouttheRoadmapareillustrativeofgovernmentsthathaveeffectivelyimplementedtherelevantstepsinthisRoadmapbutpleasenotethatnocitydoesallofthisperfectlyTogetherthestepsinthisRoadmaparemeanttohelpcitiesframetheirproblemscreateademandforevidenceandbuildsustainablecoalitionsformeaningfulpolicychange
UsingtheRoadmap
7
GatherinformationabouttheproblemInyourearliestphaseitiscriticaltodevelopacomprehensiveunderstandingoftheproblemYouwillwanttolearnaboutthepeoplewhoareaffectedbytheproblemyouaretryingtosolveandthosewhocanimpacttheproblemFromthereyouwillgetinsightintotherootcauseoftheproblemtheworkthathasbeentriedinthepastandtheeffectivenessofthoseeffortsTheworkfromthischapterwillserveasthefoundationforactivitiesinsubsequentchaptersYoumayrevisitthesestepsasyouidentifyindicatorsandplanyourimplementationandanalysis
BesuretodedicatetimeandresourcestothisstepItcanbetemptingtoskipaheadtolookingforsolutionsortoassumethatyouhaveanunderstandingoftheproblemasaresultofapublicengagementprocess
TheCityofLewisvilleTXconductedaCitywidepublicengagementprocesstodevelopitsLewisville2025StrategicPlanOneoftheBigMovesoutlinedintheplanwasExtendingtheGreenorexpandinggreenspaceandconnectingrecreationareaswithneighborhoodsTheactionstepstoachievethisgoalincludedcleaningupcreeksandremovingdebristomakethemmoreaccessibleThecleanupswentsmoothlyinseveralneighborhoodsbuttheCitymetwithoppositionfromoneneighborhoodthatpreferredthenaturalhabitatandexpressedconcernabouttheenvironmentalimpactofthecleanupsAccordingtoLewisvilleiftheCityhadmappedallofthestakeholderstodeterminewhoneededtobeatthetableandwhowasmissingfocusinginformationgatheringontheneighborhoodlevelithaveavoidedthisproblem
Startwiththewhothepeople
Establishaprojectteam
BeginbynamingtheprojectteamforthisprocessincludingaprojectleadTheprojectleadshouldbesomeoneinCitygovernmentwhoisrespectedcangalvanizeotherkeypartnersandhasthetimetoleadthisworkThisissomeonewhomayhavetohavedifficultconversationswithCityleadershipdepartmentsandotherfundedpartnersItisnotnecessaryfortheprojectleadtobeapolicyexpertItismoreimportantforthemtohavetherightrelationshipsandpositionwithintheorganizationTheprojectteamshouldincludeindividualswithdiverseperspectivesfrommultipledepartmentsThereshouldbesomeonewithpolicyexperienceontheprojectteamtogroundtheconversationinrealisticactionHavingateammemberwhocancoordinatewithresearchersonrigorousanalysisandtranslatethatworkforothersiskeyAdditionallybesuretoincludefrontlinestaffworkersTheywillhavevaluableinsightbasedontheirinteractionswithresidentsandpolicysolutions
Gatherinformationabouttheproblem
8
havetomakesensetothepeoplewhowillbechargedwithimplementingthemBuildingastrongdiverseprojectteamisimportanttoensurethattheworkdoesnotliveanddiebyoneperson
Identifykeypartners
NextidentifythepeoplewhotheprojectteamcanworkwithontheissueyouarefacingincludingpeoplewithlivedexperienceintheproblemyourcityistryingtosolvedepartmentstaffandsubjectmatterexpertsoutsideofcitygovernmentThisistheconceptbehindbuildwithnotforamovementledbyLaurenellenMcCannthatemphasizescommunityexpertiseandleadershipMappingoutkeyplayersandtheirrolesiscriticaldotoearlyintheinformationgatheringphasesincecoalitionbuildingissuchanintegralpartofthepolicychangeprocessThisprocessmightconsistofstakeholdermapping(describedbelow)communityresourcemappingandculturalmappingInyourmappingremembertoconsiderpotentialalliesandthosewhomaypresentroadblocksalongtheway
PolicymakingisnottopdownorlinearInsomecasescitiesmayhavetosharedecision-makingpowerwithotheractorsForinstanceinmanycitiesthelocalContinuumofCare(CoC)istheleadentityonhomelessnessservicesCoCsapplyforandstewardfederalfundstoaddresshomelessnessandcoordinateservicesamonglocalprovidersAlthoughcitygovernmentsdohavearoleinCoCstheyarenotalwaystheleadagencychargedwithdistributingfundsandthereforetheirroleinhomelessnesspolicymaybelimitedCountygovernmentsoftenhavehealthdepartmentswhilecitiesdonotInthosecasesthebalanceofpowerfavorscountygovernment
TheCityofDentonTXwasrecentlyfacedwithanissueofresidentcomplaintsaboutanewdocklessbikesharethatarrivedintownandisviewedbysomeasexpensivelitterInresponseDentonsCityCouncilwhichwasalreadydiscussingpotentialrulesforbikesharecompanieselevatedthisissueandchargedcitystaffwithidentifyingasolutionPresentlytheCityisreviewingordinancesfromnearbycitiestodeterminehowtheyaresolvingthisproblemandwhatsolutionsmaybefeasibleinDentonThemappingexamplebelowrepresentslocalactorsinDentonwithastakeinthisissuewhohavemultipleperspectivesandlevelsofpower
Gatherinformationabouttheproblem
9
UsethistemplatedevelopedbytheCentreforPublicImpact(CPI)tobrainstormwhoyoumightreachouttoasyouembarkontheroadtopolicychangeRemembertothinkabouthoweachpersonorgrouprelatestotheproblem
BegintodefinethewhatortheprobleminyourCity
TherestofthissectionwillhelpyourefinetheproblemstatementFornowgathersomekeypartnerstoclearlyarticulatewhychangeisneededUsethetemplatebelowdevelopedbyDenverPeakAcademytocapturetheproblemstatementYouwillbuildonthislater
ProblemStatementndashWhyChangeisNeeded(1-2Sentences)
ThroughourinterviewswithcitystaffweheardthatwhendiscussingmunicipalchallengesitcanbetemptingtojumptoasolutionbeforetakingthetimetoreallyunderstandtherootcauseoftheproblemThismakessensesincecityleadersandstaffareresponsiblefordeliveringservicesandbeingresponsivetoresidentswhileaddressingnewproblemsthatariseinthecity
TherootsoftheproblemsthatcitiesareworkingtosolvearecomplexandmaybethelegacyofhistoricalfederalpolicyorsweepingsocialtrendsTohelptheproblemseemlessdauntingitcanhelptobreakthemdownintotherootcausetoidentifyamanageablecomponentoftheproblemUsingafishboneexerciseorfivewhystechniquecanalsohelp
Gatherinformationabouttheproblem
10
tofindtherootcauseoftheproblemandavoidsimplyaddressingsymptomsoftheproblemandrepeatingthemistakesofhistoryToolssuchasjourneymapscanhelpyouunderstandtheexperienceofthosewhoareaffectedbytheproblem
TheCentreforPublicImpactwithsupportfromBloombergPhilanthropiesdevelopedanexercisetosupportcitiesinidentifyingtherightsizedproblemAccordingtoCPIarightsizedproblemisonethatislargeenoughtopotentiallyapplytoothercitiesbutsmallenoughtosolvewithinareasonabletimeframe
AtthebeginningofMayorBerkesadministrationintheCityofChattanoogaTNveteransexperiencinghomelessnesswereaskingtheCityforhelpinfindinghousingTalkingtothispopulationrevealedthatveteranshadtroubleidentifyingaffordablehousingwhichmotivatedtheAdministrationtoworkonasolutionTheproblemofveteransexperiencinghomelessnesswastherightsizebecauseitfocusedononesegmentofpeopleexperiencinghomelessness-veterans-andtheirspecificchallengeinidentifyingaffordablehousing
Developacomprehensiveunderstandingofhowyourcitycurrentlyaddressestheissue
IntheChattanoogaexamplementionedaboveCitystaffdiscoveredthatveteranswerehavingtroublenavigatingtheCitysvariousaffordablehousingprogramsandgettingaccesstothehousingandservicesthattheyneededAtthesametimenewlyelectedMayorBerkewasexploringcurrentandfutureopportunitiestoservethispopulationRatherthancontinuetousefederalfundingforemergencysheltersMayorBerkedirectedcitystafftoprioritize
Gatherinformationabouttheproblem
11
HousingFirstwhichvaluesconnectingpeoplewithpermanenthousingovertransitionalhousingandsupportiveserviceafterspeakingwithrepresentativesattheUSDepartmentofHousingandUrbanDevelopment(HUD)
ThisexaminationofwhattheCityiscurrentlyfundingandthesourcesofthatfundingwasaturningpointinChattanoogaUnderstandingthisproblemfromasystemsperspective-includingpoliciesandfundingstreamsatmultiplelevelsofgovernmentsocialnetworksthroughwhichinformationisoftenspreadtechnologythebuiltenvironmentofmarket-rateandpublichousing-helpedthenewMayoranalyzeexistingeffortsandleadstaffandotheractorsinthesamedirection
Begintogatherthedata
TotrulycapturethesuiteofservicesthatyourCityisprovidingaskdepartmentstafftolisttheprogramsrelatedtoyourproblemstatementthatyourcityrunsorfundsWhenlistingtheprogramscapturedetailssuchas
programnamethenameofthedatasetthatisrelevanttoeachprogramadescriptionofthedatasetthesourceofthedatasetifthedataisinternalorexternalwhetherthisdataisalreadybeingcollected
ThisinformationwillformthebasisofyourinventoryofadministrativedataordatasetsthatarekeptbygovernmentsforoperationalpurposesThismayincludecallcenterrecords(311911)taxdatapropertyassessmentsinspectioninformationsocialservicerecordsetcAtthispointlistallrelevantadministrativedatasetsYouwillprioritizewhichdatasetstogathermoreinformationonlaterwhenyouaredevelopingindicators
Asampletemplatethatcanbeusedtocapturethisinformationisbelow
ProblemStatement____________________________________________
Gatherinformationabouttheproblem
12
ProgramName
DatasetTitle
DatasetDescription DataSource
Departmentorexternalpartner
DoestheCityalreadycollectthisdata
Whatisthenameoftheprogram
Human-readablenameofthedataset
Shouldbeunderstandablebynon-technicalusers
Whatistheoriginalsystemapplicationfilethathousesit
Departmentorexternalpartnerthatmaintainsthedata
Isthisdataalreadycollected
TogatherthisinformationconsiderassigningonepointpersontosurveyorinterviewcitystaffSpeakingwithfrontlineworkersstaffwhoworkwithfederalgrantsdepartmentheadsandeveryoneinbetweenwillensureadeepunderstandingoftherelevantprogramsanddatasetsDuringthecourseoftheseconversationsbesuretoaskcitystaffwhichorganizationstheyworkwithexternallyandcapturethatinformationaswelleveniftheyarenotrecipientsofcityfundingInadditiontoconversationswithexpertsincitygovernmentyourcitysbudgetdocumentsareanothergreatsourceofinformation
Collectdataandobservetrends
Nexttrytogainanunderstandingofwhytheproblemstillexistsifithasbeensomethingyouhaveworkedonforawhileorwhyitisemergingnowifitisnew
AnalyzingadministrativedatacandemonstratepasteffectivenessofinitiativesrelatedtoyourproblemstatementUsingthedatasetinventorythatyoubeganintheprevioussteptalktodepartmentstaffaboutwhichdatasetsaremostrelevanttotheproblemandgatherafewyearsworthofhistoricaldataBegintolookfortrendsinthisdatadiscussingpotentialreasonsforthosetrendswithrelevantstaffCanthetrendsbeexplainedbyseasonalitychangesinfundingstaffingfluctuationsorsomeotherreasonBasedonthehistoricaldataanalysiscreateabaselineagainstwhichfuturechangescanbeobserved
OtherdatasetscanhelpprovideadditionalcontextForinstanceifthenumberofpeopleexperiencinghomelessnessinyourcityisincreasingitmaybehelpfultotakealookathousingaffordabilityIfyoudonothavethedatayouneedconsideraproxymeasureAsanexamplehousingisoftenconsideredaffordableifitcosts30orlessofahouseholdsincomeButifhouseholdincomedataisnotavailableatthecitylevelAreaMedianIncome(AMI)canbeahelpfulproxy
Gatherinformationabouttheproblem
13
QualitativedatacanalsohelpanswerquestionsaroundtheeffectivenessofpreviouseffortsTrytoobtainavarietyofperspectives--bothmainstreamandoutliers--fromnetworksbeyondyourowntoprovideyouwiththegreatestvantagepointforevaluatingpotentialimpactUnderstandinghowthoseeffortswerereceivedbydifferentpartsofyourcommunityandiftheremaybeelementsofprevioussystemstopreservewillbecrucialtothesuccessofanypolicychangeThisinformationmaybeobtainedthroughfocusgroupsinterviewsandsurveysFormoreinformationonhowtousethesequalitativeresearchtoolspleaseseetheseresourcesdevelopedbytheAgencyforHealthCareandResearchandUrbanInstitute
FinallytakealookatnationaldatatoseewhereelsethisproblemmightexistThisresearchmaybehelpfulinlaterstepswhenyouarelookingforpromisingpoliciesinothercities
Puttingitalltogether
UsethetemplatebelowdevelopedbyDenversPeakAcademytocaptureyourrefinedrightsizedproblemstatementandcurrentandfuturestatesWorkingthroughthisexercisewithpartnersishelpfultogainbuy-inTheresponseswillbeimportantwhengettingotherpartnersonboard
WhyChangeIsNeeded
Yourright-sizedproblem
CurrentState
Analysisoftrendsindata
FutureState
Ideallywhataretheoutcomesyouhopetoachieve
Checklist
Thekeystepstogatheringinformationarebelow
Atthispointfocusontheproblemratherthanjumpingtosolutions
Listentopeoplewhoareaskingforhelpandtoadvocateswhoareraisingissues
Beasspecificaspossibleinidentifyingtherootcauseandhoninginonthepartoftheproblemthatyourcitycanaddress
Identifytrendsbasedonmultiplesourcesofdataincludinghistoricaladministrativedataproxymeasuresqualitativedataandnationalmeasures
Gatherinformationabouttheproblem
14
Captureyourproblemstatementtheresultsofyouranalysisanddesiredoutcomes
Gatherinformationabouttheproblem
15
StrategicallyengagekeypartnersPolicytheoryresearchoftenfocusesonincreasingpolicymakersaccesstoevidenceasawaytoavoidtakingshortcutslikemakingdecisionsbasedonemotionImplementationsciencesuggeststhatincreasingaccesstoevidenceisnotenoughandthatpolicychangeshouldbeexaminedinthewidercontextofdebatecoalitionformingandpersuasionMeaningfulpolicychangerequiresldquoalong-termstrategytoformalliancesanddevelopknowledgeofpolicymakersandpolicymakingratherthanusingevidenceinpolicymakingonanadhocorone-offbasis
UnderstandingnetworksandsystemsisimportanttodevelopsustainablepolicychangeThissectionismeanttohelpreaderstakethestepstobuildengagementstrategiesforpartnersthatrepresentdifferentsystemsandnetworksEngagementshouldbeanongoingprocessofconnectingpeopletooneanotherandtoanissueEngagementshouldnotbeinnameonly-wehaveallexperiencedengagementactivitiesthatgivepeopleaspecificroletoplaythatleadstoaneatpredictableoutcomeAccordingtotheBuildWithmovementundervaluingandunderinvestinginengagementstrategiescanleadtoadisconnectofcertainindividualsorgroupsandmistrustofgovernmentWhendevelopingengagementactivitiesrememberthathowpeopleengageandwhethertheyengagehastodowithhowyoustructureopportunitiesforengagement
Conveneacoreadvisorygroup
AnadvisorygroupcomprisedofpeoplefrommultiplesectorscanbehelpfulasyouconsiderthelocalsocialenvironmentalandpoliticalcontextinwhichyouintendtointroducepolicychangeRevisityourstakeholdermapanddeterminewhowouldbebesttoserveonthisgroupTomakesureyouareconsideringtheproblemsandsolutionsthroughanequitylensincludepeoplewithlivedexperiencewhohaveexpertiseintheeffectivenessofsystemswhetherthosesystemsfunctionasintendedandhowtheymayimpacthumanbehaviorYoumayalsowishtoincludesomeonewithexperienceworkingwithdataandanalysis-thispersoncanhelpdeterminethetypeofdatathatmaybereadilyavailableandusefultocollectandcanalsohelpplanandshapeyouranalysistofigureoutwhichprogramsareeffectiveBesuretoinvitefront-lineworkersaswell-itscriticalthatwhateversolutionscomeoutofthisprocessaremeaningfultothosewhowillberesponsibleforimplementingthem
BringingpeopletogethercanbechallengingbutthereareafewwaystosetandmanageexpectationsFirstmakesurethatthegroupisawareoftheproblemstatementcurrentstateandfuturestate-thiscanhelpstarteveryoneoffonthesamepageNextspend
Strategicallyengagekeypartners
16
sometimegettingtoknoweachotherrsquosworktolearnabouteveryonesstrengthsThissharedunderstandingcanhelpasyouclarifyrolesandresponsibilitiesofeachpartnerinthegroup--thisisimportanttoavoidfinger-pointinginthefutureandtoensurethatnoneofthetasksslipthroughthecracksInsomecasestherearemanyactorswithdifferentresponsibilitiesasinthecaseofBaltimoreCitys38-memberYouthFundTaskForcewhichischairedbynonprofitandcorporatepartnersandincludesrepresentativesfromthenonprofitandfaithcommunitiesstudentsthebusinesssectorlocaluniversitiesCityCouncilandtheMayorsOffice
OncerolesandresponsibilitiesareclearyoumayconsiderdevelopingacharterguidelinesortermsofreferencetosetexpectationsforthegroupBesuretodistinguishthisfromotherboardsandcommissionsthatyourcitymayhaveNextyoumaywishtoestablisharegularongoingmeetingscheduleBesuretosendagendasandmeetingmaterialsaheadoftimeandclarifytheobjectiveofeachmeeting
InSeattletheCitydevelopeditsHousingAffordabilityandLivabilityAgenda(HALA)torespondtorisinghousingcostsandhomelessnessThisisatrendinotherwestcoastcitiesaswellwhichmaybeexplainedbyaninfluxofnewresidentswhoareattractedbytheemergenceoftechjobstheattractionoftheoutdoorlifestyleortherecentpassageofprogressivepolicieslikelegalizedgaymarriagea$15minimumwageandlegalizationofmarijuanaSeattlebroughttogetheramulti-stakeholderadvisorygroupcomprisedoflocalhousingexpertsrentersandhomeownersdevelopersandgovernmentofficialstoshareinformationabouttheirneedsanddeveloprecommendationsbasedontheseneedsTheinterestsandmotivationsofthisgroupwerevariedandittookabouttenmonthstofindcommongroundTheseriesofrecommendationsthattheadvisorygroupcreatedismeanttoserveresidentswithaspectrumofneeds
Buildaprocessforcommunityparticipation
Ofcoursenoteveryonecanbeincludedinanadvisorygroupandadvisorygroupmembersmaynotnecessarilyrepresentthevariedperspectivesoftheirpeers-forinstancenoteveryhomeownerhasthesamesetofneedsTogatherinputfromawiderswathofthepopulationitcanbehelpfultobuildaprocesstoinvitethepublicatlargetosharetheirideasCitiesdothisinavarietyofwaysfromonlineconversationsandsurveystoin-personmeetingsRememberthatbeingintentionalaboutgatheringtheviewpointsofpeopleismoreimportantthanusingthelatesttechnologyandtools
TheinsightgeneratedbycommunityparticipationexercisescanbeincludedinyourreviewofevidencewhichisdiscussedinasubsequentchapterInWestVirginiaastatethathasbeenhithardbytheopioidepidemicstatehealthofficialsdraftedanopioidresponseplanthatwasinformedbypublicengagementaswellaslocalandregionalsubjectmatter
Strategicallyengagekeypartners
17
expertsThepublicwasinvitedtopresentcommentsonideasfordecreasingoverdoseearlyinterventiontreatmentandrecoveryPublichealthandmedicalfacultyfromWestVirginiaUniversityMarshallUniversityandJohnsHopkinsUniversityprovidedresponsestothecommentsTheresultwasanOpioidResponsePlanthathealthcommissionerRahulGuptapresentedtoGovernorJimJustice
AsamplespectrumtoplanyourowncommunityparticipationprocesscreatedbyLaurenellenMcCannisbelow
SourceLaurenellenMcCann
Community-basedparticipatoryresearchisanotherwaytoworkwiththecommunitytoimproveoutcomesCBPRmeansthecommunityisinvolvedinallaspectsoftheresearchprocesswhichisofteniterativeInCBPRtraditionalresearchersandcommunitymembersshareknowledgeaswellasresourcesandcredit
CBPRInActionInSanFranciscoanenvironmentaljusticenonprofitteamedupwiththeCitysDepartmentofPublicHealthandanoutsideevaluatortounderstandtherootcausesofobesityandfoodinsecurityinlow-incomeneighborhoodsandidentifysolutionsThroughresearchmethodssuchasinterviewingresidentsandstoreownersdiagrammingstoreshelvesGISmappinganeconomicanalysisandreviewofsimilarpoliciesinnearbycitiestheteamobservedthatthestoresthatresidentshasveaccesstowerestockedwithlesshealthyfoodsandthatresidentswantedmorehealthyfoodandlessalcoholandtobaccoThisprojectresultedinlocalandstatelegislationtoencouragestorestostockfreshproduce
Strategicallyengagekeypartners
18
Developrelationshipswithsubjectmatterexperts
OnewaytokeepsubjectmatterexpertsengagedistoinvolvetheminthecommunityparticipationprocessasdescribedaboveIntheWestVirginiaexamplesubjectmatterexpertsrespondedtoandsynthesizedthepubliccommentsOtheropportunitiesaredescribedbelow
AdvisoryAcitydepartmentheadMayororstaffmembermaydevelopaone-on-oneadvisoryrelationshipwithuniversityfacultymemberoracollaborativebetweentheCityandUniversitytoaddressaparticularproblemAgreatplacetostartistolookforrelevantpracticecenterswithinyourlocaluniversityandreachingouttosubjectmatterexpertsInadditiontotheirexpertiseontherelevantpolicytopicheorshemayalsobeabletoidentifyotherexpertsandevenprovidesupportonhiringtherighttalenttoleadcertainprogramsAnadvisoryrelationshipwithauniversitysubjectmatterexpertcanalsohelpwhencitiesareconsideringbringinginnewprogramsTheseexpertsareoftenfamiliarwithprogramsthatarebasedonnationalmodelsandcanprovideinsightintoaprogramsoriginandlineageeffectivenessinotherplacesandthepotentialforimplementationinyourcityAcademicscanalsohelpidentifyindicatorsbasedongovernmentindicatorsorliteratureFinallyrelationshipsbetweenpolicymakersandresearchersprovidethemostbroadlyeffectiveroutetopullingresearchintopolicy
ResearchandevaluationAcademicexpertsmaysupportcitiesinevaluation-indeedevaluatingcityinitiativesisatraditionalmodelofcityuniversityrelationshipsSharingtheinformationyoucollectedintheGatherInformationphasecanhelpresearchersunderstandyourworkandgoalswhichinturnwillstrengthenthesupportthattheycanprovideDiscussingdetailsaboutyourinterventionandevaluationplanatthebeginningoftheprocessmeansthataresearchpartnercanhelpwiththeevaluationdesignanddatacollectionForinstancetheWilsonSheehanLabforEconomicOpportunitieswalkscitiesthroughconsideringthefollowingquestionsbeforepartneringonanevaluation
1 Isthereaclearpolicyprogrammodelwithobjectivescreeningcriteria
2 Isitpossibletoevaluatewhathappensbothwiththepolicyprogramandintheabsenceofthepolicyprogram
3 Arethereoneormoremeasurableoutcomesofinterest
4 Arethereenoughpeopleforrigorousquantitativeanalysis
5 Whatdatawillbenecessaryforthestudyandwhataretheprocessesforaccessingthesedatasource(s)
6 Isthereacompellingreasontoevaluate
Strategicallyengagekeypartners
19
InadditiontoassessingreadinesstoconductanevaluationresearcherscanhelpyouhoneinonthepurposeoftheevaluationTheWorldBankdescribesseveralpotentialusesofevaluationwhicharedefinedhereInbriefevaluationcanhelpdecisionmakers
makeresourceallocationdecisionsrethinkrootcausesofproblemsidentifynewproblemsdecideonabestalternativedeterminewhetherprogramsorpoliciesarebeingimplementedwithfidelitydeterminewhetherprogramsorpoliciesarehavingthedesiredimpact
Researcherscanalsoprovidesupportinsharingtheresultswiththescientificcommunitythroughpapersandconferenceswhichisvaluableincreatingapeernetworkofevidence-basedcitypractitioners
TrainingandtechnicalassistanceUniversitypartnersespeciallythoseatpracticecenterscanbehelpfulindevelopingorupdatingtrainingcurriculumtrainingthetraineranddeliveringtrainingtocitystaff
Strategicallyengagekeypartners
20
ReviewtheevidenceandcollectdetailsOnceyouhaveathoroughunderstandingoftheproblemdataandactorsyouwillneedtoconductyourownresearchonavailableevidenceandlearnaboutthelandscapeofpolicyoptionsYourgoalwillbetodevelopanaccessiblelistofinformationaboutpoliciesthathavehelpedsolvesimilarproblemsinotherplacesThischapterwillguideyouthroughthestepsofpreparingyourreviewersunderstandinglevelsofevidenceandcollectingthedetailsonevidencethatyouwillneedfordiscussionwithdecisionmakersinyourcity
Selectandprepareyourreviewers
AssignapersonorteamtoreviewrelevantinterventionsthathavedemonstratedsuccessinotherjurisdictionsTheirgoalwillbetoproduceareviewofalloftherelevantprogramspoliciesorinitiativesthatyourteamcouldconsiderimplementingTheywillbeconductingaliteraturereviewsupplementedbyinterviewswithexpertsandpeerswhohaveinformationaboutpotentialprograms
SincethepeopleyouassigntothistaskareprovidingthebaselineofpotentialoptionstoimplementitisimportantthatyouchoosepeopleyouareconfidentwillbethoughtfulandthoroughintheirworkItisalsocriticalthattheyareawareoftheimportanceofsourcecredibilityEnsureyourreviewershavefamiliarizedthemselveswithrulesforcriticallyanalyzinginformationsourcesbeforetheygetstartedIftheyareincludingnewsarticlesaspartoftheirsearchtheyshouldalsobeawareofmethodsforevaluatingnewscredibility
AskyourreviewerstodevelopaplanandtimelinetocompletethereviewThisstartswithsolidifyingagoverningresearchquestionOnewaytoknowyouarecreatingadetailedresearchquestionistousetheacronymPICOremindingyoutoincludereferencetothepopulationintervention(ifyouhaveoneinmind)comparisonandoutcomeelementsinyourresearchquestionForexampleifyouknowthatyouhaveapopulationofteenagerswhoarehomelessyouhavelearnedfromyourstakeholdersthatmanyoftheseteenagersidentifyasLGBTandyouhaveheardsomethingabouttheapproachofcreatingdrop-incentersfordeliveringsupportyoucouldask
ForLGBTyouthfacinghomelessness(population)howeffectivearedrop-incenters(intervention)relativetostandardsocialservices(comparison)atsettlingtheminpermanenthousing(outcome)
Ifyoudonothaveaninterventioninmindyoucanleaveouttheinterventionandcomparisonandsimplyask
Reviewtheevidenceandcollectdetails
21
HowcanwehelpLGBTyouthfacinghomelessness(population)obtainpermanenthousing(outcome)
OnceyourreviewershavetheirresearchquestionsusethequestionstocreatealistofkeywordstheycanuseintheirliteraturesearchThiswillhelpkeeptheircollectionofarticlesreasonablyfocusedHavethemdescribewhattheirsearchprocesswillbeincludinglistingthedatabasestheywillreviewandtheirplanforstoringtheinformationtheywillcollect(Herearesomeadditionalsuggestionsforstructuringareviewplan)
WhatyourreviewerswillbedoingisalightweightversionofarapidreviewTotheextentthattheywouldliketodevelopamethodologyforspecificsearchprotocolsincludingcriticalappraisalprocessestheycancertainlygoaheadanddoso
Usedifferentlevelsofevidenceforthemostcompletepicture
AsdiscussedintheWorkingwitharangeofevidencechaptertheNestaStandardsofEvidenceframework--reproducedbelow--ishelpfultounderstandthelevelofconfidencethataprogramorinterventionishavingapositiveimpactThisapproachletsyouputinterventionsthatyouhaveheardaboutlocallybutwhichhavenotbeenstudiedtogetherwiththosewithademonstratedcausaleffectThisishelpfulsinceanumberofpoliciesprogramsandpracticesthatyouwillwanttoconsidermaynotyethavemuchofaresearchrecord
Reviewtheevidenceandcollectdetails
22
NearlyanypieceofinformationaboutaninterventioncanbeclassifiedwithinthestandardsofevidenceForexampleifsomeonehasacrediblehypothesisabouthowaninterventioncouldproducebenefitthistypeofevidencewouldbeclassifiedasLevel1ThistypeofevidencecouldincludeinterventionsinothercitiesthatyoureadaboutinthenewsorhearaboutataconferenceForinstancewhenUSDAallowedschoolsandschooldistrictstoprovidefreebreakfastandlunchtoallstudentsthroughtheCommunityEligibilityProvision(CEP)manydistrictsimplementedthisoptionevenbeforetherewasevidenceofeffectivenessThiswouldfitintoLevel1sinceitbuildsonthelogicthatprovidingaccesstomealswilldecreasefoodinsecurity
YoumayalsofindpiecesofinformationthatqualifyasLevel2evidenceForinstanceasmoreschooldistrictsimplementedCEPprincipalstestifiedaboutpositivechangestheyobservedinstudentssuchasamorepositivedemeanorthatcanleadtobetteracademicperformanceDataalsoshowedthatthestudentsdescribedwerereceivingmoremealsthanbeforetheprogramsimplementationHoweveritwastoosoontobuildevidencetoshowacausalrelationshipbetweentheincreaseinfreemealsandincreasedacademicperformance
ldquoLower-levelrdquoevidencecanconsistofnuancedbutvaluableelementsthatrelatetoaninterventionrsquospotentialfitwithacommunityQualitativeinformationsuchastestimonyfrompeoplewhohaveexperiencedaninterventionandthosewhohavefacilitatedorimplementeditwillalwaysbeusefultoconsiderandprovidesdetailedlocalcontext
Experimentalstudies--describedstartingatLevel3withcontrolledexperimentsandgoingthroughLevel5withfullyfleshedoutprotocolsandprocessesforreplication--stilloccupythehigherlevelsofthisdescriptionofevidenceThisprioritizationshowsthevalueofrigorousstudyforevaluatingtheimpactofaparticularpolicyorprogramratherthanotherfactorsthatcouldbeinfluencingoutcomesofinterestRememberthatmultipletypesofevidencecancomplementrigorousimpactevaluationsandgivecityleadersadditionalinsightsandbroaderperspective
Accessinformation
Encourageyourreviewerstobegintheirsearchforpromisingpracticesingovernmentandnon-profitclearinghousesandinacademicpolicyjournalsWorkpublishedinthesesourceshasalreadybeensubjectedtoahighlevelofprofessionalscrutinyandgenerallycontainsagreatdealofdetailwhichmeansthatyourreviewersaremorelikelytobeabletopullouttheinformationyouneedtomakeagooddecision
Clearinghouses
Reviewtheevidenceandcollectdetails
23
Herearesomeoftheclearinghouses(collectionsofprogramevaluations)forevidence-basedpoliciesandprograms
ResultsFirstClearinghouseDatabaseThePewCharitableTrustsCollectstogethereightotherclearinghousestoprovideinformationoninterventionsinthefollowingcategoriesadultcriminaljusticeeducationchildwelfarehealthjuvenilejusticementalhealthsubstanceabuseandyouthdevelopmentIndividualclearinghousesincludedinResultsFirstare
BlueprintsforHealthyYouthDevelopment-foryouthdevelopmentandviolenceprevention
CaliforniaEvidence-BasedClearinghouse-forchildwelfareinterventions
TheLauraandJohnArnoldFoundationrsquosSocialProgramsThatWork-forsocialservicesprograms
TheUSDepartmentofEducationrsquosWhatWorksClearinghouse-foreducationinterventions
TheUSDepartmentofHealthandHumanServicesrsquoResearch-TestedInterventionPrograms(RTIPs)-forcancercontrolinterventions
TheUSDepartmentofHealthandHumanServicesrsquoTeenPregnancyPreventionEvidenceReview(TPPEvidenceReview)-forsexualriskbehaviorandsexualhealthoutcomes
TheUSDepartmentofJusticersquosCrimeSolutionsgov-forcriminaljusticejuvenilejusticeandcrimevictimservices
TheUSSubstanceAbuseandMentalHealthServicesAdministrationrsquosNationalRegistryofEvidence-basedProgramsandPractices(NREPP)-substanceuseandmentalhealthinterventions
TheUniversityofWisconsinPopulationHealthInstituteandRobertWoodJohnsonFoundationrsquosCountyHealthRankingsandRoadmapsWhatWorksforHealth(WhatWorksforHealth)-healthbehaviorsclinicalcaresocialandeconomicfactorsandthephysicalenvironment
UKWhatWorksCentresTheBritishgovernmenthascharteredanetworkofsubject-specificorganizationswhichcollectsynthesizeandreportoutevidenceontheeffectivenessofpolicyprogramsandpractices
NationalInstituteforHealthandCareExcellence(NICE)-healthandsocialcare
Reviewtheevidenceandcollectdetails
24
SuttonTrustEducationalEndowmentFoundation-educationalachievement
CollegeofPolicingWhatWorksCentreforCrimeReduction-crimereduction
EarlyInterventionFoundation-earlyintervention
WhatWorksCentreforLocalEconomicGrowth-localeconomicgrowth
CentreforAgeingBetter-qualityoflifeforolderpeople
WhatWorksCentreforWellbeing-wellbeing
WilsonSheehanLabforEconomicOpportunities(LEO)UniversityofNotreDameProvideslinkstotheirevaluationsofinterventionsincriminaljusticeself-sufficiencyeducationhealthandhousingandhomelessness
ClearinghouseforLaborEvaluationandResearchUSDepartmentofLaborProvidesresearchonworkforceandlabor-relatedpoliciesandprograms
J-PALEvaluationsAbdulLatifJameelPovertyActionLabStudiesonpoverty-reducinginterventionsfromaroundtheworld
CampbellSystematicReviewsCampbellCollaborationSystematicreviewofpolicyinterventionsbyscholarlycoordinatinggroupsinsevenissueareascrimeandjusticeeducationinternationaldevelopmentmethodssocialwelfareknowledgetranslationandimplementationanddisability
Benefit-CostResultsWashingtonStateInstituteforPublicPolicyAWashingtonState-charteredprogramtoestablishthelikelihoodofrelativecostsandbenefitsforprogramsacrossthefollowingareasjuvenilejusticeadultcriminaljusticechildwelfarepre-Kto12educationchildrensmentalhealthhealthcaresubstanceusedisordersadultmentalhealthpublichealthandPreventionworkforcedevelopmentandhighereducation
EffectiveInterventionsCentersforDiseaseControlandPreventionRepositoryforevidence-informedHIVpreventioncareinterventions(withstepsonhowtoimplement)
Journalsandindexes
OnceyouhavereviewedrelevantinformationaccessiblethroughclearinghousesitisworthtakingalookatindividualstudiespublishedasscholarlyarticlesYourreviewerswilldothisbysearchingarticleindexesandacademicjournalsUnfortunatelyanumberofusefulacademicjournalsandindexingplatformsmaynotbegenerallyaccessibletoyourreviewersifyoudonothaveasubscriptiontothemToaddressthisconsiderpartneringwithalocal
Reviewtheevidenceandcollectdetails
25
collegeoruniversitytoprovideyouwithalistofarticlesfromthefollowingsources(Thisisalsoanexcellentopportunitytoseekresearchsupportfromanyacademicstakeholdersyouidentifiedpreviously)
Indexes
GoogleScholarincludesmostpeer-reviewedonlineacademicjournalsandbooksconferencepapersthesesanddissertationspreprintsabstractstechnicalreportsandotherscholarlyliteratureincludingcourtopinionsandpatentsItallowsyoutospecifyapublicationrangeinyoursearchandprovideslinkstolikelysourcesforalllistedarticles
PolicyFileIndexisauniqueresourceforUSpublicpolicyresearchinthatitgrantsusersaccesstotimelyupdatedinformationfromover350publicpolicythinktanksnongovernmentalorganizationsresearchinstitutesuniversitycentersadvocacygroupsandotherentities
PolicyArchiveworkstocollectanddisseminatesummariesandfulltextsvideosreportsbriefsandmultimediamaterialofthinktankuniversitygovernmentandfoundation-fundedpolicyresearch
Anothersourceforhigh-qualityresearchisthereportsoftheCongressionalResearchServiceYoucanfindtheirreportsaggregatedhere
Specificjournals
Oftenthemostusefulacademicjournalsforunearthingpolicyorprogramreviewswillbeineachspecificsubjectarea--thatisineducationjournalsforeducationpoliciesYoucanuseageneralindexlikeGoogleScholartobegintolearnwhichjournalsaremostoftenusefulforyoursearchHoweveryoucanalsosearchthefollowingtwogeneralistpolicypublicationsforissuesandarticlesrelatedtoyourtopic
PublicAdministrationReviewJournalofPolicyAnalysisandManagement
Newssourcessubjectmatterexpertsandinterviews
AlthoughitisvaluabletoknowwhathasbeenexploredthroughascholarlyresearchframeworkitisalsousefultolearnwhatkindofinterventionsarebeingtriedinthefieldrightnowandhowpeoplefeelaboutthesetrialsSearchsubscriptionnewsaggregatorslikeLexisNexisorfreenewsaggregatorslikeGoogleNewsorYahooNewsforyourissueareaplusthewordpolicyorprogramtoseewhatkindsofinterventionsarebeingcoveredinthenews
Reviewtheevidenceandcollectdetails
26
DonotforgettocontacttheadvisorygroupmemberswhowerechosenfortheirsubjectmatterexpertiseInadditiontoprovidinggoodmaterialthroughinterviewsthemselvestheymayhaveideasaboutothersubjectmatterexpertsyoushouldcontacttoaskaboutpotentialinterventions
Peercitiesandnetworks
InadditiontoreadingabouttheworkofothercitiesitisidealifyouareabletospeakwithpeoplewhoareinvolvedwithcurrentimplementationsofinterestinginterventionsInyourcityyoulikelyhaverelationshipswithyourneighborsbutitcanbehelpfultolookoutsideofyourregionforsolutionsaswellTofindcounterpartsinothercitieswhoareworkingonaprogramyouwouldliketolearnmoreaboutyoucancontactstaffinintermediaryorganizations(likeGovExoryourprofessionalassociations)whomightbeinapositiontoputyouintouchimmediatelyorwhomightseeanothergoodrouteforsettingupaconversationHereisalistofnetworksthatmaybehelpful
InternationalCityCountyManagementAssociation(ICMA)NationalAssociationofCounties(NACO)NationalLeagueofCitiesUSConferenceofMayorsWhatWorksCities
Collectthedetails
Onceyourreviewershavefoundevidenceonpossiblepoliciesorprogramstoimplementtheyshouldcapturekeydetailsthatwillhelpdecisionmakersunderstandthepotentialeffectivenessandapplicabilityofeachpotentialintervention
Foreachsourcethathasprovidedinformationyouwanttoconveyusethetemplatebelowtocaptureasmanyofthefollowingdetailsaspossible(Youdonotneedtocompletethisforsourcesthatprovideduplicativeinformationorareotherwisenotthatuseful)
Programorpolicyname (egldquoHousingFirstrdquoorldquoblightedpropertiesregistryrdquo)
Sourcetype (egjournalarticleinterviewsubjectmatterexpertstudyassociationcommunityparticipationprocess)
Sourcedetails (egarticletitleandpublicationnameandtitleofinterviewee)
Evaluationmethodology
(egrandomizedevaluationcomparativeevaluationoftwodifferentgroupsbeforeafteranalysis)
Evidencelevel (egLevel12345)
Programorpolicy
Reviewtheevidenceandcollectdetails
27
description
Programorpolicygoal (1sentencecapturingtheprincipalaimoftheintervention)
Existingimplementations
(WherehastheinterventionbeentriedListasmanysitesasyouareawareof)
Characteristicsofimplementationsites
(ForeachimplementationsiteofferbasicgeographicIDmdashcityandstateThenyouwillwanttocollectthedetailswhicharerelevanttoyourdecisionmakersrsquoconsiderationsaboutwhatmakesalocationldquosimilartordquoorldquodifferentfromrdquoyoursituationYouwillatleastwanttoincludeforeachimplementationsitePopulationnumberPopulationdensity(ruralurbansuburban)Socio-economiccharacteristicsBeforestartingaskyourdecisionmakersaboutanyparticularcharacteristicsofprogramimplementationsitestheywouldwanttoknowbeforeseekingtoreplicate)
Characteristicsofimplementationpartners
(Anyavailabledetailsaboutorganizationsinstitutionsorpeoplewhowereinvolvedwithimplementingtheintervention-egindependentchildcarecenterpublicelementaryschoolcitypublichealthnurses)
Characteristicsoftargetpopulationeligibleprogramparticipants
(Anyavailabledetailsaboutpeoplewhoarethebeneficiariesoftheprogramorotherwiseincludedoraffectedbytheintervention-egnumberofprogramparticipantscategoriesofpropertyownersaffected)
Policyorprogramduration
(HowlonghastheinterventionbeeninplaceineachimplementationsiteHasitendedandifsowhy)
Whatbenefitscamefromeachimplementation
(Anyobservedpositiveeffectsoftheinterventionincludingeffectsize)
Statisticalestimatesofinterventioneffectiveness
(Anyavailablestatisticalestimatesoftheinterventionrsquoseffects)
Whatdrawbackswereobserved
(Anyavailabledetailsaboutunintendedorunavoidablenegativeeffectsoftheintervention)
Programorpolicycosts
(Anyavailabledetailsaboutprogramcostsperimplementation)
Reviewtheevidenceandcollectdetails
28
Reviewtheevidenceandcollectdetails
29
DecideonapolicytoimplementbasedonlocalfactorsTogetthecriticalbuy-infromleadershiptoimplementanewpolicyorprogramtheprojectleadshouldscheduleameetingwithdecisionmakerstopresenttheevidencesurroundingthepolicyoptionscapturedinthetemplateDependingonyourinsightintoyourcityleadershipswillingnesstomakechangeyoumaywishtosharetheresearchyouhaveconductedontherootcausesoftheproblemandsomeofyouranalysisontheimpactofcurrenteffortstodate
Considertheapplicabilityofanapproach
AsdiscussedinthepreviouschapterpolicymakersoftenlookforevidenceofeffectiveprogramsinothercitiesbutoftenthequestionarisesofhowgeneralizableaprogramisfromonecitytoanotherTheAbdulLatifJameelPovertyActionLab(J-PAL)suggeststhatinreviewingevidencefromanotherlocationitishelpfultoidentifythespecificmechanismthatwasused(deliveryofprogramorserviceinformationsharingetc)identifythelocalconditionsthatwereinplaceobservethebehaviorchangethatresultedfromthemechanismandfinallytounderstandlocalcontextandconditions(legalorcostconstraintsculturalnormsetc)Thissetofpresentationelementscanbedrawnfromthelistofdetailscollectedintheresearchtemplate
EducationpolicymakersinChicagoILusedfindingsfromprogramsinKenyaandIndiainwhichchildrenwhohadfallenbehindtheirpeersweregiveneducationalsupporttargetedtotheirleveloflearningThemechanismusedwastargetedteachingandthereweresimilarconditionspresentinallthreecountriesAlthoughsomestudentshadfallenbehindinthecurriculumteacherswereincentivizedtoteachgrade-levelmaterialsConsiderationsofhowthelocalcontextinChicagomightaffectimplementationofatargetedteachingprogramincludedunderstandinghowtheneighborhoodsthatthestudentswerefrommayhaveaffectedtheireducationaloutcomesandthehighcostofteachinginsmallgroupsInChicagoMatchEducationprovidedtwo-on-onetutoringwhichmatchedstudentswithwell-educatedindividualswhohavenothadformalteachertrainingforalower-costmodelTheresultswerepositiveStudentswhoweretutoredperformedbetterinmathandweremorelikelytopasshighschoolclasses
ThegeneralizabilityframeworkproposedbyJ-PALisbelowUsethisasatemplateasyouconsiderpotentialpolicysolutions
Decideonapolicytoimplementbasedonlocalfactors
30
SourceTheGeneralizabilityPuzzle
OnceyouhavenarrowedthepolicyoptionsdiscussthequestionsbelowforeachoptionsyouareconsideringThisstepiscriticalasunderstandinglocalcontextissoimportantinsuccessfulpolicyimplementation
CommunityContextQuestionnaire
Localcontext
DoesthisoptionmakesenseinourcommunityIsthisaonesizefitsallapproachIsanyparticulargroupabsentfromthisapproachIfsoisthatok
Youranswers
Effectiveness
WhatwouldweexpecttoobserveifthepilotisaseffectivehereasinCityX
Youranswers
Timing
DoesthetimingmakesenseIsthistherighttimetoimplementthispolicychange
Decideonapolicytoimplementbasedonlocalfactors
31
WherearethepotentialroadblocksthatcouldaffectthescheduleIsthereanywaytomitigatethisHowlongwouldittaketoseeanimpact
Youranswers
Implementation
DowehavethestaffcapacitytoimplementthisArethereadditionalpartnersthatyoucanbringtothetabletosupportimplementationDoesthismakesensegivenourformofgovernment
Youranswers
Authority
DoesourcityhavetheauthoritytoenactthisIfsowhatistheappropriatemechanismAretherelegalbarriersArethereanyexistingpreemptionissuesfromthestateorfederalgovernmentCanthosebeaddressedthroughlegislationadvocacyIstherepoliticalsupportoroppositiontothis
Youranswers
Cost
HowmuchwillthiscostannuallyHowwillwepayforitCanthepolicybeimplementedwithexistingfunding
IfsoisthisfundingreliableHowwouldthispolicyoptionaffectourcitysbudgetintheshortandlongtermWhatotherfundingoptionscanweconsiderIsthereanalternativethatwouldcostlessandgiveussimilarresultsWillanyconstituencybeadverselyaffectediffundingforprogramsthatimpactthemisshiftedtoanotherprogrampolicyorinitiative
Youranswers
Decideonapolicytoimplementbasedonlocalfactors
32
EstablishaframeworktomonitorprogressNowthatyouhavereviewedtheevidenceconsideredlocalcontextandchosenyourpolicyitistimetodeterminehowyourcitywillmonitorprogressofthisnewapproachCitystaffandleadersneedtoestablishtheinfrastructureforongoingcontinuousdecisionmakingbasedonwhetherthepolicychangeisleadingtothedesiredoutcomesforresidentsandwhetherthechangeismeaningfulandrationaltofrontlineworkersSettingarbitrarygoalsatthelocallevelthatconflictwithgoalsatotherlevelsisconfusingtotheindividualsresponsibleforachievingthegoalsGoalsthataresettoohightoachievecanbedemoralizingtostaff
ThissectionincludesstepstosetupavenueinwhichCitystaffandleadershiproutinelyvisitkeyindicatorstodeterminewhatisorisnotworkingandtomakedecisionsinreal-timeFormorereadingonperformancemanagementpleasevisittheGovExPerformanceManagementGettingStartedGuide
InperformancemanagementtermssuchasindicatormeasureandmetricmaybeusedinterchangeablyInthisguidewewilluseindicatorInyourworkthemostimportantthingistobeconsistentinwhichevertermyouuseThisvideomaybehelpfulinunderstandingthisconceptandcommunicatingittoothers
Establishthefundamentalsandgatheryourteam
FirstdeterminewhetherthereisanestablishedvenueorregularmeetingscheduleduringwhichyoucandiscussprogressIfyourcityhasanexistingperformancemanagementstructureonarelatedtopicaddingadiscussionoftheprogressofpolicychangetoyouragendacanbeagreatideaIfthiswillbeastandalonemeetingdecidehowfrequentlytheprojectteamanddecisionmakerswillmeetandputthosemeetingsonthecalendarfortheentirelengthoftheproject
DeterminewhowillstaffandattendroutinemeetingsonCityprogressThesemeetingsmayincludetheMayorandorCityManagerdepartmentheadstaffwhoareimplementingthepolicychangeandanalysts
Identifyindicators
ItiscriticalthatindicatorsbeanimportantcityandcommunitypriorityaswellassomethingtheCityplaysamajorroleinaddressingTheremaybeinstancesinwhichtheCitywillneedtoreachouttoorcollaboratewithexternalpartnerstogetamorecompletepictureofprogressontheoverallpolicychange
Establishaframeworktomonitorprogress
33
Anoutcomeindicatordescribesthedesiredeffectofthetargetedpopulation
ExampleBellevuesfamilyfriendlyneighborhoodsindicatormeasuresthepercentofresidentswhobelievetheyliveinaneighborhoodthatsupportsfamiliesparticularlythosewithchildren
WithyourkeypartnersrevisityourproblemstatementcurrentstateandfuturestateexerciseandbegintoidentifytheoutcomeindicatorsthatwillillustratesuccessFornowchooseasingletop-levelkeyperformanceindicator(KPI)ItiscriticalthattheKPI-thepieceyouwillmeasure-beapriorityforthecommunityandsomethingtheCityplaysamajorroleinaddressing
Outputindicatorsmeasuretheprocessofprogramorservicedelivery
ExampleSanDiegoMayorKevinFaulconersgoaltorepair1000milesofstreetby2020
ReviewingtheadministrativedatasetinventorybegintoparedowntherelevantprogramsanddatasetsthatsupportyourKPIWhatarethemeasuresofeffectivenessforeachprogramTheseareyouroutputindicators
Thediagrambelowillustratestherelationshipbetweenoutputandoutcomeindicators
ThinkaboutimplementationindicatorsasmeasuringstepsintheprogramsrolloutthatarenecessarytoobtaindesiredchangesImplementationoutcomescanfocusontheinterventionstaffdeliveringtheinterventionanorganizationorsystemandcanaddressfeasibilityadaptabilitysustainabilityandcostTwokeyquestionstoconsiderwhendevelopingimplementationindicatorsare
1 Istheinterventionacceptableandappropriatetothecommunityand
2 IsitfeasibletoimplementinterventionXatthistimewiththeseresources
Establishaframeworktomonitorprogress
34
Amoreextensivelistwithdefinitionsofpotentialimplementationoutcomesislinkedhere
UsethetablebelowtochoosethedesiredimplementationoutcomesforyourpilotandtoplanyourmeasurementtechniqueThisshouldincludelevelofanalysis(individualproviderorconsumerorganizationorsetting)yourmeasurementtechnique(surveysobservationinterviewfocusgroupsadministrativedataanalysis)andthephasewhenyouwillcollectthisinformation(earlymidlate)Theremaybemorethanonelevelofanalysismeasurementtechniqueandphaseofdatacollection
Implementationoutcome
Definitionofoutcome
Levelofanalysisandmeasurement
Yourmeasurementtechnique
Phaseofpilotwhenyouwillcollectdata
Usethespacebelowtocaptureyourindicatorwishlist
Indicatorwishlist
Outcomes_____________________________________________________________________________
Outputs________________________________________________________________________________
Implementationoutcomes______________________________________________________________
Setapublichigh-levelgoal
Thehigh-levelgoaliscriticalinestablishingdirectiontogalvanizethepeoplewhowillbedoingtheworkWhendiscussingthegoalusetheMayorsbullypulpitLookforkeymilestonesinCitygovernmenttosharethegoalstatementsuchasthebeginningofaMayoraladministrationaStateoftheCityspeechorotherpublicremarks
GoalsshouldbeldquoSMARTrdquo
Specificaspossible(innamingthepopulationwhowillbebetterofftheactionthecitywilltake)
Measurable
Achievable
Establishaframeworktomonitorprogress
35
Results-focused
Time-bound
ThegoalstatementshouldincludeyouroutcomeindicatorlikeChattanoogasgoalldquoWewillendchronicveteranhomelessnessbyDecember2016rdquoAdiagramtohepyoubuildaSMARTgoalaroundyouroutcomeindicatorisbelow
Component Description Yourturn
Specific WhowillbebetteroffWhatactionthecitywilltake
Measurable Howwillyoumeasuresuccess
Achievable Isthegoalattainable
Results-focused Whatistheexpectedresult
Time-bound When
Setrealisticambitioustargetsandtimelinesforeachindicator
ToestablisharealisticandambitioustargetforyourgoalstatementusetheadministrativedatacollectedforyourchosenKPItoreviewactualCityperformanceinthemostrecentyearcomparedwiththethreeorfourmostrecentyearsworthofdataBegintocalculatesummarystatistics(egtotalminimumaveragemedianmaximum)toidentifytrendsCreatecumulativeandmonthlychartstoidentifytrendsandanomaliesBasedonyouranalysisofhistoricaldataignoretheanomaliesandassumetheproductivitytrendswillfollowthesameoverallpatternfrommonth-to-monthFinallyapplyanincreasedproductivityfactortotheperformanceeachmonthtoforecastfutureprogressBasedonthisanalysischooseatargetthatinspiresimprovementwhileaccountingforreality
Manycitiesthataremorematureintheirperformancemanagementpracticesrecognizetheimportanceoftakingacommunity-levelapproachtoimprovingoutcomesforresidentsThesecitiesarecreatinggoaltaxonomiesthatincludemeasuresrelatedtopublicsafetyhousingandevenrecreationCitiesdriveprogressonthesegoalsbyimplementingstrategiesthatengagemultipledepartmentsandcollaboratewithserviceprovidersinthecommunity
EventuallyyouwilldevelopgoalstatementsfortheoutputsinordertocapturetheefforttoachievetheoutcomecontainedinyourgoalstatementYoursetofindicatorsshouldbecomposedofshort-andinterim-termsupportivemeasureswithaccuratetargetsandrelevanttimelinesAnexamplegoaltaxonomyfromtheCityofLasVegassResultsVegasprogramisbelowUsethistaxonomyasamodelforyourownsuiteofindicatorsandgoalstatements
Establishaframeworktomonitorprogress
36
Yourturn
Thetemplatebelowismeanttohelpyoucraftagoalstatementforeachsupportingmeasure
Goalstatement
Departments
Outcome
Activities
KPI
Target
Timeframe
Conductanddiscussrelevantanalysis
ThrougharoutineseriesofmeetingscontinuallymonitorprogressofKPIssupportingindicatorsandanyotherdatacollectedsuchasqualitativedatatoidentifytrendsthatwillindicatewhetherthechosenpolicyishavingthedesiredimpactThesedatapointsshoulddrivethediscussionduringthesemeetingspointingtoopportunitiesforfurtherinquiryinto
Establishaframeworktomonitorprogress
37
whatisdrivingtheobservedtrendsAnalystsshouldbeencouragedandtrainedtoaskquestionsofthedepartmentsproducingthedataandconductsitevisitswhenpossibletodeepentheirunderstandingofthedataPresentationoftheanalysisshouldfocusononeideaatatimetokeeptheconversationfocusedonkeypointsandspecificopportunitiesforimprovementYoumayconsiderdevelopingapublicfacingdashboardtosharethisprogressmorebroadly
AgeneralcomparisonofthestatusquodatawithwhatwasobservedduringthepilotcanprovidedecisionmakerswithaquickunderstandingofthedifferencesbetweentheoldandnewapproachesalthoughthiswillnotshowcausalityOftenthesecomparisonswillincludeoperationalmetricssuchasdaysordollarssavedbutrememberthattheimpactonpeoplewillbemostimportantIfyouareworkingwithanevaluationpartnertheywillsupportthisstep
Usethespacebelowtosummarizetheresultsofyouranalysis
MeasureTargetvalue(seesectiononsettingtargetsabove)
Pilotvalue(whatdidyouobserveduringthepilot)
Historicalaverage(ifavailable)
Results(WhoisbetteroffIsanyonenegativelyimpacted1-2sentencesreassessingyourgoalstatement)
DocumentyourprocessandprogressincludingfeedbackfromtheparticipantsimplementationpartnersandtheprojectteamDidanythingunexpectedoccurduringthepilotArethereinternalandexternalfactorsthatmayaffectyouroutcomesThisiswheretheimplementationoutcomescanbecanhelpfulIftheinterventiondidnotgoasplannedconsiderwhetheritisbecausetheinterventionwasnotacceptableorfeasible
UsethetemplatebelowcreatedbyDenversPeakAcademytocapturethisinformationIfthe306090daytimelinedoesnotmakesenseforyourcityfeelfreetocustomizeit
CurrentState
FutureState
30Day
60Day
90day
Whatwentwell(+)
Whatdidnotgowellshouldbechanged(-)
IfaquestionordiscussionpointarisesthatcannotbeansweredinthemeetinganassignedstaffmembershouldcapturethequestionduringthemeetingAfterwardthefollowupitemsshouldbesenttorelevantpartnersandthenextmeetingshouldbeginwithadiscussionof
Establishaframeworktomonitorprogress
38
thoseitems
Takeaction
DiscussionsaboutwhetheryourindicatorsaremovingintherightdirectionandwhatmightbedrivingtrendsmayleadtoadiscoverythattherearenotenoughresourcesallocatedtothispolicyorissueTheremaybelimitedfundingforstaffprogrammingortechnologythatisdrivingthedatainthewrongdirectionIfCityleadershipisintheroomthisisagreattimetomakethecaseforadditionalfundingduringthebudgetseasonBycollectinganalyzinganddiscussingdataonthispolicyroutinelyyouarebuildingevidencetomakethecasefortherightleveloffundingtoreachtheestablishedtargets
Establishaframeworktomonitorprogress
39
Assessdataaccessibilityqualityandsecurity
Determineaccessibilityandqualityofdata
OnceyouhavecompiledyourwishlistofindicatorsitistimetodeterminetheaccessibilityandqualityofthedataTopreventcomplicationyoumaywishtoprioritizerelevantindicatorsforwhichyouarealreadycollectingdataAvoidthetemptationofadoptingatechnologysolutiontocollectnewdataatthispointsuchasaphotosurveytoolorotherappFornowyouaretryingtodeterminewhichdatasetsarerelevantaccessibleandhighqualityBesuretoworkwithyourkeyinternalpartnerswhileyouareidentifyingyourmeasuresDepartmentstaffwhoworkonprogramsarefamiliarwiththeindicatorsthatwillaccuratelysupporttheoutcomesanddatabaseadministratorswillbevaluablewhendiscussingaccessibilityandqualityofthedata
Abrieflistofconsiderationsfortheaccessibilityofyourdataisbelow
ArewecurrentlycollectingthisdatasetorwillwebecollectingitforthefirsttimeHowfarbackdoestherecordgo(timehorizon)WouldweneednewtechnologytocollectoranalyzethedataCanthedatabecollectedmanuallyorcanitbeautomatedHowfrequentlyisthedataproduced(continuouslyhourlydailyweeklymonthlyquarterlyyearlyetc)IsthedataavailablefortheentireCitypopulationorasubsetofthepopulationHowgranularisthedata(individuallevelCensusblockneighborhoodcitywideetc)DoesthedatacontainsensitiveinformationIsthereaproxymeasurethatwecanusefornowIsthedatamachinereadable
AsnotedbytheWilsonSheehanLabforEconomicOpportunities(LEO)attheUniversityofNotreDameleveragingadministrativedatacansignificantlyreducethecostsandtimelinesofrigorousevaluationCitiescanestablishaninternaldatasharingagreementacrossdepartmentsincludinglanguagethatenablesadministrativedatatobeusedforresearchpurposesCitiescanalsoidentifyexternaladministrativedata-holderssuchasacountyschooldistrictpolicedepartmenthospitaloranotherstatethatmayhaveoutcomedataimportantformeasuringprogramimpactCitiescanleverageexistingrelationshipsandorworkwithresearchinstitutionstodevelopstreamlineddatasharingprocessesandagreementswiththeseexternalentities
Assessdataaccessibilityqualityandsecurity
40
Evaluatethequalityofyourdata
QualityisasimportantasaccessibilityToevaluatedataqualityyoumaywishtocreatearubricliketheonebelow
Value ValueDescription
3 ExcellentDataQualityThedataiscompleteaccurateandupdatedregularly
2 AdequateDataQualityThedataissomewhatcompleteaccurateandupdatedfairlyregularly
1 PoorDataQualityThedataisnotcompleteaccurateorupdatedregularly
ThisisagreattimetobuildonthedatainventorythatyoustartedinthegatherinformationpageSuggestedadditionalfieldstoaddtoyourdatainventorytemplatearebelow
Program DatasetTitle
DatasetDescription DataSource
Departmentorexternalpartner
DoestheCityalreadycollectthisdata
Whatisthenameoftheprogram
Human-readablenameofthedataset
Shouldbeunderstandablebynon-technicalusers
Whatistheoriginalsystemapplicationfilethathousesit
Departmentorexternalpartnerthatmaintainsthedata
Isthisdataalreadycollected
Updatefrequency-ThefileformatforthedatasetTypicallycsvxlsxshapefiletxtetc
Format-Howoftenthedatasetisupdated
Timerange-Theperiodoftimecoveredinthisparticulardataset
Datause-Howisthisdatasetcommonlyused
Datausers-Whotypicallyusesthisdataset
Dataownerstitle-Titleofthedataowner
Dataownersposition-DataOwnersjobtitle
Dataownersemailaddress-DataOwnersemailaddressWhomanagesthedataandorisresponsibleforgrantingpermissiontoaccessthedataWhounderstandswhatthedatasetincludesandcananswerquestionsaboutit
Dataqualityconcerns-Describetheconcernsyoumayhaveaboutthequalityofthedataset
Assessdataaccessibilityqualityandsecurity
41
Division-DivisionwherethedataismaintainedIfmorethanonedivisionisresponsiblelisttheprimaryownerofthedataset
Sensitivedatacomments-DescribetheissuesthatwouldpreventthecityfromsharingthedatapubliclyCommonsensitivityconcernsincludeprivacyviolationssecurityissuesorhighcostorstaffdemandstopostthedata
Publishingstatus-Thistheplacetoputanycommentsregardingthecurrentstatusofthedataset
Publishedlink-Thisisthelinktothepublisheddatasetontheopendataportalorcitywebsite
Reviewdatasecurityandprivacyprotectionprotocols
OftenapolicychangealsoinvolvesachangeinthedatathatisbeingcollectedhowthatdataisbeinghandledwhohasaccesstoitandahostofotheradjustmentsAsyoutakestockofthedataassetsthatareavailableforyourinitiativeitisvitaltoalsothinkthroughwhatwillbedonetoprotectthatdataandtheprivacyofthepeopleitrepresentsAbigpartofgooddatasecurityinvolvesstrongdatagovernancepracticesthatdefineandidentifysensitivedataandconsciouslyworktobalancethevalueandriskofstoringsuchsensitivedataOnewayofevaluatingyourdatacomesfromcaselawrelatedtotheUSFreedomofInformationActwhichwehavetranslatedtothechecklistbelow
1 Woulddisclosureresultinasubstantialinvasionofprivacy2 Haveweconsideredtheextentorvalueofthepublicinterestandthepurposeorobject
oftheindividualsseekingdisclosure3 Istheinformationavailablefromothersources4 Wastheinformationgivenwithanexpectationofconfidentiality5 Isitpossibletomouldreliefsoastolimittheinvasionofindividualprivacy
TheseriskscouldbemitigatedusingavarietyofstrategiesfromaggregatingdatatoobfuscatingindividualstoredactingfieldsthatmayincludePersonallyIdentifiableInformation(PII)orre-evaluatingyourdatacollectionmethodstofindalternativewaysofgettingtheinformationyouneedThesestepsshouldbecodifiedandmanagedbyadatagovernancecommitteeifpossibleForthesensitivedatathatisdeemedessentialcreateaplanaroundpotentialdatabreachesinordertoquicklyandconfidentlyrespondtoemergencysituations
ProactivelypreparingforthesesituationsisoftenthebeststrategyfordatasecuritybutdonotforgetthetechnicalsideInvolveyourITadministratorsinconversationsaboutdatastorageandaccessensuringthattheywillbeprotectedbyenterprise-gradecybersecuritysystems
Assessdataaccessibilityqualityandsecurity
42
Assessdataaccessibilityqualityandsecurity
43
Designimplementationandevaluation
Designtheimplementation
NowthatyouhaveyouraccountabilityplanandkeyindicatorsitistimetogetspecificaboutyourplanforimplementationThetemplatebelowwillhelpyouthinkthroughthespecificmechanismsofimplementingpolicychange
UnderstandingorganizationsandsystemsiscriticaltounderstandingandmakingdecisionsaboutwhereandhowinterventionsshouldbeappliedApolicyimplementedinasinglesettingmayshowpromisingresultsbutimplementingmultiplepoliciesandprogramsatthesametimewithdifferentactorsresponsibleforeachaspectcanhelpensuresustainablebehaviorchange
InSomervilleMAShapeUpSomerville(SUS)isacitywidepublichealthprogramthatisimplementedacrossmultiplelevelsofinfluenceandsystemsthroughinterventionsbeforeduringandafterschooltoincludechangestobuiltenvironmentsuchasbikelanesregionalmasstransportationandphysicaleducationequipmentinschoolsandgymstoincreasephysicalactivitywellnesspoliciesandschoolgardensnutritioneducationduringandafterschoolandhealthyrestaurantprogramstosupporthealthyfoodandhealthyeatinghabits
Usethetemplatebelowtoplanyourimplementation
Policyimplementationplanningtemplate
ListthespecificdesiredoutcomesInthepreviousstepyouworkedwithdepartmentstafftoestablishspecificmeasurableoutcomesandinterimindicatorsPrioritizetheindicatorsthatwilldemonstrateprogressUsethespacebelowtocapturethedesiredoutcomesofthepolicychange
ExampleShapeupSomervillebeganwiththegoalofpreventingobesityin1stthrough3rdgraders
Yourturn
____________________________________________________________________________
____________________________________________________________________________
Designimplementationandevaluation
44
SpecifythetargetpopulationAreyoudesigninganinterventionthattargetsonlycertainsitesAreyouplanningonimplementingaprogramorinitiativeforaspecifictargetaudienceClearlyspecifythepopulationyouplantotarget
IfyouplantoconductarigorousimpactevaluationasnotedbytheWilsonSheehanLabforEconomicOpportunitiesitisimportanttoestablishconcreteobjectiveeligibilitycriteriaandensurethatthesecriteriaareappliedconsistentlybyallstaffduringscreeningandintakeItisalsovitaltoidentifytheindividualswhoareeligibleforaparticularprogramorserviceevenifeveryoneeligiblecannotbeservedbyexistingresourcesThishelpscitiesidentifyanyunmetneedsandguidefutureresourceallocationwhilealsocreatingthepossibilityofarigorousimpactevaluationbyidentifyingapotentialcomparisongroupFinallywhenconductingscreeningandintakeitisimperativetocollectandmaintainbasicinformation(egnamedateofbirthgender)aboutallprogram-eligibleindividualsnotjustthosewhoparticipateintheprogram
ExampleIntheShapeUpSomervilleexampletheprogrambeganbytargeting1stthrough3rdgraders
Yourturn
____________________________________________________________________________
____________________________________________________________________________
DefinethelocationWherewillthepolicychangetakeplaceWillimplementationtakeplaceatonesiteormultiplesites
ExampleShapeUpSomervilleinterventionsweredeliveredinschools
Yourturn
____________________________________________________________________________
____________________________________________________________________________
ClarifyimplementationpartnersandarticulatetheirrolesWhatarethekeyorganizationsinstitutionsorpeopleresponsibleforimplementationTheprojectteamshouldbeasclearandspecificaspossibleinprovidingadescriptionoftheconcreteactivitiesthatimplementationpartnersaremeanttocarryoutPracticeprofilesmaybeahelpfultooltohelpdefinewhatisrequiredoftheactorswhoareresponsibleforimplementingtheintervention
Designimplementationandevaluation
45
ExampleInSomervillemayoralleadershipandengagementofmultiplecityagencies(publichealthprimarycareschoolsCambridgeHealthAllianceetc)streamlinedandconnectedthevariousShapeUpSomervilleinitiativesandgrantswhichhelpedtokeepthetalentandenergyfocusedonhealthyeatingandactivelivingforallsectorsofSomervilleSUSpartnersultimatelyincludedschoolsnonprofitsCityandstateagenciesandnationalfoundations
Yourturn
____________________________________________________________________________
____________________________________________________________________________
DescribetheinterventionWhatarethestepsactivitiesorprocessesRememberthatthemechanismthespecificstepsorprocessthatleadtobehaviorchangeiskeytounderstandingtheeffectivenessofthepolicychangeandsharingtheresultswithothers
ExampleGoalsofShapeUpSomervilleweretoinfluenceeverypartofanearlyelementaryschoolchildrsquosdayndashbeforeduringandafter-schooltopromotehealthyeatingphysicalactivityandtheirweightstatusInterventionincludedfoodserviceenhancementstoimprovebreakfastandlunchandstaffprofessionaldevelopmentwalktoschoolactivitiesSUSclassroomcurriculumandprofessionaldevelopmentschoolwellnesspolicydevelopmentSUSafter-schoolcurriculumandprofessionaldevelopmentoutreachandeducationtohomethroughmaterialsforumseventsldquoSUSapprovedrdquorestaurantscommunityoutreachandcapacitybuildingthroughpolicydevelopmenttrainingsmediaplacements
Yourturn
____________________________________________________________________________
____________________________________________________________________________
ClarifytimelineandmilestonesfordeliveryandevaluationHowfrequentlywilltheservicesinterventionbedeliveredHowlongwilltheprogramrunWhatisthetimetoexpectedimpacts(eg12monthsafterprogramcompletion)
Designimplementationandevaluation
46
ExampleShapeUpSomervilleledbyTuftsUniversityFriedmanSchoolofNutritionScienceandPolicyconductedaquasi-experimental(nonrandomized)interventionstudyamong1stthrough3rdgradestudentsinSomerville(intervention)andtwocomparisoncommunitiesfor9-monthsduringoneschoolyear(fall2002ndashspring2003)ThestudyshowedpositiveassociationoftheinterventiononchangeinBMISUSreleasedreportssummarizingevaluationsprogresstodateplansforadditionalprojectsandimpactstenandfifteenyearsintotheinitiative
Yourturn
____________________________________________________________________________
____________________________________________________________________________
PlanyouranalysisTheprevioussectionaddressedestablishinganaccountabilityframeworktomonitorprogressandimplementationDeterminewhichindicatorswillbediscussthroughthatprocesswhowillsharethedataandanalysisandhowfeedbackfromthoseconversationswillbeincorporatedbytheprojectteam
ExampleTheMayorrsquosofficeimplementedadata-drivendecisionmakingapproachcalledSomerStatwhichwassynergisticwiththeapproachestakenbytheresearchandevaluationteamsdeployedforthespecificgrants
Yourturn
____________________________________________________________________________
____________________________________________________________________________
PlanforsustainabilityIfthepolicyandprogramchangeissuccessfulhowwilltheprojectteamsustainandexpandtoadditionalsitesorpopulations
ExampleAlthoughmostofthepartnershipsforgedintheearlyyearsweremaintainednewgrantsandcollaborationsexpandedattentiontoolderpublicschoolstudentsactivetransportationandtheGreenLinesubwayextensionthebuiltenvironmentfarmerrsquosmarketeffortssuchasldquoGrowninSomervillerdquowithUnionMainStreeteconomicdevelopmentandhealthdisparitiesaswellasdatacollectionmonitoringandsurveillance
Yourturn
Designimplementationandevaluation
47
____________________________________________________________________________
____________________________________________________________________________
Pilotplanningtemplate
IfyouplantoconductapilotfirstdeterminethepurposeofthepilotDoyouwanttoobservewhetherthepolicychangeresultsinthedesiredbehaviorchangeAreyouconcernedaboutfeasibilityscalabilityorcostDoyouwanttotestwhetheryourdesiredresultscanbeachievedthroughthispolicyapproachClarifythegoalofthepilotandthequestionsyouhopetoanswerThiswillhelpyouidentifythekeyindicatorsthatyouwillusetomeasurewhetherthepilotissuccessfulBeforebeginningestablishabaselineofcurrentperformanceandsetyourtargetsaccordinglybasedonwhatyouhopetoaccomplishthroughthepilot
NextestablishthepilotplanforyourprojectteamincludingmilestonesforteamcheckinstodeterminewhethertomoveaheadorchangecourseThisisseparatefromthemeetingsthatwilloccuraspartoftheaccountabilityframeworkUsethespacebelowtocreateyourtimelineanddevelopyourmeetingschedule
Task Personresponsible Startdate Checkindate Deliverable
Planyourevaluation
WhileyouaredesigningtheimplementationitisalsotimetobegintodevelopyourplanforanalysisAsnotedbytheWilsonSheehanLabforEconomicOpportunitiesdecisionsaboutprogramsetup-includingoutreachscreeningintakedatatrackingandenrollment-candeterminewhetherornotarigorousimpactevaluationwillbepossibleAsdiscussedcityleadersmayfindithelpfultoincludeevaluationpartnersinearlyconversationsaboutprogramsetupDuringpreviousstagesyoumayhaveworkedwitharesearchertoplanyourproposedpolicychangeYoushouldbeclearaboutthepurposeoftheevaluationanddevelopanevaluationplanBelowareadditionalconsiderations
Causationvstrendanalysis
IntheEstablishaframeworktomonitorprogressstagetherewasguidanceontrackingtrendsovertimeItisimportanttonoteherethatmonitoringdataisdifferentfromanalyzingacausaleffectTrendanalysiswillillustratethatchangesareoccurringandmayleadtoquestionsthatcouldbeansweredbyyourevaluationbuttrendanalysisalonedoesnotprovideevidenceaboutwhatisdrivingthosechangesYoumayseechangesinyour
Designimplementationandevaluation
48
outcomeindicatorsbasedonvariablesoutsideoftheCityscontrolForinstanceachangeinfederaltradepolicycanhaveanimpactemploymentratesinyourCityoutsideofyourCitysnewjobtrainingandplacementprogram
ToaddresscausalityyoumustcomparetheoutcomesofindividualswhoreceivedtheinterventionwiththosewhodidnotWithoutacomparisongroupofsimilarindividualswhowerenotexposedtotheinterventionitisimpossibletoknowwhichfactorsareresponsiblefortheoutcomesRandomizedControlTrials(RCTs)inwhichindividualsarerandomlyassignedtotreatmentandcontrolgroupsarethegoldstandardinprovingcausalityIfyoucandemonstratethatoutcomesimprovedforthosewhoreceivedtheinterventionanddidnotimproveatallorimprovedlessforacomparisongroupyoucanbesomewhatcertainthattheinterventionwasthecause
Examplesofothernon-causalimpactevaluationsthatarerigorousandreliableincludecertainquasi-experimentalstudiesusingcomparisongroupdesignssmartinterruptedtimeseriesanalysesthatmodeltrendsovertimeinwaysthatallowestimationofinterventioneffectswelldesignedpre-posttestanalysesneedsassessmentsimplementationstudiesetcFormoreinformationondifferenttypesofevaluationsseetheSageResearchMethodswebsite
Lookfordifferencesingroups
BuildintimetoalsoexaminethepilotdatatoseeifgroupsareexperiencingdifferentresultsReviewingperformancebasedonoutcomemeasuresforapopulationinaggregatemayhidethesubtlewaysthatsomepeoplemaybeaffecteddifferentlysoitisnecessarytocarefullyanalyzethedatafordisparateoutcomesForinstancetheMovingtoOpportunity(MTO)studythatexaminedtheimpactofhousingvouchersonfamilieswholivedinhigh-povertyneighborhoodsfoundthatchildrenwhomovedbeforetheageof13hadbettereducationandhigherincomesasadultsHoweveradultsdidnotseeimprovededucationandemploymentoutcomes
Disaggregatingdataintomorespecificcategoriesofraceandethnicitycanhelpunderstandtrendsanddisparitiesinamoregranularwaywhichcanhelpdecisionmakerstargetpoliciesandinterventionsinamoreequitableway
EarlierintheRoadmapweworkedtoidentifystakeholdersandwhomightbepositivelyandnegativelyimpactedbyachangeinpolicyUsetheresultsofthatexercisetoinvolvethecommunityininterpretingdataanalysisandtogatherqualitativefeedbackfromthecommunitytocallattentiontoissuesForinstanceanincreaseinyouthdetentionmaybeexplainedbyinefficienciesorlackofaccesstotransportation
Designimplementationandevaluation
49
Designimplementationandevaluation
50
Shareyourstoryandsuccess
Sharetheresults
Storytelling
AnintegralstepinyourstrategyissharingtheresultsofyourpilotorpolicychangewithyourcommunityThisincludesdevelopingmessagingaroundtheworkofyourorganizationandcreatingaplantodisseminateyourstoriesWhytellstoriesaboutyourwork
Informandorunderstandasubjectorsituation
Giveneededcontextofdataorbitsofinformation
Relatablestoriesarememorableforreadersviewers
ReadersorviewerswillseetheinformationasmorecompellingwhenthereisademonstratedandrelatablechangeorimpactonthemselvesortheirfriendsneighborsandcolleaguesStoriesareaboutpeopleandgovernmentisfundamentallyabouthelpingpeopleSotellthestoriesofthepeopleyouandyourcitycolleagueshelped
Keepthesetipsinmindasyoudevelopyourstorywithyourteam
WhatwereyourfindingsandwhyshouldanyonecareTellyouraudiencehowtheproblemwassolvedWhathastheimpactbeenonresidentsandcitystafftodateWhatwasthesuccessandweretheremultiplesuccessesalongthewayIfyouhavepreliminaryresultsordatashowingimprovementtosharethisisthetimetodoit
WhowasinvolvedandimpactedReviewthelistofstakeholdersHowwereeachofthemimpactedbytheprogramHumanizeyourstoryandimpactbyaddingadditionalvoicesThesevoicescanbeoneofthefollowing
Apersonorfamilywhowaspositivelyaffectedbythechangesuccess
Aninvolvedthird-partydescribingboththeproblemandthesolution(ieanorganizationnonprofitprovidersbusinesstradeassociationsportsteametc)whoispartofthecommunitythatthegovernmentworkedwithonthechangesuccess
Someonefromthegovernmentitselfwhocanspeakaboutthechangesuccess
TellmoreaboutyourcityDescribeyourcityincludingpopulationdemographicsindustriesetcWhatareitschallengesWhatisitknownforWhatmakesituniqueHowisitsimilarordifferentfromothercitiesintheUS
Shareyourstoryandsuccess
51
DescribeyourcityrsquoschallengeWriteoutthechallengeyourcityisfacing(asdefinedinyourproblemstatement)HowdoesitaffectresidentsHowdoesthisproblemaffecttheworkofcitystaffmembers
AddthedetailsProvidedetailsonyourappliedresearchstrategyTalkaboutwhatyourcityhasdonenthepasttoresolvetheissueDiscusshowyourstrategyhaschangedwhenyoubegantoadoptthisapproachWasmoneysavedorinvested
Usethisstorymapexercisetoorganizeyourthoughtsandguaranteethatyouincludealloftheelementsofatraditionalstory
AfteryoudevelopyourmessagingcreateacommunicationsmarketingandoutreachstrategydesignedtospreadyourmessageYourstrategyshouldwalkyouthroughthefollowingkeyactivities
1 IdentifyandprofileyouraudienceWhencreatinganddisseminatingyourmessageitisalwaysgoodtothinkaboutyouraudienceWhoexactlywouldbeinterestedinwhatyouhavetosayThegeneralpopulationofthecityPeoplewhoworkinnonprofitsBusinessownersItmattersbecausenoteveryonewholivesinacitymaybethetargetoftheinformationBusinessownersmaybemoreinterestedindatashowingeffectsonthebusinessdistrictthanothersandsoonWewillcoverthislaterbutthisstepwillalsohelpyoutothinkofthemosteffectivewaytoreachyouraudience
2 Defineyourcalltoaction(ifyouhaveone)SpendsometimeconsideringwhyyouareinterestedintellingyourstoryDoyousimplywanttoupdatethepubliconyourprogressorcelebrateyoursuccessesAreyouinterestedinyouraudiencechangingitsbehaviororsupportingaspecificinitiativebeinglaunchedbyyourorganizationIfyouwanttohaveyouraudienceactorbehaveinacertainwaybespecificandincorporateitinyourmessaging
Shareyourstoryandsuccess
52
3 SelectperformancemeasuresThinkabouthowyouwoulddetermineifyourcampaignissuccessfulandselectappropriateperformancemeasuresYouwillrevisitthesemeasuresattheendofyourcampaigncommunicationsplanimplementationtodeterminewhetheryouneedtomodifyyourcommunicationsandoutreachstrategyforthefuture
4 MethodstocommunicateyourstoryTherearemanywaystocommunicatesuccessofyourprojectHerearesomesuggestionsonreachingaudiencesItisimportanttodeterminewhichmethodsarethemosteffectiveforyourdesiredaudiencetoreceiveinformationDoyourresearch(PewResearchCenterhasexcellentstudiesonthereachofmediadigitalmediaandmarketing)ForexampleifseniorsareyourtargetpopulationandtheyarelessinclinedtouseInstagram(thatistrue)youmaywanttoconsiderFacebookorthenightlylocalnewsbothofwhichareusedmoreoftenbyseniorsBelowisalistofsomemethodsandtoolsyoucanuse
i Pressreleasestolocalnewspapersandtvstations
ii ShortsmartphonevideosAgreatwaytoengageanaudienceisthroughvideoAskonequestionofsomeonewhoispartofthestoryMaybethereisasocialworkerwhocandiscusshungerinyourcityThesevideoscanbereusedonmultipleplatforms
iii Long-formwriting
iv TellingindividualstoriesthroughaseriesofphotosandshortnarrativesforpublicationonFacebookorInstagram
v Op-edsopinionpiecesplacedinthelocalpaper
vi Emailupdatestoyourlist
vii Connectingwithappropriateorganizationsthatareinvestedinthetopicandworkwiththecommunityandaskthemtodisseminateinformationtotheirmembers
viii EmployingldquoWordofMouthrdquo(WOM)tacticsbycreatingtownhallsorfreeeventsforresidentstoattendintheirownneighborhoods
5 DeterminethetypeofcollateralUsuallycampaignsthattargetresidentsrequireseveralavenuestoreachthemPeoplearenotalwaysinthesameplaceorspaceHavingakeenunderstandingofyouraudiencewillassistyouwithyourchoicesonwhatneedstobedevelopedSometimesthatmeansimagesforsocialmediaaonepageflierthatgivespeopleinformationonattendinganeventoranin-depthbrochure
Shareyourstoryandsuccess
53
6 ImplementyourstrategyKeepinmindstrategiescanbefluidSometimessomethingmaynotturnoutasexpectedevenifduringplanningsessionsitwasldquosupposedtordquoItisokYoucananalyzewhathappenedduringadebriefsessionandonceyoursurveysandothermeasurementshavebeenfinalized
Visualizations
Withtherightvisualizationyoucanreviewandshareinformationinsightsanalysisperformanceandtrendsallofwhichisvaluabletoacross-sectionofthepublicButwhichvisualizationisrightforyouTochoosetherighttypeofchartgraphdeterminethetypeofquestionyouwanttoanswerSomecommoncharttypesarebelowandmoreinformationcanbefoundhere
Exploreadditionalanalyticaltechniques
Duringthegoal-settingphaseyouestablishedoutputoutcomeandimplementationmeasuresBasedontheresultsofyourpilotyoumayneedtoadjustyourmeasuresWiththepilotdatainhandyoumayhavesomeideasofhowfutureanalysescouldbedifferentandpotentiallyimprovethequalityofyourprogramTakesometimetobrainstormotherrelevantdatasetsthatyoudidnothavethetimetopursueearlierincludingmessydepartmentaldatathatneedsmoreattentionorsourcesofpublicdatafromthestateandfederalgovernmentnonprofitsandevenprivatecompaniesthatcouldsupplementyourunderstandingofanissueUsingmachinelearningtechniquesyoumightbeabletobetter
Shareyourstoryandsuccess
54
predictdemandforhomelessservicesusingNOAAweatherforecastspreemptwatermainbreaksusingsoilstratigraphyorinvestineconomicdevelopmentbymonitoringrealestatelistingdata
Advancedanalyticaltechniquescanalsohelpyourimplementationrunmoreeffectivelybyintegratingdata-informedtargetinganddecisiontoolsintotheprocessByautomatingroutineworkflowsandsurfacingsubtletrendsthesetoolscanhelpyouiterateinamoreprecisemannerandprovidekeystrategicguidancetofront-lineemployeesthatwouldotherwisebeunavailableThesesortsofdata-science-enhancedprocessescanbeclassifiedasoneofsixprojecttypesasexploredfurtherbytheNewOrleansLANOLAlyticsteamFormoreguidanceonscopingasimilarprojecttheGovExadvancedanalyticsscopingguidegivesyouastep-by-stepoverviewofhowtobesuccessful
Usethespacebelowtolistyourideasforpossibleadvancedanalyticsprojectsbytype
Typeofproject Ideas
Findingtheneedleinthehaystack
Prioritizingforimpact
Earlywarningtools
Betterquickerdecisions
Optimizingresourceallocation
Experimentingforwhatworks
Keepthefeedbackloopgoing
GatherinformationonpublicsentimentaboutpolicychangethroughfocusgroupssurveysandpublicmeetingsledbyadvisorygroupmembersNotewhethernewinformationisavailablethatcanhelpanalyzeorcontextualizeyourprogress(egfederalorstatedatareports)Sharetheresultsofyourpilotandthepolicychangeprocesswithothercitiesandresearchers
Shareyourstoryandsuccess
55
WorkingwitharangeofevidencePresentlythereisalotofattentionandinvestmentinevidence-basedpolicymakingincludingthefederalbipartisanCommissiononEvidence-BasedPolicymakingthePewCharitableTrustsreportsandtoolkitsonHowStatesEngageinEvidence-BasedPolicymakingandtheLauraandJohnArnoldFoundationsupportedEvidence-BasedPolicymakingCollaborativeManyoftheseinitiativesfocusonrigorousevaluation--meaningrandomizedcontrolledtrials(RCT)orotherexperimentalmodels--asthebestkindofevidencetouseHoweverthisbodyofevidenceisofteninsufficientforthespecifickindsofquestionsthatgovernmentscanhavewhenmakingadecisionaboutpolicyInthisguideweproposeanapproachtosupportgovernmentsinincorporatingadditionaltypesofevidenceintotheirdecisionmaking
OnewaytounderstandarangeofevidenceistouseNestarsquosStandardsofEvidencetoframeanddefineevidenceTheBritishnonprofitNestaworkstoinjectpositivechangeintothewayUKgovernmentpoliciesandprogramsfunctionOneoftheirkeyareasofworkhasbeendevelopingWhatWorksCentresorganizationsthatcollectcommissionsynthesizeanddisseminateevidenceaboutinterventioneffectivenessacrossarangeofsocialimpactareasNestadevelopedthefollowingStandardsofEvidencetoprovideausefulwaytodocumenthowdifferentkindsofinformationdemonstratedifferentlevelsofevidence
Workingwitharangeofevidence
56
Theimportanceofpotentiallybiasedinformation
Onechallengepresentedbyanevidenceclassificationthatacknowledgesthevalueofnon-scientificformsofevidenceistofigureouthowtobalanceallofthepiecestogetherwhenyouusethemTounderstandhowtodothiswehavetothinkabouthowtopositionpotentiallybiasedinformationindecisionmaking
WhenwelookatinformationwhichhasnotbeenrigorouslyevaluatedweacceptthatthisinformationmightbebiasedThewholereasonweconductrigorousevaluationistoremovebiasfromourobservationsWeconductstructuredtrialstoproduceresultsthatareconsistentandgeneralizableIfafindingisunbiasedthisaccuracymdashthislackofbiasmdashcanbeprovedbecauseanyindependenttestercanreproduceitwhentheyfollowanidenticalprocess
YoucanlookatpotentiallybiasedinformationasjustinformationthatisearlieroninitspathtorefinementreplicationandverifiedaccuracyApieceofinformationbasedonanyoneindividualrsquosobservationsislikelytobebiasedbecauseitiscomingfromjustonesourceBiasisreducedbygatheringtogetherlotsofindependentsourcesofinformationandfiguringoutitsmostcommonattributes
MaybethebiasedinformationmakesaclaimthatistoolargemdashitsayssomethingistrueunderallconditionswhenitisreallyonlytrueundercertainconditionsorthataneffectofaninterventionisbiggerthanitstrueaverageorthataneffectlastsovertimewhenitdoesnotPerhapschangingtheclaimtobemorespecificmdashcriticallyreviewingtheinformationtoseeifspecificconditionsmustbemettoachievetheoutcomesmdashwouldmaketheclaimmoreaccurate
IfyoutestedyourclaimfounditdidnotholdupinsomecaseschangedittesteditagainuntilyourclaimrepeatedlyheldupthenyoucoulddemonstratethattheclaimwasnolongerbiasedbutaccurateThisprocessoftestingimprovingtestingandvalidatingiswhatwegetfromstructuredscientificstudy
Withindividualpiecesofinformation--crediblehypothesescasestudiesexpertopinion--youcannotassumethatkindofaccuracyAtthesametimetheseearlyobservationsarefoundationaltoanyreplicablescientificobservationsWewillnevergetanywhereifwedonotstartwiththesepiecesofinformationeveniftheymightbebiased
Combininglevelsofevidencetogetthemostdetailedinformationforyourcity
ItisespeciallyimportanttoacceptdifferentformsofevidencewhenyoufacequestionsthathavenotyetbeentestedbyindependentstudiesSomeofthemostimportantquestionsrelatetothingsthatareveryspecifictoyoursituationYoufaceaparticularcombinationof
Workingwitharangeofevidence
57
issuesactorstrendsandresourcesthatisultimatelydifferentfromeveryotherplaceintheworld
ThisisimportantbecauseaspolicypractitionersandnotscientistsyoucaremoreaboutyourspecificcasethanahypotheticalaveragegeneralcaseYoucarelessaboutthetotallikelyrangeofimpactsfromapolicythanhowthissingleimplementationwillworkYouwantinformationaboutpoliciesthatwillprovideyouwiththebestpossiblepredictionabouthowtheywillworkinyourcontext
ByputtingdifferentkindsofevidenceintothesameframeworkNestarsquosStandardsofEvidenceimplythatwhenyoucannotgetrigorouslytestedexperimentallyverifiedinformationitisappropriatetoworkwithwhatwedohaveThisisrelevantforthemunicipalimplementationcontextbecausenostudypulledfromaliteraturereviewwillaccountforallofthevariablesyoufaceinyourowncontext
Intheabsenceofhavingastudythatpredictshowaninterventionworksinyourspecificcontextyouwillneednon-scientificinformationtotrytofillinthosegapsYouwillneedtogatherstoriesfromresidentsstaffmembersandotherstakeholdersthatsitonLevels1or2ofthestandardsofevidence
Someofthekindsofinformationyouwillprobablygetmoreeasilyfrompeoplethanstudiesinclude
howpeopleincitieslikeyoursreactedtoaninterventioninpublicforumshowpeoplewhoimplementedaninterventioninacitylikeyoursfiguredouthowtogetitpassedhowthecitycommunicatedpubliclyabouttheapparentbenefitsanddrawbacksoftheinterventionthepoliticalacceptabilityoftheintervention
Youcanremainsimultaneouslyawarethattheinformationyouaregatheringthroughindividualstoriesisbiased--notgeneralizableinformedbyspecificperspectives--whilealsorecognizingthathavingsomeinformationaboutsomethingimportantisbetterthannoinformationEvidencefromthefirsttwoevidencelevelsareessentialbuildingblocksofknowledgeandperfectlyusefulwhenyoucannotgetdetailedunbiasedknowledge
Youhavetostartsomewhereandyoushouldalwaysstartwithevidenceyouhaveaboutyourveryspecificconditionsthatyouknowareimportantforsuccess
Workingwitharangeofevidence
58
Thankyou
Interviewswithcities
GovExheldsemi-structuredinterviewswithsixstaffmembersinfourcitiestoestablishtheinitialstepsoftheRoadmapWeaskedrespondentstotalkaboutaninitiativeorpolicythattheyhavebeenresponsibleforimplementingintheircityRespondentsfromthreecitiesspokeabouttheireffortstoaddresshomelessnessandrespondentsfromonecityspokeaboutblightAfterrespondentsidentifiedtheproblemtheyspokeabouthowthecityprioritizedthisissuethevariousstepsinvolvedandbarrierspresent
GovExconductedtextanalysisoftheseinterviewswhichrevealedthatpeoplewasthemostcommonlyusedtermduringtheseconversationswhendiscussingthestepstochangeaprogramorpolicyThisisathemethroughoutthisRoadmapThefirstsectionbeginswithidentifyingtherelevantpeopleTheconversationsaroundevidencearefocusedonhumanbehaviorratherthanmethodologyorstrengthofevidence
Thankyou
59
OncewehadadraftoftheRoadmapwefollowedupwithcitiesonanindividualbasistolearnmoreaboutwhatwouldbehelpfulWearegratefultothefollowingcitystafffortheinterviewsandindividualconversations
MichaelBaskinChiefPolicyOfficerChattanoogaTN
MelissaBridgesPerformanceandInnovationCoordinatorLittleRockAR
VictoriaCarreoacutenAdministrativeOfficerLasVegasNV
TyroneGrandstrandCommunityDevelopmentPlannerFargoND
TibLaughlinDirectorofGeneralServicesKansasCityKS
TimMorelandDirectorofPerformanceManagementandOpenDataChattanoogaTN
CharlesRosendahlManagementAnalystDentonTX
Thankyou
60
MelissaSiebenAssistantCountyAdministratorKansasCityKS
GinaThompsonDirectorofStrategicServicesLewisvilleTX
JHUFacultycollaboration
GovExalsoconvenedthecitycontactsandfacultypartnersinaseriesofAdvisoryGroupmeetingsThesemeetingsservedtoconnectcitiesandfacultytounderstandoneanothersperspective
StefanBaralBloombergSchoolofPublicHealth
AshleyBernerJHUSchoolofEducation
MelissaDavey-RothwellBloombergSchoolofPublicHealth
JonLinksBloombergSchoolofPublicHealth
TamarMendelsonBloombergSchoolofPublicHealth
KristinMmariBloombergSchoolofPublicHealth
KeshiaPollackPorterBloombergSchoolofPublicHealth
JoshSharfsteinBloombergSchoolofPublicHealth
LizStuartBloombergSchoolofPublicHealth
LindsayThompsonCareyBusinessSchool
DanielWebsterBloombergSchoolofPublicHealth
Partnersindataandevidence-basedpolicy
Wealsospokewithpartnerswhoareexpertsinthefieldofevidence-basedpolicywhosharedtheirknowledgeandresourcesofcurrentandfuturethinkinginevidence-informedthinking
WendyBarrenoAssociateDirectorWilsonSheehanLabforEconomicOpportunitiesUniversityofNotreDame
TiffanyDavenportDirectorofDataPartnershipsBaltimoresPromise
TomiHiersDirectorBaltimoreCivicSiteAnnieECaseyFoundation
ChrisKingsleySeniorAssociateAnnieECaseyFoundation
KathrynOliverAssociateProfessorLondonSchoolofHygieneandTropicalMedicine
Thankyou
61
KristenWatkinsSeniorPolicyAssociateJ-PAL
GovExParticipants
GovExAnalystMiriamMcKinneyanalyzedtheinterviewtranscriptsandconductedtextanalysisSeniorImplementationAdvisorEmilyShawwillintegrateRoadmapactivitieswithadditionalservicedeliveryactivitiesatGovEx
Thankyou
62
ReferencesAbrahamKGampHaskinsR(2017)Thepromiseofevidence-basedpolicymakingReportofthecommissiononevidence-basedpolicymakingWashingtonDCCommissiononEvidence-basedPolicymakingBigmove2Extendingthegreen(nd)Retrievedfromhttpswwwcityoflewisvillecomdoing-businesslewisville-2025big-move-2-extending-the-greenBatesMAampGlennersterR(2017)ThegeneralizabilitypuzzleRetrievedfromhttpsssirorgarticlesentrythe_generalizability_puzzleBuildwithnotfor(2018January)RetrievedfromhttpsbuildwithnotforwordpresscomCairneyPampOliverK(2017)Evidence-basedpolicymakingisnotlikeevidence-basedmedicinesohowfarshouldyougotobridgethedividebetweenevidenceandpolicyHealthResearchPolicyandSystemsdoi101186s12961-017-0192-xCairneyPOliverKampWellsteadA(2016)TobridgethedividebetweenevidenceandpolicyReduceambiguityasmuchasuncertaintyPublicAdministrationReview76(3)399-402Retrievedfromhttpdxdoiorg101111puar12555Centreforpublicimpact(nd)RetrievedfromhttpswwwcentreforpublicimpactorgCernaL(2013)ThenatureofpolicychangeandimplementationAreviewofdifferenttheoreticalapproachesOrganisationforEconomicCo-OperationandDevelopmentRetrievedfromhttpswwwoecdorgeduceriTheNatureofPolicyChangeandImplementationpdfColoradoSprings(nd)Retrievedfromhttpscoloradospringsgovcommunicationspagewhat-expect-when-you-speakupContinuumofcare(coc)program(nd)RetrievedfromhttpswwwhudexchangeinfoprogramscocCouncilpresidentkicksoff$12millionyouthfund(nd)Retrievedfromhttpwwwbaltimorecitycouncilcomcontentcouncil-president-kicks-12-million-youth-fundDurlakJAampDuPreEP(2008)ImplementationmattersAreviewofresearchontheinfluenceofimplementationonprogramoutcomesandthefactorsaffectingimplementationAmericanJournalofCommunityPsychology41327-350Retrievedfromhttpswwwncbinlmnihgovpubmed18322790Fishbonediagram(ishikawa)(nd)Retrievedfromhttpasqorglearn-about-qualitycause-analysis-toolsoverviewfishbonehtmlFivewhysGettingtotherootofaproblemquickly(nd)RetrievedfromhttpswwwmindtoolscompagesarticlenewTMC_5Whtm
References
63
FligsteinNampMcAdamD(2011)TowardageneraltheoryofstrategicactionfieldsSociologicalTheory29(1)1-26RetrievedfromhttpsociologyberkeleyedusitesdefaultfilesfacultyfligsteinFligstein20McAdam20Soc20Theory20PaperpdHeinkelP(2018January12)CitymullspermitsforbikesharingcompaniesRetrievedfromhttpwwwdentonrccomnewsdenton20180112city-mulls-permits-bike-sharing-companiesHowisareamedianincomecalculated(2012October)Retrievedfromhttpswwwhudexchangeinfofaqs843how-is-area-median-income-calculatedLampkinLMampHatryHP(2003)KeystepsinoutcomemanagementUrbanInstituteRetrievedfromhttpswwwurbanorgresearchpublicationkey-steps-outcome-managementMarshDampSmithM(2000)UnderstandingpolicynetworksTowardsadialecticalapproachPoliticalStudies484-21Retrievedfromhttpjournalssagepubcomdoiabs1011111467-924800247McCannL(nd)RetrievedfromhttplaurenellencomMoultonSampSandfortJR(2016)ThestrategicactionfieldframeworkforpolicyimplementationresearchPolicyStudiesJournal45(1)Retrievedfromhttponlinelibrarywileycomstore101111psj12147assetpsj12147pdfv=1ampt=jdkhhcp4amps=e184c625e1855a6b1615ccb99ee27b1c0b3212d8PaynterB(2018January1)What100citiesarelearningfromeachotherbysharingtheirdataRetrievedfromhttpswwwfastcompanycom40518004what-100-cities-are-learning-from-each-other-by-sharing-their-dataPeakacademy(nd)Retrievedfromhttpswwwdenvergovorgcontentdenvergovenmayors-officeprograms-initiativespeak-performancepeak-academyhtmlPinnockHBarwickMCarpenterCREldridgeSGrandesGGriffithsCJRycroft-MaloneJMeissnerPMurrayEPatelASheikhAampTaylorSJC(2017)Standardsforreportingimplementationstudies(stari)ExplanationandelaborationdocumentBMJOpenRetrievedfromhttpbmjopenbmjcomcontentbmjopen74e013318fullpdfProctorESilmereHRaghavanRHovmandPAaronsGBungerAGriffeyRampHensleyM(2011)OutcomesforimplementationresearchConceptualdistinctionsmeasurementchallengesandresearchagendaAdministrationandPolicyinMentalHealth38(2)65-76doi101007s10488-010-0319-7TandonSD(2018)ProgressincommunityhealthpartnershipsResearcheducationandactionRetrievedfromhttpswwwpressjhuedujournalsprogress-community-health-partnerships-research-education-and-actionProvanKGampKenisP(2007)ModesofnetworkgovernanceStructuremanagementandeffectivenessJournalofPublicAdministrationResearchandTheory
References
64
18(2)229-252Retrievedfromhttpsacademicoupcomjpartarticle182229935895SabatierPA(1988)Anadvocacycoalitionframeworkofpolicychangeandtheroleofpolicy-orientedlearningthereinPolicySciences21(3)129-168Retrievedfromhttpwwwjstororgstable4532139SandstromAampCarlssonL(2008)TheperformanceofpolicynetworksTherelationbetweennetworkstructureandnetworkperformancePolicyStudiesJournal36497-524Retrievedfromhttponlinelibrarywileycomdoi101111j1541-0072200800281xabstractSchwartzMampWilsonE(nd)WhocanaffordtoliveinahomeAlookatdatafromthe2006americancommunitysurveyRetrievedfromhttpswwwcensusgovhousingcensuspublicationswho-can-affordpdfShearerJCAbelsonAKouyateBLavisJNampWaltG(2016)WhydopolicieschangeInstitutionsinterestsideasandnetworksinthreecasesofpolicyreformHealthPolicyandPlanning31(9)1200-1211Retrievedfromhttpsdoiorg101093heapolczw052ThelessonsofadministrativedataHigh-profilepolicy-relevantresearchpoweredbyadministrativedata(nd)Retrievedfromhttpswwwpovertyactionlaborgsitesdefaultfilesresources20161114-The-Lessons-of-Administrative-DatapdfToolsandotherneedsforqualitativeandquantitativemethods(2015February)AgencyforHealthcareResearchandQualityRetrievedfromhttpwwwahrqgovprofessionalsquality-patient-safetytalkingqualityassesstoolshtml
WeaverRCampBagchi-SenS(2013)SpatialanalysisofurbandeclineThegeographyofblightAppliedGeography4061-70
References
65
UniversityPartners
Name Title Affiliation Area(s)ofexpertise
Contactinformation
Examplesofworkingwithnon-academicdecisionmakers
WendyPBarreno
AssociateDirector
WilsonSheehanLabforEconomicOpportunitiesUniversityofNotreDame
LEOsfocusareasincludehealtheducationhousingchildwelfareeconomicopportunitycriminaljustice
wbarrenonduduorleondedu
MelissaDavey-RothwellPhD
AssociateScientist
JohnsHopkinsBloombergSchoolofPublicHealth
Health mdavey1jhuedu410-614-5854
BaltimoreCityHealthDepartmentLeadersMarylandDepartmentofHealthAddictionRecoveryServicesinDCExecutiveDirectorsofcommunityorganizationsacrosstheUSthatfocusonHIVSTIsubstanceabuseandimplementationscience
ElizabethStuart
AssociateDeanforEducationandProfessorDepartmentsofMentalHealthBiostatisticsandHealthPolicyand
JohnsHopkinsBloombergSchoolofPublicHealth
Educationsocialpolicystatisticalmethods
estuartjhuedu
Resources
66
Management
Design-ThinkingResources
EngagementGuideforCivicTech
Journeymap
PerformanceManagementResources
DenverPeakAcademy
Proxymeasuresguide
Talkingaboutperformancewithoutcomplicatingit(video)
SettingPerformanceTargetsguide
Datavisualizationguide
Performancemanagementgettingstartedguide
Exercises
Fishboneexercise
5Whys
AdministrativeData
TheLessonsofAdministrativeData(JPAL)
EvaluationResources
Quantitativeandqualitativemethods(USDepartmentofHealthandHumanServicesAgencyforHealthcareResearchandQuality)
KeyStepsinOutcomeManagement(UrbanInstitute)
RCT-YES
Sageresearchmethods
BestPracticesforMixedMethodsResearchintheHealthSciences
ACCORDChecklist
Pilotandimplementationplanning
Benchmarkingexamplesandproxymeasures
Seattleopendataplaybook
Resources
67
Datasciencescopingguide
Resources
68
AboutGovExGovExhelpsgovernmentsusedataandevidencetoimproveresidentsrsquolivesWesupportgovernmentsinbuildingtheirinternalcapacitytooperatesustainableanalyticspracticeswhichimprovecommunityoutcomesinfieldsfrompublicsafetytoeconomicdevelopmentGovExhasaproventrackrecordofworkinginconcertwithpartnerstoprovidetechnicalassistanceandtrainingatscale
SinceourlaunchinApril2015GovExhas
Trainedover2000governmentemployeesinpersonandonlineintopicsincludingdatamanagementperformancemanagementopendatacommunityengagementandanalytics
Providedtechnicalassistancetomorethan100localgovernmentsacrosstheglobeimprovingtheiruseofdatatoachieveefficienciesinservicedeliveryandbetteroutcomesforresidents
Assessedthecapacityofmorethan130governmentalorganizationstodeterminehowleadersstaffandexternalstakeholdersmanageandusedata
Localgovernmentsareincreasinglyonthefrontlinesofaddressinglargescaleproblemswithglobalimplicationssuchasclimatechangepublicsafetyandeconomicinequality
Whileevidence-basedinterventionstohelpsolvethoseproblemsareavailablelocalgovernmentstaffhavelimitedtimeandcapacitytofindandimplementresearch-basedsolutionsAsaresultcitiescontinuetoworkthewaytheyalwayshave--makingincrementalimprovementstoprogramsbutnotalwayscreatingchangeatasystem-level
ThroughresourcedevelopmentandcollaborationwithJHUfacultyGovExissupportingcityleadersinusingdataandevidencetomakedecisionsandcreatinglastingrelationshipswithresearchers
TheRoadmapwasdevelopedbyGovExaspartofitspartnershipwithBloombergPhilanthropiesWhatWorksCitiesinitiative
AboutGovEx
69
- Title page
- Introduction and purpose of the Roadmap
- Using the Roadmap
-
- Gather information about the problem
- Strategically engage key partners
- Review the evidence and collect details
- Decide on a policy to implement based on local factors
- Establish a framework to monitor progress
- Assess data accessibility quality and security
- Design implementation and evaluation
- Share your story and success
-
- Working with a range of evidence
- Thank you
- References
- Resources
- About GovEx
-
UsingtheRoadmap
WhoshouldusethisRoadmap
ThisRoadmapisforanystaffmemberincitygovernmentwhoischargedwithsolvingaproblemorimplementingapolicyAlthoughthestepscanbeusedtochangeexistingprogramsorimplementnewprogramsthefocusoftheRoadmapisonlong-termsustainablepolicychangeWeusethetermpolicytomeanlawsregulationsfundingdecisionsandinterventions
CitystaffwhoarereadingthisguidemaybecomingfromdifferentstartingpointsItisimportanttobehonestaboutwherethatisForinstanceyoumaybelookingfornewsolutionstolongstandingseeminglyintractableproblemssuchasanannualincreaseingunviolenceormaybeyouarefacingnewerchallengesliketheopioidepidemicOrperhapsyouarelookingforadata-drivenwaytoaddressissuesrelatedtoday-to-dayservicedeliverysuchaspermittingorbikelanesThestepsareapplicablewhetheryouareworkingonalargescaleissueoralocalmunicipalissue
YoucanusethisguidewhetheryouareadecisionmakerorfrontlineworkerThroughouttheRoadmaptheresinformationaboutwhenandhowtoengageotherpeopleCollaborationisakeythemethroughthisRoadmapThegoodnewsisyoudonothavetodothisworkalone
TheapproachandthetoolscontainedintheRoadmapshouldberelevantwhetheryouareinastrongmayororcouncil-managerformofgovernmentElevatingthisprocesstothelevelofMayorCityManagerorCityCouncilmaytaketimeandmultiplestepsbutwehopeyouwillfindwaystoincorporatedataandevidenceintoyourday-to-daywork
WhatarethestepsoftheRoadmap
ThechaptersinthisRoadmaprepresentthehigh-levelstepsinvolvedinpolicychangeYouwillnoticethatthesectionsrefertooneanotherandbuildonthepreviousstepsTheRoadmapismeanttobeiterativeyouwillrevisitsomesteps
TherootsoftheproblemsthatcitiesareworkingtosolvearecomplexandmaybethelegacyofhistoricalfederalpolicyorsweepingsocialtrendsAlthoughthisRoadmapdoesnotexplicitlyincludestepsonreviewingthishistoryweencouragereaderstodevelopadeepunderstandingoftheprobleminyourowncitywhileconsideringcurrentstateandfederalcontext
UsingtheRoadmap
6
IninterviewswithGovExonthepolicychangeprocessintheircitiespeoplewasthewordmostcommonlyusedbyintervieweeswhendescribingthevariousstepstoimplementanewpolicyorprogramYouwillnoticethatthisisathemethroughoutthisRoadmapThefirstsectionbeginswithidentifyingtherelevantpeopleThechapteronidentifyingrolesforkeypartnersservesasafoundationforthefollowingstepsThediscussionofevidenceencouragesreaderstoconsiderwaysinwhichpeoplecanbuildanduseevidencetodesignstrongprogramsandinitiativeschangehumanbehaviorassessimpactensureeffectiveuseofpublicresourcesandimprovelives
Meaningfulapplicationofevidencebypolicymakersdoesnothappenwithone-timesharingofcuttingedgeevidenceRathermeaningfulinfluencerequiresalong-termstrategytoformalliancesanddevelopknowledgeofpolicymakersandpolicymakingInadditiontoworkingwiththerightpeopleandunderstandingevidenceandcontextwehopethatRoadmapuserswillformapeernetworkthatsharesnotjustwhichpolicysolutionsexistbutwhatimplementationlookslikeAsonecitywespoketoindevelopingthisRoadmapputitalotofourcitysproblemshavebeensolvedbyotherswejustneedtofigureouthowtoimplementthesolutions
TheexamplesthroughouttheRoadmapareillustrativeofgovernmentsthathaveeffectivelyimplementedtherelevantstepsinthisRoadmapbutpleasenotethatnocitydoesallofthisperfectlyTogetherthestepsinthisRoadmaparemeanttohelpcitiesframetheirproblemscreateademandforevidenceandbuildsustainablecoalitionsformeaningfulpolicychange
UsingtheRoadmap
7
GatherinformationabouttheproblemInyourearliestphaseitiscriticaltodevelopacomprehensiveunderstandingoftheproblemYouwillwanttolearnaboutthepeoplewhoareaffectedbytheproblemyouaretryingtosolveandthosewhocanimpacttheproblemFromthereyouwillgetinsightintotherootcauseoftheproblemtheworkthathasbeentriedinthepastandtheeffectivenessofthoseeffortsTheworkfromthischapterwillserveasthefoundationforactivitiesinsubsequentchaptersYoumayrevisitthesestepsasyouidentifyindicatorsandplanyourimplementationandanalysis
BesuretodedicatetimeandresourcestothisstepItcanbetemptingtoskipaheadtolookingforsolutionsortoassumethatyouhaveanunderstandingoftheproblemasaresultofapublicengagementprocess
TheCityofLewisvilleTXconductedaCitywidepublicengagementprocesstodevelopitsLewisville2025StrategicPlanOneoftheBigMovesoutlinedintheplanwasExtendingtheGreenorexpandinggreenspaceandconnectingrecreationareaswithneighborhoodsTheactionstepstoachievethisgoalincludedcleaningupcreeksandremovingdebristomakethemmoreaccessibleThecleanupswentsmoothlyinseveralneighborhoodsbuttheCitymetwithoppositionfromoneneighborhoodthatpreferredthenaturalhabitatandexpressedconcernabouttheenvironmentalimpactofthecleanupsAccordingtoLewisvilleiftheCityhadmappedallofthestakeholderstodeterminewhoneededtobeatthetableandwhowasmissingfocusinginformationgatheringontheneighborhoodlevelithaveavoidedthisproblem
Startwiththewhothepeople
Establishaprojectteam
BeginbynamingtheprojectteamforthisprocessincludingaprojectleadTheprojectleadshouldbesomeoneinCitygovernmentwhoisrespectedcangalvanizeotherkeypartnersandhasthetimetoleadthisworkThisissomeonewhomayhavetohavedifficultconversationswithCityleadershipdepartmentsandotherfundedpartnersItisnotnecessaryfortheprojectleadtobeapolicyexpertItismoreimportantforthemtohavetherightrelationshipsandpositionwithintheorganizationTheprojectteamshouldincludeindividualswithdiverseperspectivesfrommultipledepartmentsThereshouldbesomeonewithpolicyexperienceontheprojectteamtogroundtheconversationinrealisticactionHavingateammemberwhocancoordinatewithresearchersonrigorousanalysisandtranslatethatworkforothersiskeyAdditionallybesuretoincludefrontlinestaffworkersTheywillhavevaluableinsightbasedontheirinteractionswithresidentsandpolicysolutions
Gatherinformationabouttheproblem
8
havetomakesensetothepeoplewhowillbechargedwithimplementingthemBuildingastrongdiverseprojectteamisimportanttoensurethattheworkdoesnotliveanddiebyoneperson
Identifykeypartners
NextidentifythepeoplewhotheprojectteamcanworkwithontheissueyouarefacingincludingpeoplewithlivedexperienceintheproblemyourcityistryingtosolvedepartmentstaffandsubjectmatterexpertsoutsideofcitygovernmentThisistheconceptbehindbuildwithnotforamovementledbyLaurenellenMcCannthatemphasizescommunityexpertiseandleadershipMappingoutkeyplayersandtheirrolesiscriticaldotoearlyintheinformationgatheringphasesincecoalitionbuildingissuchanintegralpartofthepolicychangeprocessThisprocessmightconsistofstakeholdermapping(describedbelow)communityresourcemappingandculturalmappingInyourmappingremembertoconsiderpotentialalliesandthosewhomaypresentroadblocksalongtheway
PolicymakingisnottopdownorlinearInsomecasescitiesmayhavetosharedecision-makingpowerwithotheractorsForinstanceinmanycitiesthelocalContinuumofCare(CoC)istheleadentityonhomelessnessservicesCoCsapplyforandstewardfederalfundstoaddresshomelessnessandcoordinateservicesamonglocalprovidersAlthoughcitygovernmentsdohavearoleinCoCstheyarenotalwaystheleadagencychargedwithdistributingfundsandthereforetheirroleinhomelessnesspolicymaybelimitedCountygovernmentsoftenhavehealthdepartmentswhilecitiesdonotInthosecasesthebalanceofpowerfavorscountygovernment
TheCityofDentonTXwasrecentlyfacedwithanissueofresidentcomplaintsaboutanewdocklessbikesharethatarrivedintownandisviewedbysomeasexpensivelitterInresponseDentonsCityCouncilwhichwasalreadydiscussingpotentialrulesforbikesharecompanieselevatedthisissueandchargedcitystaffwithidentifyingasolutionPresentlytheCityisreviewingordinancesfromnearbycitiestodeterminehowtheyaresolvingthisproblemandwhatsolutionsmaybefeasibleinDentonThemappingexamplebelowrepresentslocalactorsinDentonwithastakeinthisissuewhohavemultipleperspectivesandlevelsofpower
Gatherinformationabouttheproblem
9
UsethistemplatedevelopedbytheCentreforPublicImpact(CPI)tobrainstormwhoyoumightreachouttoasyouembarkontheroadtopolicychangeRemembertothinkabouthoweachpersonorgrouprelatestotheproblem
BegintodefinethewhatortheprobleminyourCity
TherestofthissectionwillhelpyourefinetheproblemstatementFornowgathersomekeypartnerstoclearlyarticulatewhychangeisneededUsethetemplatebelowdevelopedbyDenverPeakAcademytocapturetheproblemstatementYouwillbuildonthislater
ProblemStatementndashWhyChangeisNeeded(1-2Sentences)
ThroughourinterviewswithcitystaffweheardthatwhendiscussingmunicipalchallengesitcanbetemptingtojumptoasolutionbeforetakingthetimetoreallyunderstandtherootcauseoftheproblemThismakessensesincecityleadersandstaffareresponsiblefordeliveringservicesandbeingresponsivetoresidentswhileaddressingnewproblemsthatariseinthecity
TherootsoftheproblemsthatcitiesareworkingtosolvearecomplexandmaybethelegacyofhistoricalfederalpolicyorsweepingsocialtrendsTohelptheproblemseemlessdauntingitcanhelptobreakthemdownintotherootcausetoidentifyamanageablecomponentoftheproblemUsingafishboneexerciseorfivewhystechniquecanalsohelp
Gatherinformationabouttheproblem
10
tofindtherootcauseoftheproblemandavoidsimplyaddressingsymptomsoftheproblemandrepeatingthemistakesofhistoryToolssuchasjourneymapscanhelpyouunderstandtheexperienceofthosewhoareaffectedbytheproblem
TheCentreforPublicImpactwithsupportfromBloombergPhilanthropiesdevelopedanexercisetosupportcitiesinidentifyingtherightsizedproblemAccordingtoCPIarightsizedproblemisonethatislargeenoughtopotentiallyapplytoothercitiesbutsmallenoughtosolvewithinareasonabletimeframe
AtthebeginningofMayorBerkesadministrationintheCityofChattanoogaTNveteransexperiencinghomelessnesswereaskingtheCityforhelpinfindinghousingTalkingtothispopulationrevealedthatveteranshadtroubleidentifyingaffordablehousingwhichmotivatedtheAdministrationtoworkonasolutionTheproblemofveteransexperiencinghomelessnesswastherightsizebecauseitfocusedononesegmentofpeopleexperiencinghomelessness-veterans-andtheirspecificchallengeinidentifyingaffordablehousing
Developacomprehensiveunderstandingofhowyourcitycurrentlyaddressestheissue
IntheChattanoogaexamplementionedaboveCitystaffdiscoveredthatveteranswerehavingtroublenavigatingtheCitysvariousaffordablehousingprogramsandgettingaccesstothehousingandservicesthattheyneededAtthesametimenewlyelectedMayorBerkewasexploringcurrentandfutureopportunitiestoservethispopulationRatherthancontinuetousefederalfundingforemergencysheltersMayorBerkedirectedcitystafftoprioritize
Gatherinformationabouttheproblem
11
HousingFirstwhichvaluesconnectingpeoplewithpermanenthousingovertransitionalhousingandsupportiveserviceafterspeakingwithrepresentativesattheUSDepartmentofHousingandUrbanDevelopment(HUD)
ThisexaminationofwhattheCityiscurrentlyfundingandthesourcesofthatfundingwasaturningpointinChattanoogaUnderstandingthisproblemfromasystemsperspective-includingpoliciesandfundingstreamsatmultiplelevelsofgovernmentsocialnetworksthroughwhichinformationisoftenspreadtechnologythebuiltenvironmentofmarket-rateandpublichousing-helpedthenewMayoranalyzeexistingeffortsandleadstaffandotheractorsinthesamedirection
Begintogatherthedata
TotrulycapturethesuiteofservicesthatyourCityisprovidingaskdepartmentstafftolisttheprogramsrelatedtoyourproblemstatementthatyourcityrunsorfundsWhenlistingtheprogramscapturedetailssuchas
programnamethenameofthedatasetthatisrelevanttoeachprogramadescriptionofthedatasetthesourceofthedatasetifthedataisinternalorexternalwhetherthisdataisalreadybeingcollected
ThisinformationwillformthebasisofyourinventoryofadministrativedataordatasetsthatarekeptbygovernmentsforoperationalpurposesThismayincludecallcenterrecords(311911)taxdatapropertyassessmentsinspectioninformationsocialservicerecordsetcAtthispointlistallrelevantadministrativedatasetsYouwillprioritizewhichdatasetstogathermoreinformationonlaterwhenyouaredevelopingindicators
Asampletemplatethatcanbeusedtocapturethisinformationisbelow
ProblemStatement____________________________________________
Gatherinformationabouttheproblem
12
ProgramName
DatasetTitle
DatasetDescription DataSource
Departmentorexternalpartner
DoestheCityalreadycollectthisdata
Whatisthenameoftheprogram
Human-readablenameofthedataset
Shouldbeunderstandablebynon-technicalusers
Whatistheoriginalsystemapplicationfilethathousesit
Departmentorexternalpartnerthatmaintainsthedata
Isthisdataalreadycollected
TogatherthisinformationconsiderassigningonepointpersontosurveyorinterviewcitystaffSpeakingwithfrontlineworkersstaffwhoworkwithfederalgrantsdepartmentheadsandeveryoneinbetweenwillensureadeepunderstandingoftherelevantprogramsanddatasetsDuringthecourseoftheseconversationsbesuretoaskcitystaffwhichorganizationstheyworkwithexternallyandcapturethatinformationaswelleveniftheyarenotrecipientsofcityfundingInadditiontoconversationswithexpertsincitygovernmentyourcitysbudgetdocumentsareanothergreatsourceofinformation
Collectdataandobservetrends
Nexttrytogainanunderstandingofwhytheproblemstillexistsifithasbeensomethingyouhaveworkedonforawhileorwhyitisemergingnowifitisnew
AnalyzingadministrativedatacandemonstratepasteffectivenessofinitiativesrelatedtoyourproblemstatementUsingthedatasetinventorythatyoubeganintheprevioussteptalktodepartmentstaffaboutwhichdatasetsaremostrelevanttotheproblemandgatherafewyearsworthofhistoricaldataBegintolookfortrendsinthisdatadiscussingpotentialreasonsforthosetrendswithrelevantstaffCanthetrendsbeexplainedbyseasonalitychangesinfundingstaffingfluctuationsorsomeotherreasonBasedonthehistoricaldataanalysiscreateabaselineagainstwhichfuturechangescanbeobserved
OtherdatasetscanhelpprovideadditionalcontextForinstanceifthenumberofpeopleexperiencinghomelessnessinyourcityisincreasingitmaybehelpfultotakealookathousingaffordabilityIfyoudonothavethedatayouneedconsideraproxymeasureAsanexamplehousingisoftenconsideredaffordableifitcosts30orlessofahouseholdsincomeButifhouseholdincomedataisnotavailableatthecitylevelAreaMedianIncome(AMI)canbeahelpfulproxy
Gatherinformationabouttheproblem
13
QualitativedatacanalsohelpanswerquestionsaroundtheeffectivenessofpreviouseffortsTrytoobtainavarietyofperspectives--bothmainstreamandoutliers--fromnetworksbeyondyourowntoprovideyouwiththegreatestvantagepointforevaluatingpotentialimpactUnderstandinghowthoseeffortswerereceivedbydifferentpartsofyourcommunityandiftheremaybeelementsofprevioussystemstopreservewillbecrucialtothesuccessofanypolicychangeThisinformationmaybeobtainedthroughfocusgroupsinterviewsandsurveysFormoreinformationonhowtousethesequalitativeresearchtoolspleaseseetheseresourcesdevelopedbytheAgencyforHealthCareandResearchandUrbanInstitute
FinallytakealookatnationaldatatoseewhereelsethisproblemmightexistThisresearchmaybehelpfulinlaterstepswhenyouarelookingforpromisingpoliciesinothercities
Puttingitalltogether
UsethetemplatebelowdevelopedbyDenversPeakAcademytocaptureyourrefinedrightsizedproblemstatementandcurrentandfuturestatesWorkingthroughthisexercisewithpartnersishelpfultogainbuy-inTheresponseswillbeimportantwhengettingotherpartnersonboard
WhyChangeIsNeeded
Yourright-sizedproblem
CurrentState
Analysisoftrendsindata
FutureState
Ideallywhataretheoutcomesyouhopetoachieve
Checklist
Thekeystepstogatheringinformationarebelow
Atthispointfocusontheproblemratherthanjumpingtosolutions
Listentopeoplewhoareaskingforhelpandtoadvocateswhoareraisingissues
Beasspecificaspossibleinidentifyingtherootcauseandhoninginonthepartoftheproblemthatyourcitycanaddress
Identifytrendsbasedonmultiplesourcesofdataincludinghistoricaladministrativedataproxymeasuresqualitativedataandnationalmeasures
Gatherinformationabouttheproblem
14
Captureyourproblemstatementtheresultsofyouranalysisanddesiredoutcomes
Gatherinformationabouttheproblem
15
StrategicallyengagekeypartnersPolicytheoryresearchoftenfocusesonincreasingpolicymakersaccesstoevidenceasawaytoavoidtakingshortcutslikemakingdecisionsbasedonemotionImplementationsciencesuggeststhatincreasingaccesstoevidenceisnotenoughandthatpolicychangeshouldbeexaminedinthewidercontextofdebatecoalitionformingandpersuasionMeaningfulpolicychangerequiresldquoalong-termstrategytoformalliancesanddevelopknowledgeofpolicymakersandpolicymakingratherthanusingevidenceinpolicymakingonanadhocorone-offbasis
UnderstandingnetworksandsystemsisimportanttodevelopsustainablepolicychangeThissectionismeanttohelpreaderstakethestepstobuildengagementstrategiesforpartnersthatrepresentdifferentsystemsandnetworksEngagementshouldbeanongoingprocessofconnectingpeopletooneanotherandtoanissueEngagementshouldnotbeinnameonly-wehaveallexperiencedengagementactivitiesthatgivepeopleaspecificroletoplaythatleadstoaneatpredictableoutcomeAccordingtotheBuildWithmovementundervaluingandunderinvestinginengagementstrategiescanleadtoadisconnectofcertainindividualsorgroupsandmistrustofgovernmentWhendevelopingengagementactivitiesrememberthathowpeopleengageandwhethertheyengagehastodowithhowyoustructureopportunitiesforengagement
Conveneacoreadvisorygroup
AnadvisorygroupcomprisedofpeoplefrommultiplesectorscanbehelpfulasyouconsiderthelocalsocialenvironmentalandpoliticalcontextinwhichyouintendtointroducepolicychangeRevisityourstakeholdermapanddeterminewhowouldbebesttoserveonthisgroupTomakesureyouareconsideringtheproblemsandsolutionsthroughanequitylensincludepeoplewithlivedexperiencewhohaveexpertiseintheeffectivenessofsystemswhetherthosesystemsfunctionasintendedandhowtheymayimpacthumanbehaviorYoumayalsowishtoincludesomeonewithexperienceworkingwithdataandanalysis-thispersoncanhelpdeterminethetypeofdatathatmaybereadilyavailableandusefultocollectandcanalsohelpplanandshapeyouranalysistofigureoutwhichprogramsareeffectiveBesuretoinvitefront-lineworkersaswell-itscriticalthatwhateversolutionscomeoutofthisprocessaremeaningfultothosewhowillberesponsibleforimplementingthem
BringingpeopletogethercanbechallengingbutthereareafewwaystosetandmanageexpectationsFirstmakesurethatthegroupisawareoftheproblemstatementcurrentstateandfuturestate-thiscanhelpstarteveryoneoffonthesamepageNextspend
Strategicallyengagekeypartners
16
sometimegettingtoknoweachotherrsquosworktolearnabouteveryonesstrengthsThissharedunderstandingcanhelpasyouclarifyrolesandresponsibilitiesofeachpartnerinthegroup--thisisimportanttoavoidfinger-pointinginthefutureandtoensurethatnoneofthetasksslipthroughthecracksInsomecasestherearemanyactorswithdifferentresponsibilitiesasinthecaseofBaltimoreCitys38-memberYouthFundTaskForcewhichischairedbynonprofitandcorporatepartnersandincludesrepresentativesfromthenonprofitandfaithcommunitiesstudentsthebusinesssectorlocaluniversitiesCityCouncilandtheMayorsOffice
OncerolesandresponsibilitiesareclearyoumayconsiderdevelopingacharterguidelinesortermsofreferencetosetexpectationsforthegroupBesuretodistinguishthisfromotherboardsandcommissionsthatyourcitymayhaveNextyoumaywishtoestablisharegularongoingmeetingscheduleBesuretosendagendasandmeetingmaterialsaheadoftimeandclarifytheobjectiveofeachmeeting
InSeattletheCitydevelopeditsHousingAffordabilityandLivabilityAgenda(HALA)torespondtorisinghousingcostsandhomelessnessThisisatrendinotherwestcoastcitiesaswellwhichmaybeexplainedbyaninfluxofnewresidentswhoareattractedbytheemergenceoftechjobstheattractionoftheoutdoorlifestyleortherecentpassageofprogressivepolicieslikelegalizedgaymarriagea$15minimumwageandlegalizationofmarijuanaSeattlebroughttogetheramulti-stakeholderadvisorygroupcomprisedoflocalhousingexpertsrentersandhomeownersdevelopersandgovernmentofficialstoshareinformationabouttheirneedsanddeveloprecommendationsbasedontheseneedsTheinterestsandmotivationsofthisgroupwerevariedandittookabouttenmonthstofindcommongroundTheseriesofrecommendationsthattheadvisorygroupcreatedismeanttoserveresidentswithaspectrumofneeds
Buildaprocessforcommunityparticipation
Ofcoursenoteveryonecanbeincludedinanadvisorygroupandadvisorygroupmembersmaynotnecessarilyrepresentthevariedperspectivesoftheirpeers-forinstancenoteveryhomeownerhasthesamesetofneedsTogatherinputfromawiderswathofthepopulationitcanbehelpfultobuildaprocesstoinvitethepublicatlargetosharetheirideasCitiesdothisinavarietyofwaysfromonlineconversationsandsurveystoin-personmeetingsRememberthatbeingintentionalaboutgatheringtheviewpointsofpeopleismoreimportantthanusingthelatesttechnologyandtools
TheinsightgeneratedbycommunityparticipationexercisescanbeincludedinyourreviewofevidencewhichisdiscussedinasubsequentchapterInWestVirginiaastatethathasbeenhithardbytheopioidepidemicstatehealthofficialsdraftedanopioidresponseplanthatwasinformedbypublicengagementaswellaslocalandregionalsubjectmatter
Strategicallyengagekeypartners
17
expertsThepublicwasinvitedtopresentcommentsonideasfordecreasingoverdoseearlyinterventiontreatmentandrecoveryPublichealthandmedicalfacultyfromWestVirginiaUniversityMarshallUniversityandJohnsHopkinsUniversityprovidedresponsestothecommentsTheresultwasanOpioidResponsePlanthathealthcommissionerRahulGuptapresentedtoGovernorJimJustice
AsamplespectrumtoplanyourowncommunityparticipationprocesscreatedbyLaurenellenMcCannisbelow
SourceLaurenellenMcCann
Community-basedparticipatoryresearchisanotherwaytoworkwiththecommunitytoimproveoutcomesCBPRmeansthecommunityisinvolvedinallaspectsoftheresearchprocesswhichisofteniterativeInCBPRtraditionalresearchersandcommunitymembersshareknowledgeaswellasresourcesandcredit
CBPRInActionInSanFranciscoanenvironmentaljusticenonprofitteamedupwiththeCitysDepartmentofPublicHealthandanoutsideevaluatortounderstandtherootcausesofobesityandfoodinsecurityinlow-incomeneighborhoodsandidentifysolutionsThroughresearchmethodssuchasinterviewingresidentsandstoreownersdiagrammingstoreshelvesGISmappinganeconomicanalysisandreviewofsimilarpoliciesinnearbycitiestheteamobservedthatthestoresthatresidentshasveaccesstowerestockedwithlesshealthyfoodsandthatresidentswantedmorehealthyfoodandlessalcoholandtobaccoThisprojectresultedinlocalandstatelegislationtoencouragestorestostockfreshproduce
Strategicallyengagekeypartners
18
Developrelationshipswithsubjectmatterexperts
OnewaytokeepsubjectmatterexpertsengagedistoinvolvetheminthecommunityparticipationprocessasdescribedaboveIntheWestVirginiaexamplesubjectmatterexpertsrespondedtoandsynthesizedthepubliccommentsOtheropportunitiesaredescribedbelow
AdvisoryAcitydepartmentheadMayororstaffmembermaydevelopaone-on-oneadvisoryrelationshipwithuniversityfacultymemberoracollaborativebetweentheCityandUniversitytoaddressaparticularproblemAgreatplacetostartistolookforrelevantpracticecenterswithinyourlocaluniversityandreachingouttosubjectmatterexpertsInadditiontotheirexpertiseontherelevantpolicytopicheorshemayalsobeabletoidentifyotherexpertsandevenprovidesupportonhiringtherighttalenttoleadcertainprogramsAnadvisoryrelationshipwithauniversitysubjectmatterexpertcanalsohelpwhencitiesareconsideringbringinginnewprogramsTheseexpertsareoftenfamiliarwithprogramsthatarebasedonnationalmodelsandcanprovideinsightintoaprogramsoriginandlineageeffectivenessinotherplacesandthepotentialforimplementationinyourcityAcademicscanalsohelpidentifyindicatorsbasedongovernmentindicatorsorliteratureFinallyrelationshipsbetweenpolicymakersandresearchersprovidethemostbroadlyeffectiveroutetopullingresearchintopolicy
ResearchandevaluationAcademicexpertsmaysupportcitiesinevaluation-indeedevaluatingcityinitiativesisatraditionalmodelofcityuniversityrelationshipsSharingtheinformationyoucollectedintheGatherInformationphasecanhelpresearchersunderstandyourworkandgoalswhichinturnwillstrengthenthesupportthattheycanprovideDiscussingdetailsaboutyourinterventionandevaluationplanatthebeginningoftheprocessmeansthataresearchpartnercanhelpwiththeevaluationdesignanddatacollectionForinstancetheWilsonSheehanLabforEconomicOpportunitieswalkscitiesthroughconsideringthefollowingquestionsbeforepartneringonanevaluation
1 Isthereaclearpolicyprogrammodelwithobjectivescreeningcriteria
2 Isitpossibletoevaluatewhathappensbothwiththepolicyprogramandintheabsenceofthepolicyprogram
3 Arethereoneormoremeasurableoutcomesofinterest
4 Arethereenoughpeopleforrigorousquantitativeanalysis
5 Whatdatawillbenecessaryforthestudyandwhataretheprocessesforaccessingthesedatasource(s)
6 Isthereacompellingreasontoevaluate
Strategicallyengagekeypartners
19
InadditiontoassessingreadinesstoconductanevaluationresearcherscanhelpyouhoneinonthepurposeoftheevaluationTheWorldBankdescribesseveralpotentialusesofevaluationwhicharedefinedhereInbriefevaluationcanhelpdecisionmakers
makeresourceallocationdecisionsrethinkrootcausesofproblemsidentifynewproblemsdecideonabestalternativedeterminewhetherprogramsorpoliciesarebeingimplementedwithfidelitydeterminewhetherprogramsorpoliciesarehavingthedesiredimpact
Researcherscanalsoprovidesupportinsharingtheresultswiththescientificcommunitythroughpapersandconferenceswhichisvaluableincreatingapeernetworkofevidence-basedcitypractitioners
TrainingandtechnicalassistanceUniversitypartnersespeciallythoseatpracticecenterscanbehelpfulindevelopingorupdatingtrainingcurriculumtrainingthetraineranddeliveringtrainingtocitystaff
Strategicallyengagekeypartners
20
ReviewtheevidenceandcollectdetailsOnceyouhaveathoroughunderstandingoftheproblemdataandactorsyouwillneedtoconductyourownresearchonavailableevidenceandlearnaboutthelandscapeofpolicyoptionsYourgoalwillbetodevelopanaccessiblelistofinformationaboutpoliciesthathavehelpedsolvesimilarproblemsinotherplacesThischapterwillguideyouthroughthestepsofpreparingyourreviewersunderstandinglevelsofevidenceandcollectingthedetailsonevidencethatyouwillneedfordiscussionwithdecisionmakersinyourcity
Selectandprepareyourreviewers
AssignapersonorteamtoreviewrelevantinterventionsthathavedemonstratedsuccessinotherjurisdictionsTheirgoalwillbetoproduceareviewofalloftherelevantprogramspoliciesorinitiativesthatyourteamcouldconsiderimplementingTheywillbeconductingaliteraturereviewsupplementedbyinterviewswithexpertsandpeerswhohaveinformationaboutpotentialprograms
SincethepeopleyouassigntothistaskareprovidingthebaselineofpotentialoptionstoimplementitisimportantthatyouchoosepeopleyouareconfidentwillbethoughtfulandthoroughintheirworkItisalsocriticalthattheyareawareoftheimportanceofsourcecredibilityEnsureyourreviewershavefamiliarizedthemselveswithrulesforcriticallyanalyzinginformationsourcesbeforetheygetstartedIftheyareincludingnewsarticlesaspartoftheirsearchtheyshouldalsobeawareofmethodsforevaluatingnewscredibility
AskyourreviewerstodevelopaplanandtimelinetocompletethereviewThisstartswithsolidifyingagoverningresearchquestionOnewaytoknowyouarecreatingadetailedresearchquestionistousetheacronymPICOremindingyoutoincludereferencetothepopulationintervention(ifyouhaveoneinmind)comparisonandoutcomeelementsinyourresearchquestionForexampleifyouknowthatyouhaveapopulationofteenagerswhoarehomelessyouhavelearnedfromyourstakeholdersthatmanyoftheseteenagersidentifyasLGBTandyouhaveheardsomethingabouttheapproachofcreatingdrop-incentersfordeliveringsupportyoucouldask
ForLGBTyouthfacinghomelessness(population)howeffectivearedrop-incenters(intervention)relativetostandardsocialservices(comparison)atsettlingtheminpermanenthousing(outcome)
Ifyoudonothaveaninterventioninmindyoucanleaveouttheinterventionandcomparisonandsimplyask
Reviewtheevidenceandcollectdetails
21
HowcanwehelpLGBTyouthfacinghomelessness(population)obtainpermanenthousing(outcome)
OnceyourreviewershavetheirresearchquestionsusethequestionstocreatealistofkeywordstheycanuseintheirliteraturesearchThiswillhelpkeeptheircollectionofarticlesreasonablyfocusedHavethemdescribewhattheirsearchprocesswillbeincludinglistingthedatabasestheywillreviewandtheirplanforstoringtheinformationtheywillcollect(Herearesomeadditionalsuggestionsforstructuringareviewplan)
WhatyourreviewerswillbedoingisalightweightversionofarapidreviewTotheextentthattheywouldliketodevelopamethodologyforspecificsearchprotocolsincludingcriticalappraisalprocessestheycancertainlygoaheadanddoso
Usedifferentlevelsofevidenceforthemostcompletepicture
AsdiscussedintheWorkingwitharangeofevidencechaptertheNestaStandardsofEvidenceframework--reproducedbelow--ishelpfultounderstandthelevelofconfidencethataprogramorinterventionishavingapositiveimpactThisapproachletsyouputinterventionsthatyouhaveheardaboutlocallybutwhichhavenotbeenstudiedtogetherwiththosewithademonstratedcausaleffectThisishelpfulsinceanumberofpoliciesprogramsandpracticesthatyouwillwanttoconsidermaynotyethavemuchofaresearchrecord
Reviewtheevidenceandcollectdetails
22
NearlyanypieceofinformationaboutaninterventioncanbeclassifiedwithinthestandardsofevidenceForexampleifsomeonehasacrediblehypothesisabouthowaninterventioncouldproducebenefitthistypeofevidencewouldbeclassifiedasLevel1ThistypeofevidencecouldincludeinterventionsinothercitiesthatyoureadaboutinthenewsorhearaboutataconferenceForinstancewhenUSDAallowedschoolsandschooldistrictstoprovidefreebreakfastandlunchtoallstudentsthroughtheCommunityEligibilityProvision(CEP)manydistrictsimplementedthisoptionevenbeforetherewasevidenceofeffectivenessThiswouldfitintoLevel1sinceitbuildsonthelogicthatprovidingaccesstomealswilldecreasefoodinsecurity
YoumayalsofindpiecesofinformationthatqualifyasLevel2evidenceForinstanceasmoreschooldistrictsimplementedCEPprincipalstestifiedaboutpositivechangestheyobservedinstudentssuchasamorepositivedemeanorthatcanleadtobetteracademicperformanceDataalsoshowedthatthestudentsdescribedwerereceivingmoremealsthanbeforetheprogramsimplementationHoweveritwastoosoontobuildevidencetoshowacausalrelationshipbetweentheincreaseinfreemealsandincreasedacademicperformance
ldquoLower-levelrdquoevidencecanconsistofnuancedbutvaluableelementsthatrelatetoaninterventionrsquospotentialfitwithacommunityQualitativeinformationsuchastestimonyfrompeoplewhohaveexperiencedaninterventionandthosewhohavefacilitatedorimplementeditwillalwaysbeusefultoconsiderandprovidesdetailedlocalcontext
Experimentalstudies--describedstartingatLevel3withcontrolledexperimentsandgoingthroughLevel5withfullyfleshedoutprotocolsandprocessesforreplication--stilloccupythehigherlevelsofthisdescriptionofevidenceThisprioritizationshowsthevalueofrigorousstudyforevaluatingtheimpactofaparticularpolicyorprogramratherthanotherfactorsthatcouldbeinfluencingoutcomesofinterestRememberthatmultipletypesofevidencecancomplementrigorousimpactevaluationsandgivecityleadersadditionalinsightsandbroaderperspective
Accessinformation
Encourageyourreviewerstobegintheirsearchforpromisingpracticesingovernmentandnon-profitclearinghousesandinacademicpolicyjournalsWorkpublishedinthesesourceshasalreadybeensubjectedtoahighlevelofprofessionalscrutinyandgenerallycontainsagreatdealofdetailwhichmeansthatyourreviewersaremorelikelytobeabletopullouttheinformationyouneedtomakeagooddecision
Clearinghouses
Reviewtheevidenceandcollectdetails
23
Herearesomeoftheclearinghouses(collectionsofprogramevaluations)forevidence-basedpoliciesandprograms
ResultsFirstClearinghouseDatabaseThePewCharitableTrustsCollectstogethereightotherclearinghousestoprovideinformationoninterventionsinthefollowingcategoriesadultcriminaljusticeeducationchildwelfarehealthjuvenilejusticementalhealthsubstanceabuseandyouthdevelopmentIndividualclearinghousesincludedinResultsFirstare
BlueprintsforHealthyYouthDevelopment-foryouthdevelopmentandviolenceprevention
CaliforniaEvidence-BasedClearinghouse-forchildwelfareinterventions
TheLauraandJohnArnoldFoundationrsquosSocialProgramsThatWork-forsocialservicesprograms
TheUSDepartmentofEducationrsquosWhatWorksClearinghouse-foreducationinterventions
TheUSDepartmentofHealthandHumanServicesrsquoResearch-TestedInterventionPrograms(RTIPs)-forcancercontrolinterventions
TheUSDepartmentofHealthandHumanServicesrsquoTeenPregnancyPreventionEvidenceReview(TPPEvidenceReview)-forsexualriskbehaviorandsexualhealthoutcomes
TheUSDepartmentofJusticersquosCrimeSolutionsgov-forcriminaljusticejuvenilejusticeandcrimevictimservices
TheUSSubstanceAbuseandMentalHealthServicesAdministrationrsquosNationalRegistryofEvidence-basedProgramsandPractices(NREPP)-substanceuseandmentalhealthinterventions
TheUniversityofWisconsinPopulationHealthInstituteandRobertWoodJohnsonFoundationrsquosCountyHealthRankingsandRoadmapsWhatWorksforHealth(WhatWorksforHealth)-healthbehaviorsclinicalcaresocialandeconomicfactorsandthephysicalenvironment
UKWhatWorksCentresTheBritishgovernmenthascharteredanetworkofsubject-specificorganizationswhichcollectsynthesizeandreportoutevidenceontheeffectivenessofpolicyprogramsandpractices
NationalInstituteforHealthandCareExcellence(NICE)-healthandsocialcare
Reviewtheevidenceandcollectdetails
24
SuttonTrustEducationalEndowmentFoundation-educationalachievement
CollegeofPolicingWhatWorksCentreforCrimeReduction-crimereduction
EarlyInterventionFoundation-earlyintervention
WhatWorksCentreforLocalEconomicGrowth-localeconomicgrowth
CentreforAgeingBetter-qualityoflifeforolderpeople
WhatWorksCentreforWellbeing-wellbeing
WilsonSheehanLabforEconomicOpportunities(LEO)UniversityofNotreDameProvideslinkstotheirevaluationsofinterventionsincriminaljusticeself-sufficiencyeducationhealthandhousingandhomelessness
ClearinghouseforLaborEvaluationandResearchUSDepartmentofLaborProvidesresearchonworkforceandlabor-relatedpoliciesandprograms
J-PALEvaluationsAbdulLatifJameelPovertyActionLabStudiesonpoverty-reducinginterventionsfromaroundtheworld
CampbellSystematicReviewsCampbellCollaborationSystematicreviewofpolicyinterventionsbyscholarlycoordinatinggroupsinsevenissueareascrimeandjusticeeducationinternationaldevelopmentmethodssocialwelfareknowledgetranslationandimplementationanddisability
Benefit-CostResultsWashingtonStateInstituteforPublicPolicyAWashingtonState-charteredprogramtoestablishthelikelihoodofrelativecostsandbenefitsforprogramsacrossthefollowingareasjuvenilejusticeadultcriminaljusticechildwelfarepre-Kto12educationchildrensmentalhealthhealthcaresubstanceusedisordersadultmentalhealthpublichealthandPreventionworkforcedevelopmentandhighereducation
EffectiveInterventionsCentersforDiseaseControlandPreventionRepositoryforevidence-informedHIVpreventioncareinterventions(withstepsonhowtoimplement)
Journalsandindexes
OnceyouhavereviewedrelevantinformationaccessiblethroughclearinghousesitisworthtakingalookatindividualstudiespublishedasscholarlyarticlesYourreviewerswilldothisbysearchingarticleindexesandacademicjournalsUnfortunatelyanumberofusefulacademicjournalsandindexingplatformsmaynotbegenerallyaccessibletoyourreviewersifyoudonothaveasubscriptiontothemToaddressthisconsiderpartneringwithalocal
Reviewtheevidenceandcollectdetails
25
collegeoruniversitytoprovideyouwithalistofarticlesfromthefollowingsources(Thisisalsoanexcellentopportunitytoseekresearchsupportfromanyacademicstakeholdersyouidentifiedpreviously)
Indexes
GoogleScholarincludesmostpeer-reviewedonlineacademicjournalsandbooksconferencepapersthesesanddissertationspreprintsabstractstechnicalreportsandotherscholarlyliteratureincludingcourtopinionsandpatentsItallowsyoutospecifyapublicationrangeinyoursearchandprovideslinkstolikelysourcesforalllistedarticles
PolicyFileIndexisauniqueresourceforUSpublicpolicyresearchinthatitgrantsusersaccesstotimelyupdatedinformationfromover350publicpolicythinktanksnongovernmentalorganizationsresearchinstitutesuniversitycentersadvocacygroupsandotherentities
PolicyArchiveworkstocollectanddisseminatesummariesandfulltextsvideosreportsbriefsandmultimediamaterialofthinktankuniversitygovernmentandfoundation-fundedpolicyresearch
Anothersourceforhigh-qualityresearchisthereportsoftheCongressionalResearchServiceYoucanfindtheirreportsaggregatedhere
Specificjournals
Oftenthemostusefulacademicjournalsforunearthingpolicyorprogramreviewswillbeineachspecificsubjectarea--thatisineducationjournalsforeducationpoliciesYoucanuseageneralindexlikeGoogleScholartobegintolearnwhichjournalsaremostoftenusefulforyoursearchHoweveryoucanalsosearchthefollowingtwogeneralistpolicypublicationsforissuesandarticlesrelatedtoyourtopic
PublicAdministrationReviewJournalofPolicyAnalysisandManagement
Newssourcessubjectmatterexpertsandinterviews
AlthoughitisvaluabletoknowwhathasbeenexploredthroughascholarlyresearchframeworkitisalsousefultolearnwhatkindofinterventionsarebeingtriedinthefieldrightnowandhowpeoplefeelaboutthesetrialsSearchsubscriptionnewsaggregatorslikeLexisNexisorfreenewsaggregatorslikeGoogleNewsorYahooNewsforyourissueareaplusthewordpolicyorprogramtoseewhatkindsofinterventionsarebeingcoveredinthenews
Reviewtheevidenceandcollectdetails
26
DonotforgettocontacttheadvisorygroupmemberswhowerechosenfortheirsubjectmatterexpertiseInadditiontoprovidinggoodmaterialthroughinterviewsthemselvestheymayhaveideasaboutothersubjectmatterexpertsyoushouldcontacttoaskaboutpotentialinterventions
Peercitiesandnetworks
InadditiontoreadingabouttheworkofothercitiesitisidealifyouareabletospeakwithpeoplewhoareinvolvedwithcurrentimplementationsofinterestinginterventionsInyourcityyoulikelyhaverelationshipswithyourneighborsbutitcanbehelpfultolookoutsideofyourregionforsolutionsaswellTofindcounterpartsinothercitieswhoareworkingonaprogramyouwouldliketolearnmoreaboutyoucancontactstaffinintermediaryorganizations(likeGovExoryourprofessionalassociations)whomightbeinapositiontoputyouintouchimmediatelyorwhomightseeanothergoodrouteforsettingupaconversationHereisalistofnetworksthatmaybehelpful
InternationalCityCountyManagementAssociation(ICMA)NationalAssociationofCounties(NACO)NationalLeagueofCitiesUSConferenceofMayorsWhatWorksCities
Collectthedetails
Onceyourreviewershavefoundevidenceonpossiblepoliciesorprogramstoimplementtheyshouldcapturekeydetailsthatwillhelpdecisionmakersunderstandthepotentialeffectivenessandapplicabilityofeachpotentialintervention
Foreachsourcethathasprovidedinformationyouwanttoconveyusethetemplatebelowtocaptureasmanyofthefollowingdetailsaspossible(Youdonotneedtocompletethisforsourcesthatprovideduplicativeinformationorareotherwisenotthatuseful)
Programorpolicyname (egldquoHousingFirstrdquoorldquoblightedpropertiesregistryrdquo)
Sourcetype (egjournalarticleinterviewsubjectmatterexpertstudyassociationcommunityparticipationprocess)
Sourcedetails (egarticletitleandpublicationnameandtitleofinterviewee)
Evaluationmethodology
(egrandomizedevaluationcomparativeevaluationoftwodifferentgroupsbeforeafteranalysis)
Evidencelevel (egLevel12345)
Programorpolicy
Reviewtheevidenceandcollectdetails
27
description
Programorpolicygoal (1sentencecapturingtheprincipalaimoftheintervention)
Existingimplementations
(WherehastheinterventionbeentriedListasmanysitesasyouareawareof)
Characteristicsofimplementationsites
(ForeachimplementationsiteofferbasicgeographicIDmdashcityandstateThenyouwillwanttocollectthedetailswhicharerelevanttoyourdecisionmakersrsquoconsiderationsaboutwhatmakesalocationldquosimilartordquoorldquodifferentfromrdquoyoursituationYouwillatleastwanttoincludeforeachimplementationsitePopulationnumberPopulationdensity(ruralurbansuburban)Socio-economiccharacteristicsBeforestartingaskyourdecisionmakersaboutanyparticularcharacteristicsofprogramimplementationsitestheywouldwanttoknowbeforeseekingtoreplicate)
Characteristicsofimplementationpartners
(Anyavailabledetailsaboutorganizationsinstitutionsorpeoplewhowereinvolvedwithimplementingtheintervention-egindependentchildcarecenterpublicelementaryschoolcitypublichealthnurses)
Characteristicsoftargetpopulationeligibleprogramparticipants
(Anyavailabledetailsaboutpeoplewhoarethebeneficiariesoftheprogramorotherwiseincludedoraffectedbytheintervention-egnumberofprogramparticipantscategoriesofpropertyownersaffected)
Policyorprogramduration
(HowlonghastheinterventionbeeninplaceineachimplementationsiteHasitendedandifsowhy)
Whatbenefitscamefromeachimplementation
(Anyobservedpositiveeffectsoftheinterventionincludingeffectsize)
Statisticalestimatesofinterventioneffectiveness
(Anyavailablestatisticalestimatesoftheinterventionrsquoseffects)
Whatdrawbackswereobserved
(Anyavailabledetailsaboutunintendedorunavoidablenegativeeffectsoftheintervention)
Programorpolicycosts
(Anyavailabledetailsaboutprogramcostsperimplementation)
Reviewtheevidenceandcollectdetails
28
Reviewtheevidenceandcollectdetails
29
DecideonapolicytoimplementbasedonlocalfactorsTogetthecriticalbuy-infromleadershiptoimplementanewpolicyorprogramtheprojectleadshouldscheduleameetingwithdecisionmakerstopresenttheevidencesurroundingthepolicyoptionscapturedinthetemplateDependingonyourinsightintoyourcityleadershipswillingnesstomakechangeyoumaywishtosharetheresearchyouhaveconductedontherootcausesoftheproblemandsomeofyouranalysisontheimpactofcurrenteffortstodate
Considertheapplicabilityofanapproach
AsdiscussedinthepreviouschapterpolicymakersoftenlookforevidenceofeffectiveprogramsinothercitiesbutoftenthequestionarisesofhowgeneralizableaprogramisfromonecitytoanotherTheAbdulLatifJameelPovertyActionLab(J-PAL)suggeststhatinreviewingevidencefromanotherlocationitishelpfultoidentifythespecificmechanismthatwasused(deliveryofprogramorserviceinformationsharingetc)identifythelocalconditionsthatwereinplaceobservethebehaviorchangethatresultedfromthemechanismandfinallytounderstandlocalcontextandconditions(legalorcostconstraintsculturalnormsetc)Thissetofpresentationelementscanbedrawnfromthelistofdetailscollectedintheresearchtemplate
EducationpolicymakersinChicagoILusedfindingsfromprogramsinKenyaandIndiainwhichchildrenwhohadfallenbehindtheirpeersweregiveneducationalsupporttargetedtotheirleveloflearningThemechanismusedwastargetedteachingandthereweresimilarconditionspresentinallthreecountriesAlthoughsomestudentshadfallenbehindinthecurriculumteacherswereincentivizedtoteachgrade-levelmaterialsConsiderationsofhowthelocalcontextinChicagomightaffectimplementationofatargetedteachingprogramincludedunderstandinghowtheneighborhoodsthatthestudentswerefrommayhaveaffectedtheireducationaloutcomesandthehighcostofteachinginsmallgroupsInChicagoMatchEducationprovidedtwo-on-onetutoringwhichmatchedstudentswithwell-educatedindividualswhohavenothadformalteachertrainingforalower-costmodelTheresultswerepositiveStudentswhoweretutoredperformedbetterinmathandweremorelikelytopasshighschoolclasses
ThegeneralizabilityframeworkproposedbyJ-PALisbelowUsethisasatemplateasyouconsiderpotentialpolicysolutions
Decideonapolicytoimplementbasedonlocalfactors
30
SourceTheGeneralizabilityPuzzle
OnceyouhavenarrowedthepolicyoptionsdiscussthequestionsbelowforeachoptionsyouareconsideringThisstepiscriticalasunderstandinglocalcontextissoimportantinsuccessfulpolicyimplementation
CommunityContextQuestionnaire
Localcontext
DoesthisoptionmakesenseinourcommunityIsthisaonesizefitsallapproachIsanyparticulargroupabsentfromthisapproachIfsoisthatok
Youranswers
Effectiveness
WhatwouldweexpecttoobserveifthepilotisaseffectivehereasinCityX
Youranswers
Timing
DoesthetimingmakesenseIsthistherighttimetoimplementthispolicychange
Decideonapolicytoimplementbasedonlocalfactors
31
WherearethepotentialroadblocksthatcouldaffectthescheduleIsthereanywaytomitigatethisHowlongwouldittaketoseeanimpact
Youranswers
Implementation
DowehavethestaffcapacitytoimplementthisArethereadditionalpartnersthatyoucanbringtothetabletosupportimplementationDoesthismakesensegivenourformofgovernment
Youranswers
Authority
DoesourcityhavetheauthoritytoenactthisIfsowhatistheappropriatemechanismAretherelegalbarriersArethereanyexistingpreemptionissuesfromthestateorfederalgovernmentCanthosebeaddressedthroughlegislationadvocacyIstherepoliticalsupportoroppositiontothis
Youranswers
Cost
HowmuchwillthiscostannuallyHowwillwepayforitCanthepolicybeimplementedwithexistingfunding
IfsoisthisfundingreliableHowwouldthispolicyoptionaffectourcitysbudgetintheshortandlongtermWhatotherfundingoptionscanweconsiderIsthereanalternativethatwouldcostlessandgiveussimilarresultsWillanyconstituencybeadverselyaffectediffundingforprogramsthatimpactthemisshiftedtoanotherprogrampolicyorinitiative
Youranswers
Decideonapolicytoimplementbasedonlocalfactors
32
EstablishaframeworktomonitorprogressNowthatyouhavereviewedtheevidenceconsideredlocalcontextandchosenyourpolicyitistimetodeterminehowyourcitywillmonitorprogressofthisnewapproachCitystaffandleadersneedtoestablishtheinfrastructureforongoingcontinuousdecisionmakingbasedonwhetherthepolicychangeisleadingtothedesiredoutcomesforresidentsandwhetherthechangeismeaningfulandrationaltofrontlineworkersSettingarbitrarygoalsatthelocallevelthatconflictwithgoalsatotherlevelsisconfusingtotheindividualsresponsibleforachievingthegoalsGoalsthataresettoohightoachievecanbedemoralizingtostaff
ThissectionincludesstepstosetupavenueinwhichCitystaffandleadershiproutinelyvisitkeyindicatorstodeterminewhatisorisnotworkingandtomakedecisionsinreal-timeFormorereadingonperformancemanagementpleasevisittheGovExPerformanceManagementGettingStartedGuide
InperformancemanagementtermssuchasindicatormeasureandmetricmaybeusedinterchangeablyInthisguidewewilluseindicatorInyourworkthemostimportantthingistobeconsistentinwhichevertermyouuseThisvideomaybehelpfulinunderstandingthisconceptandcommunicatingittoothers
Establishthefundamentalsandgatheryourteam
FirstdeterminewhetherthereisanestablishedvenueorregularmeetingscheduleduringwhichyoucandiscussprogressIfyourcityhasanexistingperformancemanagementstructureonarelatedtopicaddingadiscussionoftheprogressofpolicychangetoyouragendacanbeagreatideaIfthiswillbeastandalonemeetingdecidehowfrequentlytheprojectteamanddecisionmakerswillmeetandputthosemeetingsonthecalendarfortheentirelengthoftheproject
DeterminewhowillstaffandattendroutinemeetingsonCityprogressThesemeetingsmayincludetheMayorandorCityManagerdepartmentheadstaffwhoareimplementingthepolicychangeandanalysts
Identifyindicators
ItiscriticalthatindicatorsbeanimportantcityandcommunitypriorityaswellassomethingtheCityplaysamajorroleinaddressingTheremaybeinstancesinwhichtheCitywillneedtoreachouttoorcollaboratewithexternalpartnerstogetamorecompletepictureofprogressontheoverallpolicychange
Establishaframeworktomonitorprogress
33
Anoutcomeindicatordescribesthedesiredeffectofthetargetedpopulation
ExampleBellevuesfamilyfriendlyneighborhoodsindicatormeasuresthepercentofresidentswhobelievetheyliveinaneighborhoodthatsupportsfamiliesparticularlythosewithchildren
WithyourkeypartnersrevisityourproblemstatementcurrentstateandfuturestateexerciseandbegintoidentifytheoutcomeindicatorsthatwillillustratesuccessFornowchooseasingletop-levelkeyperformanceindicator(KPI)ItiscriticalthattheKPI-thepieceyouwillmeasure-beapriorityforthecommunityandsomethingtheCityplaysamajorroleinaddressing
Outputindicatorsmeasuretheprocessofprogramorservicedelivery
ExampleSanDiegoMayorKevinFaulconersgoaltorepair1000milesofstreetby2020
ReviewingtheadministrativedatasetinventorybegintoparedowntherelevantprogramsanddatasetsthatsupportyourKPIWhatarethemeasuresofeffectivenessforeachprogramTheseareyouroutputindicators
Thediagrambelowillustratestherelationshipbetweenoutputandoutcomeindicators
ThinkaboutimplementationindicatorsasmeasuringstepsintheprogramsrolloutthatarenecessarytoobtaindesiredchangesImplementationoutcomescanfocusontheinterventionstaffdeliveringtheinterventionanorganizationorsystemandcanaddressfeasibilityadaptabilitysustainabilityandcostTwokeyquestionstoconsiderwhendevelopingimplementationindicatorsare
1 Istheinterventionacceptableandappropriatetothecommunityand
2 IsitfeasibletoimplementinterventionXatthistimewiththeseresources
Establishaframeworktomonitorprogress
34
Amoreextensivelistwithdefinitionsofpotentialimplementationoutcomesislinkedhere
UsethetablebelowtochoosethedesiredimplementationoutcomesforyourpilotandtoplanyourmeasurementtechniqueThisshouldincludelevelofanalysis(individualproviderorconsumerorganizationorsetting)yourmeasurementtechnique(surveysobservationinterviewfocusgroupsadministrativedataanalysis)andthephasewhenyouwillcollectthisinformation(earlymidlate)Theremaybemorethanonelevelofanalysismeasurementtechniqueandphaseofdatacollection
Implementationoutcome
Definitionofoutcome
Levelofanalysisandmeasurement
Yourmeasurementtechnique
Phaseofpilotwhenyouwillcollectdata
Usethespacebelowtocaptureyourindicatorwishlist
Indicatorwishlist
Outcomes_____________________________________________________________________________
Outputs________________________________________________________________________________
Implementationoutcomes______________________________________________________________
Setapublichigh-levelgoal
Thehigh-levelgoaliscriticalinestablishingdirectiontogalvanizethepeoplewhowillbedoingtheworkWhendiscussingthegoalusetheMayorsbullypulpitLookforkeymilestonesinCitygovernmenttosharethegoalstatementsuchasthebeginningofaMayoraladministrationaStateoftheCityspeechorotherpublicremarks
GoalsshouldbeldquoSMARTrdquo
Specificaspossible(innamingthepopulationwhowillbebetterofftheactionthecitywilltake)
Measurable
Achievable
Establishaframeworktomonitorprogress
35
Results-focused
Time-bound
ThegoalstatementshouldincludeyouroutcomeindicatorlikeChattanoogasgoalldquoWewillendchronicveteranhomelessnessbyDecember2016rdquoAdiagramtohepyoubuildaSMARTgoalaroundyouroutcomeindicatorisbelow
Component Description Yourturn
Specific WhowillbebetteroffWhatactionthecitywilltake
Measurable Howwillyoumeasuresuccess
Achievable Isthegoalattainable
Results-focused Whatistheexpectedresult
Time-bound When
Setrealisticambitioustargetsandtimelinesforeachindicator
ToestablisharealisticandambitioustargetforyourgoalstatementusetheadministrativedatacollectedforyourchosenKPItoreviewactualCityperformanceinthemostrecentyearcomparedwiththethreeorfourmostrecentyearsworthofdataBegintocalculatesummarystatistics(egtotalminimumaveragemedianmaximum)toidentifytrendsCreatecumulativeandmonthlychartstoidentifytrendsandanomaliesBasedonyouranalysisofhistoricaldataignoretheanomaliesandassumetheproductivitytrendswillfollowthesameoverallpatternfrommonth-to-monthFinallyapplyanincreasedproductivityfactortotheperformanceeachmonthtoforecastfutureprogressBasedonthisanalysischooseatargetthatinspiresimprovementwhileaccountingforreality
Manycitiesthataremorematureintheirperformancemanagementpracticesrecognizetheimportanceoftakingacommunity-levelapproachtoimprovingoutcomesforresidentsThesecitiesarecreatinggoaltaxonomiesthatincludemeasuresrelatedtopublicsafetyhousingandevenrecreationCitiesdriveprogressonthesegoalsbyimplementingstrategiesthatengagemultipledepartmentsandcollaboratewithserviceprovidersinthecommunity
EventuallyyouwilldevelopgoalstatementsfortheoutputsinordertocapturetheefforttoachievetheoutcomecontainedinyourgoalstatementYoursetofindicatorsshouldbecomposedofshort-andinterim-termsupportivemeasureswithaccuratetargetsandrelevanttimelinesAnexamplegoaltaxonomyfromtheCityofLasVegassResultsVegasprogramisbelowUsethistaxonomyasamodelforyourownsuiteofindicatorsandgoalstatements
Establishaframeworktomonitorprogress
36
Yourturn
Thetemplatebelowismeanttohelpyoucraftagoalstatementforeachsupportingmeasure
Goalstatement
Departments
Outcome
Activities
KPI
Target
Timeframe
Conductanddiscussrelevantanalysis
ThrougharoutineseriesofmeetingscontinuallymonitorprogressofKPIssupportingindicatorsandanyotherdatacollectedsuchasqualitativedatatoidentifytrendsthatwillindicatewhetherthechosenpolicyishavingthedesiredimpactThesedatapointsshoulddrivethediscussionduringthesemeetingspointingtoopportunitiesforfurtherinquiryinto
Establishaframeworktomonitorprogress
37
whatisdrivingtheobservedtrendsAnalystsshouldbeencouragedandtrainedtoaskquestionsofthedepartmentsproducingthedataandconductsitevisitswhenpossibletodeepentheirunderstandingofthedataPresentationoftheanalysisshouldfocusononeideaatatimetokeeptheconversationfocusedonkeypointsandspecificopportunitiesforimprovementYoumayconsiderdevelopingapublicfacingdashboardtosharethisprogressmorebroadly
AgeneralcomparisonofthestatusquodatawithwhatwasobservedduringthepilotcanprovidedecisionmakerswithaquickunderstandingofthedifferencesbetweentheoldandnewapproachesalthoughthiswillnotshowcausalityOftenthesecomparisonswillincludeoperationalmetricssuchasdaysordollarssavedbutrememberthattheimpactonpeoplewillbemostimportantIfyouareworkingwithanevaluationpartnertheywillsupportthisstep
Usethespacebelowtosummarizetheresultsofyouranalysis
MeasureTargetvalue(seesectiononsettingtargetsabove)
Pilotvalue(whatdidyouobserveduringthepilot)
Historicalaverage(ifavailable)
Results(WhoisbetteroffIsanyonenegativelyimpacted1-2sentencesreassessingyourgoalstatement)
DocumentyourprocessandprogressincludingfeedbackfromtheparticipantsimplementationpartnersandtheprojectteamDidanythingunexpectedoccurduringthepilotArethereinternalandexternalfactorsthatmayaffectyouroutcomesThisiswheretheimplementationoutcomescanbecanhelpfulIftheinterventiondidnotgoasplannedconsiderwhetheritisbecausetheinterventionwasnotacceptableorfeasible
UsethetemplatebelowcreatedbyDenversPeakAcademytocapturethisinformationIfthe306090daytimelinedoesnotmakesenseforyourcityfeelfreetocustomizeit
CurrentState
FutureState
30Day
60Day
90day
Whatwentwell(+)
Whatdidnotgowellshouldbechanged(-)
IfaquestionordiscussionpointarisesthatcannotbeansweredinthemeetinganassignedstaffmembershouldcapturethequestionduringthemeetingAfterwardthefollowupitemsshouldbesenttorelevantpartnersandthenextmeetingshouldbeginwithadiscussionof
Establishaframeworktomonitorprogress
38
thoseitems
Takeaction
DiscussionsaboutwhetheryourindicatorsaremovingintherightdirectionandwhatmightbedrivingtrendsmayleadtoadiscoverythattherearenotenoughresourcesallocatedtothispolicyorissueTheremaybelimitedfundingforstaffprogrammingortechnologythatisdrivingthedatainthewrongdirectionIfCityleadershipisintheroomthisisagreattimetomakethecaseforadditionalfundingduringthebudgetseasonBycollectinganalyzinganddiscussingdataonthispolicyroutinelyyouarebuildingevidencetomakethecasefortherightleveloffundingtoreachtheestablishedtargets
Establishaframeworktomonitorprogress
39
Assessdataaccessibilityqualityandsecurity
Determineaccessibilityandqualityofdata
OnceyouhavecompiledyourwishlistofindicatorsitistimetodeterminetheaccessibilityandqualityofthedataTopreventcomplicationyoumaywishtoprioritizerelevantindicatorsforwhichyouarealreadycollectingdataAvoidthetemptationofadoptingatechnologysolutiontocollectnewdataatthispointsuchasaphotosurveytoolorotherappFornowyouaretryingtodeterminewhichdatasetsarerelevantaccessibleandhighqualityBesuretoworkwithyourkeyinternalpartnerswhileyouareidentifyingyourmeasuresDepartmentstaffwhoworkonprogramsarefamiliarwiththeindicatorsthatwillaccuratelysupporttheoutcomesanddatabaseadministratorswillbevaluablewhendiscussingaccessibilityandqualityofthedata
Abrieflistofconsiderationsfortheaccessibilityofyourdataisbelow
ArewecurrentlycollectingthisdatasetorwillwebecollectingitforthefirsttimeHowfarbackdoestherecordgo(timehorizon)WouldweneednewtechnologytocollectoranalyzethedataCanthedatabecollectedmanuallyorcanitbeautomatedHowfrequentlyisthedataproduced(continuouslyhourlydailyweeklymonthlyquarterlyyearlyetc)IsthedataavailablefortheentireCitypopulationorasubsetofthepopulationHowgranularisthedata(individuallevelCensusblockneighborhoodcitywideetc)DoesthedatacontainsensitiveinformationIsthereaproxymeasurethatwecanusefornowIsthedatamachinereadable
AsnotedbytheWilsonSheehanLabforEconomicOpportunities(LEO)attheUniversityofNotreDameleveragingadministrativedatacansignificantlyreducethecostsandtimelinesofrigorousevaluationCitiescanestablishaninternaldatasharingagreementacrossdepartmentsincludinglanguagethatenablesadministrativedatatobeusedforresearchpurposesCitiescanalsoidentifyexternaladministrativedata-holderssuchasacountyschooldistrictpolicedepartmenthospitaloranotherstatethatmayhaveoutcomedataimportantformeasuringprogramimpactCitiescanleverageexistingrelationshipsandorworkwithresearchinstitutionstodevelopstreamlineddatasharingprocessesandagreementswiththeseexternalentities
Assessdataaccessibilityqualityandsecurity
40
Evaluatethequalityofyourdata
QualityisasimportantasaccessibilityToevaluatedataqualityyoumaywishtocreatearubricliketheonebelow
Value ValueDescription
3 ExcellentDataQualityThedataiscompleteaccurateandupdatedregularly
2 AdequateDataQualityThedataissomewhatcompleteaccurateandupdatedfairlyregularly
1 PoorDataQualityThedataisnotcompleteaccurateorupdatedregularly
ThisisagreattimetobuildonthedatainventorythatyoustartedinthegatherinformationpageSuggestedadditionalfieldstoaddtoyourdatainventorytemplatearebelow
Program DatasetTitle
DatasetDescription DataSource
Departmentorexternalpartner
DoestheCityalreadycollectthisdata
Whatisthenameoftheprogram
Human-readablenameofthedataset
Shouldbeunderstandablebynon-technicalusers
Whatistheoriginalsystemapplicationfilethathousesit
Departmentorexternalpartnerthatmaintainsthedata
Isthisdataalreadycollected
Updatefrequency-ThefileformatforthedatasetTypicallycsvxlsxshapefiletxtetc
Format-Howoftenthedatasetisupdated
Timerange-Theperiodoftimecoveredinthisparticulardataset
Datause-Howisthisdatasetcommonlyused
Datausers-Whotypicallyusesthisdataset
Dataownerstitle-Titleofthedataowner
Dataownersposition-DataOwnersjobtitle
Dataownersemailaddress-DataOwnersemailaddressWhomanagesthedataandorisresponsibleforgrantingpermissiontoaccessthedataWhounderstandswhatthedatasetincludesandcananswerquestionsaboutit
Dataqualityconcerns-Describetheconcernsyoumayhaveaboutthequalityofthedataset
Assessdataaccessibilityqualityandsecurity
41
Division-DivisionwherethedataismaintainedIfmorethanonedivisionisresponsiblelisttheprimaryownerofthedataset
Sensitivedatacomments-DescribetheissuesthatwouldpreventthecityfromsharingthedatapubliclyCommonsensitivityconcernsincludeprivacyviolationssecurityissuesorhighcostorstaffdemandstopostthedata
Publishingstatus-Thistheplacetoputanycommentsregardingthecurrentstatusofthedataset
Publishedlink-Thisisthelinktothepublisheddatasetontheopendataportalorcitywebsite
Reviewdatasecurityandprivacyprotectionprotocols
OftenapolicychangealsoinvolvesachangeinthedatathatisbeingcollectedhowthatdataisbeinghandledwhohasaccesstoitandahostofotheradjustmentsAsyoutakestockofthedataassetsthatareavailableforyourinitiativeitisvitaltoalsothinkthroughwhatwillbedonetoprotectthatdataandtheprivacyofthepeopleitrepresentsAbigpartofgooddatasecurityinvolvesstrongdatagovernancepracticesthatdefineandidentifysensitivedataandconsciouslyworktobalancethevalueandriskofstoringsuchsensitivedataOnewayofevaluatingyourdatacomesfromcaselawrelatedtotheUSFreedomofInformationActwhichwehavetranslatedtothechecklistbelow
1 Woulddisclosureresultinasubstantialinvasionofprivacy2 Haveweconsideredtheextentorvalueofthepublicinterestandthepurposeorobject
oftheindividualsseekingdisclosure3 Istheinformationavailablefromothersources4 Wastheinformationgivenwithanexpectationofconfidentiality5 Isitpossibletomouldreliefsoastolimittheinvasionofindividualprivacy
TheseriskscouldbemitigatedusingavarietyofstrategiesfromaggregatingdatatoobfuscatingindividualstoredactingfieldsthatmayincludePersonallyIdentifiableInformation(PII)orre-evaluatingyourdatacollectionmethodstofindalternativewaysofgettingtheinformationyouneedThesestepsshouldbecodifiedandmanagedbyadatagovernancecommitteeifpossibleForthesensitivedatathatisdeemedessentialcreateaplanaroundpotentialdatabreachesinordertoquicklyandconfidentlyrespondtoemergencysituations
ProactivelypreparingforthesesituationsisoftenthebeststrategyfordatasecuritybutdonotforgetthetechnicalsideInvolveyourITadministratorsinconversationsaboutdatastorageandaccessensuringthattheywillbeprotectedbyenterprise-gradecybersecuritysystems
Assessdataaccessibilityqualityandsecurity
42
Assessdataaccessibilityqualityandsecurity
43
Designimplementationandevaluation
Designtheimplementation
NowthatyouhaveyouraccountabilityplanandkeyindicatorsitistimetogetspecificaboutyourplanforimplementationThetemplatebelowwillhelpyouthinkthroughthespecificmechanismsofimplementingpolicychange
UnderstandingorganizationsandsystemsiscriticaltounderstandingandmakingdecisionsaboutwhereandhowinterventionsshouldbeappliedApolicyimplementedinasinglesettingmayshowpromisingresultsbutimplementingmultiplepoliciesandprogramsatthesametimewithdifferentactorsresponsibleforeachaspectcanhelpensuresustainablebehaviorchange
InSomervilleMAShapeUpSomerville(SUS)isacitywidepublichealthprogramthatisimplementedacrossmultiplelevelsofinfluenceandsystemsthroughinterventionsbeforeduringandafterschooltoincludechangestobuiltenvironmentsuchasbikelanesregionalmasstransportationandphysicaleducationequipmentinschoolsandgymstoincreasephysicalactivitywellnesspoliciesandschoolgardensnutritioneducationduringandafterschoolandhealthyrestaurantprogramstosupporthealthyfoodandhealthyeatinghabits
Usethetemplatebelowtoplanyourimplementation
Policyimplementationplanningtemplate
ListthespecificdesiredoutcomesInthepreviousstepyouworkedwithdepartmentstafftoestablishspecificmeasurableoutcomesandinterimindicatorsPrioritizetheindicatorsthatwilldemonstrateprogressUsethespacebelowtocapturethedesiredoutcomesofthepolicychange
ExampleShapeupSomervillebeganwiththegoalofpreventingobesityin1stthrough3rdgraders
Yourturn
____________________________________________________________________________
____________________________________________________________________________
Designimplementationandevaluation
44
SpecifythetargetpopulationAreyoudesigninganinterventionthattargetsonlycertainsitesAreyouplanningonimplementingaprogramorinitiativeforaspecifictargetaudienceClearlyspecifythepopulationyouplantotarget
IfyouplantoconductarigorousimpactevaluationasnotedbytheWilsonSheehanLabforEconomicOpportunitiesitisimportanttoestablishconcreteobjectiveeligibilitycriteriaandensurethatthesecriteriaareappliedconsistentlybyallstaffduringscreeningandintakeItisalsovitaltoidentifytheindividualswhoareeligibleforaparticularprogramorserviceevenifeveryoneeligiblecannotbeservedbyexistingresourcesThishelpscitiesidentifyanyunmetneedsandguidefutureresourceallocationwhilealsocreatingthepossibilityofarigorousimpactevaluationbyidentifyingapotentialcomparisongroupFinallywhenconductingscreeningandintakeitisimperativetocollectandmaintainbasicinformation(egnamedateofbirthgender)aboutallprogram-eligibleindividualsnotjustthosewhoparticipateintheprogram
ExampleIntheShapeUpSomervilleexampletheprogrambeganbytargeting1stthrough3rdgraders
Yourturn
____________________________________________________________________________
____________________________________________________________________________
DefinethelocationWherewillthepolicychangetakeplaceWillimplementationtakeplaceatonesiteormultiplesites
ExampleShapeUpSomervilleinterventionsweredeliveredinschools
Yourturn
____________________________________________________________________________
____________________________________________________________________________
ClarifyimplementationpartnersandarticulatetheirrolesWhatarethekeyorganizationsinstitutionsorpeopleresponsibleforimplementationTheprojectteamshouldbeasclearandspecificaspossibleinprovidingadescriptionoftheconcreteactivitiesthatimplementationpartnersaremeanttocarryoutPracticeprofilesmaybeahelpfultooltohelpdefinewhatisrequiredoftheactorswhoareresponsibleforimplementingtheintervention
Designimplementationandevaluation
45
ExampleInSomervillemayoralleadershipandengagementofmultiplecityagencies(publichealthprimarycareschoolsCambridgeHealthAllianceetc)streamlinedandconnectedthevariousShapeUpSomervilleinitiativesandgrantswhichhelpedtokeepthetalentandenergyfocusedonhealthyeatingandactivelivingforallsectorsofSomervilleSUSpartnersultimatelyincludedschoolsnonprofitsCityandstateagenciesandnationalfoundations
Yourturn
____________________________________________________________________________
____________________________________________________________________________
DescribetheinterventionWhatarethestepsactivitiesorprocessesRememberthatthemechanismthespecificstepsorprocessthatleadtobehaviorchangeiskeytounderstandingtheeffectivenessofthepolicychangeandsharingtheresultswithothers
ExampleGoalsofShapeUpSomervilleweretoinfluenceeverypartofanearlyelementaryschoolchildrsquosdayndashbeforeduringandafter-schooltopromotehealthyeatingphysicalactivityandtheirweightstatusInterventionincludedfoodserviceenhancementstoimprovebreakfastandlunchandstaffprofessionaldevelopmentwalktoschoolactivitiesSUSclassroomcurriculumandprofessionaldevelopmentschoolwellnesspolicydevelopmentSUSafter-schoolcurriculumandprofessionaldevelopmentoutreachandeducationtohomethroughmaterialsforumseventsldquoSUSapprovedrdquorestaurantscommunityoutreachandcapacitybuildingthroughpolicydevelopmenttrainingsmediaplacements
Yourturn
____________________________________________________________________________
____________________________________________________________________________
ClarifytimelineandmilestonesfordeliveryandevaluationHowfrequentlywilltheservicesinterventionbedeliveredHowlongwilltheprogramrunWhatisthetimetoexpectedimpacts(eg12monthsafterprogramcompletion)
Designimplementationandevaluation
46
ExampleShapeUpSomervilleledbyTuftsUniversityFriedmanSchoolofNutritionScienceandPolicyconductedaquasi-experimental(nonrandomized)interventionstudyamong1stthrough3rdgradestudentsinSomerville(intervention)andtwocomparisoncommunitiesfor9-monthsduringoneschoolyear(fall2002ndashspring2003)ThestudyshowedpositiveassociationoftheinterventiononchangeinBMISUSreleasedreportssummarizingevaluationsprogresstodateplansforadditionalprojectsandimpactstenandfifteenyearsintotheinitiative
Yourturn
____________________________________________________________________________
____________________________________________________________________________
PlanyouranalysisTheprevioussectionaddressedestablishinganaccountabilityframeworktomonitorprogressandimplementationDeterminewhichindicatorswillbediscussthroughthatprocesswhowillsharethedataandanalysisandhowfeedbackfromthoseconversationswillbeincorporatedbytheprojectteam
ExampleTheMayorrsquosofficeimplementedadata-drivendecisionmakingapproachcalledSomerStatwhichwassynergisticwiththeapproachestakenbytheresearchandevaluationteamsdeployedforthespecificgrants
Yourturn
____________________________________________________________________________
____________________________________________________________________________
PlanforsustainabilityIfthepolicyandprogramchangeissuccessfulhowwilltheprojectteamsustainandexpandtoadditionalsitesorpopulations
ExampleAlthoughmostofthepartnershipsforgedintheearlyyearsweremaintainednewgrantsandcollaborationsexpandedattentiontoolderpublicschoolstudentsactivetransportationandtheGreenLinesubwayextensionthebuiltenvironmentfarmerrsquosmarketeffortssuchasldquoGrowninSomervillerdquowithUnionMainStreeteconomicdevelopmentandhealthdisparitiesaswellasdatacollectionmonitoringandsurveillance
Yourturn
Designimplementationandevaluation
47
____________________________________________________________________________
____________________________________________________________________________
Pilotplanningtemplate
IfyouplantoconductapilotfirstdeterminethepurposeofthepilotDoyouwanttoobservewhetherthepolicychangeresultsinthedesiredbehaviorchangeAreyouconcernedaboutfeasibilityscalabilityorcostDoyouwanttotestwhetheryourdesiredresultscanbeachievedthroughthispolicyapproachClarifythegoalofthepilotandthequestionsyouhopetoanswerThiswillhelpyouidentifythekeyindicatorsthatyouwillusetomeasurewhetherthepilotissuccessfulBeforebeginningestablishabaselineofcurrentperformanceandsetyourtargetsaccordinglybasedonwhatyouhopetoaccomplishthroughthepilot
NextestablishthepilotplanforyourprojectteamincludingmilestonesforteamcheckinstodeterminewhethertomoveaheadorchangecourseThisisseparatefromthemeetingsthatwilloccuraspartoftheaccountabilityframeworkUsethespacebelowtocreateyourtimelineanddevelopyourmeetingschedule
Task Personresponsible Startdate Checkindate Deliverable
Planyourevaluation
WhileyouaredesigningtheimplementationitisalsotimetobegintodevelopyourplanforanalysisAsnotedbytheWilsonSheehanLabforEconomicOpportunitiesdecisionsaboutprogramsetup-includingoutreachscreeningintakedatatrackingandenrollment-candeterminewhetherornotarigorousimpactevaluationwillbepossibleAsdiscussedcityleadersmayfindithelpfultoincludeevaluationpartnersinearlyconversationsaboutprogramsetupDuringpreviousstagesyoumayhaveworkedwitharesearchertoplanyourproposedpolicychangeYoushouldbeclearaboutthepurposeoftheevaluationanddevelopanevaluationplanBelowareadditionalconsiderations
Causationvstrendanalysis
IntheEstablishaframeworktomonitorprogressstagetherewasguidanceontrackingtrendsovertimeItisimportanttonoteherethatmonitoringdataisdifferentfromanalyzingacausaleffectTrendanalysiswillillustratethatchangesareoccurringandmayleadtoquestionsthatcouldbeansweredbyyourevaluationbuttrendanalysisalonedoesnotprovideevidenceaboutwhatisdrivingthosechangesYoumayseechangesinyour
Designimplementationandevaluation
48
outcomeindicatorsbasedonvariablesoutsideoftheCityscontrolForinstanceachangeinfederaltradepolicycanhaveanimpactemploymentratesinyourCityoutsideofyourCitysnewjobtrainingandplacementprogram
ToaddresscausalityyoumustcomparetheoutcomesofindividualswhoreceivedtheinterventionwiththosewhodidnotWithoutacomparisongroupofsimilarindividualswhowerenotexposedtotheinterventionitisimpossibletoknowwhichfactorsareresponsiblefortheoutcomesRandomizedControlTrials(RCTs)inwhichindividualsarerandomlyassignedtotreatmentandcontrolgroupsarethegoldstandardinprovingcausalityIfyoucandemonstratethatoutcomesimprovedforthosewhoreceivedtheinterventionanddidnotimproveatallorimprovedlessforacomparisongroupyoucanbesomewhatcertainthattheinterventionwasthecause
Examplesofothernon-causalimpactevaluationsthatarerigorousandreliableincludecertainquasi-experimentalstudiesusingcomparisongroupdesignssmartinterruptedtimeseriesanalysesthatmodeltrendsovertimeinwaysthatallowestimationofinterventioneffectswelldesignedpre-posttestanalysesneedsassessmentsimplementationstudiesetcFormoreinformationondifferenttypesofevaluationsseetheSageResearchMethodswebsite
Lookfordifferencesingroups
BuildintimetoalsoexaminethepilotdatatoseeifgroupsareexperiencingdifferentresultsReviewingperformancebasedonoutcomemeasuresforapopulationinaggregatemayhidethesubtlewaysthatsomepeoplemaybeaffecteddifferentlysoitisnecessarytocarefullyanalyzethedatafordisparateoutcomesForinstancetheMovingtoOpportunity(MTO)studythatexaminedtheimpactofhousingvouchersonfamilieswholivedinhigh-povertyneighborhoodsfoundthatchildrenwhomovedbeforetheageof13hadbettereducationandhigherincomesasadultsHoweveradultsdidnotseeimprovededucationandemploymentoutcomes
Disaggregatingdataintomorespecificcategoriesofraceandethnicitycanhelpunderstandtrendsanddisparitiesinamoregranularwaywhichcanhelpdecisionmakerstargetpoliciesandinterventionsinamoreequitableway
EarlierintheRoadmapweworkedtoidentifystakeholdersandwhomightbepositivelyandnegativelyimpactedbyachangeinpolicyUsetheresultsofthatexercisetoinvolvethecommunityininterpretingdataanalysisandtogatherqualitativefeedbackfromthecommunitytocallattentiontoissuesForinstanceanincreaseinyouthdetentionmaybeexplainedbyinefficienciesorlackofaccesstotransportation
Designimplementationandevaluation
49
Designimplementationandevaluation
50
Shareyourstoryandsuccess
Sharetheresults
Storytelling
AnintegralstepinyourstrategyissharingtheresultsofyourpilotorpolicychangewithyourcommunityThisincludesdevelopingmessagingaroundtheworkofyourorganizationandcreatingaplantodisseminateyourstoriesWhytellstoriesaboutyourwork
Informandorunderstandasubjectorsituation
Giveneededcontextofdataorbitsofinformation
Relatablestoriesarememorableforreadersviewers
ReadersorviewerswillseetheinformationasmorecompellingwhenthereisademonstratedandrelatablechangeorimpactonthemselvesortheirfriendsneighborsandcolleaguesStoriesareaboutpeopleandgovernmentisfundamentallyabouthelpingpeopleSotellthestoriesofthepeopleyouandyourcitycolleagueshelped
Keepthesetipsinmindasyoudevelopyourstorywithyourteam
WhatwereyourfindingsandwhyshouldanyonecareTellyouraudiencehowtheproblemwassolvedWhathastheimpactbeenonresidentsandcitystafftodateWhatwasthesuccessandweretheremultiplesuccessesalongthewayIfyouhavepreliminaryresultsordatashowingimprovementtosharethisisthetimetodoit
WhowasinvolvedandimpactedReviewthelistofstakeholdersHowwereeachofthemimpactedbytheprogramHumanizeyourstoryandimpactbyaddingadditionalvoicesThesevoicescanbeoneofthefollowing
Apersonorfamilywhowaspositivelyaffectedbythechangesuccess
Aninvolvedthird-partydescribingboththeproblemandthesolution(ieanorganizationnonprofitprovidersbusinesstradeassociationsportsteametc)whoispartofthecommunitythatthegovernmentworkedwithonthechangesuccess
Someonefromthegovernmentitselfwhocanspeakaboutthechangesuccess
TellmoreaboutyourcityDescribeyourcityincludingpopulationdemographicsindustriesetcWhatareitschallengesWhatisitknownforWhatmakesituniqueHowisitsimilarordifferentfromothercitiesintheUS
Shareyourstoryandsuccess
51
DescribeyourcityrsquoschallengeWriteoutthechallengeyourcityisfacing(asdefinedinyourproblemstatement)HowdoesitaffectresidentsHowdoesthisproblemaffecttheworkofcitystaffmembers
AddthedetailsProvidedetailsonyourappliedresearchstrategyTalkaboutwhatyourcityhasdonenthepasttoresolvetheissueDiscusshowyourstrategyhaschangedwhenyoubegantoadoptthisapproachWasmoneysavedorinvested
Usethisstorymapexercisetoorganizeyourthoughtsandguaranteethatyouincludealloftheelementsofatraditionalstory
AfteryoudevelopyourmessagingcreateacommunicationsmarketingandoutreachstrategydesignedtospreadyourmessageYourstrategyshouldwalkyouthroughthefollowingkeyactivities
1 IdentifyandprofileyouraudienceWhencreatinganddisseminatingyourmessageitisalwaysgoodtothinkaboutyouraudienceWhoexactlywouldbeinterestedinwhatyouhavetosayThegeneralpopulationofthecityPeoplewhoworkinnonprofitsBusinessownersItmattersbecausenoteveryonewholivesinacitymaybethetargetoftheinformationBusinessownersmaybemoreinterestedindatashowingeffectsonthebusinessdistrictthanothersandsoonWewillcoverthislaterbutthisstepwillalsohelpyoutothinkofthemosteffectivewaytoreachyouraudience
2 Defineyourcalltoaction(ifyouhaveone)SpendsometimeconsideringwhyyouareinterestedintellingyourstoryDoyousimplywanttoupdatethepubliconyourprogressorcelebrateyoursuccessesAreyouinterestedinyouraudiencechangingitsbehaviororsupportingaspecificinitiativebeinglaunchedbyyourorganizationIfyouwanttohaveyouraudienceactorbehaveinacertainwaybespecificandincorporateitinyourmessaging
Shareyourstoryandsuccess
52
3 SelectperformancemeasuresThinkabouthowyouwoulddetermineifyourcampaignissuccessfulandselectappropriateperformancemeasuresYouwillrevisitthesemeasuresattheendofyourcampaigncommunicationsplanimplementationtodeterminewhetheryouneedtomodifyyourcommunicationsandoutreachstrategyforthefuture
4 MethodstocommunicateyourstoryTherearemanywaystocommunicatesuccessofyourprojectHerearesomesuggestionsonreachingaudiencesItisimportanttodeterminewhichmethodsarethemosteffectiveforyourdesiredaudiencetoreceiveinformationDoyourresearch(PewResearchCenterhasexcellentstudiesonthereachofmediadigitalmediaandmarketing)ForexampleifseniorsareyourtargetpopulationandtheyarelessinclinedtouseInstagram(thatistrue)youmaywanttoconsiderFacebookorthenightlylocalnewsbothofwhichareusedmoreoftenbyseniorsBelowisalistofsomemethodsandtoolsyoucanuse
i Pressreleasestolocalnewspapersandtvstations
ii ShortsmartphonevideosAgreatwaytoengageanaudienceisthroughvideoAskonequestionofsomeonewhoispartofthestoryMaybethereisasocialworkerwhocandiscusshungerinyourcityThesevideoscanbereusedonmultipleplatforms
iii Long-formwriting
iv TellingindividualstoriesthroughaseriesofphotosandshortnarrativesforpublicationonFacebookorInstagram
v Op-edsopinionpiecesplacedinthelocalpaper
vi Emailupdatestoyourlist
vii Connectingwithappropriateorganizationsthatareinvestedinthetopicandworkwiththecommunityandaskthemtodisseminateinformationtotheirmembers
viii EmployingldquoWordofMouthrdquo(WOM)tacticsbycreatingtownhallsorfreeeventsforresidentstoattendintheirownneighborhoods
5 DeterminethetypeofcollateralUsuallycampaignsthattargetresidentsrequireseveralavenuestoreachthemPeoplearenotalwaysinthesameplaceorspaceHavingakeenunderstandingofyouraudiencewillassistyouwithyourchoicesonwhatneedstobedevelopedSometimesthatmeansimagesforsocialmediaaonepageflierthatgivespeopleinformationonattendinganeventoranin-depthbrochure
Shareyourstoryandsuccess
53
6 ImplementyourstrategyKeepinmindstrategiescanbefluidSometimessomethingmaynotturnoutasexpectedevenifduringplanningsessionsitwasldquosupposedtordquoItisokYoucananalyzewhathappenedduringadebriefsessionandonceyoursurveysandothermeasurementshavebeenfinalized
Visualizations
Withtherightvisualizationyoucanreviewandshareinformationinsightsanalysisperformanceandtrendsallofwhichisvaluabletoacross-sectionofthepublicButwhichvisualizationisrightforyouTochoosetherighttypeofchartgraphdeterminethetypeofquestionyouwanttoanswerSomecommoncharttypesarebelowandmoreinformationcanbefoundhere
Exploreadditionalanalyticaltechniques
Duringthegoal-settingphaseyouestablishedoutputoutcomeandimplementationmeasuresBasedontheresultsofyourpilotyoumayneedtoadjustyourmeasuresWiththepilotdatainhandyoumayhavesomeideasofhowfutureanalysescouldbedifferentandpotentiallyimprovethequalityofyourprogramTakesometimetobrainstormotherrelevantdatasetsthatyoudidnothavethetimetopursueearlierincludingmessydepartmentaldatathatneedsmoreattentionorsourcesofpublicdatafromthestateandfederalgovernmentnonprofitsandevenprivatecompaniesthatcouldsupplementyourunderstandingofanissueUsingmachinelearningtechniquesyoumightbeabletobetter
Shareyourstoryandsuccess
54
predictdemandforhomelessservicesusingNOAAweatherforecastspreemptwatermainbreaksusingsoilstratigraphyorinvestineconomicdevelopmentbymonitoringrealestatelistingdata
Advancedanalyticaltechniquescanalsohelpyourimplementationrunmoreeffectivelybyintegratingdata-informedtargetinganddecisiontoolsintotheprocessByautomatingroutineworkflowsandsurfacingsubtletrendsthesetoolscanhelpyouiterateinamoreprecisemannerandprovidekeystrategicguidancetofront-lineemployeesthatwouldotherwisebeunavailableThesesortsofdata-science-enhancedprocessescanbeclassifiedasoneofsixprojecttypesasexploredfurtherbytheNewOrleansLANOLAlyticsteamFormoreguidanceonscopingasimilarprojecttheGovExadvancedanalyticsscopingguidegivesyouastep-by-stepoverviewofhowtobesuccessful
Usethespacebelowtolistyourideasforpossibleadvancedanalyticsprojectsbytype
Typeofproject Ideas
Findingtheneedleinthehaystack
Prioritizingforimpact
Earlywarningtools
Betterquickerdecisions
Optimizingresourceallocation
Experimentingforwhatworks
Keepthefeedbackloopgoing
GatherinformationonpublicsentimentaboutpolicychangethroughfocusgroupssurveysandpublicmeetingsledbyadvisorygroupmembersNotewhethernewinformationisavailablethatcanhelpanalyzeorcontextualizeyourprogress(egfederalorstatedatareports)Sharetheresultsofyourpilotandthepolicychangeprocesswithothercitiesandresearchers
Shareyourstoryandsuccess
55
WorkingwitharangeofevidencePresentlythereisalotofattentionandinvestmentinevidence-basedpolicymakingincludingthefederalbipartisanCommissiononEvidence-BasedPolicymakingthePewCharitableTrustsreportsandtoolkitsonHowStatesEngageinEvidence-BasedPolicymakingandtheLauraandJohnArnoldFoundationsupportedEvidence-BasedPolicymakingCollaborativeManyoftheseinitiativesfocusonrigorousevaluation--meaningrandomizedcontrolledtrials(RCT)orotherexperimentalmodels--asthebestkindofevidencetouseHoweverthisbodyofevidenceisofteninsufficientforthespecifickindsofquestionsthatgovernmentscanhavewhenmakingadecisionaboutpolicyInthisguideweproposeanapproachtosupportgovernmentsinincorporatingadditionaltypesofevidenceintotheirdecisionmaking
OnewaytounderstandarangeofevidenceistouseNestarsquosStandardsofEvidencetoframeanddefineevidenceTheBritishnonprofitNestaworkstoinjectpositivechangeintothewayUKgovernmentpoliciesandprogramsfunctionOneoftheirkeyareasofworkhasbeendevelopingWhatWorksCentresorganizationsthatcollectcommissionsynthesizeanddisseminateevidenceaboutinterventioneffectivenessacrossarangeofsocialimpactareasNestadevelopedthefollowingStandardsofEvidencetoprovideausefulwaytodocumenthowdifferentkindsofinformationdemonstratedifferentlevelsofevidence
Workingwitharangeofevidence
56
Theimportanceofpotentiallybiasedinformation
Onechallengepresentedbyanevidenceclassificationthatacknowledgesthevalueofnon-scientificformsofevidenceistofigureouthowtobalanceallofthepiecestogetherwhenyouusethemTounderstandhowtodothiswehavetothinkabouthowtopositionpotentiallybiasedinformationindecisionmaking
WhenwelookatinformationwhichhasnotbeenrigorouslyevaluatedweacceptthatthisinformationmightbebiasedThewholereasonweconductrigorousevaluationistoremovebiasfromourobservationsWeconductstructuredtrialstoproduceresultsthatareconsistentandgeneralizableIfafindingisunbiasedthisaccuracymdashthislackofbiasmdashcanbeprovedbecauseanyindependenttestercanreproduceitwhentheyfollowanidenticalprocess
YoucanlookatpotentiallybiasedinformationasjustinformationthatisearlieroninitspathtorefinementreplicationandverifiedaccuracyApieceofinformationbasedonanyoneindividualrsquosobservationsislikelytobebiasedbecauseitiscomingfromjustonesourceBiasisreducedbygatheringtogetherlotsofindependentsourcesofinformationandfiguringoutitsmostcommonattributes
MaybethebiasedinformationmakesaclaimthatistoolargemdashitsayssomethingistrueunderallconditionswhenitisreallyonlytrueundercertainconditionsorthataneffectofaninterventionisbiggerthanitstrueaverageorthataneffectlastsovertimewhenitdoesnotPerhapschangingtheclaimtobemorespecificmdashcriticallyreviewingtheinformationtoseeifspecificconditionsmustbemettoachievetheoutcomesmdashwouldmaketheclaimmoreaccurate
IfyoutestedyourclaimfounditdidnotholdupinsomecaseschangedittesteditagainuntilyourclaimrepeatedlyheldupthenyoucoulddemonstratethattheclaimwasnolongerbiasedbutaccurateThisprocessoftestingimprovingtestingandvalidatingiswhatwegetfromstructuredscientificstudy
Withindividualpiecesofinformation--crediblehypothesescasestudiesexpertopinion--youcannotassumethatkindofaccuracyAtthesametimetheseearlyobservationsarefoundationaltoanyreplicablescientificobservationsWewillnevergetanywhereifwedonotstartwiththesepiecesofinformationeveniftheymightbebiased
Combininglevelsofevidencetogetthemostdetailedinformationforyourcity
ItisespeciallyimportanttoacceptdifferentformsofevidencewhenyoufacequestionsthathavenotyetbeentestedbyindependentstudiesSomeofthemostimportantquestionsrelatetothingsthatareveryspecifictoyoursituationYoufaceaparticularcombinationof
Workingwitharangeofevidence
57
issuesactorstrendsandresourcesthatisultimatelydifferentfromeveryotherplaceintheworld
ThisisimportantbecauseaspolicypractitionersandnotscientistsyoucaremoreaboutyourspecificcasethanahypotheticalaveragegeneralcaseYoucarelessaboutthetotallikelyrangeofimpactsfromapolicythanhowthissingleimplementationwillworkYouwantinformationaboutpoliciesthatwillprovideyouwiththebestpossiblepredictionabouthowtheywillworkinyourcontext
ByputtingdifferentkindsofevidenceintothesameframeworkNestarsquosStandardsofEvidenceimplythatwhenyoucannotgetrigorouslytestedexperimentallyverifiedinformationitisappropriatetoworkwithwhatwedohaveThisisrelevantforthemunicipalimplementationcontextbecausenostudypulledfromaliteraturereviewwillaccountforallofthevariablesyoufaceinyourowncontext
Intheabsenceofhavingastudythatpredictshowaninterventionworksinyourspecificcontextyouwillneednon-scientificinformationtotrytofillinthosegapsYouwillneedtogatherstoriesfromresidentsstaffmembersandotherstakeholdersthatsitonLevels1or2ofthestandardsofevidence
Someofthekindsofinformationyouwillprobablygetmoreeasilyfrompeoplethanstudiesinclude
howpeopleincitieslikeyoursreactedtoaninterventioninpublicforumshowpeoplewhoimplementedaninterventioninacitylikeyoursfiguredouthowtogetitpassedhowthecitycommunicatedpubliclyabouttheapparentbenefitsanddrawbacksoftheinterventionthepoliticalacceptabilityoftheintervention
Youcanremainsimultaneouslyawarethattheinformationyouaregatheringthroughindividualstoriesisbiased--notgeneralizableinformedbyspecificperspectives--whilealsorecognizingthathavingsomeinformationaboutsomethingimportantisbetterthannoinformationEvidencefromthefirsttwoevidencelevelsareessentialbuildingblocksofknowledgeandperfectlyusefulwhenyoucannotgetdetailedunbiasedknowledge
Youhavetostartsomewhereandyoushouldalwaysstartwithevidenceyouhaveaboutyourveryspecificconditionsthatyouknowareimportantforsuccess
Workingwitharangeofevidence
58
Thankyou
Interviewswithcities
GovExheldsemi-structuredinterviewswithsixstaffmembersinfourcitiestoestablishtheinitialstepsoftheRoadmapWeaskedrespondentstotalkaboutaninitiativeorpolicythattheyhavebeenresponsibleforimplementingintheircityRespondentsfromthreecitiesspokeabouttheireffortstoaddresshomelessnessandrespondentsfromonecityspokeaboutblightAfterrespondentsidentifiedtheproblemtheyspokeabouthowthecityprioritizedthisissuethevariousstepsinvolvedandbarrierspresent
GovExconductedtextanalysisoftheseinterviewswhichrevealedthatpeoplewasthemostcommonlyusedtermduringtheseconversationswhendiscussingthestepstochangeaprogramorpolicyThisisathemethroughoutthisRoadmapThefirstsectionbeginswithidentifyingtherelevantpeopleTheconversationsaroundevidencearefocusedonhumanbehaviorratherthanmethodologyorstrengthofevidence
Thankyou
59
OncewehadadraftoftheRoadmapwefollowedupwithcitiesonanindividualbasistolearnmoreaboutwhatwouldbehelpfulWearegratefultothefollowingcitystafffortheinterviewsandindividualconversations
MichaelBaskinChiefPolicyOfficerChattanoogaTN
MelissaBridgesPerformanceandInnovationCoordinatorLittleRockAR
VictoriaCarreoacutenAdministrativeOfficerLasVegasNV
TyroneGrandstrandCommunityDevelopmentPlannerFargoND
TibLaughlinDirectorofGeneralServicesKansasCityKS
TimMorelandDirectorofPerformanceManagementandOpenDataChattanoogaTN
CharlesRosendahlManagementAnalystDentonTX
Thankyou
60
MelissaSiebenAssistantCountyAdministratorKansasCityKS
GinaThompsonDirectorofStrategicServicesLewisvilleTX
JHUFacultycollaboration
GovExalsoconvenedthecitycontactsandfacultypartnersinaseriesofAdvisoryGroupmeetingsThesemeetingsservedtoconnectcitiesandfacultytounderstandoneanothersperspective
StefanBaralBloombergSchoolofPublicHealth
AshleyBernerJHUSchoolofEducation
MelissaDavey-RothwellBloombergSchoolofPublicHealth
JonLinksBloombergSchoolofPublicHealth
TamarMendelsonBloombergSchoolofPublicHealth
KristinMmariBloombergSchoolofPublicHealth
KeshiaPollackPorterBloombergSchoolofPublicHealth
JoshSharfsteinBloombergSchoolofPublicHealth
LizStuartBloombergSchoolofPublicHealth
LindsayThompsonCareyBusinessSchool
DanielWebsterBloombergSchoolofPublicHealth
Partnersindataandevidence-basedpolicy
Wealsospokewithpartnerswhoareexpertsinthefieldofevidence-basedpolicywhosharedtheirknowledgeandresourcesofcurrentandfuturethinkinginevidence-informedthinking
WendyBarrenoAssociateDirectorWilsonSheehanLabforEconomicOpportunitiesUniversityofNotreDame
TiffanyDavenportDirectorofDataPartnershipsBaltimoresPromise
TomiHiersDirectorBaltimoreCivicSiteAnnieECaseyFoundation
ChrisKingsleySeniorAssociateAnnieECaseyFoundation
KathrynOliverAssociateProfessorLondonSchoolofHygieneandTropicalMedicine
Thankyou
61
KristenWatkinsSeniorPolicyAssociateJ-PAL
GovExParticipants
GovExAnalystMiriamMcKinneyanalyzedtheinterviewtranscriptsandconductedtextanalysisSeniorImplementationAdvisorEmilyShawwillintegrateRoadmapactivitieswithadditionalservicedeliveryactivitiesatGovEx
Thankyou
62
ReferencesAbrahamKGampHaskinsR(2017)Thepromiseofevidence-basedpolicymakingReportofthecommissiononevidence-basedpolicymakingWashingtonDCCommissiononEvidence-basedPolicymakingBigmove2Extendingthegreen(nd)Retrievedfromhttpswwwcityoflewisvillecomdoing-businesslewisville-2025big-move-2-extending-the-greenBatesMAampGlennersterR(2017)ThegeneralizabilitypuzzleRetrievedfromhttpsssirorgarticlesentrythe_generalizability_puzzleBuildwithnotfor(2018January)RetrievedfromhttpsbuildwithnotforwordpresscomCairneyPampOliverK(2017)Evidence-basedpolicymakingisnotlikeevidence-basedmedicinesohowfarshouldyougotobridgethedividebetweenevidenceandpolicyHealthResearchPolicyandSystemsdoi101186s12961-017-0192-xCairneyPOliverKampWellsteadA(2016)TobridgethedividebetweenevidenceandpolicyReduceambiguityasmuchasuncertaintyPublicAdministrationReview76(3)399-402Retrievedfromhttpdxdoiorg101111puar12555Centreforpublicimpact(nd)RetrievedfromhttpswwwcentreforpublicimpactorgCernaL(2013)ThenatureofpolicychangeandimplementationAreviewofdifferenttheoreticalapproachesOrganisationforEconomicCo-OperationandDevelopmentRetrievedfromhttpswwwoecdorgeduceriTheNatureofPolicyChangeandImplementationpdfColoradoSprings(nd)Retrievedfromhttpscoloradospringsgovcommunicationspagewhat-expect-when-you-speakupContinuumofcare(coc)program(nd)RetrievedfromhttpswwwhudexchangeinfoprogramscocCouncilpresidentkicksoff$12millionyouthfund(nd)Retrievedfromhttpwwwbaltimorecitycouncilcomcontentcouncil-president-kicks-12-million-youth-fundDurlakJAampDuPreEP(2008)ImplementationmattersAreviewofresearchontheinfluenceofimplementationonprogramoutcomesandthefactorsaffectingimplementationAmericanJournalofCommunityPsychology41327-350Retrievedfromhttpswwwncbinlmnihgovpubmed18322790Fishbonediagram(ishikawa)(nd)Retrievedfromhttpasqorglearn-about-qualitycause-analysis-toolsoverviewfishbonehtmlFivewhysGettingtotherootofaproblemquickly(nd)RetrievedfromhttpswwwmindtoolscompagesarticlenewTMC_5Whtm
References
63
FligsteinNampMcAdamD(2011)TowardageneraltheoryofstrategicactionfieldsSociologicalTheory29(1)1-26RetrievedfromhttpsociologyberkeleyedusitesdefaultfilesfacultyfligsteinFligstein20McAdam20Soc20Theory20PaperpdHeinkelP(2018January12)CitymullspermitsforbikesharingcompaniesRetrievedfromhttpwwwdentonrccomnewsdenton20180112city-mulls-permits-bike-sharing-companiesHowisareamedianincomecalculated(2012October)Retrievedfromhttpswwwhudexchangeinfofaqs843how-is-area-median-income-calculatedLampkinLMampHatryHP(2003)KeystepsinoutcomemanagementUrbanInstituteRetrievedfromhttpswwwurbanorgresearchpublicationkey-steps-outcome-managementMarshDampSmithM(2000)UnderstandingpolicynetworksTowardsadialecticalapproachPoliticalStudies484-21Retrievedfromhttpjournalssagepubcomdoiabs1011111467-924800247McCannL(nd)RetrievedfromhttplaurenellencomMoultonSampSandfortJR(2016)ThestrategicactionfieldframeworkforpolicyimplementationresearchPolicyStudiesJournal45(1)Retrievedfromhttponlinelibrarywileycomstore101111psj12147assetpsj12147pdfv=1ampt=jdkhhcp4amps=e184c625e1855a6b1615ccb99ee27b1c0b3212d8PaynterB(2018January1)What100citiesarelearningfromeachotherbysharingtheirdataRetrievedfromhttpswwwfastcompanycom40518004what-100-cities-are-learning-from-each-other-by-sharing-their-dataPeakacademy(nd)Retrievedfromhttpswwwdenvergovorgcontentdenvergovenmayors-officeprograms-initiativespeak-performancepeak-academyhtmlPinnockHBarwickMCarpenterCREldridgeSGrandesGGriffithsCJRycroft-MaloneJMeissnerPMurrayEPatelASheikhAampTaylorSJC(2017)Standardsforreportingimplementationstudies(stari)ExplanationandelaborationdocumentBMJOpenRetrievedfromhttpbmjopenbmjcomcontentbmjopen74e013318fullpdfProctorESilmereHRaghavanRHovmandPAaronsGBungerAGriffeyRampHensleyM(2011)OutcomesforimplementationresearchConceptualdistinctionsmeasurementchallengesandresearchagendaAdministrationandPolicyinMentalHealth38(2)65-76doi101007s10488-010-0319-7TandonSD(2018)ProgressincommunityhealthpartnershipsResearcheducationandactionRetrievedfromhttpswwwpressjhuedujournalsprogress-community-health-partnerships-research-education-and-actionProvanKGampKenisP(2007)ModesofnetworkgovernanceStructuremanagementandeffectivenessJournalofPublicAdministrationResearchandTheory
References
64
18(2)229-252Retrievedfromhttpsacademicoupcomjpartarticle182229935895SabatierPA(1988)Anadvocacycoalitionframeworkofpolicychangeandtheroleofpolicy-orientedlearningthereinPolicySciences21(3)129-168Retrievedfromhttpwwwjstororgstable4532139SandstromAampCarlssonL(2008)TheperformanceofpolicynetworksTherelationbetweennetworkstructureandnetworkperformancePolicyStudiesJournal36497-524Retrievedfromhttponlinelibrarywileycomdoi101111j1541-0072200800281xabstractSchwartzMampWilsonE(nd)WhocanaffordtoliveinahomeAlookatdatafromthe2006americancommunitysurveyRetrievedfromhttpswwwcensusgovhousingcensuspublicationswho-can-affordpdfShearerJCAbelsonAKouyateBLavisJNampWaltG(2016)WhydopolicieschangeInstitutionsinterestsideasandnetworksinthreecasesofpolicyreformHealthPolicyandPlanning31(9)1200-1211Retrievedfromhttpsdoiorg101093heapolczw052ThelessonsofadministrativedataHigh-profilepolicy-relevantresearchpoweredbyadministrativedata(nd)Retrievedfromhttpswwwpovertyactionlaborgsitesdefaultfilesresources20161114-The-Lessons-of-Administrative-DatapdfToolsandotherneedsforqualitativeandquantitativemethods(2015February)AgencyforHealthcareResearchandQualityRetrievedfromhttpwwwahrqgovprofessionalsquality-patient-safetytalkingqualityassesstoolshtml
WeaverRCampBagchi-SenS(2013)SpatialanalysisofurbandeclineThegeographyofblightAppliedGeography4061-70
References
65
UniversityPartners
Name Title Affiliation Area(s)ofexpertise
Contactinformation
Examplesofworkingwithnon-academicdecisionmakers
WendyPBarreno
AssociateDirector
WilsonSheehanLabforEconomicOpportunitiesUniversityofNotreDame
LEOsfocusareasincludehealtheducationhousingchildwelfareeconomicopportunitycriminaljustice
wbarrenonduduorleondedu
MelissaDavey-RothwellPhD
AssociateScientist
JohnsHopkinsBloombergSchoolofPublicHealth
Health mdavey1jhuedu410-614-5854
BaltimoreCityHealthDepartmentLeadersMarylandDepartmentofHealthAddictionRecoveryServicesinDCExecutiveDirectorsofcommunityorganizationsacrosstheUSthatfocusonHIVSTIsubstanceabuseandimplementationscience
ElizabethStuart
AssociateDeanforEducationandProfessorDepartmentsofMentalHealthBiostatisticsandHealthPolicyand
JohnsHopkinsBloombergSchoolofPublicHealth
Educationsocialpolicystatisticalmethods
estuartjhuedu
Resources
66
Management
Design-ThinkingResources
EngagementGuideforCivicTech
Journeymap
PerformanceManagementResources
DenverPeakAcademy
Proxymeasuresguide
Talkingaboutperformancewithoutcomplicatingit(video)
SettingPerformanceTargetsguide
Datavisualizationguide
Performancemanagementgettingstartedguide
Exercises
Fishboneexercise
5Whys
AdministrativeData
TheLessonsofAdministrativeData(JPAL)
EvaluationResources
Quantitativeandqualitativemethods(USDepartmentofHealthandHumanServicesAgencyforHealthcareResearchandQuality)
KeyStepsinOutcomeManagement(UrbanInstitute)
RCT-YES
Sageresearchmethods
BestPracticesforMixedMethodsResearchintheHealthSciences
ACCORDChecklist
Pilotandimplementationplanning
Benchmarkingexamplesandproxymeasures
Seattleopendataplaybook
Resources
67
Datasciencescopingguide
Resources
68
AboutGovExGovExhelpsgovernmentsusedataandevidencetoimproveresidentsrsquolivesWesupportgovernmentsinbuildingtheirinternalcapacitytooperatesustainableanalyticspracticeswhichimprovecommunityoutcomesinfieldsfrompublicsafetytoeconomicdevelopmentGovExhasaproventrackrecordofworkinginconcertwithpartnerstoprovidetechnicalassistanceandtrainingatscale
SinceourlaunchinApril2015GovExhas
Trainedover2000governmentemployeesinpersonandonlineintopicsincludingdatamanagementperformancemanagementopendatacommunityengagementandanalytics
Providedtechnicalassistancetomorethan100localgovernmentsacrosstheglobeimprovingtheiruseofdatatoachieveefficienciesinservicedeliveryandbetteroutcomesforresidents
Assessedthecapacityofmorethan130governmentalorganizationstodeterminehowleadersstaffandexternalstakeholdersmanageandusedata
Localgovernmentsareincreasinglyonthefrontlinesofaddressinglargescaleproblemswithglobalimplicationssuchasclimatechangepublicsafetyandeconomicinequality
Whileevidence-basedinterventionstohelpsolvethoseproblemsareavailablelocalgovernmentstaffhavelimitedtimeandcapacitytofindandimplementresearch-basedsolutionsAsaresultcitiescontinuetoworkthewaytheyalwayshave--makingincrementalimprovementstoprogramsbutnotalwayscreatingchangeatasystem-level
ThroughresourcedevelopmentandcollaborationwithJHUfacultyGovExissupportingcityleadersinusingdataandevidencetomakedecisionsandcreatinglastingrelationshipswithresearchers
TheRoadmapwasdevelopedbyGovExaspartofitspartnershipwithBloombergPhilanthropiesWhatWorksCitiesinitiative
AboutGovEx
69
- Title page
- Introduction and purpose of the Roadmap
- Using the Roadmap
-
- Gather information about the problem
- Strategically engage key partners
- Review the evidence and collect details
- Decide on a policy to implement based on local factors
- Establish a framework to monitor progress
- Assess data accessibility quality and security
- Design implementation and evaluation
- Share your story and success
-
- Working with a range of evidence
- Thank you
- References
- Resources
- About GovEx
-
IninterviewswithGovExonthepolicychangeprocessintheircitiespeoplewasthewordmostcommonlyusedbyintervieweeswhendescribingthevariousstepstoimplementanewpolicyorprogramYouwillnoticethatthisisathemethroughoutthisRoadmapThefirstsectionbeginswithidentifyingtherelevantpeopleThechapteronidentifyingrolesforkeypartnersservesasafoundationforthefollowingstepsThediscussionofevidenceencouragesreaderstoconsiderwaysinwhichpeoplecanbuildanduseevidencetodesignstrongprogramsandinitiativeschangehumanbehaviorassessimpactensureeffectiveuseofpublicresourcesandimprovelives
Meaningfulapplicationofevidencebypolicymakersdoesnothappenwithone-timesharingofcuttingedgeevidenceRathermeaningfulinfluencerequiresalong-termstrategytoformalliancesanddevelopknowledgeofpolicymakersandpolicymakingInadditiontoworkingwiththerightpeopleandunderstandingevidenceandcontextwehopethatRoadmapuserswillformapeernetworkthatsharesnotjustwhichpolicysolutionsexistbutwhatimplementationlookslikeAsonecitywespoketoindevelopingthisRoadmapputitalotofourcitysproblemshavebeensolvedbyotherswejustneedtofigureouthowtoimplementthesolutions
TheexamplesthroughouttheRoadmapareillustrativeofgovernmentsthathaveeffectivelyimplementedtherelevantstepsinthisRoadmapbutpleasenotethatnocitydoesallofthisperfectlyTogetherthestepsinthisRoadmaparemeanttohelpcitiesframetheirproblemscreateademandforevidenceandbuildsustainablecoalitionsformeaningfulpolicychange
UsingtheRoadmap
7
GatherinformationabouttheproblemInyourearliestphaseitiscriticaltodevelopacomprehensiveunderstandingoftheproblemYouwillwanttolearnaboutthepeoplewhoareaffectedbytheproblemyouaretryingtosolveandthosewhocanimpacttheproblemFromthereyouwillgetinsightintotherootcauseoftheproblemtheworkthathasbeentriedinthepastandtheeffectivenessofthoseeffortsTheworkfromthischapterwillserveasthefoundationforactivitiesinsubsequentchaptersYoumayrevisitthesestepsasyouidentifyindicatorsandplanyourimplementationandanalysis
BesuretodedicatetimeandresourcestothisstepItcanbetemptingtoskipaheadtolookingforsolutionsortoassumethatyouhaveanunderstandingoftheproblemasaresultofapublicengagementprocess
TheCityofLewisvilleTXconductedaCitywidepublicengagementprocesstodevelopitsLewisville2025StrategicPlanOneoftheBigMovesoutlinedintheplanwasExtendingtheGreenorexpandinggreenspaceandconnectingrecreationareaswithneighborhoodsTheactionstepstoachievethisgoalincludedcleaningupcreeksandremovingdebristomakethemmoreaccessibleThecleanupswentsmoothlyinseveralneighborhoodsbuttheCitymetwithoppositionfromoneneighborhoodthatpreferredthenaturalhabitatandexpressedconcernabouttheenvironmentalimpactofthecleanupsAccordingtoLewisvilleiftheCityhadmappedallofthestakeholderstodeterminewhoneededtobeatthetableandwhowasmissingfocusinginformationgatheringontheneighborhoodlevelithaveavoidedthisproblem
Startwiththewhothepeople
Establishaprojectteam
BeginbynamingtheprojectteamforthisprocessincludingaprojectleadTheprojectleadshouldbesomeoneinCitygovernmentwhoisrespectedcangalvanizeotherkeypartnersandhasthetimetoleadthisworkThisissomeonewhomayhavetohavedifficultconversationswithCityleadershipdepartmentsandotherfundedpartnersItisnotnecessaryfortheprojectleadtobeapolicyexpertItismoreimportantforthemtohavetherightrelationshipsandpositionwithintheorganizationTheprojectteamshouldincludeindividualswithdiverseperspectivesfrommultipledepartmentsThereshouldbesomeonewithpolicyexperienceontheprojectteamtogroundtheconversationinrealisticactionHavingateammemberwhocancoordinatewithresearchersonrigorousanalysisandtranslatethatworkforothersiskeyAdditionallybesuretoincludefrontlinestaffworkersTheywillhavevaluableinsightbasedontheirinteractionswithresidentsandpolicysolutions
Gatherinformationabouttheproblem
8
havetomakesensetothepeoplewhowillbechargedwithimplementingthemBuildingastrongdiverseprojectteamisimportanttoensurethattheworkdoesnotliveanddiebyoneperson
Identifykeypartners
NextidentifythepeoplewhotheprojectteamcanworkwithontheissueyouarefacingincludingpeoplewithlivedexperienceintheproblemyourcityistryingtosolvedepartmentstaffandsubjectmatterexpertsoutsideofcitygovernmentThisistheconceptbehindbuildwithnotforamovementledbyLaurenellenMcCannthatemphasizescommunityexpertiseandleadershipMappingoutkeyplayersandtheirrolesiscriticaldotoearlyintheinformationgatheringphasesincecoalitionbuildingissuchanintegralpartofthepolicychangeprocessThisprocessmightconsistofstakeholdermapping(describedbelow)communityresourcemappingandculturalmappingInyourmappingremembertoconsiderpotentialalliesandthosewhomaypresentroadblocksalongtheway
PolicymakingisnottopdownorlinearInsomecasescitiesmayhavetosharedecision-makingpowerwithotheractorsForinstanceinmanycitiesthelocalContinuumofCare(CoC)istheleadentityonhomelessnessservicesCoCsapplyforandstewardfederalfundstoaddresshomelessnessandcoordinateservicesamonglocalprovidersAlthoughcitygovernmentsdohavearoleinCoCstheyarenotalwaystheleadagencychargedwithdistributingfundsandthereforetheirroleinhomelessnesspolicymaybelimitedCountygovernmentsoftenhavehealthdepartmentswhilecitiesdonotInthosecasesthebalanceofpowerfavorscountygovernment
TheCityofDentonTXwasrecentlyfacedwithanissueofresidentcomplaintsaboutanewdocklessbikesharethatarrivedintownandisviewedbysomeasexpensivelitterInresponseDentonsCityCouncilwhichwasalreadydiscussingpotentialrulesforbikesharecompanieselevatedthisissueandchargedcitystaffwithidentifyingasolutionPresentlytheCityisreviewingordinancesfromnearbycitiestodeterminehowtheyaresolvingthisproblemandwhatsolutionsmaybefeasibleinDentonThemappingexamplebelowrepresentslocalactorsinDentonwithastakeinthisissuewhohavemultipleperspectivesandlevelsofpower
Gatherinformationabouttheproblem
9
UsethistemplatedevelopedbytheCentreforPublicImpact(CPI)tobrainstormwhoyoumightreachouttoasyouembarkontheroadtopolicychangeRemembertothinkabouthoweachpersonorgrouprelatestotheproblem
BegintodefinethewhatortheprobleminyourCity
TherestofthissectionwillhelpyourefinetheproblemstatementFornowgathersomekeypartnerstoclearlyarticulatewhychangeisneededUsethetemplatebelowdevelopedbyDenverPeakAcademytocapturetheproblemstatementYouwillbuildonthislater
ProblemStatementndashWhyChangeisNeeded(1-2Sentences)
ThroughourinterviewswithcitystaffweheardthatwhendiscussingmunicipalchallengesitcanbetemptingtojumptoasolutionbeforetakingthetimetoreallyunderstandtherootcauseoftheproblemThismakessensesincecityleadersandstaffareresponsiblefordeliveringservicesandbeingresponsivetoresidentswhileaddressingnewproblemsthatariseinthecity
TherootsoftheproblemsthatcitiesareworkingtosolvearecomplexandmaybethelegacyofhistoricalfederalpolicyorsweepingsocialtrendsTohelptheproblemseemlessdauntingitcanhelptobreakthemdownintotherootcausetoidentifyamanageablecomponentoftheproblemUsingafishboneexerciseorfivewhystechniquecanalsohelp
Gatherinformationabouttheproblem
10
tofindtherootcauseoftheproblemandavoidsimplyaddressingsymptomsoftheproblemandrepeatingthemistakesofhistoryToolssuchasjourneymapscanhelpyouunderstandtheexperienceofthosewhoareaffectedbytheproblem
TheCentreforPublicImpactwithsupportfromBloombergPhilanthropiesdevelopedanexercisetosupportcitiesinidentifyingtherightsizedproblemAccordingtoCPIarightsizedproblemisonethatislargeenoughtopotentiallyapplytoothercitiesbutsmallenoughtosolvewithinareasonabletimeframe
AtthebeginningofMayorBerkesadministrationintheCityofChattanoogaTNveteransexperiencinghomelessnesswereaskingtheCityforhelpinfindinghousingTalkingtothispopulationrevealedthatveteranshadtroubleidentifyingaffordablehousingwhichmotivatedtheAdministrationtoworkonasolutionTheproblemofveteransexperiencinghomelessnesswastherightsizebecauseitfocusedononesegmentofpeopleexperiencinghomelessness-veterans-andtheirspecificchallengeinidentifyingaffordablehousing
Developacomprehensiveunderstandingofhowyourcitycurrentlyaddressestheissue
IntheChattanoogaexamplementionedaboveCitystaffdiscoveredthatveteranswerehavingtroublenavigatingtheCitysvariousaffordablehousingprogramsandgettingaccesstothehousingandservicesthattheyneededAtthesametimenewlyelectedMayorBerkewasexploringcurrentandfutureopportunitiestoservethispopulationRatherthancontinuetousefederalfundingforemergencysheltersMayorBerkedirectedcitystafftoprioritize
Gatherinformationabouttheproblem
11
HousingFirstwhichvaluesconnectingpeoplewithpermanenthousingovertransitionalhousingandsupportiveserviceafterspeakingwithrepresentativesattheUSDepartmentofHousingandUrbanDevelopment(HUD)
ThisexaminationofwhattheCityiscurrentlyfundingandthesourcesofthatfundingwasaturningpointinChattanoogaUnderstandingthisproblemfromasystemsperspective-includingpoliciesandfundingstreamsatmultiplelevelsofgovernmentsocialnetworksthroughwhichinformationisoftenspreadtechnologythebuiltenvironmentofmarket-rateandpublichousing-helpedthenewMayoranalyzeexistingeffortsandleadstaffandotheractorsinthesamedirection
Begintogatherthedata
TotrulycapturethesuiteofservicesthatyourCityisprovidingaskdepartmentstafftolisttheprogramsrelatedtoyourproblemstatementthatyourcityrunsorfundsWhenlistingtheprogramscapturedetailssuchas
programnamethenameofthedatasetthatisrelevanttoeachprogramadescriptionofthedatasetthesourceofthedatasetifthedataisinternalorexternalwhetherthisdataisalreadybeingcollected
ThisinformationwillformthebasisofyourinventoryofadministrativedataordatasetsthatarekeptbygovernmentsforoperationalpurposesThismayincludecallcenterrecords(311911)taxdatapropertyassessmentsinspectioninformationsocialservicerecordsetcAtthispointlistallrelevantadministrativedatasetsYouwillprioritizewhichdatasetstogathermoreinformationonlaterwhenyouaredevelopingindicators
Asampletemplatethatcanbeusedtocapturethisinformationisbelow
ProblemStatement____________________________________________
Gatherinformationabouttheproblem
12
ProgramName
DatasetTitle
DatasetDescription DataSource
Departmentorexternalpartner
DoestheCityalreadycollectthisdata
Whatisthenameoftheprogram
Human-readablenameofthedataset
Shouldbeunderstandablebynon-technicalusers
Whatistheoriginalsystemapplicationfilethathousesit
Departmentorexternalpartnerthatmaintainsthedata
Isthisdataalreadycollected
TogatherthisinformationconsiderassigningonepointpersontosurveyorinterviewcitystaffSpeakingwithfrontlineworkersstaffwhoworkwithfederalgrantsdepartmentheadsandeveryoneinbetweenwillensureadeepunderstandingoftherelevantprogramsanddatasetsDuringthecourseoftheseconversationsbesuretoaskcitystaffwhichorganizationstheyworkwithexternallyandcapturethatinformationaswelleveniftheyarenotrecipientsofcityfundingInadditiontoconversationswithexpertsincitygovernmentyourcitysbudgetdocumentsareanothergreatsourceofinformation
Collectdataandobservetrends
Nexttrytogainanunderstandingofwhytheproblemstillexistsifithasbeensomethingyouhaveworkedonforawhileorwhyitisemergingnowifitisnew
AnalyzingadministrativedatacandemonstratepasteffectivenessofinitiativesrelatedtoyourproblemstatementUsingthedatasetinventorythatyoubeganintheprevioussteptalktodepartmentstaffaboutwhichdatasetsaremostrelevanttotheproblemandgatherafewyearsworthofhistoricaldataBegintolookfortrendsinthisdatadiscussingpotentialreasonsforthosetrendswithrelevantstaffCanthetrendsbeexplainedbyseasonalitychangesinfundingstaffingfluctuationsorsomeotherreasonBasedonthehistoricaldataanalysiscreateabaselineagainstwhichfuturechangescanbeobserved
OtherdatasetscanhelpprovideadditionalcontextForinstanceifthenumberofpeopleexperiencinghomelessnessinyourcityisincreasingitmaybehelpfultotakealookathousingaffordabilityIfyoudonothavethedatayouneedconsideraproxymeasureAsanexamplehousingisoftenconsideredaffordableifitcosts30orlessofahouseholdsincomeButifhouseholdincomedataisnotavailableatthecitylevelAreaMedianIncome(AMI)canbeahelpfulproxy
Gatherinformationabouttheproblem
13
QualitativedatacanalsohelpanswerquestionsaroundtheeffectivenessofpreviouseffortsTrytoobtainavarietyofperspectives--bothmainstreamandoutliers--fromnetworksbeyondyourowntoprovideyouwiththegreatestvantagepointforevaluatingpotentialimpactUnderstandinghowthoseeffortswerereceivedbydifferentpartsofyourcommunityandiftheremaybeelementsofprevioussystemstopreservewillbecrucialtothesuccessofanypolicychangeThisinformationmaybeobtainedthroughfocusgroupsinterviewsandsurveysFormoreinformationonhowtousethesequalitativeresearchtoolspleaseseetheseresourcesdevelopedbytheAgencyforHealthCareandResearchandUrbanInstitute
FinallytakealookatnationaldatatoseewhereelsethisproblemmightexistThisresearchmaybehelpfulinlaterstepswhenyouarelookingforpromisingpoliciesinothercities
Puttingitalltogether
UsethetemplatebelowdevelopedbyDenversPeakAcademytocaptureyourrefinedrightsizedproblemstatementandcurrentandfuturestatesWorkingthroughthisexercisewithpartnersishelpfultogainbuy-inTheresponseswillbeimportantwhengettingotherpartnersonboard
WhyChangeIsNeeded
Yourright-sizedproblem
CurrentState
Analysisoftrendsindata
FutureState
Ideallywhataretheoutcomesyouhopetoachieve
Checklist
Thekeystepstogatheringinformationarebelow
Atthispointfocusontheproblemratherthanjumpingtosolutions
Listentopeoplewhoareaskingforhelpandtoadvocateswhoareraisingissues
Beasspecificaspossibleinidentifyingtherootcauseandhoninginonthepartoftheproblemthatyourcitycanaddress
Identifytrendsbasedonmultiplesourcesofdataincludinghistoricaladministrativedataproxymeasuresqualitativedataandnationalmeasures
Gatherinformationabouttheproblem
14
Captureyourproblemstatementtheresultsofyouranalysisanddesiredoutcomes
Gatherinformationabouttheproblem
15
StrategicallyengagekeypartnersPolicytheoryresearchoftenfocusesonincreasingpolicymakersaccesstoevidenceasawaytoavoidtakingshortcutslikemakingdecisionsbasedonemotionImplementationsciencesuggeststhatincreasingaccesstoevidenceisnotenoughandthatpolicychangeshouldbeexaminedinthewidercontextofdebatecoalitionformingandpersuasionMeaningfulpolicychangerequiresldquoalong-termstrategytoformalliancesanddevelopknowledgeofpolicymakersandpolicymakingratherthanusingevidenceinpolicymakingonanadhocorone-offbasis
UnderstandingnetworksandsystemsisimportanttodevelopsustainablepolicychangeThissectionismeanttohelpreaderstakethestepstobuildengagementstrategiesforpartnersthatrepresentdifferentsystemsandnetworksEngagementshouldbeanongoingprocessofconnectingpeopletooneanotherandtoanissueEngagementshouldnotbeinnameonly-wehaveallexperiencedengagementactivitiesthatgivepeopleaspecificroletoplaythatleadstoaneatpredictableoutcomeAccordingtotheBuildWithmovementundervaluingandunderinvestinginengagementstrategiescanleadtoadisconnectofcertainindividualsorgroupsandmistrustofgovernmentWhendevelopingengagementactivitiesrememberthathowpeopleengageandwhethertheyengagehastodowithhowyoustructureopportunitiesforengagement
Conveneacoreadvisorygroup
AnadvisorygroupcomprisedofpeoplefrommultiplesectorscanbehelpfulasyouconsiderthelocalsocialenvironmentalandpoliticalcontextinwhichyouintendtointroducepolicychangeRevisityourstakeholdermapanddeterminewhowouldbebesttoserveonthisgroupTomakesureyouareconsideringtheproblemsandsolutionsthroughanequitylensincludepeoplewithlivedexperiencewhohaveexpertiseintheeffectivenessofsystemswhetherthosesystemsfunctionasintendedandhowtheymayimpacthumanbehaviorYoumayalsowishtoincludesomeonewithexperienceworkingwithdataandanalysis-thispersoncanhelpdeterminethetypeofdatathatmaybereadilyavailableandusefultocollectandcanalsohelpplanandshapeyouranalysistofigureoutwhichprogramsareeffectiveBesuretoinvitefront-lineworkersaswell-itscriticalthatwhateversolutionscomeoutofthisprocessaremeaningfultothosewhowillberesponsibleforimplementingthem
BringingpeopletogethercanbechallengingbutthereareafewwaystosetandmanageexpectationsFirstmakesurethatthegroupisawareoftheproblemstatementcurrentstateandfuturestate-thiscanhelpstarteveryoneoffonthesamepageNextspend
Strategicallyengagekeypartners
16
sometimegettingtoknoweachotherrsquosworktolearnabouteveryonesstrengthsThissharedunderstandingcanhelpasyouclarifyrolesandresponsibilitiesofeachpartnerinthegroup--thisisimportanttoavoidfinger-pointinginthefutureandtoensurethatnoneofthetasksslipthroughthecracksInsomecasestherearemanyactorswithdifferentresponsibilitiesasinthecaseofBaltimoreCitys38-memberYouthFundTaskForcewhichischairedbynonprofitandcorporatepartnersandincludesrepresentativesfromthenonprofitandfaithcommunitiesstudentsthebusinesssectorlocaluniversitiesCityCouncilandtheMayorsOffice
OncerolesandresponsibilitiesareclearyoumayconsiderdevelopingacharterguidelinesortermsofreferencetosetexpectationsforthegroupBesuretodistinguishthisfromotherboardsandcommissionsthatyourcitymayhaveNextyoumaywishtoestablisharegularongoingmeetingscheduleBesuretosendagendasandmeetingmaterialsaheadoftimeandclarifytheobjectiveofeachmeeting
InSeattletheCitydevelopeditsHousingAffordabilityandLivabilityAgenda(HALA)torespondtorisinghousingcostsandhomelessnessThisisatrendinotherwestcoastcitiesaswellwhichmaybeexplainedbyaninfluxofnewresidentswhoareattractedbytheemergenceoftechjobstheattractionoftheoutdoorlifestyleortherecentpassageofprogressivepolicieslikelegalizedgaymarriagea$15minimumwageandlegalizationofmarijuanaSeattlebroughttogetheramulti-stakeholderadvisorygroupcomprisedoflocalhousingexpertsrentersandhomeownersdevelopersandgovernmentofficialstoshareinformationabouttheirneedsanddeveloprecommendationsbasedontheseneedsTheinterestsandmotivationsofthisgroupwerevariedandittookabouttenmonthstofindcommongroundTheseriesofrecommendationsthattheadvisorygroupcreatedismeanttoserveresidentswithaspectrumofneeds
Buildaprocessforcommunityparticipation
Ofcoursenoteveryonecanbeincludedinanadvisorygroupandadvisorygroupmembersmaynotnecessarilyrepresentthevariedperspectivesoftheirpeers-forinstancenoteveryhomeownerhasthesamesetofneedsTogatherinputfromawiderswathofthepopulationitcanbehelpfultobuildaprocesstoinvitethepublicatlargetosharetheirideasCitiesdothisinavarietyofwaysfromonlineconversationsandsurveystoin-personmeetingsRememberthatbeingintentionalaboutgatheringtheviewpointsofpeopleismoreimportantthanusingthelatesttechnologyandtools
TheinsightgeneratedbycommunityparticipationexercisescanbeincludedinyourreviewofevidencewhichisdiscussedinasubsequentchapterInWestVirginiaastatethathasbeenhithardbytheopioidepidemicstatehealthofficialsdraftedanopioidresponseplanthatwasinformedbypublicengagementaswellaslocalandregionalsubjectmatter
Strategicallyengagekeypartners
17
expertsThepublicwasinvitedtopresentcommentsonideasfordecreasingoverdoseearlyinterventiontreatmentandrecoveryPublichealthandmedicalfacultyfromWestVirginiaUniversityMarshallUniversityandJohnsHopkinsUniversityprovidedresponsestothecommentsTheresultwasanOpioidResponsePlanthathealthcommissionerRahulGuptapresentedtoGovernorJimJustice
AsamplespectrumtoplanyourowncommunityparticipationprocesscreatedbyLaurenellenMcCannisbelow
SourceLaurenellenMcCann
Community-basedparticipatoryresearchisanotherwaytoworkwiththecommunitytoimproveoutcomesCBPRmeansthecommunityisinvolvedinallaspectsoftheresearchprocesswhichisofteniterativeInCBPRtraditionalresearchersandcommunitymembersshareknowledgeaswellasresourcesandcredit
CBPRInActionInSanFranciscoanenvironmentaljusticenonprofitteamedupwiththeCitysDepartmentofPublicHealthandanoutsideevaluatortounderstandtherootcausesofobesityandfoodinsecurityinlow-incomeneighborhoodsandidentifysolutionsThroughresearchmethodssuchasinterviewingresidentsandstoreownersdiagrammingstoreshelvesGISmappinganeconomicanalysisandreviewofsimilarpoliciesinnearbycitiestheteamobservedthatthestoresthatresidentshasveaccesstowerestockedwithlesshealthyfoodsandthatresidentswantedmorehealthyfoodandlessalcoholandtobaccoThisprojectresultedinlocalandstatelegislationtoencouragestorestostockfreshproduce
Strategicallyengagekeypartners
18
Developrelationshipswithsubjectmatterexperts
OnewaytokeepsubjectmatterexpertsengagedistoinvolvetheminthecommunityparticipationprocessasdescribedaboveIntheWestVirginiaexamplesubjectmatterexpertsrespondedtoandsynthesizedthepubliccommentsOtheropportunitiesaredescribedbelow
AdvisoryAcitydepartmentheadMayororstaffmembermaydevelopaone-on-oneadvisoryrelationshipwithuniversityfacultymemberoracollaborativebetweentheCityandUniversitytoaddressaparticularproblemAgreatplacetostartistolookforrelevantpracticecenterswithinyourlocaluniversityandreachingouttosubjectmatterexpertsInadditiontotheirexpertiseontherelevantpolicytopicheorshemayalsobeabletoidentifyotherexpertsandevenprovidesupportonhiringtherighttalenttoleadcertainprogramsAnadvisoryrelationshipwithauniversitysubjectmatterexpertcanalsohelpwhencitiesareconsideringbringinginnewprogramsTheseexpertsareoftenfamiliarwithprogramsthatarebasedonnationalmodelsandcanprovideinsightintoaprogramsoriginandlineageeffectivenessinotherplacesandthepotentialforimplementationinyourcityAcademicscanalsohelpidentifyindicatorsbasedongovernmentindicatorsorliteratureFinallyrelationshipsbetweenpolicymakersandresearchersprovidethemostbroadlyeffectiveroutetopullingresearchintopolicy
ResearchandevaluationAcademicexpertsmaysupportcitiesinevaluation-indeedevaluatingcityinitiativesisatraditionalmodelofcityuniversityrelationshipsSharingtheinformationyoucollectedintheGatherInformationphasecanhelpresearchersunderstandyourworkandgoalswhichinturnwillstrengthenthesupportthattheycanprovideDiscussingdetailsaboutyourinterventionandevaluationplanatthebeginningoftheprocessmeansthataresearchpartnercanhelpwiththeevaluationdesignanddatacollectionForinstancetheWilsonSheehanLabforEconomicOpportunitieswalkscitiesthroughconsideringthefollowingquestionsbeforepartneringonanevaluation
1 Isthereaclearpolicyprogrammodelwithobjectivescreeningcriteria
2 Isitpossibletoevaluatewhathappensbothwiththepolicyprogramandintheabsenceofthepolicyprogram
3 Arethereoneormoremeasurableoutcomesofinterest
4 Arethereenoughpeopleforrigorousquantitativeanalysis
5 Whatdatawillbenecessaryforthestudyandwhataretheprocessesforaccessingthesedatasource(s)
6 Isthereacompellingreasontoevaluate
Strategicallyengagekeypartners
19
InadditiontoassessingreadinesstoconductanevaluationresearcherscanhelpyouhoneinonthepurposeoftheevaluationTheWorldBankdescribesseveralpotentialusesofevaluationwhicharedefinedhereInbriefevaluationcanhelpdecisionmakers
makeresourceallocationdecisionsrethinkrootcausesofproblemsidentifynewproblemsdecideonabestalternativedeterminewhetherprogramsorpoliciesarebeingimplementedwithfidelitydeterminewhetherprogramsorpoliciesarehavingthedesiredimpact
Researcherscanalsoprovidesupportinsharingtheresultswiththescientificcommunitythroughpapersandconferenceswhichisvaluableincreatingapeernetworkofevidence-basedcitypractitioners
TrainingandtechnicalassistanceUniversitypartnersespeciallythoseatpracticecenterscanbehelpfulindevelopingorupdatingtrainingcurriculumtrainingthetraineranddeliveringtrainingtocitystaff
Strategicallyengagekeypartners
20
ReviewtheevidenceandcollectdetailsOnceyouhaveathoroughunderstandingoftheproblemdataandactorsyouwillneedtoconductyourownresearchonavailableevidenceandlearnaboutthelandscapeofpolicyoptionsYourgoalwillbetodevelopanaccessiblelistofinformationaboutpoliciesthathavehelpedsolvesimilarproblemsinotherplacesThischapterwillguideyouthroughthestepsofpreparingyourreviewersunderstandinglevelsofevidenceandcollectingthedetailsonevidencethatyouwillneedfordiscussionwithdecisionmakersinyourcity
Selectandprepareyourreviewers
AssignapersonorteamtoreviewrelevantinterventionsthathavedemonstratedsuccessinotherjurisdictionsTheirgoalwillbetoproduceareviewofalloftherelevantprogramspoliciesorinitiativesthatyourteamcouldconsiderimplementingTheywillbeconductingaliteraturereviewsupplementedbyinterviewswithexpertsandpeerswhohaveinformationaboutpotentialprograms
SincethepeopleyouassigntothistaskareprovidingthebaselineofpotentialoptionstoimplementitisimportantthatyouchoosepeopleyouareconfidentwillbethoughtfulandthoroughintheirworkItisalsocriticalthattheyareawareoftheimportanceofsourcecredibilityEnsureyourreviewershavefamiliarizedthemselveswithrulesforcriticallyanalyzinginformationsourcesbeforetheygetstartedIftheyareincludingnewsarticlesaspartoftheirsearchtheyshouldalsobeawareofmethodsforevaluatingnewscredibility
AskyourreviewerstodevelopaplanandtimelinetocompletethereviewThisstartswithsolidifyingagoverningresearchquestionOnewaytoknowyouarecreatingadetailedresearchquestionistousetheacronymPICOremindingyoutoincludereferencetothepopulationintervention(ifyouhaveoneinmind)comparisonandoutcomeelementsinyourresearchquestionForexampleifyouknowthatyouhaveapopulationofteenagerswhoarehomelessyouhavelearnedfromyourstakeholdersthatmanyoftheseteenagersidentifyasLGBTandyouhaveheardsomethingabouttheapproachofcreatingdrop-incentersfordeliveringsupportyoucouldask
ForLGBTyouthfacinghomelessness(population)howeffectivearedrop-incenters(intervention)relativetostandardsocialservices(comparison)atsettlingtheminpermanenthousing(outcome)
Ifyoudonothaveaninterventioninmindyoucanleaveouttheinterventionandcomparisonandsimplyask
Reviewtheevidenceandcollectdetails
21
HowcanwehelpLGBTyouthfacinghomelessness(population)obtainpermanenthousing(outcome)
OnceyourreviewershavetheirresearchquestionsusethequestionstocreatealistofkeywordstheycanuseintheirliteraturesearchThiswillhelpkeeptheircollectionofarticlesreasonablyfocusedHavethemdescribewhattheirsearchprocesswillbeincludinglistingthedatabasestheywillreviewandtheirplanforstoringtheinformationtheywillcollect(Herearesomeadditionalsuggestionsforstructuringareviewplan)
WhatyourreviewerswillbedoingisalightweightversionofarapidreviewTotheextentthattheywouldliketodevelopamethodologyforspecificsearchprotocolsincludingcriticalappraisalprocessestheycancertainlygoaheadanddoso
Usedifferentlevelsofevidenceforthemostcompletepicture
AsdiscussedintheWorkingwitharangeofevidencechaptertheNestaStandardsofEvidenceframework--reproducedbelow--ishelpfultounderstandthelevelofconfidencethataprogramorinterventionishavingapositiveimpactThisapproachletsyouputinterventionsthatyouhaveheardaboutlocallybutwhichhavenotbeenstudiedtogetherwiththosewithademonstratedcausaleffectThisishelpfulsinceanumberofpoliciesprogramsandpracticesthatyouwillwanttoconsidermaynotyethavemuchofaresearchrecord
Reviewtheevidenceandcollectdetails
22
NearlyanypieceofinformationaboutaninterventioncanbeclassifiedwithinthestandardsofevidenceForexampleifsomeonehasacrediblehypothesisabouthowaninterventioncouldproducebenefitthistypeofevidencewouldbeclassifiedasLevel1ThistypeofevidencecouldincludeinterventionsinothercitiesthatyoureadaboutinthenewsorhearaboutataconferenceForinstancewhenUSDAallowedschoolsandschooldistrictstoprovidefreebreakfastandlunchtoallstudentsthroughtheCommunityEligibilityProvision(CEP)manydistrictsimplementedthisoptionevenbeforetherewasevidenceofeffectivenessThiswouldfitintoLevel1sinceitbuildsonthelogicthatprovidingaccesstomealswilldecreasefoodinsecurity
YoumayalsofindpiecesofinformationthatqualifyasLevel2evidenceForinstanceasmoreschooldistrictsimplementedCEPprincipalstestifiedaboutpositivechangestheyobservedinstudentssuchasamorepositivedemeanorthatcanleadtobetteracademicperformanceDataalsoshowedthatthestudentsdescribedwerereceivingmoremealsthanbeforetheprogramsimplementationHoweveritwastoosoontobuildevidencetoshowacausalrelationshipbetweentheincreaseinfreemealsandincreasedacademicperformance
ldquoLower-levelrdquoevidencecanconsistofnuancedbutvaluableelementsthatrelatetoaninterventionrsquospotentialfitwithacommunityQualitativeinformationsuchastestimonyfrompeoplewhohaveexperiencedaninterventionandthosewhohavefacilitatedorimplementeditwillalwaysbeusefultoconsiderandprovidesdetailedlocalcontext
Experimentalstudies--describedstartingatLevel3withcontrolledexperimentsandgoingthroughLevel5withfullyfleshedoutprotocolsandprocessesforreplication--stilloccupythehigherlevelsofthisdescriptionofevidenceThisprioritizationshowsthevalueofrigorousstudyforevaluatingtheimpactofaparticularpolicyorprogramratherthanotherfactorsthatcouldbeinfluencingoutcomesofinterestRememberthatmultipletypesofevidencecancomplementrigorousimpactevaluationsandgivecityleadersadditionalinsightsandbroaderperspective
Accessinformation
Encourageyourreviewerstobegintheirsearchforpromisingpracticesingovernmentandnon-profitclearinghousesandinacademicpolicyjournalsWorkpublishedinthesesourceshasalreadybeensubjectedtoahighlevelofprofessionalscrutinyandgenerallycontainsagreatdealofdetailwhichmeansthatyourreviewersaremorelikelytobeabletopullouttheinformationyouneedtomakeagooddecision
Clearinghouses
Reviewtheevidenceandcollectdetails
23
Herearesomeoftheclearinghouses(collectionsofprogramevaluations)forevidence-basedpoliciesandprograms
ResultsFirstClearinghouseDatabaseThePewCharitableTrustsCollectstogethereightotherclearinghousestoprovideinformationoninterventionsinthefollowingcategoriesadultcriminaljusticeeducationchildwelfarehealthjuvenilejusticementalhealthsubstanceabuseandyouthdevelopmentIndividualclearinghousesincludedinResultsFirstare
BlueprintsforHealthyYouthDevelopment-foryouthdevelopmentandviolenceprevention
CaliforniaEvidence-BasedClearinghouse-forchildwelfareinterventions
TheLauraandJohnArnoldFoundationrsquosSocialProgramsThatWork-forsocialservicesprograms
TheUSDepartmentofEducationrsquosWhatWorksClearinghouse-foreducationinterventions
TheUSDepartmentofHealthandHumanServicesrsquoResearch-TestedInterventionPrograms(RTIPs)-forcancercontrolinterventions
TheUSDepartmentofHealthandHumanServicesrsquoTeenPregnancyPreventionEvidenceReview(TPPEvidenceReview)-forsexualriskbehaviorandsexualhealthoutcomes
TheUSDepartmentofJusticersquosCrimeSolutionsgov-forcriminaljusticejuvenilejusticeandcrimevictimservices
TheUSSubstanceAbuseandMentalHealthServicesAdministrationrsquosNationalRegistryofEvidence-basedProgramsandPractices(NREPP)-substanceuseandmentalhealthinterventions
TheUniversityofWisconsinPopulationHealthInstituteandRobertWoodJohnsonFoundationrsquosCountyHealthRankingsandRoadmapsWhatWorksforHealth(WhatWorksforHealth)-healthbehaviorsclinicalcaresocialandeconomicfactorsandthephysicalenvironment
UKWhatWorksCentresTheBritishgovernmenthascharteredanetworkofsubject-specificorganizationswhichcollectsynthesizeandreportoutevidenceontheeffectivenessofpolicyprogramsandpractices
NationalInstituteforHealthandCareExcellence(NICE)-healthandsocialcare
Reviewtheevidenceandcollectdetails
24
SuttonTrustEducationalEndowmentFoundation-educationalachievement
CollegeofPolicingWhatWorksCentreforCrimeReduction-crimereduction
EarlyInterventionFoundation-earlyintervention
WhatWorksCentreforLocalEconomicGrowth-localeconomicgrowth
CentreforAgeingBetter-qualityoflifeforolderpeople
WhatWorksCentreforWellbeing-wellbeing
WilsonSheehanLabforEconomicOpportunities(LEO)UniversityofNotreDameProvideslinkstotheirevaluationsofinterventionsincriminaljusticeself-sufficiencyeducationhealthandhousingandhomelessness
ClearinghouseforLaborEvaluationandResearchUSDepartmentofLaborProvidesresearchonworkforceandlabor-relatedpoliciesandprograms
J-PALEvaluationsAbdulLatifJameelPovertyActionLabStudiesonpoverty-reducinginterventionsfromaroundtheworld
CampbellSystematicReviewsCampbellCollaborationSystematicreviewofpolicyinterventionsbyscholarlycoordinatinggroupsinsevenissueareascrimeandjusticeeducationinternationaldevelopmentmethodssocialwelfareknowledgetranslationandimplementationanddisability
Benefit-CostResultsWashingtonStateInstituteforPublicPolicyAWashingtonState-charteredprogramtoestablishthelikelihoodofrelativecostsandbenefitsforprogramsacrossthefollowingareasjuvenilejusticeadultcriminaljusticechildwelfarepre-Kto12educationchildrensmentalhealthhealthcaresubstanceusedisordersadultmentalhealthpublichealthandPreventionworkforcedevelopmentandhighereducation
EffectiveInterventionsCentersforDiseaseControlandPreventionRepositoryforevidence-informedHIVpreventioncareinterventions(withstepsonhowtoimplement)
Journalsandindexes
OnceyouhavereviewedrelevantinformationaccessiblethroughclearinghousesitisworthtakingalookatindividualstudiespublishedasscholarlyarticlesYourreviewerswilldothisbysearchingarticleindexesandacademicjournalsUnfortunatelyanumberofusefulacademicjournalsandindexingplatformsmaynotbegenerallyaccessibletoyourreviewersifyoudonothaveasubscriptiontothemToaddressthisconsiderpartneringwithalocal
Reviewtheevidenceandcollectdetails
25
collegeoruniversitytoprovideyouwithalistofarticlesfromthefollowingsources(Thisisalsoanexcellentopportunitytoseekresearchsupportfromanyacademicstakeholdersyouidentifiedpreviously)
Indexes
GoogleScholarincludesmostpeer-reviewedonlineacademicjournalsandbooksconferencepapersthesesanddissertationspreprintsabstractstechnicalreportsandotherscholarlyliteratureincludingcourtopinionsandpatentsItallowsyoutospecifyapublicationrangeinyoursearchandprovideslinkstolikelysourcesforalllistedarticles
PolicyFileIndexisauniqueresourceforUSpublicpolicyresearchinthatitgrantsusersaccesstotimelyupdatedinformationfromover350publicpolicythinktanksnongovernmentalorganizationsresearchinstitutesuniversitycentersadvocacygroupsandotherentities
PolicyArchiveworkstocollectanddisseminatesummariesandfulltextsvideosreportsbriefsandmultimediamaterialofthinktankuniversitygovernmentandfoundation-fundedpolicyresearch
Anothersourceforhigh-qualityresearchisthereportsoftheCongressionalResearchServiceYoucanfindtheirreportsaggregatedhere
Specificjournals
Oftenthemostusefulacademicjournalsforunearthingpolicyorprogramreviewswillbeineachspecificsubjectarea--thatisineducationjournalsforeducationpoliciesYoucanuseageneralindexlikeGoogleScholartobegintolearnwhichjournalsaremostoftenusefulforyoursearchHoweveryoucanalsosearchthefollowingtwogeneralistpolicypublicationsforissuesandarticlesrelatedtoyourtopic
PublicAdministrationReviewJournalofPolicyAnalysisandManagement
Newssourcessubjectmatterexpertsandinterviews
AlthoughitisvaluabletoknowwhathasbeenexploredthroughascholarlyresearchframeworkitisalsousefultolearnwhatkindofinterventionsarebeingtriedinthefieldrightnowandhowpeoplefeelaboutthesetrialsSearchsubscriptionnewsaggregatorslikeLexisNexisorfreenewsaggregatorslikeGoogleNewsorYahooNewsforyourissueareaplusthewordpolicyorprogramtoseewhatkindsofinterventionsarebeingcoveredinthenews
Reviewtheevidenceandcollectdetails
26
DonotforgettocontacttheadvisorygroupmemberswhowerechosenfortheirsubjectmatterexpertiseInadditiontoprovidinggoodmaterialthroughinterviewsthemselvestheymayhaveideasaboutothersubjectmatterexpertsyoushouldcontacttoaskaboutpotentialinterventions
Peercitiesandnetworks
InadditiontoreadingabouttheworkofothercitiesitisidealifyouareabletospeakwithpeoplewhoareinvolvedwithcurrentimplementationsofinterestinginterventionsInyourcityyoulikelyhaverelationshipswithyourneighborsbutitcanbehelpfultolookoutsideofyourregionforsolutionsaswellTofindcounterpartsinothercitieswhoareworkingonaprogramyouwouldliketolearnmoreaboutyoucancontactstaffinintermediaryorganizations(likeGovExoryourprofessionalassociations)whomightbeinapositiontoputyouintouchimmediatelyorwhomightseeanothergoodrouteforsettingupaconversationHereisalistofnetworksthatmaybehelpful
InternationalCityCountyManagementAssociation(ICMA)NationalAssociationofCounties(NACO)NationalLeagueofCitiesUSConferenceofMayorsWhatWorksCities
Collectthedetails
Onceyourreviewershavefoundevidenceonpossiblepoliciesorprogramstoimplementtheyshouldcapturekeydetailsthatwillhelpdecisionmakersunderstandthepotentialeffectivenessandapplicabilityofeachpotentialintervention
Foreachsourcethathasprovidedinformationyouwanttoconveyusethetemplatebelowtocaptureasmanyofthefollowingdetailsaspossible(Youdonotneedtocompletethisforsourcesthatprovideduplicativeinformationorareotherwisenotthatuseful)
Programorpolicyname (egldquoHousingFirstrdquoorldquoblightedpropertiesregistryrdquo)
Sourcetype (egjournalarticleinterviewsubjectmatterexpertstudyassociationcommunityparticipationprocess)
Sourcedetails (egarticletitleandpublicationnameandtitleofinterviewee)
Evaluationmethodology
(egrandomizedevaluationcomparativeevaluationoftwodifferentgroupsbeforeafteranalysis)
Evidencelevel (egLevel12345)
Programorpolicy
Reviewtheevidenceandcollectdetails
27
description
Programorpolicygoal (1sentencecapturingtheprincipalaimoftheintervention)
Existingimplementations
(WherehastheinterventionbeentriedListasmanysitesasyouareawareof)
Characteristicsofimplementationsites
(ForeachimplementationsiteofferbasicgeographicIDmdashcityandstateThenyouwillwanttocollectthedetailswhicharerelevanttoyourdecisionmakersrsquoconsiderationsaboutwhatmakesalocationldquosimilartordquoorldquodifferentfromrdquoyoursituationYouwillatleastwanttoincludeforeachimplementationsitePopulationnumberPopulationdensity(ruralurbansuburban)Socio-economiccharacteristicsBeforestartingaskyourdecisionmakersaboutanyparticularcharacteristicsofprogramimplementationsitestheywouldwanttoknowbeforeseekingtoreplicate)
Characteristicsofimplementationpartners
(Anyavailabledetailsaboutorganizationsinstitutionsorpeoplewhowereinvolvedwithimplementingtheintervention-egindependentchildcarecenterpublicelementaryschoolcitypublichealthnurses)
Characteristicsoftargetpopulationeligibleprogramparticipants
(Anyavailabledetailsaboutpeoplewhoarethebeneficiariesoftheprogramorotherwiseincludedoraffectedbytheintervention-egnumberofprogramparticipantscategoriesofpropertyownersaffected)
Policyorprogramduration
(HowlonghastheinterventionbeeninplaceineachimplementationsiteHasitendedandifsowhy)
Whatbenefitscamefromeachimplementation
(Anyobservedpositiveeffectsoftheinterventionincludingeffectsize)
Statisticalestimatesofinterventioneffectiveness
(Anyavailablestatisticalestimatesoftheinterventionrsquoseffects)
Whatdrawbackswereobserved
(Anyavailabledetailsaboutunintendedorunavoidablenegativeeffectsoftheintervention)
Programorpolicycosts
(Anyavailabledetailsaboutprogramcostsperimplementation)
Reviewtheevidenceandcollectdetails
28
Reviewtheevidenceandcollectdetails
29
DecideonapolicytoimplementbasedonlocalfactorsTogetthecriticalbuy-infromleadershiptoimplementanewpolicyorprogramtheprojectleadshouldscheduleameetingwithdecisionmakerstopresenttheevidencesurroundingthepolicyoptionscapturedinthetemplateDependingonyourinsightintoyourcityleadershipswillingnesstomakechangeyoumaywishtosharetheresearchyouhaveconductedontherootcausesoftheproblemandsomeofyouranalysisontheimpactofcurrenteffortstodate
Considertheapplicabilityofanapproach
AsdiscussedinthepreviouschapterpolicymakersoftenlookforevidenceofeffectiveprogramsinothercitiesbutoftenthequestionarisesofhowgeneralizableaprogramisfromonecitytoanotherTheAbdulLatifJameelPovertyActionLab(J-PAL)suggeststhatinreviewingevidencefromanotherlocationitishelpfultoidentifythespecificmechanismthatwasused(deliveryofprogramorserviceinformationsharingetc)identifythelocalconditionsthatwereinplaceobservethebehaviorchangethatresultedfromthemechanismandfinallytounderstandlocalcontextandconditions(legalorcostconstraintsculturalnormsetc)Thissetofpresentationelementscanbedrawnfromthelistofdetailscollectedintheresearchtemplate
EducationpolicymakersinChicagoILusedfindingsfromprogramsinKenyaandIndiainwhichchildrenwhohadfallenbehindtheirpeersweregiveneducationalsupporttargetedtotheirleveloflearningThemechanismusedwastargetedteachingandthereweresimilarconditionspresentinallthreecountriesAlthoughsomestudentshadfallenbehindinthecurriculumteacherswereincentivizedtoteachgrade-levelmaterialsConsiderationsofhowthelocalcontextinChicagomightaffectimplementationofatargetedteachingprogramincludedunderstandinghowtheneighborhoodsthatthestudentswerefrommayhaveaffectedtheireducationaloutcomesandthehighcostofteachinginsmallgroupsInChicagoMatchEducationprovidedtwo-on-onetutoringwhichmatchedstudentswithwell-educatedindividualswhohavenothadformalteachertrainingforalower-costmodelTheresultswerepositiveStudentswhoweretutoredperformedbetterinmathandweremorelikelytopasshighschoolclasses
ThegeneralizabilityframeworkproposedbyJ-PALisbelowUsethisasatemplateasyouconsiderpotentialpolicysolutions
Decideonapolicytoimplementbasedonlocalfactors
30
SourceTheGeneralizabilityPuzzle
OnceyouhavenarrowedthepolicyoptionsdiscussthequestionsbelowforeachoptionsyouareconsideringThisstepiscriticalasunderstandinglocalcontextissoimportantinsuccessfulpolicyimplementation
CommunityContextQuestionnaire
Localcontext
DoesthisoptionmakesenseinourcommunityIsthisaonesizefitsallapproachIsanyparticulargroupabsentfromthisapproachIfsoisthatok
Youranswers
Effectiveness
WhatwouldweexpecttoobserveifthepilotisaseffectivehereasinCityX
Youranswers
Timing
DoesthetimingmakesenseIsthistherighttimetoimplementthispolicychange
Decideonapolicytoimplementbasedonlocalfactors
31
WherearethepotentialroadblocksthatcouldaffectthescheduleIsthereanywaytomitigatethisHowlongwouldittaketoseeanimpact
Youranswers
Implementation
DowehavethestaffcapacitytoimplementthisArethereadditionalpartnersthatyoucanbringtothetabletosupportimplementationDoesthismakesensegivenourformofgovernment
Youranswers
Authority
DoesourcityhavetheauthoritytoenactthisIfsowhatistheappropriatemechanismAretherelegalbarriersArethereanyexistingpreemptionissuesfromthestateorfederalgovernmentCanthosebeaddressedthroughlegislationadvocacyIstherepoliticalsupportoroppositiontothis
Youranswers
Cost
HowmuchwillthiscostannuallyHowwillwepayforitCanthepolicybeimplementedwithexistingfunding
IfsoisthisfundingreliableHowwouldthispolicyoptionaffectourcitysbudgetintheshortandlongtermWhatotherfundingoptionscanweconsiderIsthereanalternativethatwouldcostlessandgiveussimilarresultsWillanyconstituencybeadverselyaffectediffundingforprogramsthatimpactthemisshiftedtoanotherprogrampolicyorinitiative
Youranswers
Decideonapolicytoimplementbasedonlocalfactors
32
EstablishaframeworktomonitorprogressNowthatyouhavereviewedtheevidenceconsideredlocalcontextandchosenyourpolicyitistimetodeterminehowyourcitywillmonitorprogressofthisnewapproachCitystaffandleadersneedtoestablishtheinfrastructureforongoingcontinuousdecisionmakingbasedonwhetherthepolicychangeisleadingtothedesiredoutcomesforresidentsandwhetherthechangeismeaningfulandrationaltofrontlineworkersSettingarbitrarygoalsatthelocallevelthatconflictwithgoalsatotherlevelsisconfusingtotheindividualsresponsibleforachievingthegoalsGoalsthataresettoohightoachievecanbedemoralizingtostaff
ThissectionincludesstepstosetupavenueinwhichCitystaffandleadershiproutinelyvisitkeyindicatorstodeterminewhatisorisnotworkingandtomakedecisionsinreal-timeFormorereadingonperformancemanagementpleasevisittheGovExPerformanceManagementGettingStartedGuide
InperformancemanagementtermssuchasindicatormeasureandmetricmaybeusedinterchangeablyInthisguidewewilluseindicatorInyourworkthemostimportantthingistobeconsistentinwhichevertermyouuseThisvideomaybehelpfulinunderstandingthisconceptandcommunicatingittoothers
Establishthefundamentalsandgatheryourteam
FirstdeterminewhetherthereisanestablishedvenueorregularmeetingscheduleduringwhichyoucandiscussprogressIfyourcityhasanexistingperformancemanagementstructureonarelatedtopicaddingadiscussionoftheprogressofpolicychangetoyouragendacanbeagreatideaIfthiswillbeastandalonemeetingdecidehowfrequentlytheprojectteamanddecisionmakerswillmeetandputthosemeetingsonthecalendarfortheentirelengthoftheproject
DeterminewhowillstaffandattendroutinemeetingsonCityprogressThesemeetingsmayincludetheMayorandorCityManagerdepartmentheadstaffwhoareimplementingthepolicychangeandanalysts
Identifyindicators
ItiscriticalthatindicatorsbeanimportantcityandcommunitypriorityaswellassomethingtheCityplaysamajorroleinaddressingTheremaybeinstancesinwhichtheCitywillneedtoreachouttoorcollaboratewithexternalpartnerstogetamorecompletepictureofprogressontheoverallpolicychange
Establishaframeworktomonitorprogress
33
Anoutcomeindicatordescribesthedesiredeffectofthetargetedpopulation
ExampleBellevuesfamilyfriendlyneighborhoodsindicatormeasuresthepercentofresidentswhobelievetheyliveinaneighborhoodthatsupportsfamiliesparticularlythosewithchildren
WithyourkeypartnersrevisityourproblemstatementcurrentstateandfuturestateexerciseandbegintoidentifytheoutcomeindicatorsthatwillillustratesuccessFornowchooseasingletop-levelkeyperformanceindicator(KPI)ItiscriticalthattheKPI-thepieceyouwillmeasure-beapriorityforthecommunityandsomethingtheCityplaysamajorroleinaddressing
Outputindicatorsmeasuretheprocessofprogramorservicedelivery
ExampleSanDiegoMayorKevinFaulconersgoaltorepair1000milesofstreetby2020
ReviewingtheadministrativedatasetinventorybegintoparedowntherelevantprogramsanddatasetsthatsupportyourKPIWhatarethemeasuresofeffectivenessforeachprogramTheseareyouroutputindicators
Thediagrambelowillustratestherelationshipbetweenoutputandoutcomeindicators
ThinkaboutimplementationindicatorsasmeasuringstepsintheprogramsrolloutthatarenecessarytoobtaindesiredchangesImplementationoutcomescanfocusontheinterventionstaffdeliveringtheinterventionanorganizationorsystemandcanaddressfeasibilityadaptabilitysustainabilityandcostTwokeyquestionstoconsiderwhendevelopingimplementationindicatorsare
1 Istheinterventionacceptableandappropriatetothecommunityand
2 IsitfeasibletoimplementinterventionXatthistimewiththeseresources
Establishaframeworktomonitorprogress
34
Amoreextensivelistwithdefinitionsofpotentialimplementationoutcomesislinkedhere
UsethetablebelowtochoosethedesiredimplementationoutcomesforyourpilotandtoplanyourmeasurementtechniqueThisshouldincludelevelofanalysis(individualproviderorconsumerorganizationorsetting)yourmeasurementtechnique(surveysobservationinterviewfocusgroupsadministrativedataanalysis)andthephasewhenyouwillcollectthisinformation(earlymidlate)Theremaybemorethanonelevelofanalysismeasurementtechniqueandphaseofdatacollection
Implementationoutcome
Definitionofoutcome
Levelofanalysisandmeasurement
Yourmeasurementtechnique
Phaseofpilotwhenyouwillcollectdata
Usethespacebelowtocaptureyourindicatorwishlist
Indicatorwishlist
Outcomes_____________________________________________________________________________
Outputs________________________________________________________________________________
Implementationoutcomes______________________________________________________________
Setapublichigh-levelgoal
Thehigh-levelgoaliscriticalinestablishingdirectiontogalvanizethepeoplewhowillbedoingtheworkWhendiscussingthegoalusetheMayorsbullypulpitLookforkeymilestonesinCitygovernmenttosharethegoalstatementsuchasthebeginningofaMayoraladministrationaStateoftheCityspeechorotherpublicremarks
GoalsshouldbeldquoSMARTrdquo
Specificaspossible(innamingthepopulationwhowillbebetterofftheactionthecitywilltake)
Measurable
Achievable
Establishaframeworktomonitorprogress
35
Results-focused
Time-bound
ThegoalstatementshouldincludeyouroutcomeindicatorlikeChattanoogasgoalldquoWewillendchronicveteranhomelessnessbyDecember2016rdquoAdiagramtohepyoubuildaSMARTgoalaroundyouroutcomeindicatorisbelow
Component Description Yourturn
Specific WhowillbebetteroffWhatactionthecitywilltake
Measurable Howwillyoumeasuresuccess
Achievable Isthegoalattainable
Results-focused Whatistheexpectedresult
Time-bound When
Setrealisticambitioustargetsandtimelinesforeachindicator
ToestablisharealisticandambitioustargetforyourgoalstatementusetheadministrativedatacollectedforyourchosenKPItoreviewactualCityperformanceinthemostrecentyearcomparedwiththethreeorfourmostrecentyearsworthofdataBegintocalculatesummarystatistics(egtotalminimumaveragemedianmaximum)toidentifytrendsCreatecumulativeandmonthlychartstoidentifytrendsandanomaliesBasedonyouranalysisofhistoricaldataignoretheanomaliesandassumetheproductivitytrendswillfollowthesameoverallpatternfrommonth-to-monthFinallyapplyanincreasedproductivityfactortotheperformanceeachmonthtoforecastfutureprogressBasedonthisanalysischooseatargetthatinspiresimprovementwhileaccountingforreality
Manycitiesthataremorematureintheirperformancemanagementpracticesrecognizetheimportanceoftakingacommunity-levelapproachtoimprovingoutcomesforresidentsThesecitiesarecreatinggoaltaxonomiesthatincludemeasuresrelatedtopublicsafetyhousingandevenrecreationCitiesdriveprogressonthesegoalsbyimplementingstrategiesthatengagemultipledepartmentsandcollaboratewithserviceprovidersinthecommunity
EventuallyyouwilldevelopgoalstatementsfortheoutputsinordertocapturetheefforttoachievetheoutcomecontainedinyourgoalstatementYoursetofindicatorsshouldbecomposedofshort-andinterim-termsupportivemeasureswithaccuratetargetsandrelevanttimelinesAnexamplegoaltaxonomyfromtheCityofLasVegassResultsVegasprogramisbelowUsethistaxonomyasamodelforyourownsuiteofindicatorsandgoalstatements
Establishaframeworktomonitorprogress
36
Yourturn
Thetemplatebelowismeanttohelpyoucraftagoalstatementforeachsupportingmeasure
Goalstatement
Departments
Outcome
Activities
KPI
Target
Timeframe
Conductanddiscussrelevantanalysis
ThrougharoutineseriesofmeetingscontinuallymonitorprogressofKPIssupportingindicatorsandanyotherdatacollectedsuchasqualitativedatatoidentifytrendsthatwillindicatewhetherthechosenpolicyishavingthedesiredimpactThesedatapointsshoulddrivethediscussionduringthesemeetingspointingtoopportunitiesforfurtherinquiryinto
Establishaframeworktomonitorprogress
37
whatisdrivingtheobservedtrendsAnalystsshouldbeencouragedandtrainedtoaskquestionsofthedepartmentsproducingthedataandconductsitevisitswhenpossibletodeepentheirunderstandingofthedataPresentationoftheanalysisshouldfocusononeideaatatimetokeeptheconversationfocusedonkeypointsandspecificopportunitiesforimprovementYoumayconsiderdevelopingapublicfacingdashboardtosharethisprogressmorebroadly
AgeneralcomparisonofthestatusquodatawithwhatwasobservedduringthepilotcanprovidedecisionmakerswithaquickunderstandingofthedifferencesbetweentheoldandnewapproachesalthoughthiswillnotshowcausalityOftenthesecomparisonswillincludeoperationalmetricssuchasdaysordollarssavedbutrememberthattheimpactonpeoplewillbemostimportantIfyouareworkingwithanevaluationpartnertheywillsupportthisstep
Usethespacebelowtosummarizetheresultsofyouranalysis
MeasureTargetvalue(seesectiononsettingtargetsabove)
Pilotvalue(whatdidyouobserveduringthepilot)
Historicalaverage(ifavailable)
Results(WhoisbetteroffIsanyonenegativelyimpacted1-2sentencesreassessingyourgoalstatement)
DocumentyourprocessandprogressincludingfeedbackfromtheparticipantsimplementationpartnersandtheprojectteamDidanythingunexpectedoccurduringthepilotArethereinternalandexternalfactorsthatmayaffectyouroutcomesThisiswheretheimplementationoutcomescanbecanhelpfulIftheinterventiondidnotgoasplannedconsiderwhetheritisbecausetheinterventionwasnotacceptableorfeasible
UsethetemplatebelowcreatedbyDenversPeakAcademytocapturethisinformationIfthe306090daytimelinedoesnotmakesenseforyourcityfeelfreetocustomizeit
CurrentState
FutureState
30Day
60Day
90day
Whatwentwell(+)
Whatdidnotgowellshouldbechanged(-)
IfaquestionordiscussionpointarisesthatcannotbeansweredinthemeetinganassignedstaffmembershouldcapturethequestionduringthemeetingAfterwardthefollowupitemsshouldbesenttorelevantpartnersandthenextmeetingshouldbeginwithadiscussionof
Establishaframeworktomonitorprogress
38
thoseitems
Takeaction
DiscussionsaboutwhetheryourindicatorsaremovingintherightdirectionandwhatmightbedrivingtrendsmayleadtoadiscoverythattherearenotenoughresourcesallocatedtothispolicyorissueTheremaybelimitedfundingforstaffprogrammingortechnologythatisdrivingthedatainthewrongdirectionIfCityleadershipisintheroomthisisagreattimetomakethecaseforadditionalfundingduringthebudgetseasonBycollectinganalyzinganddiscussingdataonthispolicyroutinelyyouarebuildingevidencetomakethecasefortherightleveloffundingtoreachtheestablishedtargets
Establishaframeworktomonitorprogress
39
Assessdataaccessibilityqualityandsecurity
Determineaccessibilityandqualityofdata
OnceyouhavecompiledyourwishlistofindicatorsitistimetodeterminetheaccessibilityandqualityofthedataTopreventcomplicationyoumaywishtoprioritizerelevantindicatorsforwhichyouarealreadycollectingdataAvoidthetemptationofadoptingatechnologysolutiontocollectnewdataatthispointsuchasaphotosurveytoolorotherappFornowyouaretryingtodeterminewhichdatasetsarerelevantaccessibleandhighqualityBesuretoworkwithyourkeyinternalpartnerswhileyouareidentifyingyourmeasuresDepartmentstaffwhoworkonprogramsarefamiliarwiththeindicatorsthatwillaccuratelysupporttheoutcomesanddatabaseadministratorswillbevaluablewhendiscussingaccessibilityandqualityofthedata
Abrieflistofconsiderationsfortheaccessibilityofyourdataisbelow
ArewecurrentlycollectingthisdatasetorwillwebecollectingitforthefirsttimeHowfarbackdoestherecordgo(timehorizon)WouldweneednewtechnologytocollectoranalyzethedataCanthedatabecollectedmanuallyorcanitbeautomatedHowfrequentlyisthedataproduced(continuouslyhourlydailyweeklymonthlyquarterlyyearlyetc)IsthedataavailablefortheentireCitypopulationorasubsetofthepopulationHowgranularisthedata(individuallevelCensusblockneighborhoodcitywideetc)DoesthedatacontainsensitiveinformationIsthereaproxymeasurethatwecanusefornowIsthedatamachinereadable
AsnotedbytheWilsonSheehanLabforEconomicOpportunities(LEO)attheUniversityofNotreDameleveragingadministrativedatacansignificantlyreducethecostsandtimelinesofrigorousevaluationCitiescanestablishaninternaldatasharingagreementacrossdepartmentsincludinglanguagethatenablesadministrativedatatobeusedforresearchpurposesCitiescanalsoidentifyexternaladministrativedata-holderssuchasacountyschooldistrictpolicedepartmenthospitaloranotherstatethatmayhaveoutcomedataimportantformeasuringprogramimpactCitiescanleverageexistingrelationshipsandorworkwithresearchinstitutionstodevelopstreamlineddatasharingprocessesandagreementswiththeseexternalentities
Assessdataaccessibilityqualityandsecurity
40
Evaluatethequalityofyourdata
QualityisasimportantasaccessibilityToevaluatedataqualityyoumaywishtocreatearubricliketheonebelow
Value ValueDescription
3 ExcellentDataQualityThedataiscompleteaccurateandupdatedregularly
2 AdequateDataQualityThedataissomewhatcompleteaccurateandupdatedfairlyregularly
1 PoorDataQualityThedataisnotcompleteaccurateorupdatedregularly
ThisisagreattimetobuildonthedatainventorythatyoustartedinthegatherinformationpageSuggestedadditionalfieldstoaddtoyourdatainventorytemplatearebelow
Program DatasetTitle
DatasetDescription DataSource
Departmentorexternalpartner
DoestheCityalreadycollectthisdata
Whatisthenameoftheprogram
Human-readablenameofthedataset
Shouldbeunderstandablebynon-technicalusers
Whatistheoriginalsystemapplicationfilethathousesit
Departmentorexternalpartnerthatmaintainsthedata
Isthisdataalreadycollected
Updatefrequency-ThefileformatforthedatasetTypicallycsvxlsxshapefiletxtetc
Format-Howoftenthedatasetisupdated
Timerange-Theperiodoftimecoveredinthisparticulardataset
Datause-Howisthisdatasetcommonlyused
Datausers-Whotypicallyusesthisdataset
Dataownerstitle-Titleofthedataowner
Dataownersposition-DataOwnersjobtitle
Dataownersemailaddress-DataOwnersemailaddressWhomanagesthedataandorisresponsibleforgrantingpermissiontoaccessthedataWhounderstandswhatthedatasetincludesandcananswerquestionsaboutit
Dataqualityconcerns-Describetheconcernsyoumayhaveaboutthequalityofthedataset
Assessdataaccessibilityqualityandsecurity
41
Division-DivisionwherethedataismaintainedIfmorethanonedivisionisresponsiblelisttheprimaryownerofthedataset
Sensitivedatacomments-DescribetheissuesthatwouldpreventthecityfromsharingthedatapubliclyCommonsensitivityconcernsincludeprivacyviolationssecurityissuesorhighcostorstaffdemandstopostthedata
Publishingstatus-Thistheplacetoputanycommentsregardingthecurrentstatusofthedataset
Publishedlink-Thisisthelinktothepublisheddatasetontheopendataportalorcitywebsite
Reviewdatasecurityandprivacyprotectionprotocols
OftenapolicychangealsoinvolvesachangeinthedatathatisbeingcollectedhowthatdataisbeinghandledwhohasaccesstoitandahostofotheradjustmentsAsyoutakestockofthedataassetsthatareavailableforyourinitiativeitisvitaltoalsothinkthroughwhatwillbedonetoprotectthatdataandtheprivacyofthepeopleitrepresentsAbigpartofgooddatasecurityinvolvesstrongdatagovernancepracticesthatdefineandidentifysensitivedataandconsciouslyworktobalancethevalueandriskofstoringsuchsensitivedataOnewayofevaluatingyourdatacomesfromcaselawrelatedtotheUSFreedomofInformationActwhichwehavetranslatedtothechecklistbelow
1 Woulddisclosureresultinasubstantialinvasionofprivacy2 Haveweconsideredtheextentorvalueofthepublicinterestandthepurposeorobject
oftheindividualsseekingdisclosure3 Istheinformationavailablefromothersources4 Wastheinformationgivenwithanexpectationofconfidentiality5 Isitpossibletomouldreliefsoastolimittheinvasionofindividualprivacy
TheseriskscouldbemitigatedusingavarietyofstrategiesfromaggregatingdatatoobfuscatingindividualstoredactingfieldsthatmayincludePersonallyIdentifiableInformation(PII)orre-evaluatingyourdatacollectionmethodstofindalternativewaysofgettingtheinformationyouneedThesestepsshouldbecodifiedandmanagedbyadatagovernancecommitteeifpossibleForthesensitivedatathatisdeemedessentialcreateaplanaroundpotentialdatabreachesinordertoquicklyandconfidentlyrespondtoemergencysituations
ProactivelypreparingforthesesituationsisoftenthebeststrategyfordatasecuritybutdonotforgetthetechnicalsideInvolveyourITadministratorsinconversationsaboutdatastorageandaccessensuringthattheywillbeprotectedbyenterprise-gradecybersecuritysystems
Assessdataaccessibilityqualityandsecurity
42
Assessdataaccessibilityqualityandsecurity
43
Designimplementationandevaluation
Designtheimplementation
NowthatyouhaveyouraccountabilityplanandkeyindicatorsitistimetogetspecificaboutyourplanforimplementationThetemplatebelowwillhelpyouthinkthroughthespecificmechanismsofimplementingpolicychange
UnderstandingorganizationsandsystemsiscriticaltounderstandingandmakingdecisionsaboutwhereandhowinterventionsshouldbeappliedApolicyimplementedinasinglesettingmayshowpromisingresultsbutimplementingmultiplepoliciesandprogramsatthesametimewithdifferentactorsresponsibleforeachaspectcanhelpensuresustainablebehaviorchange
InSomervilleMAShapeUpSomerville(SUS)isacitywidepublichealthprogramthatisimplementedacrossmultiplelevelsofinfluenceandsystemsthroughinterventionsbeforeduringandafterschooltoincludechangestobuiltenvironmentsuchasbikelanesregionalmasstransportationandphysicaleducationequipmentinschoolsandgymstoincreasephysicalactivitywellnesspoliciesandschoolgardensnutritioneducationduringandafterschoolandhealthyrestaurantprogramstosupporthealthyfoodandhealthyeatinghabits
Usethetemplatebelowtoplanyourimplementation
Policyimplementationplanningtemplate
ListthespecificdesiredoutcomesInthepreviousstepyouworkedwithdepartmentstafftoestablishspecificmeasurableoutcomesandinterimindicatorsPrioritizetheindicatorsthatwilldemonstrateprogressUsethespacebelowtocapturethedesiredoutcomesofthepolicychange
ExampleShapeupSomervillebeganwiththegoalofpreventingobesityin1stthrough3rdgraders
Yourturn
____________________________________________________________________________
____________________________________________________________________________
Designimplementationandevaluation
44
SpecifythetargetpopulationAreyoudesigninganinterventionthattargetsonlycertainsitesAreyouplanningonimplementingaprogramorinitiativeforaspecifictargetaudienceClearlyspecifythepopulationyouplantotarget
IfyouplantoconductarigorousimpactevaluationasnotedbytheWilsonSheehanLabforEconomicOpportunitiesitisimportanttoestablishconcreteobjectiveeligibilitycriteriaandensurethatthesecriteriaareappliedconsistentlybyallstaffduringscreeningandintakeItisalsovitaltoidentifytheindividualswhoareeligibleforaparticularprogramorserviceevenifeveryoneeligiblecannotbeservedbyexistingresourcesThishelpscitiesidentifyanyunmetneedsandguidefutureresourceallocationwhilealsocreatingthepossibilityofarigorousimpactevaluationbyidentifyingapotentialcomparisongroupFinallywhenconductingscreeningandintakeitisimperativetocollectandmaintainbasicinformation(egnamedateofbirthgender)aboutallprogram-eligibleindividualsnotjustthosewhoparticipateintheprogram
ExampleIntheShapeUpSomervilleexampletheprogrambeganbytargeting1stthrough3rdgraders
Yourturn
____________________________________________________________________________
____________________________________________________________________________
DefinethelocationWherewillthepolicychangetakeplaceWillimplementationtakeplaceatonesiteormultiplesites
ExampleShapeUpSomervilleinterventionsweredeliveredinschools
Yourturn
____________________________________________________________________________
____________________________________________________________________________
ClarifyimplementationpartnersandarticulatetheirrolesWhatarethekeyorganizationsinstitutionsorpeopleresponsibleforimplementationTheprojectteamshouldbeasclearandspecificaspossibleinprovidingadescriptionoftheconcreteactivitiesthatimplementationpartnersaremeanttocarryoutPracticeprofilesmaybeahelpfultooltohelpdefinewhatisrequiredoftheactorswhoareresponsibleforimplementingtheintervention
Designimplementationandevaluation
45
ExampleInSomervillemayoralleadershipandengagementofmultiplecityagencies(publichealthprimarycareschoolsCambridgeHealthAllianceetc)streamlinedandconnectedthevariousShapeUpSomervilleinitiativesandgrantswhichhelpedtokeepthetalentandenergyfocusedonhealthyeatingandactivelivingforallsectorsofSomervilleSUSpartnersultimatelyincludedschoolsnonprofitsCityandstateagenciesandnationalfoundations
Yourturn
____________________________________________________________________________
____________________________________________________________________________
DescribetheinterventionWhatarethestepsactivitiesorprocessesRememberthatthemechanismthespecificstepsorprocessthatleadtobehaviorchangeiskeytounderstandingtheeffectivenessofthepolicychangeandsharingtheresultswithothers
ExampleGoalsofShapeUpSomervilleweretoinfluenceeverypartofanearlyelementaryschoolchildrsquosdayndashbeforeduringandafter-schooltopromotehealthyeatingphysicalactivityandtheirweightstatusInterventionincludedfoodserviceenhancementstoimprovebreakfastandlunchandstaffprofessionaldevelopmentwalktoschoolactivitiesSUSclassroomcurriculumandprofessionaldevelopmentschoolwellnesspolicydevelopmentSUSafter-schoolcurriculumandprofessionaldevelopmentoutreachandeducationtohomethroughmaterialsforumseventsldquoSUSapprovedrdquorestaurantscommunityoutreachandcapacitybuildingthroughpolicydevelopmenttrainingsmediaplacements
Yourturn
____________________________________________________________________________
____________________________________________________________________________
ClarifytimelineandmilestonesfordeliveryandevaluationHowfrequentlywilltheservicesinterventionbedeliveredHowlongwilltheprogramrunWhatisthetimetoexpectedimpacts(eg12monthsafterprogramcompletion)
Designimplementationandevaluation
46
ExampleShapeUpSomervilleledbyTuftsUniversityFriedmanSchoolofNutritionScienceandPolicyconductedaquasi-experimental(nonrandomized)interventionstudyamong1stthrough3rdgradestudentsinSomerville(intervention)andtwocomparisoncommunitiesfor9-monthsduringoneschoolyear(fall2002ndashspring2003)ThestudyshowedpositiveassociationoftheinterventiononchangeinBMISUSreleasedreportssummarizingevaluationsprogresstodateplansforadditionalprojectsandimpactstenandfifteenyearsintotheinitiative
Yourturn
____________________________________________________________________________
____________________________________________________________________________
PlanyouranalysisTheprevioussectionaddressedestablishinganaccountabilityframeworktomonitorprogressandimplementationDeterminewhichindicatorswillbediscussthroughthatprocesswhowillsharethedataandanalysisandhowfeedbackfromthoseconversationswillbeincorporatedbytheprojectteam
ExampleTheMayorrsquosofficeimplementedadata-drivendecisionmakingapproachcalledSomerStatwhichwassynergisticwiththeapproachestakenbytheresearchandevaluationteamsdeployedforthespecificgrants
Yourturn
____________________________________________________________________________
____________________________________________________________________________
PlanforsustainabilityIfthepolicyandprogramchangeissuccessfulhowwilltheprojectteamsustainandexpandtoadditionalsitesorpopulations
ExampleAlthoughmostofthepartnershipsforgedintheearlyyearsweremaintainednewgrantsandcollaborationsexpandedattentiontoolderpublicschoolstudentsactivetransportationandtheGreenLinesubwayextensionthebuiltenvironmentfarmerrsquosmarketeffortssuchasldquoGrowninSomervillerdquowithUnionMainStreeteconomicdevelopmentandhealthdisparitiesaswellasdatacollectionmonitoringandsurveillance
Yourturn
Designimplementationandevaluation
47
____________________________________________________________________________
____________________________________________________________________________
Pilotplanningtemplate
IfyouplantoconductapilotfirstdeterminethepurposeofthepilotDoyouwanttoobservewhetherthepolicychangeresultsinthedesiredbehaviorchangeAreyouconcernedaboutfeasibilityscalabilityorcostDoyouwanttotestwhetheryourdesiredresultscanbeachievedthroughthispolicyapproachClarifythegoalofthepilotandthequestionsyouhopetoanswerThiswillhelpyouidentifythekeyindicatorsthatyouwillusetomeasurewhetherthepilotissuccessfulBeforebeginningestablishabaselineofcurrentperformanceandsetyourtargetsaccordinglybasedonwhatyouhopetoaccomplishthroughthepilot
NextestablishthepilotplanforyourprojectteamincludingmilestonesforteamcheckinstodeterminewhethertomoveaheadorchangecourseThisisseparatefromthemeetingsthatwilloccuraspartoftheaccountabilityframeworkUsethespacebelowtocreateyourtimelineanddevelopyourmeetingschedule
Task Personresponsible Startdate Checkindate Deliverable
Planyourevaluation
WhileyouaredesigningtheimplementationitisalsotimetobegintodevelopyourplanforanalysisAsnotedbytheWilsonSheehanLabforEconomicOpportunitiesdecisionsaboutprogramsetup-includingoutreachscreeningintakedatatrackingandenrollment-candeterminewhetherornotarigorousimpactevaluationwillbepossibleAsdiscussedcityleadersmayfindithelpfultoincludeevaluationpartnersinearlyconversationsaboutprogramsetupDuringpreviousstagesyoumayhaveworkedwitharesearchertoplanyourproposedpolicychangeYoushouldbeclearaboutthepurposeoftheevaluationanddevelopanevaluationplanBelowareadditionalconsiderations
Causationvstrendanalysis
IntheEstablishaframeworktomonitorprogressstagetherewasguidanceontrackingtrendsovertimeItisimportanttonoteherethatmonitoringdataisdifferentfromanalyzingacausaleffectTrendanalysiswillillustratethatchangesareoccurringandmayleadtoquestionsthatcouldbeansweredbyyourevaluationbuttrendanalysisalonedoesnotprovideevidenceaboutwhatisdrivingthosechangesYoumayseechangesinyour
Designimplementationandevaluation
48
outcomeindicatorsbasedonvariablesoutsideoftheCityscontrolForinstanceachangeinfederaltradepolicycanhaveanimpactemploymentratesinyourCityoutsideofyourCitysnewjobtrainingandplacementprogram
ToaddresscausalityyoumustcomparetheoutcomesofindividualswhoreceivedtheinterventionwiththosewhodidnotWithoutacomparisongroupofsimilarindividualswhowerenotexposedtotheinterventionitisimpossibletoknowwhichfactorsareresponsiblefortheoutcomesRandomizedControlTrials(RCTs)inwhichindividualsarerandomlyassignedtotreatmentandcontrolgroupsarethegoldstandardinprovingcausalityIfyoucandemonstratethatoutcomesimprovedforthosewhoreceivedtheinterventionanddidnotimproveatallorimprovedlessforacomparisongroupyoucanbesomewhatcertainthattheinterventionwasthecause
Examplesofothernon-causalimpactevaluationsthatarerigorousandreliableincludecertainquasi-experimentalstudiesusingcomparisongroupdesignssmartinterruptedtimeseriesanalysesthatmodeltrendsovertimeinwaysthatallowestimationofinterventioneffectswelldesignedpre-posttestanalysesneedsassessmentsimplementationstudiesetcFormoreinformationondifferenttypesofevaluationsseetheSageResearchMethodswebsite
Lookfordifferencesingroups
BuildintimetoalsoexaminethepilotdatatoseeifgroupsareexperiencingdifferentresultsReviewingperformancebasedonoutcomemeasuresforapopulationinaggregatemayhidethesubtlewaysthatsomepeoplemaybeaffecteddifferentlysoitisnecessarytocarefullyanalyzethedatafordisparateoutcomesForinstancetheMovingtoOpportunity(MTO)studythatexaminedtheimpactofhousingvouchersonfamilieswholivedinhigh-povertyneighborhoodsfoundthatchildrenwhomovedbeforetheageof13hadbettereducationandhigherincomesasadultsHoweveradultsdidnotseeimprovededucationandemploymentoutcomes
Disaggregatingdataintomorespecificcategoriesofraceandethnicitycanhelpunderstandtrendsanddisparitiesinamoregranularwaywhichcanhelpdecisionmakerstargetpoliciesandinterventionsinamoreequitableway
EarlierintheRoadmapweworkedtoidentifystakeholdersandwhomightbepositivelyandnegativelyimpactedbyachangeinpolicyUsetheresultsofthatexercisetoinvolvethecommunityininterpretingdataanalysisandtogatherqualitativefeedbackfromthecommunitytocallattentiontoissuesForinstanceanincreaseinyouthdetentionmaybeexplainedbyinefficienciesorlackofaccesstotransportation
Designimplementationandevaluation
49
Designimplementationandevaluation
50
Shareyourstoryandsuccess
Sharetheresults
Storytelling
AnintegralstepinyourstrategyissharingtheresultsofyourpilotorpolicychangewithyourcommunityThisincludesdevelopingmessagingaroundtheworkofyourorganizationandcreatingaplantodisseminateyourstoriesWhytellstoriesaboutyourwork
Informandorunderstandasubjectorsituation
Giveneededcontextofdataorbitsofinformation
Relatablestoriesarememorableforreadersviewers
ReadersorviewerswillseetheinformationasmorecompellingwhenthereisademonstratedandrelatablechangeorimpactonthemselvesortheirfriendsneighborsandcolleaguesStoriesareaboutpeopleandgovernmentisfundamentallyabouthelpingpeopleSotellthestoriesofthepeopleyouandyourcitycolleagueshelped
Keepthesetipsinmindasyoudevelopyourstorywithyourteam
WhatwereyourfindingsandwhyshouldanyonecareTellyouraudiencehowtheproblemwassolvedWhathastheimpactbeenonresidentsandcitystafftodateWhatwasthesuccessandweretheremultiplesuccessesalongthewayIfyouhavepreliminaryresultsordatashowingimprovementtosharethisisthetimetodoit
WhowasinvolvedandimpactedReviewthelistofstakeholdersHowwereeachofthemimpactedbytheprogramHumanizeyourstoryandimpactbyaddingadditionalvoicesThesevoicescanbeoneofthefollowing
Apersonorfamilywhowaspositivelyaffectedbythechangesuccess
Aninvolvedthird-partydescribingboththeproblemandthesolution(ieanorganizationnonprofitprovidersbusinesstradeassociationsportsteametc)whoispartofthecommunitythatthegovernmentworkedwithonthechangesuccess
Someonefromthegovernmentitselfwhocanspeakaboutthechangesuccess
TellmoreaboutyourcityDescribeyourcityincludingpopulationdemographicsindustriesetcWhatareitschallengesWhatisitknownforWhatmakesituniqueHowisitsimilarordifferentfromothercitiesintheUS
Shareyourstoryandsuccess
51
DescribeyourcityrsquoschallengeWriteoutthechallengeyourcityisfacing(asdefinedinyourproblemstatement)HowdoesitaffectresidentsHowdoesthisproblemaffecttheworkofcitystaffmembers
AddthedetailsProvidedetailsonyourappliedresearchstrategyTalkaboutwhatyourcityhasdonenthepasttoresolvetheissueDiscusshowyourstrategyhaschangedwhenyoubegantoadoptthisapproachWasmoneysavedorinvested
Usethisstorymapexercisetoorganizeyourthoughtsandguaranteethatyouincludealloftheelementsofatraditionalstory
AfteryoudevelopyourmessagingcreateacommunicationsmarketingandoutreachstrategydesignedtospreadyourmessageYourstrategyshouldwalkyouthroughthefollowingkeyactivities
1 IdentifyandprofileyouraudienceWhencreatinganddisseminatingyourmessageitisalwaysgoodtothinkaboutyouraudienceWhoexactlywouldbeinterestedinwhatyouhavetosayThegeneralpopulationofthecityPeoplewhoworkinnonprofitsBusinessownersItmattersbecausenoteveryonewholivesinacitymaybethetargetoftheinformationBusinessownersmaybemoreinterestedindatashowingeffectsonthebusinessdistrictthanothersandsoonWewillcoverthislaterbutthisstepwillalsohelpyoutothinkofthemosteffectivewaytoreachyouraudience
2 Defineyourcalltoaction(ifyouhaveone)SpendsometimeconsideringwhyyouareinterestedintellingyourstoryDoyousimplywanttoupdatethepubliconyourprogressorcelebrateyoursuccessesAreyouinterestedinyouraudiencechangingitsbehaviororsupportingaspecificinitiativebeinglaunchedbyyourorganizationIfyouwanttohaveyouraudienceactorbehaveinacertainwaybespecificandincorporateitinyourmessaging
Shareyourstoryandsuccess
52
3 SelectperformancemeasuresThinkabouthowyouwoulddetermineifyourcampaignissuccessfulandselectappropriateperformancemeasuresYouwillrevisitthesemeasuresattheendofyourcampaigncommunicationsplanimplementationtodeterminewhetheryouneedtomodifyyourcommunicationsandoutreachstrategyforthefuture
4 MethodstocommunicateyourstoryTherearemanywaystocommunicatesuccessofyourprojectHerearesomesuggestionsonreachingaudiencesItisimportanttodeterminewhichmethodsarethemosteffectiveforyourdesiredaudiencetoreceiveinformationDoyourresearch(PewResearchCenterhasexcellentstudiesonthereachofmediadigitalmediaandmarketing)ForexampleifseniorsareyourtargetpopulationandtheyarelessinclinedtouseInstagram(thatistrue)youmaywanttoconsiderFacebookorthenightlylocalnewsbothofwhichareusedmoreoftenbyseniorsBelowisalistofsomemethodsandtoolsyoucanuse
i Pressreleasestolocalnewspapersandtvstations
ii ShortsmartphonevideosAgreatwaytoengageanaudienceisthroughvideoAskonequestionofsomeonewhoispartofthestoryMaybethereisasocialworkerwhocandiscusshungerinyourcityThesevideoscanbereusedonmultipleplatforms
iii Long-formwriting
iv TellingindividualstoriesthroughaseriesofphotosandshortnarrativesforpublicationonFacebookorInstagram
v Op-edsopinionpiecesplacedinthelocalpaper
vi Emailupdatestoyourlist
vii Connectingwithappropriateorganizationsthatareinvestedinthetopicandworkwiththecommunityandaskthemtodisseminateinformationtotheirmembers
viii EmployingldquoWordofMouthrdquo(WOM)tacticsbycreatingtownhallsorfreeeventsforresidentstoattendintheirownneighborhoods
5 DeterminethetypeofcollateralUsuallycampaignsthattargetresidentsrequireseveralavenuestoreachthemPeoplearenotalwaysinthesameplaceorspaceHavingakeenunderstandingofyouraudiencewillassistyouwithyourchoicesonwhatneedstobedevelopedSometimesthatmeansimagesforsocialmediaaonepageflierthatgivespeopleinformationonattendinganeventoranin-depthbrochure
Shareyourstoryandsuccess
53
6 ImplementyourstrategyKeepinmindstrategiescanbefluidSometimessomethingmaynotturnoutasexpectedevenifduringplanningsessionsitwasldquosupposedtordquoItisokYoucananalyzewhathappenedduringadebriefsessionandonceyoursurveysandothermeasurementshavebeenfinalized
Visualizations
Withtherightvisualizationyoucanreviewandshareinformationinsightsanalysisperformanceandtrendsallofwhichisvaluabletoacross-sectionofthepublicButwhichvisualizationisrightforyouTochoosetherighttypeofchartgraphdeterminethetypeofquestionyouwanttoanswerSomecommoncharttypesarebelowandmoreinformationcanbefoundhere
Exploreadditionalanalyticaltechniques
Duringthegoal-settingphaseyouestablishedoutputoutcomeandimplementationmeasuresBasedontheresultsofyourpilotyoumayneedtoadjustyourmeasuresWiththepilotdatainhandyoumayhavesomeideasofhowfutureanalysescouldbedifferentandpotentiallyimprovethequalityofyourprogramTakesometimetobrainstormotherrelevantdatasetsthatyoudidnothavethetimetopursueearlierincludingmessydepartmentaldatathatneedsmoreattentionorsourcesofpublicdatafromthestateandfederalgovernmentnonprofitsandevenprivatecompaniesthatcouldsupplementyourunderstandingofanissueUsingmachinelearningtechniquesyoumightbeabletobetter
Shareyourstoryandsuccess
54
predictdemandforhomelessservicesusingNOAAweatherforecastspreemptwatermainbreaksusingsoilstratigraphyorinvestineconomicdevelopmentbymonitoringrealestatelistingdata
Advancedanalyticaltechniquescanalsohelpyourimplementationrunmoreeffectivelybyintegratingdata-informedtargetinganddecisiontoolsintotheprocessByautomatingroutineworkflowsandsurfacingsubtletrendsthesetoolscanhelpyouiterateinamoreprecisemannerandprovidekeystrategicguidancetofront-lineemployeesthatwouldotherwisebeunavailableThesesortsofdata-science-enhancedprocessescanbeclassifiedasoneofsixprojecttypesasexploredfurtherbytheNewOrleansLANOLAlyticsteamFormoreguidanceonscopingasimilarprojecttheGovExadvancedanalyticsscopingguidegivesyouastep-by-stepoverviewofhowtobesuccessful
Usethespacebelowtolistyourideasforpossibleadvancedanalyticsprojectsbytype
Typeofproject Ideas
Findingtheneedleinthehaystack
Prioritizingforimpact
Earlywarningtools
Betterquickerdecisions
Optimizingresourceallocation
Experimentingforwhatworks
Keepthefeedbackloopgoing
GatherinformationonpublicsentimentaboutpolicychangethroughfocusgroupssurveysandpublicmeetingsledbyadvisorygroupmembersNotewhethernewinformationisavailablethatcanhelpanalyzeorcontextualizeyourprogress(egfederalorstatedatareports)Sharetheresultsofyourpilotandthepolicychangeprocesswithothercitiesandresearchers
Shareyourstoryandsuccess
55
WorkingwitharangeofevidencePresentlythereisalotofattentionandinvestmentinevidence-basedpolicymakingincludingthefederalbipartisanCommissiononEvidence-BasedPolicymakingthePewCharitableTrustsreportsandtoolkitsonHowStatesEngageinEvidence-BasedPolicymakingandtheLauraandJohnArnoldFoundationsupportedEvidence-BasedPolicymakingCollaborativeManyoftheseinitiativesfocusonrigorousevaluation--meaningrandomizedcontrolledtrials(RCT)orotherexperimentalmodels--asthebestkindofevidencetouseHoweverthisbodyofevidenceisofteninsufficientforthespecifickindsofquestionsthatgovernmentscanhavewhenmakingadecisionaboutpolicyInthisguideweproposeanapproachtosupportgovernmentsinincorporatingadditionaltypesofevidenceintotheirdecisionmaking
OnewaytounderstandarangeofevidenceistouseNestarsquosStandardsofEvidencetoframeanddefineevidenceTheBritishnonprofitNestaworkstoinjectpositivechangeintothewayUKgovernmentpoliciesandprogramsfunctionOneoftheirkeyareasofworkhasbeendevelopingWhatWorksCentresorganizationsthatcollectcommissionsynthesizeanddisseminateevidenceaboutinterventioneffectivenessacrossarangeofsocialimpactareasNestadevelopedthefollowingStandardsofEvidencetoprovideausefulwaytodocumenthowdifferentkindsofinformationdemonstratedifferentlevelsofevidence
Workingwitharangeofevidence
56
Theimportanceofpotentiallybiasedinformation
Onechallengepresentedbyanevidenceclassificationthatacknowledgesthevalueofnon-scientificformsofevidenceistofigureouthowtobalanceallofthepiecestogetherwhenyouusethemTounderstandhowtodothiswehavetothinkabouthowtopositionpotentiallybiasedinformationindecisionmaking
WhenwelookatinformationwhichhasnotbeenrigorouslyevaluatedweacceptthatthisinformationmightbebiasedThewholereasonweconductrigorousevaluationistoremovebiasfromourobservationsWeconductstructuredtrialstoproduceresultsthatareconsistentandgeneralizableIfafindingisunbiasedthisaccuracymdashthislackofbiasmdashcanbeprovedbecauseanyindependenttestercanreproduceitwhentheyfollowanidenticalprocess
YoucanlookatpotentiallybiasedinformationasjustinformationthatisearlieroninitspathtorefinementreplicationandverifiedaccuracyApieceofinformationbasedonanyoneindividualrsquosobservationsislikelytobebiasedbecauseitiscomingfromjustonesourceBiasisreducedbygatheringtogetherlotsofindependentsourcesofinformationandfiguringoutitsmostcommonattributes
MaybethebiasedinformationmakesaclaimthatistoolargemdashitsayssomethingistrueunderallconditionswhenitisreallyonlytrueundercertainconditionsorthataneffectofaninterventionisbiggerthanitstrueaverageorthataneffectlastsovertimewhenitdoesnotPerhapschangingtheclaimtobemorespecificmdashcriticallyreviewingtheinformationtoseeifspecificconditionsmustbemettoachievetheoutcomesmdashwouldmaketheclaimmoreaccurate
IfyoutestedyourclaimfounditdidnotholdupinsomecaseschangedittesteditagainuntilyourclaimrepeatedlyheldupthenyoucoulddemonstratethattheclaimwasnolongerbiasedbutaccurateThisprocessoftestingimprovingtestingandvalidatingiswhatwegetfromstructuredscientificstudy
Withindividualpiecesofinformation--crediblehypothesescasestudiesexpertopinion--youcannotassumethatkindofaccuracyAtthesametimetheseearlyobservationsarefoundationaltoanyreplicablescientificobservationsWewillnevergetanywhereifwedonotstartwiththesepiecesofinformationeveniftheymightbebiased
Combininglevelsofevidencetogetthemostdetailedinformationforyourcity
ItisespeciallyimportanttoacceptdifferentformsofevidencewhenyoufacequestionsthathavenotyetbeentestedbyindependentstudiesSomeofthemostimportantquestionsrelatetothingsthatareveryspecifictoyoursituationYoufaceaparticularcombinationof
Workingwitharangeofevidence
57
issuesactorstrendsandresourcesthatisultimatelydifferentfromeveryotherplaceintheworld
ThisisimportantbecauseaspolicypractitionersandnotscientistsyoucaremoreaboutyourspecificcasethanahypotheticalaveragegeneralcaseYoucarelessaboutthetotallikelyrangeofimpactsfromapolicythanhowthissingleimplementationwillworkYouwantinformationaboutpoliciesthatwillprovideyouwiththebestpossiblepredictionabouthowtheywillworkinyourcontext
ByputtingdifferentkindsofevidenceintothesameframeworkNestarsquosStandardsofEvidenceimplythatwhenyoucannotgetrigorouslytestedexperimentallyverifiedinformationitisappropriatetoworkwithwhatwedohaveThisisrelevantforthemunicipalimplementationcontextbecausenostudypulledfromaliteraturereviewwillaccountforallofthevariablesyoufaceinyourowncontext
Intheabsenceofhavingastudythatpredictshowaninterventionworksinyourspecificcontextyouwillneednon-scientificinformationtotrytofillinthosegapsYouwillneedtogatherstoriesfromresidentsstaffmembersandotherstakeholdersthatsitonLevels1or2ofthestandardsofevidence
Someofthekindsofinformationyouwillprobablygetmoreeasilyfrompeoplethanstudiesinclude
howpeopleincitieslikeyoursreactedtoaninterventioninpublicforumshowpeoplewhoimplementedaninterventioninacitylikeyoursfiguredouthowtogetitpassedhowthecitycommunicatedpubliclyabouttheapparentbenefitsanddrawbacksoftheinterventionthepoliticalacceptabilityoftheintervention
Youcanremainsimultaneouslyawarethattheinformationyouaregatheringthroughindividualstoriesisbiased--notgeneralizableinformedbyspecificperspectives--whilealsorecognizingthathavingsomeinformationaboutsomethingimportantisbetterthannoinformationEvidencefromthefirsttwoevidencelevelsareessentialbuildingblocksofknowledgeandperfectlyusefulwhenyoucannotgetdetailedunbiasedknowledge
Youhavetostartsomewhereandyoushouldalwaysstartwithevidenceyouhaveaboutyourveryspecificconditionsthatyouknowareimportantforsuccess
Workingwitharangeofevidence
58
Thankyou
Interviewswithcities
GovExheldsemi-structuredinterviewswithsixstaffmembersinfourcitiestoestablishtheinitialstepsoftheRoadmapWeaskedrespondentstotalkaboutaninitiativeorpolicythattheyhavebeenresponsibleforimplementingintheircityRespondentsfromthreecitiesspokeabouttheireffortstoaddresshomelessnessandrespondentsfromonecityspokeaboutblightAfterrespondentsidentifiedtheproblemtheyspokeabouthowthecityprioritizedthisissuethevariousstepsinvolvedandbarrierspresent
GovExconductedtextanalysisoftheseinterviewswhichrevealedthatpeoplewasthemostcommonlyusedtermduringtheseconversationswhendiscussingthestepstochangeaprogramorpolicyThisisathemethroughoutthisRoadmapThefirstsectionbeginswithidentifyingtherelevantpeopleTheconversationsaroundevidencearefocusedonhumanbehaviorratherthanmethodologyorstrengthofevidence
Thankyou
59
OncewehadadraftoftheRoadmapwefollowedupwithcitiesonanindividualbasistolearnmoreaboutwhatwouldbehelpfulWearegratefultothefollowingcitystafffortheinterviewsandindividualconversations
MichaelBaskinChiefPolicyOfficerChattanoogaTN
MelissaBridgesPerformanceandInnovationCoordinatorLittleRockAR
VictoriaCarreoacutenAdministrativeOfficerLasVegasNV
TyroneGrandstrandCommunityDevelopmentPlannerFargoND
TibLaughlinDirectorofGeneralServicesKansasCityKS
TimMorelandDirectorofPerformanceManagementandOpenDataChattanoogaTN
CharlesRosendahlManagementAnalystDentonTX
Thankyou
60
MelissaSiebenAssistantCountyAdministratorKansasCityKS
GinaThompsonDirectorofStrategicServicesLewisvilleTX
JHUFacultycollaboration
GovExalsoconvenedthecitycontactsandfacultypartnersinaseriesofAdvisoryGroupmeetingsThesemeetingsservedtoconnectcitiesandfacultytounderstandoneanothersperspective
StefanBaralBloombergSchoolofPublicHealth
AshleyBernerJHUSchoolofEducation
MelissaDavey-RothwellBloombergSchoolofPublicHealth
JonLinksBloombergSchoolofPublicHealth
TamarMendelsonBloombergSchoolofPublicHealth
KristinMmariBloombergSchoolofPublicHealth
KeshiaPollackPorterBloombergSchoolofPublicHealth
JoshSharfsteinBloombergSchoolofPublicHealth
LizStuartBloombergSchoolofPublicHealth
LindsayThompsonCareyBusinessSchool
DanielWebsterBloombergSchoolofPublicHealth
Partnersindataandevidence-basedpolicy
Wealsospokewithpartnerswhoareexpertsinthefieldofevidence-basedpolicywhosharedtheirknowledgeandresourcesofcurrentandfuturethinkinginevidence-informedthinking
WendyBarrenoAssociateDirectorWilsonSheehanLabforEconomicOpportunitiesUniversityofNotreDame
TiffanyDavenportDirectorofDataPartnershipsBaltimoresPromise
TomiHiersDirectorBaltimoreCivicSiteAnnieECaseyFoundation
ChrisKingsleySeniorAssociateAnnieECaseyFoundation
KathrynOliverAssociateProfessorLondonSchoolofHygieneandTropicalMedicine
Thankyou
61
KristenWatkinsSeniorPolicyAssociateJ-PAL
GovExParticipants
GovExAnalystMiriamMcKinneyanalyzedtheinterviewtranscriptsandconductedtextanalysisSeniorImplementationAdvisorEmilyShawwillintegrateRoadmapactivitieswithadditionalservicedeliveryactivitiesatGovEx
Thankyou
62
ReferencesAbrahamKGampHaskinsR(2017)Thepromiseofevidence-basedpolicymakingReportofthecommissiononevidence-basedpolicymakingWashingtonDCCommissiononEvidence-basedPolicymakingBigmove2Extendingthegreen(nd)Retrievedfromhttpswwwcityoflewisvillecomdoing-businesslewisville-2025big-move-2-extending-the-greenBatesMAampGlennersterR(2017)ThegeneralizabilitypuzzleRetrievedfromhttpsssirorgarticlesentrythe_generalizability_puzzleBuildwithnotfor(2018January)RetrievedfromhttpsbuildwithnotforwordpresscomCairneyPampOliverK(2017)Evidence-basedpolicymakingisnotlikeevidence-basedmedicinesohowfarshouldyougotobridgethedividebetweenevidenceandpolicyHealthResearchPolicyandSystemsdoi101186s12961-017-0192-xCairneyPOliverKampWellsteadA(2016)TobridgethedividebetweenevidenceandpolicyReduceambiguityasmuchasuncertaintyPublicAdministrationReview76(3)399-402Retrievedfromhttpdxdoiorg101111puar12555Centreforpublicimpact(nd)RetrievedfromhttpswwwcentreforpublicimpactorgCernaL(2013)ThenatureofpolicychangeandimplementationAreviewofdifferenttheoreticalapproachesOrganisationforEconomicCo-OperationandDevelopmentRetrievedfromhttpswwwoecdorgeduceriTheNatureofPolicyChangeandImplementationpdfColoradoSprings(nd)Retrievedfromhttpscoloradospringsgovcommunicationspagewhat-expect-when-you-speakupContinuumofcare(coc)program(nd)RetrievedfromhttpswwwhudexchangeinfoprogramscocCouncilpresidentkicksoff$12millionyouthfund(nd)Retrievedfromhttpwwwbaltimorecitycouncilcomcontentcouncil-president-kicks-12-million-youth-fundDurlakJAampDuPreEP(2008)ImplementationmattersAreviewofresearchontheinfluenceofimplementationonprogramoutcomesandthefactorsaffectingimplementationAmericanJournalofCommunityPsychology41327-350Retrievedfromhttpswwwncbinlmnihgovpubmed18322790Fishbonediagram(ishikawa)(nd)Retrievedfromhttpasqorglearn-about-qualitycause-analysis-toolsoverviewfishbonehtmlFivewhysGettingtotherootofaproblemquickly(nd)RetrievedfromhttpswwwmindtoolscompagesarticlenewTMC_5Whtm
References
63
FligsteinNampMcAdamD(2011)TowardageneraltheoryofstrategicactionfieldsSociologicalTheory29(1)1-26RetrievedfromhttpsociologyberkeleyedusitesdefaultfilesfacultyfligsteinFligstein20McAdam20Soc20Theory20PaperpdHeinkelP(2018January12)CitymullspermitsforbikesharingcompaniesRetrievedfromhttpwwwdentonrccomnewsdenton20180112city-mulls-permits-bike-sharing-companiesHowisareamedianincomecalculated(2012October)Retrievedfromhttpswwwhudexchangeinfofaqs843how-is-area-median-income-calculatedLampkinLMampHatryHP(2003)KeystepsinoutcomemanagementUrbanInstituteRetrievedfromhttpswwwurbanorgresearchpublicationkey-steps-outcome-managementMarshDampSmithM(2000)UnderstandingpolicynetworksTowardsadialecticalapproachPoliticalStudies484-21Retrievedfromhttpjournalssagepubcomdoiabs1011111467-924800247McCannL(nd)RetrievedfromhttplaurenellencomMoultonSampSandfortJR(2016)ThestrategicactionfieldframeworkforpolicyimplementationresearchPolicyStudiesJournal45(1)Retrievedfromhttponlinelibrarywileycomstore101111psj12147assetpsj12147pdfv=1ampt=jdkhhcp4amps=e184c625e1855a6b1615ccb99ee27b1c0b3212d8PaynterB(2018January1)What100citiesarelearningfromeachotherbysharingtheirdataRetrievedfromhttpswwwfastcompanycom40518004what-100-cities-are-learning-from-each-other-by-sharing-their-dataPeakacademy(nd)Retrievedfromhttpswwwdenvergovorgcontentdenvergovenmayors-officeprograms-initiativespeak-performancepeak-academyhtmlPinnockHBarwickMCarpenterCREldridgeSGrandesGGriffithsCJRycroft-MaloneJMeissnerPMurrayEPatelASheikhAampTaylorSJC(2017)Standardsforreportingimplementationstudies(stari)ExplanationandelaborationdocumentBMJOpenRetrievedfromhttpbmjopenbmjcomcontentbmjopen74e013318fullpdfProctorESilmereHRaghavanRHovmandPAaronsGBungerAGriffeyRampHensleyM(2011)OutcomesforimplementationresearchConceptualdistinctionsmeasurementchallengesandresearchagendaAdministrationandPolicyinMentalHealth38(2)65-76doi101007s10488-010-0319-7TandonSD(2018)ProgressincommunityhealthpartnershipsResearcheducationandactionRetrievedfromhttpswwwpressjhuedujournalsprogress-community-health-partnerships-research-education-and-actionProvanKGampKenisP(2007)ModesofnetworkgovernanceStructuremanagementandeffectivenessJournalofPublicAdministrationResearchandTheory
References
64
18(2)229-252Retrievedfromhttpsacademicoupcomjpartarticle182229935895SabatierPA(1988)Anadvocacycoalitionframeworkofpolicychangeandtheroleofpolicy-orientedlearningthereinPolicySciences21(3)129-168Retrievedfromhttpwwwjstororgstable4532139SandstromAampCarlssonL(2008)TheperformanceofpolicynetworksTherelationbetweennetworkstructureandnetworkperformancePolicyStudiesJournal36497-524Retrievedfromhttponlinelibrarywileycomdoi101111j1541-0072200800281xabstractSchwartzMampWilsonE(nd)WhocanaffordtoliveinahomeAlookatdatafromthe2006americancommunitysurveyRetrievedfromhttpswwwcensusgovhousingcensuspublicationswho-can-affordpdfShearerJCAbelsonAKouyateBLavisJNampWaltG(2016)WhydopolicieschangeInstitutionsinterestsideasandnetworksinthreecasesofpolicyreformHealthPolicyandPlanning31(9)1200-1211Retrievedfromhttpsdoiorg101093heapolczw052ThelessonsofadministrativedataHigh-profilepolicy-relevantresearchpoweredbyadministrativedata(nd)Retrievedfromhttpswwwpovertyactionlaborgsitesdefaultfilesresources20161114-The-Lessons-of-Administrative-DatapdfToolsandotherneedsforqualitativeandquantitativemethods(2015February)AgencyforHealthcareResearchandQualityRetrievedfromhttpwwwahrqgovprofessionalsquality-patient-safetytalkingqualityassesstoolshtml
WeaverRCampBagchi-SenS(2013)SpatialanalysisofurbandeclineThegeographyofblightAppliedGeography4061-70
References
65
UniversityPartners
Name Title Affiliation Area(s)ofexpertise
Contactinformation
Examplesofworkingwithnon-academicdecisionmakers
WendyPBarreno
AssociateDirector
WilsonSheehanLabforEconomicOpportunitiesUniversityofNotreDame
LEOsfocusareasincludehealtheducationhousingchildwelfareeconomicopportunitycriminaljustice
wbarrenonduduorleondedu
MelissaDavey-RothwellPhD
AssociateScientist
JohnsHopkinsBloombergSchoolofPublicHealth
Health mdavey1jhuedu410-614-5854
BaltimoreCityHealthDepartmentLeadersMarylandDepartmentofHealthAddictionRecoveryServicesinDCExecutiveDirectorsofcommunityorganizationsacrosstheUSthatfocusonHIVSTIsubstanceabuseandimplementationscience
ElizabethStuart
AssociateDeanforEducationandProfessorDepartmentsofMentalHealthBiostatisticsandHealthPolicyand
JohnsHopkinsBloombergSchoolofPublicHealth
Educationsocialpolicystatisticalmethods
estuartjhuedu
Resources
66
Management
Design-ThinkingResources
EngagementGuideforCivicTech
Journeymap
PerformanceManagementResources
DenverPeakAcademy
Proxymeasuresguide
Talkingaboutperformancewithoutcomplicatingit(video)
SettingPerformanceTargetsguide
Datavisualizationguide
Performancemanagementgettingstartedguide
Exercises
Fishboneexercise
5Whys
AdministrativeData
TheLessonsofAdministrativeData(JPAL)
EvaluationResources
Quantitativeandqualitativemethods(USDepartmentofHealthandHumanServicesAgencyforHealthcareResearchandQuality)
KeyStepsinOutcomeManagement(UrbanInstitute)
RCT-YES
Sageresearchmethods
BestPracticesforMixedMethodsResearchintheHealthSciences
ACCORDChecklist
Pilotandimplementationplanning
Benchmarkingexamplesandproxymeasures
Seattleopendataplaybook
Resources
67
Datasciencescopingguide
Resources
68
AboutGovExGovExhelpsgovernmentsusedataandevidencetoimproveresidentsrsquolivesWesupportgovernmentsinbuildingtheirinternalcapacitytooperatesustainableanalyticspracticeswhichimprovecommunityoutcomesinfieldsfrompublicsafetytoeconomicdevelopmentGovExhasaproventrackrecordofworkinginconcertwithpartnerstoprovidetechnicalassistanceandtrainingatscale
SinceourlaunchinApril2015GovExhas
Trainedover2000governmentemployeesinpersonandonlineintopicsincludingdatamanagementperformancemanagementopendatacommunityengagementandanalytics
Providedtechnicalassistancetomorethan100localgovernmentsacrosstheglobeimprovingtheiruseofdatatoachieveefficienciesinservicedeliveryandbetteroutcomesforresidents
Assessedthecapacityofmorethan130governmentalorganizationstodeterminehowleadersstaffandexternalstakeholdersmanageandusedata
Localgovernmentsareincreasinglyonthefrontlinesofaddressinglargescaleproblemswithglobalimplicationssuchasclimatechangepublicsafetyandeconomicinequality
Whileevidence-basedinterventionstohelpsolvethoseproblemsareavailablelocalgovernmentstaffhavelimitedtimeandcapacitytofindandimplementresearch-basedsolutionsAsaresultcitiescontinuetoworkthewaytheyalwayshave--makingincrementalimprovementstoprogramsbutnotalwayscreatingchangeatasystem-level
ThroughresourcedevelopmentandcollaborationwithJHUfacultyGovExissupportingcityleadersinusingdataandevidencetomakedecisionsandcreatinglastingrelationshipswithresearchers
TheRoadmapwasdevelopedbyGovExaspartofitspartnershipwithBloombergPhilanthropiesWhatWorksCitiesinitiative
AboutGovEx
69
- Title page
- Introduction and purpose of the Roadmap
- Using the Roadmap
-
- Gather information about the problem
- Strategically engage key partners
- Review the evidence and collect details
- Decide on a policy to implement based on local factors
- Establish a framework to monitor progress
- Assess data accessibility quality and security
- Design implementation and evaluation
- Share your story and success
-
- Working with a range of evidence
- Thank you
- References
- Resources
- About GovEx
-
GatherinformationabouttheproblemInyourearliestphaseitiscriticaltodevelopacomprehensiveunderstandingoftheproblemYouwillwanttolearnaboutthepeoplewhoareaffectedbytheproblemyouaretryingtosolveandthosewhocanimpacttheproblemFromthereyouwillgetinsightintotherootcauseoftheproblemtheworkthathasbeentriedinthepastandtheeffectivenessofthoseeffortsTheworkfromthischapterwillserveasthefoundationforactivitiesinsubsequentchaptersYoumayrevisitthesestepsasyouidentifyindicatorsandplanyourimplementationandanalysis
BesuretodedicatetimeandresourcestothisstepItcanbetemptingtoskipaheadtolookingforsolutionsortoassumethatyouhaveanunderstandingoftheproblemasaresultofapublicengagementprocess
TheCityofLewisvilleTXconductedaCitywidepublicengagementprocesstodevelopitsLewisville2025StrategicPlanOneoftheBigMovesoutlinedintheplanwasExtendingtheGreenorexpandinggreenspaceandconnectingrecreationareaswithneighborhoodsTheactionstepstoachievethisgoalincludedcleaningupcreeksandremovingdebristomakethemmoreaccessibleThecleanupswentsmoothlyinseveralneighborhoodsbuttheCitymetwithoppositionfromoneneighborhoodthatpreferredthenaturalhabitatandexpressedconcernabouttheenvironmentalimpactofthecleanupsAccordingtoLewisvilleiftheCityhadmappedallofthestakeholderstodeterminewhoneededtobeatthetableandwhowasmissingfocusinginformationgatheringontheneighborhoodlevelithaveavoidedthisproblem
Startwiththewhothepeople
Establishaprojectteam
BeginbynamingtheprojectteamforthisprocessincludingaprojectleadTheprojectleadshouldbesomeoneinCitygovernmentwhoisrespectedcangalvanizeotherkeypartnersandhasthetimetoleadthisworkThisissomeonewhomayhavetohavedifficultconversationswithCityleadershipdepartmentsandotherfundedpartnersItisnotnecessaryfortheprojectleadtobeapolicyexpertItismoreimportantforthemtohavetherightrelationshipsandpositionwithintheorganizationTheprojectteamshouldincludeindividualswithdiverseperspectivesfrommultipledepartmentsThereshouldbesomeonewithpolicyexperienceontheprojectteamtogroundtheconversationinrealisticactionHavingateammemberwhocancoordinatewithresearchersonrigorousanalysisandtranslatethatworkforothersiskeyAdditionallybesuretoincludefrontlinestaffworkersTheywillhavevaluableinsightbasedontheirinteractionswithresidentsandpolicysolutions
Gatherinformationabouttheproblem
8
havetomakesensetothepeoplewhowillbechargedwithimplementingthemBuildingastrongdiverseprojectteamisimportanttoensurethattheworkdoesnotliveanddiebyoneperson
Identifykeypartners
NextidentifythepeoplewhotheprojectteamcanworkwithontheissueyouarefacingincludingpeoplewithlivedexperienceintheproblemyourcityistryingtosolvedepartmentstaffandsubjectmatterexpertsoutsideofcitygovernmentThisistheconceptbehindbuildwithnotforamovementledbyLaurenellenMcCannthatemphasizescommunityexpertiseandleadershipMappingoutkeyplayersandtheirrolesiscriticaldotoearlyintheinformationgatheringphasesincecoalitionbuildingissuchanintegralpartofthepolicychangeprocessThisprocessmightconsistofstakeholdermapping(describedbelow)communityresourcemappingandculturalmappingInyourmappingremembertoconsiderpotentialalliesandthosewhomaypresentroadblocksalongtheway
PolicymakingisnottopdownorlinearInsomecasescitiesmayhavetosharedecision-makingpowerwithotheractorsForinstanceinmanycitiesthelocalContinuumofCare(CoC)istheleadentityonhomelessnessservicesCoCsapplyforandstewardfederalfundstoaddresshomelessnessandcoordinateservicesamonglocalprovidersAlthoughcitygovernmentsdohavearoleinCoCstheyarenotalwaystheleadagencychargedwithdistributingfundsandthereforetheirroleinhomelessnesspolicymaybelimitedCountygovernmentsoftenhavehealthdepartmentswhilecitiesdonotInthosecasesthebalanceofpowerfavorscountygovernment
TheCityofDentonTXwasrecentlyfacedwithanissueofresidentcomplaintsaboutanewdocklessbikesharethatarrivedintownandisviewedbysomeasexpensivelitterInresponseDentonsCityCouncilwhichwasalreadydiscussingpotentialrulesforbikesharecompanieselevatedthisissueandchargedcitystaffwithidentifyingasolutionPresentlytheCityisreviewingordinancesfromnearbycitiestodeterminehowtheyaresolvingthisproblemandwhatsolutionsmaybefeasibleinDentonThemappingexamplebelowrepresentslocalactorsinDentonwithastakeinthisissuewhohavemultipleperspectivesandlevelsofpower
Gatherinformationabouttheproblem
9
UsethistemplatedevelopedbytheCentreforPublicImpact(CPI)tobrainstormwhoyoumightreachouttoasyouembarkontheroadtopolicychangeRemembertothinkabouthoweachpersonorgrouprelatestotheproblem
BegintodefinethewhatortheprobleminyourCity
TherestofthissectionwillhelpyourefinetheproblemstatementFornowgathersomekeypartnerstoclearlyarticulatewhychangeisneededUsethetemplatebelowdevelopedbyDenverPeakAcademytocapturetheproblemstatementYouwillbuildonthislater
ProblemStatementndashWhyChangeisNeeded(1-2Sentences)
ThroughourinterviewswithcitystaffweheardthatwhendiscussingmunicipalchallengesitcanbetemptingtojumptoasolutionbeforetakingthetimetoreallyunderstandtherootcauseoftheproblemThismakessensesincecityleadersandstaffareresponsiblefordeliveringservicesandbeingresponsivetoresidentswhileaddressingnewproblemsthatariseinthecity
TherootsoftheproblemsthatcitiesareworkingtosolvearecomplexandmaybethelegacyofhistoricalfederalpolicyorsweepingsocialtrendsTohelptheproblemseemlessdauntingitcanhelptobreakthemdownintotherootcausetoidentifyamanageablecomponentoftheproblemUsingafishboneexerciseorfivewhystechniquecanalsohelp
Gatherinformationabouttheproblem
10
tofindtherootcauseoftheproblemandavoidsimplyaddressingsymptomsoftheproblemandrepeatingthemistakesofhistoryToolssuchasjourneymapscanhelpyouunderstandtheexperienceofthosewhoareaffectedbytheproblem
TheCentreforPublicImpactwithsupportfromBloombergPhilanthropiesdevelopedanexercisetosupportcitiesinidentifyingtherightsizedproblemAccordingtoCPIarightsizedproblemisonethatislargeenoughtopotentiallyapplytoothercitiesbutsmallenoughtosolvewithinareasonabletimeframe
AtthebeginningofMayorBerkesadministrationintheCityofChattanoogaTNveteransexperiencinghomelessnesswereaskingtheCityforhelpinfindinghousingTalkingtothispopulationrevealedthatveteranshadtroubleidentifyingaffordablehousingwhichmotivatedtheAdministrationtoworkonasolutionTheproblemofveteransexperiencinghomelessnesswastherightsizebecauseitfocusedononesegmentofpeopleexperiencinghomelessness-veterans-andtheirspecificchallengeinidentifyingaffordablehousing
Developacomprehensiveunderstandingofhowyourcitycurrentlyaddressestheissue
IntheChattanoogaexamplementionedaboveCitystaffdiscoveredthatveteranswerehavingtroublenavigatingtheCitysvariousaffordablehousingprogramsandgettingaccesstothehousingandservicesthattheyneededAtthesametimenewlyelectedMayorBerkewasexploringcurrentandfutureopportunitiestoservethispopulationRatherthancontinuetousefederalfundingforemergencysheltersMayorBerkedirectedcitystafftoprioritize
Gatherinformationabouttheproblem
11
HousingFirstwhichvaluesconnectingpeoplewithpermanenthousingovertransitionalhousingandsupportiveserviceafterspeakingwithrepresentativesattheUSDepartmentofHousingandUrbanDevelopment(HUD)
ThisexaminationofwhattheCityiscurrentlyfundingandthesourcesofthatfundingwasaturningpointinChattanoogaUnderstandingthisproblemfromasystemsperspective-includingpoliciesandfundingstreamsatmultiplelevelsofgovernmentsocialnetworksthroughwhichinformationisoftenspreadtechnologythebuiltenvironmentofmarket-rateandpublichousing-helpedthenewMayoranalyzeexistingeffortsandleadstaffandotheractorsinthesamedirection
Begintogatherthedata
TotrulycapturethesuiteofservicesthatyourCityisprovidingaskdepartmentstafftolisttheprogramsrelatedtoyourproblemstatementthatyourcityrunsorfundsWhenlistingtheprogramscapturedetailssuchas
programnamethenameofthedatasetthatisrelevanttoeachprogramadescriptionofthedatasetthesourceofthedatasetifthedataisinternalorexternalwhetherthisdataisalreadybeingcollected
ThisinformationwillformthebasisofyourinventoryofadministrativedataordatasetsthatarekeptbygovernmentsforoperationalpurposesThismayincludecallcenterrecords(311911)taxdatapropertyassessmentsinspectioninformationsocialservicerecordsetcAtthispointlistallrelevantadministrativedatasetsYouwillprioritizewhichdatasetstogathermoreinformationonlaterwhenyouaredevelopingindicators
Asampletemplatethatcanbeusedtocapturethisinformationisbelow
ProblemStatement____________________________________________
Gatherinformationabouttheproblem
12
ProgramName
DatasetTitle
DatasetDescription DataSource
Departmentorexternalpartner
DoestheCityalreadycollectthisdata
Whatisthenameoftheprogram
Human-readablenameofthedataset
Shouldbeunderstandablebynon-technicalusers
Whatistheoriginalsystemapplicationfilethathousesit
Departmentorexternalpartnerthatmaintainsthedata
Isthisdataalreadycollected
TogatherthisinformationconsiderassigningonepointpersontosurveyorinterviewcitystaffSpeakingwithfrontlineworkersstaffwhoworkwithfederalgrantsdepartmentheadsandeveryoneinbetweenwillensureadeepunderstandingoftherelevantprogramsanddatasetsDuringthecourseoftheseconversationsbesuretoaskcitystaffwhichorganizationstheyworkwithexternallyandcapturethatinformationaswelleveniftheyarenotrecipientsofcityfundingInadditiontoconversationswithexpertsincitygovernmentyourcitysbudgetdocumentsareanothergreatsourceofinformation
Collectdataandobservetrends
Nexttrytogainanunderstandingofwhytheproblemstillexistsifithasbeensomethingyouhaveworkedonforawhileorwhyitisemergingnowifitisnew
AnalyzingadministrativedatacandemonstratepasteffectivenessofinitiativesrelatedtoyourproblemstatementUsingthedatasetinventorythatyoubeganintheprevioussteptalktodepartmentstaffaboutwhichdatasetsaremostrelevanttotheproblemandgatherafewyearsworthofhistoricaldataBegintolookfortrendsinthisdatadiscussingpotentialreasonsforthosetrendswithrelevantstaffCanthetrendsbeexplainedbyseasonalitychangesinfundingstaffingfluctuationsorsomeotherreasonBasedonthehistoricaldataanalysiscreateabaselineagainstwhichfuturechangescanbeobserved
OtherdatasetscanhelpprovideadditionalcontextForinstanceifthenumberofpeopleexperiencinghomelessnessinyourcityisincreasingitmaybehelpfultotakealookathousingaffordabilityIfyoudonothavethedatayouneedconsideraproxymeasureAsanexamplehousingisoftenconsideredaffordableifitcosts30orlessofahouseholdsincomeButifhouseholdincomedataisnotavailableatthecitylevelAreaMedianIncome(AMI)canbeahelpfulproxy
Gatherinformationabouttheproblem
13
QualitativedatacanalsohelpanswerquestionsaroundtheeffectivenessofpreviouseffortsTrytoobtainavarietyofperspectives--bothmainstreamandoutliers--fromnetworksbeyondyourowntoprovideyouwiththegreatestvantagepointforevaluatingpotentialimpactUnderstandinghowthoseeffortswerereceivedbydifferentpartsofyourcommunityandiftheremaybeelementsofprevioussystemstopreservewillbecrucialtothesuccessofanypolicychangeThisinformationmaybeobtainedthroughfocusgroupsinterviewsandsurveysFormoreinformationonhowtousethesequalitativeresearchtoolspleaseseetheseresourcesdevelopedbytheAgencyforHealthCareandResearchandUrbanInstitute
FinallytakealookatnationaldatatoseewhereelsethisproblemmightexistThisresearchmaybehelpfulinlaterstepswhenyouarelookingforpromisingpoliciesinothercities
Puttingitalltogether
UsethetemplatebelowdevelopedbyDenversPeakAcademytocaptureyourrefinedrightsizedproblemstatementandcurrentandfuturestatesWorkingthroughthisexercisewithpartnersishelpfultogainbuy-inTheresponseswillbeimportantwhengettingotherpartnersonboard
WhyChangeIsNeeded
Yourright-sizedproblem
CurrentState
Analysisoftrendsindata
FutureState
Ideallywhataretheoutcomesyouhopetoachieve
Checklist
Thekeystepstogatheringinformationarebelow
Atthispointfocusontheproblemratherthanjumpingtosolutions
Listentopeoplewhoareaskingforhelpandtoadvocateswhoareraisingissues
Beasspecificaspossibleinidentifyingtherootcauseandhoninginonthepartoftheproblemthatyourcitycanaddress
Identifytrendsbasedonmultiplesourcesofdataincludinghistoricaladministrativedataproxymeasuresqualitativedataandnationalmeasures
Gatherinformationabouttheproblem
14
Captureyourproblemstatementtheresultsofyouranalysisanddesiredoutcomes
Gatherinformationabouttheproblem
15
StrategicallyengagekeypartnersPolicytheoryresearchoftenfocusesonincreasingpolicymakersaccesstoevidenceasawaytoavoidtakingshortcutslikemakingdecisionsbasedonemotionImplementationsciencesuggeststhatincreasingaccesstoevidenceisnotenoughandthatpolicychangeshouldbeexaminedinthewidercontextofdebatecoalitionformingandpersuasionMeaningfulpolicychangerequiresldquoalong-termstrategytoformalliancesanddevelopknowledgeofpolicymakersandpolicymakingratherthanusingevidenceinpolicymakingonanadhocorone-offbasis
UnderstandingnetworksandsystemsisimportanttodevelopsustainablepolicychangeThissectionismeanttohelpreaderstakethestepstobuildengagementstrategiesforpartnersthatrepresentdifferentsystemsandnetworksEngagementshouldbeanongoingprocessofconnectingpeopletooneanotherandtoanissueEngagementshouldnotbeinnameonly-wehaveallexperiencedengagementactivitiesthatgivepeopleaspecificroletoplaythatleadstoaneatpredictableoutcomeAccordingtotheBuildWithmovementundervaluingandunderinvestinginengagementstrategiescanleadtoadisconnectofcertainindividualsorgroupsandmistrustofgovernmentWhendevelopingengagementactivitiesrememberthathowpeopleengageandwhethertheyengagehastodowithhowyoustructureopportunitiesforengagement
Conveneacoreadvisorygroup
AnadvisorygroupcomprisedofpeoplefrommultiplesectorscanbehelpfulasyouconsiderthelocalsocialenvironmentalandpoliticalcontextinwhichyouintendtointroducepolicychangeRevisityourstakeholdermapanddeterminewhowouldbebesttoserveonthisgroupTomakesureyouareconsideringtheproblemsandsolutionsthroughanequitylensincludepeoplewithlivedexperiencewhohaveexpertiseintheeffectivenessofsystemswhetherthosesystemsfunctionasintendedandhowtheymayimpacthumanbehaviorYoumayalsowishtoincludesomeonewithexperienceworkingwithdataandanalysis-thispersoncanhelpdeterminethetypeofdatathatmaybereadilyavailableandusefultocollectandcanalsohelpplanandshapeyouranalysistofigureoutwhichprogramsareeffectiveBesuretoinvitefront-lineworkersaswell-itscriticalthatwhateversolutionscomeoutofthisprocessaremeaningfultothosewhowillberesponsibleforimplementingthem
BringingpeopletogethercanbechallengingbutthereareafewwaystosetandmanageexpectationsFirstmakesurethatthegroupisawareoftheproblemstatementcurrentstateandfuturestate-thiscanhelpstarteveryoneoffonthesamepageNextspend
Strategicallyengagekeypartners
16
sometimegettingtoknoweachotherrsquosworktolearnabouteveryonesstrengthsThissharedunderstandingcanhelpasyouclarifyrolesandresponsibilitiesofeachpartnerinthegroup--thisisimportanttoavoidfinger-pointinginthefutureandtoensurethatnoneofthetasksslipthroughthecracksInsomecasestherearemanyactorswithdifferentresponsibilitiesasinthecaseofBaltimoreCitys38-memberYouthFundTaskForcewhichischairedbynonprofitandcorporatepartnersandincludesrepresentativesfromthenonprofitandfaithcommunitiesstudentsthebusinesssectorlocaluniversitiesCityCouncilandtheMayorsOffice
OncerolesandresponsibilitiesareclearyoumayconsiderdevelopingacharterguidelinesortermsofreferencetosetexpectationsforthegroupBesuretodistinguishthisfromotherboardsandcommissionsthatyourcitymayhaveNextyoumaywishtoestablisharegularongoingmeetingscheduleBesuretosendagendasandmeetingmaterialsaheadoftimeandclarifytheobjectiveofeachmeeting
InSeattletheCitydevelopeditsHousingAffordabilityandLivabilityAgenda(HALA)torespondtorisinghousingcostsandhomelessnessThisisatrendinotherwestcoastcitiesaswellwhichmaybeexplainedbyaninfluxofnewresidentswhoareattractedbytheemergenceoftechjobstheattractionoftheoutdoorlifestyleortherecentpassageofprogressivepolicieslikelegalizedgaymarriagea$15minimumwageandlegalizationofmarijuanaSeattlebroughttogetheramulti-stakeholderadvisorygroupcomprisedoflocalhousingexpertsrentersandhomeownersdevelopersandgovernmentofficialstoshareinformationabouttheirneedsanddeveloprecommendationsbasedontheseneedsTheinterestsandmotivationsofthisgroupwerevariedandittookabouttenmonthstofindcommongroundTheseriesofrecommendationsthattheadvisorygroupcreatedismeanttoserveresidentswithaspectrumofneeds
Buildaprocessforcommunityparticipation
Ofcoursenoteveryonecanbeincludedinanadvisorygroupandadvisorygroupmembersmaynotnecessarilyrepresentthevariedperspectivesoftheirpeers-forinstancenoteveryhomeownerhasthesamesetofneedsTogatherinputfromawiderswathofthepopulationitcanbehelpfultobuildaprocesstoinvitethepublicatlargetosharetheirideasCitiesdothisinavarietyofwaysfromonlineconversationsandsurveystoin-personmeetingsRememberthatbeingintentionalaboutgatheringtheviewpointsofpeopleismoreimportantthanusingthelatesttechnologyandtools
TheinsightgeneratedbycommunityparticipationexercisescanbeincludedinyourreviewofevidencewhichisdiscussedinasubsequentchapterInWestVirginiaastatethathasbeenhithardbytheopioidepidemicstatehealthofficialsdraftedanopioidresponseplanthatwasinformedbypublicengagementaswellaslocalandregionalsubjectmatter
Strategicallyengagekeypartners
17
expertsThepublicwasinvitedtopresentcommentsonideasfordecreasingoverdoseearlyinterventiontreatmentandrecoveryPublichealthandmedicalfacultyfromWestVirginiaUniversityMarshallUniversityandJohnsHopkinsUniversityprovidedresponsestothecommentsTheresultwasanOpioidResponsePlanthathealthcommissionerRahulGuptapresentedtoGovernorJimJustice
AsamplespectrumtoplanyourowncommunityparticipationprocesscreatedbyLaurenellenMcCannisbelow
SourceLaurenellenMcCann
Community-basedparticipatoryresearchisanotherwaytoworkwiththecommunitytoimproveoutcomesCBPRmeansthecommunityisinvolvedinallaspectsoftheresearchprocesswhichisofteniterativeInCBPRtraditionalresearchersandcommunitymembersshareknowledgeaswellasresourcesandcredit
CBPRInActionInSanFranciscoanenvironmentaljusticenonprofitteamedupwiththeCitysDepartmentofPublicHealthandanoutsideevaluatortounderstandtherootcausesofobesityandfoodinsecurityinlow-incomeneighborhoodsandidentifysolutionsThroughresearchmethodssuchasinterviewingresidentsandstoreownersdiagrammingstoreshelvesGISmappinganeconomicanalysisandreviewofsimilarpoliciesinnearbycitiestheteamobservedthatthestoresthatresidentshasveaccesstowerestockedwithlesshealthyfoodsandthatresidentswantedmorehealthyfoodandlessalcoholandtobaccoThisprojectresultedinlocalandstatelegislationtoencouragestorestostockfreshproduce
Strategicallyengagekeypartners
18
Developrelationshipswithsubjectmatterexperts
OnewaytokeepsubjectmatterexpertsengagedistoinvolvetheminthecommunityparticipationprocessasdescribedaboveIntheWestVirginiaexamplesubjectmatterexpertsrespondedtoandsynthesizedthepubliccommentsOtheropportunitiesaredescribedbelow
AdvisoryAcitydepartmentheadMayororstaffmembermaydevelopaone-on-oneadvisoryrelationshipwithuniversityfacultymemberoracollaborativebetweentheCityandUniversitytoaddressaparticularproblemAgreatplacetostartistolookforrelevantpracticecenterswithinyourlocaluniversityandreachingouttosubjectmatterexpertsInadditiontotheirexpertiseontherelevantpolicytopicheorshemayalsobeabletoidentifyotherexpertsandevenprovidesupportonhiringtherighttalenttoleadcertainprogramsAnadvisoryrelationshipwithauniversitysubjectmatterexpertcanalsohelpwhencitiesareconsideringbringinginnewprogramsTheseexpertsareoftenfamiliarwithprogramsthatarebasedonnationalmodelsandcanprovideinsightintoaprogramsoriginandlineageeffectivenessinotherplacesandthepotentialforimplementationinyourcityAcademicscanalsohelpidentifyindicatorsbasedongovernmentindicatorsorliteratureFinallyrelationshipsbetweenpolicymakersandresearchersprovidethemostbroadlyeffectiveroutetopullingresearchintopolicy
ResearchandevaluationAcademicexpertsmaysupportcitiesinevaluation-indeedevaluatingcityinitiativesisatraditionalmodelofcityuniversityrelationshipsSharingtheinformationyoucollectedintheGatherInformationphasecanhelpresearchersunderstandyourworkandgoalswhichinturnwillstrengthenthesupportthattheycanprovideDiscussingdetailsaboutyourinterventionandevaluationplanatthebeginningoftheprocessmeansthataresearchpartnercanhelpwiththeevaluationdesignanddatacollectionForinstancetheWilsonSheehanLabforEconomicOpportunitieswalkscitiesthroughconsideringthefollowingquestionsbeforepartneringonanevaluation
1 Isthereaclearpolicyprogrammodelwithobjectivescreeningcriteria
2 Isitpossibletoevaluatewhathappensbothwiththepolicyprogramandintheabsenceofthepolicyprogram
3 Arethereoneormoremeasurableoutcomesofinterest
4 Arethereenoughpeopleforrigorousquantitativeanalysis
5 Whatdatawillbenecessaryforthestudyandwhataretheprocessesforaccessingthesedatasource(s)
6 Isthereacompellingreasontoevaluate
Strategicallyengagekeypartners
19
InadditiontoassessingreadinesstoconductanevaluationresearcherscanhelpyouhoneinonthepurposeoftheevaluationTheWorldBankdescribesseveralpotentialusesofevaluationwhicharedefinedhereInbriefevaluationcanhelpdecisionmakers
makeresourceallocationdecisionsrethinkrootcausesofproblemsidentifynewproblemsdecideonabestalternativedeterminewhetherprogramsorpoliciesarebeingimplementedwithfidelitydeterminewhetherprogramsorpoliciesarehavingthedesiredimpact
Researcherscanalsoprovidesupportinsharingtheresultswiththescientificcommunitythroughpapersandconferenceswhichisvaluableincreatingapeernetworkofevidence-basedcitypractitioners
TrainingandtechnicalassistanceUniversitypartnersespeciallythoseatpracticecenterscanbehelpfulindevelopingorupdatingtrainingcurriculumtrainingthetraineranddeliveringtrainingtocitystaff
Strategicallyengagekeypartners
20
ReviewtheevidenceandcollectdetailsOnceyouhaveathoroughunderstandingoftheproblemdataandactorsyouwillneedtoconductyourownresearchonavailableevidenceandlearnaboutthelandscapeofpolicyoptionsYourgoalwillbetodevelopanaccessiblelistofinformationaboutpoliciesthathavehelpedsolvesimilarproblemsinotherplacesThischapterwillguideyouthroughthestepsofpreparingyourreviewersunderstandinglevelsofevidenceandcollectingthedetailsonevidencethatyouwillneedfordiscussionwithdecisionmakersinyourcity
Selectandprepareyourreviewers
AssignapersonorteamtoreviewrelevantinterventionsthathavedemonstratedsuccessinotherjurisdictionsTheirgoalwillbetoproduceareviewofalloftherelevantprogramspoliciesorinitiativesthatyourteamcouldconsiderimplementingTheywillbeconductingaliteraturereviewsupplementedbyinterviewswithexpertsandpeerswhohaveinformationaboutpotentialprograms
SincethepeopleyouassigntothistaskareprovidingthebaselineofpotentialoptionstoimplementitisimportantthatyouchoosepeopleyouareconfidentwillbethoughtfulandthoroughintheirworkItisalsocriticalthattheyareawareoftheimportanceofsourcecredibilityEnsureyourreviewershavefamiliarizedthemselveswithrulesforcriticallyanalyzinginformationsourcesbeforetheygetstartedIftheyareincludingnewsarticlesaspartoftheirsearchtheyshouldalsobeawareofmethodsforevaluatingnewscredibility
AskyourreviewerstodevelopaplanandtimelinetocompletethereviewThisstartswithsolidifyingagoverningresearchquestionOnewaytoknowyouarecreatingadetailedresearchquestionistousetheacronymPICOremindingyoutoincludereferencetothepopulationintervention(ifyouhaveoneinmind)comparisonandoutcomeelementsinyourresearchquestionForexampleifyouknowthatyouhaveapopulationofteenagerswhoarehomelessyouhavelearnedfromyourstakeholdersthatmanyoftheseteenagersidentifyasLGBTandyouhaveheardsomethingabouttheapproachofcreatingdrop-incentersfordeliveringsupportyoucouldask
ForLGBTyouthfacinghomelessness(population)howeffectivearedrop-incenters(intervention)relativetostandardsocialservices(comparison)atsettlingtheminpermanenthousing(outcome)
Ifyoudonothaveaninterventioninmindyoucanleaveouttheinterventionandcomparisonandsimplyask
Reviewtheevidenceandcollectdetails
21
HowcanwehelpLGBTyouthfacinghomelessness(population)obtainpermanenthousing(outcome)
OnceyourreviewershavetheirresearchquestionsusethequestionstocreatealistofkeywordstheycanuseintheirliteraturesearchThiswillhelpkeeptheircollectionofarticlesreasonablyfocusedHavethemdescribewhattheirsearchprocesswillbeincludinglistingthedatabasestheywillreviewandtheirplanforstoringtheinformationtheywillcollect(Herearesomeadditionalsuggestionsforstructuringareviewplan)
WhatyourreviewerswillbedoingisalightweightversionofarapidreviewTotheextentthattheywouldliketodevelopamethodologyforspecificsearchprotocolsincludingcriticalappraisalprocessestheycancertainlygoaheadanddoso
Usedifferentlevelsofevidenceforthemostcompletepicture
AsdiscussedintheWorkingwitharangeofevidencechaptertheNestaStandardsofEvidenceframework--reproducedbelow--ishelpfultounderstandthelevelofconfidencethataprogramorinterventionishavingapositiveimpactThisapproachletsyouputinterventionsthatyouhaveheardaboutlocallybutwhichhavenotbeenstudiedtogetherwiththosewithademonstratedcausaleffectThisishelpfulsinceanumberofpoliciesprogramsandpracticesthatyouwillwanttoconsidermaynotyethavemuchofaresearchrecord
Reviewtheevidenceandcollectdetails
22
NearlyanypieceofinformationaboutaninterventioncanbeclassifiedwithinthestandardsofevidenceForexampleifsomeonehasacrediblehypothesisabouthowaninterventioncouldproducebenefitthistypeofevidencewouldbeclassifiedasLevel1ThistypeofevidencecouldincludeinterventionsinothercitiesthatyoureadaboutinthenewsorhearaboutataconferenceForinstancewhenUSDAallowedschoolsandschooldistrictstoprovidefreebreakfastandlunchtoallstudentsthroughtheCommunityEligibilityProvision(CEP)manydistrictsimplementedthisoptionevenbeforetherewasevidenceofeffectivenessThiswouldfitintoLevel1sinceitbuildsonthelogicthatprovidingaccesstomealswilldecreasefoodinsecurity
YoumayalsofindpiecesofinformationthatqualifyasLevel2evidenceForinstanceasmoreschooldistrictsimplementedCEPprincipalstestifiedaboutpositivechangestheyobservedinstudentssuchasamorepositivedemeanorthatcanleadtobetteracademicperformanceDataalsoshowedthatthestudentsdescribedwerereceivingmoremealsthanbeforetheprogramsimplementationHoweveritwastoosoontobuildevidencetoshowacausalrelationshipbetweentheincreaseinfreemealsandincreasedacademicperformance
ldquoLower-levelrdquoevidencecanconsistofnuancedbutvaluableelementsthatrelatetoaninterventionrsquospotentialfitwithacommunityQualitativeinformationsuchastestimonyfrompeoplewhohaveexperiencedaninterventionandthosewhohavefacilitatedorimplementeditwillalwaysbeusefultoconsiderandprovidesdetailedlocalcontext
Experimentalstudies--describedstartingatLevel3withcontrolledexperimentsandgoingthroughLevel5withfullyfleshedoutprotocolsandprocessesforreplication--stilloccupythehigherlevelsofthisdescriptionofevidenceThisprioritizationshowsthevalueofrigorousstudyforevaluatingtheimpactofaparticularpolicyorprogramratherthanotherfactorsthatcouldbeinfluencingoutcomesofinterestRememberthatmultipletypesofevidencecancomplementrigorousimpactevaluationsandgivecityleadersadditionalinsightsandbroaderperspective
Accessinformation
Encourageyourreviewerstobegintheirsearchforpromisingpracticesingovernmentandnon-profitclearinghousesandinacademicpolicyjournalsWorkpublishedinthesesourceshasalreadybeensubjectedtoahighlevelofprofessionalscrutinyandgenerallycontainsagreatdealofdetailwhichmeansthatyourreviewersaremorelikelytobeabletopullouttheinformationyouneedtomakeagooddecision
Clearinghouses
Reviewtheevidenceandcollectdetails
23
Herearesomeoftheclearinghouses(collectionsofprogramevaluations)forevidence-basedpoliciesandprograms
ResultsFirstClearinghouseDatabaseThePewCharitableTrustsCollectstogethereightotherclearinghousestoprovideinformationoninterventionsinthefollowingcategoriesadultcriminaljusticeeducationchildwelfarehealthjuvenilejusticementalhealthsubstanceabuseandyouthdevelopmentIndividualclearinghousesincludedinResultsFirstare
BlueprintsforHealthyYouthDevelopment-foryouthdevelopmentandviolenceprevention
CaliforniaEvidence-BasedClearinghouse-forchildwelfareinterventions
TheLauraandJohnArnoldFoundationrsquosSocialProgramsThatWork-forsocialservicesprograms
TheUSDepartmentofEducationrsquosWhatWorksClearinghouse-foreducationinterventions
TheUSDepartmentofHealthandHumanServicesrsquoResearch-TestedInterventionPrograms(RTIPs)-forcancercontrolinterventions
TheUSDepartmentofHealthandHumanServicesrsquoTeenPregnancyPreventionEvidenceReview(TPPEvidenceReview)-forsexualriskbehaviorandsexualhealthoutcomes
TheUSDepartmentofJusticersquosCrimeSolutionsgov-forcriminaljusticejuvenilejusticeandcrimevictimservices
TheUSSubstanceAbuseandMentalHealthServicesAdministrationrsquosNationalRegistryofEvidence-basedProgramsandPractices(NREPP)-substanceuseandmentalhealthinterventions
TheUniversityofWisconsinPopulationHealthInstituteandRobertWoodJohnsonFoundationrsquosCountyHealthRankingsandRoadmapsWhatWorksforHealth(WhatWorksforHealth)-healthbehaviorsclinicalcaresocialandeconomicfactorsandthephysicalenvironment
UKWhatWorksCentresTheBritishgovernmenthascharteredanetworkofsubject-specificorganizationswhichcollectsynthesizeandreportoutevidenceontheeffectivenessofpolicyprogramsandpractices
NationalInstituteforHealthandCareExcellence(NICE)-healthandsocialcare
Reviewtheevidenceandcollectdetails
24
SuttonTrustEducationalEndowmentFoundation-educationalachievement
CollegeofPolicingWhatWorksCentreforCrimeReduction-crimereduction
EarlyInterventionFoundation-earlyintervention
WhatWorksCentreforLocalEconomicGrowth-localeconomicgrowth
CentreforAgeingBetter-qualityoflifeforolderpeople
WhatWorksCentreforWellbeing-wellbeing
WilsonSheehanLabforEconomicOpportunities(LEO)UniversityofNotreDameProvideslinkstotheirevaluationsofinterventionsincriminaljusticeself-sufficiencyeducationhealthandhousingandhomelessness
ClearinghouseforLaborEvaluationandResearchUSDepartmentofLaborProvidesresearchonworkforceandlabor-relatedpoliciesandprograms
J-PALEvaluationsAbdulLatifJameelPovertyActionLabStudiesonpoverty-reducinginterventionsfromaroundtheworld
CampbellSystematicReviewsCampbellCollaborationSystematicreviewofpolicyinterventionsbyscholarlycoordinatinggroupsinsevenissueareascrimeandjusticeeducationinternationaldevelopmentmethodssocialwelfareknowledgetranslationandimplementationanddisability
Benefit-CostResultsWashingtonStateInstituteforPublicPolicyAWashingtonState-charteredprogramtoestablishthelikelihoodofrelativecostsandbenefitsforprogramsacrossthefollowingareasjuvenilejusticeadultcriminaljusticechildwelfarepre-Kto12educationchildrensmentalhealthhealthcaresubstanceusedisordersadultmentalhealthpublichealthandPreventionworkforcedevelopmentandhighereducation
EffectiveInterventionsCentersforDiseaseControlandPreventionRepositoryforevidence-informedHIVpreventioncareinterventions(withstepsonhowtoimplement)
Journalsandindexes
OnceyouhavereviewedrelevantinformationaccessiblethroughclearinghousesitisworthtakingalookatindividualstudiespublishedasscholarlyarticlesYourreviewerswilldothisbysearchingarticleindexesandacademicjournalsUnfortunatelyanumberofusefulacademicjournalsandindexingplatformsmaynotbegenerallyaccessibletoyourreviewersifyoudonothaveasubscriptiontothemToaddressthisconsiderpartneringwithalocal
Reviewtheevidenceandcollectdetails
25
collegeoruniversitytoprovideyouwithalistofarticlesfromthefollowingsources(Thisisalsoanexcellentopportunitytoseekresearchsupportfromanyacademicstakeholdersyouidentifiedpreviously)
Indexes
GoogleScholarincludesmostpeer-reviewedonlineacademicjournalsandbooksconferencepapersthesesanddissertationspreprintsabstractstechnicalreportsandotherscholarlyliteratureincludingcourtopinionsandpatentsItallowsyoutospecifyapublicationrangeinyoursearchandprovideslinkstolikelysourcesforalllistedarticles
PolicyFileIndexisauniqueresourceforUSpublicpolicyresearchinthatitgrantsusersaccesstotimelyupdatedinformationfromover350publicpolicythinktanksnongovernmentalorganizationsresearchinstitutesuniversitycentersadvocacygroupsandotherentities
PolicyArchiveworkstocollectanddisseminatesummariesandfulltextsvideosreportsbriefsandmultimediamaterialofthinktankuniversitygovernmentandfoundation-fundedpolicyresearch
Anothersourceforhigh-qualityresearchisthereportsoftheCongressionalResearchServiceYoucanfindtheirreportsaggregatedhere
Specificjournals
Oftenthemostusefulacademicjournalsforunearthingpolicyorprogramreviewswillbeineachspecificsubjectarea--thatisineducationjournalsforeducationpoliciesYoucanuseageneralindexlikeGoogleScholartobegintolearnwhichjournalsaremostoftenusefulforyoursearchHoweveryoucanalsosearchthefollowingtwogeneralistpolicypublicationsforissuesandarticlesrelatedtoyourtopic
PublicAdministrationReviewJournalofPolicyAnalysisandManagement
Newssourcessubjectmatterexpertsandinterviews
AlthoughitisvaluabletoknowwhathasbeenexploredthroughascholarlyresearchframeworkitisalsousefultolearnwhatkindofinterventionsarebeingtriedinthefieldrightnowandhowpeoplefeelaboutthesetrialsSearchsubscriptionnewsaggregatorslikeLexisNexisorfreenewsaggregatorslikeGoogleNewsorYahooNewsforyourissueareaplusthewordpolicyorprogramtoseewhatkindsofinterventionsarebeingcoveredinthenews
Reviewtheevidenceandcollectdetails
26
DonotforgettocontacttheadvisorygroupmemberswhowerechosenfortheirsubjectmatterexpertiseInadditiontoprovidinggoodmaterialthroughinterviewsthemselvestheymayhaveideasaboutothersubjectmatterexpertsyoushouldcontacttoaskaboutpotentialinterventions
Peercitiesandnetworks
InadditiontoreadingabouttheworkofothercitiesitisidealifyouareabletospeakwithpeoplewhoareinvolvedwithcurrentimplementationsofinterestinginterventionsInyourcityyoulikelyhaverelationshipswithyourneighborsbutitcanbehelpfultolookoutsideofyourregionforsolutionsaswellTofindcounterpartsinothercitieswhoareworkingonaprogramyouwouldliketolearnmoreaboutyoucancontactstaffinintermediaryorganizations(likeGovExoryourprofessionalassociations)whomightbeinapositiontoputyouintouchimmediatelyorwhomightseeanothergoodrouteforsettingupaconversationHereisalistofnetworksthatmaybehelpful
InternationalCityCountyManagementAssociation(ICMA)NationalAssociationofCounties(NACO)NationalLeagueofCitiesUSConferenceofMayorsWhatWorksCities
Collectthedetails
Onceyourreviewershavefoundevidenceonpossiblepoliciesorprogramstoimplementtheyshouldcapturekeydetailsthatwillhelpdecisionmakersunderstandthepotentialeffectivenessandapplicabilityofeachpotentialintervention
Foreachsourcethathasprovidedinformationyouwanttoconveyusethetemplatebelowtocaptureasmanyofthefollowingdetailsaspossible(Youdonotneedtocompletethisforsourcesthatprovideduplicativeinformationorareotherwisenotthatuseful)
Programorpolicyname (egldquoHousingFirstrdquoorldquoblightedpropertiesregistryrdquo)
Sourcetype (egjournalarticleinterviewsubjectmatterexpertstudyassociationcommunityparticipationprocess)
Sourcedetails (egarticletitleandpublicationnameandtitleofinterviewee)
Evaluationmethodology
(egrandomizedevaluationcomparativeevaluationoftwodifferentgroupsbeforeafteranalysis)
Evidencelevel (egLevel12345)
Programorpolicy
Reviewtheevidenceandcollectdetails
27
description
Programorpolicygoal (1sentencecapturingtheprincipalaimoftheintervention)
Existingimplementations
(WherehastheinterventionbeentriedListasmanysitesasyouareawareof)
Characteristicsofimplementationsites
(ForeachimplementationsiteofferbasicgeographicIDmdashcityandstateThenyouwillwanttocollectthedetailswhicharerelevanttoyourdecisionmakersrsquoconsiderationsaboutwhatmakesalocationldquosimilartordquoorldquodifferentfromrdquoyoursituationYouwillatleastwanttoincludeforeachimplementationsitePopulationnumberPopulationdensity(ruralurbansuburban)Socio-economiccharacteristicsBeforestartingaskyourdecisionmakersaboutanyparticularcharacteristicsofprogramimplementationsitestheywouldwanttoknowbeforeseekingtoreplicate)
Characteristicsofimplementationpartners
(Anyavailabledetailsaboutorganizationsinstitutionsorpeoplewhowereinvolvedwithimplementingtheintervention-egindependentchildcarecenterpublicelementaryschoolcitypublichealthnurses)
Characteristicsoftargetpopulationeligibleprogramparticipants
(Anyavailabledetailsaboutpeoplewhoarethebeneficiariesoftheprogramorotherwiseincludedoraffectedbytheintervention-egnumberofprogramparticipantscategoriesofpropertyownersaffected)
Policyorprogramduration
(HowlonghastheinterventionbeeninplaceineachimplementationsiteHasitendedandifsowhy)
Whatbenefitscamefromeachimplementation
(Anyobservedpositiveeffectsoftheinterventionincludingeffectsize)
Statisticalestimatesofinterventioneffectiveness
(Anyavailablestatisticalestimatesoftheinterventionrsquoseffects)
Whatdrawbackswereobserved
(Anyavailabledetailsaboutunintendedorunavoidablenegativeeffectsoftheintervention)
Programorpolicycosts
(Anyavailabledetailsaboutprogramcostsperimplementation)
Reviewtheevidenceandcollectdetails
28
Reviewtheevidenceandcollectdetails
29
DecideonapolicytoimplementbasedonlocalfactorsTogetthecriticalbuy-infromleadershiptoimplementanewpolicyorprogramtheprojectleadshouldscheduleameetingwithdecisionmakerstopresenttheevidencesurroundingthepolicyoptionscapturedinthetemplateDependingonyourinsightintoyourcityleadershipswillingnesstomakechangeyoumaywishtosharetheresearchyouhaveconductedontherootcausesoftheproblemandsomeofyouranalysisontheimpactofcurrenteffortstodate
Considertheapplicabilityofanapproach
AsdiscussedinthepreviouschapterpolicymakersoftenlookforevidenceofeffectiveprogramsinothercitiesbutoftenthequestionarisesofhowgeneralizableaprogramisfromonecitytoanotherTheAbdulLatifJameelPovertyActionLab(J-PAL)suggeststhatinreviewingevidencefromanotherlocationitishelpfultoidentifythespecificmechanismthatwasused(deliveryofprogramorserviceinformationsharingetc)identifythelocalconditionsthatwereinplaceobservethebehaviorchangethatresultedfromthemechanismandfinallytounderstandlocalcontextandconditions(legalorcostconstraintsculturalnormsetc)Thissetofpresentationelementscanbedrawnfromthelistofdetailscollectedintheresearchtemplate
EducationpolicymakersinChicagoILusedfindingsfromprogramsinKenyaandIndiainwhichchildrenwhohadfallenbehindtheirpeersweregiveneducationalsupporttargetedtotheirleveloflearningThemechanismusedwastargetedteachingandthereweresimilarconditionspresentinallthreecountriesAlthoughsomestudentshadfallenbehindinthecurriculumteacherswereincentivizedtoteachgrade-levelmaterialsConsiderationsofhowthelocalcontextinChicagomightaffectimplementationofatargetedteachingprogramincludedunderstandinghowtheneighborhoodsthatthestudentswerefrommayhaveaffectedtheireducationaloutcomesandthehighcostofteachinginsmallgroupsInChicagoMatchEducationprovidedtwo-on-onetutoringwhichmatchedstudentswithwell-educatedindividualswhohavenothadformalteachertrainingforalower-costmodelTheresultswerepositiveStudentswhoweretutoredperformedbetterinmathandweremorelikelytopasshighschoolclasses
ThegeneralizabilityframeworkproposedbyJ-PALisbelowUsethisasatemplateasyouconsiderpotentialpolicysolutions
Decideonapolicytoimplementbasedonlocalfactors
30
SourceTheGeneralizabilityPuzzle
OnceyouhavenarrowedthepolicyoptionsdiscussthequestionsbelowforeachoptionsyouareconsideringThisstepiscriticalasunderstandinglocalcontextissoimportantinsuccessfulpolicyimplementation
CommunityContextQuestionnaire
Localcontext
DoesthisoptionmakesenseinourcommunityIsthisaonesizefitsallapproachIsanyparticulargroupabsentfromthisapproachIfsoisthatok
Youranswers
Effectiveness
WhatwouldweexpecttoobserveifthepilotisaseffectivehereasinCityX
Youranswers
Timing
DoesthetimingmakesenseIsthistherighttimetoimplementthispolicychange
Decideonapolicytoimplementbasedonlocalfactors
31
WherearethepotentialroadblocksthatcouldaffectthescheduleIsthereanywaytomitigatethisHowlongwouldittaketoseeanimpact
Youranswers
Implementation
DowehavethestaffcapacitytoimplementthisArethereadditionalpartnersthatyoucanbringtothetabletosupportimplementationDoesthismakesensegivenourformofgovernment
Youranswers
Authority
DoesourcityhavetheauthoritytoenactthisIfsowhatistheappropriatemechanismAretherelegalbarriersArethereanyexistingpreemptionissuesfromthestateorfederalgovernmentCanthosebeaddressedthroughlegislationadvocacyIstherepoliticalsupportoroppositiontothis
Youranswers
Cost
HowmuchwillthiscostannuallyHowwillwepayforitCanthepolicybeimplementedwithexistingfunding
IfsoisthisfundingreliableHowwouldthispolicyoptionaffectourcitysbudgetintheshortandlongtermWhatotherfundingoptionscanweconsiderIsthereanalternativethatwouldcostlessandgiveussimilarresultsWillanyconstituencybeadverselyaffectediffundingforprogramsthatimpactthemisshiftedtoanotherprogrampolicyorinitiative
Youranswers
Decideonapolicytoimplementbasedonlocalfactors
32
EstablishaframeworktomonitorprogressNowthatyouhavereviewedtheevidenceconsideredlocalcontextandchosenyourpolicyitistimetodeterminehowyourcitywillmonitorprogressofthisnewapproachCitystaffandleadersneedtoestablishtheinfrastructureforongoingcontinuousdecisionmakingbasedonwhetherthepolicychangeisleadingtothedesiredoutcomesforresidentsandwhetherthechangeismeaningfulandrationaltofrontlineworkersSettingarbitrarygoalsatthelocallevelthatconflictwithgoalsatotherlevelsisconfusingtotheindividualsresponsibleforachievingthegoalsGoalsthataresettoohightoachievecanbedemoralizingtostaff
ThissectionincludesstepstosetupavenueinwhichCitystaffandleadershiproutinelyvisitkeyindicatorstodeterminewhatisorisnotworkingandtomakedecisionsinreal-timeFormorereadingonperformancemanagementpleasevisittheGovExPerformanceManagementGettingStartedGuide
InperformancemanagementtermssuchasindicatormeasureandmetricmaybeusedinterchangeablyInthisguidewewilluseindicatorInyourworkthemostimportantthingistobeconsistentinwhichevertermyouuseThisvideomaybehelpfulinunderstandingthisconceptandcommunicatingittoothers
Establishthefundamentalsandgatheryourteam
FirstdeterminewhetherthereisanestablishedvenueorregularmeetingscheduleduringwhichyoucandiscussprogressIfyourcityhasanexistingperformancemanagementstructureonarelatedtopicaddingadiscussionoftheprogressofpolicychangetoyouragendacanbeagreatideaIfthiswillbeastandalonemeetingdecidehowfrequentlytheprojectteamanddecisionmakerswillmeetandputthosemeetingsonthecalendarfortheentirelengthoftheproject
DeterminewhowillstaffandattendroutinemeetingsonCityprogressThesemeetingsmayincludetheMayorandorCityManagerdepartmentheadstaffwhoareimplementingthepolicychangeandanalysts
Identifyindicators
ItiscriticalthatindicatorsbeanimportantcityandcommunitypriorityaswellassomethingtheCityplaysamajorroleinaddressingTheremaybeinstancesinwhichtheCitywillneedtoreachouttoorcollaboratewithexternalpartnerstogetamorecompletepictureofprogressontheoverallpolicychange
Establishaframeworktomonitorprogress
33
Anoutcomeindicatordescribesthedesiredeffectofthetargetedpopulation
ExampleBellevuesfamilyfriendlyneighborhoodsindicatormeasuresthepercentofresidentswhobelievetheyliveinaneighborhoodthatsupportsfamiliesparticularlythosewithchildren
WithyourkeypartnersrevisityourproblemstatementcurrentstateandfuturestateexerciseandbegintoidentifytheoutcomeindicatorsthatwillillustratesuccessFornowchooseasingletop-levelkeyperformanceindicator(KPI)ItiscriticalthattheKPI-thepieceyouwillmeasure-beapriorityforthecommunityandsomethingtheCityplaysamajorroleinaddressing
Outputindicatorsmeasuretheprocessofprogramorservicedelivery
ExampleSanDiegoMayorKevinFaulconersgoaltorepair1000milesofstreetby2020
ReviewingtheadministrativedatasetinventorybegintoparedowntherelevantprogramsanddatasetsthatsupportyourKPIWhatarethemeasuresofeffectivenessforeachprogramTheseareyouroutputindicators
Thediagrambelowillustratestherelationshipbetweenoutputandoutcomeindicators
ThinkaboutimplementationindicatorsasmeasuringstepsintheprogramsrolloutthatarenecessarytoobtaindesiredchangesImplementationoutcomescanfocusontheinterventionstaffdeliveringtheinterventionanorganizationorsystemandcanaddressfeasibilityadaptabilitysustainabilityandcostTwokeyquestionstoconsiderwhendevelopingimplementationindicatorsare
1 Istheinterventionacceptableandappropriatetothecommunityand
2 IsitfeasibletoimplementinterventionXatthistimewiththeseresources
Establishaframeworktomonitorprogress
34
Amoreextensivelistwithdefinitionsofpotentialimplementationoutcomesislinkedhere
UsethetablebelowtochoosethedesiredimplementationoutcomesforyourpilotandtoplanyourmeasurementtechniqueThisshouldincludelevelofanalysis(individualproviderorconsumerorganizationorsetting)yourmeasurementtechnique(surveysobservationinterviewfocusgroupsadministrativedataanalysis)andthephasewhenyouwillcollectthisinformation(earlymidlate)Theremaybemorethanonelevelofanalysismeasurementtechniqueandphaseofdatacollection
Implementationoutcome
Definitionofoutcome
Levelofanalysisandmeasurement
Yourmeasurementtechnique
Phaseofpilotwhenyouwillcollectdata
Usethespacebelowtocaptureyourindicatorwishlist
Indicatorwishlist
Outcomes_____________________________________________________________________________
Outputs________________________________________________________________________________
Implementationoutcomes______________________________________________________________
Setapublichigh-levelgoal
Thehigh-levelgoaliscriticalinestablishingdirectiontogalvanizethepeoplewhowillbedoingtheworkWhendiscussingthegoalusetheMayorsbullypulpitLookforkeymilestonesinCitygovernmenttosharethegoalstatementsuchasthebeginningofaMayoraladministrationaStateoftheCityspeechorotherpublicremarks
GoalsshouldbeldquoSMARTrdquo
Specificaspossible(innamingthepopulationwhowillbebetterofftheactionthecitywilltake)
Measurable
Achievable
Establishaframeworktomonitorprogress
35
Results-focused
Time-bound
ThegoalstatementshouldincludeyouroutcomeindicatorlikeChattanoogasgoalldquoWewillendchronicveteranhomelessnessbyDecember2016rdquoAdiagramtohepyoubuildaSMARTgoalaroundyouroutcomeindicatorisbelow
Component Description Yourturn
Specific WhowillbebetteroffWhatactionthecitywilltake
Measurable Howwillyoumeasuresuccess
Achievable Isthegoalattainable
Results-focused Whatistheexpectedresult
Time-bound When
Setrealisticambitioustargetsandtimelinesforeachindicator
ToestablisharealisticandambitioustargetforyourgoalstatementusetheadministrativedatacollectedforyourchosenKPItoreviewactualCityperformanceinthemostrecentyearcomparedwiththethreeorfourmostrecentyearsworthofdataBegintocalculatesummarystatistics(egtotalminimumaveragemedianmaximum)toidentifytrendsCreatecumulativeandmonthlychartstoidentifytrendsandanomaliesBasedonyouranalysisofhistoricaldataignoretheanomaliesandassumetheproductivitytrendswillfollowthesameoverallpatternfrommonth-to-monthFinallyapplyanincreasedproductivityfactortotheperformanceeachmonthtoforecastfutureprogressBasedonthisanalysischooseatargetthatinspiresimprovementwhileaccountingforreality
Manycitiesthataremorematureintheirperformancemanagementpracticesrecognizetheimportanceoftakingacommunity-levelapproachtoimprovingoutcomesforresidentsThesecitiesarecreatinggoaltaxonomiesthatincludemeasuresrelatedtopublicsafetyhousingandevenrecreationCitiesdriveprogressonthesegoalsbyimplementingstrategiesthatengagemultipledepartmentsandcollaboratewithserviceprovidersinthecommunity
EventuallyyouwilldevelopgoalstatementsfortheoutputsinordertocapturetheefforttoachievetheoutcomecontainedinyourgoalstatementYoursetofindicatorsshouldbecomposedofshort-andinterim-termsupportivemeasureswithaccuratetargetsandrelevanttimelinesAnexamplegoaltaxonomyfromtheCityofLasVegassResultsVegasprogramisbelowUsethistaxonomyasamodelforyourownsuiteofindicatorsandgoalstatements
Establishaframeworktomonitorprogress
36
Yourturn
Thetemplatebelowismeanttohelpyoucraftagoalstatementforeachsupportingmeasure
Goalstatement
Departments
Outcome
Activities
KPI
Target
Timeframe
Conductanddiscussrelevantanalysis
ThrougharoutineseriesofmeetingscontinuallymonitorprogressofKPIssupportingindicatorsandanyotherdatacollectedsuchasqualitativedatatoidentifytrendsthatwillindicatewhetherthechosenpolicyishavingthedesiredimpactThesedatapointsshoulddrivethediscussionduringthesemeetingspointingtoopportunitiesforfurtherinquiryinto
Establishaframeworktomonitorprogress
37
whatisdrivingtheobservedtrendsAnalystsshouldbeencouragedandtrainedtoaskquestionsofthedepartmentsproducingthedataandconductsitevisitswhenpossibletodeepentheirunderstandingofthedataPresentationoftheanalysisshouldfocusononeideaatatimetokeeptheconversationfocusedonkeypointsandspecificopportunitiesforimprovementYoumayconsiderdevelopingapublicfacingdashboardtosharethisprogressmorebroadly
AgeneralcomparisonofthestatusquodatawithwhatwasobservedduringthepilotcanprovidedecisionmakerswithaquickunderstandingofthedifferencesbetweentheoldandnewapproachesalthoughthiswillnotshowcausalityOftenthesecomparisonswillincludeoperationalmetricssuchasdaysordollarssavedbutrememberthattheimpactonpeoplewillbemostimportantIfyouareworkingwithanevaluationpartnertheywillsupportthisstep
Usethespacebelowtosummarizetheresultsofyouranalysis
MeasureTargetvalue(seesectiononsettingtargetsabove)
Pilotvalue(whatdidyouobserveduringthepilot)
Historicalaverage(ifavailable)
Results(WhoisbetteroffIsanyonenegativelyimpacted1-2sentencesreassessingyourgoalstatement)
DocumentyourprocessandprogressincludingfeedbackfromtheparticipantsimplementationpartnersandtheprojectteamDidanythingunexpectedoccurduringthepilotArethereinternalandexternalfactorsthatmayaffectyouroutcomesThisiswheretheimplementationoutcomescanbecanhelpfulIftheinterventiondidnotgoasplannedconsiderwhetheritisbecausetheinterventionwasnotacceptableorfeasible
UsethetemplatebelowcreatedbyDenversPeakAcademytocapturethisinformationIfthe306090daytimelinedoesnotmakesenseforyourcityfeelfreetocustomizeit
CurrentState
FutureState
30Day
60Day
90day
Whatwentwell(+)
Whatdidnotgowellshouldbechanged(-)
IfaquestionordiscussionpointarisesthatcannotbeansweredinthemeetinganassignedstaffmembershouldcapturethequestionduringthemeetingAfterwardthefollowupitemsshouldbesenttorelevantpartnersandthenextmeetingshouldbeginwithadiscussionof
Establishaframeworktomonitorprogress
38
thoseitems
Takeaction
DiscussionsaboutwhetheryourindicatorsaremovingintherightdirectionandwhatmightbedrivingtrendsmayleadtoadiscoverythattherearenotenoughresourcesallocatedtothispolicyorissueTheremaybelimitedfundingforstaffprogrammingortechnologythatisdrivingthedatainthewrongdirectionIfCityleadershipisintheroomthisisagreattimetomakethecaseforadditionalfundingduringthebudgetseasonBycollectinganalyzinganddiscussingdataonthispolicyroutinelyyouarebuildingevidencetomakethecasefortherightleveloffundingtoreachtheestablishedtargets
Establishaframeworktomonitorprogress
39
Assessdataaccessibilityqualityandsecurity
Determineaccessibilityandqualityofdata
OnceyouhavecompiledyourwishlistofindicatorsitistimetodeterminetheaccessibilityandqualityofthedataTopreventcomplicationyoumaywishtoprioritizerelevantindicatorsforwhichyouarealreadycollectingdataAvoidthetemptationofadoptingatechnologysolutiontocollectnewdataatthispointsuchasaphotosurveytoolorotherappFornowyouaretryingtodeterminewhichdatasetsarerelevantaccessibleandhighqualityBesuretoworkwithyourkeyinternalpartnerswhileyouareidentifyingyourmeasuresDepartmentstaffwhoworkonprogramsarefamiliarwiththeindicatorsthatwillaccuratelysupporttheoutcomesanddatabaseadministratorswillbevaluablewhendiscussingaccessibilityandqualityofthedata
Abrieflistofconsiderationsfortheaccessibilityofyourdataisbelow
ArewecurrentlycollectingthisdatasetorwillwebecollectingitforthefirsttimeHowfarbackdoestherecordgo(timehorizon)WouldweneednewtechnologytocollectoranalyzethedataCanthedatabecollectedmanuallyorcanitbeautomatedHowfrequentlyisthedataproduced(continuouslyhourlydailyweeklymonthlyquarterlyyearlyetc)IsthedataavailablefortheentireCitypopulationorasubsetofthepopulationHowgranularisthedata(individuallevelCensusblockneighborhoodcitywideetc)DoesthedatacontainsensitiveinformationIsthereaproxymeasurethatwecanusefornowIsthedatamachinereadable
AsnotedbytheWilsonSheehanLabforEconomicOpportunities(LEO)attheUniversityofNotreDameleveragingadministrativedatacansignificantlyreducethecostsandtimelinesofrigorousevaluationCitiescanestablishaninternaldatasharingagreementacrossdepartmentsincludinglanguagethatenablesadministrativedatatobeusedforresearchpurposesCitiescanalsoidentifyexternaladministrativedata-holderssuchasacountyschooldistrictpolicedepartmenthospitaloranotherstatethatmayhaveoutcomedataimportantformeasuringprogramimpactCitiescanleverageexistingrelationshipsandorworkwithresearchinstitutionstodevelopstreamlineddatasharingprocessesandagreementswiththeseexternalentities
Assessdataaccessibilityqualityandsecurity
40
Evaluatethequalityofyourdata
QualityisasimportantasaccessibilityToevaluatedataqualityyoumaywishtocreatearubricliketheonebelow
Value ValueDescription
3 ExcellentDataQualityThedataiscompleteaccurateandupdatedregularly
2 AdequateDataQualityThedataissomewhatcompleteaccurateandupdatedfairlyregularly
1 PoorDataQualityThedataisnotcompleteaccurateorupdatedregularly
ThisisagreattimetobuildonthedatainventorythatyoustartedinthegatherinformationpageSuggestedadditionalfieldstoaddtoyourdatainventorytemplatearebelow
Program DatasetTitle
DatasetDescription DataSource
Departmentorexternalpartner
DoestheCityalreadycollectthisdata
Whatisthenameoftheprogram
Human-readablenameofthedataset
Shouldbeunderstandablebynon-technicalusers
Whatistheoriginalsystemapplicationfilethathousesit
Departmentorexternalpartnerthatmaintainsthedata
Isthisdataalreadycollected
Updatefrequency-ThefileformatforthedatasetTypicallycsvxlsxshapefiletxtetc
Format-Howoftenthedatasetisupdated
Timerange-Theperiodoftimecoveredinthisparticulardataset
Datause-Howisthisdatasetcommonlyused
Datausers-Whotypicallyusesthisdataset
Dataownerstitle-Titleofthedataowner
Dataownersposition-DataOwnersjobtitle
Dataownersemailaddress-DataOwnersemailaddressWhomanagesthedataandorisresponsibleforgrantingpermissiontoaccessthedataWhounderstandswhatthedatasetincludesandcananswerquestionsaboutit
Dataqualityconcerns-Describetheconcernsyoumayhaveaboutthequalityofthedataset
Assessdataaccessibilityqualityandsecurity
41
Division-DivisionwherethedataismaintainedIfmorethanonedivisionisresponsiblelisttheprimaryownerofthedataset
Sensitivedatacomments-DescribetheissuesthatwouldpreventthecityfromsharingthedatapubliclyCommonsensitivityconcernsincludeprivacyviolationssecurityissuesorhighcostorstaffdemandstopostthedata
Publishingstatus-Thistheplacetoputanycommentsregardingthecurrentstatusofthedataset
Publishedlink-Thisisthelinktothepublisheddatasetontheopendataportalorcitywebsite
Reviewdatasecurityandprivacyprotectionprotocols
OftenapolicychangealsoinvolvesachangeinthedatathatisbeingcollectedhowthatdataisbeinghandledwhohasaccesstoitandahostofotheradjustmentsAsyoutakestockofthedataassetsthatareavailableforyourinitiativeitisvitaltoalsothinkthroughwhatwillbedonetoprotectthatdataandtheprivacyofthepeopleitrepresentsAbigpartofgooddatasecurityinvolvesstrongdatagovernancepracticesthatdefineandidentifysensitivedataandconsciouslyworktobalancethevalueandriskofstoringsuchsensitivedataOnewayofevaluatingyourdatacomesfromcaselawrelatedtotheUSFreedomofInformationActwhichwehavetranslatedtothechecklistbelow
1 Woulddisclosureresultinasubstantialinvasionofprivacy2 Haveweconsideredtheextentorvalueofthepublicinterestandthepurposeorobject
oftheindividualsseekingdisclosure3 Istheinformationavailablefromothersources4 Wastheinformationgivenwithanexpectationofconfidentiality5 Isitpossibletomouldreliefsoastolimittheinvasionofindividualprivacy
TheseriskscouldbemitigatedusingavarietyofstrategiesfromaggregatingdatatoobfuscatingindividualstoredactingfieldsthatmayincludePersonallyIdentifiableInformation(PII)orre-evaluatingyourdatacollectionmethodstofindalternativewaysofgettingtheinformationyouneedThesestepsshouldbecodifiedandmanagedbyadatagovernancecommitteeifpossibleForthesensitivedatathatisdeemedessentialcreateaplanaroundpotentialdatabreachesinordertoquicklyandconfidentlyrespondtoemergencysituations
ProactivelypreparingforthesesituationsisoftenthebeststrategyfordatasecuritybutdonotforgetthetechnicalsideInvolveyourITadministratorsinconversationsaboutdatastorageandaccessensuringthattheywillbeprotectedbyenterprise-gradecybersecuritysystems
Assessdataaccessibilityqualityandsecurity
42
Assessdataaccessibilityqualityandsecurity
43
Designimplementationandevaluation
Designtheimplementation
NowthatyouhaveyouraccountabilityplanandkeyindicatorsitistimetogetspecificaboutyourplanforimplementationThetemplatebelowwillhelpyouthinkthroughthespecificmechanismsofimplementingpolicychange
UnderstandingorganizationsandsystemsiscriticaltounderstandingandmakingdecisionsaboutwhereandhowinterventionsshouldbeappliedApolicyimplementedinasinglesettingmayshowpromisingresultsbutimplementingmultiplepoliciesandprogramsatthesametimewithdifferentactorsresponsibleforeachaspectcanhelpensuresustainablebehaviorchange
InSomervilleMAShapeUpSomerville(SUS)isacitywidepublichealthprogramthatisimplementedacrossmultiplelevelsofinfluenceandsystemsthroughinterventionsbeforeduringandafterschooltoincludechangestobuiltenvironmentsuchasbikelanesregionalmasstransportationandphysicaleducationequipmentinschoolsandgymstoincreasephysicalactivitywellnesspoliciesandschoolgardensnutritioneducationduringandafterschoolandhealthyrestaurantprogramstosupporthealthyfoodandhealthyeatinghabits
Usethetemplatebelowtoplanyourimplementation
Policyimplementationplanningtemplate
ListthespecificdesiredoutcomesInthepreviousstepyouworkedwithdepartmentstafftoestablishspecificmeasurableoutcomesandinterimindicatorsPrioritizetheindicatorsthatwilldemonstrateprogressUsethespacebelowtocapturethedesiredoutcomesofthepolicychange
ExampleShapeupSomervillebeganwiththegoalofpreventingobesityin1stthrough3rdgraders
Yourturn
____________________________________________________________________________
____________________________________________________________________________
Designimplementationandevaluation
44
SpecifythetargetpopulationAreyoudesigninganinterventionthattargetsonlycertainsitesAreyouplanningonimplementingaprogramorinitiativeforaspecifictargetaudienceClearlyspecifythepopulationyouplantotarget
IfyouplantoconductarigorousimpactevaluationasnotedbytheWilsonSheehanLabforEconomicOpportunitiesitisimportanttoestablishconcreteobjectiveeligibilitycriteriaandensurethatthesecriteriaareappliedconsistentlybyallstaffduringscreeningandintakeItisalsovitaltoidentifytheindividualswhoareeligibleforaparticularprogramorserviceevenifeveryoneeligiblecannotbeservedbyexistingresourcesThishelpscitiesidentifyanyunmetneedsandguidefutureresourceallocationwhilealsocreatingthepossibilityofarigorousimpactevaluationbyidentifyingapotentialcomparisongroupFinallywhenconductingscreeningandintakeitisimperativetocollectandmaintainbasicinformation(egnamedateofbirthgender)aboutallprogram-eligibleindividualsnotjustthosewhoparticipateintheprogram
ExampleIntheShapeUpSomervilleexampletheprogrambeganbytargeting1stthrough3rdgraders
Yourturn
____________________________________________________________________________
____________________________________________________________________________
DefinethelocationWherewillthepolicychangetakeplaceWillimplementationtakeplaceatonesiteormultiplesites
ExampleShapeUpSomervilleinterventionsweredeliveredinschools
Yourturn
____________________________________________________________________________
____________________________________________________________________________
ClarifyimplementationpartnersandarticulatetheirrolesWhatarethekeyorganizationsinstitutionsorpeopleresponsibleforimplementationTheprojectteamshouldbeasclearandspecificaspossibleinprovidingadescriptionoftheconcreteactivitiesthatimplementationpartnersaremeanttocarryoutPracticeprofilesmaybeahelpfultooltohelpdefinewhatisrequiredoftheactorswhoareresponsibleforimplementingtheintervention
Designimplementationandevaluation
45
ExampleInSomervillemayoralleadershipandengagementofmultiplecityagencies(publichealthprimarycareschoolsCambridgeHealthAllianceetc)streamlinedandconnectedthevariousShapeUpSomervilleinitiativesandgrantswhichhelpedtokeepthetalentandenergyfocusedonhealthyeatingandactivelivingforallsectorsofSomervilleSUSpartnersultimatelyincludedschoolsnonprofitsCityandstateagenciesandnationalfoundations
Yourturn
____________________________________________________________________________
____________________________________________________________________________
DescribetheinterventionWhatarethestepsactivitiesorprocessesRememberthatthemechanismthespecificstepsorprocessthatleadtobehaviorchangeiskeytounderstandingtheeffectivenessofthepolicychangeandsharingtheresultswithothers
ExampleGoalsofShapeUpSomervilleweretoinfluenceeverypartofanearlyelementaryschoolchildrsquosdayndashbeforeduringandafter-schooltopromotehealthyeatingphysicalactivityandtheirweightstatusInterventionincludedfoodserviceenhancementstoimprovebreakfastandlunchandstaffprofessionaldevelopmentwalktoschoolactivitiesSUSclassroomcurriculumandprofessionaldevelopmentschoolwellnesspolicydevelopmentSUSafter-schoolcurriculumandprofessionaldevelopmentoutreachandeducationtohomethroughmaterialsforumseventsldquoSUSapprovedrdquorestaurantscommunityoutreachandcapacitybuildingthroughpolicydevelopmenttrainingsmediaplacements
Yourturn
____________________________________________________________________________
____________________________________________________________________________
ClarifytimelineandmilestonesfordeliveryandevaluationHowfrequentlywilltheservicesinterventionbedeliveredHowlongwilltheprogramrunWhatisthetimetoexpectedimpacts(eg12monthsafterprogramcompletion)
Designimplementationandevaluation
46
ExampleShapeUpSomervilleledbyTuftsUniversityFriedmanSchoolofNutritionScienceandPolicyconductedaquasi-experimental(nonrandomized)interventionstudyamong1stthrough3rdgradestudentsinSomerville(intervention)andtwocomparisoncommunitiesfor9-monthsduringoneschoolyear(fall2002ndashspring2003)ThestudyshowedpositiveassociationoftheinterventiononchangeinBMISUSreleasedreportssummarizingevaluationsprogresstodateplansforadditionalprojectsandimpactstenandfifteenyearsintotheinitiative
Yourturn
____________________________________________________________________________
____________________________________________________________________________
PlanyouranalysisTheprevioussectionaddressedestablishinganaccountabilityframeworktomonitorprogressandimplementationDeterminewhichindicatorswillbediscussthroughthatprocesswhowillsharethedataandanalysisandhowfeedbackfromthoseconversationswillbeincorporatedbytheprojectteam
ExampleTheMayorrsquosofficeimplementedadata-drivendecisionmakingapproachcalledSomerStatwhichwassynergisticwiththeapproachestakenbytheresearchandevaluationteamsdeployedforthespecificgrants
Yourturn
____________________________________________________________________________
____________________________________________________________________________
PlanforsustainabilityIfthepolicyandprogramchangeissuccessfulhowwilltheprojectteamsustainandexpandtoadditionalsitesorpopulations
ExampleAlthoughmostofthepartnershipsforgedintheearlyyearsweremaintainednewgrantsandcollaborationsexpandedattentiontoolderpublicschoolstudentsactivetransportationandtheGreenLinesubwayextensionthebuiltenvironmentfarmerrsquosmarketeffortssuchasldquoGrowninSomervillerdquowithUnionMainStreeteconomicdevelopmentandhealthdisparitiesaswellasdatacollectionmonitoringandsurveillance
Yourturn
Designimplementationandevaluation
47
____________________________________________________________________________
____________________________________________________________________________
Pilotplanningtemplate
IfyouplantoconductapilotfirstdeterminethepurposeofthepilotDoyouwanttoobservewhetherthepolicychangeresultsinthedesiredbehaviorchangeAreyouconcernedaboutfeasibilityscalabilityorcostDoyouwanttotestwhetheryourdesiredresultscanbeachievedthroughthispolicyapproachClarifythegoalofthepilotandthequestionsyouhopetoanswerThiswillhelpyouidentifythekeyindicatorsthatyouwillusetomeasurewhetherthepilotissuccessfulBeforebeginningestablishabaselineofcurrentperformanceandsetyourtargetsaccordinglybasedonwhatyouhopetoaccomplishthroughthepilot
NextestablishthepilotplanforyourprojectteamincludingmilestonesforteamcheckinstodeterminewhethertomoveaheadorchangecourseThisisseparatefromthemeetingsthatwilloccuraspartoftheaccountabilityframeworkUsethespacebelowtocreateyourtimelineanddevelopyourmeetingschedule
Task Personresponsible Startdate Checkindate Deliverable
Planyourevaluation
WhileyouaredesigningtheimplementationitisalsotimetobegintodevelopyourplanforanalysisAsnotedbytheWilsonSheehanLabforEconomicOpportunitiesdecisionsaboutprogramsetup-includingoutreachscreeningintakedatatrackingandenrollment-candeterminewhetherornotarigorousimpactevaluationwillbepossibleAsdiscussedcityleadersmayfindithelpfultoincludeevaluationpartnersinearlyconversationsaboutprogramsetupDuringpreviousstagesyoumayhaveworkedwitharesearchertoplanyourproposedpolicychangeYoushouldbeclearaboutthepurposeoftheevaluationanddevelopanevaluationplanBelowareadditionalconsiderations
Causationvstrendanalysis
IntheEstablishaframeworktomonitorprogressstagetherewasguidanceontrackingtrendsovertimeItisimportanttonoteherethatmonitoringdataisdifferentfromanalyzingacausaleffectTrendanalysiswillillustratethatchangesareoccurringandmayleadtoquestionsthatcouldbeansweredbyyourevaluationbuttrendanalysisalonedoesnotprovideevidenceaboutwhatisdrivingthosechangesYoumayseechangesinyour
Designimplementationandevaluation
48
outcomeindicatorsbasedonvariablesoutsideoftheCityscontrolForinstanceachangeinfederaltradepolicycanhaveanimpactemploymentratesinyourCityoutsideofyourCitysnewjobtrainingandplacementprogram
ToaddresscausalityyoumustcomparetheoutcomesofindividualswhoreceivedtheinterventionwiththosewhodidnotWithoutacomparisongroupofsimilarindividualswhowerenotexposedtotheinterventionitisimpossibletoknowwhichfactorsareresponsiblefortheoutcomesRandomizedControlTrials(RCTs)inwhichindividualsarerandomlyassignedtotreatmentandcontrolgroupsarethegoldstandardinprovingcausalityIfyoucandemonstratethatoutcomesimprovedforthosewhoreceivedtheinterventionanddidnotimproveatallorimprovedlessforacomparisongroupyoucanbesomewhatcertainthattheinterventionwasthecause
Examplesofothernon-causalimpactevaluationsthatarerigorousandreliableincludecertainquasi-experimentalstudiesusingcomparisongroupdesignssmartinterruptedtimeseriesanalysesthatmodeltrendsovertimeinwaysthatallowestimationofinterventioneffectswelldesignedpre-posttestanalysesneedsassessmentsimplementationstudiesetcFormoreinformationondifferenttypesofevaluationsseetheSageResearchMethodswebsite
Lookfordifferencesingroups
BuildintimetoalsoexaminethepilotdatatoseeifgroupsareexperiencingdifferentresultsReviewingperformancebasedonoutcomemeasuresforapopulationinaggregatemayhidethesubtlewaysthatsomepeoplemaybeaffecteddifferentlysoitisnecessarytocarefullyanalyzethedatafordisparateoutcomesForinstancetheMovingtoOpportunity(MTO)studythatexaminedtheimpactofhousingvouchersonfamilieswholivedinhigh-povertyneighborhoodsfoundthatchildrenwhomovedbeforetheageof13hadbettereducationandhigherincomesasadultsHoweveradultsdidnotseeimprovededucationandemploymentoutcomes
Disaggregatingdataintomorespecificcategoriesofraceandethnicitycanhelpunderstandtrendsanddisparitiesinamoregranularwaywhichcanhelpdecisionmakerstargetpoliciesandinterventionsinamoreequitableway
EarlierintheRoadmapweworkedtoidentifystakeholdersandwhomightbepositivelyandnegativelyimpactedbyachangeinpolicyUsetheresultsofthatexercisetoinvolvethecommunityininterpretingdataanalysisandtogatherqualitativefeedbackfromthecommunitytocallattentiontoissuesForinstanceanincreaseinyouthdetentionmaybeexplainedbyinefficienciesorlackofaccesstotransportation
Designimplementationandevaluation
49
Designimplementationandevaluation
50
Shareyourstoryandsuccess
Sharetheresults
Storytelling
AnintegralstepinyourstrategyissharingtheresultsofyourpilotorpolicychangewithyourcommunityThisincludesdevelopingmessagingaroundtheworkofyourorganizationandcreatingaplantodisseminateyourstoriesWhytellstoriesaboutyourwork
Informandorunderstandasubjectorsituation
Giveneededcontextofdataorbitsofinformation
Relatablestoriesarememorableforreadersviewers
ReadersorviewerswillseetheinformationasmorecompellingwhenthereisademonstratedandrelatablechangeorimpactonthemselvesortheirfriendsneighborsandcolleaguesStoriesareaboutpeopleandgovernmentisfundamentallyabouthelpingpeopleSotellthestoriesofthepeopleyouandyourcitycolleagueshelped
Keepthesetipsinmindasyoudevelopyourstorywithyourteam
WhatwereyourfindingsandwhyshouldanyonecareTellyouraudiencehowtheproblemwassolvedWhathastheimpactbeenonresidentsandcitystafftodateWhatwasthesuccessandweretheremultiplesuccessesalongthewayIfyouhavepreliminaryresultsordatashowingimprovementtosharethisisthetimetodoit
WhowasinvolvedandimpactedReviewthelistofstakeholdersHowwereeachofthemimpactedbytheprogramHumanizeyourstoryandimpactbyaddingadditionalvoicesThesevoicescanbeoneofthefollowing
Apersonorfamilywhowaspositivelyaffectedbythechangesuccess
Aninvolvedthird-partydescribingboththeproblemandthesolution(ieanorganizationnonprofitprovidersbusinesstradeassociationsportsteametc)whoispartofthecommunitythatthegovernmentworkedwithonthechangesuccess
Someonefromthegovernmentitselfwhocanspeakaboutthechangesuccess
TellmoreaboutyourcityDescribeyourcityincludingpopulationdemographicsindustriesetcWhatareitschallengesWhatisitknownforWhatmakesituniqueHowisitsimilarordifferentfromothercitiesintheUS
Shareyourstoryandsuccess
51
DescribeyourcityrsquoschallengeWriteoutthechallengeyourcityisfacing(asdefinedinyourproblemstatement)HowdoesitaffectresidentsHowdoesthisproblemaffecttheworkofcitystaffmembers
AddthedetailsProvidedetailsonyourappliedresearchstrategyTalkaboutwhatyourcityhasdonenthepasttoresolvetheissueDiscusshowyourstrategyhaschangedwhenyoubegantoadoptthisapproachWasmoneysavedorinvested
Usethisstorymapexercisetoorganizeyourthoughtsandguaranteethatyouincludealloftheelementsofatraditionalstory
AfteryoudevelopyourmessagingcreateacommunicationsmarketingandoutreachstrategydesignedtospreadyourmessageYourstrategyshouldwalkyouthroughthefollowingkeyactivities
1 IdentifyandprofileyouraudienceWhencreatinganddisseminatingyourmessageitisalwaysgoodtothinkaboutyouraudienceWhoexactlywouldbeinterestedinwhatyouhavetosayThegeneralpopulationofthecityPeoplewhoworkinnonprofitsBusinessownersItmattersbecausenoteveryonewholivesinacitymaybethetargetoftheinformationBusinessownersmaybemoreinterestedindatashowingeffectsonthebusinessdistrictthanothersandsoonWewillcoverthislaterbutthisstepwillalsohelpyoutothinkofthemosteffectivewaytoreachyouraudience
2 Defineyourcalltoaction(ifyouhaveone)SpendsometimeconsideringwhyyouareinterestedintellingyourstoryDoyousimplywanttoupdatethepubliconyourprogressorcelebrateyoursuccessesAreyouinterestedinyouraudiencechangingitsbehaviororsupportingaspecificinitiativebeinglaunchedbyyourorganizationIfyouwanttohaveyouraudienceactorbehaveinacertainwaybespecificandincorporateitinyourmessaging
Shareyourstoryandsuccess
52
3 SelectperformancemeasuresThinkabouthowyouwoulddetermineifyourcampaignissuccessfulandselectappropriateperformancemeasuresYouwillrevisitthesemeasuresattheendofyourcampaigncommunicationsplanimplementationtodeterminewhetheryouneedtomodifyyourcommunicationsandoutreachstrategyforthefuture
4 MethodstocommunicateyourstoryTherearemanywaystocommunicatesuccessofyourprojectHerearesomesuggestionsonreachingaudiencesItisimportanttodeterminewhichmethodsarethemosteffectiveforyourdesiredaudiencetoreceiveinformationDoyourresearch(PewResearchCenterhasexcellentstudiesonthereachofmediadigitalmediaandmarketing)ForexampleifseniorsareyourtargetpopulationandtheyarelessinclinedtouseInstagram(thatistrue)youmaywanttoconsiderFacebookorthenightlylocalnewsbothofwhichareusedmoreoftenbyseniorsBelowisalistofsomemethodsandtoolsyoucanuse
i Pressreleasestolocalnewspapersandtvstations
ii ShortsmartphonevideosAgreatwaytoengageanaudienceisthroughvideoAskonequestionofsomeonewhoispartofthestoryMaybethereisasocialworkerwhocandiscusshungerinyourcityThesevideoscanbereusedonmultipleplatforms
iii Long-formwriting
iv TellingindividualstoriesthroughaseriesofphotosandshortnarrativesforpublicationonFacebookorInstagram
v Op-edsopinionpiecesplacedinthelocalpaper
vi Emailupdatestoyourlist
vii Connectingwithappropriateorganizationsthatareinvestedinthetopicandworkwiththecommunityandaskthemtodisseminateinformationtotheirmembers
viii EmployingldquoWordofMouthrdquo(WOM)tacticsbycreatingtownhallsorfreeeventsforresidentstoattendintheirownneighborhoods
5 DeterminethetypeofcollateralUsuallycampaignsthattargetresidentsrequireseveralavenuestoreachthemPeoplearenotalwaysinthesameplaceorspaceHavingakeenunderstandingofyouraudiencewillassistyouwithyourchoicesonwhatneedstobedevelopedSometimesthatmeansimagesforsocialmediaaonepageflierthatgivespeopleinformationonattendinganeventoranin-depthbrochure
Shareyourstoryandsuccess
53
6 ImplementyourstrategyKeepinmindstrategiescanbefluidSometimessomethingmaynotturnoutasexpectedevenifduringplanningsessionsitwasldquosupposedtordquoItisokYoucananalyzewhathappenedduringadebriefsessionandonceyoursurveysandothermeasurementshavebeenfinalized
Visualizations
Withtherightvisualizationyoucanreviewandshareinformationinsightsanalysisperformanceandtrendsallofwhichisvaluabletoacross-sectionofthepublicButwhichvisualizationisrightforyouTochoosetherighttypeofchartgraphdeterminethetypeofquestionyouwanttoanswerSomecommoncharttypesarebelowandmoreinformationcanbefoundhere
Exploreadditionalanalyticaltechniques
Duringthegoal-settingphaseyouestablishedoutputoutcomeandimplementationmeasuresBasedontheresultsofyourpilotyoumayneedtoadjustyourmeasuresWiththepilotdatainhandyoumayhavesomeideasofhowfutureanalysescouldbedifferentandpotentiallyimprovethequalityofyourprogramTakesometimetobrainstormotherrelevantdatasetsthatyoudidnothavethetimetopursueearlierincludingmessydepartmentaldatathatneedsmoreattentionorsourcesofpublicdatafromthestateandfederalgovernmentnonprofitsandevenprivatecompaniesthatcouldsupplementyourunderstandingofanissueUsingmachinelearningtechniquesyoumightbeabletobetter
Shareyourstoryandsuccess
54
predictdemandforhomelessservicesusingNOAAweatherforecastspreemptwatermainbreaksusingsoilstratigraphyorinvestineconomicdevelopmentbymonitoringrealestatelistingdata
Advancedanalyticaltechniquescanalsohelpyourimplementationrunmoreeffectivelybyintegratingdata-informedtargetinganddecisiontoolsintotheprocessByautomatingroutineworkflowsandsurfacingsubtletrendsthesetoolscanhelpyouiterateinamoreprecisemannerandprovidekeystrategicguidancetofront-lineemployeesthatwouldotherwisebeunavailableThesesortsofdata-science-enhancedprocessescanbeclassifiedasoneofsixprojecttypesasexploredfurtherbytheNewOrleansLANOLAlyticsteamFormoreguidanceonscopingasimilarprojecttheGovExadvancedanalyticsscopingguidegivesyouastep-by-stepoverviewofhowtobesuccessful
Usethespacebelowtolistyourideasforpossibleadvancedanalyticsprojectsbytype
Typeofproject Ideas
Findingtheneedleinthehaystack
Prioritizingforimpact
Earlywarningtools
Betterquickerdecisions
Optimizingresourceallocation
Experimentingforwhatworks
Keepthefeedbackloopgoing
GatherinformationonpublicsentimentaboutpolicychangethroughfocusgroupssurveysandpublicmeetingsledbyadvisorygroupmembersNotewhethernewinformationisavailablethatcanhelpanalyzeorcontextualizeyourprogress(egfederalorstatedatareports)Sharetheresultsofyourpilotandthepolicychangeprocesswithothercitiesandresearchers
Shareyourstoryandsuccess
55
WorkingwitharangeofevidencePresentlythereisalotofattentionandinvestmentinevidence-basedpolicymakingincludingthefederalbipartisanCommissiononEvidence-BasedPolicymakingthePewCharitableTrustsreportsandtoolkitsonHowStatesEngageinEvidence-BasedPolicymakingandtheLauraandJohnArnoldFoundationsupportedEvidence-BasedPolicymakingCollaborativeManyoftheseinitiativesfocusonrigorousevaluation--meaningrandomizedcontrolledtrials(RCT)orotherexperimentalmodels--asthebestkindofevidencetouseHoweverthisbodyofevidenceisofteninsufficientforthespecifickindsofquestionsthatgovernmentscanhavewhenmakingadecisionaboutpolicyInthisguideweproposeanapproachtosupportgovernmentsinincorporatingadditionaltypesofevidenceintotheirdecisionmaking
OnewaytounderstandarangeofevidenceistouseNestarsquosStandardsofEvidencetoframeanddefineevidenceTheBritishnonprofitNestaworkstoinjectpositivechangeintothewayUKgovernmentpoliciesandprogramsfunctionOneoftheirkeyareasofworkhasbeendevelopingWhatWorksCentresorganizationsthatcollectcommissionsynthesizeanddisseminateevidenceaboutinterventioneffectivenessacrossarangeofsocialimpactareasNestadevelopedthefollowingStandardsofEvidencetoprovideausefulwaytodocumenthowdifferentkindsofinformationdemonstratedifferentlevelsofevidence
Workingwitharangeofevidence
56
Theimportanceofpotentiallybiasedinformation
Onechallengepresentedbyanevidenceclassificationthatacknowledgesthevalueofnon-scientificformsofevidenceistofigureouthowtobalanceallofthepiecestogetherwhenyouusethemTounderstandhowtodothiswehavetothinkabouthowtopositionpotentiallybiasedinformationindecisionmaking
WhenwelookatinformationwhichhasnotbeenrigorouslyevaluatedweacceptthatthisinformationmightbebiasedThewholereasonweconductrigorousevaluationistoremovebiasfromourobservationsWeconductstructuredtrialstoproduceresultsthatareconsistentandgeneralizableIfafindingisunbiasedthisaccuracymdashthislackofbiasmdashcanbeprovedbecauseanyindependenttestercanreproduceitwhentheyfollowanidenticalprocess
YoucanlookatpotentiallybiasedinformationasjustinformationthatisearlieroninitspathtorefinementreplicationandverifiedaccuracyApieceofinformationbasedonanyoneindividualrsquosobservationsislikelytobebiasedbecauseitiscomingfromjustonesourceBiasisreducedbygatheringtogetherlotsofindependentsourcesofinformationandfiguringoutitsmostcommonattributes
MaybethebiasedinformationmakesaclaimthatistoolargemdashitsayssomethingistrueunderallconditionswhenitisreallyonlytrueundercertainconditionsorthataneffectofaninterventionisbiggerthanitstrueaverageorthataneffectlastsovertimewhenitdoesnotPerhapschangingtheclaimtobemorespecificmdashcriticallyreviewingtheinformationtoseeifspecificconditionsmustbemettoachievetheoutcomesmdashwouldmaketheclaimmoreaccurate
IfyoutestedyourclaimfounditdidnotholdupinsomecaseschangedittesteditagainuntilyourclaimrepeatedlyheldupthenyoucoulddemonstratethattheclaimwasnolongerbiasedbutaccurateThisprocessoftestingimprovingtestingandvalidatingiswhatwegetfromstructuredscientificstudy
Withindividualpiecesofinformation--crediblehypothesescasestudiesexpertopinion--youcannotassumethatkindofaccuracyAtthesametimetheseearlyobservationsarefoundationaltoanyreplicablescientificobservationsWewillnevergetanywhereifwedonotstartwiththesepiecesofinformationeveniftheymightbebiased
Combininglevelsofevidencetogetthemostdetailedinformationforyourcity
ItisespeciallyimportanttoacceptdifferentformsofevidencewhenyoufacequestionsthathavenotyetbeentestedbyindependentstudiesSomeofthemostimportantquestionsrelatetothingsthatareveryspecifictoyoursituationYoufaceaparticularcombinationof
Workingwitharangeofevidence
57
issuesactorstrendsandresourcesthatisultimatelydifferentfromeveryotherplaceintheworld
ThisisimportantbecauseaspolicypractitionersandnotscientistsyoucaremoreaboutyourspecificcasethanahypotheticalaveragegeneralcaseYoucarelessaboutthetotallikelyrangeofimpactsfromapolicythanhowthissingleimplementationwillworkYouwantinformationaboutpoliciesthatwillprovideyouwiththebestpossiblepredictionabouthowtheywillworkinyourcontext
ByputtingdifferentkindsofevidenceintothesameframeworkNestarsquosStandardsofEvidenceimplythatwhenyoucannotgetrigorouslytestedexperimentallyverifiedinformationitisappropriatetoworkwithwhatwedohaveThisisrelevantforthemunicipalimplementationcontextbecausenostudypulledfromaliteraturereviewwillaccountforallofthevariablesyoufaceinyourowncontext
Intheabsenceofhavingastudythatpredictshowaninterventionworksinyourspecificcontextyouwillneednon-scientificinformationtotrytofillinthosegapsYouwillneedtogatherstoriesfromresidentsstaffmembersandotherstakeholdersthatsitonLevels1or2ofthestandardsofevidence
Someofthekindsofinformationyouwillprobablygetmoreeasilyfrompeoplethanstudiesinclude
howpeopleincitieslikeyoursreactedtoaninterventioninpublicforumshowpeoplewhoimplementedaninterventioninacitylikeyoursfiguredouthowtogetitpassedhowthecitycommunicatedpubliclyabouttheapparentbenefitsanddrawbacksoftheinterventionthepoliticalacceptabilityoftheintervention
Youcanremainsimultaneouslyawarethattheinformationyouaregatheringthroughindividualstoriesisbiased--notgeneralizableinformedbyspecificperspectives--whilealsorecognizingthathavingsomeinformationaboutsomethingimportantisbetterthannoinformationEvidencefromthefirsttwoevidencelevelsareessentialbuildingblocksofknowledgeandperfectlyusefulwhenyoucannotgetdetailedunbiasedknowledge
Youhavetostartsomewhereandyoushouldalwaysstartwithevidenceyouhaveaboutyourveryspecificconditionsthatyouknowareimportantforsuccess
Workingwitharangeofevidence
58
Thankyou
Interviewswithcities
GovExheldsemi-structuredinterviewswithsixstaffmembersinfourcitiestoestablishtheinitialstepsoftheRoadmapWeaskedrespondentstotalkaboutaninitiativeorpolicythattheyhavebeenresponsibleforimplementingintheircityRespondentsfromthreecitiesspokeabouttheireffortstoaddresshomelessnessandrespondentsfromonecityspokeaboutblightAfterrespondentsidentifiedtheproblemtheyspokeabouthowthecityprioritizedthisissuethevariousstepsinvolvedandbarrierspresent
GovExconductedtextanalysisoftheseinterviewswhichrevealedthatpeoplewasthemostcommonlyusedtermduringtheseconversationswhendiscussingthestepstochangeaprogramorpolicyThisisathemethroughoutthisRoadmapThefirstsectionbeginswithidentifyingtherelevantpeopleTheconversationsaroundevidencearefocusedonhumanbehaviorratherthanmethodologyorstrengthofevidence
Thankyou
59
OncewehadadraftoftheRoadmapwefollowedupwithcitiesonanindividualbasistolearnmoreaboutwhatwouldbehelpfulWearegratefultothefollowingcitystafffortheinterviewsandindividualconversations
MichaelBaskinChiefPolicyOfficerChattanoogaTN
MelissaBridgesPerformanceandInnovationCoordinatorLittleRockAR
VictoriaCarreoacutenAdministrativeOfficerLasVegasNV
TyroneGrandstrandCommunityDevelopmentPlannerFargoND
TibLaughlinDirectorofGeneralServicesKansasCityKS
TimMorelandDirectorofPerformanceManagementandOpenDataChattanoogaTN
CharlesRosendahlManagementAnalystDentonTX
Thankyou
60
MelissaSiebenAssistantCountyAdministratorKansasCityKS
GinaThompsonDirectorofStrategicServicesLewisvilleTX
JHUFacultycollaboration
GovExalsoconvenedthecitycontactsandfacultypartnersinaseriesofAdvisoryGroupmeetingsThesemeetingsservedtoconnectcitiesandfacultytounderstandoneanothersperspective
StefanBaralBloombergSchoolofPublicHealth
AshleyBernerJHUSchoolofEducation
MelissaDavey-RothwellBloombergSchoolofPublicHealth
JonLinksBloombergSchoolofPublicHealth
TamarMendelsonBloombergSchoolofPublicHealth
KristinMmariBloombergSchoolofPublicHealth
KeshiaPollackPorterBloombergSchoolofPublicHealth
JoshSharfsteinBloombergSchoolofPublicHealth
LizStuartBloombergSchoolofPublicHealth
LindsayThompsonCareyBusinessSchool
DanielWebsterBloombergSchoolofPublicHealth
Partnersindataandevidence-basedpolicy
Wealsospokewithpartnerswhoareexpertsinthefieldofevidence-basedpolicywhosharedtheirknowledgeandresourcesofcurrentandfuturethinkinginevidence-informedthinking
WendyBarrenoAssociateDirectorWilsonSheehanLabforEconomicOpportunitiesUniversityofNotreDame
TiffanyDavenportDirectorofDataPartnershipsBaltimoresPromise
TomiHiersDirectorBaltimoreCivicSiteAnnieECaseyFoundation
ChrisKingsleySeniorAssociateAnnieECaseyFoundation
KathrynOliverAssociateProfessorLondonSchoolofHygieneandTropicalMedicine
Thankyou
61
KristenWatkinsSeniorPolicyAssociateJ-PAL
GovExParticipants
GovExAnalystMiriamMcKinneyanalyzedtheinterviewtranscriptsandconductedtextanalysisSeniorImplementationAdvisorEmilyShawwillintegrateRoadmapactivitieswithadditionalservicedeliveryactivitiesatGovEx
Thankyou
62
ReferencesAbrahamKGampHaskinsR(2017)Thepromiseofevidence-basedpolicymakingReportofthecommissiononevidence-basedpolicymakingWashingtonDCCommissiononEvidence-basedPolicymakingBigmove2Extendingthegreen(nd)Retrievedfromhttpswwwcityoflewisvillecomdoing-businesslewisville-2025big-move-2-extending-the-greenBatesMAampGlennersterR(2017)ThegeneralizabilitypuzzleRetrievedfromhttpsssirorgarticlesentrythe_generalizability_puzzleBuildwithnotfor(2018January)RetrievedfromhttpsbuildwithnotforwordpresscomCairneyPampOliverK(2017)Evidence-basedpolicymakingisnotlikeevidence-basedmedicinesohowfarshouldyougotobridgethedividebetweenevidenceandpolicyHealthResearchPolicyandSystemsdoi101186s12961-017-0192-xCairneyPOliverKampWellsteadA(2016)TobridgethedividebetweenevidenceandpolicyReduceambiguityasmuchasuncertaintyPublicAdministrationReview76(3)399-402Retrievedfromhttpdxdoiorg101111puar12555Centreforpublicimpact(nd)RetrievedfromhttpswwwcentreforpublicimpactorgCernaL(2013)ThenatureofpolicychangeandimplementationAreviewofdifferenttheoreticalapproachesOrganisationforEconomicCo-OperationandDevelopmentRetrievedfromhttpswwwoecdorgeduceriTheNatureofPolicyChangeandImplementationpdfColoradoSprings(nd)Retrievedfromhttpscoloradospringsgovcommunicationspagewhat-expect-when-you-speakupContinuumofcare(coc)program(nd)RetrievedfromhttpswwwhudexchangeinfoprogramscocCouncilpresidentkicksoff$12millionyouthfund(nd)Retrievedfromhttpwwwbaltimorecitycouncilcomcontentcouncil-president-kicks-12-million-youth-fundDurlakJAampDuPreEP(2008)ImplementationmattersAreviewofresearchontheinfluenceofimplementationonprogramoutcomesandthefactorsaffectingimplementationAmericanJournalofCommunityPsychology41327-350Retrievedfromhttpswwwncbinlmnihgovpubmed18322790Fishbonediagram(ishikawa)(nd)Retrievedfromhttpasqorglearn-about-qualitycause-analysis-toolsoverviewfishbonehtmlFivewhysGettingtotherootofaproblemquickly(nd)RetrievedfromhttpswwwmindtoolscompagesarticlenewTMC_5Whtm
References
63
FligsteinNampMcAdamD(2011)TowardageneraltheoryofstrategicactionfieldsSociologicalTheory29(1)1-26RetrievedfromhttpsociologyberkeleyedusitesdefaultfilesfacultyfligsteinFligstein20McAdam20Soc20Theory20PaperpdHeinkelP(2018January12)CitymullspermitsforbikesharingcompaniesRetrievedfromhttpwwwdentonrccomnewsdenton20180112city-mulls-permits-bike-sharing-companiesHowisareamedianincomecalculated(2012October)Retrievedfromhttpswwwhudexchangeinfofaqs843how-is-area-median-income-calculatedLampkinLMampHatryHP(2003)KeystepsinoutcomemanagementUrbanInstituteRetrievedfromhttpswwwurbanorgresearchpublicationkey-steps-outcome-managementMarshDampSmithM(2000)UnderstandingpolicynetworksTowardsadialecticalapproachPoliticalStudies484-21Retrievedfromhttpjournalssagepubcomdoiabs1011111467-924800247McCannL(nd)RetrievedfromhttplaurenellencomMoultonSampSandfortJR(2016)ThestrategicactionfieldframeworkforpolicyimplementationresearchPolicyStudiesJournal45(1)Retrievedfromhttponlinelibrarywileycomstore101111psj12147assetpsj12147pdfv=1ampt=jdkhhcp4amps=e184c625e1855a6b1615ccb99ee27b1c0b3212d8PaynterB(2018January1)What100citiesarelearningfromeachotherbysharingtheirdataRetrievedfromhttpswwwfastcompanycom40518004what-100-cities-are-learning-from-each-other-by-sharing-their-dataPeakacademy(nd)Retrievedfromhttpswwwdenvergovorgcontentdenvergovenmayors-officeprograms-initiativespeak-performancepeak-academyhtmlPinnockHBarwickMCarpenterCREldridgeSGrandesGGriffithsCJRycroft-MaloneJMeissnerPMurrayEPatelASheikhAampTaylorSJC(2017)Standardsforreportingimplementationstudies(stari)ExplanationandelaborationdocumentBMJOpenRetrievedfromhttpbmjopenbmjcomcontentbmjopen74e013318fullpdfProctorESilmereHRaghavanRHovmandPAaronsGBungerAGriffeyRampHensleyM(2011)OutcomesforimplementationresearchConceptualdistinctionsmeasurementchallengesandresearchagendaAdministrationandPolicyinMentalHealth38(2)65-76doi101007s10488-010-0319-7TandonSD(2018)ProgressincommunityhealthpartnershipsResearcheducationandactionRetrievedfromhttpswwwpressjhuedujournalsprogress-community-health-partnerships-research-education-and-actionProvanKGampKenisP(2007)ModesofnetworkgovernanceStructuremanagementandeffectivenessJournalofPublicAdministrationResearchandTheory
References
64
18(2)229-252Retrievedfromhttpsacademicoupcomjpartarticle182229935895SabatierPA(1988)Anadvocacycoalitionframeworkofpolicychangeandtheroleofpolicy-orientedlearningthereinPolicySciences21(3)129-168Retrievedfromhttpwwwjstororgstable4532139SandstromAampCarlssonL(2008)TheperformanceofpolicynetworksTherelationbetweennetworkstructureandnetworkperformancePolicyStudiesJournal36497-524Retrievedfromhttponlinelibrarywileycomdoi101111j1541-0072200800281xabstractSchwartzMampWilsonE(nd)WhocanaffordtoliveinahomeAlookatdatafromthe2006americancommunitysurveyRetrievedfromhttpswwwcensusgovhousingcensuspublicationswho-can-affordpdfShearerJCAbelsonAKouyateBLavisJNampWaltG(2016)WhydopolicieschangeInstitutionsinterestsideasandnetworksinthreecasesofpolicyreformHealthPolicyandPlanning31(9)1200-1211Retrievedfromhttpsdoiorg101093heapolczw052ThelessonsofadministrativedataHigh-profilepolicy-relevantresearchpoweredbyadministrativedata(nd)Retrievedfromhttpswwwpovertyactionlaborgsitesdefaultfilesresources20161114-The-Lessons-of-Administrative-DatapdfToolsandotherneedsforqualitativeandquantitativemethods(2015February)AgencyforHealthcareResearchandQualityRetrievedfromhttpwwwahrqgovprofessionalsquality-patient-safetytalkingqualityassesstoolshtml
WeaverRCampBagchi-SenS(2013)SpatialanalysisofurbandeclineThegeographyofblightAppliedGeography4061-70
References
65
UniversityPartners
Name Title Affiliation Area(s)ofexpertise
Contactinformation
Examplesofworkingwithnon-academicdecisionmakers
WendyPBarreno
AssociateDirector
WilsonSheehanLabforEconomicOpportunitiesUniversityofNotreDame
LEOsfocusareasincludehealtheducationhousingchildwelfareeconomicopportunitycriminaljustice
wbarrenonduduorleondedu
MelissaDavey-RothwellPhD
AssociateScientist
JohnsHopkinsBloombergSchoolofPublicHealth
Health mdavey1jhuedu410-614-5854
BaltimoreCityHealthDepartmentLeadersMarylandDepartmentofHealthAddictionRecoveryServicesinDCExecutiveDirectorsofcommunityorganizationsacrosstheUSthatfocusonHIVSTIsubstanceabuseandimplementationscience
ElizabethStuart
AssociateDeanforEducationandProfessorDepartmentsofMentalHealthBiostatisticsandHealthPolicyand
JohnsHopkinsBloombergSchoolofPublicHealth
Educationsocialpolicystatisticalmethods
estuartjhuedu
Resources
66
Management
Design-ThinkingResources
EngagementGuideforCivicTech
Journeymap
PerformanceManagementResources
DenverPeakAcademy
Proxymeasuresguide
Talkingaboutperformancewithoutcomplicatingit(video)
SettingPerformanceTargetsguide
Datavisualizationguide
Performancemanagementgettingstartedguide
Exercises
Fishboneexercise
5Whys
AdministrativeData
TheLessonsofAdministrativeData(JPAL)
EvaluationResources
Quantitativeandqualitativemethods(USDepartmentofHealthandHumanServicesAgencyforHealthcareResearchandQuality)
KeyStepsinOutcomeManagement(UrbanInstitute)
RCT-YES
Sageresearchmethods
BestPracticesforMixedMethodsResearchintheHealthSciences
ACCORDChecklist
Pilotandimplementationplanning
Benchmarkingexamplesandproxymeasures
Seattleopendataplaybook
Resources
67
Datasciencescopingguide
Resources
68
AboutGovExGovExhelpsgovernmentsusedataandevidencetoimproveresidentsrsquolivesWesupportgovernmentsinbuildingtheirinternalcapacitytooperatesustainableanalyticspracticeswhichimprovecommunityoutcomesinfieldsfrompublicsafetytoeconomicdevelopmentGovExhasaproventrackrecordofworkinginconcertwithpartnerstoprovidetechnicalassistanceandtrainingatscale
SinceourlaunchinApril2015GovExhas
Trainedover2000governmentemployeesinpersonandonlineintopicsincludingdatamanagementperformancemanagementopendatacommunityengagementandanalytics
Providedtechnicalassistancetomorethan100localgovernmentsacrosstheglobeimprovingtheiruseofdatatoachieveefficienciesinservicedeliveryandbetteroutcomesforresidents
Assessedthecapacityofmorethan130governmentalorganizationstodeterminehowleadersstaffandexternalstakeholdersmanageandusedata
Localgovernmentsareincreasinglyonthefrontlinesofaddressinglargescaleproblemswithglobalimplicationssuchasclimatechangepublicsafetyandeconomicinequality
Whileevidence-basedinterventionstohelpsolvethoseproblemsareavailablelocalgovernmentstaffhavelimitedtimeandcapacitytofindandimplementresearch-basedsolutionsAsaresultcitiescontinuetoworkthewaytheyalwayshave--makingincrementalimprovementstoprogramsbutnotalwayscreatingchangeatasystem-level
ThroughresourcedevelopmentandcollaborationwithJHUfacultyGovExissupportingcityleadersinusingdataandevidencetomakedecisionsandcreatinglastingrelationshipswithresearchers
TheRoadmapwasdevelopedbyGovExaspartofitspartnershipwithBloombergPhilanthropiesWhatWorksCitiesinitiative
AboutGovEx
69
- Title page
- Introduction and purpose of the Roadmap
- Using the Roadmap
-
- Gather information about the problem
- Strategically engage key partners
- Review the evidence and collect details
- Decide on a policy to implement based on local factors
- Establish a framework to monitor progress
- Assess data accessibility quality and security
- Design implementation and evaluation
- Share your story and success
-
- Working with a range of evidence
- Thank you
- References
- Resources
- About GovEx
-
havetomakesensetothepeoplewhowillbechargedwithimplementingthemBuildingastrongdiverseprojectteamisimportanttoensurethattheworkdoesnotliveanddiebyoneperson
Identifykeypartners
NextidentifythepeoplewhotheprojectteamcanworkwithontheissueyouarefacingincludingpeoplewithlivedexperienceintheproblemyourcityistryingtosolvedepartmentstaffandsubjectmatterexpertsoutsideofcitygovernmentThisistheconceptbehindbuildwithnotforamovementledbyLaurenellenMcCannthatemphasizescommunityexpertiseandleadershipMappingoutkeyplayersandtheirrolesiscriticaldotoearlyintheinformationgatheringphasesincecoalitionbuildingissuchanintegralpartofthepolicychangeprocessThisprocessmightconsistofstakeholdermapping(describedbelow)communityresourcemappingandculturalmappingInyourmappingremembertoconsiderpotentialalliesandthosewhomaypresentroadblocksalongtheway
PolicymakingisnottopdownorlinearInsomecasescitiesmayhavetosharedecision-makingpowerwithotheractorsForinstanceinmanycitiesthelocalContinuumofCare(CoC)istheleadentityonhomelessnessservicesCoCsapplyforandstewardfederalfundstoaddresshomelessnessandcoordinateservicesamonglocalprovidersAlthoughcitygovernmentsdohavearoleinCoCstheyarenotalwaystheleadagencychargedwithdistributingfundsandthereforetheirroleinhomelessnesspolicymaybelimitedCountygovernmentsoftenhavehealthdepartmentswhilecitiesdonotInthosecasesthebalanceofpowerfavorscountygovernment
TheCityofDentonTXwasrecentlyfacedwithanissueofresidentcomplaintsaboutanewdocklessbikesharethatarrivedintownandisviewedbysomeasexpensivelitterInresponseDentonsCityCouncilwhichwasalreadydiscussingpotentialrulesforbikesharecompanieselevatedthisissueandchargedcitystaffwithidentifyingasolutionPresentlytheCityisreviewingordinancesfromnearbycitiestodeterminehowtheyaresolvingthisproblemandwhatsolutionsmaybefeasibleinDentonThemappingexamplebelowrepresentslocalactorsinDentonwithastakeinthisissuewhohavemultipleperspectivesandlevelsofpower
Gatherinformationabouttheproblem
9
UsethistemplatedevelopedbytheCentreforPublicImpact(CPI)tobrainstormwhoyoumightreachouttoasyouembarkontheroadtopolicychangeRemembertothinkabouthoweachpersonorgrouprelatestotheproblem
BegintodefinethewhatortheprobleminyourCity
TherestofthissectionwillhelpyourefinetheproblemstatementFornowgathersomekeypartnerstoclearlyarticulatewhychangeisneededUsethetemplatebelowdevelopedbyDenverPeakAcademytocapturetheproblemstatementYouwillbuildonthislater
ProblemStatementndashWhyChangeisNeeded(1-2Sentences)
ThroughourinterviewswithcitystaffweheardthatwhendiscussingmunicipalchallengesitcanbetemptingtojumptoasolutionbeforetakingthetimetoreallyunderstandtherootcauseoftheproblemThismakessensesincecityleadersandstaffareresponsiblefordeliveringservicesandbeingresponsivetoresidentswhileaddressingnewproblemsthatariseinthecity
TherootsoftheproblemsthatcitiesareworkingtosolvearecomplexandmaybethelegacyofhistoricalfederalpolicyorsweepingsocialtrendsTohelptheproblemseemlessdauntingitcanhelptobreakthemdownintotherootcausetoidentifyamanageablecomponentoftheproblemUsingafishboneexerciseorfivewhystechniquecanalsohelp
Gatherinformationabouttheproblem
10
tofindtherootcauseoftheproblemandavoidsimplyaddressingsymptomsoftheproblemandrepeatingthemistakesofhistoryToolssuchasjourneymapscanhelpyouunderstandtheexperienceofthosewhoareaffectedbytheproblem
TheCentreforPublicImpactwithsupportfromBloombergPhilanthropiesdevelopedanexercisetosupportcitiesinidentifyingtherightsizedproblemAccordingtoCPIarightsizedproblemisonethatislargeenoughtopotentiallyapplytoothercitiesbutsmallenoughtosolvewithinareasonabletimeframe
AtthebeginningofMayorBerkesadministrationintheCityofChattanoogaTNveteransexperiencinghomelessnesswereaskingtheCityforhelpinfindinghousingTalkingtothispopulationrevealedthatveteranshadtroubleidentifyingaffordablehousingwhichmotivatedtheAdministrationtoworkonasolutionTheproblemofveteransexperiencinghomelessnesswastherightsizebecauseitfocusedononesegmentofpeopleexperiencinghomelessness-veterans-andtheirspecificchallengeinidentifyingaffordablehousing
Developacomprehensiveunderstandingofhowyourcitycurrentlyaddressestheissue
IntheChattanoogaexamplementionedaboveCitystaffdiscoveredthatveteranswerehavingtroublenavigatingtheCitysvariousaffordablehousingprogramsandgettingaccesstothehousingandservicesthattheyneededAtthesametimenewlyelectedMayorBerkewasexploringcurrentandfutureopportunitiestoservethispopulationRatherthancontinuetousefederalfundingforemergencysheltersMayorBerkedirectedcitystafftoprioritize
Gatherinformationabouttheproblem
11
HousingFirstwhichvaluesconnectingpeoplewithpermanenthousingovertransitionalhousingandsupportiveserviceafterspeakingwithrepresentativesattheUSDepartmentofHousingandUrbanDevelopment(HUD)
ThisexaminationofwhattheCityiscurrentlyfundingandthesourcesofthatfundingwasaturningpointinChattanoogaUnderstandingthisproblemfromasystemsperspective-includingpoliciesandfundingstreamsatmultiplelevelsofgovernmentsocialnetworksthroughwhichinformationisoftenspreadtechnologythebuiltenvironmentofmarket-rateandpublichousing-helpedthenewMayoranalyzeexistingeffortsandleadstaffandotheractorsinthesamedirection
Begintogatherthedata
TotrulycapturethesuiteofservicesthatyourCityisprovidingaskdepartmentstafftolisttheprogramsrelatedtoyourproblemstatementthatyourcityrunsorfundsWhenlistingtheprogramscapturedetailssuchas
programnamethenameofthedatasetthatisrelevanttoeachprogramadescriptionofthedatasetthesourceofthedatasetifthedataisinternalorexternalwhetherthisdataisalreadybeingcollected
ThisinformationwillformthebasisofyourinventoryofadministrativedataordatasetsthatarekeptbygovernmentsforoperationalpurposesThismayincludecallcenterrecords(311911)taxdatapropertyassessmentsinspectioninformationsocialservicerecordsetcAtthispointlistallrelevantadministrativedatasetsYouwillprioritizewhichdatasetstogathermoreinformationonlaterwhenyouaredevelopingindicators
Asampletemplatethatcanbeusedtocapturethisinformationisbelow
ProblemStatement____________________________________________
Gatherinformationabouttheproblem
12
ProgramName
DatasetTitle
DatasetDescription DataSource
Departmentorexternalpartner
DoestheCityalreadycollectthisdata
Whatisthenameoftheprogram
Human-readablenameofthedataset
Shouldbeunderstandablebynon-technicalusers
Whatistheoriginalsystemapplicationfilethathousesit
Departmentorexternalpartnerthatmaintainsthedata
Isthisdataalreadycollected
TogatherthisinformationconsiderassigningonepointpersontosurveyorinterviewcitystaffSpeakingwithfrontlineworkersstaffwhoworkwithfederalgrantsdepartmentheadsandeveryoneinbetweenwillensureadeepunderstandingoftherelevantprogramsanddatasetsDuringthecourseoftheseconversationsbesuretoaskcitystaffwhichorganizationstheyworkwithexternallyandcapturethatinformationaswelleveniftheyarenotrecipientsofcityfundingInadditiontoconversationswithexpertsincitygovernmentyourcitysbudgetdocumentsareanothergreatsourceofinformation
Collectdataandobservetrends
Nexttrytogainanunderstandingofwhytheproblemstillexistsifithasbeensomethingyouhaveworkedonforawhileorwhyitisemergingnowifitisnew
AnalyzingadministrativedatacandemonstratepasteffectivenessofinitiativesrelatedtoyourproblemstatementUsingthedatasetinventorythatyoubeganintheprevioussteptalktodepartmentstaffaboutwhichdatasetsaremostrelevanttotheproblemandgatherafewyearsworthofhistoricaldataBegintolookfortrendsinthisdatadiscussingpotentialreasonsforthosetrendswithrelevantstaffCanthetrendsbeexplainedbyseasonalitychangesinfundingstaffingfluctuationsorsomeotherreasonBasedonthehistoricaldataanalysiscreateabaselineagainstwhichfuturechangescanbeobserved
OtherdatasetscanhelpprovideadditionalcontextForinstanceifthenumberofpeopleexperiencinghomelessnessinyourcityisincreasingitmaybehelpfultotakealookathousingaffordabilityIfyoudonothavethedatayouneedconsideraproxymeasureAsanexamplehousingisoftenconsideredaffordableifitcosts30orlessofahouseholdsincomeButifhouseholdincomedataisnotavailableatthecitylevelAreaMedianIncome(AMI)canbeahelpfulproxy
Gatherinformationabouttheproblem
13
QualitativedatacanalsohelpanswerquestionsaroundtheeffectivenessofpreviouseffortsTrytoobtainavarietyofperspectives--bothmainstreamandoutliers--fromnetworksbeyondyourowntoprovideyouwiththegreatestvantagepointforevaluatingpotentialimpactUnderstandinghowthoseeffortswerereceivedbydifferentpartsofyourcommunityandiftheremaybeelementsofprevioussystemstopreservewillbecrucialtothesuccessofanypolicychangeThisinformationmaybeobtainedthroughfocusgroupsinterviewsandsurveysFormoreinformationonhowtousethesequalitativeresearchtoolspleaseseetheseresourcesdevelopedbytheAgencyforHealthCareandResearchandUrbanInstitute
FinallytakealookatnationaldatatoseewhereelsethisproblemmightexistThisresearchmaybehelpfulinlaterstepswhenyouarelookingforpromisingpoliciesinothercities
Puttingitalltogether
UsethetemplatebelowdevelopedbyDenversPeakAcademytocaptureyourrefinedrightsizedproblemstatementandcurrentandfuturestatesWorkingthroughthisexercisewithpartnersishelpfultogainbuy-inTheresponseswillbeimportantwhengettingotherpartnersonboard
WhyChangeIsNeeded
Yourright-sizedproblem
CurrentState
Analysisoftrendsindata
FutureState
Ideallywhataretheoutcomesyouhopetoachieve
Checklist
Thekeystepstogatheringinformationarebelow
Atthispointfocusontheproblemratherthanjumpingtosolutions
Listentopeoplewhoareaskingforhelpandtoadvocateswhoareraisingissues
Beasspecificaspossibleinidentifyingtherootcauseandhoninginonthepartoftheproblemthatyourcitycanaddress
Identifytrendsbasedonmultiplesourcesofdataincludinghistoricaladministrativedataproxymeasuresqualitativedataandnationalmeasures
Gatherinformationabouttheproblem
14
Captureyourproblemstatementtheresultsofyouranalysisanddesiredoutcomes
Gatherinformationabouttheproblem
15
StrategicallyengagekeypartnersPolicytheoryresearchoftenfocusesonincreasingpolicymakersaccesstoevidenceasawaytoavoidtakingshortcutslikemakingdecisionsbasedonemotionImplementationsciencesuggeststhatincreasingaccesstoevidenceisnotenoughandthatpolicychangeshouldbeexaminedinthewidercontextofdebatecoalitionformingandpersuasionMeaningfulpolicychangerequiresldquoalong-termstrategytoformalliancesanddevelopknowledgeofpolicymakersandpolicymakingratherthanusingevidenceinpolicymakingonanadhocorone-offbasis
UnderstandingnetworksandsystemsisimportanttodevelopsustainablepolicychangeThissectionismeanttohelpreaderstakethestepstobuildengagementstrategiesforpartnersthatrepresentdifferentsystemsandnetworksEngagementshouldbeanongoingprocessofconnectingpeopletooneanotherandtoanissueEngagementshouldnotbeinnameonly-wehaveallexperiencedengagementactivitiesthatgivepeopleaspecificroletoplaythatleadstoaneatpredictableoutcomeAccordingtotheBuildWithmovementundervaluingandunderinvestinginengagementstrategiescanleadtoadisconnectofcertainindividualsorgroupsandmistrustofgovernmentWhendevelopingengagementactivitiesrememberthathowpeopleengageandwhethertheyengagehastodowithhowyoustructureopportunitiesforengagement
Conveneacoreadvisorygroup
AnadvisorygroupcomprisedofpeoplefrommultiplesectorscanbehelpfulasyouconsiderthelocalsocialenvironmentalandpoliticalcontextinwhichyouintendtointroducepolicychangeRevisityourstakeholdermapanddeterminewhowouldbebesttoserveonthisgroupTomakesureyouareconsideringtheproblemsandsolutionsthroughanequitylensincludepeoplewithlivedexperiencewhohaveexpertiseintheeffectivenessofsystemswhetherthosesystemsfunctionasintendedandhowtheymayimpacthumanbehaviorYoumayalsowishtoincludesomeonewithexperienceworkingwithdataandanalysis-thispersoncanhelpdeterminethetypeofdatathatmaybereadilyavailableandusefultocollectandcanalsohelpplanandshapeyouranalysistofigureoutwhichprogramsareeffectiveBesuretoinvitefront-lineworkersaswell-itscriticalthatwhateversolutionscomeoutofthisprocessaremeaningfultothosewhowillberesponsibleforimplementingthem
BringingpeopletogethercanbechallengingbutthereareafewwaystosetandmanageexpectationsFirstmakesurethatthegroupisawareoftheproblemstatementcurrentstateandfuturestate-thiscanhelpstarteveryoneoffonthesamepageNextspend
Strategicallyengagekeypartners
16
sometimegettingtoknoweachotherrsquosworktolearnabouteveryonesstrengthsThissharedunderstandingcanhelpasyouclarifyrolesandresponsibilitiesofeachpartnerinthegroup--thisisimportanttoavoidfinger-pointinginthefutureandtoensurethatnoneofthetasksslipthroughthecracksInsomecasestherearemanyactorswithdifferentresponsibilitiesasinthecaseofBaltimoreCitys38-memberYouthFundTaskForcewhichischairedbynonprofitandcorporatepartnersandincludesrepresentativesfromthenonprofitandfaithcommunitiesstudentsthebusinesssectorlocaluniversitiesCityCouncilandtheMayorsOffice
OncerolesandresponsibilitiesareclearyoumayconsiderdevelopingacharterguidelinesortermsofreferencetosetexpectationsforthegroupBesuretodistinguishthisfromotherboardsandcommissionsthatyourcitymayhaveNextyoumaywishtoestablisharegularongoingmeetingscheduleBesuretosendagendasandmeetingmaterialsaheadoftimeandclarifytheobjectiveofeachmeeting
InSeattletheCitydevelopeditsHousingAffordabilityandLivabilityAgenda(HALA)torespondtorisinghousingcostsandhomelessnessThisisatrendinotherwestcoastcitiesaswellwhichmaybeexplainedbyaninfluxofnewresidentswhoareattractedbytheemergenceoftechjobstheattractionoftheoutdoorlifestyleortherecentpassageofprogressivepolicieslikelegalizedgaymarriagea$15minimumwageandlegalizationofmarijuanaSeattlebroughttogetheramulti-stakeholderadvisorygroupcomprisedoflocalhousingexpertsrentersandhomeownersdevelopersandgovernmentofficialstoshareinformationabouttheirneedsanddeveloprecommendationsbasedontheseneedsTheinterestsandmotivationsofthisgroupwerevariedandittookabouttenmonthstofindcommongroundTheseriesofrecommendationsthattheadvisorygroupcreatedismeanttoserveresidentswithaspectrumofneeds
Buildaprocessforcommunityparticipation
Ofcoursenoteveryonecanbeincludedinanadvisorygroupandadvisorygroupmembersmaynotnecessarilyrepresentthevariedperspectivesoftheirpeers-forinstancenoteveryhomeownerhasthesamesetofneedsTogatherinputfromawiderswathofthepopulationitcanbehelpfultobuildaprocesstoinvitethepublicatlargetosharetheirideasCitiesdothisinavarietyofwaysfromonlineconversationsandsurveystoin-personmeetingsRememberthatbeingintentionalaboutgatheringtheviewpointsofpeopleismoreimportantthanusingthelatesttechnologyandtools
TheinsightgeneratedbycommunityparticipationexercisescanbeincludedinyourreviewofevidencewhichisdiscussedinasubsequentchapterInWestVirginiaastatethathasbeenhithardbytheopioidepidemicstatehealthofficialsdraftedanopioidresponseplanthatwasinformedbypublicengagementaswellaslocalandregionalsubjectmatter
Strategicallyengagekeypartners
17
expertsThepublicwasinvitedtopresentcommentsonideasfordecreasingoverdoseearlyinterventiontreatmentandrecoveryPublichealthandmedicalfacultyfromWestVirginiaUniversityMarshallUniversityandJohnsHopkinsUniversityprovidedresponsestothecommentsTheresultwasanOpioidResponsePlanthathealthcommissionerRahulGuptapresentedtoGovernorJimJustice
AsamplespectrumtoplanyourowncommunityparticipationprocesscreatedbyLaurenellenMcCannisbelow
SourceLaurenellenMcCann
Community-basedparticipatoryresearchisanotherwaytoworkwiththecommunitytoimproveoutcomesCBPRmeansthecommunityisinvolvedinallaspectsoftheresearchprocesswhichisofteniterativeInCBPRtraditionalresearchersandcommunitymembersshareknowledgeaswellasresourcesandcredit
CBPRInActionInSanFranciscoanenvironmentaljusticenonprofitteamedupwiththeCitysDepartmentofPublicHealthandanoutsideevaluatortounderstandtherootcausesofobesityandfoodinsecurityinlow-incomeneighborhoodsandidentifysolutionsThroughresearchmethodssuchasinterviewingresidentsandstoreownersdiagrammingstoreshelvesGISmappinganeconomicanalysisandreviewofsimilarpoliciesinnearbycitiestheteamobservedthatthestoresthatresidentshasveaccesstowerestockedwithlesshealthyfoodsandthatresidentswantedmorehealthyfoodandlessalcoholandtobaccoThisprojectresultedinlocalandstatelegislationtoencouragestorestostockfreshproduce
Strategicallyengagekeypartners
18
Developrelationshipswithsubjectmatterexperts
OnewaytokeepsubjectmatterexpertsengagedistoinvolvetheminthecommunityparticipationprocessasdescribedaboveIntheWestVirginiaexamplesubjectmatterexpertsrespondedtoandsynthesizedthepubliccommentsOtheropportunitiesaredescribedbelow
AdvisoryAcitydepartmentheadMayororstaffmembermaydevelopaone-on-oneadvisoryrelationshipwithuniversityfacultymemberoracollaborativebetweentheCityandUniversitytoaddressaparticularproblemAgreatplacetostartistolookforrelevantpracticecenterswithinyourlocaluniversityandreachingouttosubjectmatterexpertsInadditiontotheirexpertiseontherelevantpolicytopicheorshemayalsobeabletoidentifyotherexpertsandevenprovidesupportonhiringtherighttalenttoleadcertainprogramsAnadvisoryrelationshipwithauniversitysubjectmatterexpertcanalsohelpwhencitiesareconsideringbringinginnewprogramsTheseexpertsareoftenfamiliarwithprogramsthatarebasedonnationalmodelsandcanprovideinsightintoaprogramsoriginandlineageeffectivenessinotherplacesandthepotentialforimplementationinyourcityAcademicscanalsohelpidentifyindicatorsbasedongovernmentindicatorsorliteratureFinallyrelationshipsbetweenpolicymakersandresearchersprovidethemostbroadlyeffectiveroutetopullingresearchintopolicy
ResearchandevaluationAcademicexpertsmaysupportcitiesinevaluation-indeedevaluatingcityinitiativesisatraditionalmodelofcityuniversityrelationshipsSharingtheinformationyoucollectedintheGatherInformationphasecanhelpresearchersunderstandyourworkandgoalswhichinturnwillstrengthenthesupportthattheycanprovideDiscussingdetailsaboutyourinterventionandevaluationplanatthebeginningoftheprocessmeansthataresearchpartnercanhelpwiththeevaluationdesignanddatacollectionForinstancetheWilsonSheehanLabforEconomicOpportunitieswalkscitiesthroughconsideringthefollowingquestionsbeforepartneringonanevaluation
1 Isthereaclearpolicyprogrammodelwithobjectivescreeningcriteria
2 Isitpossibletoevaluatewhathappensbothwiththepolicyprogramandintheabsenceofthepolicyprogram
3 Arethereoneormoremeasurableoutcomesofinterest
4 Arethereenoughpeopleforrigorousquantitativeanalysis
5 Whatdatawillbenecessaryforthestudyandwhataretheprocessesforaccessingthesedatasource(s)
6 Isthereacompellingreasontoevaluate
Strategicallyengagekeypartners
19
InadditiontoassessingreadinesstoconductanevaluationresearcherscanhelpyouhoneinonthepurposeoftheevaluationTheWorldBankdescribesseveralpotentialusesofevaluationwhicharedefinedhereInbriefevaluationcanhelpdecisionmakers
makeresourceallocationdecisionsrethinkrootcausesofproblemsidentifynewproblemsdecideonabestalternativedeterminewhetherprogramsorpoliciesarebeingimplementedwithfidelitydeterminewhetherprogramsorpoliciesarehavingthedesiredimpact
Researcherscanalsoprovidesupportinsharingtheresultswiththescientificcommunitythroughpapersandconferenceswhichisvaluableincreatingapeernetworkofevidence-basedcitypractitioners
TrainingandtechnicalassistanceUniversitypartnersespeciallythoseatpracticecenterscanbehelpfulindevelopingorupdatingtrainingcurriculumtrainingthetraineranddeliveringtrainingtocitystaff
Strategicallyengagekeypartners
20
ReviewtheevidenceandcollectdetailsOnceyouhaveathoroughunderstandingoftheproblemdataandactorsyouwillneedtoconductyourownresearchonavailableevidenceandlearnaboutthelandscapeofpolicyoptionsYourgoalwillbetodevelopanaccessiblelistofinformationaboutpoliciesthathavehelpedsolvesimilarproblemsinotherplacesThischapterwillguideyouthroughthestepsofpreparingyourreviewersunderstandinglevelsofevidenceandcollectingthedetailsonevidencethatyouwillneedfordiscussionwithdecisionmakersinyourcity
Selectandprepareyourreviewers
AssignapersonorteamtoreviewrelevantinterventionsthathavedemonstratedsuccessinotherjurisdictionsTheirgoalwillbetoproduceareviewofalloftherelevantprogramspoliciesorinitiativesthatyourteamcouldconsiderimplementingTheywillbeconductingaliteraturereviewsupplementedbyinterviewswithexpertsandpeerswhohaveinformationaboutpotentialprograms
SincethepeopleyouassigntothistaskareprovidingthebaselineofpotentialoptionstoimplementitisimportantthatyouchoosepeopleyouareconfidentwillbethoughtfulandthoroughintheirworkItisalsocriticalthattheyareawareoftheimportanceofsourcecredibilityEnsureyourreviewershavefamiliarizedthemselveswithrulesforcriticallyanalyzinginformationsourcesbeforetheygetstartedIftheyareincludingnewsarticlesaspartoftheirsearchtheyshouldalsobeawareofmethodsforevaluatingnewscredibility
AskyourreviewerstodevelopaplanandtimelinetocompletethereviewThisstartswithsolidifyingagoverningresearchquestionOnewaytoknowyouarecreatingadetailedresearchquestionistousetheacronymPICOremindingyoutoincludereferencetothepopulationintervention(ifyouhaveoneinmind)comparisonandoutcomeelementsinyourresearchquestionForexampleifyouknowthatyouhaveapopulationofteenagerswhoarehomelessyouhavelearnedfromyourstakeholdersthatmanyoftheseteenagersidentifyasLGBTandyouhaveheardsomethingabouttheapproachofcreatingdrop-incentersfordeliveringsupportyoucouldask
ForLGBTyouthfacinghomelessness(population)howeffectivearedrop-incenters(intervention)relativetostandardsocialservices(comparison)atsettlingtheminpermanenthousing(outcome)
Ifyoudonothaveaninterventioninmindyoucanleaveouttheinterventionandcomparisonandsimplyask
Reviewtheevidenceandcollectdetails
21
HowcanwehelpLGBTyouthfacinghomelessness(population)obtainpermanenthousing(outcome)
OnceyourreviewershavetheirresearchquestionsusethequestionstocreatealistofkeywordstheycanuseintheirliteraturesearchThiswillhelpkeeptheircollectionofarticlesreasonablyfocusedHavethemdescribewhattheirsearchprocesswillbeincludinglistingthedatabasestheywillreviewandtheirplanforstoringtheinformationtheywillcollect(Herearesomeadditionalsuggestionsforstructuringareviewplan)
WhatyourreviewerswillbedoingisalightweightversionofarapidreviewTotheextentthattheywouldliketodevelopamethodologyforspecificsearchprotocolsincludingcriticalappraisalprocessestheycancertainlygoaheadanddoso
Usedifferentlevelsofevidenceforthemostcompletepicture
AsdiscussedintheWorkingwitharangeofevidencechaptertheNestaStandardsofEvidenceframework--reproducedbelow--ishelpfultounderstandthelevelofconfidencethataprogramorinterventionishavingapositiveimpactThisapproachletsyouputinterventionsthatyouhaveheardaboutlocallybutwhichhavenotbeenstudiedtogetherwiththosewithademonstratedcausaleffectThisishelpfulsinceanumberofpoliciesprogramsandpracticesthatyouwillwanttoconsidermaynotyethavemuchofaresearchrecord
Reviewtheevidenceandcollectdetails
22
NearlyanypieceofinformationaboutaninterventioncanbeclassifiedwithinthestandardsofevidenceForexampleifsomeonehasacrediblehypothesisabouthowaninterventioncouldproducebenefitthistypeofevidencewouldbeclassifiedasLevel1ThistypeofevidencecouldincludeinterventionsinothercitiesthatyoureadaboutinthenewsorhearaboutataconferenceForinstancewhenUSDAallowedschoolsandschooldistrictstoprovidefreebreakfastandlunchtoallstudentsthroughtheCommunityEligibilityProvision(CEP)manydistrictsimplementedthisoptionevenbeforetherewasevidenceofeffectivenessThiswouldfitintoLevel1sinceitbuildsonthelogicthatprovidingaccesstomealswilldecreasefoodinsecurity
YoumayalsofindpiecesofinformationthatqualifyasLevel2evidenceForinstanceasmoreschooldistrictsimplementedCEPprincipalstestifiedaboutpositivechangestheyobservedinstudentssuchasamorepositivedemeanorthatcanleadtobetteracademicperformanceDataalsoshowedthatthestudentsdescribedwerereceivingmoremealsthanbeforetheprogramsimplementationHoweveritwastoosoontobuildevidencetoshowacausalrelationshipbetweentheincreaseinfreemealsandincreasedacademicperformance
ldquoLower-levelrdquoevidencecanconsistofnuancedbutvaluableelementsthatrelatetoaninterventionrsquospotentialfitwithacommunityQualitativeinformationsuchastestimonyfrompeoplewhohaveexperiencedaninterventionandthosewhohavefacilitatedorimplementeditwillalwaysbeusefultoconsiderandprovidesdetailedlocalcontext
Experimentalstudies--describedstartingatLevel3withcontrolledexperimentsandgoingthroughLevel5withfullyfleshedoutprotocolsandprocessesforreplication--stilloccupythehigherlevelsofthisdescriptionofevidenceThisprioritizationshowsthevalueofrigorousstudyforevaluatingtheimpactofaparticularpolicyorprogramratherthanotherfactorsthatcouldbeinfluencingoutcomesofinterestRememberthatmultipletypesofevidencecancomplementrigorousimpactevaluationsandgivecityleadersadditionalinsightsandbroaderperspective
Accessinformation
Encourageyourreviewerstobegintheirsearchforpromisingpracticesingovernmentandnon-profitclearinghousesandinacademicpolicyjournalsWorkpublishedinthesesourceshasalreadybeensubjectedtoahighlevelofprofessionalscrutinyandgenerallycontainsagreatdealofdetailwhichmeansthatyourreviewersaremorelikelytobeabletopullouttheinformationyouneedtomakeagooddecision
Clearinghouses
Reviewtheevidenceandcollectdetails
23
Herearesomeoftheclearinghouses(collectionsofprogramevaluations)forevidence-basedpoliciesandprograms
ResultsFirstClearinghouseDatabaseThePewCharitableTrustsCollectstogethereightotherclearinghousestoprovideinformationoninterventionsinthefollowingcategoriesadultcriminaljusticeeducationchildwelfarehealthjuvenilejusticementalhealthsubstanceabuseandyouthdevelopmentIndividualclearinghousesincludedinResultsFirstare
BlueprintsforHealthyYouthDevelopment-foryouthdevelopmentandviolenceprevention
CaliforniaEvidence-BasedClearinghouse-forchildwelfareinterventions
TheLauraandJohnArnoldFoundationrsquosSocialProgramsThatWork-forsocialservicesprograms
TheUSDepartmentofEducationrsquosWhatWorksClearinghouse-foreducationinterventions
TheUSDepartmentofHealthandHumanServicesrsquoResearch-TestedInterventionPrograms(RTIPs)-forcancercontrolinterventions
TheUSDepartmentofHealthandHumanServicesrsquoTeenPregnancyPreventionEvidenceReview(TPPEvidenceReview)-forsexualriskbehaviorandsexualhealthoutcomes
TheUSDepartmentofJusticersquosCrimeSolutionsgov-forcriminaljusticejuvenilejusticeandcrimevictimservices
TheUSSubstanceAbuseandMentalHealthServicesAdministrationrsquosNationalRegistryofEvidence-basedProgramsandPractices(NREPP)-substanceuseandmentalhealthinterventions
TheUniversityofWisconsinPopulationHealthInstituteandRobertWoodJohnsonFoundationrsquosCountyHealthRankingsandRoadmapsWhatWorksforHealth(WhatWorksforHealth)-healthbehaviorsclinicalcaresocialandeconomicfactorsandthephysicalenvironment
UKWhatWorksCentresTheBritishgovernmenthascharteredanetworkofsubject-specificorganizationswhichcollectsynthesizeandreportoutevidenceontheeffectivenessofpolicyprogramsandpractices
NationalInstituteforHealthandCareExcellence(NICE)-healthandsocialcare
Reviewtheevidenceandcollectdetails
24
SuttonTrustEducationalEndowmentFoundation-educationalachievement
CollegeofPolicingWhatWorksCentreforCrimeReduction-crimereduction
EarlyInterventionFoundation-earlyintervention
WhatWorksCentreforLocalEconomicGrowth-localeconomicgrowth
CentreforAgeingBetter-qualityoflifeforolderpeople
WhatWorksCentreforWellbeing-wellbeing
WilsonSheehanLabforEconomicOpportunities(LEO)UniversityofNotreDameProvideslinkstotheirevaluationsofinterventionsincriminaljusticeself-sufficiencyeducationhealthandhousingandhomelessness
ClearinghouseforLaborEvaluationandResearchUSDepartmentofLaborProvidesresearchonworkforceandlabor-relatedpoliciesandprograms
J-PALEvaluationsAbdulLatifJameelPovertyActionLabStudiesonpoverty-reducinginterventionsfromaroundtheworld
CampbellSystematicReviewsCampbellCollaborationSystematicreviewofpolicyinterventionsbyscholarlycoordinatinggroupsinsevenissueareascrimeandjusticeeducationinternationaldevelopmentmethodssocialwelfareknowledgetranslationandimplementationanddisability
Benefit-CostResultsWashingtonStateInstituteforPublicPolicyAWashingtonState-charteredprogramtoestablishthelikelihoodofrelativecostsandbenefitsforprogramsacrossthefollowingareasjuvenilejusticeadultcriminaljusticechildwelfarepre-Kto12educationchildrensmentalhealthhealthcaresubstanceusedisordersadultmentalhealthpublichealthandPreventionworkforcedevelopmentandhighereducation
EffectiveInterventionsCentersforDiseaseControlandPreventionRepositoryforevidence-informedHIVpreventioncareinterventions(withstepsonhowtoimplement)
Journalsandindexes
OnceyouhavereviewedrelevantinformationaccessiblethroughclearinghousesitisworthtakingalookatindividualstudiespublishedasscholarlyarticlesYourreviewerswilldothisbysearchingarticleindexesandacademicjournalsUnfortunatelyanumberofusefulacademicjournalsandindexingplatformsmaynotbegenerallyaccessibletoyourreviewersifyoudonothaveasubscriptiontothemToaddressthisconsiderpartneringwithalocal
Reviewtheevidenceandcollectdetails
25
collegeoruniversitytoprovideyouwithalistofarticlesfromthefollowingsources(Thisisalsoanexcellentopportunitytoseekresearchsupportfromanyacademicstakeholdersyouidentifiedpreviously)
Indexes
GoogleScholarincludesmostpeer-reviewedonlineacademicjournalsandbooksconferencepapersthesesanddissertationspreprintsabstractstechnicalreportsandotherscholarlyliteratureincludingcourtopinionsandpatentsItallowsyoutospecifyapublicationrangeinyoursearchandprovideslinkstolikelysourcesforalllistedarticles
PolicyFileIndexisauniqueresourceforUSpublicpolicyresearchinthatitgrantsusersaccesstotimelyupdatedinformationfromover350publicpolicythinktanksnongovernmentalorganizationsresearchinstitutesuniversitycentersadvocacygroupsandotherentities
PolicyArchiveworkstocollectanddisseminatesummariesandfulltextsvideosreportsbriefsandmultimediamaterialofthinktankuniversitygovernmentandfoundation-fundedpolicyresearch
Anothersourceforhigh-qualityresearchisthereportsoftheCongressionalResearchServiceYoucanfindtheirreportsaggregatedhere
Specificjournals
Oftenthemostusefulacademicjournalsforunearthingpolicyorprogramreviewswillbeineachspecificsubjectarea--thatisineducationjournalsforeducationpoliciesYoucanuseageneralindexlikeGoogleScholartobegintolearnwhichjournalsaremostoftenusefulforyoursearchHoweveryoucanalsosearchthefollowingtwogeneralistpolicypublicationsforissuesandarticlesrelatedtoyourtopic
PublicAdministrationReviewJournalofPolicyAnalysisandManagement
Newssourcessubjectmatterexpertsandinterviews
AlthoughitisvaluabletoknowwhathasbeenexploredthroughascholarlyresearchframeworkitisalsousefultolearnwhatkindofinterventionsarebeingtriedinthefieldrightnowandhowpeoplefeelaboutthesetrialsSearchsubscriptionnewsaggregatorslikeLexisNexisorfreenewsaggregatorslikeGoogleNewsorYahooNewsforyourissueareaplusthewordpolicyorprogramtoseewhatkindsofinterventionsarebeingcoveredinthenews
Reviewtheevidenceandcollectdetails
26
DonotforgettocontacttheadvisorygroupmemberswhowerechosenfortheirsubjectmatterexpertiseInadditiontoprovidinggoodmaterialthroughinterviewsthemselvestheymayhaveideasaboutothersubjectmatterexpertsyoushouldcontacttoaskaboutpotentialinterventions
Peercitiesandnetworks
InadditiontoreadingabouttheworkofothercitiesitisidealifyouareabletospeakwithpeoplewhoareinvolvedwithcurrentimplementationsofinterestinginterventionsInyourcityyoulikelyhaverelationshipswithyourneighborsbutitcanbehelpfultolookoutsideofyourregionforsolutionsaswellTofindcounterpartsinothercitieswhoareworkingonaprogramyouwouldliketolearnmoreaboutyoucancontactstaffinintermediaryorganizations(likeGovExoryourprofessionalassociations)whomightbeinapositiontoputyouintouchimmediatelyorwhomightseeanothergoodrouteforsettingupaconversationHereisalistofnetworksthatmaybehelpful
InternationalCityCountyManagementAssociation(ICMA)NationalAssociationofCounties(NACO)NationalLeagueofCitiesUSConferenceofMayorsWhatWorksCities
Collectthedetails
Onceyourreviewershavefoundevidenceonpossiblepoliciesorprogramstoimplementtheyshouldcapturekeydetailsthatwillhelpdecisionmakersunderstandthepotentialeffectivenessandapplicabilityofeachpotentialintervention
Foreachsourcethathasprovidedinformationyouwanttoconveyusethetemplatebelowtocaptureasmanyofthefollowingdetailsaspossible(Youdonotneedtocompletethisforsourcesthatprovideduplicativeinformationorareotherwisenotthatuseful)
Programorpolicyname (egldquoHousingFirstrdquoorldquoblightedpropertiesregistryrdquo)
Sourcetype (egjournalarticleinterviewsubjectmatterexpertstudyassociationcommunityparticipationprocess)
Sourcedetails (egarticletitleandpublicationnameandtitleofinterviewee)
Evaluationmethodology
(egrandomizedevaluationcomparativeevaluationoftwodifferentgroupsbeforeafteranalysis)
Evidencelevel (egLevel12345)
Programorpolicy
Reviewtheevidenceandcollectdetails
27
description
Programorpolicygoal (1sentencecapturingtheprincipalaimoftheintervention)
Existingimplementations
(WherehastheinterventionbeentriedListasmanysitesasyouareawareof)
Characteristicsofimplementationsites
(ForeachimplementationsiteofferbasicgeographicIDmdashcityandstateThenyouwillwanttocollectthedetailswhicharerelevanttoyourdecisionmakersrsquoconsiderationsaboutwhatmakesalocationldquosimilartordquoorldquodifferentfromrdquoyoursituationYouwillatleastwanttoincludeforeachimplementationsitePopulationnumberPopulationdensity(ruralurbansuburban)Socio-economiccharacteristicsBeforestartingaskyourdecisionmakersaboutanyparticularcharacteristicsofprogramimplementationsitestheywouldwanttoknowbeforeseekingtoreplicate)
Characteristicsofimplementationpartners
(Anyavailabledetailsaboutorganizationsinstitutionsorpeoplewhowereinvolvedwithimplementingtheintervention-egindependentchildcarecenterpublicelementaryschoolcitypublichealthnurses)
Characteristicsoftargetpopulationeligibleprogramparticipants
(Anyavailabledetailsaboutpeoplewhoarethebeneficiariesoftheprogramorotherwiseincludedoraffectedbytheintervention-egnumberofprogramparticipantscategoriesofpropertyownersaffected)
Policyorprogramduration
(HowlonghastheinterventionbeeninplaceineachimplementationsiteHasitendedandifsowhy)
Whatbenefitscamefromeachimplementation
(Anyobservedpositiveeffectsoftheinterventionincludingeffectsize)
Statisticalestimatesofinterventioneffectiveness
(Anyavailablestatisticalestimatesoftheinterventionrsquoseffects)
Whatdrawbackswereobserved
(Anyavailabledetailsaboutunintendedorunavoidablenegativeeffectsoftheintervention)
Programorpolicycosts
(Anyavailabledetailsaboutprogramcostsperimplementation)
Reviewtheevidenceandcollectdetails
28
Reviewtheevidenceandcollectdetails
29
DecideonapolicytoimplementbasedonlocalfactorsTogetthecriticalbuy-infromleadershiptoimplementanewpolicyorprogramtheprojectleadshouldscheduleameetingwithdecisionmakerstopresenttheevidencesurroundingthepolicyoptionscapturedinthetemplateDependingonyourinsightintoyourcityleadershipswillingnesstomakechangeyoumaywishtosharetheresearchyouhaveconductedontherootcausesoftheproblemandsomeofyouranalysisontheimpactofcurrenteffortstodate
Considertheapplicabilityofanapproach
AsdiscussedinthepreviouschapterpolicymakersoftenlookforevidenceofeffectiveprogramsinothercitiesbutoftenthequestionarisesofhowgeneralizableaprogramisfromonecitytoanotherTheAbdulLatifJameelPovertyActionLab(J-PAL)suggeststhatinreviewingevidencefromanotherlocationitishelpfultoidentifythespecificmechanismthatwasused(deliveryofprogramorserviceinformationsharingetc)identifythelocalconditionsthatwereinplaceobservethebehaviorchangethatresultedfromthemechanismandfinallytounderstandlocalcontextandconditions(legalorcostconstraintsculturalnormsetc)Thissetofpresentationelementscanbedrawnfromthelistofdetailscollectedintheresearchtemplate
EducationpolicymakersinChicagoILusedfindingsfromprogramsinKenyaandIndiainwhichchildrenwhohadfallenbehindtheirpeersweregiveneducationalsupporttargetedtotheirleveloflearningThemechanismusedwastargetedteachingandthereweresimilarconditionspresentinallthreecountriesAlthoughsomestudentshadfallenbehindinthecurriculumteacherswereincentivizedtoteachgrade-levelmaterialsConsiderationsofhowthelocalcontextinChicagomightaffectimplementationofatargetedteachingprogramincludedunderstandinghowtheneighborhoodsthatthestudentswerefrommayhaveaffectedtheireducationaloutcomesandthehighcostofteachinginsmallgroupsInChicagoMatchEducationprovidedtwo-on-onetutoringwhichmatchedstudentswithwell-educatedindividualswhohavenothadformalteachertrainingforalower-costmodelTheresultswerepositiveStudentswhoweretutoredperformedbetterinmathandweremorelikelytopasshighschoolclasses
ThegeneralizabilityframeworkproposedbyJ-PALisbelowUsethisasatemplateasyouconsiderpotentialpolicysolutions
Decideonapolicytoimplementbasedonlocalfactors
30
SourceTheGeneralizabilityPuzzle
OnceyouhavenarrowedthepolicyoptionsdiscussthequestionsbelowforeachoptionsyouareconsideringThisstepiscriticalasunderstandinglocalcontextissoimportantinsuccessfulpolicyimplementation
CommunityContextQuestionnaire
Localcontext
DoesthisoptionmakesenseinourcommunityIsthisaonesizefitsallapproachIsanyparticulargroupabsentfromthisapproachIfsoisthatok
Youranswers
Effectiveness
WhatwouldweexpecttoobserveifthepilotisaseffectivehereasinCityX
Youranswers
Timing
DoesthetimingmakesenseIsthistherighttimetoimplementthispolicychange
Decideonapolicytoimplementbasedonlocalfactors
31
WherearethepotentialroadblocksthatcouldaffectthescheduleIsthereanywaytomitigatethisHowlongwouldittaketoseeanimpact
Youranswers
Implementation
DowehavethestaffcapacitytoimplementthisArethereadditionalpartnersthatyoucanbringtothetabletosupportimplementationDoesthismakesensegivenourformofgovernment
Youranswers
Authority
DoesourcityhavetheauthoritytoenactthisIfsowhatistheappropriatemechanismAretherelegalbarriersArethereanyexistingpreemptionissuesfromthestateorfederalgovernmentCanthosebeaddressedthroughlegislationadvocacyIstherepoliticalsupportoroppositiontothis
Youranswers
Cost
HowmuchwillthiscostannuallyHowwillwepayforitCanthepolicybeimplementedwithexistingfunding
IfsoisthisfundingreliableHowwouldthispolicyoptionaffectourcitysbudgetintheshortandlongtermWhatotherfundingoptionscanweconsiderIsthereanalternativethatwouldcostlessandgiveussimilarresultsWillanyconstituencybeadverselyaffectediffundingforprogramsthatimpactthemisshiftedtoanotherprogrampolicyorinitiative
Youranswers
Decideonapolicytoimplementbasedonlocalfactors
32
EstablishaframeworktomonitorprogressNowthatyouhavereviewedtheevidenceconsideredlocalcontextandchosenyourpolicyitistimetodeterminehowyourcitywillmonitorprogressofthisnewapproachCitystaffandleadersneedtoestablishtheinfrastructureforongoingcontinuousdecisionmakingbasedonwhetherthepolicychangeisleadingtothedesiredoutcomesforresidentsandwhetherthechangeismeaningfulandrationaltofrontlineworkersSettingarbitrarygoalsatthelocallevelthatconflictwithgoalsatotherlevelsisconfusingtotheindividualsresponsibleforachievingthegoalsGoalsthataresettoohightoachievecanbedemoralizingtostaff
ThissectionincludesstepstosetupavenueinwhichCitystaffandleadershiproutinelyvisitkeyindicatorstodeterminewhatisorisnotworkingandtomakedecisionsinreal-timeFormorereadingonperformancemanagementpleasevisittheGovExPerformanceManagementGettingStartedGuide
InperformancemanagementtermssuchasindicatormeasureandmetricmaybeusedinterchangeablyInthisguidewewilluseindicatorInyourworkthemostimportantthingistobeconsistentinwhichevertermyouuseThisvideomaybehelpfulinunderstandingthisconceptandcommunicatingittoothers
Establishthefundamentalsandgatheryourteam
FirstdeterminewhetherthereisanestablishedvenueorregularmeetingscheduleduringwhichyoucandiscussprogressIfyourcityhasanexistingperformancemanagementstructureonarelatedtopicaddingadiscussionoftheprogressofpolicychangetoyouragendacanbeagreatideaIfthiswillbeastandalonemeetingdecidehowfrequentlytheprojectteamanddecisionmakerswillmeetandputthosemeetingsonthecalendarfortheentirelengthoftheproject
DeterminewhowillstaffandattendroutinemeetingsonCityprogressThesemeetingsmayincludetheMayorandorCityManagerdepartmentheadstaffwhoareimplementingthepolicychangeandanalysts
Identifyindicators
ItiscriticalthatindicatorsbeanimportantcityandcommunitypriorityaswellassomethingtheCityplaysamajorroleinaddressingTheremaybeinstancesinwhichtheCitywillneedtoreachouttoorcollaboratewithexternalpartnerstogetamorecompletepictureofprogressontheoverallpolicychange
Establishaframeworktomonitorprogress
33
Anoutcomeindicatordescribesthedesiredeffectofthetargetedpopulation
ExampleBellevuesfamilyfriendlyneighborhoodsindicatormeasuresthepercentofresidentswhobelievetheyliveinaneighborhoodthatsupportsfamiliesparticularlythosewithchildren
WithyourkeypartnersrevisityourproblemstatementcurrentstateandfuturestateexerciseandbegintoidentifytheoutcomeindicatorsthatwillillustratesuccessFornowchooseasingletop-levelkeyperformanceindicator(KPI)ItiscriticalthattheKPI-thepieceyouwillmeasure-beapriorityforthecommunityandsomethingtheCityplaysamajorroleinaddressing
Outputindicatorsmeasuretheprocessofprogramorservicedelivery
ExampleSanDiegoMayorKevinFaulconersgoaltorepair1000milesofstreetby2020
ReviewingtheadministrativedatasetinventorybegintoparedowntherelevantprogramsanddatasetsthatsupportyourKPIWhatarethemeasuresofeffectivenessforeachprogramTheseareyouroutputindicators
Thediagrambelowillustratestherelationshipbetweenoutputandoutcomeindicators
ThinkaboutimplementationindicatorsasmeasuringstepsintheprogramsrolloutthatarenecessarytoobtaindesiredchangesImplementationoutcomescanfocusontheinterventionstaffdeliveringtheinterventionanorganizationorsystemandcanaddressfeasibilityadaptabilitysustainabilityandcostTwokeyquestionstoconsiderwhendevelopingimplementationindicatorsare
1 Istheinterventionacceptableandappropriatetothecommunityand
2 IsitfeasibletoimplementinterventionXatthistimewiththeseresources
Establishaframeworktomonitorprogress
34
Amoreextensivelistwithdefinitionsofpotentialimplementationoutcomesislinkedhere
UsethetablebelowtochoosethedesiredimplementationoutcomesforyourpilotandtoplanyourmeasurementtechniqueThisshouldincludelevelofanalysis(individualproviderorconsumerorganizationorsetting)yourmeasurementtechnique(surveysobservationinterviewfocusgroupsadministrativedataanalysis)andthephasewhenyouwillcollectthisinformation(earlymidlate)Theremaybemorethanonelevelofanalysismeasurementtechniqueandphaseofdatacollection
Implementationoutcome
Definitionofoutcome
Levelofanalysisandmeasurement
Yourmeasurementtechnique
Phaseofpilotwhenyouwillcollectdata
Usethespacebelowtocaptureyourindicatorwishlist
Indicatorwishlist
Outcomes_____________________________________________________________________________
Outputs________________________________________________________________________________
Implementationoutcomes______________________________________________________________
Setapublichigh-levelgoal
Thehigh-levelgoaliscriticalinestablishingdirectiontogalvanizethepeoplewhowillbedoingtheworkWhendiscussingthegoalusetheMayorsbullypulpitLookforkeymilestonesinCitygovernmenttosharethegoalstatementsuchasthebeginningofaMayoraladministrationaStateoftheCityspeechorotherpublicremarks
GoalsshouldbeldquoSMARTrdquo
Specificaspossible(innamingthepopulationwhowillbebetterofftheactionthecitywilltake)
Measurable
Achievable
Establishaframeworktomonitorprogress
35
Results-focused
Time-bound
ThegoalstatementshouldincludeyouroutcomeindicatorlikeChattanoogasgoalldquoWewillendchronicveteranhomelessnessbyDecember2016rdquoAdiagramtohepyoubuildaSMARTgoalaroundyouroutcomeindicatorisbelow
Component Description Yourturn
Specific WhowillbebetteroffWhatactionthecitywilltake
Measurable Howwillyoumeasuresuccess
Achievable Isthegoalattainable
Results-focused Whatistheexpectedresult
Time-bound When
Setrealisticambitioustargetsandtimelinesforeachindicator
ToestablisharealisticandambitioustargetforyourgoalstatementusetheadministrativedatacollectedforyourchosenKPItoreviewactualCityperformanceinthemostrecentyearcomparedwiththethreeorfourmostrecentyearsworthofdataBegintocalculatesummarystatistics(egtotalminimumaveragemedianmaximum)toidentifytrendsCreatecumulativeandmonthlychartstoidentifytrendsandanomaliesBasedonyouranalysisofhistoricaldataignoretheanomaliesandassumetheproductivitytrendswillfollowthesameoverallpatternfrommonth-to-monthFinallyapplyanincreasedproductivityfactortotheperformanceeachmonthtoforecastfutureprogressBasedonthisanalysischooseatargetthatinspiresimprovementwhileaccountingforreality
Manycitiesthataremorematureintheirperformancemanagementpracticesrecognizetheimportanceoftakingacommunity-levelapproachtoimprovingoutcomesforresidentsThesecitiesarecreatinggoaltaxonomiesthatincludemeasuresrelatedtopublicsafetyhousingandevenrecreationCitiesdriveprogressonthesegoalsbyimplementingstrategiesthatengagemultipledepartmentsandcollaboratewithserviceprovidersinthecommunity
EventuallyyouwilldevelopgoalstatementsfortheoutputsinordertocapturetheefforttoachievetheoutcomecontainedinyourgoalstatementYoursetofindicatorsshouldbecomposedofshort-andinterim-termsupportivemeasureswithaccuratetargetsandrelevanttimelinesAnexamplegoaltaxonomyfromtheCityofLasVegassResultsVegasprogramisbelowUsethistaxonomyasamodelforyourownsuiteofindicatorsandgoalstatements
Establishaframeworktomonitorprogress
36
Yourturn
Thetemplatebelowismeanttohelpyoucraftagoalstatementforeachsupportingmeasure
Goalstatement
Departments
Outcome
Activities
KPI
Target
Timeframe
Conductanddiscussrelevantanalysis
ThrougharoutineseriesofmeetingscontinuallymonitorprogressofKPIssupportingindicatorsandanyotherdatacollectedsuchasqualitativedatatoidentifytrendsthatwillindicatewhetherthechosenpolicyishavingthedesiredimpactThesedatapointsshoulddrivethediscussionduringthesemeetingspointingtoopportunitiesforfurtherinquiryinto
Establishaframeworktomonitorprogress
37
whatisdrivingtheobservedtrendsAnalystsshouldbeencouragedandtrainedtoaskquestionsofthedepartmentsproducingthedataandconductsitevisitswhenpossibletodeepentheirunderstandingofthedataPresentationoftheanalysisshouldfocusononeideaatatimetokeeptheconversationfocusedonkeypointsandspecificopportunitiesforimprovementYoumayconsiderdevelopingapublicfacingdashboardtosharethisprogressmorebroadly
AgeneralcomparisonofthestatusquodatawithwhatwasobservedduringthepilotcanprovidedecisionmakerswithaquickunderstandingofthedifferencesbetweentheoldandnewapproachesalthoughthiswillnotshowcausalityOftenthesecomparisonswillincludeoperationalmetricssuchasdaysordollarssavedbutrememberthattheimpactonpeoplewillbemostimportantIfyouareworkingwithanevaluationpartnertheywillsupportthisstep
Usethespacebelowtosummarizetheresultsofyouranalysis
MeasureTargetvalue(seesectiononsettingtargetsabove)
Pilotvalue(whatdidyouobserveduringthepilot)
Historicalaverage(ifavailable)
Results(WhoisbetteroffIsanyonenegativelyimpacted1-2sentencesreassessingyourgoalstatement)
DocumentyourprocessandprogressincludingfeedbackfromtheparticipantsimplementationpartnersandtheprojectteamDidanythingunexpectedoccurduringthepilotArethereinternalandexternalfactorsthatmayaffectyouroutcomesThisiswheretheimplementationoutcomescanbecanhelpfulIftheinterventiondidnotgoasplannedconsiderwhetheritisbecausetheinterventionwasnotacceptableorfeasible
UsethetemplatebelowcreatedbyDenversPeakAcademytocapturethisinformationIfthe306090daytimelinedoesnotmakesenseforyourcityfeelfreetocustomizeit
CurrentState
FutureState
30Day
60Day
90day
Whatwentwell(+)
Whatdidnotgowellshouldbechanged(-)
IfaquestionordiscussionpointarisesthatcannotbeansweredinthemeetinganassignedstaffmembershouldcapturethequestionduringthemeetingAfterwardthefollowupitemsshouldbesenttorelevantpartnersandthenextmeetingshouldbeginwithadiscussionof
Establishaframeworktomonitorprogress
38
thoseitems
Takeaction
DiscussionsaboutwhetheryourindicatorsaremovingintherightdirectionandwhatmightbedrivingtrendsmayleadtoadiscoverythattherearenotenoughresourcesallocatedtothispolicyorissueTheremaybelimitedfundingforstaffprogrammingortechnologythatisdrivingthedatainthewrongdirectionIfCityleadershipisintheroomthisisagreattimetomakethecaseforadditionalfundingduringthebudgetseasonBycollectinganalyzinganddiscussingdataonthispolicyroutinelyyouarebuildingevidencetomakethecasefortherightleveloffundingtoreachtheestablishedtargets
Establishaframeworktomonitorprogress
39
Assessdataaccessibilityqualityandsecurity
Determineaccessibilityandqualityofdata
OnceyouhavecompiledyourwishlistofindicatorsitistimetodeterminetheaccessibilityandqualityofthedataTopreventcomplicationyoumaywishtoprioritizerelevantindicatorsforwhichyouarealreadycollectingdataAvoidthetemptationofadoptingatechnologysolutiontocollectnewdataatthispointsuchasaphotosurveytoolorotherappFornowyouaretryingtodeterminewhichdatasetsarerelevantaccessibleandhighqualityBesuretoworkwithyourkeyinternalpartnerswhileyouareidentifyingyourmeasuresDepartmentstaffwhoworkonprogramsarefamiliarwiththeindicatorsthatwillaccuratelysupporttheoutcomesanddatabaseadministratorswillbevaluablewhendiscussingaccessibilityandqualityofthedata
Abrieflistofconsiderationsfortheaccessibilityofyourdataisbelow
ArewecurrentlycollectingthisdatasetorwillwebecollectingitforthefirsttimeHowfarbackdoestherecordgo(timehorizon)WouldweneednewtechnologytocollectoranalyzethedataCanthedatabecollectedmanuallyorcanitbeautomatedHowfrequentlyisthedataproduced(continuouslyhourlydailyweeklymonthlyquarterlyyearlyetc)IsthedataavailablefortheentireCitypopulationorasubsetofthepopulationHowgranularisthedata(individuallevelCensusblockneighborhoodcitywideetc)DoesthedatacontainsensitiveinformationIsthereaproxymeasurethatwecanusefornowIsthedatamachinereadable
AsnotedbytheWilsonSheehanLabforEconomicOpportunities(LEO)attheUniversityofNotreDameleveragingadministrativedatacansignificantlyreducethecostsandtimelinesofrigorousevaluationCitiescanestablishaninternaldatasharingagreementacrossdepartmentsincludinglanguagethatenablesadministrativedatatobeusedforresearchpurposesCitiescanalsoidentifyexternaladministrativedata-holderssuchasacountyschooldistrictpolicedepartmenthospitaloranotherstatethatmayhaveoutcomedataimportantformeasuringprogramimpactCitiescanleverageexistingrelationshipsandorworkwithresearchinstitutionstodevelopstreamlineddatasharingprocessesandagreementswiththeseexternalentities
Assessdataaccessibilityqualityandsecurity
40
Evaluatethequalityofyourdata
QualityisasimportantasaccessibilityToevaluatedataqualityyoumaywishtocreatearubricliketheonebelow
Value ValueDescription
3 ExcellentDataQualityThedataiscompleteaccurateandupdatedregularly
2 AdequateDataQualityThedataissomewhatcompleteaccurateandupdatedfairlyregularly
1 PoorDataQualityThedataisnotcompleteaccurateorupdatedregularly
ThisisagreattimetobuildonthedatainventorythatyoustartedinthegatherinformationpageSuggestedadditionalfieldstoaddtoyourdatainventorytemplatearebelow
Program DatasetTitle
DatasetDescription DataSource
Departmentorexternalpartner
DoestheCityalreadycollectthisdata
Whatisthenameoftheprogram
Human-readablenameofthedataset
Shouldbeunderstandablebynon-technicalusers
Whatistheoriginalsystemapplicationfilethathousesit
Departmentorexternalpartnerthatmaintainsthedata
Isthisdataalreadycollected
Updatefrequency-ThefileformatforthedatasetTypicallycsvxlsxshapefiletxtetc
Format-Howoftenthedatasetisupdated
Timerange-Theperiodoftimecoveredinthisparticulardataset
Datause-Howisthisdatasetcommonlyused
Datausers-Whotypicallyusesthisdataset
Dataownerstitle-Titleofthedataowner
Dataownersposition-DataOwnersjobtitle
Dataownersemailaddress-DataOwnersemailaddressWhomanagesthedataandorisresponsibleforgrantingpermissiontoaccessthedataWhounderstandswhatthedatasetincludesandcananswerquestionsaboutit
Dataqualityconcerns-Describetheconcernsyoumayhaveaboutthequalityofthedataset
Assessdataaccessibilityqualityandsecurity
41
Division-DivisionwherethedataismaintainedIfmorethanonedivisionisresponsiblelisttheprimaryownerofthedataset
Sensitivedatacomments-DescribetheissuesthatwouldpreventthecityfromsharingthedatapubliclyCommonsensitivityconcernsincludeprivacyviolationssecurityissuesorhighcostorstaffdemandstopostthedata
Publishingstatus-Thistheplacetoputanycommentsregardingthecurrentstatusofthedataset
Publishedlink-Thisisthelinktothepublisheddatasetontheopendataportalorcitywebsite
Reviewdatasecurityandprivacyprotectionprotocols
OftenapolicychangealsoinvolvesachangeinthedatathatisbeingcollectedhowthatdataisbeinghandledwhohasaccesstoitandahostofotheradjustmentsAsyoutakestockofthedataassetsthatareavailableforyourinitiativeitisvitaltoalsothinkthroughwhatwillbedonetoprotectthatdataandtheprivacyofthepeopleitrepresentsAbigpartofgooddatasecurityinvolvesstrongdatagovernancepracticesthatdefineandidentifysensitivedataandconsciouslyworktobalancethevalueandriskofstoringsuchsensitivedataOnewayofevaluatingyourdatacomesfromcaselawrelatedtotheUSFreedomofInformationActwhichwehavetranslatedtothechecklistbelow
1 Woulddisclosureresultinasubstantialinvasionofprivacy2 Haveweconsideredtheextentorvalueofthepublicinterestandthepurposeorobject
oftheindividualsseekingdisclosure3 Istheinformationavailablefromothersources4 Wastheinformationgivenwithanexpectationofconfidentiality5 Isitpossibletomouldreliefsoastolimittheinvasionofindividualprivacy
TheseriskscouldbemitigatedusingavarietyofstrategiesfromaggregatingdatatoobfuscatingindividualstoredactingfieldsthatmayincludePersonallyIdentifiableInformation(PII)orre-evaluatingyourdatacollectionmethodstofindalternativewaysofgettingtheinformationyouneedThesestepsshouldbecodifiedandmanagedbyadatagovernancecommitteeifpossibleForthesensitivedatathatisdeemedessentialcreateaplanaroundpotentialdatabreachesinordertoquicklyandconfidentlyrespondtoemergencysituations
ProactivelypreparingforthesesituationsisoftenthebeststrategyfordatasecuritybutdonotforgetthetechnicalsideInvolveyourITadministratorsinconversationsaboutdatastorageandaccessensuringthattheywillbeprotectedbyenterprise-gradecybersecuritysystems
Assessdataaccessibilityqualityandsecurity
42
Assessdataaccessibilityqualityandsecurity
43
Designimplementationandevaluation
Designtheimplementation
NowthatyouhaveyouraccountabilityplanandkeyindicatorsitistimetogetspecificaboutyourplanforimplementationThetemplatebelowwillhelpyouthinkthroughthespecificmechanismsofimplementingpolicychange
UnderstandingorganizationsandsystemsiscriticaltounderstandingandmakingdecisionsaboutwhereandhowinterventionsshouldbeappliedApolicyimplementedinasinglesettingmayshowpromisingresultsbutimplementingmultiplepoliciesandprogramsatthesametimewithdifferentactorsresponsibleforeachaspectcanhelpensuresustainablebehaviorchange
InSomervilleMAShapeUpSomerville(SUS)isacitywidepublichealthprogramthatisimplementedacrossmultiplelevelsofinfluenceandsystemsthroughinterventionsbeforeduringandafterschooltoincludechangestobuiltenvironmentsuchasbikelanesregionalmasstransportationandphysicaleducationequipmentinschoolsandgymstoincreasephysicalactivitywellnesspoliciesandschoolgardensnutritioneducationduringandafterschoolandhealthyrestaurantprogramstosupporthealthyfoodandhealthyeatinghabits
Usethetemplatebelowtoplanyourimplementation
Policyimplementationplanningtemplate
ListthespecificdesiredoutcomesInthepreviousstepyouworkedwithdepartmentstafftoestablishspecificmeasurableoutcomesandinterimindicatorsPrioritizetheindicatorsthatwilldemonstrateprogressUsethespacebelowtocapturethedesiredoutcomesofthepolicychange
ExampleShapeupSomervillebeganwiththegoalofpreventingobesityin1stthrough3rdgraders
Yourturn
____________________________________________________________________________
____________________________________________________________________________
Designimplementationandevaluation
44
SpecifythetargetpopulationAreyoudesigninganinterventionthattargetsonlycertainsitesAreyouplanningonimplementingaprogramorinitiativeforaspecifictargetaudienceClearlyspecifythepopulationyouplantotarget
IfyouplantoconductarigorousimpactevaluationasnotedbytheWilsonSheehanLabforEconomicOpportunitiesitisimportanttoestablishconcreteobjectiveeligibilitycriteriaandensurethatthesecriteriaareappliedconsistentlybyallstaffduringscreeningandintakeItisalsovitaltoidentifytheindividualswhoareeligibleforaparticularprogramorserviceevenifeveryoneeligiblecannotbeservedbyexistingresourcesThishelpscitiesidentifyanyunmetneedsandguidefutureresourceallocationwhilealsocreatingthepossibilityofarigorousimpactevaluationbyidentifyingapotentialcomparisongroupFinallywhenconductingscreeningandintakeitisimperativetocollectandmaintainbasicinformation(egnamedateofbirthgender)aboutallprogram-eligibleindividualsnotjustthosewhoparticipateintheprogram
ExampleIntheShapeUpSomervilleexampletheprogrambeganbytargeting1stthrough3rdgraders
Yourturn
____________________________________________________________________________
____________________________________________________________________________
DefinethelocationWherewillthepolicychangetakeplaceWillimplementationtakeplaceatonesiteormultiplesites
ExampleShapeUpSomervilleinterventionsweredeliveredinschools
Yourturn
____________________________________________________________________________
____________________________________________________________________________
ClarifyimplementationpartnersandarticulatetheirrolesWhatarethekeyorganizationsinstitutionsorpeopleresponsibleforimplementationTheprojectteamshouldbeasclearandspecificaspossibleinprovidingadescriptionoftheconcreteactivitiesthatimplementationpartnersaremeanttocarryoutPracticeprofilesmaybeahelpfultooltohelpdefinewhatisrequiredoftheactorswhoareresponsibleforimplementingtheintervention
Designimplementationandevaluation
45
ExampleInSomervillemayoralleadershipandengagementofmultiplecityagencies(publichealthprimarycareschoolsCambridgeHealthAllianceetc)streamlinedandconnectedthevariousShapeUpSomervilleinitiativesandgrantswhichhelpedtokeepthetalentandenergyfocusedonhealthyeatingandactivelivingforallsectorsofSomervilleSUSpartnersultimatelyincludedschoolsnonprofitsCityandstateagenciesandnationalfoundations
Yourturn
____________________________________________________________________________
____________________________________________________________________________
DescribetheinterventionWhatarethestepsactivitiesorprocessesRememberthatthemechanismthespecificstepsorprocessthatleadtobehaviorchangeiskeytounderstandingtheeffectivenessofthepolicychangeandsharingtheresultswithothers
ExampleGoalsofShapeUpSomervilleweretoinfluenceeverypartofanearlyelementaryschoolchildrsquosdayndashbeforeduringandafter-schooltopromotehealthyeatingphysicalactivityandtheirweightstatusInterventionincludedfoodserviceenhancementstoimprovebreakfastandlunchandstaffprofessionaldevelopmentwalktoschoolactivitiesSUSclassroomcurriculumandprofessionaldevelopmentschoolwellnesspolicydevelopmentSUSafter-schoolcurriculumandprofessionaldevelopmentoutreachandeducationtohomethroughmaterialsforumseventsldquoSUSapprovedrdquorestaurantscommunityoutreachandcapacitybuildingthroughpolicydevelopmenttrainingsmediaplacements
Yourturn
____________________________________________________________________________
____________________________________________________________________________
ClarifytimelineandmilestonesfordeliveryandevaluationHowfrequentlywilltheservicesinterventionbedeliveredHowlongwilltheprogramrunWhatisthetimetoexpectedimpacts(eg12monthsafterprogramcompletion)
Designimplementationandevaluation
46
ExampleShapeUpSomervilleledbyTuftsUniversityFriedmanSchoolofNutritionScienceandPolicyconductedaquasi-experimental(nonrandomized)interventionstudyamong1stthrough3rdgradestudentsinSomerville(intervention)andtwocomparisoncommunitiesfor9-monthsduringoneschoolyear(fall2002ndashspring2003)ThestudyshowedpositiveassociationoftheinterventiononchangeinBMISUSreleasedreportssummarizingevaluationsprogresstodateplansforadditionalprojectsandimpactstenandfifteenyearsintotheinitiative
Yourturn
____________________________________________________________________________
____________________________________________________________________________
PlanyouranalysisTheprevioussectionaddressedestablishinganaccountabilityframeworktomonitorprogressandimplementationDeterminewhichindicatorswillbediscussthroughthatprocesswhowillsharethedataandanalysisandhowfeedbackfromthoseconversationswillbeincorporatedbytheprojectteam
ExampleTheMayorrsquosofficeimplementedadata-drivendecisionmakingapproachcalledSomerStatwhichwassynergisticwiththeapproachestakenbytheresearchandevaluationteamsdeployedforthespecificgrants
Yourturn
____________________________________________________________________________
____________________________________________________________________________
PlanforsustainabilityIfthepolicyandprogramchangeissuccessfulhowwilltheprojectteamsustainandexpandtoadditionalsitesorpopulations
ExampleAlthoughmostofthepartnershipsforgedintheearlyyearsweremaintainednewgrantsandcollaborationsexpandedattentiontoolderpublicschoolstudentsactivetransportationandtheGreenLinesubwayextensionthebuiltenvironmentfarmerrsquosmarketeffortssuchasldquoGrowninSomervillerdquowithUnionMainStreeteconomicdevelopmentandhealthdisparitiesaswellasdatacollectionmonitoringandsurveillance
Yourturn
Designimplementationandevaluation
47
____________________________________________________________________________
____________________________________________________________________________
Pilotplanningtemplate
IfyouplantoconductapilotfirstdeterminethepurposeofthepilotDoyouwanttoobservewhetherthepolicychangeresultsinthedesiredbehaviorchangeAreyouconcernedaboutfeasibilityscalabilityorcostDoyouwanttotestwhetheryourdesiredresultscanbeachievedthroughthispolicyapproachClarifythegoalofthepilotandthequestionsyouhopetoanswerThiswillhelpyouidentifythekeyindicatorsthatyouwillusetomeasurewhetherthepilotissuccessfulBeforebeginningestablishabaselineofcurrentperformanceandsetyourtargetsaccordinglybasedonwhatyouhopetoaccomplishthroughthepilot
NextestablishthepilotplanforyourprojectteamincludingmilestonesforteamcheckinstodeterminewhethertomoveaheadorchangecourseThisisseparatefromthemeetingsthatwilloccuraspartoftheaccountabilityframeworkUsethespacebelowtocreateyourtimelineanddevelopyourmeetingschedule
Task Personresponsible Startdate Checkindate Deliverable
Planyourevaluation
WhileyouaredesigningtheimplementationitisalsotimetobegintodevelopyourplanforanalysisAsnotedbytheWilsonSheehanLabforEconomicOpportunitiesdecisionsaboutprogramsetup-includingoutreachscreeningintakedatatrackingandenrollment-candeterminewhetherornotarigorousimpactevaluationwillbepossibleAsdiscussedcityleadersmayfindithelpfultoincludeevaluationpartnersinearlyconversationsaboutprogramsetupDuringpreviousstagesyoumayhaveworkedwitharesearchertoplanyourproposedpolicychangeYoushouldbeclearaboutthepurposeoftheevaluationanddevelopanevaluationplanBelowareadditionalconsiderations
Causationvstrendanalysis
IntheEstablishaframeworktomonitorprogressstagetherewasguidanceontrackingtrendsovertimeItisimportanttonoteherethatmonitoringdataisdifferentfromanalyzingacausaleffectTrendanalysiswillillustratethatchangesareoccurringandmayleadtoquestionsthatcouldbeansweredbyyourevaluationbuttrendanalysisalonedoesnotprovideevidenceaboutwhatisdrivingthosechangesYoumayseechangesinyour
Designimplementationandevaluation
48
outcomeindicatorsbasedonvariablesoutsideoftheCityscontrolForinstanceachangeinfederaltradepolicycanhaveanimpactemploymentratesinyourCityoutsideofyourCitysnewjobtrainingandplacementprogram
ToaddresscausalityyoumustcomparetheoutcomesofindividualswhoreceivedtheinterventionwiththosewhodidnotWithoutacomparisongroupofsimilarindividualswhowerenotexposedtotheinterventionitisimpossibletoknowwhichfactorsareresponsiblefortheoutcomesRandomizedControlTrials(RCTs)inwhichindividualsarerandomlyassignedtotreatmentandcontrolgroupsarethegoldstandardinprovingcausalityIfyoucandemonstratethatoutcomesimprovedforthosewhoreceivedtheinterventionanddidnotimproveatallorimprovedlessforacomparisongroupyoucanbesomewhatcertainthattheinterventionwasthecause
Examplesofothernon-causalimpactevaluationsthatarerigorousandreliableincludecertainquasi-experimentalstudiesusingcomparisongroupdesignssmartinterruptedtimeseriesanalysesthatmodeltrendsovertimeinwaysthatallowestimationofinterventioneffectswelldesignedpre-posttestanalysesneedsassessmentsimplementationstudiesetcFormoreinformationondifferenttypesofevaluationsseetheSageResearchMethodswebsite
Lookfordifferencesingroups
BuildintimetoalsoexaminethepilotdatatoseeifgroupsareexperiencingdifferentresultsReviewingperformancebasedonoutcomemeasuresforapopulationinaggregatemayhidethesubtlewaysthatsomepeoplemaybeaffecteddifferentlysoitisnecessarytocarefullyanalyzethedatafordisparateoutcomesForinstancetheMovingtoOpportunity(MTO)studythatexaminedtheimpactofhousingvouchersonfamilieswholivedinhigh-povertyneighborhoodsfoundthatchildrenwhomovedbeforetheageof13hadbettereducationandhigherincomesasadultsHoweveradultsdidnotseeimprovededucationandemploymentoutcomes
Disaggregatingdataintomorespecificcategoriesofraceandethnicitycanhelpunderstandtrendsanddisparitiesinamoregranularwaywhichcanhelpdecisionmakerstargetpoliciesandinterventionsinamoreequitableway
EarlierintheRoadmapweworkedtoidentifystakeholdersandwhomightbepositivelyandnegativelyimpactedbyachangeinpolicyUsetheresultsofthatexercisetoinvolvethecommunityininterpretingdataanalysisandtogatherqualitativefeedbackfromthecommunitytocallattentiontoissuesForinstanceanincreaseinyouthdetentionmaybeexplainedbyinefficienciesorlackofaccesstotransportation
Designimplementationandevaluation
49
Designimplementationandevaluation
50
Shareyourstoryandsuccess
Sharetheresults
Storytelling
AnintegralstepinyourstrategyissharingtheresultsofyourpilotorpolicychangewithyourcommunityThisincludesdevelopingmessagingaroundtheworkofyourorganizationandcreatingaplantodisseminateyourstoriesWhytellstoriesaboutyourwork
Informandorunderstandasubjectorsituation
Giveneededcontextofdataorbitsofinformation
Relatablestoriesarememorableforreadersviewers
ReadersorviewerswillseetheinformationasmorecompellingwhenthereisademonstratedandrelatablechangeorimpactonthemselvesortheirfriendsneighborsandcolleaguesStoriesareaboutpeopleandgovernmentisfundamentallyabouthelpingpeopleSotellthestoriesofthepeopleyouandyourcitycolleagueshelped
Keepthesetipsinmindasyoudevelopyourstorywithyourteam
WhatwereyourfindingsandwhyshouldanyonecareTellyouraudiencehowtheproblemwassolvedWhathastheimpactbeenonresidentsandcitystafftodateWhatwasthesuccessandweretheremultiplesuccessesalongthewayIfyouhavepreliminaryresultsordatashowingimprovementtosharethisisthetimetodoit
WhowasinvolvedandimpactedReviewthelistofstakeholdersHowwereeachofthemimpactedbytheprogramHumanizeyourstoryandimpactbyaddingadditionalvoicesThesevoicescanbeoneofthefollowing
Apersonorfamilywhowaspositivelyaffectedbythechangesuccess
Aninvolvedthird-partydescribingboththeproblemandthesolution(ieanorganizationnonprofitprovidersbusinesstradeassociationsportsteametc)whoispartofthecommunitythatthegovernmentworkedwithonthechangesuccess
Someonefromthegovernmentitselfwhocanspeakaboutthechangesuccess
TellmoreaboutyourcityDescribeyourcityincludingpopulationdemographicsindustriesetcWhatareitschallengesWhatisitknownforWhatmakesituniqueHowisitsimilarordifferentfromothercitiesintheUS
Shareyourstoryandsuccess
51
DescribeyourcityrsquoschallengeWriteoutthechallengeyourcityisfacing(asdefinedinyourproblemstatement)HowdoesitaffectresidentsHowdoesthisproblemaffecttheworkofcitystaffmembers
AddthedetailsProvidedetailsonyourappliedresearchstrategyTalkaboutwhatyourcityhasdonenthepasttoresolvetheissueDiscusshowyourstrategyhaschangedwhenyoubegantoadoptthisapproachWasmoneysavedorinvested
Usethisstorymapexercisetoorganizeyourthoughtsandguaranteethatyouincludealloftheelementsofatraditionalstory
AfteryoudevelopyourmessagingcreateacommunicationsmarketingandoutreachstrategydesignedtospreadyourmessageYourstrategyshouldwalkyouthroughthefollowingkeyactivities
1 IdentifyandprofileyouraudienceWhencreatinganddisseminatingyourmessageitisalwaysgoodtothinkaboutyouraudienceWhoexactlywouldbeinterestedinwhatyouhavetosayThegeneralpopulationofthecityPeoplewhoworkinnonprofitsBusinessownersItmattersbecausenoteveryonewholivesinacitymaybethetargetoftheinformationBusinessownersmaybemoreinterestedindatashowingeffectsonthebusinessdistrictthanothersandsoonWewillcoverthislaterbutthisstepwillalsohelpyoutothinkofthemosteffectivewaytoreachyouraudience
2 Defineyourcalltoaction(ifyouhaveone)SpendsometimeconsideringwhyyouareinterestedintellingyourstoryDoyousimplywanttoupdatethepubliconyourprogressorcelebrateyoursuccessesAreyouinterestedinyouraudiencechangingitsbehaviororsupportingaspecificinitiativebeinglaunchedbyyourorganizationIfyouwanttohaveyouraudienceactorbehaveinacertainwaybespecificandincorporateitinyourmessaging
Shareyourstoryandsuccess
52
3 SelectperformancemeasuresThinkabouthowyouwoulddetermineifyourcampaignissuccessfulandselectappropriateperformancemeasuresYouwillrevisitthesemeasuresattheendofyourcampaigncommunicationsplanimplementationtodeterminewhetheryouneedtomodifyyourcommunicationsandoutreachstrategyforthefuture
4 MethodstocommunicateyourstoryTherearemanywaystocommunicatesuccessofyourprojectHerearesomesuggestionsonreachingaudiencesItisimportanttodeterminewhichmethodsarethemosteffectiveforyourdesiredaudiencetoreceiveinformationDoyourresearch(PewResearchCenterhasexcellentstudiesonthereachofmediadigitalmediaandmarketing)ForexampleifseniorsareyourtargetpopulationandtheyarelessinclinedtouseInstagram(thatistrue)youmaywanttoconsiderFacebookorthenightlylocalnewsbothofwhichareusedmoreoftenbyseniorsBelowisalistofsomemethodsandtoolsyoucanuse
i Pressreleasestolocalnewspapersandtvstations
ii ShortsmartphonevideosAgreatwaytoengageanaudienceisthroughvideoAskonequestionofsomeonewhoispartofthestoryMaybethereisasocialworkerwhocandiscusshungerinyourcityThesevideoscanbereusedonmultipleplatforms
iii Long-formwriting
iv TellingindividualstoriesthroughaseriesofphotosandshortnarrativesforpublicationonFacebookorInstagram
v Op-edsopinionpiecesplacedinthelocalpaper
vi Emailupdatestoyourlist
vii Connectingwithappropriateorganizationsthatareinvestedinthetopicandworkwiththecommunityandaskthemtodisseminateinformationtotheirmembers
viii EmployingldquoWordofMouthrdquo(WOM)tacticsbycreatingtownhallsorfreeeventsforresidentstoattendintheirownneighborhoods
5 DeterminethetypeofcollateralUsuallycampaignsthattargetresidentsrequireseveralavenuestoreachthemPeoplearenotalwaysinthesameplaceorspaceHavingakeenunderstandingofyouraudiencewillassistyouwithyourchoicesonwhatneedstobedevelopedSometimesthatmeansimagesforsocialmediaaonepageflierthatgivespeopleinformationonattendinganeventoranin-depthbrochure
Shareyourstoryandsuccess
53
6 ImplementyourstrategyKeepinmindstrategiescanbefluidSometimessomethingmaynotturnoutasexpectedevenifduringplanningsessionsitwasldquosupposedtordquoItisokYoucananalyzewhathappenedduringadebriefsessionandonceyoursurveysandothermeasurementshavebeenfinalized
Visualizations
Withtherightvisualizationyoucanreviewandshareinformationinsightsanalysisperformanceandtrendsallofwhichisvaluabletoacross-sectionofthepublicButwhichvisualizationisrightforyouTochoosetherighttypeofchartgraphdeterminethetypeofquestionyouwanttoanswerSomecommoncharttypesarebelowandmoreinformationcanbefoundhere
Exploreadditionalanalyticaltechniques
Duringthegoal-settingphaseyouestablishedoutputoutcomeandimplementationmeasuresBasedontheresultsofyourpilotyoumayneedtoadjustyourmeasuresWiththepilotdatainhandyoumayhavesomeideasofhowfutureanalysescouldbedifferentandpotentiallyimprovethequalityofyourprogramTakesometimetobrainstormotherrelevantdatasetsthatyoudidnothavethetimetopursueearlierincludingmessydepartmentaldatathatneedsmoreattentionorsourcesofpublicdatafromthestateandfederalgovernmentnonprofitsandevenprivatecompaniesthatcouldsupplementyourunderstandingofanissueUsingmachinelearningtechniquesyoumightbeabletobetter
Shareyourstoryandsuccess
54
predictdemandforhomelessservicesusingNOAAweatherforecastspreemptwatermainbreaksusingsoilstratigraphyorinvestineconomicdevelopmentbymonitoringrealestatelistingdata
Advancedanalyticaltechniquescanalsohelpyourimplementationrunmoreeffectivelybyintegratingdata-informedtargetinganddecisiontoolsintotheprocessByautomatingroutineworkflowsandsurfacingsubtletrendsthesetoolscanhelpyouiterateinamoreprecisemannerandprovidekeystrategicguidancetofront-lineemployeesthatwouldotherwisebeunavailableThesesortsofdata-science-enhancedprocessescanbeclassifiedasoneofsixprojecttypesasexploredfurtherbytheNewOrleansLANOLAlyticsteamFormoreguidanceonscopingasimilarprojecttheGovExadvancedanalyticsscopingguidegivesyouastep-by-stepoverviewofhowtobesuccessful
Usethespacebelowtolistyourideasforpossibleadvancedanalyticsprojectsbytype
Typeofproject Ideas
Findingtheneedleinthehaystack
Prioritizingforimpact
Earlywarningtools
Betterquickerdecisions
Optimizingresourceallocation
Experimentingforwhatworks
Keepthefeedbackloopgoing
GatherinformationonpublicsentimentaboutpolicychangethroughfocusgroupssurveysandpublicmeetingsledbyadvisorygroupmembersNotewhethernewinformationisavailablethatcanhelpanalyzeorcontextualizeyourprogress(egfederalorstatedatareports)Sharetheresultsofyourpilotandthepolicychangeprocesswithothercitiesandresearchers
Shareyourstoryandsuccess
55
WorkingwitharangeofevidencePresentlythereisalotofattentionandinvestmentinevidence-basedpolicymakingincludingthefederalbipartisanCommissiononEvidence-BasedPolicymakingthePewCharitableTrustsreportsandtoolkitsonHowStatesEngageinEvidence-BasedPolicymakingandtheLauraandJohnArnoldFoundationsupportedEvidence-BasedPolicymakingCollaborativeManyoftheseinitiativesfocusonrigorousevaluation--meaningrandomizedcontrolledtrials(RCT)orotherexperimentalmodels--asthebestkindofevidencetouseHoweverthisbodyofevidenceisofteninsufficientforthespecifickindsofquestionsthatgovernmentscanhavewhenmakingadecisionaboutpolicyInthisguideweproposeanapproachtosupportgovernmentsinincorporatingadditionaltypesofevidenceintotheirdecisionmaking
OnewaytounderstandarangeofevidenceistouseNestarsquosStandardsofEvidencetoframeanddefineevidenceTheBritishnonprofitNestaworkstoinjectpositivechangeintothewayUKgovernmentpoliciesandprogramsfunctionOneoftheirkeyareasofworkhasbeendevelopingWhatWorksCentresorganizationsthatcollectcommissionsynthesizeanddisseminateevidenceaboutinterventioneffectivenessacrossarangeofsocialimpactareasNestadevelopedthefollowingStandardsofEvidencetoprovideausefulwaytodocumenthowdifferentkindsofinformationdemonstratedifferentlevelsofevidence
Workingwitharangeofevidence
56
Theimportanceofpotentiallybiasedinformation
Onechallengepresentedbyanevidenceclassificationthatacknowledgesthevalueofnon-scientificformsofevidenceistofigureouthowtobalanceallofthepiecestogetherwhenyouusethemTounderstandhowtodothiswehavetothinkabouthowtopositionpotentiallybiasedinformationindecisionmaking
WhenwelookatinformationwhichhasnotbeenrigorouslyevaluatedweacceptthatthisinformationmightbebiasedThewholereasonweconductrigorousevaluationistoremovebiasfromourobservationsWeconductstructuredtrialstoproduceresultsthatareconsistentandgeneralizableIfafindingisunbiasedthisaccuracymdashthislackofbiasmdashcanbeprovedbecauseanyindependenttestercanreproduceitwhentheyfollowanidenticalprocess
YoucanlookatpotentiallybiasedinformationasjustinformationthatisearlieroninitspathtorefinementreplicationandverifiedaccuracyApieceofinformationbasedonanyoneindividualrsquosobservationsislikelytobebiasedbecauseitiscomingfromjustonesourceBiasisreducedbygatheringtogetherlotsofindependentsourcesofinformationandfiguringoutitsmostcommonattributes
MaybethebiasedinformationmakesaclaimthatistoolargemdashitsayssomethingistrueunderallconditionswhenitisreallyonlytrueundercertainconditionsorthataneffectofaninterventionisbiggerthanitstrueaverageorthataneffectlastsovertimewhenitdoesnotPerhapschangingtheclaimtobemorespecificmdashcriticallyreviewingtheinformationtoseeifspecificconditionsmustbemettoachievetheoutcomesmdashwouldmaketheclaimmoreaccurate
IfyoutestedyourclaimfounditdidnotholdupinsomecaseschangedittesteditagainuntilyourclaimrepeatedlyheldupthenyoucoulddemonstratethattheclaimwasnolongerbiasedbutaccurateThisprocessoftestingimprovingtestingandvalidatingiswhatwegetfromstructuredscientificstudy
Withindividualpiecesofinformation--crediblehypothesescasestudiesexpertopinion--youcannotassumethatkindofaccuracyAtthesametimetheseearlyobservationsarefoundationaltoanyreplicablescientificobservationsWewillnevergetanywhereifwedonotstartwiththesepiecesofinformationeveniftheymightbebiased
Combininglevelsofevidencetogetthemostdetailedinformationforyourcity
ItisespeciallyimportanttoacceptdifferentformsofevidencewhenyoufacequestionsthathavenotyetbeentestedbyindependentstudiesSomeofthemostimportantquestionsrelatetothingsthatareveryspecifictoyoursituationYoufaceaparticularcombinationof
Workingwitharangeofevidence
57
issuesactorstrendsandresourcesthatisultimatelydifferentfromeveryotherplaceintheworld
ThisisimportantbecauseaspolicypractitionersandnotscientistsyoucaremoreaboutyourspecificcasethanahypotheticalaveragegeneralcaseYoucarelessaboutthetotallikelyrangeofimpactsfromapolicythanhowthissingleimplementationwillworkYouwantinformationaboutpoliciesthatwillprovideyouwiththebestpossiblepredictionabouthowtheywillworkinyourcontext
ByputtingdifferentkindsofevidenceintothesameframeworkNestarsquosStandardsofEvidenceimplythatwhenyoucannotgetrigorouslytestedexperimentallyverifiedinformationitisappropriatetoworkwithwhatwedohaveThisisrelevantforthemunicipalimplementationcontextbecausenostudypulledfromaliteraturereviewwillaccountforallofthevariablesyoufaceinyourowncontext
Intheabsenceofhavingastudythatpredictshowaninterventionworksinyourspecificcontextyouwillneednon-scientificinformationtotrytofillinthosegapsYouwillneedtogatherstoriesfromresidentsstaffmembersandotherstakeholdersthatsitonLevels1or2ofthestandardsofevidence
Someofthekindsofinformationyouwillprobablygetmoreeasilyfrompeoplethanstudiesinclude
howpeopleincitieslikeyoursreactedtoaninterventioninpublicforumshowpeoplewhoimplementedaninterventioninacitylikeyoursfiguredouthowtogetitpassedhowthecitycommunicatedpubliclyabouttheapparentbenefitsanddrawbacksoftheinterventionthepoliticalacceptabilityoftheintervention
Youcanremainsimultaneouslyawarethattheinformationyouaregatheringthroughindividualstoriesisbiased--notgeneralizableinformedbyspecificperspectives--whilealsorecognizingthathavingsomeinformationaboutsomethingimportantisbetterthannoinformationEvidencefromthefirsttwoevidencelevelsareessentialbuildingblocksofknowledgeandperfectlyusefulwhenyoucannotgetdetailedunbiasedknowledge
Youhavetostartsomewhereandyoushouldalwaysstartwithevidenceyouhaveaboutyourveryspecificconditionsthatyouknowareimportantforsuccess
Workingwitharangeofevidence
58
Thankyou
Interviewswithcities
GovExheldsemi-structuredinterviewswithsixstaffmembersinfourcitiestoestablishtheinitialstepsoftheRoadmapWeaskedrespondentstotalkaboutaninitiativeorpolicythattheyhavebeenresponsibleforimplementingintheircityRespondentsfromthreecitiesspokeabouttheireffortstoaddresshomelessnessandrespondentsfromonecityspokeaboutblightAfterrespondentsidentifiedtheproblemtheyspokeabouthowthecityprioritizedthisissuethevariousstepsinvolvedandbarrierspresent
GovExconductedtextanalysisoftheseinterviewswhichrevealedthatpeoplewasthemostcommonlyusedtermduringtheseconversationswhendiscussingthestepstochangeaprogramorpolicyThisisathemethroughoutthisRoadmapThefirstsectionbeginswithidentifyingtherelevantpeopleTheconversationsaroundevidencearefocusedonhumanbehaviorratherthanmethodologyorstrengthofevidence
Thankyou
59
OncewehadadraftoftheRoadmapwefollowedupwithcitiesonanindividualbasistolearnmoreaboutwhatwouldbehelpfulWearegratefultothefollowingcitystafffortheinterviewsandindividualconversations
MichaelBaskinChiefPolicyOfficerChattanoogaTN
MelissaBridgesPerformanceandInnovationCoordinatorLittleRockAR
VictoriaCarreoacutenAdministrativeOfficerLasVegasNV
TyroneGrandstrandCommunityDevelopmentPlannerFargoND
TibLaughlinDirectorofGeneralServicesKansasCityKS
TimMorelandDirectorofPerformanceManagementandOpenDataChattanoogaTN
CharlesRosendahlManagementAnalystDentonTX
Thankyou
60
MelissaSiebenAssistantCountyAdministratorKansasCityKS
GinaThompsonDirectorofStrategicServicesLewisvilleTX
JHUFacultycollaboration
GovExalsoconvenedthecitycontactsandfacultypartnersinaseriesofAdvisoryGroupmeetingsThesemeetingsservedtoconnectcitiesandfacultytounderstandoneanothersperspective
StefanBaralBloombergSchoolofPublicHealth
AshleyBernerJHUSchoolofEducation
MelissaDavey-RothwellBloombergSchoolofPublicHealth
JonLinksBloombergSchoolofPublicHealth
TamarMendelsonBloombergSchoolofPublicHealth
KristinMmariBloombergSchoolofPublicHealth
KeshiaPollackPorterBloombergSchoolofPublicHealth
JoshSharfsteinBloombergSchoolofPublicHealth
LizStuartBloombergSchoolofPublicHealth
LindsayThompsonCareyBusinessSchool
DanielWebsterBloombergSchoolofPublicHealth
Partnersindataandevidence-basedpolicy
Wealsospokewithpartnerswhoareexpertsinthefieldofevidence-basedpolicywhosharedtheirknowledgeandresourcesofcurrentandfuturethinkinginevidence-informedthinking
WendyBarrenoAssociateDirectorWilsonSheehanLabforEconomicOpportunitiesUniversityofNotreDame
TiffanyDavenportDirectorofDataPartnershipsBaltimoresPromise
TomiHiersDirectorBaltimoreCivicSiteAnnieECaseyFoundation
ChrisKingsleySeniorAssociateAnnieECaseyFoundation
KathrynOliverAssociateProfessorLondonSchoolofHygieneandTropicalMedicine
Thankyou
61
KristenWatkinsSeniorPolicyAssociateJ-PAL
GovExParticipants
GovExAnalystMiriamMcKinneyanalyzedtheinterviewtranscriptsandconductedtextanalysisSeniorImplementationAdvisorEmilyShawwillintegrateRoadmapactivitieswithadditionalservicedeliveryactivitiesatGovEx
Thankyou
62
ReferencesAbrahamKGampHaskinsR(2017)Thepromiseofevidence-basedpolicymakingReportofthecommissiononevidence-basedpolicymakingWashingtonDCCommissiononEvidence-basedPolicymakingBigmove2Extendingthegreen(nd)Retrievedfromhttpswwwcityoflewisvillecomdoing-businesslewisville-2025big-move-2-extending-the-greenBatesMAampGlennersterR(2017)ThegeneralizabilitypuzzleRetrievedfromhttpsssirorgarticlesentrythe_generalizability_puzzleBuildwithnotfor(2018January)RetrievedfromhttpsbuildwithnotforwordpresscomCairneyPampOliverK(2017)Evidence-basedpolicymakingisnotlikeevidence-basedmedicinesohowfarshouldyougotobridgethedividebetweenevidenceandpolicyHealthResearchPolicyandSystemsdoi101186s12961-017-0192-xCairneyPOliverKampWellsteadA(2016)TobridgethedividebetweenevidenceandpolicyReduceambiguityasmuchasuncertaintyPublicAdministrationReview76(3)399-402Retrievedfromhttpdxdoiorg101111puar12555Centreforpublicimpact(nd)RetrievedfromhttpswwwcentreforpublicimpactorgCernaL(2013)ThenatureofpolicychangeandimplementationAreviewofdifferenttheoreticalapproachesOrganisationforEconomicCo-OperationandDevelopmentRetrievedfromhttpswwwoecdorgeduceriTheNatureofPolicyChangeandImplementationpdfColoradoSprings(nd)Retrievedfromhttpscoloradospringsgovcommunicationspagewhat-expect-when-you-speakupContinuumofcare(coc)program(nd)RetrievedfromhttpswwwhudexchangeinfoprogramscocCouncilpresidentkicksoff$12millionyouthfund(nd)Retrievedfromhttpwwwbaltimorecitycouncilcomcontentcouncil-president-kicks-12-million-youth-fundDurlakJAampDuPreEP(2008)ImplementationmattersAreviewofresearchontheinfluenceofimplementationonprogramoutcomesandthefactorsaffectingimplementationAmericanJournalofCommunityPsychology41327-350Retrievedfromhttpswwwncbinlmnihgovpubmed18322790Fishbonediagram(ishikawa)(nd)Retrievedfromhttpasqorglearn-about-qualitycause-analysis-toolsoverviewfishbonehtmlFivewhysGettingtotherootofaproblemquickly(nd)RetrievedfromhttpswwwmindtoolscompagesarticlenewTMC_5Whtm
References
63
FligsteinNampMcAdamD(2011)TowardageneraltheoryofstrategicactionfieldsSociologicalTheory29(1)1-26RetrievedfromhttpsociologyberkeleyedusitesdefaultfilesfacultyfligsteinFligstein20McAdam20Soc20Theory20PaperpdHeinkelP(2018January12)CitymullspermitsforbikesharingcompaniesRetrievedfromhttpwwwdentonrccomnewsdenton20180112city-mulls-permits-bike-sharing-companiesHowisareamedianincomecalculated(2012October)Retrievedfromhttpswwwhudexchangeinfofaqs843how-is-area-median-income-calculatedLampkinLMampHatryHP(2003)KeystepsinoutcomemanagementUrbanInstituteRetrievedfromhttpswwwurbanorgresearchpublicationkey-steps-outcome-managementMarshDampSmithM(2000)UnderstandingpolicynetworksTowardsadialecticalapproachPoliticalStudies484-21Retrievedfromhttpjournalssagepubcomdoiabs1011111467-924800247McCannL(nd)RetrievedfromhttplaurenellencomMoultonSampSandfortJR(2016)ThestrategicactionfieldframeworkforpolicyimplementationresearchPolicyStudiesJournal45(1)Retrievedfromhttponlinelibrarywileycomstore101111psj12147assetpsj12147pdfv=1ampt=jdkhhcp4amps=e184c625e1855a6b1615ccb99ee27b1c0b3212d8PaynterB(2018January1)What100citiesarelearningfromeachotherbysharingtheirdataRetrievedfromhttpswwwfastcompanycom40518004what-100-cities-are-learning-from-each-other-by-sharing-their-dataPeakacademy(nd)Retrievedfromhttpswwwdenvergovorgcontentdenvergovenmayors-officeprograms-initiativespeak-performancepeak-academyhtmlPinnockHBarwickMCarpenterCREldridgeSGrandesGGriffithsCJRycroft-MaloneJMeissnerPMurrayEPatelASheikhAampTaylorSJC(2017)Standardsforreportingimplementationstudies(stari)ExplanationandelaborationdocumentBMJOpenRetrievedfromhttpbmjopenbmjcomcontentbmjopen74e013318fullpdfProctorESilmereHRaghavanRHovmandPAaronsGBungerAGriffeyRampHensleyM(2011)OutcomesforimplementationresearchConceptualdistinctionsmeasurementchallengesandresearchagendaAdministrationandPolicyinMentalHealth38(2)65-76doi101007s10488-010-0319-7TandonSD(2018)ProgressincommunityhealthpartnershipsResearcheducationandactionRetrievedfromhttpswwwpressjhuedujournalsprogress-community-health-partnerships-research-education-and-actionProvanKGampKenisP(2007)ModesofnetworkgovernanceStructuremanagementandeffectivenessJournalofPublicAdministrationResearchandTheory
References
64
18(2)229-252Retrievedfromhttpsacademicoupcomjpartarticle182229935895SabatierPA(1988)Anadvocacycoalitionframeworkofpolicychangeandtheroleofpolicy-orientedlearningthereinPolicySciences21(3)129-168Retrievedfromhttpwwwjstororgstable4532139SandstromAampCarlssonL(2008)TheperformanceofpolicynetworksTherelationbetweennetworkstructureandnetworkperformancePolicyStudiesJournal36497-524Retrievedfromhttponlinelibrarywileycomdoi101111j1541-0072200800281xabstractSchwartzMampWilsonE(nd)WhocanaffordtoliveinahomeAlookatdatafromthe2006americancommunitysurveyRetrievedfromhttpswwwcensusgovhousingcensuspublicationswho-can-affordpdfShearerJCAbelsonAKouyateBLavisJNampWaltG(2016)WhydopolicieschangeInstitutionsinterestsideasandnetworksinthreecasesofpolicyreformHealthPolicyandPlanning31(9)1200-1211Retrievedfromhttpsdoiorg101093heapolczw052ThelessonsofadministrativedataHigh-profilepolicy-relevantresearchpoweredbyadministrativedata(nd)Retrievedfromhttpswwwpovertyactionlaborgsitesdefaultfilesresources20161114-The-Lessons-of-Administrative-DatapdfToolsandotherneedsforqualitativeandquantitativemethods(2015February)AgencyforHealthcareResearchandQualityRetrievedfromhttpwwwahrqgovprofessionalsquality-patient-safetytalkingqualityassesstoolshtml
WeaverRCampBagchi-SenS(2013)SpatialanalysisofurbandeclineThegeographyofblightAppliedGeography4061-70
References
65
UniversityPartners
Name Title Affiliation Area(s)ofexpertise
Contactinformation
Examplesofworkingwithnon-academicdecisionmakers
WendyPBarreno
AssociateDirector
WilsonSheehanLabforEconomicOpportunitiesUniversityofNotreDame
LEOsfocusareasincludehealtheducationhousingchildwelfareeconomicopportunitycriminaljustice
wbarrenonduduorleondedu
MelissaDavey-RothwellPhD
AssociateScientist
JohnsHopkinsBloombergSchoolofPublicHealth
Health mdavey1jhuedu410-614-5854
BaltimoreCityHealthDepartmentLeadersMarylandDepartmentofHealthAddictionRecoveryServicesinDCExecutiveDirectorsofcommunityorganizationsacrosstheUSthatfocusonHIVSTIsubstanceabuseandimplementationscience
ElizabethStuart
AssociateDeanforEducationandProfessorDepartmentsofMentalHealthBiostatisticsandHealthPolicyand
JohnsHopkinsBloombergSchoolofPublicHealth
Educationsocialpolicystatisticalmethods
estuartjhuedu
Resources
66
Management
Design-ThinkingResources
EngagementGuideforCivicTech
Journeymap
PerformanceManagementResources
DenverPeakAcademy
Proxymeasuresguide
Talkingaboutperformancewithoutcomplicatingit(video)
SettingPerformanceTargetsguide
Datavisualizationguide
Performancemanagementgettingstartedguide
Exercises
Fishboneexercise
5Whys
AdministrativeData
TheLessonsofAdministrativeData(JPAL)
EvaluationResources
Quantitativeandqualitativemethods(USDepartmentofHealthandHumanServicesAgencyforHealthcareResearchandQuality)
KeyStepsinOutcomeManagement(UrbanInstitute)
RCT-YES
Sageresearchmethods
BestPracticesforMixedMethodsResearchintheHealthSciences
ACCORDChecklist
Pilotandimplementationplanning
Benchmarkingexamplesandproxymeasures
Seattleopendataplaybook
Resources
67
Datasciencescopingguide
Resources
68
AboutGovExGovExhelpsgovernmentsusedataandevidencetoimproveresidentsrsquolivesWesupportgovernmentsinbuildingtheirinternalcapacitytooperatesustainableanalyticspracticeswhichimprovecommunityoutcomesinfieldsfrompublicsafetytoeconomicdevelopmentGovExhasaproventrackrecordofworkinginconcertwithpartnerstoprovidetechnicalassistanceandtrainingatscale
SinceourlaunchinApril2015GovExhas
Trainedover2000governmentemployeesinpersonandonlineintopicsincludingdatamanagementperformancemanagementopendatacommunityengagementandanalytics
Providedtechnicalassistancetomorethan100localgovernmentsacrosstheglobeimprovingtheiruseofdatatoachieveefficienciesinservicedeliveryandbetteroutcomesforresidents
Assessedthecapacityofmorethan130governmentalorganizationstodeterminehowleadersstaffandexternalstakeholdersmanageandusedata
Localgovernmentsareincreasinglyonthefrontlinesofaddressinglargescaleproblemswithglobalimplicationssuchasclimatechangepublicsafetyandeconomicinequality
Whileevidence-basedinterventionstohelpsolvethoseproblemsareavailablelocalgovernmentstaffhavelimitedtimeandcapacitytofindandimplementresearch-basedsolutionsAsaresultcitiescontinuetoworkthewaytheyalwayshave--makingincrementalimprovementstoprogramsbutnotalwayscreatingchangeatasystem-level
ThroughresourcedevelopmentandcollaborationwithJHUfacultyGovExissupportingcityleadersinusingdataandevidencetomakedecisionsandcreatinglastingrelationshipswithresearchers
TheRoadmapwasdevelopedbyGovExaspartofitspartnershipwithBloombergPhilanthropiesWhatWorksCitiesinitiative
AboutGovEx
69
- Title page
- Introduction and purpose of the Roadmap
- Using the Roadmap
-
- Gather information about the problem
- Strategically engage key partners
- Review the evidence and collect details
- Decide on a policy to implement based on local factors
- Establish a framework to monitor progress
- Assess data accessibility quality and security
- Design implementation and evaluation
- Share your story and success
-
- Working with a range of evidence
- Thank you
- References
- Resources
- About GovEx
-
UsethistemplatedevelopedbytheCentreforPublicImpact(CPI)tobrainstormwhoyoumightreachouttoasyouembarkontheroadtopolicychangeRemembertothinkabouthoweachpersonorgrouprelatestotheproblem
BegintodefinethewhatortheprobleminyourCity
TherestofthissectionwillhelpyourefinetheproblemstatementFornowgathersomekeypartnerstoclearlyarticulatewhychangeisneededUsethetemplatebelowdevelopedbyDenverPeakAcademytocapturetheproblemstatementYouwillbuildonthislater
ProblemStatementndashWhyChangeisNeeded(1-2Sentences)
ThroughourinterviewswithcitystaffweheardthatwhendiscussingmunicipalchallengesitcanbetemptingtojumptoasolutionbeforetakingthetimetoreallyunderstandtherootcauseoftheproblemThismakessensesincecityleadersandstaffareresponsiblefordeliveringservicesandbeingresponsivetoresidentswhileaddressingnewproblemsthatariseinthecity
TherootsoftheproblemsthatcitiesareworkingtosolvearecomplexandmaybethelegacyofhistoricalfederalpolicyorsweepingsocialtrendsTohelptheproblemseemlessdauntingitcanhelptobreakthemdownintotherootcausetoidentifyamanageablecomponentoftheproblemUsingafishboneexerciseorfivewhystechniquecanalsohelp
Gatherinformationabouttheproblem
10
tofindtherootcauseoftheproblemandavoidsimplyaddressingsymptomsoftheproblemandrepeatingthemistakesofhistoryToolssuchasjourneymapscanhelpyouunderstandtheexperienceofthosewhoareaffectedbytheproblem
TheCentreforPublicImpactwithsupportfromBloombergPhilanthropiesdevelopedanexercisetosupportcitiesinidentifyingtherightsizedproblemAccordingtoCPIarightsizedproblemisonethatislargeenoughtopotentiallyapplytoothercitiesbutsmallenoughtosolvewithinareasonabletimeframe
AtthebeginningofMayorBerkesadministrationintheCityofChattanoogaTNveteransexperiencinghomelessnesswereaskingtheCityforhelpinfindinghousingTalkingtothispopulationrevealedthatveteranshadtroubleidentifyingaffordablehousingwhichmotivatedtheAdministrationtoworkonasolutionTheproblemofveteransexperiencinghomelessnesswastherightsizebecauseitfocusedononesegmentofpeopleexperiencinghomelessness-veterans-andtheirspecificchallengeinidentifyingaffordablehousing
Developacomprehensiveunderstandingofhowyourcitycurrentlyaddressestheissue
IntheChattanoogaexamplementionedaboveCitystaffdiscoveredthatveteranswerehavingtroublenavigatingtheCitysvariousaffordablehousingprogramsandgettingaccesstothehousingandservicesthattheyneededAtthesametimenewlyelectedMayorBerkewasexploringcurrentandfutureopportunitiestoservethispopulationRatherthancontinuetousefederalfundingforemergencysheltersMayorBerkedirectedcitystafftoprioritize
Gatherinformationabouttheproblem
11
HousingFirstwhichvaluesconnectingpeoplewithpermanenthousingovertransitionalhousingandsupportiveserviceafterspeakingwithrepresentativesattheUSDepartmentofHousingandUrbanDevelopment(HUD)
ThisexaminationofwhattheCityiscurrentlyfundingandthesourcesofthatfundingwasaturningpointinChattanoogaUnderstandingthisproblemfromasystemsperspective-includingpoliciesandfundingstreamsatmultiplelevelsofgovernmentsocialnetworksthroughwhichinformationisoftenspreadtechnologythebuiltenvironmentofmarket-rateandpublichousing-helpedthenewMayoranalyzeexistingeffortsandleadstaffandotheractorsinthesamedirection
Begintogatherthedata
TotrulycapturethesuiteofservicesthatyourCityisprovidingaskdepartmentstafftolisttheprogramsrelatedtoyourproblemstatementthatyourcityrunsorfundsWhenlistingtheprogramscapturedetailssuchas
programnamethenameofthedatasetthatisrelevanttoeachprogramadescriptionofthedatasetthesourceofthedatasetifthedataisinternalorexternalwhetherthisdataisalreadybeingcollected
ThisinformationwillformthebasisofyourinventoryofadministrativedataordatasetsthatarekeptbygovernmentsforoperationalpurposesThismayincludecallcenterrecords(311911)taxdatapropertyassessmentsinspectioninformationsocialservicerecordsetcAtthispointlistallrelevantadministrativedatasetsYouwillprioritizewhichdatasetstogathermoreinformationonlaterwhenyouaredevelopingindicators
Asampletemplatethatcanbeusedtocapturethisinformationisbelow
ProblemStatement____________________________________________
Gatherinformationabouttheproblem
12
ProgramName
DatasetTitle
DatasetDescription DataSource
Departmentorexternalpartner
DoestheCityalreadycollectthisdata
Whatisthenameoftheprogram
Human-readablenameofthedataset
Shouldbeunderstandablebynon-technicalusers
Whatistheoriginalsystemapplicationfilethathousesit
Departmentorexternalpartnerthatmaintainsthedata
Isthisdataalreadycollected
TogatherthisinformationconsiderassigningonepointpersontosurveyorinterviewcitystaffSpeakingwithfrontlineworkersstaffwhoworkwithfederalgrantsdepartmentheadsandeveryoneinbetweenwillensureadeepunderstandingoftherelevantprogramsanddatasetsDuringthecourseoftheseconversationsbesuretoaskcitystaffwhichorganizationstheyworkwithexternallyandcapturethatinformationaswelleveniftheyarenotrecipientsofcityfundingInadditiontoconversationswithexpertsincitygovernmentyourcitysbudgetdocumentsareanothergreatsourceofinformation
Collectdataandobservetrends
Nexttrytogainanunderstandingofwhytheproblemstillexistsifithasbeensomethingyouhaveworkedonforawhileorwhyitisemergingnowifitisnew
AnalyzingadministrativedatacandemonstratepasteffectivenessofinitiativesrelatedtoyourproblemstatementUsingthedatasetinventorythatyoubeganintheprevioussteptalktodepartmentstaffaboutwhichdatasetsaremostrelevanttotheproblemandgatherafewyearsworthofhistoricaldataBegintolookfortrendsinthisdatadiscussingpotentialreasonsforthosetrendswithrelevantstaffCanthetrendsbeexplainedbyseasonalitychangesinfundingstaffingfluctuationsorsomeotherreasonBasedonthehistoricaldataanalysiscreateabaselineagainstwhichfuturechangescanbeobserved
OtherdatasetscanhelpprovideadditionalcontextForinstanceifthenumberofpeopleexperiencinghomelessnessinyourcityisincreasingitmaybehelpfultotakealookathousingaffordabilityIfyoudonothavethedatayouneedconsideraproxymeasureAsanexamplehousingisoftenconsideredaffordableifitcosts30orlessofahouseholdsincomeButifhouseholdincomedataisnotavailableatthecitylevelAreaMedianIncome(AMI)canbeahelpfulproxy
Gatherinformationabouttheproblem
13
QualitativedatacanalsohelpanswerquestionsaroundtheeffectivenessofpreviouseffortsTrytoobtainavarietyofperspectives--bothmainstreamandoutliers--fromnetworksbeyondyourowntoprovideyouwiththegreatestvantagepointforevaluatingpotentialimpactUnderstandinghowthoseeffortswerereceivedbydifferentpartsofyourcommunityandiftheremaybeelementsofprevioussystemstopreservewillbecrucialtothesuccessofanypolicychangeThisinformationmaybeobtainedthroughfocusgroupsinterviewsandsurveysFormoreinformationonhowtousethesequalitativeresearchtoolspleaseseetheseresourcesdevelopedbytheAgencyforHealthCareandResearchandUrbanInstitute
FinallytakealookatnationaldatatoseewhereelsethisproblemmightexistThisresearchmaybehelpfulinlaterstepswhenyouarelookingforpromisingpoliciesinothercities
Puttingitalltogether
UsethetemplatebelowdevelopedbyDenversPeakAcademytocaptureyourrefinedrightsizedproblemstatementandcurrentandfuturestatesWorkingthroughthisexercisewithpartnersishelpfultogainbuy-inTheresponseswillbeimportantwhengettingotherpartnersonboard
WhyChangeIsNeeded
Yourright-sizedproblem
CurrentState
Analysisoftrendsindata
FutureState
Ideallywhataretheoutcomesyouhopetoachieve
Checklist
Thekeystepstogatheringinformationarebelow
Atthispointfocusontheproblemratherthanjumpingtosolutions
Listentopeoplewhoareaskingforhelpandtoadvocateswhoareraisingissues
Beasspecificaspossibleinidentifyingtherootcauseandhoninginonthepartoftheproblemthatyourcitycanaddress
Identifytrendsbasedonmultiplesourcesofdataincludinghistoricaladministrativedataproxymeasuresqualitativedataandnationalmeasures
Gatherinformationabouttheproblem
14
Captureyourproblemstatementtheresultsofyouranalysisanddesiredoutcomes
Gatherinformationabouttheproblem
15
StrategicallyengagekeypartnersPolicytheoryresearchoftenfocusesonincreasingpolicymakersaccesstoevidenceasawaytoavoidtakingshortcutslikemakingdecisionsbasedonemotionImplementationsciencesuggeststhatincreasingaccesstoevidenceisnotenoughandthatpolicychangeshouldbeexaminedinthewidercontextofdebatecoalitionformingandpersuasionMeaningfulpolicychangerequiresldquoalong-termstrategytoformalliancesanddevelopknowledgeofpolicymakersandpolicymakingratherthanusingevidenceinpolicymakingonanadhocorone-offbasis
UnderstandingnetworksandsystemsisimportanttodevelopsustainablepolicychangeThissectionismeanttohelpreaderstakethestepstobuildengagementstrategiesforpartnersthatrepresentdifferentsystemsandnetworksEngagementshouldbeanongoingprocessofconnectingpeopletooneanotherandtoanissueEngagementshouldnotbeinnameonly-wehaveallexperiencedengagementactivitiesthatgivepeopleaspecificroletoplaythatleadstoaneatpredictableoutcomeAccordingtotheBuildWithmovementundervaluingandunderinvestinginengagementstrategiescanleadtoadisconnectofcertainindividualsorgroupsandmistrustofgovernmentWhendevelopingengagementactivitiesrememberthathowpeopleengageandwhethertheyengagehastodowithhowyoustructureopportunitiesforengagement
Conveneacoreadvisorygroup
AnadvisorygroupcomprisedofpeoplefrommultiplesectorscanbehelpfulasyouconsiderthelocalsocialenvironmentalandpoliticalcontextinwhichyouintendtointroducepolicychangeRevisityourstakeholdermapanddeterminewhowouldbebesttoserveonthisgroupTomakesureyouareconsideringtheproblemsandsolutionsthroughanequitylensincludepeoplewithlivedexperiencewhohaveexpertiseintheeffectivenessofsystemswhetherthosesystemsfunctionasintendedandhowtheymayimpacthumanbehaviorYoumayalsowishtoincludesomeonewithexperienceworkingwithdataandanalysis-thispersoncanhelpdeterminethetypeofdatathatmaybereadilyavailableandusefultocollectandcanalsohelpplanandshapeyouranalysistofigureoutwhichprogramsareeffectiveBesuretoinvitefront-lineworkersaswell-itscriticalthatwhateversolutionscomeoutofthisprocessaremeaningfultothosewhowillberesponsibleforimplementingthem
BringingpeopletogethercanbechallengingbutthereareafewwaystosetandmanageexpectationsFirstmakesurethatthegroupisawareoftheproblemstatementcurrentstateandfuturestate-thiscanhelpstarteveryoneoffonthesamepageNextspend
Strategicallyengagekeypartners
16
sometimegettingtoknoweachotherrsquosworktolearnabouteveryonesstrengthsThissharedunderstandingcanhelpasyouclarifyrolesandresponsibilitiesofeachpartnerinthegroup--thisisimportanttoavoidfinger-pointinginthefutureandtoensurethatnoneofthetasksslipthroughthecracksInsomecasestherearemanyactorswithdifferentresponsibilitiesasinthecaseofBaltimoreCitys38-memberYouthFundTaskForcewhichischairedbynonprofitandcorporatepartnersandincludesrepresentativesfromthenonprofitandfaithcommunitiesstudentsthebusinesssectorlocaluniversitiesCityCouncilandtheMayorsOffice
OncerolesandresponsibilitiesareclearyoumayconsiderdevelopingacharterguidelinesortermsofreferencetosetexpectationsforthegroupBesuretodistinguishthisfromotherboardsandcommissionsthatyourcitymayhaveNextyoumaywishtoestablisharegularongoingmeetingscheduleBesuretosendagendasandmeetingmaterialsaheadoftimeandclarifytheobjectiveofeachmeeting
InSeattletheCitydevelopeditsHousingAffordabilityandLivabilityAgenda(HALA)torespondtorisinghousingcostsandhomelessnessThisisatrendinotherwestcoastcitiesaswellwhichmaybeexplainedbyaninfluxofnewresidentswhoareattractedbytheemergenceoftechjobstheattractionoftheoutdoorlifestyleortherecentpassageofprogressivepolicieslikelegalizedgaymarriagea$15minimumwageandlegalizationofmarijuanaSeattlebroughttogetheramulti-stakeholderadvisorygroupcomprisedoflocalhousingexpertsrentersandhomeownersdevelopersandgovernmentofficialstoshareinformationabouttheirneedsanddeveloprecommendationsbasedontheseneedsTheinterestsandmotivationsofthisgroupwerevariedandittookabouttenmonthstofindcommongroundTheseriesofrecommendationsthattheadvisorygroupcreatedismeanttoserveresidentswithaspectrumofneeds
Buildaprocessforcommunityparticipation
Ofcoursenoteveryonecanbeincludedinanadvisorygroupandadvisorygroupmembersmaynotnecessarilyrepresentthevariedperspectivesoftheirpeers-forinstancenoteveryhomeownerhasthesamesetofneedsTogatherinputfromawiderswathofthepopulationitcanbehelpfultobuildaprocesstoinvitethepublicatlargetosharetheirideasCitiesdothisinavarietyofwaysfromonlineconversationsandsurveystoin-personmeetingsRememberthatbeingintentionalaboutgatheringtheviewpointsofpeopleismoreimportantthanusingthelatesttechnologyandtools
TheinsightgeneratedbycommunityparticipationexercisescanbeincludedinyourreviewofevidencewhichisdiscussedinasubsequentchapterInWestVirginiaastatethathasbeenhithardbytheopioidepidemicstatehealthofficialsdraftedanopioidresponseplanthatwasinformedbypublicengagementaswellaslocalandregionalsubjectmatter
Strategicallyengagekeypartners
17
expertsThepublicwasinvitedtopresentcommentsonideasfordecreasingoverdoseearlyinterventiontreatmentandrecoveryPublichealthandmedicalfacultyfromWestVirginiaUniversityMarshallUniversityandJohnsHopkinsUniversityprovidedresponsestothecommentsTheresultwasanOpioidResponsePlanthathealthcommissionerRahulGuptapresentedtoGovernorJimJustice
AsamplespectrumtoplanyourowncommunityparticipationprocesscreatedbyLaurenellenMcCannisbelow
SourceLaurenellenMcCann
Community-basedparticipatoryresearchisanotherwaytoworkwiththecommunitytoimproveoutcomesCBPRmeansthecommunityisinvolvedinallaspectsoftheresearchprocesswhichisofteniterativeInCBPRtraditionalresearchersandcommunitymembersshareknowledgeaswellasresourcesandcredit
CBPRInActionInSanFranciscoanenvironmentaljusticenonprofitteamedupwiththeCitysDepartmentofPublicHealthandanoutsideevaluatortounderstandtherootcausesofobesityandfoodinsecurityinlow-incomeneighborhoodsandidentifysolutionsThroughresearchmethodssuchasinterviewingresidentsandstoreownersdiagrammingstoreshelvesGISmappinganeconomicanalysisandreviewofsimilarpoliciesinnearbycitiestheteamobservedthatthestoresthatresidentshasveaccesstowerestockedwithlesshealthyfoodsandthatresidentswantedmorehealthyfoodandlessalcoholandtobaccoThisprojectresultedinlocalandstatelegislationtoencouragestorestostockfreshproduce
Strategicallyengagekeypartners
18
Developrelationshipswithsubjectmatterexperts
OnewaytokeepsubjectmatterexpertsengagedistoinvolvetheminthecommunityparticipationprocessasdescribedaboveIntheWestVirginiaexamplesubjectmatterexpertsrespondedtoandsynthesizedthepubliccommentsOtheropportunitiesaredescribedbelow
AdvisoryAcitydepartmentheadMayororstaffmembermaydevelopaone-on-oneadvisoryrelationshipwithuniversityfacultymemberoracollaborativebetweentheCityandUniversitytoaddressaparticularproblemAgreatplacetostartistolookforrelevantpracticecenterswithinyourlocaluniversityandreachingouttosubjectmatterexpertsInadditiontotheirexpertiseontherelevantpolicytopicheorshemayalsobeabletoidentifyotherexpertsandevenprovidesupportonhiringtherighttalenttoleadcertainprogramsAnadvisoryrelationshipwithauniversitysubjectmatterexpertcanalsohelpwhencitiesareconsideringbringinginnewprogramsTheseexpertsareoftenfamiliarwithprogramsthatarebasedonnationalmodelsandcanprovideinsightintoaprogramsoriginandlineageeffectivenessinotherplacesandthepotentialforimplementationinyourcityAcademicscanalsohelpidentifyindicatorsbasedongovernmentindicatorsorliteratureFinallyrelationshipsbetweenpolicymakersandresearchersprovidethemostbroadlyeffectiveroutetopullingresearchintopolicy
ResearchandevaluationAcademicexpertsmaysupportcitiesinevaluation-indeedevaluatingcityinitiativesisatraditionalmodelofcityuniversityrelationshipsSharingtheinformationyoucollectedintheGatherInformationphasecanhelpresearchersunderstandyourworkandgoalswhichinturnwillstrengthenthesupportthattheycanprovideDiscussingdetailsaboutyourinterventionandevaluationplanatthebeginningoftheprocessmeansthataresearchpartnercanhelpwiththeevaluationdesignanddatacollectionForinstancetheWilsonSheehanLabforEconomicOpportunitieswalkscitiesthroughconsideringthefollowingquestionsbeforepartneringonanevaluation
1 Isthereaclearpolicyprogrammodelwithobjectivescreeningcriteria
2 Isitpossibletoevaluatewhathappensbothwiththepolicyprogramandintheabsenceofthepolicyprogram
3 Arethereoneormoremeasurableoutcomesofinterest
4 Arethereenoughpeopleforrigorousquantitativeanalysis
5 Whatdatawillbenecessaryforthestudyandwhataretheprocessesforaccessingthesedatasource(s)
6 Isthereacompellingreasontoevaluate
Strategicallyengagekeypartners
19
InadditiontoassessingreadinesstoconductanevaluationresearcherscanhelpyouhoneinonthepurposeoftheevaluationTheWorldBankdescribesseveralpotentialusesofevaluationwhicharedefinedhereInbriefevaluationcanhelpdecisionmakers
makeresourceallocationdecisionsrethinkrootcausesofproblemsidentifynewproblemsdecideonabestalternativedeterminewhetherprogramsorpoliciesarebeingimplementedwithfidelitydeterminewhetherprogramsorpoliciesarehavingthedesiredimpact
Researcherscanalsoprovidesupportinsharingtheresultswiththescientificcommunitythroughpapersandconferenceswhichisvaluableincreatingapeernetworkofevidence-basedcitypractitioners
TrainingandtechnicalassistanceUniversitypartnersespeciallythoseatpracticecenterscanbehelpfulindevelopingorupdatingtrainingcurriculumtrainingthetraineranddeliveringtrainingtocitystaff
Strategicallyengagekeypartners
20
ReviewtheevidenceandcollectdetailsOnceyouhaveathoroughunderstandingoftheproblemdataandactorsyouwillneedtoconductyourownresearchonavailableevidenceandlearnaboutthelandscapeofpolicyoptionsYourgoalwillbetodevelopanaccessiblelistofinformationaboutpoliciesthathavehelpedsolvesimilarproblemsinotherplacesThischapterwillguideyouthroughthestepsofpreparingyourreviewersunderstandinglevelsofevidenceandcollectingthedetailsonevidencethatyouwillneedfordiscussionwithdecisionmakersinyourcity
Selectandprepareyourreviewers
AssignapersonorteamtoreviewrelevantinterventionsthathavedemonstratedsuccessinotherjurisdictionsTheirgoalwillbetoproduceareviewofalloftherelevantprogramspoliciesorinitiativesthatyourteamcouldconsiderimplementingTheywillbeconductingaliteraturereviewsupplementedbyinterviewswithexpertsandpeerswhohaveinformationaboutpotentialprograms
SincethepeopleyouassigntothistaskareprovidingthebaselineofpotentialoptionstoimplementitisimportantthatyouchoosepeopleyouareconfidentwillbethoughtfulandthoroughintheirworkItisalsocriticalthattheyareawareoftheimportanceofsourcecredibilityEnsureyourreviewershavefamiliarizedthemselveswithrulesforcriticallyanalyzinginformationsourcesbeforetheygetstartedIftheyareincludingnewsarticlesaspartoftheirsearchtheyshouldalsobeawareofmethodsforevaluatingnewscredibility
AskyourreviewerstodevelopaplanandtimelinetocompletethereviewThisstartswithsolidifyingagoverningresearchquestionOnewaytoknowyouarecreatingadetailedresearchquestionistousetheacronymPICOremindingyoutoincludereferencetothepopulationintervention(ifyouhaveoneinmind)comparisonandoutcomeelementsinyourresearchquestionForexampleifyouknowthatyouhaveapopulationofteenagerswhoarehomelessyouhavelearnedfromyourstakeholdersthatmanyoftheseteenagersidentifyasLGBTandyouhaveheardsomethingabouttheapproachofcreatingdrop-incentersfordeliveringsupportyoucouldask
ForLGBTyouthfacinghomelessness(population)howeffectivearedrop-incenters(intervention)relativetostandardsocialservices(comparison)atsettlingtheminpermanenthousing(outcome)
Ifyoudonothaveaninterventioninmindyoucanleaveouttheinterventionandcomparisonandsimplyask
Reviewtheevidenceandcollectdetails
21
HowcanwehelpLGBTyouthfacinghomelessness(population)obtainpermanenthousing(outcome)
OnceyourreviewershavetheirresearchquestionsusethequestionstocreatealistofkeywordstheycanuseintheirliteraturesearchThiswillhelpkeeptheircollectionofarticlesreasonablyfocusedHavethemdescribewhattheirsearchprocesswillbeincludinglistingthedatabasestheywillreviewandtheirplanforstoringtheinformationtheywillcollect(Herearesomeadditionalsuggestionsforstructuringareviewplan)
WhatyourreviewerswillbedoingisalightweightversionofarapidreviewTotheextentthattheywouldliketodevelopamethodologyforspecificsearchprotocolsincludingcriticalappraisalprocessestheycancertainlygoaheadanddoso
Usedifferentlevelsofevidenceforthemostcompletepicture
AsdiscussedintheWorkingwitharangeofevidencechaptertheNestaStandardsofEvidenceframework--reproducedbelow--ishelpfultounderstandthelevelofconfidencethataprogramorinterventionishavingapositiveimpactThisapproachletsyouputinterventionsthatyouhaveheardaboutlocallybutwhichhavenotbeenstudiedtogetherwiththosewithademonstratedcausaleffectThisishelpfulsinceanumberofpoliciesprogramsandpracticesthatyouwillwanttoconsidermaynotyethavemuchofaresearchrecord
Reviewtheevidenceandcollectdetails
22
NearlyanypieceofinformationaboutaninterventioncanbeclassifiedwithinthestandardsofevidenceForexampleifsomeonehasacrediblehypothesisabouthowaninterventioncouldproducebenefitthistypeofevidencewouldbeclassifiedasLevel1ThistypeofevidencecouldincludeinterventionsinothercitiesthatyoureadaboutinthenewsorhearaboutataconferenceForinstancewhenUSDAallowedschoolsandschooldistrictstoprovidefreebreakfastandlunchtoallstudentsthroughtheCommunityEligibilityProvision(CEP)manydistrictsimplementedthisoptionevenbeforetherewasevidenceofeffectivenessThiswouldfitintoLevel1sinceitbuildsonthelogicthatprovidingaccesstomealswilldecreasefoodinsecurity
YoumayalsofindpiecesofinformationthatqualifyasLevel2evidenceForinstanceasmoreschooldistrictsimplementedCEPprincipalstestifiedaboutpositivechangestheyobservedinstudentssuchasamorepositivedemeanorthatcanleadtobetteracademicperformanceDataalsoshowedthatthestudentsdescribedwerereceivingmoremealsthanbeforetheprogramsimplementationHoweveritwastoosoontobuildevidencetoshowacausalrelationshipbetweentheincreaseinfreemealsandincreasedacademicperformance
ldquoLower-levelrdquoevidencecanconsistofnuancedbutvaluableelementsthatrelatetoaninterventionrsquospotentialfitwithacommunityQualitativeinformationsuchastestimonyfrompeoplewhohaveexperiencedaninterventionandthosewhohavefacilitatedorimplementeditwillalwaysbeusefultoconsiderandprovidesdetailedlocalcontext
Experimentalstudies--describedstartingatLevel3withcontrolledexperimentsandgoingthroughLevel5withfullyfleshedoutprotocolsandprocessesforreplication--stilloccupythehigherlevelsofthisdescriptionofevidenceThisprioritizationshowsthevalueofrigorousstudyforevaluatingtheimpactofaparticularpolicyorprogramratherthanotherfactorsthatcouldbeinfluencingoutcomesofinterestRememberthatmultipletypesofevidencecancomplementrigorousimpactevaluationsandgivecityleadersadditionalinsightsandbroaderperspective
Accessinformation
Encourageyourreviewerstobegintheirsearchforpromisingpracticesingovernmentandnon-profitclearinghousesandinacademicpolicyjournalsWorkpublishedinthesesourceshasalreadybeensubjectedtoahighlevelofprofessionalscrutinyandgenerallycontainsagreatdealofdetailwhichmeansthatyourreviewersaremorelikelytobeabletopullouttheinformationyouneedtomakeagooddecision
Clearinghouses
Reviewtheevidenceandcollectdetails
23
Herearesomeoftheclearinghouses(collectionsofprogramevaluations)forevidence-basedpoliciesandprograms
ResultsFirstClearinghouseDatabaseThePewCharitableTrustsCollectstogethereightotherclearinghousestoprovideinformationoninterventionsinthefollowingcategoriesadultcriminaljusticeeducationchildwelfarehealthjuvenilejusticementalhealthsubstanceabuseandyouthdevelopmentIndividualclearinghousesincludedinResultsFirstare
BlueprintsforHealthyYouthDevelopment-foryouthdevelopmentandviolenceprevention
CaliforniaEvidence-BasedClearinghouse-forchildwelfareinterventions
TheLauraandJohnArnoldFoundationrsquosSocialProgramsThatWork-forsocialservicesprograms
TheUSDepartmentofEducationrsquosWhatWorksClearinghouse-foreducationinterventions
TheUSDepartmentofHealthandHumanServicesrsquoResearch-TestedInterventionPrograms(RTIPs)-forcancercontrolinterventions
TheUSDepartmentofHealthandHumanServicesrsquoTeenPregnancyPreventionEvidenceReview(TPPEvidenceReview)-forsexualriskbehaviorandsexualhealthoutcomes
TheUSDepartmentofJusticersquosCrimeSolutionsgov-forcriminaljusticejuvenilejusticeandcrimevictimservices
TheUSSubstanceAbuseandMentalHealthServicesAdministrationrsquosNationalRegistryofEvidence-basedProgramsandPractices(NREPP)-substanceuseandmentalhealthinterventions
TheUniversityofWisconsinPopulationHealthInstituteandRobertWoodJohnsonFoundationrsquosCountyHealthRankingsandRoadmapsWhatWorksforHealth(WhatWorksforHealth)-healthbehaviorsclinicalcaresocialandeconomicfactorsandthephysicalenvironment
UKWhatWorksCentresTheBritishgovernmenthascharteredanetworkofsubject-specificorganizationswhichcollectsynthesizeandreportoutevidenceontheeffectivenessofpolicyprogramsandpractices
NationalInstituteforHealthandCareExcellence(NICE)-healthandsocialcare
Reviewtheevidenceandcollectdetails
24
SuttonTrustEducationalEndowmentFoundation-educationalachievement
CollegeofPolicingWhatWorksCentreforCrimeReduction-crimereduction
EarlyInterventionFoundation-earlyintervention
WhatWorksCentreforLocalEconomicGrowth-localeconomicgrowth
CentreforAgeingBetter-qualityoflifeforolderpeople
WhatWorksCentreforWellbeing-wellbeing
WilsonSheehanLabforEconomicOpportunities(LEO)UniversityofNotreDameProvideslinkstotheirevaluationsofinterventionsincriminaljusticeself-sufficiencyeducationhealthandhousingandhomelessness
ClearinghouseforLaborEvaluationandResearchUSDepartmentofLaborProvidesresearchonworkforceandlabor-relatedpoliciesandprograms
J-PALEvaluationsAbdulLatifJameelPovertyActionLabStudiesonpoverty-reducinginterventionsfromaroundtheworld
CampbellSystematicReviewsCampbellCollaborationSystematicreviewofpolicyinterventionsbyscholarlycoordinatinggroupsinsevenissueareascrimeandjusticeeducationinternationaldevelopmentmethodssocialwelfareknowledgetranslationandimplementationanddisability
Benefit-CostResultsWashingtonStateInstituteforPublicPolicyAWashingtonState-charteredprogramtoestablishthelikelihoodofrelativecostsandbenefitsforprogramsacrossthefollowingareasjuvenilejusticeadultcriminaljusticechildwelfarepre-Kto12educationchildrensmentalhealthhealthcaresubstanceusedisordersadultmentalhealthpublichealthandPreventionworkforcedevelopmentandhighereducation
EffectiveInterventionsCentersforDiseaseControlandPreventionRepositoryforevidence-informedHIVpreventioncareinterventions(withstepsonhowtoimplement)
Journalsandindexes
OnceyouhavereviewedrelevantinformationaccessiblethroughclearinghousesitisworthtakingalookatindividualstudiespublishedasscholarlyarticlesYourreviewerswilldothisbysearchingarticleindexesandacademicjournalsUnfortunatelyanumberofusefulacademicjournalsandindexingplatformsmaynotbegenerallyaccessibletoyourreviewersifyoudonothaveasubscriptiontothemToaddressthisconsiderpartneringwithalocal
Reviewtheevidenceandcollectdetails
25
collegeoruniversitytoprovideyouwithalistofarticlesfromthefollowingsources(Thisisalsoanexcellentopportunitytoseekresearchsupportfromanyacademicstakeholdersyouidentifiedpreviously)
Indexes
GoogleScholarincludesmostpeer-reviewedonlineacademicjournalsandbooksconferencepapersthesesanddissertationspreprintsabstractstechnicalreportsandotherscholarlyliteratureincludingcourtopinionsandpatentsItallowsyoutospecifyapublicationrangeinyoursearchandprovideslinkstolikelysourcesforalllistedarticles
PolicyFileIndexisauniqueresourceforUSpublicpolicyresearchinthatitgrantsusersaccesstotimelyupdatedinformationfromover350publicpolicythinktanksnongovernmentalorganizationsresearchinstitutesuniversitycentersadvocacygroupsandotherentities
PolicyArchiveworkstocollectanddisseminatesummariesandfulltextsvideosreportsbriefsandmultimediamaterialofthinktankuniversitygovernmentandfoundation-fundedpolicyresearch
Anothersourceforhigh-qualityresearchisthereportsoftheCongressionalResearchServiceYoucanfindtheirreportsaggregatedhere
Specificjournals
Oftenthemostusefulacademicjournalsforunearthingpolicyorprogramreviewswillbeineachspecificsubjectarea--thatisineducationjournalsforeducationpoliciesYoucanuseageneralindexlikeGoogleScholartobegintolearnwhichjournalsaremostoftenusefulforyoursearchHoweveryoucanalsosearchthefollowingtwogeneralistpolicypublicationsforissuesandarticlesrelatedtoyourtopic
PublicAdministrationReviewJournalofPolicyAnalysisandManagement
Newssourcessubjectmatterexpertsandinterviews
AlthoughitisvaluabletoknowwhathasbeenexploredthroughascholarlyresearchframeworkitisalsousefultolearnwhatkindofinterventionsarebeingtriedinthefieldrightnowandhowpeoplefeelaboutthesetrialsSearchsubscriptionnewsaggregatorslikeLexisNexisorfreenewsaggregatorslikeGoogleNewsorYahooNewsforyourissueareaplusthewordpolicyorprogramtoseewhatkindsofinterventionsarebeingcoveredinthenews
Reviewtheevidenceandcollectdetails
26
DonotforgettocontacttheadvisorygroupmemberswhowerechosenfortheirsubjectmatterexpertiseInadditiontoprovidinggoodmaterialthroughinterviewsthemselvestheymayhaveideasaboutothersubjectmatterexpertsyoushouldcontacttoaskaboutpotentialinterventions
Peercitiesandnetworks
InadditiontoreadingabouttheworkofothercitiesitisidealifyouareabletospeakwithpeoplewhoareinvolvedwithcurrentimplementationsofinterestinginterventionsInyourcityyoulikelyhaverelationshipswithyourneighborsbutitcanbehelpfultolookoutsideofyourregionforsolutionsaswellTofindcounterpartsinothercitieswhoareworkingonaprogramyouwouldliketolearnmoreaboutyoucancontactstaffinintermediaryorganizations(likeGovExoryourprofessionalassociations)whomightbeinapositiontoputyouintouchimmediatelyorwhomightseeanothergoodrouteforsettingupaconversationHereisalistofnetworksthatmaybehelpful
InternationalCityCountyManagementAssociation(ICMA)NationalAssociationofCounties(NACO)NationalLeagueofCitiesUSConferenceofMayorsWhatWorksCities
Collectthedetails
Onceyourreviewershavefoundevidenceonpossiblepoliciesorprogramstoimplementtheyshouldcapturekeydetailsthatwillhelpdecisionmakersunderstandthepotentialeffectivenessandapplicabilityofeachpotentialintervention
Foreachsourcethathasprovidedinformationyouwanttoconveyusethetemplatebelowtocaptureasmanyofthefollowingdetailsaspossible(Youdonotneedtocompletethisforsourcesthatprovideduplicativeinformationorareotherwisenotthatuseful)
Programorpolicyname (egldquoHousingFirstrdquoorldquoblightedpropertiesregistryrdquo)
Sourcetype (egjournalarticleinterviewsubjectmatterexpertstudyassociationcommunityparticipationprocess)
Sourcedetails (egarticletitleandpublicationnameandtitleofinterviewee)
Evaluationmethodology
(egrandomizedevaluationcomparativeevaluationoftwodifferentgroupsbeforeafteranalysis)
Evidencelevel (egLevel12345)
Programorpolicy
Reviewtheevidenceandcollectdetails
27
description
Programorpolicygoal (1sentencecapturingtheprincipalaimoftheintervention)
Existingimplementations
(WherehastheinterventionbeentriedListasmanysitesasyouareawareof)
Characteristicsofimplementationsites
(ForeachimplementationsiteofferbasicgeographicIDmdashcityandstateThenyouwillwanttocollectthedetailswhicharerelevanttoyourdecisionmakersrsquoconsiderationsaboutwhatmakesalocationldquosimilartordquoorldquodifferentfromrdquoyoursituationYouwillatleastwanttoincludeforeachimplementationsitePopulationnumberPopulationdensity(ruralurbansuburban)Socio-economiccharacteristicsBeforestartingaskyourdecisionmakersaboutanyparticularcharacteristicsofprogramimplementationsitestheywouldwanttoknowbeforeseekingtoreplicate)
Characteristicsofimplementationpartners
(Anyavailabledetailsaboutorganizationsinstitutionsorpeoplewhowereinvolvedwithimplementingtheintervention-egindependentchildcarecenterpublicelementaryschoolcitypublichealthnurses)
Characteristicsoftargetpopulationeligibleprogramparticipants
(Anyavailabledetailsaboutpeoplewhoarethebeneficiariesoftheprogramorotherwiseincludedoraffectedbytheintervention-egnumberofprogramparticipantscategoriesofpropertyownersaffected)
Policyorprogramduration
(HowlonghastheinterventionbeeninplaceineachimplementationsiteHasitendedandifsowhy)
Whatbenefitscamefromeachimplementation
(Anyobservedpositiveeffectsoftheinterventionincludingeffectsize)
Statisticalestimatesofinterventioneffectiveness
(Anyavailablestatisticalestimatesoftheinterventionrsquoseffects)
Whatdrawbackswereobserved
(Anyavailabledetailsaboutunintendedorunavoidablenegativeeffectsoftheintervention)
Programorpolicycosts
(Anyavailabledetailsaboutprogramcostsperimplementation)
Reviewtheevidenceandcollectdetails
28
Reviewtheevidenceandcollectdetails
29
DecideonapolicytoimplementbasedonlocalfactorsTogetthecriticalbuy-infromleadershiptoimplementanewpolicyorprogramtheprojectleadshouldscheduleameetingwithdecisionmakerstopresenttheevidencesurroundingthepolicyoptionscapturedinthetemplateDependingonyourinsightintoyourcityleadershipswillingnesstomakechangeyoumaywishtosharetheresearchyouhaveconductedontherootcausesoftheproblemandsomeofyouranalysisontheimpactofcurrenteffortstodate
Considertheapplicabilityofanapproach
AsdiscussedinthepreviouschapterpolicymakersoftenlookforevidenceofeffectiveprogramsinothercitiesbutoftenthequestionarisesofhowgeneralizableaprogramisfromonecitytoanotherTheAbdulLatifJameelPovertyActionLab(J-PAL)suggeststhatinreviewingevidencefromanotherlocationitishelpfultoidentifythespecificmechanismthatwasused(deliveryofprogramorserviceinformationsharingetc)identifythelocalconditionsthatwereinplaceobservethebehaviorchangethatresultedfromthemechanismandfinallytounderstandlocalcontextandconditions(legalorcostconstraintsculturalnormsetc)Thissetofpresentationelementscanbedrawnfromthelistofdetailscollectedintheresearchtemplate
EducationpolicymakersinChicagoILusedfindingsfromprogramsinKenyaandIndiainwhichchildrenwhohadfallenbehindtheirpeersweregiveneducationalsupporttargetedtotheirleveloflearningThemechanismusedwastargetedteachingandthereweresimilarconditionspresentinallthreecountriesAlthoughsomestudentshadfallenbehindinthecurriculumteacherswereincentivizedtoteachgrade-levelmaterialsConsiderationsofhowthelocalcontextinChicagomightaffectimplementationofatargetedteachingprogramincludedunderstandinghowtheneighborhoodsthatthestudentswerefrommayhaveaffectedtheireducationaloutcomesandthehighcostofteachinginsmallgroupsInChicagoMatchEducationprovidedtwo-on-onetutoringwhichmatchedstudentswithwell-educatedindividualswhohavenothadformalteachertrainingforalower-costmodelTheresultswerepositiveStudentswhoweretutoredperformedbetterinmathandweremorelikelytopasshighschoolclasses
ThegeneralizabilityframeworkproposedbyJ-PALisbelowUsethisasatemplateasyouconsiderpotentialpolicysolutions
Decideonapolicytoimplementbasedonlocalfactors
30
SourceTheGeneralizabilityPuzzle
OnceyouhavenarrowedthepolicyoptionsdiscussthequestionsbelowforeachoptionsyouareconsideringThisstepiscriticalasunderstandinglocalcontextissoimportantinsuccessfulpolicyimplementation
CommunityContextQuestionnaire
Localcontext
DoesthisoptionmakesenseinourcommunityIsthisaonesizefitsallapproachIsanyparticulargroupabsentfromthisapproachIfsoisthatok
Youranswers
Effectiveness
WhatwouldweexpecttoobserveifthepilotisaseffectivehereasinCityX
Youranswers
Timing
DoesthetimingmakesenseIsthistherighttimetoimplementthispolicychange
Decideonapolicytoimplementbasedonlocalfactors
31
WherearethepotentialroadblocksthatcouldaffectthescheduleIsthereanywaytomitigatethisHowlongwouldittaketoseeanimpact
Youranswers
Implementation
DowehavethestaffcapacitytoimplementthisArethereadditionalpartnersthatyoucanbringtothetabletosupportimplementationDoesthismakesensegivenourformofgovernment
Youranswers
Authority
DoesourcityhavetheauthoritytoenactthisIfsowhatistheappropriatemechanismAretherelegalbarriersArethereanyexistingpreemptionissuesfromthestateorfederalgovernmentCanthosebeaddressedthroughlegislationadvocacyIstherepoliticalsupportoroppositiontothis
Youranswers
Cost
HowmuchwillthiscostannuallyHowwillwepayforitCanthepolicybeimplementedwithexistingfunding
IfsoisthisfundingreliableHowwouldthispolicyoptionaffectourcitysbudgetintheshortandlongtermWhatotherfundingoptionscanweconsiderIsthereanalternativethatwouldcostlessandgiveussimilarresultsWillanyconstituencybeadverselyaffectediffundingforprogramsthatimpactthemisshiftedtoanotherprogrampolicyorinitiative
Youranswers
Decideonapolicytoimplementbasedonlocalfactors
32
EstablishaframeworktomonitorprogressNowthatyouhavereviewedtheevidenceconsideredlocalcontextandchosenyourpolicyitistimetodeterminehowyourcitywillmonitorprogressofthisnewapproachCitystaffandleadersneedtoestablishtheinfrastructureforongoingcontinuousdecisionmakingbasedonwhetherthepolicychangeisleadingtothedesiredoutcomesforresidentsandwhetherthechangeismeaningfulandrationaltofrontlineworkersSettingarbitrarygoalsatthelocallevelthatconflictwithgoalsatotherlevelsisconfusingtotheindividualsresponsibleforachievingthegoalsGoalsthataresettoohightoachievecanbedemoralizingtostaff
ThissectionincludesstepstosetupavenueinwhichCitystaffandleadershiproutinelyvisitkeyindicatorstodeterminewhatisorisnotworkingandtomakedecisionsinreal-timeFormorereadingonperformancemanagementpleasevisittheGovExPerformanceManagementGettingStartedGuide
InperformancemanagementtermssuchasindicatormeasureandmetricmaybeusedinterchangeablyInthisguidewewilluseindicatorInyourworkthemostimportantthingistobeconsistentinwhichevertermyouuseThisvideomaybehelpfulinunderstandingthisconceptandcommunicatingittoothers
Establishthefundamentalsandgatheryourteam
FirstdeterminewhetherthereisanestablishedvenueorregularmeetingscheduleduringwhichyoucandiscussprogressIfyourcityhasanexistingperformancemanagementstructureonarelatedtopicaddingadiscussionoftheprogressofpolicychangetoyouragendacanbeagreatideaIfthiswillbeastandalonemeetingdecidehowfrequentlytheprojectteamanddecisionmakerswillmeetandputthosemeetingsonthecalendarfortheentirelengthoftheproject
DeterminewhowillstaffandattendroutinemeetingsonCityprogressThesemeetingsmayincludetheMayorandorCityManagerdepartmentheadstaffwhoareimplementingthepolicychangeandanalysts
Identifyindicators
ItiscriticalthatindicatorsbeanimportantcityandcommunitypriorityaswellassomethingtheCityplaysamajorroleinaddressingTheremaybeinstancesinwhichtheCitywillneedtoreachouttoorcollaboratewithexternalpartnerstogetamorecompletepictureofprogressontheoverallpolicychange
Establishaframeworktomonitorprogress
33
Anoutcomeindicatordescribesthedesiredeffectofthetargetedpopulation
ExampleBellevuesfamilyfriendlyneighborhoodsindicatormeasuresthepercentofresidentswhobelievetheyliveinaneighborhoodthatsupportsfamiliesparticularlythosewithchildren
WithyourkeypartnersrevisityourproblemstatementcurrentstateandfuturestateexerciseandbegintoidentifytheoutcomeindicatorsthatwillillustratesuccessFornowchooseasingletop-levelkeyperformanceindicator(KPI)ItiscriticalthattheKPI-thepieceyouwillmeasure-beapriorityforthecommunityandsomethingtheCityplaysamajorroleinaddressing
Outputindicatorsmeasuretheprocessofprogramorservicedelivery
ExampleSanDiegoMayorKevinFaulconersgoaltorepair1000milesofstreetby2020
ReviewingtheadministrativedatasetinventorybegintoparedowntherelevantprogramsanddatasetsthatsupportyourKPIWhatarethemeasuresofeffectivenessforeachprogramTheseareyouroutputindicators
Thediagrambelowillustratestherelationshipbetweenoutputandoutcomeindicators
ThinkaboutimplementationindicatorsasmeasuringstepsintheprogramsrolloutthatarenecessarytoobtaindesiredchangesImplementationoutcomescanfocusontheinterventionstaffdeliveringtheinterventionanorganizationorsystemandcanaddressfeasibilityadaptabilitysustainabilityandcostTwokeyquestionstoconsiderwhendevelopingimplementationindicatorsare
1 Istheinterventionacceptableandappropriatetothecommunityand
2 IsitfeasibletoimplementinterventionXatthistimewiththeseresources
Establishaframeworktomonitorprogress
34
Amoreextensivelistwithdefinitionsofpotentialimplementationoutcomesislinkedhere
UsethetablebelowtochoosethedesiredimplementationoutcomesforyourpilotandtoplanyourmeasurementtechniqueThisshouldincludelevelofanalysis(individualproviderorconsumerorganizationorsetting)yourmeasurementtechnique(surveysobservationinterviewfocusgroupsadministrativedataanalysis)andthephasewhenyouwillcollectthisinformation(earlymidlate)Theremaybemorethanonelevelofanalysismeasurementtechniqueandphaseofdatacollection
Implementationoutcome
Definitionofoutcome
Levelofanalysisandmeasurement
Yourmeasurementtechnique
Phaseofpilotwhenyouwillcollectdata
Usethespacebelowtocaptureyourindicatorwishlist
Indicatorwishlist
Outcomes_____________________________________________________________________________
Outputs________________________________________________________________________________
Implementationoutcomes______________________________________________________________
Setapublichigh-levelgoal
Thehigh-levelgoaliscriticalinestablishingdirectiontogalvanizethepeoplewhowillbedoingtheworkWhendiscussingthegoalusetheMayorsbullypulpitLookforkeymilestonesinCitygovernmenttosharethegoalstatementsuchasthebeginningofaMayoraladministrationaStateoftheCityspeechorotherpublicremarks
GoalsshouldbeldquoSMARTrdquo
Specificaspossible(innamingthepopulationwhowillbebetterofftheactionthecitywilltake)
Measurable
Achievable
Establishaframeworktomonitorprogress
35
Results-focused
Time-bound
ThegoalstatementshouldincludeyouroutcomeindicatorlikeChattanoogasgoalldquoWewillendchronicveteranhomelessnessbyDecember2016rdquoAdiagramtohepyoubuildaSMARTgoalaroundyouroutcomeindicatorisbelow
Component Description Yourturn
Specific WhowillbebetteroffWhatactionthecitywilltake
Measurable Howwillyoumeasuresuccess
Achievable Isthegoalattainable
Results-focused Whatistheexpectedresult
Time-bound When
Setrealisticambitioustargetsandtimelinesforeachindicator
ToestablisharealisticandambitioustargetforyourgoalstatementusetheadministrativedatacollectedforyourchosenKPItoreviewactualCityperformanceinthemostrecentyearcomparedwiththethreeorfourmostrecentyearsworthofdataBegintocalculatesummarystatistics(egtotalminimumaveragemedianmaximum)toidentifytrendsCreatecumulativeandmonthlychartstoidentifytrendsandanomaliesBasedonyouranalysisofhistoricaldataignoretheanomaliesandassumetheproductivitytrendswillfollowthesameoverallpatternfrommonth-to-monthFinallyapplyanincreasedproductivityfactortotheperformanceeachmonthtoforecastfutureprogressBasedonthisanalysischooseatargetthatinspiresimprovementwhileaccountingforreality
Manycitiesthataremorematureintheirperformancemanagementpracticesrecognizetheimportanceoftakingacommunity-levelapproachtoimprovingoutcomesforresidentsThesecitiesarecreatinggoaltaxonomiesthatincludemeasuresrelatedtopublicsafetyhousingandevenrecreationCitiesdriveprogressonthesegoalsbyimplementingstrategiesthatengagemultipledepartmentsandcollaboratewithserviceprovidersinthecommunity
EventuallyyouwilldevelopgoalstatementsfortheoutputsinordertocapturetheefforttoachievetheoutcomecontainedinyourgoalstatementYoursetofindicatorsshouldbecomposedofshort-andinterim-termsupportivemeasureswithaccuratetargetsandrelevanttimelinesAnexamplegoaltaxonomyfromtheCityofLasVegassResultsVegasprogramisbelowUsethistaxonomyasamodelforyourownsuiteofindicatorsandgoalstatements
Establishaframeworktomonitorprogress
36
Yourturn
Thetemplatebelowismeanttohelpyoucraftagoalstatementforeachsupportingmeasure
Goalstatement
Departments
Outcome
Activities
KPI
Target
Timeframe
Conductanddiscussrelevantanalysis
ThrougharoutineseriesofmeetingscontinuallymonitorprogressofKPIssupportingindicatorsandanyotherdatacollectedsuchasqualitativedatatoidentifytrendsthatwillindicatewhetherthechosenpolicyishavingthedesiredimpactThesedatapointsshoulddrivethediscussionduringthesemeetingspointingtoopportunitiesforfurtherinquiryinto
Establishaframeworktomonitorprogress
37
whatisdrivingtheobservedtrendsAnalystsshouldbeencouragedandtrainedtoaskquestionsofthedepartmentsproducingthedataandconductsitevisitswhenpossibletodeepentheirunderstandingofthedataPresentationoftheanalysisshouldfocusononeideaatatimetokeeptheconversationfocusedonkeypointsandspecificopportunitiesforimprovementYoumayconsiderdevelopingapublicfacingdashboardtosharethisprogressmorebroadly
AgeneralcomparisonofthestatusquodatawithwhatwasobservedduringthepilotcanprovidedecisionmakerswithaquickunderstandingofthedifferencesbetweentheoldandnewapproachesalthoughthiswillnotshowcausalityOftenthesecomparisonswillincludeoperationalmetricssuchasdaysordollarssavedbutrememberthattheimpactonpeoplewillbemostimportantIfyouareworkingwithanevaluationpartnertheywillsupportthisstep
Usethespacebelowtosummarizetheresultsofyouranalysis
MeasureTargetvalue(seesectiononsettingtargetsabove)
Pilotvalue(whatdidyouobserveduringthepilot)
Historicalaverage(ifavailable)
Results(WhoisbetteroffIsanyonenegativelyimpacted1-2sentencesreassessingyourgoalstatement)
DocumentyourprocessandprogressincludingfeedbackfromtheparticipantsimplementationpartnersandtheprojectteamDidanythingunexpectedoccurduringthepilotArethereinternalandexternalfactorsthatmayaffectyouroutcomesThisiswheretheimplementationoutcomescanbecanhelpfulIftheinterventiondidnotgoasplannedconsiderwhetheritisbecausetheinterventionwasnotacceptableorfeasible
UsethetemplatebelowcreatedbyDenversPeakAcademytocapturethisinformationIfthe306090daytimelinedoesnotmakesenseforyourcityfeelfreetocustomizeit
CurrentState
FutureState
30Day
60Day
90day
Whatwentwell(+)
Whatdidnotgowellshouldbechanged(-)
IfaquestionordiscussionpointarisesthatcannotbeansweredinthemeetinganassignedstaffmembershouldcapturethequestionduringthemeetingAfterwardthefollowupitemsshouldbesenttorelevantpartnersandthenextmeetingshouldbeginwithadiscussionof
Establishaframeworktomonitorprogress
38
thoseitems
Takeaction
DiscussionsaboutwhetheryourindicatorsaremovingintherightdirectionandwhatmightbedrivingtrendsmayleadtoadiscoverythattherearenotenoughresourcesallocatedtothispolicyorissueTheremaybelimitedfundingforstaffprogrammingortechnologythatisdrivingthedatainthewrongdirectionIfCityleadershipisintheroomthisisagreattimetomakethecaseforadditionalfundingduringthebudgetseasonBycollectinganalyzinganddiscussingdataonthispolicyroutinelyyouarebuildingevidencetomakethecasefortherightleveloffundingtoreachtheestablishedtargets
Establishaframeworktomonitorprogress
39
Assessdataaccessibilityqualityandsecurity
Determineaccessibilityandqualityofdata
OnceyouhavecompiledyourwishlistofindicatorsitistimetodeterminetheaccessibilityandqualityofthedataTopreventcomplicationyoumaywishtoprioritizerelevantindicatorsforwhichyouarealreadycollectingdataAvoidthetemptationofadoptingatechnologysolutiontocollectnewdataatthispointsuchasaphotosurveytoolorotherappFornowyouaretryingtodeterminewhichdatasetsarerelevantaccessibleandhighqualityBesuretoworkwithyourkeyinternalpartnerswhileyouareidentifyingyourmeasuresDepartmentstaffwhoworkonprogramsarefamiliarwiththeindicatorsthatwillaccuratelysupporttheoutcomesanddatabaseadministratorswillbevaluablewhendiscussingaccessibilityandqualityofthedata
Abrieflistofconsiderationsfortheaccessibilityofyourdataisbelow
ArewecurrentlycollectingthisdatasetorwillwebecollectingitforthefirsttimeHowfarbackdoestherecordgo(timehorizon)WouldweneednewtechnologytocollectoranalyzethedataCanthedatabecollectedmanuallyorcanitbeautomatedHowfrequentlyisthedataproduced(continuouslyhourlydailyweeklymonthlyquarterlyyearlyetc)IsthedataavailablefortheentireCitypopulationorasubsetofthepopulationHowgranularisthedata(individuallevelCensusblockneighborhoodcitywideetc)DoesthedatacontainsensitiveinformationIsthereaproxymeasurethatwecanusefornowIsthedatamachinereadable
AsnotedbytheWilsonSheehanLabforEconomicOpportunities(LEO)attheUniversityofNotreDameleveragingadministrativedatacansignificantlyreducethecostsandtimelinesofrigorousevaluationCitiescanestablishaninternaldatasharingagreementacrossdepartmentsincludinglanguagethatenablesadministrativedatatobeusedforresearchpurposesCitiescanalsoidentifyexternaladministrativedata-holderssuchasacountyschooldistrictpolicedepartmenthospitaloranotherstatethatmayhaveoutcomedataimportantformeasuringprogramimpactCitiescanleverageexistingrelationshipsandorworkwithresearchinstitutionstodevelopstreamlineddatasharingprocessesandagreementswiththeseexternalentities
Assessdataaccessibilityqualityandsecurity
40
Evaluatethequalityofyourdata
QualityisasimportantasaccessibilityToevaluatedataqualityyoumaywishtocreatearubricliketheonebelow
Value ValueDescription
3 ExcellentDataQualityThedataiscompleteaccurateandupdatedregularly
2 AdequateDataQualityThedataissomewhatcompleteaccurateandupdatedfairlyregularly
1 PoorDataQualityThedataisnotcompleteaccurateorupdatedregularly
ThisisagreattimetobuildonthedatainventorythatyoustartedinthegatherinformationpageSuggestedadditionalfieldstoaddtoyourdatainventorytemplatearebelow
Program DatasetTitle
DatasetDescription DataSource
Departmentorexternalpartner
DoestheCityalreadycollectthisdata
Whatisthenameoftheprogram
Human-readablenameofthedataset
Shouldbeunderstandablebynon-technicalusers
Whatistheoriginalsystemapplicationfilethathousesit
Departmentorexternalpartnerthatmaintainsthedata
Isthisdataalreadycollected
Updatefrequency-ThefileformatforthedatasetTypicallycsvxlsxshapefiletxtetc
Format-Howoftenthedatasetisupdated
Timerange-Theperiodoftimecoveredinthisparticulardataset
Datause-Howisthisdatasetcommonlyused
Datausers-Whotypicallyusesthisdataset
Dataownerstitle-Titleofthedataowner
Dataownersposition-DataOwnersjobtitle
Dataownersemailaddress-DataOwnersemailaddressWhomanagesthedataandorisresponsibleforgrantingpermissiontoaccessthedataWhounderstandswhatthedatasetincludesandcananswerquestionsaboutit
Dataqualityconcerns-Describetheconcernsyoumayhaveaboutthequalityofthedataset
Assessdataaccessibilityqualityandsecurity
41
Division-DivisionwherethedataismaintainedIfmorethanonedivisionisresponsiblelisttheprimaryownerofthedataset
Sensitivedatacomments-DescribetheissuesthatwouldpreventthecityfromsharingthedatapubliclyCommonsensitivityconcernsincludeprivacyviolationssecurityissuesorhighcostorstaffdemandstopostthedata
Publishingstatus-Thistheplacetoputanycommentsregardingthecurrentstatusofthedataset
Publishedlink-Thisisthelinktothepublisheddatasetontheopendataportalorcitywebsite
Reviewdatasecurityandprivacyprotectionprotocols
OftenapolicychangealsoinvolvesachangeinthedatathatisbeingcollectedhowthatdataisbeinghandledwhohasaccesstoitandahostofotheradjustmentsAsyoutakestockofthedataassetsthatareavailableforyourinitiativeitisvitaltoalsothinkthroughwhatwillbedonetoprotectthatdataandtheprivacyofthepeopleitrepresentsAbigpartofgooddatasecurityinvolvesstrongdatagovernancepracticesthatdefineandidentifysensitivedataandconsciouslyworktobalancethevalueandriskofstoringsuchsensitivedataOnewayofevaluatingyourdatacomesfromcaselawrelatedtotheUSFreedomofInformationActwhichwehavetranslatedtothechecklistbelow
1 Woulddisclosureresultinasubstantialinvasionofprivacy2 Haveweconsideredtheextentorvalueofthepublicinterestandthepurposeorobject
oftheindividualsseekingdisclosure3 Istheinformationavailablefromothersources4 Wastheinformationgivenwithanexpectationofconfidentiality5 Isitpossibletomouldreliefsoastolimittheinvasionofindividualprivacy
TheseriskscouldbemitigatedusingavarietyofstrategiesfromaggregatingdatatoobfuscatingindividualstoredactingfieldsthatmayincludePersonallyIdentifiableInformation(PII)orre-evaluatingyourdatacollectionmethodstofindalternativewaysofgettingtheinformationyouneedThesestepsshouldbecodifiedandmanagedbyadatagovernancecommitteeifpossibleForthesensitivedatathatisdeemedessentialcreateaplanaroundpotentialdatabreachesinordertoquicklyandconfidentlyrespondtoemergencysituations
ProactivelypreparingforthesesituationsisoftenthebeststrategyfordatasecuritybutdonotforgetthetechnicalsideInvolveyourITadministratorsinconversationsaboutdatastorageandaccessensuringthattheywillbeprotectedbyenterprise-gradecybersecuritysystems
Assessdataaccessibilityqualityandsecurity
42
Assessdataaccessibilityqualityandsecurity
43
Designimplementationandevaluation
Designtheimplementation
NowthatyouhaveyouraccountabilityplanandkeyindicatorsitistimetogetspecificaboutyourplanforimplementationThetemplatebelowwillhelpyouthinkthroughthespecificmechanismsofimplementingpolicychange
UnderstandingorganizationsandsystemsiscriticaltounderstandingandmakingdecisionsaboutwhereandhowinterventionsshouldbeappliedApolicyimplementedinasinglesettingmayshowpromisingresultsbutimplementingmultiplepoliciesandprogramsatthesametimewithdifferentactorsresponsibleforeachaspectcanhelpensuresustainablebehaviorchange
InSomervilleMAShapeUpSomerville(SUS)isacitywidepublichealthprogramthatisimplementedacrossmultiplelevelsofinfluenceandsystemsthroughinterventionsbeforeduringandafterschooltoincludechangestobuiltenvironmentsuchasbikelanesregionalmasstransportationandphysicaleducationequipmentinschoolsandgymstoincreasephysicalactivitywellnesspoliciesandschoolgardensnutritioneducationduringandafterschoolandhealthyrestaurantprogramstosupporthealthyfoodandhealthyeatinghabits
Usethetemplatebelowtoplanyourimplementation
Policyimplementationplanningtemplate
ListthespecificdesiredoutcomesInthepreviousstepyouworkedwithdepartmentstafftoestablishspecificmeasurableoutcomesandinterimindicatorsPrioritizetheindicatorsthatwilldemonstrateprogressUsethespacebelowtocapturethedesiredoutcomesofthepolicychange
ExampleShapeupSomervillebeganwiththegoalofpreventingobesityin1stthrough3rdgraders
Yourturn
____________________________________________________________________________
____________________________________________________________________________
Designimplementationandevaluation
44
SpecifythetargetpopulationAreyoudesigninganinterventionthattargetsonlycertainsitesAreyouplanningonimplementingaprogramorinitiativeforaspecifictargetaudienceClearlyspecifythepopulationyouplantotarget
IfyouplantoconductarigorousimpactevaluationasnotedbytheWilsonSheehanLabforEconomicOpportunitiesitisimportanttoestablishconcreteobjectiveeligibilitycriteriaandensurethatthesecriteriaareappliedconsistentlybyallstaffduringscreeningandintakeItisalsovitaltoidentifytheindividualswhoareeligibleforaparticularprogramorserviceevenifeveryoneeligiblecannotbeservedbyexistingresourcesThishelpscitiesidentifyanyunmetneedsandguidefutureresourceallocationwhilealsocreatingthepossibilityofarigorousimpactevaluationbyidentifyingapotentialcomparisongroupFinallywhenconductingscreeningandintakeitisimperativetocollectandmaintainbasicinformation(egnamedateofbirthgender)aboutallprogram-eligibleindividualsnotjustthosewhoparticipateintheprogram
ExampleIntheShapeUpSomervilleexampletheprogrambeganbytargeting1stthrough3rdgraders
Yourturn
____________________________________________________________________________
____________________________________________________________________________
DefinethelocationWherewillthepolicychangetakeplaceWillimplementationtakeplaceatonesiteormultiplesites
ExampleShapeUpSomervilleinterventionsweredeliveredinschools
Yourturn
____________________________________________________________________________
____________________________________________________________________________
ClarifyimplementationpartnersandarticulatetheirrolesWhatarethekeyorganizationsinstitutionsorpeopleresponsibleforimplementationTheprojectteamshouldbeasclearandspecificaspossibleinprovidingadescriptionoftheconcreteactivitiesthatimplementationpartnersaremeanttocarryoutPracticeprofilesmaybeahelpfultooltohelpdefinewhatisrequiredoftheactorswhoareresponsibleforimplementingtheintervention
Designimplementationandevaluation
45
ExampleInSomervillemayoralleadershipandengagementofmultiplecityagencies(publichealthprimarycareschoolsCambridgeHealthAllianceetc)streamlinedandconnectedthevariousShapeUpSomervilleinitiativesandgrantswhichhelpedtokeepthetalentandenergyfocusedonhealthyeatingandactivelivingforallsectorsofSomervilleSUSpartnersultimatelyincludedschoolsnonprofitsCityandstateagenciesandnationalfoundations
Yourturn
____________________________________________________________________________
____________________________________________________________________________
DescribetheinterventionWhatarethestepsactivitiesorprocessesRememberthatthemechanismthespecificstepsorprocessthatleadtobehaviorchangeiskeytounderstandingtheeffectivenessofthepolicychangeandsharingtheresultswithothers
ExampleGoalsofShapeUpSomervilleweretoinfluenceeverypartofanearlyelementaryschoolchildrsquosdayndashbeforeduringandafter-schooltopromotehealthyeatingphysicalactivityandtheirweightstatusInterventionincludedfoodserviceenhancementstoimprovebreakfastandlunchandstaffprofessionaldevelopmentwalktoschoolactivitiesSUSclassroomcurriculumandprofessionaldevelopmentschoolwellnesspolicydevelopmentSUSafter-schoolcurriculumandprofessionaldevelopmentoutreachandeducationtohomethroughmaterialsforumseventsldquoSUSapprovedrdquorestaurantscommunityoutreachandcapacitybuildingthroughpolicydevelopmenttrainingsmediaplacements
Yourturn
____________________________________________________________________________
____________________________________________________________________________
ClarifytimelineandmilestonesfordeliveryandevaluationHowfrequentlywilltheservicesinterventionbedeliveredHowlongwilltheprogramrunWhatisthetimetoexpectedimpacts(eg12monthsafterprogramcompletion)
Designimplementationandevaluation
46
ExampleShapeUpSomervilleledbyTuftsUniversityFriedmanSchoolofNutritionScienceandPolicyconductedaquasi-experimental(nonrandomized)interventionstudyamong1stthrough3rdgradestudentsinSomerville(intervention)andtwocomparisoncommunitiesfor9-monthsduringoneschoolyear(fall2002ndashspring2003)ThestudyshowedpositiveassociationoftheinterventiononchangeinBMISUSreleasedreportssummarizingevaluationsprogresstodateplansforadditionalprojectsandimpactstenandfifteenyearsintotheinitiative
Yourturn
____________________________________________________________________________
____________________________________________________________________________
PlanyouranalysisTheprevioussectionaddressedestablishinganaccountabilityframeworktomonitorprogressandimplementationDeterminewhichindicatorswillbediscussthroughthatprocesswhowillsharethedataandanalysisandhowfeedbackfromthoseconversationswillbeincorporatedbytheprojectteam
ExampleTheMayorrsquosofficeimplementedadata-drivendecisionmakingapproachcalledSomerStatwhichwassynergisticwiththeapproachestakenbytheresearchandevaluationteamsdeployedforthespecificgrants
Yourturn
____________________________________________________________________________
____________________________________________________________________________
PlanforsustainabilityIfthepolicyandprogramchangeissuccessfulhowwilltheprojectteamsustainandexpandtoadditionalsitesorpopulations
ExampleAlthoughmostofthepartnershipsforgedintheearlyyearsweremaintainednewgrantsandcollaborationsexpandedattentiontoolderpublicschoolstudentsactivetransportationandtheGreenLinesubwayextensionthebuiltenvironmentfarmerrsquosmarketeffortssuchasldquoGrowninSomervillerdquowithUnionMainStreeteconomicdevelopmentandhealthdisparitiesaswellasdatacollectionmonitoringandsurveillance
Yourturn
Designimplementationandevaluation
47
____________________________________________________________________________
____________________________________________________________________________
Pilotplanningtemplate
IfyouplantoconductapilotfirstdeterminethepurposeofthepilotDoyouwanttoobservewhetherthepolicychangeresultsinthedesiredbehaviorchangeAreyouconcernedaboutfeasibilityscalabilityorcostDoyouwanttotestwhetheryourdesiredresultscanbeachievedthroughthispolicyapproachClarifythegoalofthepilotandthequestionsyouhopetoanswerThiswillhelpyouidentifythekeyindicatorsthatyouwillusetomeasurewhetherthepilotissuccessfulBeforebeginningestablishabaselineofcurrentperformanceandsetyourtargetsaccordinglybasedonwhatyouhopetoaccomplishthroughthepilot
NextestablishthepilotplanforyourprojectteamincludingmilestonesforteamcheckinstodeterminewhethertomoveaheadorchangecourseThisisseparatefromthemeetingsthatwilloccuraspartoftheaccountabilityframeworkUsethespacebelowtocreateyourtimelineanddevelopyourmeetingschedule
Task Personresponsible Startdate Checkindate Deliverable
Planyourevaluation
WhileyouaredesigningtheimplementationitisalsotimetobegintodevelopyourplanforanalysisAsnotedbytheWilsonSheehanLabforEconomicOpportunitiesdecisionsaboutprogramsetup-includingoutreachscreeningintakedatatrackingandenrollment-candeterminewhetherornotarigorousimpactevaluationwillbepossibleAsdiscussedcityleadersmayfindithelpfultoincludeevaluationpartnersinearlyconversationsaboutprogramsetupDuringpreviousstagesyoumayhaveworkedwitharesearchertoplanyourproposedpolicychangeYoushouldbeclearaboutthepurposeoftheevaluationanddevelopanevaluationplanBelowareadditionalconsiderations
Causationvstrendanalysis
IntheEstablishaframeworktomonitorprogressstagetherewasguidanceontrackingtrendsovertimeItisimportanttonoteherethatmonitoringdataisdifferentfromanalyzingacausaleffectTrendanalysiswillillustratethatchangesareoccurringandmayleadtoquestionsthatcouldbeansweredbyyourevaluationbuttrendanalysisalonedoesnotprovideevidenceaboutwhatisdrivingthosechangesYoumayseechangesinyour
Designimplementationandevaluation
48
outcomeindicatorsbasedonvariablesoutsideoftheCityscontrolForinstanceachangeinfederaltradepolicycanhaveanimpactemploymentratesinyourCityoutsideofyourCitysnewjobtrainingandplacementprogram
ToaddresscausalityyoumustcomparetheoutcomesofindividualswhoreceivedtheinterventionwiththosewhodidnotWithoutacomparisongroupofsimilarindividualswhowerenotexposedtotheinterventionitisimpossibletoknowwhichfactorsareresponsiblefortheoutcomesRandomizedControlTrials(RCTs)inwhichindividualsarerandomlyassignedtotreatmentandcontrolgroupsarethegoldstandardinprovingcausalityIfyoucandemonstratethatoutcomesimprovedforthosewhoreceivedtheinterventionanddidnotimproveatallorimprovedlessforacomparisongroupyoucanbesomewhatcertainthattheinterventionwasthecause
Examplesofothernon-causalimpactevaluationsthatarerigorousandreliableincludecertainquasi-experimentalstudiesusingcomparisongroupdesignssmartinterruptedtimeseriesanalysesthatmodeltrendsovertimeinwaysthatallowestimationofinterventioneffectswelldesignedpre-posttestanalysesneedsassessmentsimplementationstudiesetcFormoreinformationondifferenttypesofevaluationsseetheSageResearchMethodswebsite
Lookfordifferencesingroups
BuildintimetoalsoexaminethepilotdatatoseeifgroupsareexperiencingdifferentresultsReviewingperformancebasedonoutcomemeasuresforapopulationinaggregatemayhidethesubtlewaysthatsomepeoplemaybeaffecteddifferentlysoitisnecessarytocarefullyanalyzethedatafordisparateoutcomesForinstancetheMovingtoOpportunity(MTO)studythatexaminedtheimpactofhousingvouchersonfamilieswholivedinhigh-povertyneighborhoodsfoundthatchildrenwhomovedbeforetheageof13hadbettereducationandhigherincomesasadultsHoweveradultsdidnotseeimprovededucationandemploymentoutcomes
Disaggregatingdataintomorespecificcategoriesofraceandethnicitycanhelpunderstandtrendsanddisparitiesinamoregranularwaywhichcanhelpdecisionmakerstargetpoliciesandinterventionsinamoreequitableway
EarlierintheRoadmapweworkedtoidentifystakeholdersandwhomightbepositivelyandnegativelyimpactedbyachangeinpolicyUsetheresultsofthatexercisetoinvolvethecommunityininterpretingdataanalysisandtogatherqualitativefeedbackfromthecommunitytocallattentiontoissuesForinstanceanincreaseinyouthdetentionmaybeexplainedbyinefficienciesorlackofaccesstotransportation
Designimplementationandevaluation
49
Designimplementationandevaluation
50
Shareyourstoryandsuccess
Sharetheresults
Storytelling
AnintegralstepinyourstrategyissharingtheresultsofyourpilotorpolicychangewithyourcommunityThisincludesdevelopingmessagingaroundtheworkofyourorganizationandcreatingaplantodisseminateyourstoriesWhytellstoriesaboutyourwork
Informandorunderstandasubjectorsituation
Giveneededcontextofdataorbitsofinformation
Relatablestoriesarememorableforreadersviewers
ReadersorviewerswillseetheinformationasmorecompellingwhenthereisademonstratedandrelatablechangeorimpactonthemselvesortheirfriendsneighborsandcolleaguesStoriesareaboutpeopleandgovernmentisfundamentallyabouthelpingpeopleSotellthestoriesofthepeopleyouandyourcitycolleagueshelped
Keepthesetipsinmindasyoudevelopyourstorywithyourteam
WhatwereyourfindingsandwhyshouldanyonecareTellyouraudiencehowtheproblemwassolvedWhathastheimpactbeenonresidentsandcitystafftodateWhatwasthesuccessandweretheremultiplesuccessesalongthewayIfyouhavepreliminaryresultsordatashowingimprovementtosharethisisthetimetodoit
WhowasinvolvedandimpactedReviewthelistofstakeholdersHowwereeachofthemimpactedbytheprogramHumanizeyourstoryandimpactbyaddingadditionalvoicesThesevoicescanbeoneofthefollowing
Apersonorfamilywhowaspositivelyaffectedbythechangesuccess
Aninvolvedthird-partydescribingboththeproblemandthesolution(ieanorganizationnonprofitprovidersbusinesstradeassociationsportsteametc)whoispartofthecommunitythatthegovernmentworkedwithonthechangesuccess
Someonefromthegovernmentitselfwhocanspeakaboutthechangesuccess
TellmoreaboutyourcityDescribeyourcityincludingpopulationdemographicsindustriesetcWhatareitschallengesWhatisitknownforWhatmakesituniqueHowisitsimilarordifferentfromothercitiesintheUS
Shareyourstoryandsuccess
51
DescribeyourcityrsquoschallengeWriteoutthechallengeyourcityisfacing(asdefinedinyourproblemstatement)HowdoesitaffectresidentsHowdoesthisproblemaffecttheworkofcitystaffmembers
AddthedetailsProvidedetailsonyourappliedresearchstrategyTalkaboutwhatyourcityhasdonenthepasttoresolvetheissueDiscusshowyourstrategyhaschangedwhenyoubegantoadoptthisapproachWasmoneysavedorinvested
Usethisstorymapexercisetoorganizeyourthoughtsandguaranteethatyouincludealloftheelementsofatraditionalstory
AfteryoudevelopyourmessagingcreateacommunicationsmarketingandoutreachstrategydesignedtospreadyourmessageYourstrategyshouldwalkyouthroughthefollowingkeyactivities
1 IdentifyandprofileyouraudienceWhencreatinganddisseminatingyourmessageitisalwaysgoodtothinkaboutyouraudienceWhoexactlywouldbeinterestedinwhatyouhavetosayThegeneralpopulationofthecityPeoplewhoworkinnonprofitsBusinessownersItmattersbecausenoteveryonewholivesinacitymaybethetargetoftheinformationBusinessownersmaybemoreinterestedindatashowingeffectsonthebusinessdistrictthanothersandsoonWewillcoverthislaterbutthisstepwillalsohelpyoutothinkofthemosteffectivewaytoreachyouraudience
2 Defineyourcalltoaction(ifyouhaveone)SpendsometimeconsideringwhyyouareinterestedintellingyourstoryDoyousimplywanttoupdatethepubliconyourprogressorcelebrateyoursuccessesAreyouinterestedinyouraudiencechangingitsbehaviororsupportingaspecificinitiativebeinglaunchedbyyourorganizationIfyouwanttohaveyouraudienceactorbehaveinacertainwaybespecificandincorporateitinyourmessaging
Shareyourstoryandsuccess
52
3 SelectperformancemeasuresThinkabouthowyouwoulddetermineifyourcampaignissuccessfulandselectappropriateperformancemeasuresYouwillrevisitthesemeasuresattheendofyourcampaigncommunicationsplanimplementationtodeterminewhetheryouneedtomodifyyourcommunicationsandoutreachstrategyforthefuture
4 MethodstocommunicateyourstoryTherearemanywaystocommunicatesuccessofyourprojectHerearesomesuggestionsonreachingaudiencesItisimportanttodeterminewhichmethodsarethemosteffectiveforyourdesiredaudiencetoreceiveinformationDoyourresearch(PewResearchCenterhasexcellentstudiesonthereachofmediadigitalmediaandmarketing)ForexampleifseniorsareyourtargetpopulationandtheyarelessinclinedtouseInstagram(thatistrue)youmaywanttoconsiderFacebookorthenightlylocalnewsbothofwhichareusedmoreoftenbyseniorsBelowisalistofsomemethodsandtoolsyoucanuse
i Pressreleasestolocalnewspapersandtvstations
ii ShortsmartphonevideosAgreatwaytoengageanaudienceisthroughvideoAskonequestionofsomeonewhoispartofthestoryMaybethereisasocialworkerwhocandiscusshungerinyourcityThesevideoscanbereusedonmultipleplatforms
iii Long-formwriting
iv TellingindividualstoriesthroughaseriesofphotosandshortnarrativesforpublicationonFacebookorInstagram
v Op-edsopinionpiecesplacedinthelocalpaper
vi Emailupdatestoyourlist
vii Connectingwithappropriateorganizationsthatareinvestedinthetopicandworkwiththecommunityandaskthemtodisseminateinformationtotheirmembers
viii EmployingldquoWordofMouthrdquo(WOM)tacticsbycreatingtownhallsorfreeeventsforresidentstoattendintheirownneighborhoods
5 DeterminethetypeofcollateralUsuallycampaignsthattargetresidentsrequireseveralavenuestoreachthemPeoplearenotalwaysinthesameplaceorspaceHavingakeenunderstandingofyouraudiencewillassistyouwithyourchoicesonwhatneedstobedevelopedSometimesthatmeansimagesforsocialmediaaonepageflierthatgivespeopleinformationonattendinganeventoranin-depthbrochure
Shareyourstoryandsuccess
53
6 ImplementyourstrategyKeepinmindstrategiescanbefluidSometimessomethingmaynotturnoutasexpectedevenifduringplanningsessionsitwasldquosupposedtordquoItisokYoucananalyzewhathappenedduringadebriefsessionandonceyoursurveysandothermeasurementshavebeenfinalized
Visualizations
Withtherightvisualizationyoucanreviewandshareinformationinsightsanalysisperformanceandtrendsallofwhichisvaluabletoacross-sectionofthepublicButwhichvisualizationisrightforyouTochoosetherighttypeofchartgraphdeterminethetypeofquestionyouwanttoanswerSomecommoncharttypesarebelowandmoreinformationcanbefoundhere
Exploreadditionalanalyticaltechniques
Duringthegoal-settingphaseyouestablishedoutputoutcomeandimplementationmeasuresBasedontheresultsofyourpilotyoumayneedtoadjustyourmeasuresWiththepilotdatainhandyoumayhavesomeideasofhowfutureanalysescouldbedifferentandpotentiallyimprovethequalityofyourprogramTakesometimetobrainstormotherrelevantdatasetsthatyoudidnothavethetimetopursueearlierincludingmessydepartmentaldatathatneedsmoreattentionorsourcesofpublicdatafromthestateandfederalgovernmentnonprofitsandevenprivatecompaniesthatcouldsupplementyourunderstandingofanissueUsingmachinelearningtechniquesyoumightbeabletobetter
Shareyourstoryandsuccess
54
predictdemandforhomelessservicesusingNOAAweatherforecastspreemptwatermainbreaksusingsoilstratigraphyorinvestineconomicdevelopmentbymonitoringrealestatelistingdata
Advancedanalyticaltechniquescanalsohelpyourimplementationrunmoreeffectivelybyintegratingdata-informedtargetinganddecisiontoolsintotheprocessByautomatingroutineworkflowsandsurfacingsubtletrendsthesetoolscanhelpyouiterateinamoreprecisemannerandprovidekeystrategicguidancetofront-lineemployeesthatwouldotherwisebeunavailableThesesortsofdata-science-enhancedprocessescanbeclassifiedasoneofsixprojecttypesasexploredfurtherbytheNewOrleansLANOLAlyticsteamFormoreguidanceonscopingasimilarprojecttheGovExadvancedanalyticsscopingguidegivesyouastep-by-stepoverviewofhowtobesuccessful
Usethespacebelowtolistyourideasforpossibleadvancedanalyticsprojectsbytype
Typeofproject Ideas
Findingtheneedleinthehaystack
Prioritizingforimpact
Earlywarningtools
Betterquickerdecisions
Optimizingresourceallocation
Experimentingforwhatworks
Keepthefeedbackloopgoing
GatherinformationonpublicsentimentaboutpolicychangethroughfocusgroupssurveysandpublicmeetingsledbyadvisorygroupmembersNotewhethernewinformationisavailablethatcanhelpanalyzeorcontextualizeyourprogress(egfederalorstatedatareports)Sharetheresultsofyourpilotandthepolicychangeprocesswithothercitiesandresearchers
Shareyourstoryandsuccess
55
WorkingwitharangeofevidencePresentlythereisalotofattentionandinvestmentinevidence-basedpolicymakingincludingthefederalbipartisanCommissiononEvidence-BasedPolicymakingthePewCharitableTrustsreportsandtoolkitsonHowStatesEngageinEvidence-BasedPolicymakingandtheLauraandJohnArnoldFoundationsupportedEvidence-BasedPolicymakingCollaborativeManyoftheseinitiativesfocusonrigorousevaluation--meaningrandomizedcontrolledtrials(RCT)orotherexperimentalmodels--asthebestkindofevidencetouseHoweverthisbodyofevidenceisofteninsufficientforthespecifickindsofquestionsthatgovernmentscanhavewhenmakingadecisionaboutpolicyInthisguideweproposeanapproachtosupportgovernmentsinincorporatingadditionaltypesofevidenceintotheirdecisionmaking
OnewaytounderstandarangeofevidenceistouseNestarsquosStandardsofEvidencetoframeanddefineevidenceTheBritishnonprofitNestaworkstoinjectpositivechangeintothewayUKgovernmentpoliciesandprogramsfunctionOneoftheirkeyareasofworkhasbeendevelopingWhatWorksCentresorganizationsthatcollectcommissionsynthesizeanddisseminateevidenceaboutinterventioneffectivenessacrossarangeofsocialimpactareasNestadevelopedthefollowingStandardsofEvidencetoprovideausefulwaytodocumenthowdifferentkindsofinformationdemonstratedifferentlevelsofevidence
Workingwitharangeofevidence
56
Theimportanceofpotentiallybiasedinformation
Onechallengepresentedbyanevidenceclassificationthatacknowledgesthevalueofnon-scientificformsofevidenceistofigureouthowtobalanceallofthepiecestogetherwhenyouusethemTounderstandhowtodothiswehavetothinkabouthowtopositionpotentiallybiasedinformationindecisionmaking
WhenwelookatinformationwhichhasnotbeenrigorouslyevaluatedweacceptthatthisinformationmightbebiasedThewholereasonweconductrigorousevaluationistoremovebiasfromourobservationsWeconductstructuredtrialstoproduceresultsthatareconsistentandgeneralizableIfafindingisunbiasedthisaccuracymdashthislackofbiasmdashcanbeprovedbecauseanyindependenttestercanreproduceitwhentheyfollowanidenticalprocess
YoucanlookatpotentiallybiasedinformationasjustinformationthatisearlieroninitspathtorefinementreplicationandverifiedaccuracyApieceofinformationbasedonanyoneindividualrsquosobservationsislikelytobebiasedbecauseitiscomingfromjustonesourceBiasisreducedbygatheringtogetherlotsofindependentsourcesofinformationandfiguringoutitsmostcommonattributes
MaybethebiasedinformationmakesaclaimthatistoolargemdashitsayssomethingistrueunderallconditionswhenitisreallyonlytrueundercertainconditionsorthataneffectofaninterventionisbiggerthanitstrueaverageorthataneffectlastsovertimewhenitdoesnotPerhapschangingtheclaimtobemorespecificmdashcriticallyreviewingtheinformationtoseeifspecificconditionsmustbemettoachievetheoutcomesmdashwouldmaketheclaimmoreaccurate
IfyoutestedyourclaimfounditdidnotholdupinsomecaseschangedittesteditagainuntilyourclaimrepeatedlyheldupthenyoucoulddemonstratethattheclaimwasnolongerbiasedbutaccurateThisprocessoftestingimprovingtestingandvalidatingiswhatwegetfromstructuredscientificstudy
Withindividualpiecesofinformation--crediblehypothesescasestudiesexpertopinion--youcannotassumethatkindofaccuracyAtthesametimetheseearlyobservationsarefoundationaltoanyreplicablescientificobservationsWewillnevergetanywhereifwedonotstartwiththesepiecesofinformationeveniftheymightbebiased
Combininglevelsofevidencetogetthemostdetailedinformationforyourcity
ItisespeciallyimportanttoacceptdifferentformsofevidencewhenyoufacequestionsthathavenotyetbeentestedbyindependentstudiesSomeofthemostimportantquestionsrelatetothingsthatareveryspecifictoyoursituationYoufaceaparticularcombinationof
Workingwitharangeofevidence
57
issuesactorstrendsandresourcesthatisultimatelydifferentfromeveryotherplaceintheworld
ThisisimportantbecauseaspolicypractitionersandnotscientistsyoucaremoreaboutyourspecificcasethanahypotheticalaveragegeneralcaseYoucarelessaboutthetotallikelyrangeofimpactsfromapolicythanhowthissingleimplementationwillworkYouwantinformationaboutpoliciesthatwillprovideyouwiththebestpossiblepredictionabouthowtheywillworkinyourcontext
ByputtingdifferentkindsofevidenceintothesameframeworkNestarsquosStandardsofEvidenceimplythatwhenyoucannotgetrigorouslytestedexperimentallyverifiedinformationitisappropriatetoworkwithwhatwedohaveThisisrelevantforthemunicipalimplementationcontextbecausenostudypulledfromaliteraturereviewwillaccountforallofthevariablesyoufaceinyourowncontext
Intheabsenceofhavingastudythatpredictshowaninterventionworksinyourspecificcontextyouwillneednon-scientificinformationtotrytofillinthosegapsYouwillneedtogatherstoriesfromresidentsstaffmembersandotherstakeholdersthatsitonLevels1or2ofthestandardsofevidence
Someofthekindsofinformationyouwillprobablygetmoreeasilyfrompeoplethanstudiesinclude
howpeopleincitieslikeyoursreactedtoaninterventioninpublicforumshowpeoplewhoimplementedaninterventioninacitylikeyoursfiguredouthowtogetitpassedhowthecitycommunicatedpubliclyabouttheapparentbenefitsanddrawbacksoftheinterventionthepoliticalacceptabilityoftheintervention
Youcanremainsimultaneouslyawarethattheinformationyouaregatheringthroughindividualstoriesisbiased--notgeneralizableinformedbyspecificperspectives--whilealsorecognizingthathavingsomeinformationaboutsomethingimportantisbetterthannoinformationEvidencefromthefirsttwoevidencelevelsareessentialbuildingblocksofknowledgeandperfectlyusefulwhenyoucannotgetdetailedunbiasedknowledge
Youhavetostartsomewhereandyoushouldalwaysstartwithevidenceyouhaveaboutyourveryspecificconditionsthatyouknowareimportantforsuccess
Workingwitharangeofevidence
58
Thankyou
Interviewswithcities
GovExheldsemi-structuredinterviewswithsixstaffmembersinfourcitiestoestablishtheinitialstepsoftheRoadmapWeaskedrespondentstotalkaboutaninitiativeorpolicythattheyhavebeenresponsibleforimplementingintheircityRespondentsfromthreecitiesspokeabouttheireffortstoaddresshomelessnessandrespondentsfromonecityspokeaboutblightAfterrespondentsidentifiedtheproblemtheyspokeabouthowthecityprioritizedthisissuethevariousstepsinvolvedandbarrierspresent
GovExconductedtextanalysisoftheseinterviewswhichrevealedthatpeoplewasthemostcommonlyusedtermduringtheseconversationswhendiscussingthestepstochangeaprogramorpolicyThisisathemethroughoutthisRoadmapThefirstsectionbeginswithidentifyingtherelevantpeopleTheconversationsaroundevidencearefocusedonhumanbehaviorratherthanmethodologyorstrengthofevidence
Thankyou
59
OncewehadadraftoftheRoadmapwefollowedupwithcitiesonanindividualbasistolearnmoreaboutwhatwouldbehelpfulWearegratefultothefollowingcitystafffortheinterviewsandindividualconversations
MichaelBaskinChiefPolicyOfficerChattanoogaTN
MelissaBridgesPerformanceandInnovationCoordinatorLittleRockAR
VictoriaCarreoacutenAdministrativeOfficerLasVegasNV
TyroneGrandstrandCommunityDevelopmentPlannerFargoND
TibLaughlinDirectorofGeneralServicesKansasCityKS
TimMorelandDirectorofPerformanceManagementandOpenDataChattanoogaTN
CharlesRosendahlManagementAnalystDentonTX
Thankyou
60
MelissaSiebenAssistantCountyAdministratorKansasCityKS
GinaThompsonDirectorofStrategicServicesLewisvilleTX
JHUFacultycollaboration
GovExalsoconvenedthecitycontactsandfacultypartnersinaseriesofAdvisoryGroupmeetingsThesemeetingsservedtoconnectcitiesandfacultytounderstandoneanothersperspective
StefanBaralBloombergSchoolofPublicHealth
AshleyBernerJHUSchoolofEducation
MelissaDavey-RothwellBloombergSchoolofPublicHealth
JonLinksBloombergSchoolofPublicHealth
TamarMendelsonBloombergSchoolofPublicHealth
KristinMmariBloombergSchoolofPublicHealth
KeshiaPollackPorterBloombergSchoolofPublicHealth
JoshSharfsteinBloombergSchoolofPublicHealth
LizStuartBloombergSchoolofPublicHealth
LindsayThompsonCareyBusinessSchool
DanielWebsterBloombergSchoolofPublicHealth
Partnersindataandevidence-basedpolicy
Wealsospokewithpartnerswhoareexpertsinthefieldofevidence-basedpolicywhosharedtheirknowledgeandresourcesofcurrentandfuturethinkinginevidence-informedthinking
WendyBarrenoAssociateDirectorWilsonSheehanLabforEconomicOpportunitiesUniversityofNotreDame
TiffanyDavenportDirectorofDataPartnershipsBaltimoresPromise
TomiHiersDirectorBaltimoreCivicSiteAnnieECaseyFoundation
ChrisKingsleySeniorAssociateAnnieECaseyFoundation
KathrynOliverAssociateProfessorLondonSchoolofHygieneandTropicalMedicine
Thankyou
61
KristenWatkinsSeniorPolicyAssociateJ-PAL
GovExParticipants
GovExAnalystMiriamMcKinneyanalyzedtheinterviewtranscriptsandconductedtextanalysisSeniorImplementationAdvisorEmilyShawwillintegrateRoadmapactivitieswithadditionalservicedeliveryactivitiesatGovEx
Thankyou
62
ReferencesAbrahamKGampHaskinsR(2017)Thepromiseofevidence-basedpolicymakingReportofthecommissiononevidence-basedpolicymakingWashingtonDCCommissiononEvidence-basedPolicymakingBigmove2Extendingthegreen(nd)Retrievedfromhttpswwwcityoflewisvillecomdoing-businesslewisville-2025big-move-2-extending-the-greenBatesMAampGlennersterR(2017)ThegeneralizabilitypuzzleRetrievedfromhttpsssirorgarticlesentrythe_generalizability_puzzleBuildwithnotfor(2018January)RetrievedfromhttpsbuildwithnotforwordpresscomCairneyPampOliverK(2017)Evidence-basedpolicymakingisnotlikeevidence-basedmedicinesohowfarshouldyougotobridgethedividebetweenevidenceandpolicyHealthResearchPolicyandSystemsdoi101186s12961-017-0192-xCairneyPOliverKampWellsteadA(2016)TobridgethedividebetweenevidenceandpolicyReduceambiguityasmuchasuncertaintyPublicAdministrationReview76(3)399-402Retrievedfromhttpdxdoiorg101111puar12555Centreforpublicimpact(nd)RetrievedfromhttpswwwcentreforpublicimpactorgCernaL(2013)ThenatureofpolicychangeandimplementationAreviewofdifferenttheoreticalapproachesOrganisationforEconomicCo-OperationandDevelopmentRetrievedfromhttpswwwoecdorgeduceriTheNatureofPolicyChangeandImplementationpdfColoradoSprings(nd)Retrievedfromhttpscoloradospringsgovcommunicationspagewhat-expect-when-you-speakupContinuumofcare(coc)program(nd)RetrievedfromhttpswwwhudexchangeinfoprogramscocCouncilpresidentkicksoff$12millionyouthfund(nd)Retrievedfromhttpwwwbaltimorecitycouncilcomcontentcouncil-president-kicks-12-million-youth-fundDurlakJAampDuPreEP(2008)ImplementationmattersAreviewofresearchontheinfluenceofimplementationonprogramoutcomesandthefactorsaffectingimplementationAmericanJournalofCommunityPsychology41327-350Retrievedfromhttpswwwncbinlmnihgovpubmed18322790Fishbonediagram(ishikawa)(nd)Retrievedfromhttpasqorglearn-about-qualitycause-analysis-toolsoverviewfishbonehtmlFivewhysGettingtotherootofaproblemquickly(nd)RetrievedfromhttpswwwmindtoolscompagesarticlenewTMC_5Whtm
References
63
FligsteinNampMcAdamD(2011)TowardageneraltheoryofstrategicactionfieldsSociologicalTheory29(1)1-26RetrievedfromhttpsociologyberkeleyedusitesdefaultfilesfacultyfligsteinFligstein20McAdam20Soc20Theory20PaperpdHeinkelP(2018January12)CitymullspermitsforbikesharingcompaniesRetrievedfromhttpwwwdentonrccomnewsdenton20180112city-mulls-permits-bike-sharing-companiesHowisareamedianincomecalculated(2012October)Retrievedfromhttpswwwhudexchangeinfofaqs843how-is-area-median-income-calculatedLampkinLMampHatryHP(2003)KeystepsinoutcomemanagementUrbanInstituteRetrievedfromhttpswwwurbanorgresearchpublicationkey-steps-outcome-managementMarshDampSmithM(2000)UnderstandingpolicynetworksTowardsadialecticalapproachPoliticalStudies484-21Retrievedfromhttpjournalssagepubcomdoiabs1011111467-924800247McCannL(nd)RetrievedfromhttplaurenellencomMoultonSampSandfortJR(2016)ThestrategicactionfieldframeworkforpolicyimplementationresearchPolicyStudiesJournal45(1)Retrievedfromhttponlinelibrarywileycomstore101111psj12147assetpsj12147pdfv=1ampt=jdkhhcp4amps=e184c625e1855a6b1615ccb99ee27b1c0b3212d8PaynterB(2018January1)What100citiesarelearningfromeachotherbysharingtheirdataRetrievedfromhttpswwwfastcompanycom40518004what-100-cities-are-learning-from-each-other-by-sharing-their-dataPeakacademy(nd)Retrievedfromhttpswwwdenvergovorgcontentdenvergovenmayors-officeprograms-initiativespeak-performancepeak-academyhtmlPinnockHBarwickMCarpenterCREldridgeSGrandesGGriffithsCJRycroft-MaloneJMeissnerPMurrayEPatelASheikhAampTaylorSJC(2017)Standardsforreportingimplementationstudies(stari)ExplanationandelaborationdocumentBMJOpenRetrievedfromhttpbmjopenbmjcomcontentbmjopen74e013318fullpdfProctorESilmereHRaghavanRHovmandPAaronsGBungerAGriffeyRampHensleyM(2011)OutcomesforimplementationresearchConceptualdistinctionsmeasurementchallengesandresearchagendaAdministrationandPolicyinMentalHealth38(2)65-76doi101007s10488-010-0319-7TandonSD(2018)ProgressincommunityhealthpartnershipsResearcheducationandactionRetrievedfromhttpswwwpressjhuedujournalsprogress-community-health-partnerships-research-education-and-actionProvanKGampKenisP(2007)ModesofnetworkgovernanceStructuremanagementandeffectivenessJournalofPublicAdministrationResearchandTheory
References
64
18(2)229-252Retrievedfromhttpsacademicoupcomjpartarticle182229935895SabatierPA(1988)Anadvocacycoalitionframeworkofpolicychangeandtheroleofpolicy-orientedlearningthereinPolicySciences21(3)129-168Retrievedfromhttpwwwjstororgstable4532139SandstromAampCarlssonL(2008)TheperformanceofpolicynetworksTherelationbetweennetworkstructureandnetworkperformancePolicyStudiesJournal36497-524Retrievedfromhttponlinelibrarywileycomdoi101111j1541-0072200800281xabstractSchwartzMampWilsonE(nd)WhocanaffordtoliveinahomeAlookatdatafromthe2006americancommunitysurveyRetrievedfromhttpswwwcensusgovhousingcensuspublicationswho-can-affordpdfShearerJCAbelsonAKouyateBLavisJNampWaltG(2016)WhydopolicieschangeInstitutionsinterestsideasandnetworksinthreecasesofpolicyreformHealthPolicyandPlanning31(9)1200-1211Retrievedfromhttpsdoiorg101093heapolczw052ThelessonsofadministrativedataHigh-profilepolicy-relevantresearchpoweredbyadministrativedata(nd)Retrievedfromhttpswwwpovertyactionlaborgsitesdefaultfilesresources20161114-The-Lessons-of-Administrative-DatapdfToolsandotherneedsforqualitativeandquantitativemethods(2015February)AgencyforHealthcareResearchandQualityRetrievedfromhttpwwwahrqgovprofessionalsquality-patient-safetytalkingqualityassesstoolshtml
WeaverRCampBagchi-SenS(2013)SpatialanalysisofurbandeclineThegeographyofblightAppliedGeography4061-70
References
65
UniversityPartners
Name Title Affiliation Area(s)ofexpertise
Contactinformation
Examplesofworkingwithnon-academicdecisionmakers
WendyPBarreno
AssociateDirector
WilsonSheehanLabforEconomicOpportunitiesUniversityofNotreDame
LEOsfocusareasincludehealtheducationhousingchildwelfareeconomicopportunitycriminaljustice
wbarrenonduduorleondedu
MelissaDavey-RothwellPhD
AssociateScientist
JohnsHopkinsBloombergSchoolofPublicHealth
Health mdavey1jhuedu410-614-5854
BaltimoreCityHealthDepartmentLeadersMarylandDepartmentofHealthAddictionRecoveryServicesinDCExecutiveDirectorsofcommunityorganizationsacrosstheUSthatfocusonHIVSTIsubstanceabuseandimplementationscience
ElizabethStuart
AssociateDeanforEducationandProfessorDepartmentsofMentalHealthBiostatisticsandHealthPolicyand
JohnsHopkinsBloombergSchoolofPublicHealth
Educationsocialpolicystatisticalmethods
estuartjhuedu
Resources
66
Management
Design-ThinkingResources
EngagementGuideforCivicTech
Journeymap
PerformanceManagementResources
DenverPeakAcademy
Proxymeasuresguide
Talkingaboutperformancewithoutcomplicatingit(video)
SettingPerformanceTargetsguide
Datavisualizationguide
Performancemanagementgettingstartedguide
Exercises
Fishboneexercise
5Whys
AdministrativeData
TheLessonsofAdministrativeData(JPAL)
EvaluationResources
Quantitativeandqualitativemethods(USDepartmentofHealthandHumanServicesAgencyforHealthcareResearchandQuality)
KeyStepsinOutcomeManagement(UrbanInstitute)
RCT-YES
Sageresearchmethods
BestPracticesforMixedMethodsResearchintheHealthSciences
ACCORDChecklist
Pilotandimplementationplanning
Benchmarkingexamplesandproxymeasures
Seattleopendataplaybook
Resources
67
Datasciencescopingguide
Resources
68
AboutGovExGovExhelpsgovernmentsusedataandevidencetoimproveresidentsrsquolivesWesupportgovernmentsinbuildingtheirinternalcapacitytooperatesustainableanalyticspracticeswhichimprovecommunityoutcomesinfieldsfrompublicsafetytoeconomicdevelopmentGovExhasaproventrackrecordofworkinginconcertwithpartnerstoprovidetechnicalassistanceandtrainingatscale
SinceourlaunchinApril2015GovExhas
Trainedover2000governmentemployeesinpersonandonlineintopicsincludingdatamanagementperformancemanagementopendatacommunityengagementandanalytics
Providedtechnicalassistancetomorethan100localgovernmentsacrosstheglobeimprovingtheiruseofdatatoachieveefficienciesinservicedeliveryandbetteroutcomesforresidents
Assessedthecapacityofmorethan130governmentalorganizationstodeterminehowleadersstaffandexternalstakeholdersmanageandusedata
Localgovernmentsareincreasinglyonthefrontlinesofaddressinglargescaleproblemswithglobalimplicationssuchasclimatechangepublicsafetyandeconomicinequality
Whileevidence-basedinterventionstohelpsolvethoseproblemsareavailablelocalgovernmentstaffhavelimitedtimeandcapacitytofindandimplementresearch-basedsolutionsAsaresultcitiescontinuetoworkthewaytheyalwayshave--makingincrementalimprovementstoprogramsbutnotalwayscreatingchangeatasystem-level
ThroughresourcedevelopmentandcollaborationwithJHUfacultyGovExissupportingcityleadersinusingdataandevidencetomakedecisionsandcreatinglastingrelationshipswithresearchers
TheRoadmapwasdevelopedbyGovExaspartofitspartnershipwithBloombergPhilanthropiesWhatWorksCitiesinitiative
AboutGovEx
69
- Title page
- Introduction and purpose of the Roadmap
- Using the Roadmap
-
- Gather information about the problem
- Strategically engage key partners
- Review the evidence and collect details
- Decide on a policy to implement based on local factors
- Establish a framework to monitor progress
- Assess data accessibility quality and security
- Design implementation and evaluation
- Share your story and success
-
- Working with a range of evidence
- Thank you
- References
- Resources
- About GovEx
-
tofindtherootcauseoftheproblemandavoidsimplyaddressingsymptomsoftheproblemandrepeatingthemistakesofhistoryToolssuchasjourneymapscanhelpyouunderstandtheexperienceofthosewhoareaffectedbytheproblem
TheCentreforPublicImpactwithsupportfromBloombergPhilanthropiesdevelopedanexercisetosupportcitiesinidentifyingtherightsizedproblemAccordingtoCPIarightsizedproblemisonethatislargeenoughtopotentiallyapplytoothercitiesbutsmallenoughtosolvewithinareasonabletimeframe
AtthebeginningofMayorBerkesadministrationintheCityofChattanoogaTNveteransexperiencinghomelessnesswereaskingtheCityforhelpinfindinghousingTalkingtothispopulationrevealedthatveteranshadtroubleidentifyingaffordablehousingwhichmotivatedtheAdministrationtoworkonasolutionTheproblemofveteransexperiencinghomelessnesswastherightsizebecauseitfocusedononesegmentofpeopleexperiencinghomelessness-veterans-andtheirspecificchallengeinidentifyingaffordablehousing
Developacomprehensiveunderstandingofhowyourcitycurrentlyaddressestheissue
IntheChattanoogaexamplementionedaboveCitystaffdiscoveredthatveteranswerehavingtroublenavigatingtheCitysvariousaffordablehousingprogramsandgettingaccesstothehousingandservicesthattheyneededAtthesametimenewlyelectedMayorBerkewasexploringcurrentandfutureopportunitiestoservethispopulationRatherthancontinuetousefederalfundingforemergencysheltersMayorBerkedirectedcitystafftoprioritize
Gatherinformationabouttheproblem
11
HousingFirstwhichvaluesconnectingpeoplewithpermanenthousingovertransitionalhousingandsupportiveserviceafterspeakingwithrepresentativesattheUSDepartmentofHousingandUrbanDevelopment(HUD)
ThisexaminationofwhattheCityiscurrentlyfundingandthesourcesofthatfundingwasaturningpointinChattanoogaUnderstandingthisproblemfromasystemsperspective-includingpoliciesandfundingstreamsatmultiplelevelsofgovernmentsocialnetworksthroughwhichinformationisoftenspreadtechnologythebuiltenvironmentofmarket-rateandpublichousing-helpedthenewMayoranalyzeexistingeffortsandleadstaffandotheractorsinthesamedirection
Begintogatherthedata
TotrulycapturethesuiteofservicesthatyourCityisprovidingaskdepartmentstafftolisttheprogramsrelatedtoyourproblemstatementthatyourcityrunsorfundsWhenlistingtheprogramscapturedetailssuchas
programnamethenameofthedatasetthatisrelevanttoeachprogramadescriptionofthedatasetthesourceofthedatasetifthedataisinternalorexternalwhetherthisdataisalreadybeingcollected
ThisinformationwillformthebasisofyourinventoryofadministrativedataordatasetsthatarekeptbygovernmentsforoperationalpurposesThismayincludecallcenterrecords(311911)taxdatapropertyassessmentsinspectioninformationsocialservicerecordsetcAtthispointlistallrelevantadministrativedatasetsYouwillprioritizewhichdatasetstogathermoreinformationonlaterwhenyouaredevelopingindicators
Asampletemplatethatcanbeusedtocapturethisinformationisbelow
ProblemStatement____________________________________________
Gatherinformationabouttheproblem
12
ProgramName
DatasetTitle
DatasetDescription DataSource
Departmentorexternalpartner
DoestheCityalreadycollectthisdata
Whatisthenameoftheprogram
Human-readablenameofthedataset
Shouldbeunderstandablebynon-technicalusers
Whatistheoriginalsystemapplicationfilethathousesit
Departmentorexternalpartnerthatmaintainsthedata
Isthisdataalreadycollected
TogatherthisinformationconsiderassigningonepointpersontosurveyorinterviewcitystaffSpeakingwithfrontlineworkersstaffwhoworkwithfederalgrantsdepartmentheadsandeveryoneinbetweenwillensureadeepunderstandingoftherelevantprogramsanddatasetsDuringthecourseoftheseconversationsbesuretoaskcitystaffwhichorganizationstheyworkwithexternallyandcapturethatinformationaswelleveniftheyarenotrecipientsofcityfundingInadditiontoconversationswithexpertsincitygovernmentyourcitysbudgetdocumentsareanothergreatsourceofinformation
Collectdataandobservetrends
Nexttrytogainanunderstandingofwhytheproblemstillexistsifithasbeensomethingyouhaveworkedonforawhileorwhyitisemergingnowifitisnew
AnalyzingadministrativedatacandemonstratepasteffectivenessofinitiativesrelatedtoyourproblemstatementUsingthedatasetinventorythatyoubeganintheprevioussteptalktodepartmentstaffaboutwhichdatasetsaremostrelevanttotheproblemandgatherafewyearsworthofhistoricaldataBegintolookfortrendsinthisdatadiscussingpotentialreasonsforthosetrendswithrelevantstaffCanthetrendsbeexplainedbyseasonalitychangesinfundingstaffingfluctuationsorsomeotherreasonBasedonthehistoricaldataanalysiscreateabaselineagainstwhichfuturechangescanbeobserved
OtherdatasetscanhelpprovideadditionalcontextForinstanceifthenumberofpeopleexperiencinghomelessnessinyourcityisincreasingitmaybehelpfultotakealookathousingaffordabilityIfyoudonothavethedatayouneedconsideraproxymeasureAsanexamplehousingisoftenconsideredaffordableifitcosts30orlessofahouseholdsincomeButifhouseholdincomedataisnotavailableatthecitylevelAreaMedianIncome(AMI)canbeahelpfulproxy
Gatherinformationabouttheproblem
13
QualitativedatacanalsohelpanswerquestionsaroundtheeffectivenessofpreviouseffortsTrytoobtainavarietyofperspectives--bothmainstreamandoutliers--fromnetworksbeyondyourowntoprovideyouwiththegreatestvantagepointforevaluatingpotentialimpactUnderstandinghowthoseeffortswerereceivedbydifferentpartsofyourcommunityandiftheremaybeelementsofprevioussystemstopreservewillbecrucialtothesuccessofanypolicychangeThisinformationmaybeobtainedthroughfocusgroupsinterviewsandsurveysFormoreinformationonhowtousethesequalitativeresearchtoolspleaseseetheseresourcesdevelopedbytheAgencyforHealthCareandResearchandUrbanInstitute
FinallytakealookatnationaldatatoseewhereelsethisproblemmightexistThisresearchmaybehelpfulinlaterstepswhenyouarelookingforpromisingpoliciesinothercities
Puttingitalltogether
UsethetemplatebelowdevelopedbyDenversPeakAcademytocaptureyourrefinedrightsizedproblemstatementandcurrentandfuturestatesWorkingthroughthisexercisewithpartnersishelpfultogainbuy-inTheresponseswillbeimportantwhengettingotherpartnersonboard
WhyChangeIsNeeded
Yourright-sizedproblem
CurrentState
Analysisoftrendsindata
FutureState
Ideallywhataretheoutcomesyouhopetoachieve
Checklist
Thekeystepstogatheringinformationarebelow
Atthispointfocusontheproblemratherthanjumpingtosolutions
Listentopeoplewhoareaskingforhelpandtoadvocateswhoareraisingissues
Beasspecificaspossibleinidentifyingtherootcauseandhoninginonthepartoftheproblemthatyourcitycanaddress
Identifytrendsbasedonmultiplesourcesofdataincludinghistoricaladministrativedataproxymeasuresqualitativedataandnationalmeasures
Gatherinformationabouttheproblem
14
Captureyourproblemstatementtheresultsofyouranalysisanddesiredoutcomes
Gatherinformationabouttheproblem
15
StrategicallyengagekeypartnersPolicytheoryresearchoftenfocusesonincreasingpolicymakersaccesstoevidenceasawaytoavoidtakingshortcutslikemakingdecisionsbasedonemotionImplementationsciencesuggeststhatincreasingaccesstoevidenceisnotenoughandthatpolicychangeshouldbeexaminedinthewidercontextofdebatecoalitionformingandpersuasionMeaningfulpolicychangerequiresldquoalong-termstrategytoformalliancesanddevelopknowledgeofpolicymakersandpolicymakingratherthanusingevidenceinpolicymakingonanadhocorone-offbasis
UnderstandingnetworksandsystemsisimportanttodevelopsustainablepolicychangeThissectionismeanttohelpreaderstakethestepstobuildengagementstrategiesforpartnersthatrepresentdifferentsystemsandnetworksEngagementshouldbeanongoingprocessofconnectingpeopletooneanotherandtoanissueEngagementshouldnotbeinnameonly-wehaveallexperiencedengagementactivitiesthatgivepeopleaspecificroletoplaythatleadstoaneatpredictableoutcomeAccordingtotheBuildWithmovementundervaluingandunderinvestinginengagementstrategiescanleadtoadisconnectofcertainindividualsorgroupsandmistrustofgovernmentWhendevelopingengagementactivitiesrememberthathowpeopleengageandwhethertheyengagehastodowithhowyoustructureopportunitiesforengagement
Conveneacoreadvisorygroup
AnadvisorygroupcomprisedofpeoplefrommultiplesectorscanbehelpfulasyouconsiderthelocalsocialenvironmentalandpoliticalcontextinwhichyouintendtointroducepolicychangeRevisityourstakeholdermapanddeterminewhowouldbebesttoserveonthisgroupTomakesureyouareconsideringtheproblemsandsolutionsthroughanequitylensincludepeoplewithlivedexperiencewhohaveexpertiseintheeffectivenessofsystemswhetherthosesystemsfunctionasintendedandhowtheymayimpacthumanbehaviorYoumayalsowishtoincludesomeonewithexperienceworkingwithdataandanalysis-thispersoncanhelpdeterminethetypeofdatathatmaybereadilyavailableandusefultocollectandcanalsohelpplanandshapeyouranalysistofigureoutwhichprogramsareeffectiveBesuretoinvitefront-lineworkersaswell-itscriticalthatwhateversolutionscomeoutofthisprocessaremeaningfultothosewhowillberesponsibleforimplementingthem
BringingpeopletogethercanbechallengingbutthereareafewwaystosetandmanageexpectationsFirstmakesurethatthegroupisawareoftheproblemstatementcurrentstateandfuturestate-thiscanhelpstarteveryoneoffonthesamepageNextspend
Strategicallyengagekeypartners
16
sometimegettingtoknoweachotherrsquosworktolearnabouteveryonesstrengthsThissharedunderstandingcanhelpasyouclarifyrolesandresponsibilitiesofeachpartnerinthegroup--thisisimportanttoavoidfinger-pointinginthefutureandtoensurethatnoneofthetasksslipthroughthecracksInsomecasestherearemanyactorswithdifferentresponsibilitiesasinthecaseofBaltimoreCitys38-memberYouthFundTaskForcewhichischairedbynonprofitandcorporatepartnersandincludesrepresentativesfromthenonprofitandfaithcommunitiesstudentsthebusinesssectorlocaluniversitiesCityCouncilandtheMayorsOffice
OncerolesandresponsibilitiesareclearyoumayconsiderdevelopingacharterguidelinesortermsofreferencetosetexpectationsforthegroupBesuretodistinguishthisfromotherboardsandcommissionsthatyourcitymayhaveNextyoumaywishtoestablisharegularongoingmeetingscheduleBesuretosendagendasandmeetingmaterialsaheadoftimeandclarifytheobjectiveofeachmeeting
InSeattletheCitydevelopeditsHousingAffordabilityandLivabilityAgenda(HALA)torespondtorisinghousingcostsandhomelessnessThisisatrendinotherwestcoastcitiesaswellwhichmaybeexplainedbyaninfluxofnewresidentswhoareattractedbytheemergenceoftechjobstheattractionoftheoutdoorlifestyleortherecentpassageofprogressivepolicieslikelegalizedgaymarriagea$15minimumwageandlegalizationofmarijuanaSeattlebroughttogetheramulti-stakeholderadvisorygroupcomprisedoflocalhousingexpertsrentersandhomeownersdevelopersandgovernmentofficialstoshareinformationabouttheirneedsanddeveloprecommendationsbasedontheseneedsTheinterestsandmotivationsofthisgroupwerevariedandittookabouttenmonthstofindcommongroundTheseriesofrecommendationsthattheadvisorygroupcreatedismeanttoserveresidentswithaspectrumofneeds
Buildaprocessforcommunityparticipation
Ofcoursenoteveryonecanbeincludedinanadvisorygroupandadvisorygroupmembersmaynotnecessarilyrepresentthevariedperspectivesoftheirpeers-forinstancenoteveryhomeownerhasthesamesetofneedsTogatherinputfromawiderswathofthepopulationitcanbehelpfultobuildaprocesstoinvitethepublicatlargetosharetheirideasCitiesdothisinavarietyofwaysfromonlineconversationsandsurveystoin-personmeetingsRememberthatbeingintentionalaboutgatheringtheviewpointsofpeopleismoreimportantthanusingthelatesttechnologyandtools
TheinsightgeneratedbycommunityparticipationexercisescanbeincludedinyourreviewofevidencewhichisdiscussedinasubsequentchapterInWestVirginiaastatethathasbeenhithardbytheopioidepidemicstatehealthofficialsdraftedanopioidresponseplanthatwasinformedbypublicengagementaswellaslocalandregionalsubjectmatter
Strategicallyengagekeypartners
17
expertsThepublicwasinvitedtopresentcommentsonideasfordecreasingoverdoseearlyinterventiontreatmentandrecoveryPublichealthandmedicalfacultyfromWestVirginiaUniversityMarshallUniversityandJohnsHopkinsUniversityprovidedresponsestothecommentsTheresultwasanOpioidResponsePlanthathealthcommissionerRahulGuptapresentedtoGovernorJimJustice
AsamplespectrumtoplanyourowncommunityparticipationprocesscreatedbyLaurenellenMcCannisbelow
SourceLaurenellenMcCann
Community-basedparticipatoryresearchisanotherwaytoworkwiththecommunitytoimproveoutcomesCBPRmeansthecommunityisinvolvedinallaspectsoftheresearchprocesswhichisofteniterativeInCBPRtraditionalresearchersandcommunitymembersshareknowledgeaswellasresourcesandcredit
CBPRInActionInSanFranciscoanenvironmentaljusticenonprofitteamedupwiththeCitysDepartmentofPublicHealthandanoutsideevaluatortounderstandtherootcausesofobesityandfoodinsecurityinlow-incomeneighborhoodsandidentifysolutionsThroughresearchmethodssuchasinterviewingresidentsandstoreownersdiagrammingstoreshelvesGISmappinganeconomicanalysisandreviewofsimilarpoliciesinnearbycitiestheteamobservedthatthestoresthatresidentshasveaccesstowerestockedwithlesshealthyfoodsandthatresidentswantedmorehealthyfoodandlessalcoholandtobaccoThisprojectresultedinlocalandstatelegislationtoencouragestorestostockfreshproduce
Strategicallyengagekeypartners
18
Developrelationshipswithsubjectmatterexperts
OnewaytokeepsubjectmatterexpertsengagedistoinvolvetheminthecommunityparticipationprocessasdescribedaboveIntheWestVirginiaexamplesubjectmatterexpertsrespondedtoandsynthesizedthepubliccommentsOtheropportunitiesaredescribedbelow
AdvisoryAcitydepartmentheadMayororstaffmembermaydevelopaone-on-oneadvisoryrelationshipwithuniversityfacultymemberoracollaborativebetweentheCityandUniversitytoaddressaparticularproblemAgreatplacetostartistolookforrelevantpracticecenterswithinyourlocaluniversityandreachingouttosubjectmatterexpertsInadditiontotheirexpertiseontherelevantpolicytopicheorshemayalsobeabletoidentifyotherexpertsandevenprovidesupportonhiringtherighttalenttoleadcertainprogramsAnadvisoryrelationshipwithauniversitysubjectmatterexpertcanalsohelpwhencitiesareconsideringbringinginnewprogramsTheseexpertsareoftenfamiliarwithprogramsthatarebasedonnationalmodelsandcanprovideinsightintoaprogramsoriginandlineageeffectivenessinotherplacesandthepotentialforimplementationinyourcityAcademicscanalsohelpidentifyindicatorsbasedongovernmentindicatorsorliteratureFinallyrelationshipsbetweenpolicymakersandresearchersprovidethemostbroadlyeffectiveroutetopullingresearchintopolicy
ResearchandevaluationAcademicexpertsmaysupportcitiesinevaluation-indeedevaluatingcityinitiativesisatraditionalmodelofcityuniversityrelationshipsSharingtheinformationyoucollectedintheGatherInformationphasecanhelpresearchersunderstandyourworkandgoalswhichinturnwillstrengthenthesupportthattheycanprovideDiscussingdetailsaboutyourinterventionandevaluationplanatthebeginningoftheprocessmeansthataresearchpartnercanhelpwiththeevaluationdesignanddatacollectionForinstancetheWilsonSheehanLabforEconomicOpportunitieswalkscitiesthroughconsideringthefollowingquestionsbeforepartneringonanevaluation
1 Isthereaclearpolicyprogrammodelwithobjectivescreeningcriteria
2 Isitpossibletoevaluatewhathappensbothwiththepolicyprogramandintheabsenceofthepolicyprogram
3 Arethereoneormoremeasurableoutcomesofinterest
4 Arethereenoughpeopleforrigorousquantitativeanalysis
5 Whatdatawillbenecessaryforthestudyandwhataretheprocessesforaccessingthesedatasource(s)
6 Isthereacompellingreasontoevaluate
Strategicallyengagekeypartners
19
InadditiontoassessingreadinesstoconductanevaluationresearcherscanhelpyouhoneinonthepurposeoftheevaluationTheWorldBankdescribesseveralpotentialusesofevaluationwhicharedefinedhereInbriefevaluationcanhelpdecisionmakers
makeresourceallocationdecisionsrethinkrootcausesofproblemsidentifynewproblemsdecideonabestalternativedeterminewhetherprogramsorpoliciesarebeingimplementedwithfidelitydeterminewhetherprogramsorpoliciesarehavingthedesiredimpact
Researcherscanalsoprovidesupportinsharingtheresultswiththescientificcommunitythroughpapersandconferenceswhichisvaluableincreatingapeernetworkofevidence-basedcitypractitioners
TrainingandtechnicalassistanceUniversitypartnersespeciallythoseatpracticecenterscanbehelpfulindevelopingorupdatingtrainingcurriculumtrainingthetraineranddeliveringtrainingtocitystaff
Strategicallyengagekeypartners
20
ReviewtheevidenceandcollectdetailsOnceyouhaveathoroughunderstandingoftheproblemdataandactorsyouwillneedtoconductyourownresearchonavailableevidenceandlearnaboutthelandscapeofpolicyoptionsYourgoalwillbetodevelopanaccessiblelistofinformationaboutpoliciesthathavehelpedsolvesimilarproblemsinotherplacesThischapterwillguideyouthroughthestepsofpreparingyourreviewersunderstandinglevelsofevidenceandcollectingthedetailsonevidencethatyouwillneedfordiscussionwithdecisionmakersinyourcity
Selectandprepareyourreviewers
AssignapersonorteamtoreviewrelevantinterventionsthathavedemonstratedsuccessinotherjurisdictionsTheirgoalwillbetoproduceareviewofalloftherelevantprogramspoliciesorinitiativesthatyourteamcouldconsiderimplementingTheywillbeconductingaliteraturereviewsupplementedbyinterviewswithexpertsandpeerswhohaveinformationaboutpotentialprograms
SincethepeopleyouassigntothistaskareprovidingthebaselineofpotentialoptionstoimplementitisimportantthatyouchoosepeopleyouareconfidentwillbethoughtfulandthoroughintheirworkItisalsocriticalthattheyareawareoftheimportanceofsourcecredibilityEnsureyourreviewershavefamiliarizedthemselveswithrulesforcriticallyanalyzinginformationsourcesbeforetheygetstartedIftheyareincludingnewsarticlesaspartoftheirsearchtheyshouldalsobeawareofmethodsforevaluatingnewscredibility
AskyourreviewerstodevelopaplanandtimelinetocompletethereviewThisstartswithsolidifyingagoverningresearchquestionOnewaytoknowyouarecreatingadetailedresearchquestionistousetheacronymPICOremindingyoutoincludereferencetothepopulationintervention(ifyouhaveoneinmind)comparisonandoutcomeelementsinyourresearchquestionForexampleifyouknowthatyouhaveapopulationofteenagerswhoarehomelessyouhavelearnedfromyourstakeholdersthatmanyoftheseteenagersidentifyasLGBTandyouhaveheardsomethingabouttheapproachofcreatingdrop-incentersfordeliveringsupportyoucouldask
ForLGBTyouthfacinghomelessness(population)howeffectivearedrop-incenters(intervention)relativetostandardsocialservices(comparison)atsettlingtheminpermanenthousing(outcome)
Ifyoudonothaveaninterventioninmindyoucanleaveouttheinterventionandcomparisonandsimplyask
Reviewtheevidenceandcollectdetails
21
HowcanwehelpLGBTyouthfacinghomelessness(population)obtainpermanenthousing(outcome)
OnceyourreviewershavetheirresearchquestionsusethequestionstocreatealistofkeywordstheycanuseintheirliteraturesearchThiswillhelpkeeptheircollectionofarticlesreasonablyfocusedHavethemdescribewhattheirsearchprocesswillbeincludinglistingthedatabasestheywillreviewandtheirplanforstoringtheinformationtheywillcollect(Herearesomeadditionalsuggestionsforstructuringareviewplan)
WhatyourreviewerswillbedoingisalightweightversionofarapidreviewTotheextentthattheywouldliketodevelopamethodologyforspecificsearchprotocolsincludingcriticalappraisalprocessestheycancertainlygoaheadanddoso
Usedifferentlevelsofevidenceforthemostcompletepicture
AsdiscussedintheWorkingwitharangeofevidencechaptertheNestaStandardsofEvidenceframework--reproducedbelow--ishelpfultounderstandthelevelofconfidencethataprogramorinterventionishavingapositiveimpactThisapproachletsyouputinterventionsthatyouhaveheardaboutlocallybutwhichhavenotbeenstudiedtogetherwiththosewithademonstratedcausaleffectThisishelpfulsinceanumberofpoliciesprogramsandpracticesthatyouwillwanttoconsidermaynotyethavemuchofaresearchrecord
Reviewtheevidenceandcollectdetails
22
NearlyanypieceofinformationaboutaninterventioncanbeclassifiedwithinthestandardsofevidenceForexampleifsomeonehasacrediblehypothesisabouthowaninterventioncouldproducebenefitthistypeofevidencewouldbeclassifiedasLevel1ThistypeofevidencecouldincludeinterventionsinothercitiesthatyoureadaboutinthenewsorhearaboutataconferenceForinstancewhenUSDAallowedschoolsandschooldistrictstoprovidefreebreakfastandlunchtoallstudentsthroughtheCommunityEligibilityProvision(CEP)manydistrictsimplementedthisoptionevenbeforetherewasevidenceofeffectivenessThiswouldfitintoLevel1sinceitbuildsonthelogicthatprovidingaccesstomealswilldecreasefoodinsecurity
YoumayalsofindpiecesofinformationthatqualifyasLevel2evidenceForinstanceasmoreschooldistrictsimplementedCEPprincipalstestifiedaboutpositivechangestheyobservedinstudentssuchasamorepositivedemeanorthatcanleadtobetteracademicperformanceDataalsoshowedthatthestudentsdescribedwerereceivingmoremealsthanbeforetheprogramsimplementationHoweveritwastoosoontobuildevidencetoshowacausalrelationshipbetweentheincreaseinfreemealsandincreasedacademicperformance
ldquoLower-levelrdquoevidencecanconsistofnuancedbutvaluableelementsthatrelatetoaninterventionrsquospotentialfitwithacommunityQualitativeinformationsuchastestimonyfrompeoplewhohaveexperiencedaninterventionandthosewhohavefacilitatedorimplementeditwillalwaysbeusefultoconsiderandprovidesdetailedlocalcontext
Experimentalstudies--describedstartingatLevel3withcontrolledexperimentsandgoingthroughLevel5withfullyfleshedoutprotocolsandprocessesforreplication--stilloccupythehigherlevelsofthisdescriptionofevidenceThisprioritizationshowsthevalueofrigorousstudyforevaluatingtheimpactofaparticularpolicyorprogramratherthanotherfactorsthatcouldbeinfluencingoutcomesofinterestRememberthatmultipletypesofevidencecancomplementrigorousimpactevaluationsandgivecityleadersadditionalinsightsandbroaderperspective
Accessinformation
Encourageyourreviewerstobegintheirsearchforpromisingpracticesingovernmentandnon-profitclearinghousesandinacademicpolicyjournalsWorkpublishedinthesesourceshasalreadybeensubjectedtoahighlevelofprofessionalscrutinyandgenerallycontainsagreatdealofdetailwhichmeansthatyourreviewersaremorelikelytobeabletopullouttheinformationyouneedtomakeagooddecision
Clearinghouses
Reviewtheevidenceandcollectdetails
23
Herearesomeoftheclearinghouses(collectionsofprogramevaluations)forevidence-basedpoliciesandprograms
ResultsFirstClearinghouseDatabaseThePewCharitableTrustsCollectstogethereightotherclearinghousestoprovideinformationoninterventionsinthefollowingcategoriesadultcriminaljusticeeducationchildwelfarehealthjuvenilejusticementalhealthsubstanceabuseandyouthdevelopmentIndividualclearinghousesincludedinResultsFirstare
BlueprintsforHealthyYouthDevelopment-foryouthdevelopmentandviolenceprevention
CaliforniaEvidence-BasedClearinghouse-forchildwelfareinterventions
TheLauraandJohnArnoldFoundationrsquosSocialProgramsThatWork-forsocialservicesprograms
TheUSDepartmentofEducationrsquosWhatWorksClearinghouse-foreducationinterventions
TheUSDepartmentofHealthandHumanServicesrsquoResearch-TestedInterventionPrograms(RTIPs)-forcancercontrolinterventions
TheUSDepartmentofHealthandHumanServicesrsquoTeenPregnancyPreventionEvidenceReview(TPPEvidenceReview)-forsexualriskbehaviorandsexualhealthoutcomes
TheUSDepartmentofJusticersquosCrimeSolutionsgov-forcriminaljusticejuvenilejusticeandcrimevictimservices
TheUSSubstanceAbuseandMentalHealthServicesAdministrationrsquosNationalRegistryofEvidence-basedProgramsandPractices(NREPP)-substanceuseandmentalhealthinterventions
TheUniversityofWisconsinPopulationHealthInstituteandRobertWoodJohnsonFoundationrsquosCountyHealthRankingsandRoadmapsWhatWorksforHealth(WhatWorksforHealth)-healthbehaviorsclinicalcaresocialandeconomicfactorsandthephysicalenvironment
UKWhatWorksCentresTheBritishgovernmenthascharteredanetworkofsubject-specificorganizationswhichcollectsynthesizeandreportoutevidenceontheeffectivenessofpolicyprogramsandpractices
NationalInstituteforHealthandCareExcellence(NICE)-healthandsocialcare
Reviewtheevidenceandcollectdetails
24
SuttonTrustEducationalEndowmentFoundation-educationalachievement
CollegeofPolicingWhatWorksCentreforCrimeReduction-crimereduction
EarlyInterventionFoundation-earlyintervention
WhatWorksCentreforLocalEconomicGrowth-localeconomicgrowth
CentreforAgeingBetter-qualityoflifeforolderpeople
WhatWorksCentreforWellbeing-wellbeing
WilsonSheehanLabforEconomicOpportunities(LEO)UniversityofNotreDameProvideslinkstotheirevaluationsofinterventionsincriminaljusticeself-sufficiencyeducationhealthandhousingandhomelessness
ClearinghouseforLaborEvaluationandResearchUSDepartmentofLaborProvidesresearchonworkforceandlabor-relatedpoliciesandprograms
J-PALEvaluationsAbdulLatifJameelPovertyActionLabStudiesonpoverty-reducinginterventionsfromaroundtheworld
CampbellSystematicReviewsCampbellCollaborationSystematicreviewofpolicyinterventionsbyscholarlycoordinatinggroupsinsevenissueareascrimeandjusticeeducationinternationaldevelopmentmethodssocialwelfareknowledgetranslationandimplementationanddisability
Benefit-CostResultsWashingtonStateInstituteforPublicPolicyAWashingtonState-charteredprogramtoestablishthelikelihoodofrelativecostsandbenefitsforprogramsacrossthefollowingareasjuvenilejusticeadultcriminaljusticechildwelfarepre-Kto12educationchildrensmentalhealthhealthcaresubstanceusedisordersadultmentalhealthpublichealthandPreventionworkforcedevelopmentandhighereducation
EffectiveInterventionsCentersforDiseaseControlandPreventionRepositoryforevidence-informedHIVpreventioncareinterventions(withstepsonhowtoimplement)
Journalsandindexes
OnceyouhavereviewedrelevantinformationaccessiblethroughclearinghousesitisworthtakingalookatindividualstudiespublishedasscholarlyarticlesYourreviewerswilldothisbysearchingarticleindexesandacademicjournalsUnfortunatelyanumberofusefulacademicjournalsandindexingplatformsmaynotbegenerallyaccessibletoyourreviewersifyoudonothaveasubscriptiontothemToaddressthisconsiderpartneringwithalocal
Reviewtheevidenceandcollectdetails
25
collegeoruniversitytoprovideyouwithalistofarticlesfromthefollowingsources(Thisisalsoanexcellentopportunitytoseekresearchsupportfromanyacademicstakeholdersyouidentifiedpreviously)
Indexes
GoogleScholarincludesmostpeer-reviewedonlineacademicjournalsandbooksconferencepapersthesesanddissertationspreprintsabstractstechnicalreportsandotherscholarlyliteratureincludingcourtopinionsandpatentsItallowsyoutospecifyapublicationrangeinyoursearchandprovideslinkstolikelysourcesforalllistedarticles
PolicyFileIndexisauniqueresourceforUSpublicpolicyresearchinthatitgrantsusersaccesstotimelyupdatedinformationfromover350publicpolicythinktanksnongovernmentalorganizationsresearchinstitutesuniversitycentersadvocacygroupsandotherentities
PolicyArchiveworkstocollectanddisseminatesummariesandfulltextsvideosreportsbriefsandmultimediamaterialofthinktankuniversitygovernmentandfoundation-fundedpolicyresearch
Anothersourceforhigh-qualityresearchisthereportsoftheCongressionalResearchServiceYoucanfindtheirreportsaggregatedhere
Specificjournals
Oftenthemostusefulacademicjournalsforunearthingpolicyorprogramreviewswillbeineachspecificsubjectarea--thatisineducationjournalsforeducationpoliciesYoucanuseageneralindexlikeGoogleScholartobegintolearnwhichjournalsaremostoftenusefulforyoursearchHoweveryoucanalsosearchthefollowingtwogeneralistpolicypublicationsforissuesandarticlesrelatedtoyourtopic
PublicAdministrationReviewJournalofPolicyAnalysisandManagement
Newssourcessubjectmatterexpertsandinterviews
AlthoughitisvaluabletoknowwhathasbeenexploredthroughascholarlyresearchframeworkitisalsousefultolearnwhatkindofinterventionsarebeingtriedinthefieldrightnowandhowpeoplefeelaboutthesetrialsSearchsubscriptionnewsaggregatorslikeLexisNexisorfreenewsaggregatorslikeGoogleNewsorYahooNewsforyourissueareaplusthewordpolicyorprogramtoseewhatkindsofinterventionsarebeingcoveredinthenews
Reviewtheevidenceandcollectdetails
26
DonotforgettocontacttheadvisorygroupmemberswhowerechosenfortheirsubjectmatterexpertiseInadditiontoprovidinggoodmaterialthroughinterviewsthemselvestheymayhaveideasaboutothersubjectmatterexpertsyoushouldcontacttoaskaboutpotentialinterventions
Peercitiesandnetworks
InadditiontoreadingabouttheworkofothercitiesitisidealifyouareabletospeakwithpeoplewhoareinvolvedwithcurrentimplementationsofinterestinginterventionsInyourcityyoulikelyhaverelationshipswithyourneighborsbutitcanbehelpfultolookoutsideofyourregionforsolutionsaswellTofindcounterpartsinothercitieswhoareworkingonaprogramyouwouldliketolearnmoreaboutyoucancontactstaffinintermediaryorganizations(likeGovExoryourprofessionalassociations)whomightbeinapositiontoputyouintouchimmediatelyorwhomightseeanothergoodrouteforsettingupaconversationHereisalistofnetworksthatmaybehelpful
InternationalCityCountyManagementAssociation(ICMA)NationalAssociationofCounties(NACO)NationalLeagueofCitiesUSConferenceofMayorsWhatWorksCities
Collectthedetails
Onceyourreviewershavefoundevidenceonpossiblepoliciesorprogramstoimplementtheyshouldcapturekeydetailsthatwillhelpdecisionmakersunderstandthepotentialeffectivenessandapplicabilityofeachpotentialintervention
Foreachsourcethathasprovidedinformationyouwanttoconveyusethetemplatebelowtocaptureasmanyofthefollowingdetailsaspossible(Youdonotneedtocompletethisforsourcesthatprovideduplicativeinformationorareotherwisenotthatuseful)
Programorpolicyname (egldquoHousingFirstrdquoorldquoblightedpropertiesregistryrdquo)
Sourcetype (egjournalarticleinterviewsubjectmatterexpertstudyassociationcommunityparticipationprocess)
Sourcedetails (egarticletitleandpublicationnameandtitleofinterviewee)
Evaluationmethodology
(egrandomizedevaluationcomparativeevaluationoftwodifferentgroupsbeforeafteranalysis)
Evidencelevel (egLevel12345)
Programorpolicy
Reviewtheevidenceandcollectdetails
27
description
Programorpolicygoal (1sentencecapturingtheprincipalaimoftheintervention)
Existingimplementations
(WherehastheinterventionbeentriedListasmanysitesasyouareawareof)
Characteristicsofimplementationsites
(ForeachimplementationsiteofferbasicgeographicIDmdashcityandstateThenyouwillwanttocollectthedetailswhicharerelevanttoyourdecisionmakersrsquoconsiderationsaboutwhatmakesalocationldquosimilartordquoorldquodifferentfromrdquoyoursituationYouwillatleastwanttoincludeforeachimplementationsitePopulationnumberPopulationdensity(ruralurbansuburban)Socio-economiccharacteristicsBeforestartingaskyourdecisionmakersaboutanyparticularcharacteristicsofprogramimplementationsitestheywouldwanttoknowbeforeseekingtoreplicate)
Characteristicsofimplementationpartners
(Anyavailabledetailsaboutorganizationsinstitutionsorpeoplewhowereinvolvedwithimplementingtheintervention-egindependentchildcarecenterpublicelementaryschoolcitypublichealthnurses)
Characteristicsoftargetpopulationeligibleprogramparticipants
(Anyavailabledetailsaboutpeoplewhoarethebeneficiariesoftheprogramorotherwiseincludedoraffectedbytheintervention-egnumberofprogramparticipantscategoriesofpropertyownersaffected)
Policyorprogramduration
(HowlonghastheinterventionbeeninplaceineachimplementationsiteHasitendedandifsowhy)
Whatbenefitscamefromeachimplementation
(Anyobservedpositiveeffectsoftheinterventionincludingeffectsize)
Statisticalestimatesofinterventioneffectiveness
(Anyavailablestatisticalestimatesoftheinterventionrsquoseffects)
Whatdrawbackswereobserved
(Anyavailabledetailsaboutunintendedorunavoidablenegativeeffectsoftheintervention)
Programorpolicycosts
(Anyavailabledetailsaboutprogramcostsperimplementation)
Reviewtheevidenceandcollectdetails
28
Reviewtheevidenceandcollectdetails
29
DecideonapolicytoimplementbasedonlocalfactorsTogetthecriticalbuy-infromleadershiptoimplementanewpolicyorprogramtheprojectleadshouldscheduleameetingwithdecisionmakerstopresenttheevidencesurroundingthepolicyoptionscapturedinthetemplateDependingonyourinsightintoyourcityleadershipswillingnesstomakechangeyoumaywishtosharetheresearchyouhaveconductedontherootcausesoftheproblemandsomeofyouranalysisontheimpactofcurrenteffortstodate
Considertheapplicabilityofanapproach
AsdiscussedinthepreviouschapterpolicymakersoftenlookforevidenceofeffectiveprogramsinothercitiesbutoftenthequestionarisesofhowgeneralizableaprogramisfromonecitytoanotherTheAbdulLatifJameelPovertyActionLab(J-PAL)suggeststhatinreviewingevidencefromanotherlocationitishelpfultoidentifythespecificmechanismthatwasused(deliveryofprogramorserviceinformationsharingetc)identifythelocalconditionsthatwereinplaceobservethebehaviorchangethatresultedfromthemechanismandfinallytounderstandlocalcontextandconditions(legalorcostconstraintsculturalnormsetc)Thissetofpresentationelementscanbedrawnfromthelistofdetailscollectedintheresearchtemplate
EducationpolicymakersinChicagoILusedfindingsfromprogramsinKenyaandIndiainwhichchildrenwhohadfallenbehindtheirpeersweregiveneducationalsupporttargetedtotheirleveloflearningThemechanismusedwastargetedteachingandthereweresimilarconditionspresentinallthreecountriesAlthoughsomestudentshadfallenbehindinthecurriculumteacherswereincentivizedtoteachgrade-levelmaterialsConsiderationsofhowthelocalcontextinChicagomightaffectimplementationofatargetedteachingprogramincludedunderstandinghowtheneighborhoodsthatthestudentswerefrommayhaveaffectedtheireducationaloutcomesandthehighcostofteachinginsmallgroupsInChicagoMatchEducationprovidedtwo-on-onetutoringwhichmatchedstudentswithwell-educatedindividualswhohavenothadformalteachertrainingforalower-costmodelTheresultswerepositiveStudentswhoweretutoredperformedbetterinmathandweremorelikelytopasshighschoolclasses
ThegeneralizabilityframeworkproposedbyJ-PALisbelowUsethisasatemplateasyouconsiderpotentialpolicysolutions
Decideonapolicytoimplementbasedonlocalfactors
30
SourceTheGeneralizabilityPuzzle
OnceyouhavenarrowedthepolicyoptionsdiscussthequestionsbelowforeachoptionsyouareconsideringThisstepiscriticalasunderstandinglocalcontextissoimportantinsuccessfulpolicyimplementation
CommunityContextQuestionnaire
Localcontext
DoesthisoptionmakesenseinourcommunityIsthisaonesizefitsallapproachIsanyparticulargroupabsentfromthisapproachIfsoisthatok
Youranswers
Effectiveness
WhatwouldweexpecttoobserveifthepilotisaseffectivehereasinCityX
Youranswers
Timing
DoesthetimingmakesenseIsthistherighttimetoimplementthispolicychange
Decideonapolicytoimplementbasedonlocalfactors
31
WherearethepotentialroadblocksthatcouldaffectthescheduleIsthereanywaytomitigatethisHowlongwouldittaketoseeanimpact
Youranswers
Implementation
DowehavethestaffcapacitytoimplementthisArethereadditionalpartnersthatyoucanbringtothetabletosupportimplementationDoesthismakesensegivenourformofgovernment
Youranswers
Authority
DoesourcityhavetheauthoritytoenactthisIfsowhatistheappropriatemechanismAretherelegalbarriersArethereanyexistingpreemptionissuesfromthestateorfederalgovernmentCanthosebeaddressedthroughlegislationadvocacyIstherepoliticalsupportoroppositiontothis
Youranswers
Cost
HowmuchwillthiscostannuallyHowwillwepayforitCanthepolicybeimplementedwithexistingfunding
IfsoisthisfundingreliableHowwouldthispolicyoptionaffectourcitysbudgetintheshortandlongtermWhatotherfundingoptionscanweconsiderIsthereanalternativethatwouldcostlessandgiveussimilarresultsWillanyconstituencybeadverselyaffectediffundingforprogramsthatimpactthemisshiftedtoanotherprogrampolicyorinitiative
Youranswers
Decideonapolicytoimplementbasedonlocalfactors
32
EstablishaframeworktomonitorprogressNowthatyouhavereviewedtheevidenceconsideredlocalcontextandchosenyourpolicyitistimetodeterminehowyourcitywillmonitorprogressofthisnewapproachCitystaffandleadersneedtoestablishtheinfrastructureforongoingcontinuousdecisionmakingbasedonwhetherthepolicychangeisleadingtothedesiredoutcomesforresidentsandwhetherthechangeismeaningfulandrationaltofrontlineworkersSettingarbitrarygoalsatthelocallevelthatconflictwithgoalsatotherlevelsisconfusingtotheindividualsresponsibleforachievingthegoalsGoalsthataresettoohightoachievecanbedemoralizingtostaff
ThissectionincludesstepstosetupavenueinwhichCitystaffandleadershiproutinelyvisitkeyindicatorstodeterminewhatisorisnotworkingandtomakedecisionsinreal-timeFormorereadingonperformancemanagementpleasevisittheGovExPerformanceManagementGettingStartedGuide
InperformancemanagementtermssuchasindicatormeasureandmetricmaybeusedinterchangeablyInthisguidewewilluseindicatorInyourworkthemostimportantthingistobeconsistentinwhichevertermyouuseThisvideomaybehelpfulinunderstandingthisconceptandcommunicatingittoothers
Establishthefundamentalsandgatheryourteam
FirstdeterminewhetherthereisanestablishedvenueorregularmeetingscheduleduringwhichyoucandiscussprogressIfyourcityhasanexistingperformancemanagementstructureonarelatedtopicaddingadiscussionoftheprogressofpolicychangetoyouragendacanbeagreatideaIfthiswillbeastandalonemeetingdecidehowfrequentlytheprojectteamanddecisionmakerswillmeetandputthosemeetingsonthecalendarfortheentirelengthoftheproject
DeterminewhowillstaffandattendroutinemeetingsonCityprogressThesemeetingsmayincludetheMayorandorCityManagerdepartmentheadstaffwhoareimplementingthepolicychangeandanalysts
Identifyindicators
ItiscriticalthatindicatorsbeanimportantcityandcommunitypriorityaswellassomethingtheCityplaysamajorroleinaddressingTheremaybeinstancesinwhichtheCitywillneedtoreachouttoorcollaboratewithexternalpartnerstogetamorecompletepictureofprogressontheoverallpolicychange
Establishaframeworktomonitorprogress
33
Anoutcomeindicatordescribesthedesiredeffectofthetargetedpopulation
ExampleBellevuesfamilyfriendlyneighborhoodsindicatormeasuresthepercentofresidentswhobelievetheyliveinaneighborhoodthatsupportsfamiliesparticularlythosewithchildren
WithyourkeypartnersrevisityourproblemstatementcurrentstateandfuturestateexerciseandbegintoidentifytheoutcomeindicatorsthatwillillustratesuccessFornowchooseasingletop-levelkeyperformanceindicator(KPI)ItiscriticalthattheKPI-thepieceyouwillmeasure-beapriorityforthecommunityandsomethingtheCityplaysamajorroleinaddressing
Outputindicatorsmeasuretheprocessofprogramorservicedelivery
ExampleSanDiegoMayorKevinFaulconersgoaltorepair1000milesofstreetby2020
ReviewingtheadministrativedatasetinventorybegintoparedowntherelevantprogramsanddatasetsthatsupportyourKPIWhatarethemeasuresofeffectivenessforeachprogramTheseareyouroutputindicators
Thediagrambelowillustratestherelationshipbetweenoutputandoutcomeindicators
ThinkaboutimplementationindicatorsasmeasuringstepsintheprogramsrolloutthatarenecessarytoobtaindesiredchangesImplementationoutcomescanfocusontheinterventionstaffdeliveringtheinterventionanorganizationorsystemandcanaddressfeasibilityadaptabilitysustainabilityandcostTwokeyquestionstoconsiderwhendevelopingimplementationindicatorsare
1 Istheinterventionacceptableandappropriatetothecommunityand
2 IsitfeasibletoimplementinterventionXatthistimewiththeseresources
Establishaframeworktomonitorprogress
34
Amoreextensivelistwithdefinitionsofpotentialimplementationoutcomesislinkedhere
UsethetablebelowtochoosethedesiredimplementationoutcomesforyourpilotandtoplanyourmeasurementtechniqueThisshouldincludelevelofanalysis(individualproviderorconsumerorganizationorsetting)yourmeasurementtechnique(surveysobservationinterviewfocusgroupsadministrativedataanalysis)andthephasewhenyouwillcollectthisinformation(earlymidlate)Theremaybemorethanonelevelofanalysismeasurementtechniqueandphaseofdatacollection
Implementationoutcome
Definitionofoutcome
Levelofanalysisandmeasurement
Yourmeasurementtechnique
Phaseofpilotwhenyouwillcollectdata
Usethespacebelowtocaptureyourindicatorwishlist
Indicatorwishlist
Outcomes_____________________________________________________________________________
Outputs________________________________________________________________________________
Implementationoutcomes______________________________________________________________
Setapublichigh-levelgoal
Thehigh-levelgoaliscriticalinestablishingdirectiontogalvanizethepeoplewhowillbedoingtheworkWhendiscussingthegoalusetheMayorsbullypulpitLookforkeymilestonesinCitygovernmenttosharethegoalstatementsuchasthebeginningofaMayoraladministrationaStateoftheCityspeechorotherpublicremarks
GoalsshouldbeldquoSMARTrdquo
Specificaspossible(innamingthepopulationwhowillbebetterofftheactionthecitywilltake)
Measurable
Achievable
Establishaframeworktomonitorprogress
35
Results-focused
Time-bound
ThegoalstatementshouldincludeyouroutcomeindicatorlikeChattanoogasgoalldquoWewillendchronicveteranhomelessnessbyDecember2016rdquoAdiagramtohepyoubuildaSMARTgoalaroundyouroutcomeindicatorisbelow
Component Description Yourturn
Specific WhowillbebetteroffWhatactionthecitywilltake
Measurable Howwillyoumeasuresuccess
Achievable Isthegoalattainable
Results-focused Whatistheexpectedresult
Time-bound When
Setrealisticambitioustargetsandtimelinesforeachindicator
ToestablisharealisticandambitioustargetforyourgoalstatementusetheadministrativedatacollectedforyourchosenKPItoreviewactualCityperformanceinthemostrecentyearcomparedwiththethreeorfourmostrecentyearsworthofdataBegintocalculatesummarystatistics(egtotalminimumaveragemedianmaximum)toidentifytrendsCreatecumulativeandmonthlychartstoidentifytrendsandanomaliesBasedonyouranalysisofhistoricaldataignoretheanomaliesandassumetheproductivitytrendswillfollowthesameoverallpatternfrommonth-to-monthFinallyapplyanincreasedproductivityfactortotheperformanceeachmonthtoforecastfutureprogressBasedonthisanalysischooseatargetthatinspiresimprovementwhileaccountingforreality
Manycitiesthataremorematureintheirperformancemanagementpracticesrecognizetheimportanceoftakingacommunity-levelapproachtoimprovingoutcomesforresidentsThesecitiesarecreatinggoaltaxonomiesthatincludemeasuresrelatedtopublicsafetyhousingandevenrecreationCitiesdriveprogressonthesegoalsbyimplementingstrategiesthatengagemultipledepartmentsandcollaboratewithserviceprovidersinthecommunity
EventuallyyouwilldevelopgoalstatementsfortheoutputsinordertocapturetheefforttoachievetheoutcomecontainedinyourgoalstatementYoursetofindicatorsshouldbecomposedofshort-andinterim-termsupportivemeasureswithaccuratetargetsandrelevanttimelinesAnexamplegoaltaxonomyfromtheCityofLasVegassResultsVegasprogramisbelowUsethistaxonomyasamodelforyourownsuiteofindicatorsandgoalstatements
Establishaframeworktomonitorprogress
36
Yourturn
Thetemplatebelowismeanttohelpyoucraftagoalstatementforeachsupportingmeasure
Goalstatement
Departments
Outcome
Activities
KPI
Target
Timeframe
Conductanddiscussrelevantanalysis
ThrougharoutineseriesofmeetingscontinuallymonitorprogressofKPIssupportingindicatorsandanyotherdatacollectedsuchasqualitativedatatoidentifytrendsthatwillindicatewhetherthechosenpolicyishavingthedesiredimpactThesedatapointsshoulddrivethediscussionduringthesemeetingspointingtoopportunitiesforfurtherinquiryinto
Establishaframeworktomonitorprogress
37
whatisdrivingtheobservedtrendsAnalystsshouldbeencouragedandtrainedtoaskquestionsofthedepartmentsproducingthedataandconductsitevisitswhenpossibletodeepentheirunderstandingofthedataPresentationoftheanalysisshouldfocusononeideaatatimetokeeptheconversationfocusedonkeypointsandspecificopportunitiesforimprovementYoumayconsiderdevelopingapublicfacingdashboardtosharethisprogressmorebroadly
AgeneralcomparisonofthestatusquodatawithwhatwasobservedduringthepilotcanprovidedecisionmakerswithaquickunderstandingofthedifferencesbetweentheoldandnewapproachesalthoughthiswillnotshowcausalityOftenthesecomparisonswillincludeoperationalmetricssuchasdaysordollarssavedbutrememberthattheimpactonpeoplewillbemostimportantIfyouareworkingwithanevaluationpartnertheywillsupportthisstep
Usethespacebelowtosummarizetheresultsofyouranalysis
MeasureTargetvalue(seesectiononsettingtargetsabove)
Pilotvalue(whatdidyouobserveduringthepilot)
Historicalaverage(ifavailable)
Results(WhoisbetteroffIsanyonenegativelyimpacted1-2sentencesreassessingyourgoalstatement)
DocumentyourprocessandprogressincludingfeedbackfromtheparticipantsimplementationpartnersandtheprojectteamDidanythingunexpectedoccurduringthepilotArethereinternalandexternalfactorsthatmayaffectyouroutcomesThisiswheretheimplementationoutcomescanbecanhelpfulIftheinterventiondidnotgoasplannedconsiderwhetheritisbecausetheinterventionwasnotacceptableorfeasible
UsethetemplatebelowcreatedbyDenversPeakAcademytocapturethisinformationIfthe306090daytimelinedoesnotmakesenseforyourcityfeelfreetocustomizeit
CurrentState
FutureState
30Day
60Day
90day
Whatwentwell(+)
Whatdidnotgowellshouldbechanged(-)
IfaquestionordiscussionpointarisesthatcannotbeansweredinthemeetinganassignedstaffmembershouldcapturethequestionduringthemeetingAfterwardthefollowupitemsshouldbesenttorelevantpartnersandthenextmeetingshouldbeginwithadiscussionof
Establishaframeworktomonitorprogress
38
thoseitems
Takeaction
DiscussionsaboutwhetheryourindicatorsaremovingintherightdirectionandwhatmightbedrivingtrendsmayleadtoadiscoverythattherearenotenoughresourcesallocatedtothispolicyorissueTheremaybelimitedfundingforstaffprogrammingortechnologythatisdrivingthedatainthewrongdirectionIfCityleadershipisintheroomthisisagreattimetomakethecaseforadditionalfundingduringthebudgetseasonBycollectinganalyzinganddiscussingdataonthispolicyroutinelyyouarebuildingevidencetomakethecasefortherightleveloffundingtoreachtheestablishedtargets
Establishaframeworktomonitorprogress
39
Assessdataaccessibilityqualityandsecurity
Determineaccessibilityandqualityofdata
OnceyouhavecompiledyourwishlistofindicatorsitistimetodeterminetheaccessibilityandqualityofthedataTopreventcomplicationyoumaywishtoprioritizerelevantindicatorsforwhichyouarealreadycollectingdataAvoidthetemptationofadoptingatechnologysolutiontocollectnewdataatthispointsuchasaphotosurveytoolorotherappFornowyouaretryingtodeterminewhichdatasetsarerelevantaccessibleandhighqualityBesuretoworkwithyourkeyinternalpartnerswhileyouareidentifyingyourmeasuresDepartmentstaffwhoworkonprogramsarefamiliarwiththeindicatorsthatwillaccuratelysupporttheoutcomesanddatabaseadministratorswillbevaluablewhendiscussingaccessibilityandqualityofthedata
Abrieflistofconsiderationsfortheaccessibilityofyourdataisbelow
ArewecurrentlycollectingthisdatasetorwillwebecollectingitforthefirsttimeHowfarbackdoestherecordgo(timehorizon)WouldweneednewtechnologytocollectoranalyzethedataCanthedatabecollectedmanuallyorcanitbeautomatedHowfrequentlyisthedataproduced(continuouslyhourlydailyweeklymonthlyquarterlyyearlyetc)IsthedataavailablefortheentireCitypopulationorasubsetofthepopulationHowgranularisthedata(individuallevelCensusblockneighborhoodcitywideetc)DoesthedatacontainsensitiveinformationIsthereaproxymeasurethatwecanusefornowIsthedatamachinereadable
AsnotedbytheWilsonSheehanLabforEconomicOpportunities(LEO)attheUniversityofNotreDameleveragingadministrativedatacansignificantlyreducethecostsandtimelinesofrigorousevaluationCitiescanestablishaninternaldatasharingagreementacrossdepartmentsincludinglanguagethatenablesadministrativedatatobeusedforresearchpurposesCitiescanalsoidentifyexternaladministrativedata-holderssuchasacountyschooldistrictpolicedepartmenthospitaloranotherstatethatmayhaveoutcomedataimportantformeasuringprogramimpactCitiescanleverageexistingrelationshipsandorworkwithresearchinstitutionstodevelopstreamlineddatasharingprocessesandagreementswiththeseexternalentities
Assessdataaccessibilityqualityandsecurity
40
Evaluatethequalityofyourdata
QualityisasimportantasaccessibilityToevaluatedataqualityyoumaywishtocreatearubricliketheonebelow
Value ValueDescription
3 ExcellentDataQualityThedataiscompleteaccurateandupdatedregularly
2 AdequateDataQualityThedataissomewhatcompleteaccurateandupdatedfairlyregularly
1 PoorDataQualityThedataisnotcompleteaccurateorupdatedregularly
ThisisagreattimetobuildonthedatainventorythatyoustartedinthegatherinformationpageSuggestedadditionalfieldstoaddtoyourdatainventorytemplatearebelow
Program DatasetTitle
DatasetDescription DataSource
Departmentorexternalpartner
DoestheCityalreadycollectthisdata
Whatisthenameoftheprogram
Human-readablenameofthedataset
Shouldbeunderstandablebynon-technicalusers
Whatistheoriginalsystemapplicationfilethathousesit
Departmentorexternalpartnerthatmaintainsthedata
Isthisdataalreadycollected
Updatefrequency-ThefileformatforthedatasetTypicallycsvxlsxshapefiletxtetc
Format-Howoftenthedatasetisupdated
Timerange-Theperiodoftimecoveredinthisparticulardataset
Datause-Howisthisdatasetcommonlyused
Datausers-Whotypicallyusesthisdataset
Dataownerstitle-Titleofthedataowner
Dataownersposition-DataOwnersjobtitle
Dataownersemailaddress-DataOwnersemailaddressWhomanagesthedataandorisresponsibleforgrantingpermissiontoaccessthedataWhounderstandswhatthedatasetincludesandcananswerquestionsaboutit
Dataqualityconcerns-Describetheconcernsyoumayhaveaboutthequalityofthedataset
Assessdataaccessibilityqualityandsecurity
41
Division-DivisionwherethedataismaintainedIfmorethanonedivisionisresponsiblelisttheprimaryownerofthedataset
Sensitivedatacomments-DescribetheissuesthatwouldpreventthecityfromsharingthedatapubliclyCommonsensitivityconcernsincludeprivacyviolationssecurityissuesorhighcostorstaffdemandstopostthedata
Publishingstatus-Thistheplacetoputanycommentsregardingthecurrentstatusofthedataset
Publishedlink-Thisisthelinktothepublisheddatasetontheopendataportalorcitywebsite
Reviewdatasecurityandprivacyprotectionprotocols
OftenapolicychangealsoinvolvesachangeinthedatathatisbeingcollectedhowthatdataisbeinghandledwhohasaccesstoitandahostofotheradjustmentsAsyoutakestockofthedataassetsthatareavailableforyourinitiativeitisvitaltoalsothinkthroughwhatwillbedonetoprotectthatdataandtheprivacyofthepeopleitrepresentsAbigpartofgooddatasecurityinvolvesstrongdatagovernancepracticesthatdefineandidentifysensitivedataandconsciouslyworktobalancethevalueandriskofstoringsuchsensitivedataOnewayofevaluatingyourdatacomesfromcaselawrelatedtotheUSFreedomofInformationActwhichwehavetranslatedtothechecklistbelow
1 Woulddisclosureresultinasubstantialinvasionofprivacy2 Haveweconsideredtheextentorvalueofthepublicinterestandthepurposeorobject
oftheindividualsseekingdisclosure3 Istheinformationavailablefromothersources4 Wastheinformationgivenwithanexpectationofconfidentiality5 Isitpossibletomouldreliefsoastolimittheinvasionofindividualprivacy
TheseriskscouldbemitigatedusingavarietyofstrategiesfromaggregatingdatatoobfuscatingindividualstoredactingfieldsthatmayincludePersonallyIdentifiableInformation(PII)orre-evaluatingyourdatacollectionmethodstofindalternativewaysofgettingtheinformationyouneedThesestepsshouldbecodifiedandmanagedbyadatagovernancecommitteeifpossibleForthesensitivedatathatisdeemedessentialcreateaplanaroundpotentialdatabreachesinordertoquicklyandconfidentlyrespondtoemergencysituations
ProactivelypreparingforthesesituationsisoftenthebeststrategyfordatasecuritybutdonotforgetthetechnicalsideInvolveyourITadministratorsinconversationsaboutdatastorageandaccessensuringthattheywillbeprotectedbyenterprise-gradecybersecuritysystems
Assessdataaccessibilityqualityandsecurity
42
Assessdataaccessibilityqualityandsecurity
43
Designimplementationandevaluation
Designtheimplementation
NowthatyouhaveyouraccountabilityplanandkeyindicatorsitistimetogetspecificaboutyourplanforimplementationThetemplatebelowwillhelpyouthinkthroughthespecificmechanismsofimplementingpolicychange
UnderstandingorganizationsandsystemsiscriticaltounderstandingandmakingdecisionsaboutwhereandhowinterventionsshouldbeappliedApolicyimplementedinasinglesettingmayshowpromisingresultsbutimplementingmultiplepoliciesandprogramsatthesametimewithdifferentactorsresponsibleforeachaspectcanhelpensuresustainablebehaviorchange
InSomervilleMAShapeUpSomerville(SUS)isacitywidepublichealthprogramthatisimplementedacrossmultiplelevelsofinfluenceandsystemsthroughinterventionsbeforeduringandafterschooltoincludechangestobuiltenvironmentsuchasbikelanesregionalmasstransportationandphysicaleducationequipmentinschoolsandgymstoincreasephysicalactivitywellnesspoliciesandschoolgardensnutritioneducationduringandafterschoolandhealthyrestaurantprogramstosupporthealthyfoodandhealthyeatinghabits
Usethetemplatebelowtoplanyourimplementation
Policyimplementationplanningtemplate
ListthespecificdesiredoutcomesInthepreviousstepyouworkedwithdepartmentstafftoestablishspecificmeasurableoutcomesandinterimindicatorsPrioritizetheindicatorsthatwilldemonstrateprogressUsethespacebelowtocapturethedesiredoutcomesofthepolicychange
ExampleShapeupSomervillebeganwiththegoalofpreventingobesityin1stthrough3rdgraders
Yourturn
____________________________________________________________________________
____________________________________________________________________________
Designimplementationandevaluation
44
SpecifythetargetpopulationAreyoudesigninganinterventionthattargetsonlycertainsitesAreyouplanningonimplementingaprogramorinitiativeforaspecifictargetaudienceClearlyspecifythepopulationyouplantotarget
IfyouplantoconductarigorousimpactevaluationasnotedbytheWilsonSheehanLabforEconomicOpportunitiesitisimportanttoestablishconcreteobjectiveeligibilitycriteriaandensurethatthesecriteriaareappliedconsistentlybyallstaffduringscreeningandintakeItisalsovitaltoidentifytheindividualswhoareeligibleforaparticularprogramorserviceevenifeveryoneeligiblecannotbeservedbyexistingresourcesThishelpscitiesidentifyanyunmetneedsandguidefutureresourceallocationwhilealsocreatingthepossibilityofarigorousimpactevaluationbyidentifyingapotentialcomparisongroupFinallywhenconductingscreeningandintakeitisimperativetocollectandmaintainbasicinformation(egnamedateofbirthgender)aboutallprogram-eligibleindividualsnotjustthosewhoparticipateintheprogram
ExampleIntheShapeUpSomervilleexampletheprogrambeganbytargeting1stthrough3rdgraders
Yourturn
____________________________________________________________________________
____________________________________________________________________________
DefinethelocationWherewillthepolicychangetakeplaceWillimplementationtakeplaceatonesiteormultiplesites
ExampleShapeUpSomervilleinterventionsweredeliveredinschools
Yourturn
____________________________________________________________________________
____________________________________________________________________________
ClarifyimplementationpartnersandarticulatetheirrolesWhatarethekeyorganizationsinstitutionsorpeopleresponsibleforimplementationTheprojectteamshouldbeasclearandspecificaspossibleinprovidingadescriptionoftheconcreteactivitiesthatimplementationpartnersaremeanttocarryoutPracticeprofilesmaybeahelpfultooltohelpdefinewhatisrequiredoftheactorswhoareresponsibleforimplementingtheintervention
Designimplementationandevaluation
45
ExampleInSomervillemayoralleadershipandengagementofmultiplecityagencies(publichealthprimarycareschoolsCambridgeHealthAllianceetc)streamlinedandconnectedthevariousShapeUpSomervilleinitiativesandgrantswhichhelpedtokeepthetalentandenergyfocusedonhealthyeatingandactivelivingforallsectorsofSomervilleSUSpartnersultimatelyincludedschoolsnonprofitsCityandstateagenciesandnationalfoundations
Yourturn
____________________________________________________________________________
____________________________________________________________________________
DescribetheinterventionWhatarethestepsactivitiesorprocessesRememberthatthemechanismthespecificstepsorprocessthatleadtobehaviorchangeiskeytounderstandingtheeffectivenessofthepolicychangeandsharingtheresultswithothers
ExampleGoalsofShapeUpSomervilleweretoinfluenceeverypartofanearlyelementaryschoolchildrsquosdayndashbeforeduringandafter-schooltopromotehealthyeatingphysicalactivityandtheirweightstatusInterventionincludedfoodserviceenhancementstoimprovebreakfastandlunchandstaffprofessionaldevelopmentwalktoschoolactivitiesSUSclassroomcurriculumandprofessionaldevelopmentschoolwellnesspolicydevelopmentSUSafter-schoolcurriculumandprofessionaldevelopmentoutreachandeducationtohomethroughmaterialsforumseventsldquoSUSapprovedrdquorestaurantscommunityoutreachandcapacitybuildingthroughpolicydevelopmenttrainingsmediaplacements
Yourturn
____________________________________________________________________________
____________________________________________________________________________
ClarifytimelineandmilestonesfordeliveryandevaluationHowfrequentlywilltheservicesinterventionbedeliveredHowlongwilltheprogramrunWhatisthetimetoexpectedimpacts(eg12monthsafterprogramcompletion)
Designimplementationandevaluation
46
ExampleShapeUpSomervilleledbyTuftsUniversityFriedmanSchoolofNutritionScienceandPolicyconductedaquasi-experimental(nonrandomized)interventionstudyamong1stthrough3rdgradestudentsinSomerville(intervention)andtwocomparisoncommunitiesfor9-monthsduringoneschoolyear(fall2002ndashspring2003)ThestudyshowedpositiveassociationoftheinterventiononchangeinBMISUSreleasedreportssummarizingevaluationsprogresstodateplansforadditionalprojectsandimpactstenandfifteenyearsintotheinitiative
Yourturn
____________________________________________________________________________
____________________________________________________________________________
PlanyouranalysisTheprevioussectionaddressedestablishinganaccountabilityframeworktomonitorprogressandimplementationDeterminewhichindicatorswillbediscussthroughthatprocesswhowillsharethedataandanalysisandhowfeedbackfromthoseconversationswillbeincorporatedbytheprojectteam
ExampleTheMayorrsquosofficeimplementedadata-drivendecisionmakingapproachcalledSomerStatwhichwassynergisticwiththeapproachestakenbytheresearchandevaluationteamsdeployedforthespecificgrants
Yourturn
____________________________________________________________________________
____________________________________________________________________________
PlanforsustainabilityIfthepolicyandprogramchangeissuccessfulhowwilltheprojectteamsustainandexpandtoadditionalsitesorpopulations
ExampleAlthoughmostofthepartnershipsforgedintheearlyyearsweremaintainednewgrantsandcollaborationsexpandedattentiontoolderpublicschoolstudentsactivetransportationandtheGreenLinesubwayextensionthebuiltenvironmentfarmerrsquosmarketeffortssuchasldquoGrowninSomervillerdquowithUnionMainStreeteconomicdevelopmentandhealthdisparitiesaswellasdatacollectionmonitoringandsurveillance
Yourturn
Designimplementationandevaluation
47
____________________________________________________________________________
____________________________________________________________________________
Pilotplanningtemplate
IfyouplantoconductapilotfirstdeterminethepurposeofthepilotDoyouwanttoobservewhetherthepolicychangeresultsinthedesiredbehaviorchangeAreyouconcernedaboutfeasibilityscalabilityorcostDoyouwanttotestwhetheryourdesiredresultscanbeachievedthroughthispolicyapproachClarifythegoalofthepilotandthequestionsyouhopetoanswerThiswillhelpyouidentifythekeyindicatorsthatyouwillusetomeasurewhetherthepilotissuccessfulBeforebeginningestablishabaselineofcurrentperformanceandsetyourtargetsaccordinglybasedonwhatyouhopetoaccomplishthroughthepilot
NextestablishthepilotplanforyourprojectteamincludingmilestonesforteamcheckinstodeterminewhethertomoveaheadorchangecourseThisisseparatefromthemeetingsthatwilloccuraspartoftheaccountabilityframeworkUsethespacebelowtocreateyourtimelineanddevelopyourmeetingschedule
Task Personresponsible Startdate Checkindate Deliverable
Planyourevaluation
WhileyouaredesigningtheimplementationitisalsotimetobegintodevelopyourplanforanalysisAsnotedbytheWilsonSheehanLabforEconomicOpportunitiesdecisionsaboutprogramsetup-includingoutreachscreeningintakedatatrackingandenrollment-candeterminewhetherornotarigorousimpactevaluationwillbepossibleAsdiscussedcityleadersmayfindithelpfultoincludeevaluationpartnersinearlyconversationsaboutprogramsetupDuringpreviousstagesyoumayhaveworkedwitharesearchertoplanyourproposedpolicychangeYoushouldbeclearaboutthepurposeoftheevaluationanddevelopanevaluationplanBelowareadditionalconsiderations
Causationvstrendanalysis
IntheEstablishaframeworktomonitorprogressstagetherewasguidanceontrackingtrendsovertimeItisimportanttonoteherethatmonitoringdataisdifferentfromanalyzingacausaleffectTrendanalysiswillillustratethatchangesareoccurringandmayleadtoquestionsthatcouldbeansweredbyyourevaluationbuttrendanalysisalonedoesnotprovideevidenceaboutwhatisdrivingthosechangesYoumayseechangesinyour
Designimplementationandevaluation
48
outcomeindicatorsbasedonvariablesoutsideoftheCityscontrolForinstanceachangeinfederaltradepolicycanhaveanimpactemploymentratesinyourCityoutsideofyourCitysnewjobtrainingandplacementprogram
ToaddresscausalityyoumustcomparetheoutcomesofindividualswhoreceivedtheinterventionwiththosewhodidnotWithoutacomparisongroupofsimilarindividualswhowerenotexposedtotheinterventionitisimpossibletoknowwhichfactorsareresponsiblefortheoutcomesRandomizedControlTrials(RCTs)inwhichindividualsarerandomlyassignedtotreatmentandcontrolgroupsarethegoldstandardinprovingcausalityIfyoucandemonstratethatoutcomesimprovedforthosewhoreceivedtheinterventionanddidnotimproveatallorimprovedlessforacomparisongroupyoucanbesomewhatcertainthattheinterventionwasthecause
Examplesofothernon-causalimpactevaluationsthatarerigorousandreliableincludecertainquasi-experimentalstudiesusingcomparisongroupdesignssmartinterruptedtimeseriesanalysesthatmodeltrendsovertimeinwaysthatallowestimationofinterventioneffectswelldesignedpre-posttestanalysesneedsassessmentsimplementationstudiesetcFormoreinformationondifferenttypesofevaluationsseetheSageResearchMethodswebsite
Lookfordifferencesingroups
BuildintimetoalsoexaminethepilotdatatoseeifgroupsareexperiencingdifferentresultsReviewingperformancebasedonoutcomemeasuresforapopulationinaggregatemayhidethesubtlewaysthatsomepeoplemaybeaffecteddifferentlysoitisnecessarytocarefullyanalyzethedatafordisparateoutcomesForinstancetheMovingtoOpportunity(MTO)studythatexaminedtheimpactofhousingvouchersonfamilieswholivedinhigh-povertyneighborhoodsfoundthatchildrenwhomovedbeforetheageof13hadbettereducationandhigherincomesasadultsHoweveradultsdidnotseeimprovededucationandemploymentoutcomes
Disaggregatingdataintomorespecificcategoriesofraceandethnicitycanhelpunderstandtrendsanddisparitiesinamoregranularwaywhichcanhelpdecisionmakerstargetpoliciesandinterventionsinamoreequitableway
EarlierintheRoadmapweworkedtoidentifystakeholdersandwhomightbepositivelyandnegativelyimpactedbyachangeinpolicyUsetheresultsofthatexercisetoinvolvethecommunityininterpretingdataanalysisandtogatherqualitativefeedbackfromthecommunitytocallattentiontoissuesForinstanceanincreaseinyouthdetentionmaybeexplainedbyinefficienciesorlackofaccesstotransportation
Designimplementationandevaluation
49
Designimplementationandevaluation
50
Shareyourstoryandsuccess
Sharetheresults
Storytelling
AnintegralstepinyourstrategyissharingtheresultsofyourpilotorpolicychangewithyourcommunityThisincludesdevelopingmessagingaroundtheworkofyourorganizationandcreatingaplantodisseminateyourstoriesWhytellstoriesaboutyourwork
Informandorunderstandasubjectorsituation
Giveneededcontextofdataorbitsofinformation
Relatablestoriesarememorableforreadersviewers
ReadersorviewerswillseetheinformationasmorecompellingwhenthereisademonstratedandrelatablechangeorimpactonthemselvesortheirfriendsneighborsandcolleaguesStoriesareaboutpeopleandgovernmentisfundamentallyabouthelpingpeopleSotellthestoriesofthepeopleyouandyourcitycolleagueshelped
Keepthesetipsinmindasyoudevelopyourstorywithyourteam
WhatwereyourfindingsandwhyshouldanyonecareTellyouraudiencehowtheproblemwassolvedWhathastheimpactbeenonresidentsandcitystafftodateWhatwasthesuccessandweretheremultiplesuccessesalongthewayIfyouhavepreliminaryresultsordatashowingimprovementtosharethisisthetimetodoit
WhowasinvolvedandimpactedReviewthelistofstakeholdersHowwereeachofthemimpactedbytheprogramHumanizeyourstoryandimpactbyaddingadditionalvoicesThesevoicescanbeoneofthefollowing
Apersonorfamilywhowaspositivelyaffectedbythechangesuccess
Aninvolvedthird-partydescribingboththeproblemandthesolution(ieanorganizationnonprofitprovidersbusinesstradeassociationsportsteametc)whoispartofthecommunitythatthegovernmentworkedwithonthechangesuccess
Someonefromthegovernmentitselfwhocanspeakaboutthechangesuccess
TellmoreaboutyourcityDescribeyourcityincludingpopulationdemographicsindustriesetcWhatareitschallengesWhatisitknownforWhatmakesituniqueHowisitsimilarordifferentfromothercitiesintheUS
Shareyourstoryandsuccess
51
DescribeyourcityrsquoschallengeWriteoutthechallengeyourcityisfacing(asdefinedinyourproblemstatement)HowdoesitaffectresidentsHowdoesthisproblemaffecttheworkofcitystaffmembers
AddthedetailsProvidedetailsonyourappliedresearchstrategyTalkaboutwhatyourcityhasdonenthepasttoresolvetheissueDiscusshowyourstrategyhaschangedwhenyoubegantoadoptthisapproachWasmoneysavedorinvested
Usethisstorymapexercisetoorganizeyourthoughtsandguaranteethatyouincludealloftheelementsofatraditionalstory
AfteryoudevelopyourmessagingcreateacommunicationsmarketingandoutreachstrategydesignedtospreadyourmessageYourstrategyshouldwalkyouthroughthefollowingkeyactivities
1 IdentifyandprofileyouraudienceWhencreatinganddisseminatingyourmessageitisalwaysgoodtothinkaboutyouraudienceWhoexactlywouldbeinterestedinwhatyouhavetosayThegeneralpopulationofthecityPeoplewhoworkinnonprofitsBusinessownersItmattersbecausenoteveryonewholivesinacitymaybethetargetoftheinformationBusinessownersmaybemoreinterestedindatashowingeffectsonthebusinessdistrictthanothersandsoonWewillcoverthislaterbutthisstepwillalsohelpyoutothinkofthemosteffectivewaytoreachyouraudience
2 Defineyourcalltoaction(ifyouhaveone)SpendsometimeconsideringwhyyouareinterestedintellingyourstoryDoyousimplywanttoupdatethepubliconyourprogressorcelebrateyoursuccessesAreyouinterestedinyouraudiencechangingitsbehaviororsupportingaspecificinitiativebeinglaunchedbyyourorganizationIfyouwanttohaveyouraudienceactorbehaveinacertainwaybespecificandincorporateitinyourmessaging
Shareyourstoryandsuccess
52
3 SelectperformancemeasuresThinkabouthowyouwoulddetermineifyourcampaignissuccessfulandselectappropriateperformancemeasuresYouwillrevisitthesemeasuresattheendofyourcampaigncommunicationsplanimplementationtodeterminewhetheryouneedtomodifyyourcommunicationsandoutreachstrategyforthefuture
4 MethodstocommunicateyourstoryTherearemanywaystocommunicatesuccessofyourprojectHerearesomesuggestionsonreachingaudiencesItisimportanttodeterminewhichmethodsarethemosteffectiveforyourdesiredaudiencetoreceiveinformationDoyourresearch(PewResearchCenterhasexcellentstudiesonthereachofmediadigitalmediaandmarketing)ForexampleifseniorsareyourtargetpopulationandtheyarelessinclinedtouseInstagram(thatistrue)youmaywanttoconsiderFacebookorthenightlylocalnewsbothofwhichareusedmoreoftenbyseniorsBelowisalistofsomemethodsandtoolsyoucanuse
i Pressreleasestolocalnewspapersandtvstations
ii ShortsmartphonevideosAgreatwaytoengageanaudienceisthroughvideoAskonequestionofsomeonewhoispartofthestoryMaybethereisasocialworkerwhocandiscusshungerinyourcityThesevideoscanbereusedonmultipleplatforms
iii Long-formwriting
iv TellingindividualstoriesthroughaseriesofphotosandshortnarrativesforpublicationonFacebookorInstagram
v Op-edsopinionpiecesplacedinthelocalpaper
vi Emailupdatestoyourlist
vii Connectingwithappropriateorganizationsthatareinvestedinthetopicandworkwiththecommunityandaskthemtodisseminateinformationtotheirmembers
viii EmployingldquoWordofMouthrdquo(WOM)tacticsbycreatingtownhallsorfreeeventsforresidentstoattendintheirownneighborhoods
5 DeterminethetypeofcollateralUsuallycampaignsthattargetresidentsrequireseveralavenuestoreachthemPeoplearenotalwaysinthesameplaceorspaceHavingakeenunderstandingofyouraudiencewillassistyouwithyourchoicesonwhatneedstobedevelopedSometimesthatmeansimagesforsocialmediaaonepageflierthatgivespeopleinformationonattendinganeventoranin-depthbrochure
Shareyourstoryandsuccess
53
6 ImplementyourstrategyKeepinmindstrategiescanbefluidSometimessomethingmaynotturnoutasexpectedevenifduringplanningsessionsitwasldquosupposedtordquoItisokYoucananalyzewhathappenedduringadebriefsessionandonceyoursurveysandothermeasurementshavebeenfinalized
Visualizations
Withtherightvisualizationyoucanreviewandshareinformationinsightsanalysisperformanceandtrendsallofwhichisvaluabletoacross-sectionofthepublicButwhichvisualizationisrightforyouTochoosetherighttypeofchartgraphdeterminethetypeofquestionyouwanttoanswerSomecommoncharttypesarebelowandmoreinformationcanbefoundhere
Exploreadditionalanalyticaltechniques
Duringthegoal-settingphaseyouestablishedoutputoutcomeandimplementationmeasuresBasedontheresultsofyourpilotyoumayneedtoadjustyourmeasuresWiththepilotdatainhandyoumayhavesomeideasofhowfutureanalysescouldbedifferentandpotentiallyimprovethequalityofyourprogramTakesometimetobrainstormotherrelevantdatasetsthatyoudidnothavethetimetopursueearlierincludingmessydepartmentaldatathatneedsmoreattentionorsourcesofpublicdatafromthestateandfederalgovernmentnonprofitsandevenprivatecompaniesthatcouldsupplementyourunderstandingofanissueUsingmachinelearningtechniquesyoumightbeabletobetter
Shareyourstoryandsuccess
54
predictdemandforhomelessservicesusingNOAAweatherforecastspreemptwatermainbreaksusingsoilstratigraphyorinvestineconomicdevelopmentbymonitoringrealestatelistingdata
Advancedanalyticaltechniquescanalsohelpyourimplementationrunmoreeffectivelybyintegratingdata-informedtargetinganddecisiontoolsintotheprocessByautomatingroutineworkflowsandsurfacingsubtletrendsthesetoolscanhelpyouiterateinamoreprecisemannerandprovidekeystrategicguidancetofront-lineemployeesthatwouldotherwisebeunavailableThesesortsofdata-science-enhancedprocessescanbeclassifiedasoneofsixprojecttypesasexploredfurtherbytheNewOrleansLANOLAlyticsteamFormoreguidanceonscopingasimilarprojecttheGovExadvancedanalyticsscopingguidegivesyouastep-by-stepoverviewofhowtobesuccessful
Usethespacebelowtolistyourideasforpossibleadvancedanalyticsprojectsbytype
Typeofproject Ideas
Findingtheneedleinthehaystack
Prioritizingforimpact
Earlywarningtools
Betterquickerdecisions
Optimizingresourceallocation
Experimentingforwhatworks
Keepthefeedbackloopgoing
GatherinformationonpublicsentimentaboutpolicychangethroughfocusgroupssurveysandpublicmeetingsledbyadvisorygroupmembersNotewhethernewinformationisavailablethatcanhelpanalyzeorcontextualizeyourprogress(egfederalorstatedatareports)Sharetheresultsofyourpilotandthepolicychangeprocesswithothercitiesandresearchers
Shareyourstoryandsuccess
55
WorkingwitharangeofevidencePresentlythereisalotofattentionandinvestmentinevidence-basedpolicymakingincludingthefederalbipartisanCommissiononEvidence-BasedPolicymakingthePewCharitableTrustsreportsandtoolkitsonHowStatesEngageinEvidence-BasedPolicymakingandtheLauraandJohnArnoldFoundationsupportedEvidence-BasedPolicymakingCollaborativeManyoftheseinitiativesfocusonrigorousevaluation--meaningrandomizedcontrolledtrials(RCT)orotherexperimentalmodels--asthebestkindofevidencetouseHoweverthisbodyofevidenceisofteninsufficientforthespecifickindsofquestionsthatgovernmentscanhavewhenmakingadecisionaboutpolicyInthisguideweproposeanapproachtosupportgovernmentsinincorporatingadditionaltypesofevidenceintotheirdecisionmaking
OnewaytounderstandarangeofevidenceistouseNestarsquosStandardsofEvidencetoframeanddefineevidenceTheBritishnonprofitNestaworkstoinjectpositivechangeintothewayUKgovernmentpoliciesandprogramsfunctionOneoftheirkeyareasofworkhasbeendevelopingWhatWorksCentresorganizationsthatcollectcommissionsynthesizeanddisseminateevidenceaboutinterventioneffectivenessacrossarangeofsocialimpactareasNestadevelopedthefollowingStandardsofEvidencetoprovideausefulwaytodocumenthowdifferentkindsofinformationdemonstratedifferentlevelsofevidence
Workingwitharangeofevidence
56
Theimportanceofpotentiallybiasedinformation
Onechallengepresentedbyanevidenceclassificationthatacknowledgesthevalueofnon-scientificformsofevidenceistofigureouthowtobalanceallofthepiecestogetherwhenyouusethemTounderstandhowtodothiswehavetothinkabouthowtopositionpotentiallybiasedinformationindecisionmaking
WhenwelookatinformationwhichhasnotbeenrigorouslyevaluatedweacceptthatthisinformationmightbebiasedThewholereasonweconductrigorousevaluationistoremovebiasfromourobservationsWeconductstructuredtrialstoproduceresultsthatareconsistentandgeneralizableIfafindingisunbiasedthisaccuracymdashthislackofbiasmdashcanbeprovedbecauseanyindependenttestercanreproduceitwhentheyfollowanidenticalprocess
YoucanlookatpotentiallybiasedinformationasjustinformationthatisearlieroninitspathtorefinementreplicationandverifiedaccuracyApieceofinformationbasedonanyoneindividualrsquosobservationsislikelytobebiasedbecauseitiscomingfromjustonesourceBiasisreducedbygatheringtogetherlotsofindependentsourcesofinformationandfiguringoutitsmostcommonattributes
MaybethebiasedinformationmakesaclaimthatistoolargemdashitsayssomethingistrueunderallconditionswhenitisreallyonlytrueundercertainconditionsorthataneffectofaninterventionisbiggerthanitstrueaverageorthataneffectlastsovertimewhenitdoesnotPerhapschangingtheclaimtobemorespecificmdashcriticallyreviewingtheinformationtoseeifspecificconditionsmustbemettoachievetheoutcomesmdashwouldmaketheclaimmoreaccurate
IfyoutestedyourclaimfounditdidnotholdupinsomecaseschangedittesteditagainuntilyourclaimrepeatedlyheldupthenyoucoulddemonstratethattheclaimwasnolongerbiasedbutaccurateThisprocessoftestingimprovingtestingandvalidatingiswhatwegetfromstructuredscientificstudy
Withindividualpiecesofinformation--crediblehypothesescasestudiesexpertopinion--youcannotassumethatkindofaccuracyAtthesametimetheseearlyobservationsarefoundationaltoanyreplicablescientificobservationsWewillnevergetanywhereifwedonotstartwiththesepiecesofinformationeveniftheymightbebiased
Combininglevelsofevidencetogetthemostdetailedinformationforyourcity
ItisespeciallyimportanttoacceptdifferentformsofevidencewhenyoufacequestionsthathavenotyetbeentestedbyindependentstudiesSomeofthemostimportantquestionsrelatetothingsthatareveryspecifictoyoursituationYoufaceaparticularcombinationof
Workingwitharangeofevidence
57
issuesactorstrendsandresourcesthatisultimatelydifferentfromeveryotherplaceintheworld
ThisisimportantbecauseaspolicypractitionersandnotscientistsyoucaremoreaboutyourspecificcasethanahypotheticalaveragegeneralcaseYoucarelessaboutthetotallikelyrangeofimpactsfromapolicythanhowthissingleimplementationwillworkYouwantinformationaboutpoliciesthatwillprovideyouwiththebestpossiblepredictionabouthowtheywillworkinyourcontext
ByputtingdifferentkindsofevidenceintothesameframeworkNestarsquosStandardsofEvidenceimplythatwhenyoucannotgetrigorouslytestedexperimentallyverifiedinformationitisappropriatetoworkwithwhatwedohaveThisisrelevantforthemunicipalimplementationcontextbecausenostudypulledfromaliteraturereviewwillaccountforallofthevariablesyoufaceinyourowncontext
Intheabsenceofhavingastudythatpredictshowaninterventionworksinyourspecificcontextyouwillneednon-scientificinformationtotrytofillinthosegapsYouwillneedtogatherstoriesfromresidentsstaffmembersandotherstakeholdersthatsitonLevels1or2ofthestandardsofevidence
Someofthekindsofinformationyouwillprobablygetmoreeasilyfrompeoplethanstudiesinclude
howpeopleincitieslikeyoursreactedtoaninterventioninpublicforumshowpeoplewhoimplementedaninterventioninacitylikeyoursfiguredouthowtogetitpassedhowthecitycommunicatedpubliclyabouttheapparentbenefitsanddrawbacksoftheinterventionthepoliticalacceptabilityoftheintervention
Youcanremainsimultaneouslyawarethattheinformationyouaregatheringthroughindividualstoriesisbiased--notgeneralizableinformedbyspecificperspectives--whilealsorecognizingthathavingsomeinformationaboutsomethingimportantisbetterthannoinformationEvidencefromthefirsttwoevidencelevelsareessentialbuildingblocksofknowledgeandperfectlyusefulwhenyoucannotgetdetailedunbiasedknowledge
Youhavetostartsomewhereandyoushouldalwaysstartwithevidenceyouhaveaboutyourveryspecificconditionsthatyouknowareimportantforsuccess
Workingwitharangeofevidence
58
Thankyou
Interviewswithcities
GovExheldsemi-structuredinterviewswithsixstaffmembersinfourcitiestoestablishtheinitialstepsoftheRoadmapWeaskedrespondentstotalkaboutaninitiativeorpolicythattheyhavebeenresponsibleforimplementingintheircityRespondentsfromthreecitiesspokeabouttheireffortstoaddresshomelessnessandrespondentsfromonecityspokeaboutblightAfterrespondentsidentifiedtheproblemtheyspokeabouthowthecityprioritizedthisissuethevariousstepsinvolvedandbarrierspresent
GovExconductedtextanalysisoftheseinterviewswhichrevealedthatpeoplewasthemostcommonlyusedtermduringtheseconversationswhendiscussingthestepstochangeaprogramorpolicyThisisathemethroughoutthisRoadmapThefirstsectionbeginswithidentifyingtherelevantpeopleTheconversationsaroundevidencearefocusedonhumanbehaviorratherthanmethodologyorstrengthofevidence
Thankyou
59
OncewehadadraftoftheRoadmapwefollowedupwithcitiesonanindividualbasistolearnmoreaboutwhatwouldbehelpfulWearegratefultothefollowingcitystafffortheinterviewsandindividualconversations
MichaelBaskinChiefPolicyOfficerChattanoogaTN
MelissaBridgesPerformanceandInnovationCoordinatorLittleRockAR
VictoriaCarreoacutenAdministrativeOfficerLasVegasNV
TyroneGrandstrandCommunityDevelopmentPlannerFargoND
TibLaughlinDirectorofGeneralServicesKansasCityKS
TimMorelandDirectorofPerformanceManagementandOpenDataChattanoogaTN
CharlesRosendahlManagementAnalystDentonTX
Thankyou
60
MelissaSiebenAssistantCountyAdministratorKansasCityKS
GinaThompsonDirectorofStrategicServicesLewisvilleTX
JHUFacultycollaboration
GovExalsoconvenedthecitycontactsandfacultypartnersinaseriesofAdvisoryGroupmeetingsThesemeetingsservedtoconnectcitiesandfacultytounderstandoneanothersperspective
StefanBaralBloombergSchoolofPublicHealth
AshleyBernerJHUSchoolofEducation
MelissaDavey-RothwellBloombergSchoolofPublicHealth
JonLinksBloombergSchoolofPublicHealth
TamarMendelsonBloombergSchoolofPublicHealth
KristinMmariBloombergSchoolofPublicHealth
KeshiaPollackPorterBloombergSchoolofPublicHealth
JoshSharfsteinBloombergSchoolofPublicHealth
LizStuartBloombergSchoolofPublicHealth
LindsayThompsonCareyBusinessSchool
DanielWebsterBloombergSchoolofPublicHealth
Partnersindataandevidence-basedpolicy
Wealsospokewithpartnerswhoareexpertsinthefieldofevidence-basedpolicywhosharedtheirknowledgeandresourcesofcurrentandfuturethinkinginevidence-informedthinking
WendyBarrenoAssociateDirectorWilsonSheehanLabforEconomicOpportunitiesUniversityofNotreDame
TiffanyDavenportDirectorofDataPartnershipsBaltimoresPromise
TomiHiersDirectorBaltimoreCivicSiteAnnieECaseyFoundation
ChrisKingsleySeniorAssociateAnnieECaseyFoundation
KathrynOliverAssociateProfessorLondonSchoolofHygieneandTropicalMedicine
Thankyou
61
KristenWatkinsSeniorPolicyAssociateJ-PAL
GovExParticipants
GovExAnalystMiriamMcKinneyanalyzedtheinterviewtranscriptsandconductedtextanalysisSeniorImplementationAdvisorEmilyShawwillintegrateRoadmapactivitieswithadditionalservicedeliveryactivitiesatGovEx
Thankyou
62
ReferencesAbrahamKGampHaskinsR(2017)Thepromiseofevidence-basedpolicymakingReportofthecommissiononevidence-basedpolicymakingWashingtonDCCommissiononEvidence-basedPolicymakingBigmove2Extendingthegreen(nd)Retrievedfromhttpswwwcityoflewisvillecomdoing-businesslewisville-2025big-move-2-extending-the-greenBatesMAampGlennersterR(2017)ThegeneralizabilitypuzzleRetrievedfromhttpsssirorgarticlesentrythe_generalizability_puzzleBuildwithnotfor(2018January)RetrievedfromhttpsbuildwithnotforwordpresscomCairneyPampOliverK(2017)Evidence-basedpolicymakingisnotlikeevidence-basedmedicinesohowfarshouldyougotobridgethedividebetweenevidenceandpolicyHealthResearchPolicyandSystemsdoi101186s12961-017-0192-xCairneyPOliverKampWellsteadA(2016)TobridgethedividebetweenevidenceandpolicyReduceambiguityasmuchasuncertaintyPublicAdministrationReview76(3)399-402Retrievedfromhttpdxdoiorg101111puar12555Centreforpublicimpact(nd)RetrievedfromhttpswwwcentreforpublicimpactorgCernaL(2013)ThenatureofpolicychangeandimplementationAreviewofdifferenttheoreticalapproachesOrganisationforEconomicCo-OperationandDevelopmentRetrievedfromhttpswwwoecdorgeduceriTheNatureofPolicyChangeandImplementationpdfColoradoSprings(nd)Retrievedfromhttpscoloradospringsgovcommunicationspagewhat-expect-when-you-speakupContinuumofcare(coc)program(nd)RetrievedfromhttpswwwhudexchangeinfoprogramscocCouncilpresidentkicksoff$12millionyouthfund(nd)Retrievedfromhttpwwwbaltimorecitycouncilcomcontentcouncil-president-kicks-12-million-youth-fundDurlakJAampDuPreEP(2008)ImplementationmattersAreviewofresearchontheinfluenceofimplementationonprogramoutcomesandthefactorsaffectingimplementationAmericanJournalofCommunityPsychology41327-350Retrievedfromhttpswwwncbinlmnihgovpubmed18322790Fishbonediagram(ishikawa)(nd)Retrievedfromhttpasqorglearn-about-qualitycause-analysis-toolsoverviewfishbonehtmlFivewhysGettingtotherootofaproblemquickly(nd)RetrievedfromhttpswwwmindtoolscompagesarticlenewTMC_5Whtm
References
63
FligsteinNampMcAdamD(2011)TowardageneraltheoryofstrategicactionfieldsSociologicalTheory29(1)1-26RetrievedfromhttpsociologyberkeleyedusitesdefaultfilesfacultyfligsteinFligstein20McAdam20Soc20Theory20PaperpdHeinkelP(2018January12)CitymullspermitsforbikesharingcompaniesRetrievedfromhttpwwwdentonrccomnewsdenton20180112city-mulls-permits-bike-sharing-companiesHowisareamedianincomecalculated(2012October)Retrievedfromhttpswwwhudexchangeinfofaqs843how-is-area-median-income-calculatedLampkinLMampHatryHP(2003)KeystepsinoutcomemanagementUrbanInstituteRetrievedfromhttpswwwurbanorgresearchpublicationkey-steps-outcome-managementMarshDampSmithM(2000)UnderstandingpolicynetworksTowardsadialecticalapproachPoliticalStudies484-21Retrievedfromhttpjournalssagepubcomdoiabs1011111467-924800247McCannL(nd)RetrievedfromhttplaurenellencomMoultonSampSandfortJR(2016)ThestrategicactionfieldframeworkforpolicyimplementationresearchPolicyStudiesJournal45(1)Retrievedfromhttponlinelibrarywileycomstore101111psj12147assetpsj12147pdfv=1ampt=jdkhhcp4amps=e184c625e1855a6b1615ccb99ee27b1c0b3212d8PaynterB(2018January1)What100citiesarelearningfromeachotherbysharingtheirdataRetrievedfromhttpswwwfastcompanycom40518004what-100-cities-are-learning-from-each-other-by-sharing-their-dataPeakacademy(nd)Retrievedfromhttpswwwdenvergovorgcontentdenvergovenmayors-officeprograms-initiativespeak-performancepeak-academyhtmlPinnockHBarwickMCarpenterCREldridgeSGrandesGGriffithsCJRycroft-MaloneJMeissnerPMurrayEPatelASheikhAampTaylorSJC(2017)Standardsforreportingimplementationstudies(stari)ExplanationandelaborationdocumentBMJOpenRetrievedfromhttpbmjopenbmjcomcontentbmjopen74e013318fullpdfProctorESilmereHRaghavanRHovmandPAaronsGBungerAGriffeyRampHensleyM(2011)OutcomesforimplementationresearchConceptualdistinctionsmeasurementchallengesandresearchagendaAdministrationandPolicyinMentalHealth38(2)65-76doi101007s10488-010-0319-7TandonSD(2018)ProgressincommunityhealthpartnershipsResearcheducationandactionRetrievedfromhttpswwwpressjhuedujournalsprogress-community-health-partnerships-research-education-and-actionProvanKGampKenisP(2007)ModesofnetworkgovernanceStructuremanagementandeffectivenessJournalofPublicAdministrationResearchandTheory
References
64
18(2)229-252Retrievedfromhttpsacademicoupcomjpartarticle182229935895SabatierPA(1988)Anadvocacycoalitionframeworkofpolicychangeandtheroleofpolicy-orientedlearningthereinPolicySciences21(3)129-168Retrievedfromhttpwwwjstororgstable4532139SandstromAampCarlssonL(2008)TheperformanceofpolicynetworksTherelationbetweennetworkstructureandnetworkperformancePolicyStudiesJournal36497-524Retrievedfromhttponlinelibrarywileycomdoi101111j1541-0072200800281xabstractSchwartzMampWilsonE(nd)WhocanaffordtoliveinahomeAlookatdatafromthe2006americancommunitysurveyRetrievedfromhttpswwwcensusgovhousingcensuspublicationswho-can-affordpdfShearerJCAbelsonAKouyateBLavisJNampWaltG(2016)WhydopolicieschangeInstitutionsinterestsideasandnetworksinthreecasesofpolicyreformHealthPolicyandPlanning31(9)1200-1211Retrievedfromhttpsdoiorg101093heapolczw052ThelessonsofadministrativedataHigh-profilepolicy-relevantresearchpoweredbyadministrativedata(nd)Retrievedfromhttpswwwpovertyactionlaborgsitesdefaultfilesresources20161114-The-Lessons-of-Administrative-DatapdfToolsandotherneedsforqualitativeandquantitativemethods(2015February)AgencyforHealthcareResearchandQualityRetrievedfromhttpwwwahrqgovprofessionalsquality-patient-safetytalkingqualityassesstoolshtml
WeaverRCampBagchi-SenS(2013)SpatialanalysisofurbandeclineThegeographyofblightAppliedGeography4061-70
References
65
UniversityPartners
Name Title Affiliation Area(s)ofexpertise
Contactinformation
Examplesofworkingwithnon-academicdecisionmakers
WendyPBarreno
AssociateDirector
WilsonSheehanLabforEconomicOpportunitiesUniversityofNotreDame
LEOsfocusareasincludehealtheducationhousingchildwelfareeconomicopportunitycriminaljustice
wbarrenonduduorleondedu
MelissaDavey-RothwellPhD
AssociateScientist
JohnsHopkinsBloombergSchoolofPublicHealth
Health mdavey1jhuedu410-614-5854
BaltimoreCityHealthDepartmentLeadersMarylandDepartmentofHealthAddictionRecoveryServicesinDCExecutiveDirectorsofcommunityorganizationsacrosstheUSthatfocusonHIVSTIsubstanceabuseandimplementationscience
ElizabethStuart
AssociateDeanforEducationandProfessorDepartmentsofMentalHealthBiostatisticsandHealthPolicyand
JohnsHopkinsBloombergSchoolofPublicHealth
Educationsocialpolicystatisticalmethods
estuartjhuedu
Resources
66
Management
Design-ThinkingResources
EngagementGuideforCivicTech
Journeymap
PerformanceManagementResources
DenverPeakAcademy
Proxymeasuresguide
Talkingaboutperformancewithoutcomplicatingit(video)
SettingPerformanceTargetsguide
Datavisualizationguide
Performancemanagementgettingstartedguide
Exercises
Fishboneexercise
5Whys
AdministrativeData
TheLessonsofAdministrativeData(JPAL)
EvaluationResources
Quantitativeandqualitativemethods(USDepartmentofHealthandHumanServicesAgencyforHealthcareResearchandQuality)
KeyStepsinOutcomeManagement(UrbanInstitute)
RCT-YES
Sageresearchmethods
BestPracticesforMixedMethodsResearchintheHealthSciences
ACCORDChecklist
Pilotandimplementationplanning
Benchmarkingexamplesandproxymeasures
Seattleopendataplaybook
Resources
67
Datasciencescopingguide
Resources
68
AboutGovExGovExhelpsgovernmentsusedataandevidencetoimproveresidentsrsquolivesWesupportgovernmentsinbuildingtheirinternalcapacitytooperatesustainableanalyticspracticeswhichimprovecommunityoutcomesinfieldsfrompublicsafetytoeconomicdevelopmentGovExhasaproventrackrecordofworkinginconcertwithpartnerstoprovidetechnicalassistanceandtrainingatscale
SinceourlaunchinApril2015GovExhas
Trainedover2000governmentemployeesinpersonandonlineintopicsincludingdatamanagementperformancemanagementopendatacommunityengagementandanalytics
Providedtechnicalassistancetomorethan100localgovernmentsacrosstheglobeimprovingtheiruseofdatatoachieveefficienciesinservicedeliveryandbetteroutcomesforresidents
Assessedthecapacityofmorethan130governmentalorganizationstodeterminehowleadersstaffandexternalstakeholdersmanageandusedata
Localgovernmentsareincreasinglyonthefrontlinesofaddressinglargescaleproblemswithglobalimplicationssuchasclimatechangepublicsafetyandeconomicinequality
Whileevidence-basedinterventionstohelpsolvethoseproblemsareavailablelocalgovernmentstaffhavelimitedtimeandcapacitytofindandimplementresearch-basedsolutionsAsaresultcitiescontinuetoworkthewaytheyalwayshave--makingincrementalimprovementstoprogramsbutnotalwayscreatingchangeatasystem-level
ThroughresourcedevelopmentandcollaborationwithJHUfacultyGovExissupportingcityleadersinusingdataandevidencetomakedecisionsandcreatinglastingrelationshipswithresearchers
TheRoadmapwasdevelopedbyGovExaspartofitspartnershipwithBloombergPhilanthropiesWhatWorksCitiesinitiative
AboutGovEx
69
- Title page
- Introduction and purpose of the Roadmap
- Using the Roadmap
-
- Gather information about the problem
- Strategically engage key partners
- Review the evidence and collect details
- Decide on a policy to implement based on local factors
- Establish a framework to monitor progress
- Assess data accessibility quality and security
- Design implementation and evaluation
- Share your story and success
-
- Working with a range of evidence
- Thank you
- References
- Resources
- About GovEx
-
HousingFirstwhichvaluesconnectingpeoplewithpermanenthousingovertransitionalhousingandsupportiveserviceafterspeakingwithrepresentativesattheUSDepartmentofHousingandUrbanDevelopment(HUD)
ThisexaminationofwhattheCityiscurrentlyfundingandthesourcesofthatfundingwasaturningpointinChattanoogaUnderstandingthisproblemfromasystemsperspective-includingpoliciesandfundingstreamsatmultiplelevelsofgovernmentsocialnetworksthroughwhichinformationisoftenspreadtechnologythebuiltenvironmentofmarket-rateandpublichousing-helpedthenewMayoranalyzeexistingeffortsandleadstaffandotheractorsinthesamedirection
Begintogatherthedata
TotrulycapturethesuiteofservicesthatyourCityisprovidingaskdepartmentstafftolisttheprogramsrelatedtoyourproblemstatementthatyourcityrunsorfundsWhenlistingtheprogramscapturedetailssuchas
programnamethenameofthedatasetthatisrelevanttoeachprogramadescriptionofthedatasetthesourceofthedatasetifthedataisinternalorexternalwhetherthisdataisalreadybeingcollected
ThisinformationwillformthebasisofyourinventoryofadministrativedataordatasetsthatarekeptbygovernmentsforoperationalpurposesThismayincludecallcenterrecords(311911)taxdatapropertyassessmentsinspectioninformationsocialservicerecordsetcAtthispointlistallrelevantadministrativedatasetsYouwillprioritizewhichdatasetstogathermoreinformationonlaterwhenyouaredevelopingindicators
Asampletemplatethatcanbeusedtocapturethisinformationisbelow
ProblemStatement____________________________________________
Gatherinformationabouttheproblem
12
ProgramName
DatasetTitle
DatasetDescription DataSource
Departmentorexternalpartner
DoestheCityalreadycollectthisdata
Whatisthenameoftheprogram
Human-readablenameofthedataset
Shouldbeunderstandablebynon-technicalusers
Whatistheoriginalsystemapplicationfilethathousesit
Departmentorexternalpartnerthatmaintainsthedata
Isthisdataalreadycollected
TogatherthisinformationconsiderassigningonepointpersontosurveyorinterviewcitystaffSpeakingwithfrontlineworkersstaffwhoworkwithfederalgrantsdepartmentheadsandeveryoneinbetweenwillensureadeepunderstandingoftherelevantprogramsanddatasetsDuringthecourseoftheseconversationsbesuretoaskcitystaffwhichorganizationstheyworkwithexternallyandcapturethatinformationaswelleveniftheyarenotrecipientsofcityfundingInadditiontoconversationswithexpertsincitygovernmentyourcitysbudgetdocumentsareanothergreatsourceofinformation
Collectdataandobservetrends
Nexttrytogainanunderstandingofwhytheproblemstillexistsifithasbeensomethingyouhaveworkedonforawhileorwhyitisemergingnowifitisnew
AnalyzingadministrativedatacandemonstratepasteffectivenessofinitiativesrelatedtoyourproblemstatementUsingthedatasetinventorythatyoubeganintheprevioussteptalktodepartmentstaffaboutwhichdatasetsaremostrelevanttotheproblemandgatherafewyearsworthofhistoricaldataBegintolookfortrendsinthisdatadiscussingpotentialreasonsforthosetrendswithrelevantstaffCanthetrendsbeexplainedbyseasonalitychangesinfundingstaffingfluctuationsorsomeotherreasonBasedonthehistoricaldataanalysiscreateabaselineagainstwhichfuturechangescanbeobserved
OtherdatasetscanhelpprovideadditionalcontextForinstanceifthenumberofpeopleexperiencinghomelessnessinyourcityisincreasingitmaybehelpfultotakealookathousingaffordabilityIfyoudonothavethedatayouneedconsideraproxymeasureAsanexamplehousingisoftenconsideredaffordableifitcosts30orlessofahouseholdsincomeButifhouseholdincomedataisnotavailableatthecitylevelAreaMedianIncome(AMI)canbeahelpfulproxy
Gatherinformationabouttheproblem
13
QualitativedatacanalsohelpanswerquestionsaroundtheeffectivenessofpreviouseffortsTrytoobtainavarietyofperspectives--bothmainstreamandoutliers--fromnetworksbeyondyourowntoprovideyouwiththegreatestvantagepointforevaluatingpotentialimpactUnderstandinghowthoseeffortswerereceivedbydifferentpartsofyourcommunityandiftheremaybeelementsofprevioussystemstopreservewillbecrucialtothesuccessofanypolicychangeThisinformationmaybeobtainedthroughfocusgroupsinterviewsandsurveysFormoreinformationonhowtousethesequalitativeresearchtoolspleaseseetheseresourcesdevelopedbytheAgencyforHealthCareandResearchandUrbanInstitute
FinallytakealookatnationaldatatoseewhereelsethisproblemmightexistThisresearchmaybehelpfulinlaterstepswhenyouarelookingforpromisingpoliciesinothercities
Puttingitalltogether
UsethetemplatebelowdevelopedbyDenversPeakAcademytocaptureyourrefinedrightsizedproblemstatementandcurrentandfuturestatesWorkingthroughthisexercisewithpartnersishelpfultogainbuy-inTheresponseswillbeimportantwhengettingotherpartnersonboard
WhyChangeIsNeeded
Yourright-sizedproblem
CurrentState
Analysisoftrendsindata
FutureState
Ideallywhataretheoutcomesyouhopetoachieve
Checklist
Thekeystepstogatheringinformationarebelow
Atthispointfocusontheproblemratherthanjumpingtosolutions
Listentopeoplewhoareaskingforhelpandtoadvocateswhoareraisingissues
Beasspecificaspossibleinidentifyingtherootcauseandhoninginonthepartoftheproblemthatyourcitycanaddress
Identifytrendsbasedonmultiplesourcesofdataincludinghistoricaladministrativedataproxymeasuresqualitativedataandnationalmeasures
Gatherinformationabouttheproblem
14
Captureyourproblemstatementtheresultsofyouranalysisanddesiredoutcomes
Gatherinformationabouttheproblem
15
StrategicallyengagekeypartnersPolicytheoryresearchoftenfocusesonincreasingpolicymakersaccesstoevidenceasawaytoavoidtakingshortcutslikemakingdecisionsbasedonemotionImplementationsciencesuggeststhatincreasingaccesstoevidenceisnotenoughandthatpolicychangeshouldbeexaminedinthewidercontextofdebatecoalitionformingandpersuasionMeaningfulpolicychangerequiresldquoalong-termstrategytoformalliancesanddevelopknowledgeofpolicymakersandpolicymakingratherthanusingevidenceinpolicymakingonanadhocorone-offbasis
UnderstandingnetworksandsystemsisimportanttodevelopsustainablepolicychangeThissectionismeanttohelpreaderstakethestepstobuildengagementstrategiesforpartnersthatrepresentdifferentsystemsandnetworksEngagementshouldbeanongoingprocessofconnectingpeopletooneanotherandtoanissueEngagementshouldnotbeinnameonly-wehaveallexperiencedengagementactivitiesthatgivepeopleaspecificroletoplaythatleadstoaneatpredictableoutcomeAccordingtotheBuildWithmovementundervaluingandunderinvestinginengagementstrategiescanleadtoadisconnectofcertainindividualsorgroupsandmistrustofgovernmentWhendevelopingengagementactivitiesrememberthathowpeopleengageandwhethertheyengagehastodowithhowyoustructureopportunitiesforengagement
Conveneacoreadvisorygroup
AnadvisorygroupcomprisedofpeoplefrommultiplesectorscanbehelpfulasyouconsiderthelocalsocialenvironmentalandpoliticalcontextinwhichyouintendtointroducepolicychangeRevisityourstakeholdermapanddeterminewhowouldbebesttoserveonthisgroupTomakesureyouareconsideringtheproblemsandsolutionsthroughanequitylensincludepeoplewithlivedexperiencewhohaveexpertiseintheeffectivenessofsystemswhetherthosesystemsfunctionasintendedandhowtheymayimpacthumanbehaviorYoumayalsowishtoincludesomeonewithexperienceworkingwithdataandanalysis-thispersoncanhelpdeterminethetypeofdatathatmaybereadilyavailableandusefultocollectandcanalsohelpplanandshapeyouranalysistofigureoutwhichprogramsareeffectiveBesuretoinvitefront-lineworkersaswell-itscriticalthatwhateversolutionscomeoutofthisprocessaremeaningfultothosewhowillberesponsibleforimplementingthem
BringingpeopletogethercanbechallengingbutthereareafewwaystosetandmanageexpectationsFirstmakesurethatthegroupisawareoftheproblemstatementcurrentstateandfuturestate-thiscanhelpstarteveryoneoffonthesamepageNextspend
Strategicallyengagekeypartners
16
sometimegettingtoknoweachotherrsquosworktolearnabouteveryonesstrengthsThissharedunderstandingcanhelpasyouclarifyrolesandresponsibilitiesofeachpartnerinthegroup--thisisimportanttoavoidfinger-pointinginthefutureandtoensurethatnoneofthetasksslipthroughthecracksInsomecasestherearemanyactorswithdifferentresponsibilitiesasinthecaseofBaltimoreCitys38-memberYouthFundTaskForcewhichischairedbynonprofitandcorporatepartnersandincludesrepresentativesfromthenonprofitandfaithcommunitiesstudentsthebusinesssectorlocaluniversitiesCityCouncilandtheMayorsOffice
OncerolesandresponsibilitiesareclearyoumayconsiderdevelopingacharterguidelinesortermsofreferencetosetexpectationsforthegroupBesuretodistinguishthisfromotherboardsandcommissionsthatyourcitymayhaveNextyoumaywishtoestablisharegularongoingmeetingscheduleBesuretosendagendasandmeetingmaterialsaheadoftimeandclarifytheobjectiveofeachmeeting
InSeattletheCitydevelopeditsHousingAffordabilityandLivabilityAgenda(HALA)torespondtorisinghousingcostsandhomelessnessThisisatrendinotherwestcoastcitiesaswellwhichmaybeexplainedbyaninfluxofnewresidentswhoareattractedbytheemergenceoftechjobstheattractionoftheoutdoorlifestyleortherecentpassageofprogressivepolicieslikelegalizedgaymarriagea$15minimumwageandlegalizationofmarijuanaSeattlebroughttogetheramulti-stakeholderadvisorygroupcomprisedoflocalhousingexpertsrentersandhomeownersdevelopersandgovernmentofficialstoshareinformationabouttheirneedsanddeveloprecommendationsbasedontheseneedsTheinterestsandmotivationsofthisgroupwerevariedandittookabouttenmonthstofindcommongroundTheseriesofrecommendationsthattheadvisorygroupcreatedismeanttoserveresidentswithaspectrumofneeds
Buildaprocessforcommunityparticipation
Ofcoursenoteveryonecanbeincludedinanadvisorygroupandadvisorygroupmembersmaynotnecessarilyrepresentthevariedperspectivesoftheirpeers-forinstancenoteveryhomeownerhasthesamesetofneedsTogatherinputfromawiderswathofthepopulationitcanbehelpfultobuildaprocesstoinvitethepublicatlargetosharetheirideasCitiesdothisinavarietyofwaysfromonlineconversationsandsurveystoin-personmeetingsRememberthatbeingintentionalaboutgatheringtheviewpointsofpeopleismoreimportantthanusingthelatesttechnologyandtools
TheinsightgeneratedbycommunityparticipationexercisescanbeincludedinyourreviewofevidencewhichisdiscussedinasubsequentchapterInWestVirginiaastatethathasbeenhithardbytheopioidepidemicstatehealthofficialsdraftedanopioidresponseplanthatwasinformedbypublicengagementaswellaslocalandregionalsubjectmatter
Strategicallyengagekeypartners
17
expertsThepublicwasinvitedtopresentcommentsonideasfordecreasingoverdoseearlyinterventiontreatmentandrecoveryPublichealthandmedicalfacultyfromWestVirginiaUniversityMarshallUniversityandJohnsHopkinsUniversityprovidedresponsestothecommentsTheresultwasanOpioidResponsePlanthathealthcommissionerRahulGuptapresentedtoGovernorJimJustice
AsamplespectrumtoplanyourowncommunityparticipationprocesscreatedbyLaurenellenMcCannisbelow
SourceLaurenellenMcCann
Community-basedparticipatoryresearchisanotherwaytoworkwiththecommunitytoimproveoutcomesCBPRmeansthecommunityisinvolvedinallaspectsoftheresearchprocesswhichisofteniterativeInCBPRtraditionalresearchersandcommunitymembersshareknowledgeaswellasresourcesandcredit
CBPRInActionInSanFranciscoanenvironmentaljusticenonprofitteamedupwiththeCitysDepartmentofPublicHealthandanoutsideevaluatortounderstandtherootcausesofobesityandfoodinsecurityinlow-incomeneighborhoodsandidentifysolutionsThroughresearchmethodssuchasinterviewingresidentsandstoreownersdiagrammingstoreshelvesGISmappinganeconomicanalysisandreviewofsimilarpoliciesinnearbycitiestheteamobservedthatthestoresthatresidentshasveaccesstowerestockedwithlesshealthyfoodsandthatresidentswantedmorehealthyfoodandlessalcoholandtobaccoThisprojectresultedinlocalandstatelegislationtoencouragestorestostockfreshproduce
Strategicallyengagekeypartners
18
Developrelationshipswithsubjectmatterexperts
OnewaytokeepsubjectmatterexpertsengagedistoinvolvetheminthecommunityparticipationprocessasdescribedaboveIntheWestVirginiaexamplesubjectmatterexpertsrespondedtoandsynthesizedthepubliccommentsOtheropportunitiesaredescribedbelow
AdvisoryAcitydepartmentheadMayororstaffmembermaydevelopaone-on-oneadvisoryrelationshipwithuniversityfacultymemberoracollaborativebetweentheCityandUniversitytoaddressaparticularproblemAgreatplacetostartistolookforrelevantpracticecenterswithinyourlocaluniversityandreachingouttosubjectmatterexpertsInadditiontotheirexpertiseontherelevantpolicytopicheorshemayalsobeabletoidentifyotherexpertsandevenprovidesupportonhiringtherighttalenttoleadcertainprogramsAnadvisoryrelationshipwithauniversitysubjectmatterexpertcanalsohelpwhencitiesareconsideringbringinginnewprogramsTheseexpertsareoftenfamiliarwithprogramsthatarebasedonnationalmodelsandcanprovideinsightintoaprogramsoriginandlineageeffectivenessinotherplacesandthepotentialforimplementationinyourcityAcademicscanalsohelpidentifyindicatorsbasedongovernmentindicatorsorliteratureFinallyrelationshipsbetweenpolicymakersandresearchersprovidethemostbroadlyeffectiveroutetopullingresearchintopolicy
ResearchandevaluationAcademicexpertsmaysupportcitiesinevaluation-indeedevaluatingcityinitiativesisatraditionalmodelofcityuniversityrelationshipsSharingtheinformationyoucollectedintheGatherInformationphasecanhelpresearchersunderstandyourworkandgoalswhichinturnwillstrengthenthesupportthattheycanprovideDiscussingdetailsaboutyourinterventionandevaluationplanatthebeginningoftheprocessmeansthataresearchpartnercanhelpwiththeevaluationdesignanddatacollectionForinstancetheWilsonSheehanLabforEconomicOpportunitieswalkscitiesthroughconsideringthefollowingquestionsbeforepartneringonanevaluation
1 Isthereaclearpolicyprogrammodelwithobjectivescreeningcriteria
2 Isitpossibletoevaluatewhathappensbothwiththepolicyprogramandintheabsenceofthepolicyprogram
3 Arethereoneormoremeasurableoutcomesofinterest
4 Arethereenoughpeopleforrigorousquantitativeanalysis
5 Whatdatawillbenecessaryforthestudyandwhataretheprocessesforaccessingthesedatasource(s)
6 Isthereacompellingreasontoevaluate
Strategicallyengagekeypartners
19
InadditiontoassessingreadinesstoconductanevaluationresearcherscanhelpyouhoneinonthepurposeoftheevaluationTheWorldBankdescribesseveralpotentialusesofevaluationwhicharedefinedhereInbriefevaluationcanhelpdecisionmakers
makeresourceallocationdecisionsrethinkrootcausesofproblemsidentifynewproblemsdecideonabestalternativedeterminewhetherprogramsorpoliciesarebeingimplementedwithfidelitydeterminewhetherprogramsorpoliciesarehavingthedesiredimpact
Researcherscanalsoprovidesupportinsharingtheresultswiththescientificcommunitythroughpapersandconferenceswhichisvaluableincreatingapeernetworkofevidence-basedcitypractitioners
TrainingandtechnicalassistanceUniversitypartnersespeciallythoseatpracticecenterscanbehelpfulindevelopingorupdatingtrainingcurriculumtrainingthetraineranddeliveringtrainingtocitystaff
Strategicallyengagekeypartners
20
ReviewtheevidenceandcollectdetailsOnceyouhaveathoroughunderstandingoftheproblemdataandactorsyouwillneedtoconductyourownresearchonavailableevidenceandlearnaboutthelandscapeofpolicyoptionsYourgoalwillbetodevelopanaccessiblelistofinformationaboutpoliciesthathavehelpedsolvesimilarproblemsinotherplacesThischapterwillguideyouthroughthestepsofpreparingyourreviewersunderstandinglevelsofevidenceandcollectingthedetailsonevidencethatyouwillneedfordiscussionwithdecisionmakersinyourcity
Selectandprepareyourreviewers
AssignapersonorteamtoreviewrelevantinterventionsthathavedemonstratedsuccessinotherjurisdictionsTheirgoalwillbetoproduceareviewofalloftherelevantprogramspoliciesorinitiativesthatyourteamcouldconsiderimplementingTheywillbeconductingaliteraturereviewsupplementedbyinterviewswithexpertsandpeerswhohaveinformationaboutpotentialprograms
SincethepeopleyouassigntothistaskareprovidingthebaselineofpotentialoptionstoimplementitisimportantthatyouchoosepeopleyouareconfidentwillbethoughtfulandthoroughintheirworkItisalsocriticalthattheyareawareoftheimportanceofsourcecredibilityEnsureyourreviewershavefamiliarizedthemselveswithrulesforcriticallyanalyzinginformationsourcesbeforetheygetstartedIftheyareincludingnewsarticlesaspartoftheirsearchtheyshouldalsobeawareofmethodsforevaluatingnewscredibility
AskyourreviewerstodevelopaplanandtimelinetocompletethereviewThisstartswithsolidifyingagoverningresearchquestionOnewaytoknowyouarecreatingadetailedresearchquestionistousetheacronymPICOremindingyoutoincludereferencetothepopulationintervention(ifyouhaveoneinmind)comparisonandoutcomeelementsinyourresearchquestionForexampleifyouknowthatyouhaveapopulationofteenagerswhoarehomelessyouhavelearnedfromyourstakeholdersthatmanyoftheseteenagersidentifyasLGBTandyouhaveheardsomethingabouttheapproachofcreatingdrop-incentersfordeliveringsupportyoucouldask
ForLGBTyouthfacinghomelessness(population)howeffectivearedrop-incenters(intervention)relativetostandardsocialservices(comparison)atsettlingtheminpermanenthousing(outcome)
Ifyoudonothaveaninterventioninmindyoucanleaveouttheinterventionandcomparisonandsimplyask
Reviewtheevidenceandcollectdetails
21
HowcanwehelpLGBTyouthfacinghomelessness(population)obtainpermanenthousing(outcome)
OnceyourreviewershavetheirresearchquestionsusethequestionstocreatealistofkeywordstheycanuseintheirliteraturesearchThiswillhelpkeeptheircollectionofarticlesreasonablyfocusedHavethemdescribewhattheirsearchprocesswillbeincludinglistingthedatabasestheywillreviewandtheirplanforstoringtheinformationtheywillcollect(Herearesomeadditionalsuggestionsforstructuringareviewplan)
WhatyourreviewerswillbedoingisalightweightversionofarapidreviewTotheextentthattheywouldliketodevelopamethodologyforspecificsearchprotocolsincludingcriticalappraisalprocessestheycancertainlygoaheadanddoso
Usedifferentlevelsofevidenceforthemostcompletepicture
AsdiscussedintheWorkingwitharangeofevidencechaptertheNestaStandardsofEvidenceframework--reproducedbelow--ishelpfultounderstandthelevelofconfidencethataprogramorinterventionishavingapositiveimpactThisapproachletsyouputinterventionsthatyouhaveheardaboutlocallybutwhichhavenotbeenstudiedtogetherwiththosewithademonstratedcausaleffectThisishelpfulsinceanumberofpoliciesprogramsandpracticesthatyouwillwanttoconsidermaynotyethavemuchofaresearchrecord
Reviewtheevidenceandcollectdetails
22
NearlyanypieceofinformationaboutaninterventioncanbeclassifiedwithinthestandardsofevidenceForexampleifsomeonehasacrediblehypothesisabouthowaninterventioncouldproducebenefitthistypeofevidencewouldbeclassifiedasLevel1ThistypeofevidencecouldincludeinterventionsinothercitiesthatyoureadaboutinthenewsorhearaboutataconferenceForinstancewhenUSDAallowedschoolsandschooldistrictstoprovidefreebreakfastandlunchtoallstudentsthroughtheCommunityEligibilityProvision(CEP)manydistrictsimplementedthisoptionevenbeforetherewasevidenceofeffectivenessThiswouldfitintoLevel1sinceitbuildsonthelogicthatprovidingaccesstomealswilldecreasefoodinsecurity
YoumayalsofindpiecesofinformationthatqualifyasLevel2evidenceForinstanceasmoreschooldistrictsimplementedCEPprincipalstestifiedaboutpositivechangestheyobservedinstudentssuchasamorepositivedemeanorthatcanleadtobetteracademicperformanceDataalsoshowedthatthestudentsdescribedwerereceivingmoremealsthanbeforetheprogramsimplementationHoweveritwastoosoontobuildevidencetoshowacausalrelationshipbetweentheincreaseinfreemealsandincreasedacademicperformance
ldquoLower-levelrdquoevidencecanconsistofnuancedbutvaluableelementsthatrelatetoaninterventionrsquospotentialfitwithacommunityQualitativeinformationsuchastestimonyfrompeoplewhohaveexperiencedaninterventionandthosewhohavefacilitatedorimplementeditwillalwaysbeusefultoconsiderandprovidesdetailedlocalcontext
Experimentalstudies--describedstartingatLevel3withcontrolledexperimentsandgoingthroughLevel5withfullyfleshedoutprotocolsandprocessesforreplication--stilloccupythehigherlevelsofthisdescriptionofevidenceThisprioritizationshowsthevalueofrigorousstudyforevaluatingtheimpactofaparticularpolicyorprogramratherthanotherfactorsthatcouldbeinfluencingoutcomesofinterestRememberthatmultipletypesofevidencecancomplementrigorousimpactevaluationsandgivecityleadersadditionalinsightsandbroaderperspective
Accessinformation
Encourageyourreviewerstobegintheirsearchforpromisingpracticesingovernmentandnon-profitclearinghousesandinacademicpolicyjournalsWorkpublishedinthesesourceshasalreadybeensubjectedtoahighlevelofprofessionalscrutinyandgenerallycontainsagreatdealofdetailwhichmeansthatyourreviewersaremorelikelytobeabletopullouttheinformationyouneedtomakeagooddecision
Clearinghouses
Reviewtheevidenceandcollectdetails
23
Herearesomeoftheclearinghouses(collectionsofprogramevaluations)forevidence-basedpoliciesandprograms
ResultsFirstClearinghouseDatabaseThePewCharitableTrustsCollectstogethereightotherclearinghousestoprovideinformationoninterventionsinthefollowingcategoriesadultcriminaljusticeeducationchildwelfarehealthjuvenilejusticementalhealthsubstanceabuseandyouthdevelopmentIndividualclearinghousesincludedinResultsFirstare
BlueprintsforHealthyYouthDevelopment-foryouthdevelopmentandviolenceprevention
CaliforniaEvidence-BasedClearinghouse-forchildwelfareinterventions
TheLauraandJohnArnoldFoundationrsquosSocialProgramsThatWork-forsocialservicesprograms
TheUSDepartmentofEducationrsquosWhatWorksClearinghouse-foreducationinterventions
TheUSDepartmentofHealthandHumanServicesrsquoResearch-TestedInterventionPrograms(RTIPs)-forcancercontrolinterventions
TheUSDepartmentofHealthandHumanServicesrsquoTeenPregnancyPreventionEvidenceReview(TPPEvidenceReview)-forsexualriskbehaviorandsexualhealthoutcomes
TheUSDepartmentofJusticersquosCrimeSolutionsgov-forcriminaljusticejuvenilejusticeandcrimevictimservices
TheUSSubstanceAbuseandMentalHealthServicesAdministrationrsquosNationalRegistryofEvidence-basedProgramsandPractices(NREPP)-substanceuseandmentalhealthinterventions
TheUniversityofWisconsinPopulationHealthInstituteandRobertWoodJohnsonFoundationrsquosCountyHealthRankingsandRoadmapsWhatWorksforHealth(WhatWorksforHealth)-healthbehaviorsclinicalcaresocialandeconomicfactorsandthephysicalenvironment
UKWhatWorksCentresTheBritishgovernmenthascharteredanetworkofsubject-specificorganizationswhichcollectsynthesizeandreportoutevidenceontheeffectivenessofpolicyprogramsandpractices
NationalInstituteforHealthandCareExcellence(NICE)-healthandsocialcare
Reviewtheevidenceandcollectdetails
24
SuttonTrustEducationalEndowmentFoundation-educationalachievement
CollegeofPolicingWhatWorksCentreforCrimeReduction-crimereduction
EarlyInterventionFoundation-earlyintervention
WhatWorksCentreforLocalEconomicGrowth-localeconomicgrowth
CentreforAgeingBetter-qualityoflifeforolderpeople
WhatWorksCentreforWellbeing-wellbeing
WilsonSheehanLabforEconomicOpportunities(LEO)UniversityofNotreDameProvideslinkstotheirevaluationsofinterventionsincriminaljusticeself-sufficiencyeducationhealthandhousingandhomelessness
ClearinghouseforLaborEvaluationandResearchUSDepartmentofLaborProvidesresearchonworkforceandlabor-relatedpoliciesandprograms
J-PALEvaluationsAbdulLatifJameelPovertyActionLabStudiesonpoverty-reducinginterventionsfromaroundtheworld
CampbellSystematicReviewsCampbellCollaborationSystematicreviewofpolicyinterventionsbyscholarlycoordinatinggroupsinsevenissueareascrimeandjusticeeducationinternationaldevelopmentmethodssocialwelfareknowledgetranslationandimplementationanddisability
Benefit-CostResultsWashingtonStateInstituteforPublicPolicyAWashingtonState-charteredprogramtoestablishthelikelihoodofrelativecostsandbenefitsforprogramsacrossthefollowingareasjuvenilejusticeadultcriminaljusticechildwelfarepre-Kto12educationchildrensmentalhealthhealthcaresubstanceusedisordersadultmentalhealthpublichealthandPreventionworkforcedevelopmentandhighereducation
EffectiveInterventionsCentersforDiseaseControlandPreventionRepositoryforevidence-informedHIVpreventioncareinterventions(withstepsonhowtoimplement)
Journalsandindexes
OnceyouhavereviewedrelevantinformationaccessiblethroughclearinghousesitisworthtakingalookatindividualstudiespublishedasscholarlyarticlesYourreviewerswilldothisbysearchingarticleindexesandacademicjournalsUnfortunatelyanumberofusefulacademicjournalsandindexingplatformsmaynotbegenerallyaccessibletoyourreviewersifyoudonothaveasubscriptiontothemToaddressthisconsiderpartneringwithalocal
Reviewtheevidenceandcollectdetails
25
collegeoruniversitytoprovideyouwithalistofarticlesfromthefollowingsources(Thisisalsoanexcellentopportunitytoseekresearchsupportfromanyacademicstakeholdersyouidentifiedpreviously)
Indexes
GoogleScholarincludesmostpeer-reviewedonlineacademicjournalsandbooksconferencepapersthesesanddissertationspreprintsabstractstechnicalreportsandotherscholarlyliteratureincludingcourtopinionsandpatentsItallowsyoutospecifyapublicationrangeinyoursearchandprovideslinkstolikelysourcesforalllistedarticles
PolicyFileIndexisauniqueresourceforUSpublicpolicyresearchinthatitgrantsusersaccesstotimelyupdatedinformationfromover350publicpolicythinktanksnongovernmentalorganizationsresearchinstitutesuniversitycentersadvocacygroupsandotherentities
PolicyArchiveworkstocollectanddisseminatesummariesandfulltextsvideosreportsbriefsandmultimediamaterialofthinktankuniversitygovernmentandfoundation-fundedpolicyresearch
Anothersourceforhigh-qualityresearchisthereportsoftheCongressionalResearchServiceYoucanfindtheirreportsaggregatedhere
Specificjournals
Oftenthemostusefulacademicjournalsforunearthingpolicyorprogramreviewswillbeineachspecificsubjectarea--thatisineducationjournalsforeducationpoliciesYoucanuseageneralindexlikeGoogleScholartobegintolearnwhichjournalsaremostoftenusefulforyoursearchHoweveryoucanalsosearchthefollowingtwogeneralistpolicypublicationsforissuesandarticlesrelatedtoyourtopic
PublicAdministrationReviewJournalofPolicyAnalysisandManagement
Newssourcessubjectmatterexpertsandinterviews
AlthoughitisvaluabletoknowwhathasbeenexploredthroughascholarlyresearchframeworkitisalsousefultolearnwhatkindofinterventionsarebeingtriedinthefieldrightnowandhowpeoplefeelaboutthesetrialsSearchsubscriptionnewsaggregatorslikeLexisNexisorfreenewsaggregatorslikeGoogleNewsorYahooNewsforyourissueareaplusthewordpolicyorprogramtoseewhatkindsofinterventionsarebeingcoveredinthenews
Reviewtheevidenceandcollectdetails
26
DonotforgettocontacttheadvisorygroupmemberswhowerechosenfortheirsubjectmatterexpertiseInadditiontoprovidinggoodmaterialthroughinterviewsthemselvestheymayhaveideasaboutothersubjectmatterexpertsyoushouldcontacttoaskaboutpotentialinterventions
Peercitiesandnetworks
InadditiontoreadingabouttheworkofothercitiesitisidealifyouareabletospeakwithpeoplewhoareinvolvedwithcurrentimplementationsofinterestinginterventionsInyourcityyoulikelyhaverelationshipswithyourneighborsbutitcanbehelpfultolookoutsideofyourregionforsolutionsaswellTofindcounterpartsinothercitieswhoareworkingonaprogramyouwouldliketolearnmoreaboutyoucancontactstaffinintermediaryorganizations(likeGovExoryourprofessionalassociations)whomightbeinapositiontoputyouintouchimmediatelyorwhomightseeanothergoodrouteforsettingupaconversationHereisalistofnetworksthatmaybehelpful
InternationalCityCountyManagementAssociation(ICMA)NationalAssociationofCounties(NACO)NationalLeagueofCitiesUSConferenceofMayorsWhatWorksCities
Collectthedetails
Onceyourreviewershavefoundevidenceonpossiblepoliciesorprogramstoimplementtheyshouldcapturekeydetailsthatwillhelpdecisionmakersunderstandthepotentialeffectivenessandapplicabilityofeachpotentialintervention
Foreachsourcethathasprovidedinformationyouwanttoconveyusethetemplatebelowtocaptureasmanyofthefollowingdetailsaspossible(Youdonotneedtocompletethisforsourcesthatprovideduplicativeinformationorareotherwisenotthatuseful)
Programorpolicyname (egldquoHousingFirstrdquoorldquoblightedpropertiesregistryrdquo)
Sourcetype (egjournalarticleinterviewsubjectmatterexpertstudyassociationcommunityparticipationprocess)
Sourcedetails (egarticletitleandpublicationnameandtitleofinterviewee)
Evaluationmethodology
(egrandomizedevaluationcomparativeevaluationoftwodifferentgroupsbeforeafteranalysis)
Evidencelevel (egLevel12345)
Programorpolicy
Reviewtheevidenceandcollectdetails
27
description
Programorpolicygoal (1sentencecapturingtheprincipalaimoftheintervention)
Existingimplementations
(WherehastheinterventionbeentriedListasmanysitesasyouareawareof)
Characteristicsofimplementationsites
(ForeachimplementationsiteofferbasicgeographicIDmdashcityandstateThenyouwillwanttocollectthedetailswhicharerelevanttoyourdecisionmakersrsquoconsiderationsaboutwhatmakesalocationldquosimilartordquoorldquodifferentfromrdquoyoursituationYouwillatleastwanttoincludeforeachimplementationsitePopulationnumberPopulationdensity(ruralurbansuburban)Socio-economiccharacteristicsBeforestartingaskyourdecisionmakersaboutanyparticularcharacteristicsofprogramimplementationsitestheywouldwanttoknowbeforeseekingtoreplicate)
Characteristicsofimplementationpartners
(Anyavailabledetailsaboutorganizationsinstitutionsorpeoplewhowereinvolvedwithimplementingtheintervention-egindependentchildcarecenterpublicelementaryschoolcitypublichealthnurses)
Characteristicsoftargetpopulationeligibleprogramparticipants
(Anyavailabledetailsaboutpeoplewhoarethebeneficiariesoftheprogramorotherwiseincludedoraffectedbytheintervention-egnumberofprogramparticipantscategoriesofpropertyownersaffected)
Policyorprogramduration
(HowlonghastheinterventionbeeninplaceineachimplementationsiteHasitendedandifsowhy)
Whatbenefitscamefromeachimplementation
(Anyobservedpositiveeffectsoftheinterventionincludingeffectsize)
Statisticalestimatesofinterventioneffectiveness
(Anyavailablestatisticalestimatesoftheinterventionrsquoseffects)
Whatdrawbackswereobserved
(Anyavailabledetailsaboutunintendedorunavoidablenegativeeffectsoftheintervention)
Programorpolicycosts
(Anyavailabledetailsaboutprogramcostsperimplementation)
Reviewtheevidenceandcollectdetails
28
Reviewtheevidenceandcollectdetails
29
DecideonapolicytoimplementbasedonlocalfactorsTogetthecriticalbuy-infromleadershiptoimplementanewpolicyorprogramtheprojectleadshouldscheduleameetingwithdecisionmakerstopresenttheevidencesurroundingthepolicyoptionscapturedinthetemplateDependingonyourinsightintoyourcityleadershipswillingnesstomakechangeyoumaywishtosharetheresearchyouhaveconductedontherootcausesoftheproblemandsomeofyouranalysisontheimpactofcurrenteffortstodate
Considertheapplicabilityofanapproach
AsdiscussedinthepreviouschapterpolicymakersoftenlookforevidenceofeffectiveprogramsinothercitiesbutoftenthequestionarisesofhowgeneralizableaprogramisfromonecitytoanotherTheAbdulLatifJameelPovertyActionLab(J-PAL)suggeststhatinreviewingevidencefromanotherlocationitishelpfultoidentifythespecificmechanismthatwasused(deliveryofprogramorserviceinformationsharingetc)identifythelocalconditionsthatwereinplaceobservethebehaviorchangethatresultedfromthemechanismandfinallytounderstandlocalcontextandconditions(legalorcostconstraintsculturalnormsetc)Thissetofpresentationelementscanbedrawnfromthelistofdetailscollectedintheresearchtemplate
EducationpolicymakersinChicagoILusedfindingsfromprogramsinKenyaandIndiainwhichchildrenwhohadfallenbehindtheirpeersweregiveneducationalsupporttargetedtotheirleveloflearningThemechanismusedwastargetedteachingandthereweresimilarconditionspresentinallthreecountriesAlthoughsomestudentshadfallenbehindinthecurriculumteacherswereincentivizedtoteachgrade-levelmaterialsConsiderationsofhowthelocalcontextinChicagomightaffectimplementationofatargetedteachingprogramincludedunderstandinghowtheneighborhoodsthatthestudentswerefrommayhaveaffectedtheireducationaloutcomesandthehighcostofteachinginsmallgroupsInChicagoMatchEducationprovidedtwo-on-onetutoringwhichmatchedstudentswithwell-educatedindividualswhohavenothadformalteachertrainingforalower-costmodelTheresultswerepositiveStudentswhoweretutoredperformedbetterinmathandweremorelikelytopasshighschoolclasses
ThegeneralizabilityframeworkproposedbyJ-PALisbelowUsethisasatemplateasyouconsiderpotentialpolicysolutions
Decideonapolicytoimplementbasedonlocalfactors
30
SourceTheGeneralizabilityPuzzle
OnceyouhavenarrowedthepolicyoptionsdiscussthequestionsbelowforeachoptionsyouareconsideringThisstepiscriticalasunderstandinglocalcontextissoimportantinsuccessfulpolicyimplementation
CommunityContextQuestionnaire
Localcontext
DoesthisoptionmakesenseinourcommunityIsthisaonesizefitsallapproachIsanyparticulargroupabsentfromthisapproachIfsoisthatok
Youranswers
Effectiveness
WhatwouldweexpecttoobserveifthepilotisaseffectivehereasinCityX
Youranswers
Timing
DoesthetimingmakesenseIsthistherighttimetoimplementthispolicychange
Decideonapolicytoimplementbasedonlocalfactors
31
WherearethepotentialroadblocksthatcouldaffectthescheduleIsthereanywaytomitigatethisHowlongwouldittaketoseeanimpact
Youranswers
Implementation
DowehavethestaffcapacitytoimplementthisArethereadditionalpartnersthatyoucanbringtothetabletosupportimplementationDoesthismakesensegivenourformofgovernment
Youranswers
Authority
DoesourcityhavetheauthoritytoenactthisIfsowhatistheappropriatemechanismAretherelegalbarriersArethereanyexistingpreemptionissuesfromthestateorfederalgovernmentCanthosebeaddressedthroughlegislationadvocacyIstherepoliticalsupportoroppositiontothis
Youranswers
Cost
HowmuchwillthiscostannuallyHowwillwepayforitCanthepolicybeimplementedwithexistingfunding
IfsoisthisfundingreliableHowwouldthispolicyoptionaffectourcitysbudgetintheshortandlongtermWhatotherfundingoptionscanweconsiderIsthereanalternativethatwouldcostlessandgiveussimilarresultsWillanyconstituencybeadverselyaffectediffundingforprogramsthatimpactthemisshiftedtoanotherprogrampolicyorinitiative
Youranswers
Decideonapolicytoimplementbasedonlocalfactors
32
EstablishaframeworktomonitorprogressNowthatyouhavereviewedtheevidenceconsideredlocalcontextandchosenyourpolicyitistimetodeterminehowyourcitywillmonitorprogressofthisnewapproachCitystaffandleadersneedtoestablishtheinfrastructureforongoingcontinuousdecisionmakingbasedonwhetherthepolicychangeisleadingtothedesiredoutcomesforresidentsandwhetherthechangeismeaningfulandrationaltofrontlineworkersSettingarbitrarygoalsatthelocallevelthatconflictwithgoalsatotherlevelsisconfusingtotheindividualsresponsibleforachievingthegoalsGoalsthataresettoohightoachievecanbedemoralizingtostaff
ThissectionincludesstepstosetupavenueinwhichCitystaffandleadershiproutinelyvisitkeyindicatorstodeterminewhatisorisnotworkingandtomakedecisionsinreal-timeFormorereadingonperformancemanagementpleasevisittheGovExPerformanceManagementGettingStartedGuide
InperformancemanagementtermssuchasindicatormeasureandmetricmaybeusedinterchangeablyInthisguidewewilluseindicatorInyourworkthemostimportantthingistobeconsistentinwhichevertermyouuseThisvideomaybehelpfulinunderstandingthisconceptandcommunicatingittoothers
Establishthefundamentalsandgatheryourteam
FirstdeterminewhetherthereisanestablishedvenueorregularmeetingscheduleduringwhichyoucandiscussprogressIfyourcityhasanexistingperformancemanagementstructureonarelatedtopicaddingadiscussionoftheprogressofpolicychangetoyouragendacanbeagreatideaIfthiswillbeastandalonemeetingdecidehowfrequentlytheprojectteamanddecisionmakerswillmeetandputthosemeetingsonthecalendarfortheentirelengthoftheproject
DeterminewhowillstaffandattendroutinemeetingsonCityprogressThesemeetingsmayincludetheMayorandorCityManagerdepartmentheadstaffwhoareimplementingthepolicychangeandanalysts
Identifyindicators
ItiscriticalthatindicatorsbeanimportantcityandcommunitypriorityaswellassomethingtheCityplaysamajorroleinaddressingTheremaybeinstancesinwhichtheCitywillneedtoreachouttoorcollaboratewithexternalpartnerstogetamorecompletepictureofprogressontheoverallpolicychange
Establishaframeworktomonitorprogress
33
Anoutcomeindicatordescribesthedesiredeffectofthetargetedpopulation
ExampleBellevuesfamilyfriendlyneighborhoodsindicatormeasuresthepercentofresidentswhobelievetheyliveinaneighborhoodthatsupportsfamiliesparticularlythosewithchildren
WithyourkeypartnersrevisityourproblemstatementcurrentstateandfuturestateexerciseandbegintoidentifytheoutcomeindicatorsthatwillillustratesuccessFornowchooseasingletop-levelkeyperformanceindicator(KPI)ItiscriticalthattheKPI-thepieceyouwillmeasure-beapriorityforthecommunityandsomethingtheCityplaysamajorroleinaddressing
Outputindicatorsmeasuretheprocessofprogramorservicedelivery
ExampleSanDiegoMayorKevinFaulconersgoaltorepair1000milesofstreetby2020
ReviewingtheadministrativedatasetinventorybegintoparedowntherelevantprogramsanddatasetsthatsupportyourKPIWhatarethemeasuresofeffectivenessforeachprogramTheseareyouroutputindicators
Thediagrambelowillustratestherelationshipbetweenoutputandoutcomeindicators
ThinkaboutimplementationindicatorsasmeasuringstepsintheprogramsrolloutthatarenecessarytoobtaindesiredchangesImplementationoutcomescanfocusontheinterventionstaffdeliveringtheinterventionanorganizationorsystemandcanaddressfeasibilityadaptabilitysustainabilityandcostTwokeyquestionstoconsiderwhendevelopingimplementationindicatorsare
1 Istheinterventionacceptableandappropriatetothecommunityand
2 IsitfeasibletoimplementinterventionXatthistimewiththeseresources
Establishaframeworktomonitorprogress
34
Amoreextensivelistwithdefinitionsofpotentialimplementationoutcomesislinkedhere
UsethetablebelowtochoosethedesiredimplementationoutcomesforyourpilotandtoplanyourmeasurementtechniqueThisshouldincludelevelofanalysis(individualproviderorconsumerorganizationorsetting)yourmeasurementtechnique(surveysobservationinterviewfocusgroupsadministrativedataanalysis)andthephasewhenyouwillcollectthisinformation(earlymidlate)Theremaybemorethanonelevelofanalysismeasurementtechniqueandphaseofdatacollection
Implementationoutcome
Definitionofoutcome
Levelofanalysisandmeasurement
Yourmeasurementtechnique
Phaseofpilotwhenyouwillcollectdata
Usethespacebelowtocaptureyourindicatorwishlist
Indicatorwishlist
Outcomes_____________________________________________________________________________
Outputs________________________________________________________________________________
Implementationoutcomes______________________________________________________________
Setapublichigh-levelgoal
Thehigh-levelgoaliscriticalinestablishingdirectiontogalvanizethepeoplewhowillbedoingtheworkWhendiscussingthegoalusetheMayorsbullypulpitLookforkeymilestonesinCitygovernmenttosharethegoalstatementsuchasthebeginningofaMayoraladministrationaStateoftheCityspeechorotherpublicremarks
GoalsshouldbeldquoSMARTrdquo
Specificaspossible(innamingthepopulationwhowillbebetterofftheactionthecitywilltake)
Measurable
Achievable
Establishaframeworktomonitorprogress
35
Results-focused
Time-bound
ThegoalstatementshouldincludeyouroutcomeindicatorlikeChattanoogasgoalldquoWewillendchronicveteranhomelessnessbyDecember2016rdquoAdiagramtohepyoubuildaSMARTgoalaroundyouroutcomeindicatorisbelow
Component Description Yourturn
Specific WhowillbebetteroffWhatactionthecitywilltake
Measurable Howwillyoumeasuresuccess
Achievable Isthegoalattainable
Results-focused Whatistheexpectedresult
Time-bound When
Setrealisticambitioustargetsandtimelinesforeachindicator
ToestablisharealisticandambitioustargetforyourgoalstatementusetheadministrativedatacollectedforyourchosenKPItoreviewactualCityperformanceinthemostrecentyearcomparedwiththethreeorfourmostrecentyearsworthofdataBegintocalculatesummarystatistics(egtotalminimumaveragemedianmaximum)toidentifytrendsCreatecumulativeandmonthlychartstoidentifytrendsandanomaliesBasedonyouranalysisofhistoricaldataignoretheanomaliesandassumetheproductivitytrendswillfollowthesameoverallpatternfrommonth-to-monthFinallyapplyanincreasedproductivityfactortotheperformanceeachmonthtoforecastfutureprogressBasedonthisanalysischooseatargetthatinspiresimprovementwhileaccountingforreality
Manycitiesthataremorematureintheirperformancemanagementpracticesrecognizetheimportanceoftakingacommunity-levelapproachtoimprovingoutcomesforresidentsThesecitiesarecreatinggoaltaxonomiesthatincludemeasuresrelatedtopublicsafetyhousingandevenrecreationCitiesdriveprogressonthesegoalsbyimplementingstrategiesthatengagemultipledepartmentsandcollaboratewithserviceprovidersinthecommunity
EventuallyyouwilldevelopgoalstatementsfortheoutputsinordertocapturetheefforttoachievetheoutcomecontainedinyourgoalstatementYoursetofindicatorsshouldbecomposedofshort-andinterim-termsupportivemeasureswithaccuratetargetsandrelevanttimelinesAnexamplegoaltaxonomyfromtheCityofLasVegassResultsVegasprogramisbelowUsethistaxonomyasamodelforyourownsuiteofindicatorsandgoalstatements
Establishaframeworktomonitorprogress
36
Yourturn
Thetemplatebelowismeanttohelpyoucraftagoalstatementforeachsupportingmeasure
Goalstatement
Departments
Outcome
Activities
KPI
Target
Timeframe
Conductanddiscussrelevantanalysis
ThrougharoutineseriesofmeetingscontinuallymonitorprogressofKPIssupportingindicatorsandanyotherdatacollectedsuchasqualitativedatatoidentifytrendsthatwillindicatewhetherthechosenpolicyishavingthedesiredimpactThesedatapointsshoulddrivethediscussionduringthesemeetingspointingtoopportunitiesforfurtherinquiryinto
Establishaframeworktomonitorprogress
37
whatisdrivingtheobservedtrendsAnalystsshouldbeencouragedandtrainedtoaskquestionsofthedepartmentsproducingthedataandconductsitevisitswhenpossibletodeepentheirunderstandingofthedataPresentationoftheanalysisshouldfocusononeideaatatimetokeeptheconversationfocusedonkeypointsandspecificopportunitiesforimprovementYoumayconsiderdevelopingapublicfacingdashboardtosharethisprogressmorebroadly
AgeneralcomparisonofthestatusquodatawithwhatwasobservedduringthepilotcanprovidedecisionmakerswithaquickunderstandingofthedifferencesbetweentheoldandnewapproachesalthoughthiswillnotshowcausalityOftenthesecomparisonswillincludeoperationalmetricssuchasdaysordollarssavedbutrememberthattheimpactonpeoplewillbemostimportantIfyouareworkingwithanevaluationpartnertheywillsupportthisstep
Usethespacebelowtosummarizetheresultsofyouranalysis
MeasureTargetvalue(seesectiononsettingtargetsabove)
Pilotvalue(whatdidyouobserveduringthepilot)
Historicalaverage(ifavailable)
Results(WhoisbetteroffIsanyonenegativelyimpacted1-2sentencesreassessingyourgoalstatement)
DocumentyourprocessandprogressincludingfeedbackfromtheparticipantsimplementationpartnersandtheprojectteamDidanythingunexpectedoccurduringthepilotArethereinternalandexternalfactorsthatmayaffectyouroutcomesThisiswheretheimplementationoutcomescanbecanhelpfulIftheinterventiondidnotgoasplannedconsiderwhetheritisbecausetheinterventionwasnotacceptableorfeasible
UsethetemplatebelowcreatedbyDenversPeakAcademytocapturethisinformationIfthe306090daytimelinedoesnotmakesenseforyourcityfeelfreetocustomizeit
CurrentState
FutureState
30Day
60Day
90day
Whatwentwell(+)
Whatdidnotgowellshouldbechanged(-)
IfaquestionordiscussionpointarisesthatcannotbeansweredinthemeetinganassignedstaffmembershouldcapturethequestionduringthemeetingAfterwardthefollowupitemsshouldbesenttorelevantpartnersandthenextmeetingshouldbeginwithadiscussionof
Establishaframeworktomonitorprogress
38
thoseitems
Takeaction
DiscussionsaboutwhetheryourindicatorsaremovingintherightdirectionandwhatmightbedrivingtrendsmayleadtoadiscoverythattherearenotenoughresourcesallocatedtothispolicyorissueTheremaybelimitedfundingforstaffprogrammingortechnologythatisdrivingthedatainthewrongdirectionIfCityleadershipisintheroomthisisagreattimetomakethecaseforadditionalfundingduringthebudgetseasonBycollectinganalyzinganddiscussingdataonthispolicyroutinelyyouarebuildingevidencetomakethecasefortherightleveloffundingtoreachtheestablishedtargets
Establishaframeworktomonitorprogress
39
Assessdataaccessibilityqualityandsecurity
Determineaccessibilityandqualityofdata
OnceyouhavecompiledyourwishlistofindicatorsitistimetodeterminetheaccessibilityandqualityofthedataTopreventcomplicationyoumaywishtoprioritizerelevantindicatorsforwhichyouarealreadycollectingdataAvoidthetemptationofadoptingatechnologysolutiontocollectnewdataatthispointsuchasaphotosurveytoolorotherappFornowyouaretryingtodeterminewhichdatasetsarerelevantaccessibleandhighqualityBesuretoworkwithyourkeyinternalpartnerswhileyouareidentifyingyourmeasuresDepartmentstaffwhoworkonprogramsarefamiliarwiththeindicatorsthatwillaccuratelysupporttheoutcomesanddatabaseadministratorswillbevaluablewhendiscussingaccessibilityandqualityofthedata
Abrieflistofconsiderationsfortheaccessibilityofyourdataisbelow
ArewecurrentlycollectingthisdatasetorwillwebecollectingitforthefirsttimeHowfarbackdoestherecordgo(timehorizon)WouldweneednewtechnologytocollectoranalyzethedataCanthedatabecollectedmanuallyorcanitbeautomatedHowfrequentlyisthedataproduced(continuouslyhourlydailyweeklymonthlyquarterlyyearlyetc)IsthedataavailablefortheentireCitypopulationorasubsetofthepopulationHowgranularisthedata(individuallevelCensusblockneighborhoodcitywideetc)DoesthedatacontainsensitiveinformationIsthereaproxymeasurethatwecanusefornowIsthedatamachinereadable
AsnotedbytheWilsonSheehanLabforEconomicOpportunities(LEO)attheUniversityofNotreDameleveragingadministrativedatacansignificantlyreducethecostsandtimelinesofrigorousevaluationCitiescanestablishaninternaldatasharingagreementacrossdepartmentsincludinglanguagethatenablesadministrativedatatobeusedforresearchpurposesCitiescanalsoidentifyexternaladministrativedata-holderssuchasacountyschooldistrictpolicedepartmenthospitaloranotherstatethatmayhaveoutcomedataimportantformeasuringprogramimpactCitiescanleverageexistingrelationshipsandorworkwithresearchinstitutionstodevelopstreamlineddatasharingprocessesandagreementswiththeseexternalentities
Assessdataaccessibilityqualityandsecurity
40
Evaluatethequalityofyourdata
QualityisasimportantasaccessibilityToevaluatedataqualityyoumaywishtocreatearubricliketheonebelow
Value ValueDescription
3 ExcellentDataQualityThedataiscompleteaccurateandupdatedregularly
2 AdequateDataQualityThedataissomewhatcompleteaccurateandupdatedfairlyregularly
1 PoorDataQualityThedataisnotcompleteaccurateorupdatedregularly
ThisisagreattimetobuildonthedatainventorythatyoustartedinthegatherinformationpageSuggestedadditionalfieldstoaddtoyourdatainventorytemplatearebelow
Program DatasetTitle
DatasetDescription DataSource
Departmentorexternalpartner
DoestheCityalreadycollectthisdata
Whatisthenameoftheprogram
Human-readablenameofthedataset
Shouldbeunderstandablebynon-technicalusers
Whatistheoriginalsystemapplicationfilethathousesit
Departmentorexternalpartnerthatmaintainsthedata
Isthisdataalreadycollected
Updatefrequency-ThefileformatforthedatasetTypicallycsvxlsxshapefiletxtetc
Format-Howoftenthedatasetisupdated
Timerange-Theperiodoftimecoveredinthisparticulardataset
Datause-Howisthisdatasetcommonlyused
Datausers-Whotypicallyusesthisdataset
Dataownerstitle-Titleofthedataowner
Dataownersposition-DataOwnersjobtitle
Dataownersemailaddress-DataOwnersemailaddressWhomanagesthedataandorisresponsibleforgrantingpermissiontoaccessthedataWhounderstandswhatthedatasetincludesandcananswerquestionsaboutit
Dataqualityconcerns-Describetheconcernsyoumayhaveaboutthequalityofthedataset
Assessdataaccessibilityqualityandsecurity
41
Division-DivisionwherethedataismaintainedIfmorethanonedivisionisresponsiblelisttheprimaryownerofthedataset
Sensitivedatacomments-DescribetheissuesthatwouldpreventthecityfromsharingthedatapubliclyCommonsensitivityconcernsincludeprivacyviolationssecurityissuesorhighcostorstaffdemandstopostthedata
Publishingstatus-Thistheplacetoputanycommentsregardingthecurrentstatusofthedataset
Publishedlink-Thisisthelinktothepublisheddatasetontheopendataportalorcitywebsite
Reviewdatasecurityandprivacyprotectionprotocols
OftenapolicychangealsoinvolvesachangeinthedatathatisbeingcollectedhowthatdataisbeinghandledwhohasaccesstoitandahostofotheradjustmentsAsyoutakestockofthedataassetsthatareavailableforyourinitiativeitisvitaltoalsothinkthroughwhatwillbedonetoprotectthatdataandtheprivacyofthepeopleitrepresentsAbigpartofgooddatasecurityinvolvesstrongdatagovernancepracticesthatdefineandidentifysensitivedataandconsciouslyworktobalancethevalueandriskofstoringsuchsensitivedataOnewayofevaluatingyourdatacomesfromcaselawrelatedtotheUSFreedomofInformationActwhichwehavetranslatedtothechecklistbelow
1 Woulddisclosureresultinasubstantialinvasionofprivacy2 Haveweconsideredtheextentorvalueofthepublicinterestandthepurposeorobject
oftheindividualsseekingdisclosure3 Istheinformationavailablefromothersources4 Wastheinformationgivenwithanexpectationofconfidentiality5 Isitpossibletomouldreliefsoastolimittheinvasionofindividualprivacy
TheseriskscouldbemitigatedusingavarietyofstrategiesfromaggregatingdatatoobfuscatingindividualstoredactingfieldsthatmayincludePersonallyIdentifiableInformation(PII)orre-evaluatingyourdatacollectionmethodstofindalternativewaysofgettingtheinformationyouneedThesestepsshouldbecodifiedandmanagedbyadatagovernancecommitteeifpossibleForthesensitivedatathatisdeemedessentialcreateaplanaroundpotentialdatabreachesinordertoquicklyandconfidentlyrespondtoemergencysituations
ProactivelypreparingforthesesituationsisoftenthebeststrategyfordatasecuritybutdonotforgetthetechnicalsideInvolveyourITadministratorsinconversationsaboutdatastorageandaccessensuringthattheywillbeprotectedbyenterprise-gradecybersecuritysystems
Assessdataaccessibilityqualityandsecurity
42
Assessdataaccessibilityqualityandsecurity
43
Designimplementationandevaluation
Designtheimplementation
NowthatyouhaveyouraccountabilityplanandkeyindicatorsitistimetogetspecificaboutyourplanforimplementationThetemplatebelowwillhelpyouthinkthroughthespecificmechanismsofimplementingpolicychange
UnderstandingorganizationsandsystemsiscriticaltounderstandingandmakingdecisionsaboutwhereandhowinterventionsshouldbeappliedApolicyimplementedinasinglesettingmayshowpromisingresultsbutimplementingmultiplepoliciesandprogramsatthesametimewithdifferentactorsresponsibleforeachaspectcanhelpensuresustainablebehaviorchange
InSomervilleMAShapeUpSomerville(SUS)isacitywidepublichealthprogramthatisimplementedacrossmultiplelevelsofinfluenceandsystemsthroughinterventionsbeforeduringandafterschooltoincludechangestobuiltenvironmentsuchasbikelanesregionalmasstransportationandphysicaleducationequipmentinschoolsandgymstoincreasephysicalactivitywellnesspoliciesandschoolgardensnutritioneducationduringandafterschoolandhealthyrestaurantprogramstosupporthealthyfoodandhealthyeatinghabits
Usethetemplatebelowtoplanyourimplementation
Policyimplementationplanningtemplate
ListthespecificdesiredoutcomesInthepreviousstepyouworkedwithdepartmentstafftoestablishspecificmeasurableoutcomesandinterimindicatorsPrioritizetheindicatorsthatwilldemonstrateprogressUsethespacebelowtocapturethedesiredoutcomesofthepolicychange
ExampleShapeupSomervillebeganwiththegoalofpreventingobesityin1stthrough3rdgraders
Yourturn
____________________________________________________________________________
____________________________________________________________________________
Designimplementationandevaluation
44
SpecifythetargetpopulationAreyoudesigninganinterventionthattargetsonlycertainsitesAreyouplanningonimplementingaprogramorinitiativeforaspecifictargetaudienceClearlyspecifythepopulationyouplantotarget
IfyouplantoconductarigorousimpactevaluationasnotedbytheWilsonSheehanLabforEconomicOpportunitiesitisimportanttoestablishconcreteobjectiveeligibilitycriteriaandensurethatthesecriteriaareappliedconsistentlybyallstaffduringscreeningandintakeItisalsovitaltoidentifytheindividualswhoareeligibleforaparticularprogramorserviceevenifeveryoneeligiblecannotbeservedbyexistingresourcesThishelpscitiesidentifyanyunmetneedsandguidefutureresourceallocationwhilealsocreatingthepossibilityofarigorousimpactevaluationbyidentifyingapotentialcomparisongroupFinallywhenconductingscreeningandintakeitisimperativetocollectandmaintainbasicinformation(egnamedateofbirthgender)aboutallprogram-eligibleindividualsnotjustthosewhoparticipateintheprogram
ExampleIntheShapeUpSomervilleexampletheprogrambeganbytargeting1stthrough3rdgraders
Yourturn
____________________________________________________________________________
____________________________________________________________________________
DefinethelocationWherewillthepolicychangetakeplaceWillimplementationtakeplaceatonesiteormultiplesites
ExampleShapeUpSomervilleinterventionsweredeliveredinschools
Yourturn
____________________________________________________________________________
____________________________________________________________________________
ClarifyimplementationpartnersandarticulatetheirrolesWhatarethekeyorganizationsinstitutionsorpeopleresponsibleforimplementationTheprojectteamshouldbeasclearandspecificaspossibleinprovidingadescriptionoftheconcreteactivitiesthatimplementationpartnersaremeanttocarryoutPracticeprofilesmaybeahelpfultooltohelpdefinewhatisrequiredoftheactorswhoareresponsibleforimplementingtheintervention
Designimplementationandevaluation
45
ExampleInSomervillemayoralleadershipandengagementofmultiplecityagencies(publichealthprimarycareschoolsCambridgeHealthAllianceetc)streamlinedandconnectedthevariousShapeUpSomervilleinitiativesandgrantswhichhelpedtokeepthetalentandenergyfocusedonhealthyeatingandactivelivingforallsectorsofSomervilleSUSpartnersultimatelyincludedschoolsnonprofitsCityandstateagenciesandnationalfoundations
Yourturn
____________________________________________________________________________
____________________________________________________________________________
DescribetheinterventionWhatarethestepsactivitiesorprocessesRememberthatthemechanismthespecificstepsorprocessthatleadtobehaviorchangeiskeytounderstandingtheeffectivenessofthepolicychangeandsharingtheresultswithothers
ExampleGoalsofShapeUpSomervilleweretoinfluenceeverypartofanearlyelementaryschoolchildrsquosdayndashbeforeduringandafter-schooltopromotehealthyeatingphysicalactivityandtheirweightstatusInterventionincludedfoodserviceenhancementstoimprovebreakfastandlunchandstaffprofessionaldevelopmentwalktoschoolactivitiesSUSclassroomcurriculumandprofessionaldevelopmentschoolwellnesspolicydevelopmentSUSafter-schoolcurriculumandprofessionaldevelopmentoutreachandeducationtohomethroughmaterialsforumseventsldquoSUSapprovedrdquorestaurantscommunityoutreachandcapacitybuildingthroughpolicydevelopmenttrainingsmediaplacements
Yourturn
____________________________________________________________________________
____________________________________________________________________________
ClarifytimelineandmilestonesfordeliveryandevaluationHowfrequentlywilltheservicesinterventionbedeliveredHowlongwilltheprogramrunWhatisthetimetoexpectedimpacts(eg12monthsafterprogramcompletion)
Designimplementationandevaluation
46
ExampleShapeUpSomervilleledbyTuftsUniversityFriedmanSchoolofNutritionScienceandPolicyconductedaquasi-experimental(nonrandomized)interventionstudyamong1stthrough3rdgradestudentsinSomerville(intervention)andtwocomparisoncommunitiesfor9-monthsduringoneschoolyear(fall2002ndashspring2003)ThestudyshowedpositiveassociationoftheinterventiononchangeinBMISUSreleasedreportssummarizingevaluationsprogresstodateplansforadditionalprojectsandimpactstenandfifteenyearsintotheinitiative
Yourturn
____________________________________________________________________________
____________________________________________________________________________
PlanyouranalysisTheprevioussectionaddressedestablishinganaccountabilityframeworktomonitorprogressandimplementationDeterminewhichindicatorswillbediscussthroughthatprocesswhowillsharethedataandanalysisandhowfeedbackfromthoseconversationswillbeincorporatedbytheprojectteam
ExampleTheMayorrsquosofficeimplementedadata-drivendecisionmakingapproachcalledSomerStatwhichwassynergisticwiththeapproachestakenbytheresearchandevaluationteamsdeployedforthespecificgrants
Yourturn
____________________________________________________________________________
____________________________________________________________________________
PlanforsustainabilityIfthepolicyandprogramchangeissuccessfulhowwilltheprojectteamsustainandexpandtoadditionalsitesorpopulations
ExampleAlthoughmostofthepartnershipsforgedintheearlyyearsweremaintainednewgrantsandcollaborationsexpandedattentiontoolderpublicschoolstudentsactivetransportationandtheGreenLinesubwayextensionthebuiltenvironmentfarmerrsquosmarketeffortssuchasldquoGrowninSomervillerdquowithUnionMainStreeteconomicdevelopmentandhealthdisparitiesaswellasdatacollectionmonitoringandsurveillance
Yourturn
Designimplementationandevaluation
47
____________________________________________________________________________
____________________________________________________________________________
Pilotplanningtemplate
IfyouplantoconductapilotfirstdeterminethepurposeofthepilotDoyouwanttoobservewhetherthepolicychangeresultsinthedesiredbehaviorchangeAreyouconcernedaboutfeasibilityscalabilityorcostDoyouwanttotestwhetheryourdesiredresultscanbeachievedthroughthispolicyapproachClarifythegoalofthepilotandthequestionsyouhopetoanswerThiswillhelpyouidentifythekeyindicatorsthatyouwillusetomeasurewhetherthepilotissuccessfulBeforebeginningestablishabaselineofcurrentperformanceandsetyourtargetsaccordinglybasedonwhatyouhopetoaccomplishthroughthepilot
NextestablishthepilotplanforyourprojectteamincludingmilestonesforteamcheckinstodeterminewhethertomoveaheadorchangecourseThisisseparatefromthemeetingsthatwilloccuraspartoftheaccountabilityframeworkUsethespacebelowtocreateyourtimelineanddevelopyourmeetingschedule
Task Personresponsible Startdate Checkindate Deliverable
Planyourevaluation
WhileyouaredesigningtheimplementationitisalsotimetobegintodevelopyourplanforanalysisAsnotedbytheWilsonSheehanLabforEconomicOpportunitiesdecisionsaboutprogramsetup-includingoutreachscreeningintakedatatrackingandenrollment-candeterminewhetherornotarigorousimpactevaluationwillbepossibleAsdiscussedcityleadersmayfindithelpfultoincludeevaluationpartnersinearlyconversationsaboutprogramsetupDuringpreviousstagesyoumayhaveworkedwitharesearchertoplanyourproposedpolicychangeYoushouldbeclearaboutthepurposeoftheevaluationanddevelopanevaluationplanBelowareadditionalconsiderations
Causationvstrendanalysis
IntheEstablishaframeworktomonitorprogressstagetherewasguidanceontrackingtrendsovertimeItisimportanttonoteherethatmonitoringdataisdifferentfromanalyzingacausaleffectTrendanalysiswillillustratethatchangesareoccurringandmayleadtoquestionsthatcouldbeansweredbyyourevaluationbuttrendanalysisalonedoesnotprovideevidenceaboutwhatisdrivingthosechangesYoumayseechangesinyour
Designimplementationandevaluation
48
outcomeindicatorsbasedonvariablesoutsideoftheCityscontrolForinstanceachangeinfederaltradepolicycanhaveanimpactemploymentratesinyourCityoutsideofyourCitysnewjobtrainingandplacementprogram
ToaddresscausalityyoumustcomparetheoutcomesofindividualswhoreceivedtheinterventionwiththosewhodidnotWithoutacomparisongroupofsimilarindividualswhowerenotexposedtotheinterventionitisimpossibletoknowwhichfactorsareresponsiblefortheoutcomesRandomizedControlTrials(RCTs)inwhichindividualsarerandomlyassignedtotreatmentandcontrolgroupsarethegoldstandardinprovingcausalityIfyoucandemonstratethatoutcomesimprovedforthosewhoreceivedtheinterventionanddidnotimproveatallorimprovedlessforacomparisongroupyoucanbesomewhatcertainthattheinterventionwasthecause
Examplesofothernon-causalimpactevaluationsthatarerigorousandreliableincludecertainquasi-experimentalstudiesusingcomparisongroupdesignssmartinterruptedtimeseriesanalysesthatmodeltrendsovertimeinwaysthatallowestimationofinterventioneffectswelldesignedpre-posttestanalysesneedsassessmentsimplementationstudiesetcFormoreinformationondifferenttypesofevaluationsseetheSageResearchMethodswebsite
Lookfordifferencesingroups
BuildintimetoalsoexaminethepilotdatatoseeifgroupsareexperiencingdifferentresultsReviewingperformancebasedonoutcomemeasuresforapopulationinaggregatemayhidethesubtlewaysthatsomepeoplemaybeaffecteddifferentlysoitisnecessarytocarefullyanalyzethedatafordisparateoutcomesForinstancetheMovingtoOpportunity(MTO)studythatexaminedtheimpactofhousingvouchersonfamilieswholivedinhigh-povertyneighborhoodsfoundthatchildrenwhomovedbeforetheageof13hadbettereducationandhigherincomesasadultsHoweveradultsdidnotseeimprovededucationandemploymentoutcomes
Disaggregatingdataintomorespecificcategoriesofraceandethnicitycanhelpunderstandtrendsanddisparitiesinamoregranularwaywhichcanhelpdecisionmakerstargetpoliciesandinterventionsinamoreequitableway
EarlierintheRoadmapweworkedtoidentifystakeholdersandwhomightbepositivelyandnegativelyimpactedbyachangeinpolicyUsetheresultsofthatexercisetoinvolvethecommunityininterpretingdataanalysisandtogatherqualitativefeedbackfromthecommunitytocallattentiontoissuesForinstanceanincreaseinyouthdetentionmaybeexplainedbyinefficienciesorlackofaccesstotransportation
Designimplementationandevaluation
49
Designimplementationandevaluation
50
Shareyourstoryandsuccess
Sharetheresults
Storytelling
AnintegralstepinyourstrategyissharingtheresultsofyourpilotorpolicychangewithyourcommunityThisincludesdevelopingmessagingaroundtheworkofyourorganizationandcreatingaplantodisseminateyourstoriesWhytellstoriesaboutyourwork
Informandorunderstandasubjectorsituation
Giveneededcontextofdataorbitsofinformation
Relatablestoriesarememorableforreadersviewers
ReadersorviewerswillseetheinformationasmorecompellingwhenthereisademonstratedandrelatablechangeorimpactonthemselvesortheirfriendsneighborsandcolleaguesStoriesareaboutpeopleandgovernmentisfundamentallyabouthelpingpeopleSotellthestoriesofthepeopleyouandyourcitycolleagueshelped
Keepthesetipsinmindasyoudevelopyourstorywithyourteam
WhatwereyourfindingsandwhyshouldanyonecareTellyouraudiencehowtheproblemwassolvedWhathastheimpactbeenonresidentsandcitystafftodateWhatwasthesuccessandweretheremultiplesuccessesalongthewayIfyouhavepreliminaryresultsordatashowingimprovementtosharethisisthetimetodoit
WhowasinvolvedandimpactedReviewthelistofstakeholdersHowwereeachofthemimpactedbytheprogramHumanizeyourstoryandimpactbyaddingadditionalvoicesThesevoicescanbeoneofthefollowing
Apersonorfamilywhowaspositivelyaffectedbythechangesuccess
Aninvolvedthird-partydescribingboththeproblemandthesolution(ieanorganizationnonprofitprovidersbusinesstradeassociationsportsteametc)whoispartofthecommunitythatthegovernmentworkedwithonthechangesuccess
Someonefromthegovernmentitselfwhocanspeakaboutthechangesuccess
TellmoreaboutyourcityDescribeyourcityincludingpopulationdemographicsindustriesetcWhatareitschallengesWhatisitknownforWhatmakesituniqueHowisitsimilarordifferentfromothercitiesintheUS
Shareyourstoryandsuccess
51
DescribeyourcityrsquoschallengeWriteoutthechallengeyourcityisfacing(asdefinedinyourproblemstatement)HowdoesitaffectresidentsHowdoesthisproblemaffecttheworkofcitystaffmembers
AddthedetailsProvidedetailsonyourappliedresearchstrategyTalkaboutwhatyourcityhasdonenthepasttoresolvetheissueDiscusshowyourstrategyhaschangedwhenyoubegantoadoptthisapproachWasmoneysavedorinvested
Usethisstorymapexercisetoorganizeyourthoughtsandguaranteethatyouincludealloftheelementsofatraditionalstory
AfteryoudevelopyourmessagingcreateacommunicationsmarketingandoutreachstrategydesignedtospreadyourmessageYourstrategyshouldwalkyouthroughthefollowingkeyactivities
1 IdentifyandprofileyouraudienceWhencreatinganddisseminatingyourmessageitisalwaysgoodtothinkaboutyouraudienceWhoexactlywouldbeinterestedinwhatyouhavetosayThegeneralpopulationofthecityPeoplewhoworkinnonprofitsBusinessownersItmattersbecausenoteveryonewholivesinacitymaybethetargetoftheinformationBusinessownersmaybemoreinterestedindatashowingeffectsonthebusinessdistrictthanothersandsoonWewillcoverthislaterbutthisstepwillalsohelpyoutothinkofthemosteffectivewaytoreachyouraudience
2 Defineyourcalltoaction(ifyouhaveone)SpendsometimeconsideringwhyyouareinterestedintellingyourstoryDoyousimplywanttoupdatethepubliconyourprogressorcelebrateyoursuccessesAreyouinterestedinyouraudiencechangingitsbehaviororsupportingaspecificinitiativebeinglaunchedbyyourorganizationIfyouwanttohaveyouraudienceactorbehaveinacertainwaybespecificandincorporateitinyourmessaging
Shareyourstoryandsuccess
52
3 SelectperformancemeasuresThinkabouthowyouwoulddetermineifyourcampaignissuccessfulandselectappropriateperformancemeasuresYouwillrevisitthesemeasuresattheendofyourcampaigncommunicationsplanimplementationtodeterminewhetheryouneedtomodifyyourcommunicationsandoutreachstrategyforthefuture
4 MethodstocommunicateyourstoryTherearemanywaystocommunicatesuccessofyourprojectHerearesomesuggestionsonreachingaudiencesItisimportanttodeterminewhichmethodsarethemosteffectiveforyourdesiredaudiencetoreceiveinformationDoyourresearch(PewResearchCenterhasexcellentstudiesonthereachofmediadigitalmediaandmarketing)ForexampleifseniorsareyourtargetpopulationandtheyarelessinclinedtouseInstagram(thatistrue)youmaywanttoconsiderFacebookorthenightlylocalnewsbothofwhichareusedmoreoftenbyseniorsBelowisalistofsomemethodsandtoolsyoucanuse
i Pressreleasestolocalnewspapersandtvstations
ii ShortsmartphonevideosAgreatwaytoengageanaudienceisthroughvideoAskonequestionofsomeonewhoispartofthestoryMaybethereisasocialworkerwhocandiscusshungerinyourcityThesevideoscanbereusedonmultipleplatforms
iii Long-formwriting
iv TellingindividualstoriesthroughaseriesofphotosandshortnarrativesforpublicationonFacebookorInstagram
v Op-edsopinionpiecesplacedinthelocalpaper
vi Emailupdatestoyourlist
vii Connectingwithappropriateorganizationsthatareinvestedinthetopicandworkwiththecommunityandaskthemtodisseminateinformationtotheirmembers
viii EmployingldquoWordofMouthrdquo(WOM)tacticsbycreatingtownhallsorfreeeventsforresidentstoattendintheirownneighborhoods
5 DeterminethetypeofcollateralUsuallycampaignsthattargetresidentsrequireseveralavenuestoreachthemPeoplearenotalwaysinthesameplaceorspaceHavingakeenunderstandingofyouraudiencewillassistyouwithyourchoicesonwhatneedstobedevelopedSometimesthatmeansimagesforsocialmediaaonepageflierthatgivespeopleinformationonattendinganeventoranin-depthbrochure
Shareyourstoryandsuccess
53
6 ImplementyourstrategyKeepinmindstrategiescanbefluidSometimessomethingmaynotturnoutasexpectedevenifduringplanningsessionsitwasldquosupposedtordquoItisokYoucananalyzewhathappenedduringadebriefsessionandonceyoursurveysandothermeasurementshavebeenfinalized
Visualizations
Withtherightvisualizationyoucanreviewandshareinformationinsightsanalysisperformanceandtrendsallofwhichisvaluabletoacross-sectionofthepublicButwhichvisualizationisrightforyouTochoosetherighttypeofchartgraphdeterminethetypeofquestionyouwanttoanswerSomecommoncharttypesarebelowandmoreinformationcanbefoundhere
Exploreadditionalanalyticaltechniques
Duringthegoal-settingphaseyouestablishedoutputoutcomeandimplementationmeasuresBasedontheresultsofyourpilotyoumayneedtoadjustyourmeasuresWiththepilotdatainhandyoumayhavesomeideasofhowfutureanalysescouldbedifferentandpotentiallyimprovethequalityofyourprogramTakesometimetobrainstormotherrelevantdatasetsthatyoudidnothavethetimetopursueearlierincludingmessydepartmentaldatathatneedsmoreattentionorsourcesofpublicdatafromthestateandfederalgovernmentnonprofitsandevenprivatecompaniesthatcouldsupplementyourunderstandingofanissueUsingmachinelearningtechniquesyoumightbeabletobetter
Shareyourstoryandsuccess
54
predictdemandforhomelessservicesusingNOAAweatherforecastspreemptwatermainbreaksusingsoilstratigraphyorinvestineconomicdevelopmentbymonitoringrealestatelistingdata
Advancedanalyticaltechniquescanalsohelpyourimplementationrunmoreeffectivelybyintegratingdata-informedtargetinganddecisiontoolsintotheprocessByautomatingroutineworkflowsandsurfacingsubtletrendsthesetoolscanhelpyouiterateinamoreprecisemannerandprovidekeystrategicguidancetofront-lineemployeesthatwouldotherwisebeunavailableThesesortsofdata-science-enhancedprocessescanbeclassifiedasoneofsixprojecttypesasexploredfurtherbytheNewOrleansLANOLAlyticsteamFormoreguidanceonscopingasimilarprojecttheGovExadvancedanalyticsscopingguidegivesyouastep-by-stepoverviewofhowtobesuccessful
Usethespacebelowtolistyourideasforpossibleadvancedanalyticsprojectsbytype
Typeofproject Ideas
Findingtheneedleinthehaystack
Prioritizingforimpact
Earlywarningtools
Betterquickerdecisions
Optimizingresourceallocation
Experimentingforwhatworks
Keepthefeedbackloopgoing
GatherinformationonpublicsentimentaboutpolicychangethroughfocusgroupssurveysandpublicmeetingsledbyadvisorygroupmembersNotewhethernewinformationisavailablethatcanhelpanalyzeorcontextualizeyourprogress(egfederalorstatedatareports)Sharetheresultsofyourpilotandthepolicychangeprocesswithothercitiesandresearchers
Shareyourstoryandsuccess
55
WorkingwitharangeofevidencePresentlythereisalotofattentionandinvestmentinevidence-basedpolicymakingincludingthefederalbipartisanCommissiononEvidence-BasedPolicymakingthePewCharitableTrustsreportsandtoolkitsonHowStatesEngageinEvidence-BasedPolicymakingandtheLauraandJohnArnoldFoundationsupportedEvidence-BasedPolicymakingCollaborativeManyoftheseinitiativesfocusonrigorousevaluation--meaningrandomizedcontrolledtrials(RCT)orotherexperimentalmodels--asthebestkindofevidencetouseHoweverthisbodyofevidenceisofteninsufficientforthespecifickindsofquestionsthatgovernmentscanhavewhenmakingadecisionaboutpolicyInthisguideweproposeanapproachtosupportgovernmentsinincorporatingadditionaltypesofevidenceintotheirdecisionmaking
OnewaytounderstandarangeofevidenceistouseNestarsquosStandardsofEvidencetoframeanddefineevidenceTheBritishnonprofitNestaworkstoinjectpositivechangeintothewayUKgovernmentpoliciesandprogramsfunctionOneoftheirkeyareasofworkhasbeendevelopingWhatWorksCentresorganizationsthatcollectcommissionsynthesizeanddisseminateevidenceaboutinterventioneffectivenessacrossarangeofsocialimpactareasNestadevelopedthefollowingStandardsofEvidencetoprovideausefulwaytodocumenthowdifferentkindsofinformationdemonstratedifferentlevelsofevidence
Workingwitharangeofevidence
56
Theimportanceofpotentiallybiasedinformation
Onechallengepresentedbyanevidenceclassificationthatacknowledgesthevalueofnon-scientificformsofevidenceistofigureouthowtobalanceallofthepiecestogetherwhenyouusethemTounderstandhowtodothiswehavetothinkabouthowtopositionpotentiallybiasedinformationindecisionmaking
WhenwelookatinformationwhichhasnotbeenrigorouslyevaluatedweacceptthatthisinformationmightbebiasedThewholereasonweconductrigorousevaluationistoremovebiasfromourobservationsWeconductstructuredtrialstoproduceresultsthatareconsistentandgeneralizableIfafindingisunbiasedthisaccuracymdashthislackofbiasmdashcanbeprovedbecauseanyindependenttestercanreproduceitwhentheyfollowanidenticalprocess
YoucanlookatpotentiallybiasedinformationasjustinformationthatisearlieroninitspathtorefinementreplicationandverifiedaccuracyApieceofinformationbasedonanyoneindividualrsquosobservationsislikelytobebiasedbecauseitiscomingfromjustonesourceBiasisreducedbygatheringtogetherlotsofindependentsourcesofinformationandfiguringoutitsmostcommonattributes
MaybethebiasedinformationmakesaclaimthatistoolargemdashitsayssomethingistrueunderallconditionswhenitisreallyonlytrueundercertainconditionsorthataneffectofaninterventionisbiggerthanitstrueaverageorthataneffectlastsovertimewhenitdoesnotPerhapschangingtheclaimtobemorespecificmdashcriticallyreviewingtheinformationtoseeifspecificconditionsmustbemettoachievetheoutcomesmdashwouldmaketheclaimmoreaccurate
IfyoutestedyourclaimfounditdidnotholdupinsomecaseschangedittesteditagainuntilyourclaimrepeatedlyheldupthenyoucoulddemonstratethattheclaimwasnolongerbiasedbutaccurateThisprocessoftestingimprovingtestingandvalidatingiswhatwegetfromstructuredscientificstudy
Withindividualpiecesofinformation--crediblehypothesescasestudiesexpertopinion--youcannotassumethatkindofaccuracyAtthesametimetheseearlyobservationsarefoundationaltoanyreplicablescientificobservationsWewillnevergetanywhereifwedonotstartwiththesepiecesofinformationeveniftheymightbebiased
Combininglevelsofevidencetogetthemostdetailedinformationforyourcity
ItisespeciallyimportanttoacceptdifferentformsofevidencewhenyoufacequestionsthathavenotyetbeentestedbyindependentstudiesSomeofthemostimportantquestionsrelatetothingsthatareveryspecifictoyoursituationYoufaceaparticularcombinationof
Workingwitharangeofevidence
57
issuesactorstrendsandresourcesthatisultimatelydifferentfromeveryotherplaceintheworld
ThisisimportantbecauseaspolicypractitionersandnotscientistsyoucaremoreaboutyourspecificcasethanahypotheticalaveragegeneralcaseYoucarelessaboutthetotallikelyrangeofimpactsfromapolicythanhowthissingleimplementationwillworkYouwantinformationaboutpoliciesthatwillprovideyouwiththebestpossiblepredictionabouthowtheywillworkinyourcontext
ByputtingdifferentkindsofevidenceintothesameframeworkNestarsquosStandardsofEvidenceimplythatwhenyoucannotgetrigorouslytestedexperimentallyverifiedinformationitisappropriatetoworkwithwhatwedohaveThisisrelevantforthemunicipalimplementationcontextbecausenostudypulledfromaliteraturereviewwillaccountforallofthevariablesyoufaceinyourowncontext
Intheabsenceofhavingastudythatpredictshowaninterventionworksinyourspecificcontextyouwillneednon-scientificinformationtotrytofillinthosegapsYouwillneedtogatherstoriesfromresidentsstaffmembersandotherstakeholdersthatsitonLevels1or2ofthestandardsofevidence
Someofthekindsofinformationyouwillprobablygetmoreeasilyfrompeoplethanstudiesinclude
howpeopleincitieslikeyoursreactedtoaninterventioninpublicforumshowpeoplewhoimplementedaninterventioninacitylikeyoursfiguredouthowtogetitpassedhowthecitycommunicatedpubliclyabouttheapparentbenefitsanddrawbacksoftheinterventionthepoliticalacceptabilityoftheintervention
Youcanremainsimultaneouslyawarethattheinformationyouaregatheringthroughindividualstoriesisbiased--notgeneralizableinformedbyspecificperspectives--whilealsorecognizingthathavingsomeinformationaboutsomethingimportantisbetterthannoinformationEvidencefromthefirsttwoevidencelevelsareessentialbuildingblocksofknowledgeandperfectlyusefulwhenyoucannotgetdetailedunbiasedknowledge
Youhavetostartsomewhereandyoushouldalwaysstartwithevidenceyouhaveaboutyourveryspecificconditionsthatyouknowareimportantforsuccess
Workingwitharangeofevidence
58
Thankyou
Interviewswithcities
GovExheldsemi-structuredinterviewswithsixstaffmembersinfourcitiestoestablishtheinitialstepsoftheRoadmapWeaskedrespondentstotalkaboutaninitiativeorpolicythattheyhavebeenresponsibleforimplementingintheircityRespondentsfromthreecitiesspokeabouttheireffortstoaddresshomelessnessandrespondentsfromonecityspokeaboutblightAfterrespondentsidentifiedtheproblemtheyspokeabouthowthecityprioritizedthisissuethevariousstepsinvolvedandbarrierspresent
GovExconductedtextanalysisoftheseinterviewswhichrevealedthatpeoplewasthemostcommonlyusedtermduringtheseconversationswhendiscussingthestepstochangeaprogramorpolicyThisisathemethroughoutthisRoadmapThefirstsectionbeginswithidentifyingtherelevantpeopleTheconversationsaroundevidencearefocusedonhumanbehaviorratherthanmethodologyorstrengthofevidence
Thankyou
59
OncewehadadraftoftheRoadmapwefollowedupwithcitiesonanindividualbasistolearnmoreaboutwhatwouldbehelpfulWearegratefultothefollowingcitystafffortheinterviewsandindividualconversations
MichaelBaskinChiefPolicyOfficerChattanoogaTN
MelissaBridgesPerformanceandInnovationCoordinatorLittleRockAR
VictoriaCarreoacutenAdministrativeOfficerLasVegasNV
TyroneGrandstrandCommunityDevelopmentPlannerFargoND
TibLaughlinDirectorofGeneralServicesKansasCityKS
TimMorelandDirectorofPerformanceManagementandOpenDataChattanoogaTN
CharlesRosendahlManagementAnalystDentonTX
Thankyou
60
MelissaSiebenAssistantCountyAdministratorKansasCityKS
GinaThompsonDirectorofStrategicServicesLewisvilleTX
JHUFacultycollaboration
GovExalsoconvenedthecitycontactsandfacultypartnersinaseriesofAdvisoryGroupmeetingsThesemeetingsservedtoconnectcitiesandfacultytounderstandoneanothersperspective
StefanBaralBloombergSchoolofPublicHealth
AshleyBernerJHUSchoolofEducation
MelissaDavey-RothwellBloombergSchoolofPublicHealth
JonLinksBloombergSchoolofPublicHealth
TamarMendelsonBloombergSchoolofPublicHealth
KristinMmariBloombergSchoolofPublicHealth
KeshiaPollackPorterBloombergSchoolofPublicHealth
JoshSharfsteinBloombergSchoolofPublicHealth
LizStuartBloombergSchoolofPublicHealth
LindsayThompsonCareyBusinessSchool
DanielWebsterBloombergSchoolofPublicHealth
Partnersindataandevidence-basedpolicy
Wealsospokewithpartnerswhoareexpertsinthefieldofevidence-basedpolicywhosharedtheirknowledgeandresourcesofcurrentandfuturethinkinginevidence-informedthinking
WendyBarrenoAssociateDirectorWilsonSheehanLabforEconomicOpportunitiesUniversityofNotreDame
TiffanyDavenportDirectorofDataPartnershipsBaltimoresPromise
TomiHiersDirectorBaltimoreCivicSiteAnnieECaseyFoundation
ChrisKingsleySeniorAssociateAnnieECaseyFoundation
KathrynOliverAssociateProfessorLondonSchoolofHygieneandTropicalMedicine
Thankyou
61
KristenWatkinsSeniorPolicyAssociateJ-PAL
GovExParticipants
GovExAnalystMiriamMcKinneyanalyzedtheinterviewtranscriptsandconductedtextanalysisSeniorImplementationAdvisorEmilyShawwillintegrateRoadmapactivitieswithadditionalservicedeliveryactivitiesatGovEx
Thankyou
62
ReferencesAbrahamKGampHaskinsR(2017)Thepromiseofevidence-basedpolicymakingReportofthecommissiononevidence-basedpolicymakingWashingtonDCCommissiononEvidence-basedPolicymakingBigmove2Extendingthegreen(nd)Retrievedfromhttpswwwcityoflewisvillecomdoing-businesslewisville-2025big-move-2-extending-the-greenBatesMAampGlennersterR(2017)ThegeneralizabilitypuzzleRetrievedfromhttpsssirorgarticlesentrythe_generalizability_puzzleBuildwithnotfor(2018January)RetrievedfromhttpsbuildwithnotforwordpresscomCairneyPampOliverK(2017)Evidence-basedpolicymakingisnotlikeevidence-basedmedicinesohowfarshouldyougotobridgethedividebetweenevidenceandpolicyHealthResearchPolicyandSystemsdoi101186s12961-017-0192-xCairneyPOliverKampWellsteadA(2016)TobridgethedividebetweenevidenceandpolicyReduceambiguityasmuchasuncertaintyPublicAdministrationReview76(3)399-402Retrievedfromhttpdxdoiorg101111puar12555Centreforpublicimpact(nd)RetrievedfromhttpswwwcentreforpublicimpactorgCernaL(2013)ThenatureofpolicychangeandimplementationAreviewofdifferenttheoreticalapproachesOrganisationforEconomicCo-OperationandDevelopmentRetrievedfromhttpswwwoecdorgeduceriTheNatureofPolicyChangeandImplementationpdfColoradoSprings(nd)Retrievedfromhttpscoloradospringsgovcommunicationspagewhat-expect-when-you-speakupContinuumofcare(coc)program(nd)RetrievedfromhttpswwwhudexchangeinfoprogramscocCouncilpresidentkicksoff$12millionyouthfund(nd)Retrievedfromhttpwwwbaltimorecitycouncilcomcontentcouncil-president-kicks-12-million-youth-fundDurlakJAampDuPreEP(2008)ImplementationmattersAreviewofresearchontheinfluenceofimplementationonprogramoutcomesandthefactorsaffectingimplementationAmericanJournalofCommunityPsychology41327-350Retrievedfromhttpswwwncbinlmnihgovpubmed18322790Fishbonediagram(ishikawa)(nd)Retrievedfromhttpasqorglearn-about-qualitycause-analysis-toolsoverviewfishbonehtmlFivewhysGettingtotherootofaproblemquickly(nd)RetrievedfromhttpswwwmindtoolscompagesarticlenewTMC_5Whtm
References
63
FligsteinNampMcAdamD(2011)TowardageneraltheoryofstrategicactionfieldsSociologicalTheory29(1)1-26RetrievedfromhttpsociologyberkeleyedusitesdefaultfilesfacultyfligsteinFligstein20McAdam20Soc20Theory20PaperpdHeinkelP(2018January12)CitymullspermitsforbikesharingcompaniesRetrievedfromhttpwwwdentonrccomnewsdenton20180112city-mulls-permits-bike-sharing-companiesHowisareamedianincomecalculated(2012October)Retrievedfromhttpswwwhudexchangeinfofaqs843how-is-area-median-income-calculatedLampkinLMampHatryHP(2003)KeystepsinoutcomemanagementUrbanInstituteRetrievedfromhttpswwwurbanorgresearchpublicationkey-steps-outcome-managementMarshDampSmithM(2000)UnderstandingpolicynetworksTowardsadialecticalapproachPoliticalStudies484-21Retrievedfromhttpjournalssagepubcomdoiabs1011111467-924800247McCannL(nd)RetrievedfromhttplaurenellencomMoultonSampSandfortJR(2016)ThestrategicactionfieldframeworkforpolicyimplementationresearchPolicyStudiesJournal45(1)Retrievedfromhttponlinelibrarywileycomstore101111psj12147assetpsj12147pdfv=1ampt=jdkhhcp4amps=e184c625e1855a6b1615ccb99ee27b1c0b3212d8PaynterB(2018January1)What100citiesarelearningfromeachotherbysharingtheirdataRetrievedfromhttpswwwfastcompanycom40518004what-100-cities-are-learning-from-each-other-by-sharing-their-dataPeakacademy(nd)Retrievedfromhttpswwwdenvergovorgcontentdenvergovenmayors-officeprograms-initiativespeak-performancepeak-academyhtmlPinnockHBarwickMCarpenterCREldridgeSGrandesGGriffithsCJRycroft-MaloneJMeissnerPMurrayEPatelASheikhAampTaylorSJC(2017)Standardsforreportingimplementationstudies(stari)ExplanationandelaborationdocumentBMJOpenRetrievedfromhttpbmjopenbmjcomcontentbmjopen74e013318fullpdfProctorESilmereHRaghavanRHovmandPAaronsGBungerAGriffeyRampHensleyM(2011)OutcomesforimplementationresearchConceptualdistinctionsmeasurementchallengesandresearchagendaAdministrationandPolicyinMentalHealth38(2)65-76doi101007s10488-010-0319-7TandonSD(2018)ProgressincommunityhealthpartnershipsResearcheducationandactionRetrievedfromhttpswwwpressjhuedujournalsprogress-community-health-partnerships-research-education-and-actionProvanKGampKenisP(2007)ModesofnetworkgovernanceStructuremanagementandeffectivenessJournalofPublicAdministrationResearchandTheory
References
64
18(2)229-252Retrievedfromhttpsacademicoupcomjpartarticle182229935895SabatierPA(1988)Anadvocacycoalitionframeworkofpolicychangeandtheroleofpolicy-orientedlearningthereinPolicySciences21(3)129-168Retrievedfromhttpwwwjstororgstable4532139SandstromAampCarlssonL(2008)TheperformanceofpolicynetworksTherelationbetweennetworkstructureandnetworkperformancePolicyStudiesJournal36497-524Retrievedfromhttponlinelibrarywileycomdoi101111j1541-0072200800281xabstractSchwartzMampWilsonE(nd)WhocanaffordtoliveinahomeAlookatdatafromthe2006americancommunitysurveyRetrievedfromhttpswwwcensusgovhousingcensuspublicationswho-can-affordpdfShearerJCAbelsonAKouyateBLavisJNampWaltG(2016)WhydopolicieschangeInstitutionsinterestsideasandnetworksinthreecasesofpolicyreformHealthPolicyandPlanning31(9)1200-1211Retrievedfromhttpsdoiorg101093heapolczw052ThelessonsofadministrativedataHigh-profilepolicy-relevantresearchpoweredbyadministrativedata(nd)Retrievedfromhttpswwwpovertyactionlaborgsitesdefaultfilesresources20161114-The-Lessons-of-Administrative-DatapdfToolsandotherneedsforqualitativeandquantitativemethods(2015February)AgencyforHealthcareResearchandQualityRetrievedfromhttpwwwahrqgovprofessionalsquality-patient-safetytalkingqualityassesstoolshtml
WeaverRCampBagchi-SenS(2013)SpatialanalysisofurbandeclineThegeographyofblightAppliedGeography4061-70
References
65
UniversityPartners
Name Title Affiliation Area(s)ofexpertise
Contactinformation
Examplesofworkingwithnon-academicdecisionmakers
WendyPBarreno
AssociateDirector
WilsonSheehanLabforEconomicOpportunitiesUniversityofNotreDame
LEOsfocusareasincludehealtheducationhousingchildwelfareeconomicopportunitycriminaljustice
wbarrenonduduorleondedu
MelissaDavey-RothwellPhD
AssociateScientist
JohnsHopkinsBloombergSchoolofPublicHealth
Health mdavey1jhuedu410-614-5854
BaltimoreCityHealthDepartmentLeadersMarylandDepartmentofHealthAddictionRecoveryServicesinDCExecutiveDirectorsofcommunityorganizationsacrosstheUSthatfocusonHIVSTIsubstanceabuseandimplementationscience
ElizabethStuart
AssociateDeanforEducationandProfessorDepartmentsofMentalHealthBiostatisticsandHealthPolicyand
JohnsHopkinsBloombergSchoolofPublicHealth
Educationsocialpolicystatisticalmethods
estuartjhuedu
Resources
66
Management
Design-ThinkingResources
EngagementGuideforCivicTech
Journeymap
PerformanceManagementResources
DenverPeakAcademy
Proxymeasuresguide
Talkingaboutperformancewithoutcomplicatingit(video)
SettingPerformanceTargetsguide
Datavisualizationguide
Performancemanagementgettingstartedguide
Exercises
Fishboneexercise
5Whys
AdministrativeData
TheLessonsofAdministrativeData(JPAL)
EvaluationResources
Quantitativeandqualitativemethods(USDepartmentofHealthandHumanServicesAgencyforHealthcareResearchandQuality)
KeyStepsinOutcomeManagement(UrbanInstitute)
RCT-YES
Sageresearchmethods
BestPracticesforMixedMethodsResearchintheHealthSciences
ACCORDChecklist
Pilotandimplementationplanning
Benchmarkingexamplesandproxymeasures
Seattleopendataplaybook
Resources
67
Datasciencescopingguide
Resources
68
AboutGovExGovExhelpsgovernmentsusedataandevidencetoimproveresidentsrsquolivesWesupportgovernmentsinbuildingtheirinternalcapacitytooperatesustainableanalyticspracticeswhichimprovecommunityoutcomesinfieldsfrompublicsafetytoeconomicdevelopmentGovExhasaproventrackrecordofworkinginconcertwithpartnerstoprovidetechnicalassistanceandtrainingatscale
SinceourlaunchinApril2015GovExhas
Trainedover2000governmentemployeesinpersonandonlineintopicsincludingdatamanagementperformancemanagementopendatacommunityengagementandanalytics
Providedtechnicalassistancetomorethan100localgovernmentsacrosstheglobeimprovingtheiruseofdatatoachieveefficienciesinservicedeliveryandbetteroutcomesforresidents
Assessedthecapacityofmorethan130governmentalorganizationstodeterminehowleadersstaffandexternalstakeholdersmanageandusedata
Localgovernmentsareincreasinglyonthefrontlinesofaddressinglargescaleproblemswithglobalimplicationssuchasclimatechangepublicsafetyandeconomicinequality
Whileevidence-basedinterventionstohelpsolvethoseproblemsareavailablelocalgovernmentstaffhavelimitedtimeandcapacitytofindandimplementresearch-basedsolutionsAsaresultcitiescontinuetoworkthewaytheyalwayshave--makingincrementalimprovementstoprogramsbutnotalwayscreatingchangeatasystem-level
ThroughresourcedevelopmentandcollaborationwithJHUfacultyGovExissupportingcityleadersinusingdataandevidencetomakedecisionsandcreatinglastingrelationshipswithresearchers
TheRoadmapwasdevelopedbyGovExaspartofitspartnershipwithBloombergPhilanthropiesWhatWorksCitiesinitiative
AboutGovEx
69
- Title page
- Introduction and purpose of the Roadmap
- Using the Roadmap
-
- Gather information about the problem
- Strategically engage key partners
- Review the evidence and collect details
- Decide on a policy to implement based on local factors
- Establish a framework to monitor progress
- Assess data accessibility quality and security
- Design implementation and evaluation
- Share your story and success
-
- Working with a range of evidence
- Thank you
- References
- Resources
- About GovEx
-
ProgramName
DatasetTitle
DatasetDescription DataSource
Departmentorexternalpartner
DoestheCityalreadycollectthisdata
Whatisthenameoftheprogram
Human-readablenameofthedataset
Shouldbeunderstandablebynon-technicalusers
Whatistheoriginalsystemapplicationfilethathousesit
Departmentorexternalpartnerthatmaintainsthedata
Isthisdataalreadycollected
TogatherthisinformationconsiderassigningonepointpersontosurveyorinterviewcitystaffSpeakingwithfrontlineworkersstaffwhoworkwithfederalgrantsdepartmentheadsandeveryoneinbetweenwillensureadeepunderstandingoftherelevantprogramsanddatasetsDuringthecourseoftheseconversationsbesuretoaskcitystaffwhichorganizationstheyworkwithexternallyandcapturethatinformationaswelleveniftheyarenotrecipientsofcityfundingInadditiontoconversationswithexpertsincitygovernmentyourcitysbudgetdocumentsareanothergreatsourceofinformation
Collectdataandobservetrends
Nexttrytogainanunderstandingofwhytheproblemstillexistsifithasbeensomethingyouhaveworkedonforawhileorwhyitisemergingnowifitisnew
AnalyzingadministrativedatacandemonstratepasteffectivenessofinitiativesrelatedtoyourproblemstatementUsingthedatasetinventorythatyoubeganintheprevioussteptalktodepartmentstaffaboutwhichdatasetsaremostrelevanttotheproblemandgatherafewyearsworthofhistoricaldataBegintolookfortrendsinthisdatadiscussingpotentialreasonsforthosetrendswithrelevantstaffCanthetrendsbeexplainedbyseasonalitychangesinfundingstaffingfluctuationsorsomeotherreasonBasedonthehistoricaldataanalysiscreateabaselineagainstwhichfuturechangescanbeobserved
OtherdatasetscanhelpprovideadditionalcontextForinstanceifthenumberofpeopleexperiencinghomelessnessinyourcityisincreasingitmaybehelpfultotakealookathousingaffordabilityIfyoudonothavethedatayouneedconsideraproxymeasureAsanexamplehousingisoftenconsideredaffordableifitcosts30orlessofahouseholdsincomeButifhouseholdincomedataisnotavailableatthecitylevelAreaMedianIncome(AMI)canbeahelpfulproxy
Gatherinformationabouttheproblem
13
QualitativedatacanalsohelpanswerquestionsaroundtheeffectivenessofpreviouseffortsTrytoobtainavarietyofperspectives--bothmainstreamandoutliers--fromnetworksbeyondyourowntoprovideyouwiththegreatestvantagepointforevaluatingpotentialimpactUnderstandinghowthoseeffortswerereceivedbydifferentpartsofyourcommunityandiftheremaybeelementsofprevioussystemstopreservewillbecrucialtothesuccessofanypolicychangeThisinformationmaybeobtainedthroughfocusgroupsinterviewsandsurveysFormoreinformationonhowtousethesequalitativeresearchtoolspleaseseetheseresourcesdevelopedbytheAgencyforHealthCareandResearchandUrbanInstitute
FinallytakealookatnationaldatatoseewhereelsethisproblemmightexistThisresearchmaybehelpfulinlaterstepswhenyouarelookingforpromisingpoliciesinothercities
Puttingitalltogether
UsethetemplatebelowdevelopedbyDenversPeakAcademytocaptureyourrefinedrightsizedproblemstatementandcurrentandfuturestatesWorkingthroughthisexercisewithpartnersishelpfultogainbuy-inTheresponseswillbeimportantwhengettingotherpartnersonboard
WhyChangeIsNeeded
Yourright-sizedproblem
CurrentState
Analysisoftrendsindata
FutureState
Ideallywhataretheoutcomesyouhopetoachieve
Checklist
Thekeystepstogatheringinformationarebelow
Atthispointfocusontheproblemratherthanjumpingtosolutions
Listentopeoplewhoareaskingforhelpandtoadvocateswhoareraisingissues
Beasspecificaspossibleinidentifyingtherootcauseandhoninginonthepartoftheproblemthatyourcitycanaddress
Identifytrendsbasedonmultiplesourcesofdataincludinghistoricaladministrativedataproxymeasuresqualitativedataandnationalmeasures
Gatherinformationabouttheproblem
14
Captureyourproblemstatementtheresultsofyouranalysisanddesiredoutcomes
Gatherinformationabouttheproblem
15
StrategicallyengagekeypartnersPolicytheoryresearchoftenfocusesonincreasingpolicymakersaccesstoevidenceasawaytoavoidtakingshortcutslikemakingdecisionsbasedonemotionImplementationsciencesuggeststhatincreasingaccesstoevidenceisnotenoughandthatpolicychangeshouldbeexaminedinthewidercontextofdebatecoalitionformingandpersuasionMeaningfulpolicychangerequiresldquoalong-termstrategytoformalliancesanddevelopknowledgeofpolicymakersandpolicymakingratherthanusingevidenceinpolicymakingonanadhocorone-offbasis
UnderstandingnetworksandsystemsisimportanttodevelopsustainablepolicychangeThissectionismeanttohelpreaderstakethestepstobuildengagementstrategiesforpartnersthatrepresentdifferentsystemsandnetworksEngagementshouldbeanongoingprocessofconnectingpeopletooneanotherandtoanissueEngagementshouldnotbeinnameonly-wehaveallexperiencedengagementactivitiesthatgivepeopleaspecificroletoplaythatleadstoaneatpredictableoutcomeAccordingtotheBuildWithmovementundervaluingandunderinvestinginengagementstrategiescanleadtoadisconnectofcertainindividualsorgroupsandmistrustofgovernmentWhendevelopingengagementactivitiesrememberthathowpeopleengageandwhethertheyengagehastodowithhowyoustructureopportunitiesforengagement
Conveneacoreadvisorygroup
AnadvisorygroupcomprisedofpeoplefrommultiplesectorscanbehelpfulasyouconsiderthelocalsocialenvironmentalandpoliticalcontextinwhichyouintendtointroducepolicychangeRevisityourstakeholdermapanddeterminewhowouldbebesttoserveonthisgroupTomakesureyouareconsideringtheproblemsandsolutionsthroughanequitylensincludepeoplewithlivedexperiencewhohaveexpertiseintheeffectivenessofsystemswhetherthosesystemsfunctionasintendedandhowtheymayimpacthumanbehaviorYoumayalsowishtoincludesomeonewithexperienceworkingwithdataandanalysis-thispersoncanhelpdeterminethetypeofdatathatmaybereadilyavailableandusefultocollectandcanalsohelpplanandshapeyouranalysistofigureoutwhichprogramsareeffectiveBesuretoinvitefront-lineworkersaswell-itscriticalthatwhateversolutionscomeoutofthisprocessaremeaningfultothosewhowillberesponsibleforimplementingthem
BringingpeopletogethercanbechallengingbutthereareafewwaystosetandmanageexpectationsFirstmakesurethatthegroupisawareoftheproblemstatementcurrentstateandfuturestate-thiscanhelpstarteveryoneoffonthesamepageNextspend
Strategicallyengagekeypartners
16
sometimegettingtoknoweachotherrsquosworktolearnabouteveryonesstrengthsThissharedunderstandingcanhelpasyouclarifyrolesandresponsibilitiesofeachpartnerinthegroup--thisisimportanttoavoidfinger-pointinginthefutureandtoensurethatnoneofthetasksslipthroughthecracksInsomecasestherearemanyactorswithdifferentresponsibilitiesasinthecaseofBaltimoreCitys38-memberYouthFundTaskForcewhichischairedbynonprofitandcorporatepartnersandincludesrepresentativesfromthenonprofitandfaithcommunitiesstudentsthebusinesssectorlocaluniversitiesCityCouncilandtheMayorsOffice
OncerolesandresponsibilitiesareclearyoumayconsiderdevelopingacharterguidelinesortermsofreferencetosetexpectationsforthegroupBesuretodistinguishthisfromotherboardsandcommissionsthatyourcitymayhaveNextyoumaywishtoestablisharegularongoingmeetingscheduleBesuretosendagendasandmeetingmaterialsaheadoftimeandclarifytheobjectiveofeachmeeting
InSeattletheCitydevelopeditsHousingAffordabilityandLivabilityAgenda(HALA)torespondtorisinghousingcostsandhomelessnessThisisatrendinotherwestcoastcitiesaswellwhichmaybeexplainedbyaninfluxofnewresidentswhoareattractedbytheemergenceoftechjobstheattractionoftheoutdoorlifestyleortherecentpassageofprogressivepolicieslikelegalizedgaymarriagea$15minimumwageandlegalizationofmarijuanaSeattlebroughttogetheramulti-stakeholderadvisorygroupcomprisedoflocalhousingexpertsrentersandhomeownersdevelopersandgovernmentofficialstoshareinformationabouttheirneedsanddeveloprecommendationsbasedontheseneedsTheinterestsandmotivationsofthisgroupwerevariedandittookabouttenmonthstofindcommongroundTheseriesofrecommendationsthattheadvisorygroupcreatedismeanttoserveresidentswithaspectrumofneeds
Buildaprocessforcommunityparticipation
Ofcoursenoteveryonecanbeincludedinanadvisorygroupandadvisorygroupmembersmaynotnecessarilyrepresentthevariedperspectivesoftheirpeers-forinstancenoteveryhomeownerhasthesamesetofneedsTogatherinputfromawiderswathofthepopulationitcanbehelpfultobuildaprocesstoinvitethepublicatlargetosharetheirideasCitiesdothisinavarietyofwaysfromonlineconversationsandsurveystoin-personmeetingsRememberthatbeingintentionalaboutgatheringtheviewpointsofpeopleismoreimportantthanusingthelatesttechnologyandtools
TheinsightgeneratedbycommunityparticipationexercisescanbeincludedinyourreviewofevidencewhichisdiscussedinasubsequentchapterInWestVirginiaastatethathasbeenhithardbytheopioidepidemicstatehealthofficialsdraftedanopioidresponseplanthatwasinformedbypublicengagementaswellaslocalandregionalsubjectmatter
Strategicallyengagekeypartners
17
expertsThepublicwasinvitedtopresentcommentsonideasfordecreasingoverdoseearlyinterventiontreatmentandrecoveryPublichealthandmedicalfacultyfromWestVirginiaUniversityMarshallUniversityandJohnsHopkinsUniversityprovidedresponsestothecommentsTheresultwasanOpioidResponsePlanthathealthcommissionerRahulGuptapresentedtoGovernorJimJustice
AsamplespectrumtoplanyourowncommunityparticipationprocesscreatedbyLaurenellenMcCannisbelow
SourceLaurenellenMcCann
Community-basedparticipatoryresearchisanotherwaytoworkwiththecommunitytoimproveoutcomesCBPRmeansthecommunityisinvolvedinallaspectsoftheresearchprocesswhichisofteniterativeInCBPRtraditionalresearchersandcommunitymembersshareknowledgeaswellasresourcesandcredit
CBPRInActionInSanFranciscoanenvironmentaljusticenonprofitteamedupwiththeCitysDepartmentofPublicHealthandanoutsideevaluatortounderstandtherootcausesofobesityandfoodinsecurityinlow-incomeneighborhoodsandidentifysolutionsThroughresearchmethodssuchasinterviewingresidentsandstoreownersdiagrammingstoreshelvesGISmappinganeconomicanalysisandreviewofsimilarpoliciesinnearbycitiestheteamobservedthatthestoresthatresidentshasveaccesstowerestockedwithlesshealthyfoodsandthatresidentswantedmorehealthyfoodandlessalcoholandtobaccoThisprojectresultedinlocalandstatelegislationtoencouragestorestostockfreshproduce
Strategicallyengagekeypartners
18
Developrelationshipswithsubjectmatterexperts
OnewaytokeepsubjectmatterexpertsengagedistoinvolvetheminthecommunityparticipationprocessasdescribedaboveIntheWestVirginiaexamplesubjectmatterexpertsrespondedtoandsynthesizedthepubliccommentsOtheropportunitiesaredescribedbelow
AdvisoryAcitydepartmentheadMayororstaffmembermaydevelopaone-on-oneadvisoryrelationshipwithuniversityfacultymemberoracollaborativebetweentheCityandUniversitytoaddressaparticularproblemAgreatplacetostartistolookforrelevantpracticecenterswithinyourlocaluniversityandreachingouttosubjectmatterexpertsInadditiontotheirexpertiseontherelevantpolicytopicheorshemayalsobeabletoidentifyotherexpertsandevenprovidesupportonhiringtherighttalenttoleadcertainprogramsAnadvisoryrelationshipwithauniversitysubjectmatterexpertcanalsohelpwhencitiesareconsideringbringinginnewprogramsTheseexpertsareoftenfamiliarwithprogramsthatarebasedonnationalmodelsandcanprovideinsightintoaprogramsoriginandlineageeffectivenessinotherplacesandthepotentialforimplementationinyourcityAcademicscanalsohelpidentifyindicatorsbasedongovernmentindicatorsorliteratureFinallyrelationshipsbetweenpolicymakersandresearchersprovidethemostbroadlyeffectiveroutetopullingresearchintopolicy
ResearchandevaluationAcademicexpertsmaysupportcitiesinevaluation-indeedevaluatingcityinitiativesisatraditionalmodelofcityuniversityrelationshipsSharingtheinformationyoucollectedintheGatherInformationphasecanhelpresearchersunderstandyourworkandgoalswhichinturnwillstrengthenthesupportthattheycanprovideDiscussingdetailsaboutyourinterventionandevaluationplanatthebeginningoftheprocessmeansthataresearchpartnercanhelpwiththeevaluationdesignanddatacollectionForinstancetheWilsonSheehanLabforEconomicOpportunitieswalkscitiesthroughconsideringthefollowingquestionsbeforepartneringonanevaluation
1 Isthereaclearpolicyprogrammodelwithobjectivescreeningcriteria
2 Isitpossibletoevaluatewhathappensbothwiththepolicyprogramandintheabsenceofthepolicyprogram
3 Arethereoneormoremeasurableoutcomesofinterest
4 Arethereenoughpeopleforrigorousquantitativeanalysis
5 Whatdatawillbenecessaryforthestudyandwhataretheprocessesforaccessingthesedatasource(s)
6 Isthereacompellingreasontoevaluate
Strategicallyengagekeypartners
19
InadditiontoassessingreadinesstoconductanevaluationresearcherscanhelpyouhoneinonthepurposeoftheevaluationTheWorldBankdescribesseveralpotentialusesofevaluationwhicharedefinedhereInbriefevaluationcanhelpdecisionmakers
makeresourceallocationdecisionsrethinkrootcausesofproblemsidentifynewproblemsdecideonabestalternativedeterminewhetherprogramsorpoliciesarebeingimplementedwithfidelitydeterminewhetherprogramsorpoliciesarehavingthedesiredimpact
Researcherscanalsoprovidesupportinsharingtheresultswiththescientificcommunitythroughpapersandconferenceswhichisvaluableincreatingapeernetworkofevidence-basedcitypractitioners
TrainingandtechnicalassistanceUniversitypartnersespeciallythoseatpracticecenterscanbehelpfulindevelopingorupdatingtrainingcurriculumtrainingthetraineranddeliveringtrainingtocitystaff
Strategicallyengagekeypartners
20
ReviewtheevidenceandcollectdetailsOnceyouhaveathoroughunderstandingoftheproblemdataandactorsyouwillneedtoconductyourownresearchonavailableevidenceandlearnaboutthelandscapeofpolicyoptionsYourgoalwillbetodevelopanaccessiblelistofinformationaboutpoliciesthathavehelpedsolvesimilarproblemsinotherplacesThischapterwillguideyouthroughthestepsofpreparingyourreviewersunderstandinglevelsofevidenceandcollectingthedetailsonevidencethatyouwillneedfordiscussionwithdecisionmakersinyourcity
Selectandprepareyourreviewers
AssignapersonorteamtoreviewrelevantinterventionsthathavedemonstratedsuccessinotherjurisdictionsTheirgoalwillbetoproduceareviewofalloftherelevantprogramspoliciesorinitiativesthatyourteamcouldconsiderimplementingTheywillbeconductingaliteraturereviewsupplementedbyinterviewswithexpertsandpeerswhohaveinformationaboutpotentialprograms
SincethepeopleyouassigntothistaskareprovidingthebaselineofpotentialoptionstoimplementitisimportantthatyouchoosepeopleyouareconfidentwillbethoughtfulandthoroughintheirworkItisalsocriticalthattheyareawareoftheimportanceofsourcecredibilityEnsureyourreviewershavefamiliarizedthemselveswithrulesforcriticallyanalyzinginformationsourcesbeforetheygetstartedIftheyareincludingnewsarticlesaspartoftheirsearchtheyshouldalsobeawareofmethodsforevaluatingnewscredibility
AskyourreviewerstodevelopaplanandtimelinetocompletethereviewThisstartswithsolidifyingagoverningresearchquestionOnewaytoknowyouarecreatingadetailedresearchquestionistousetheacronymPICOremindingyoutoincludereferencetothepopulationintervention(ifyouhaveoneinmind)comparisonandoutcomeelementsinyourresearchquestionForexampleifyouknowthatyouhaveapopulationofteenagerswhoarehomelessyouhavelearnedfromyourstakeholdersthatmanyoftheseteenagersidentifyasLGBTandyouhaveheardsomethingabouttheapproachofcreatingdrop-incentersfordeliveringsupportyoucouldask
ForLGBTyouthfacinghomelessness(population)howeffectivearedrop-incenters(intervention)relativetostandardsocialservices(comparison)atsettlingtheminpermanenthousing(outcome)
Ifyoudonothaveaninterventioninmindyoucanleaveouttheinterventionandcomparisonandsimplyask
Reviewtheevidenceandcollectdetails
21
HowcanwehelpLGBTyouthfacinghomelessness(population)obtainpermanenthousing(outcome)
OnceyourreviewershavetheirresearchquestionsusethequestionstocreatealistofkeywordstheycanuseintheirliteraturesearchThiswillhelpkeeptheircollectionofarticlesreasonablyfocusedHavethemdescribewhattheirsearchprocesswillbeincludinglistingthedatabasestheywillreviewandtheirplanforstoringtheinformationtheywillcollect(Herearesomeadditionalsuggestionsforstructuringareviewplan)
WhatyourreviewerswillbedoingisalightweightversionofarapidreviewTotheextentthattheywouldliketodevelopamethodologyforspecificsearchprotocolsincludingcriticalappraisalprocessestheycancertainlygoaheadanddoso
Usedifferentlevelsofevidenceforthemostcompletepicture
AsdiscussedintheWorkingwitharangeofevidencechaptertheNestaStandardsofEvidenceframework--reproducedbelow--ishelpfultounderstandthelevelofconfidencethataprogramorinterventionishavingapositiveimpactThisapproachletsyouputinterventionsthatyouhaveheardaboutlocallybutwhichhavenotbeenstudiedtogetherwiththosewithademonstratedcausaleffectThisishelpfulsinceanumberofpoliciesprogramsandpracticesthatyouwillwanttoconsidermaynotyethavemuchofaresearchrecord
Reviewtheevidenceandcollectdetails
22
NearlyanypieceofinformationaboutaninterventioncanbeclassifiedwithinthestandardsofevidenceForexampleifsomeonehasacrediblehypothesisabouthowaninterventioncouldproducebenefitthistypeofevidencewouldbeclassifiedasLevel1ThistypeofevidencecouldincludeinterventionsinothercitiesthatyoureadaboutinthenewsorhearaboutataconferenceForinstancewhenUSDAallowedschoolsandschooldistrictstoprovidefreebreakfastandlunchtoallstudentsthroughtheCommunityEligibilityProvision(CEP)manydistrictsimplementedthisoptionevenbeforetherewasevidenceofeffectivenessThiswouldfitintoLevel1sinceitbuildsonthelogicthatprovidingaccesstomealswilldecreasefoodinsecurity
YoumayalsofindpiecesofinformationthatqualifyasLevel2evidenceForinstanceasmoreschooldistrictsimplementedCEPprincipalstestifiedaboutpositivechangestheyobservedinstudentssuchasamorepositivedemeanorthatcanleadtobetteracademicperformanceDataalsoshowedthatthestudentsdescribedwerereceivingmoremealsthanbeforetheprogramsimplementationHoweveritwastoosoontobuildevidencetoshowacausalrelationshipbetweentheincreaseinfreemealsandincreasedacademicperformance
ldquoLower-levelrdquoevidencecanconsistofnuancedbutvaluableelementsthatrelatetoaninterventionrsquospotentialfitwithacommunityQualitativeinformationsuchastestimonyfrompeoplewhohaveexperiencedaninterventionandthosewhohavefacilitatedorimplementeditwillalwaysbeusefultoconsiderandprovidesdetailedlocalcontext
Experimentalstudies--describedstartingatLevel3withcontrolledexperimentsandgoingthroughLevel5withfullyfleshedoutprotocolsandprocessesforreplication--stilloccupythehigherlevelsofthisdescriptionofevidenceThisprioritizationshowsthevalueofrigorousstudyforevaluatingtheimpactofaparticularpolicyorprogramratherthanotherfactorsthatcouldbeinfluencingoutcomesofinterestRememberthatmultipletypesofevidencecancomplementrigorousimpactevaluationsandgivecityleadersadditionalinsightsandbroaderperspective
Accessinformation
Encourageyourreviewerstobegintheirsearchforpromisingpracticesingovernmentandnon-profitclearinghousesandinacademicpolicyjournalsWorkpublishedinthesesourceshasalreadybeensubjectedtoahighlevelofprofessionalscrutinyandgenerallycontainsagreatdealofdetailwhichmeansthatyourreviewersaremorelikelytobeabletopullouttheinformationyouneedtomakeagooddecision
Clearinghouses
Reviewtheevidenceandcollectdetails
23
Herearesomeoftheclearinghouses(collectionsofprogramevaluations)forevidence-basedpoliciesandprograms
ResultsFirstClearinghouseDatabaseThePewCharitableTrustsCollectstogethereightotherclearinghousestoprovideinformationoninterventionsinthefollowingcategoriesadultcriminaljusticeeducationchildwelfarehealthjuvenilejusticementalhealthsubstanceabuseandyouthdevelopmentIndividualclearinghousesincludedinResultsFirstare
BlueprintsforHealthyYouthDevelopment-foryouthdevelopmentandviolenceprevention
CaliforniaEvidence-BasedClearinghouse-forchildwelfareinterventions
TheLauraandJohnArnoldFoundationrsquosSocialProgramsThatWork-forsocialservicesprograms
TheUSDepartmentofEducationrsquosWhatWorksClearinghouse-foreducationinterventions
TheUSDepartmentofHealthandHumanServicesrsquoResearch-TestedInterventionPrograms(RTIPs)-forcancercontrolinterventions
TheUSDepartmentofHealthandHumanServicesrsquoTeenPregnancyPreventionEvidenceReview(TPPEvidenceReview)-forsexualriskbehaviorandsexualhealthoutcomes
TheUSDepartmentofJusticersquosCrimeSolutionsgov-forcriminaljusticejuvenilejusticeandcrimevictimservices
TheUSSubstanceAbuseandMentalHealthServicesAdministrationrsquosNationalRegistryofEvidence-basedProgramsandPractices(NREPP)-substanceuseandmentalhealthinterventions
TheUniversityofWisconsinPopulationHealthInstituteandRobertWoodJohnsonFoundationrsquosCountyHealthRankingsandRoadmapsWhatWorksforHealth(WhatWorksforHealth)-healthbehaviorsclinicalcaresocialandeconomicfactorsandthephysicalenvironment
UKWhatWorksCentresTheBritishgovernmenthascharteredanetworkofsubject-specificorganizationswhichcollectsynthesizeandreportoutevidenceontheeffectivenessofpolicyprogramsandpractices
NationalInstituteforHealthandCareExcellence(NICE)-healthandsocialcare
Reviewtheevidenceandcollectdetails
24
SuttonTrustEducationalEndowmentFoundation-educationalachievement
CollegeofPolicingWhatWorksCentreforCrimeReduction-crimereduction
EarlyInterventionFoundation-earlyintervention
WhatWorksCentreforLocalEconomicGrowth-localeconomicgrowth
CentreforAgeingBetter-qualityoflifeforolderpeople
WhatWorksCentreforWellbeing-wellbeing
WilsonSheehanLabforEconomicOpportunities(LEO)UniversityofNotreDameProvideslinkstotheirevaluationsofinterventionsincriminaljusticeself-sufficiencyeducationhealthandhousingandhomelessness
ClearinghouseforLaborEvaluationandResearchUSDepartmentofLaborProvidesresearchonworkforceandlabor-relatedpoliciesandprograms
J-PALEvaluationsAbdulLatifJameelPovertyActionLabStudiesonpoverty-reducinginterventionsfromaroundtheworld
CampbellSystematicReviewsCampbellCollaborationSystematicreviewofpolicyinterventionsbyscholarlycoordinatinggroupsinsevenissueareascrimeandjusticeeducationinternationaldevelopmentmethodssocialwelfareknowledgetranslationandimplementationanddisability
Benefit-CostResultsWashingtonStateInstituteforPublicPolicyAWashingtonState-charteredprogramtoestablishthelikelihoodofrelativecostsandbenefitsforprogramsacrossthefollowingareasjuvenilejusticeadultcriminaljusticechildwelfarepre-Kto12educationchildrensmentalhealthhealthcaresubstanceusedisordersadultmentalhealthpublichealthandPreventionworkforcedevelopmentandhighereducation
EffectiveInterventionsCentersforDiseaseControlandPreventionRepositoryforevidence-informedHIVpreventioncareinterventions(withstepsonhowtoimplement)
Journalsandindexes
OnceyouhavereviewedrelevantinformationaccessiblethroughclearinghousesitisworthtakingalookatindividualstudiespublishedasscholarlyarticlesYourreviewerswilldothisbysearchingarticleindexesandacademicjournalsUnfortunatelyanumberofusefulacademicjournalsandindexingplatformsmaynotbegenerallyaccessibletoyourreviewersifyoudonothaveasubscriptiontothemToaddressthisconsiderpartneringwithalocal
Reviewtheevidenceandcollectdetails
25
collegeoruniversitytoprovideyouwithalistofarticlesfromthefollowingsources(Thisisalsoanexcellentopportunitytoseekresearchsupportfromanyacademicstakeholdersyouidentifiedpreviously)
Indexes
GoogleScholarincludesmostpeer-reviewedonlineacademicjournalsandbooksconferencepapersthesesanddissertationspreprintsabstractstechnicalreportsandotherscholarlyliteratureincludingcourtopinionsandpatentsItallowsyoutospecifyapublicationrangeinyoursearchandprovideslinkstolikelysourcesforalllistedarticles
PolicyFileIndexisauniqueresourceforUSpublicpolicyresearchinthatitgrantsusersaccesstotimelyupdatedinformationfromover350publicpolicythinktanksnongovernmentalorganizationsresearchinstitutesuniversitycentersadvocacygroupsandotherentities
PolicyArchiveworkstocollectanddisseminatesummariesandfulltextsvideosreportsbriefsandmultimediamaterialofthinktankuniversitygovernmentandfoundation-fundedpolicyresearch
Anothersourceforhigh-qualityresearchisthereportsoftheCongressionalResearchServiceYoucanfindtheirreportsaggregatedhere
Specificjournals
Oftenthemostusefulacademicjournalsforunearthingpolicyorprogramreviewswillbeineachspecificsubjectarea--thatisineducationjournalsforeducationpoliciesYoucanuseageneralindexlikeGoogleScholartobegintolearnwhichjournalsaremostoftenusefulforyoursearchHoweveryoucanalsosearchthefollowingtwogeneralistpolicypublicationsforissuesandarticlesrelatedtoyourtopic
PublicAdministrationReviewJournalofPolicyAnalysisandManagement
Newssourcessubjectmatterexpertsandinterviews
AlthoughitisvaluabletoknowwhathasbeenexploredthroughascholarlyresearchframeworkitisalsousefultolearnwhatkindofinterventionsarebeingtriedinthefieldrightnowandhowpeoplefeelaboutthesetrialsSearchsubscriptionnewsaggregatorslikeLexisNexisorfreenewsaggregatorslikeGoogleNewsorYahooNewsforyourissueareaplusthewordpolicyorprogramtoseewhatkindsofinterventionsarebeingcoveredinthenews
Reviewtheevidenceandcollectdetails
26
DonotforgettocontacttheadvisorygroupmemberswhowerechosenfortheirsubjectmatterexpertiseInadditiontoprovidinggoodmaterialthroughinterviewsthemselvestheymayhaveideasaboutothersubjectmatterexpertsyoushouldcontacttoaskaboutpotentialinterventions
Peercitiesandnetworks
InadditiontoreadingabouttheworkofothercitiesitisidealifyouareabletospeakwithpeoplewhoareinvolvedwithcurrentimplementationsofinterestinginterventionsInyourcityyoulikelyhaverelationshipswithyourneighborsbutitcanbehelpfultolookoutsideofyourregionforsolutionsaswellTofindcounterpartsinothercitieswhoareworkingonaprogramyouwouldliketolearnmoreaboutyoucancontactstaffinintermediaryorganizations(likeGovExoryourprofessionalassociations)whomightbeinapositiontoputyouintouchimmediatelyorwhomightseeanothergoodrouteforsettingupaconversationHereisalistofnetworksthatmaybehelpful
InternationalCityCountyManagementAssociation(ICMA)NationalAssociationofCounties(NACO)NationalLeagueofCitiesUSConferenceofMayorsWhatWorksCities
Collectthedetails
Onceyourreviewershavefoundevidenceonpossiblepoliciesorprogramstoimplementtheyshouldcapturekeydetailsthatwillhelpdecisionmakersunderstandthepotentialeffectivenessandapplicabilityofeachpotentialintervention
Foreachsourcethathasprovidedinformationyouwanttoconveyusethetemplatebelowtocaptureasmanyofthefollowingdetailsaspossible(Youdonotneedtocompletethisforsourcesthatprovideduplicativeinformationorareotherwisenotthatuseful)
Programorpolicyname (egldquoHousingFirstrdquoorldquoblightedpropertiesregistryrdquo)
Sourcetype (egjournalarticleinterviewsubjectmatterexpertstudyassociationcommunityparticipationprocess)
Sourcedetails (egarticletitleandpublicationnameandtitleofinterviewee)
Evaluationmethodology
(egrandomizedevaluationcomparativeevaluationoftwodifferentgroupsbeforeafteranalysis)
Evidencelevel (egLevel12345)
Programorpolicy
Reviewtheevidenceandcollectdetails
27
description
Programorpolicygoal (1sentencecapturingtheprincipalaimoftheintervention)
Existingimplementations
(WherehastheinterventionbeentriedListasmanysitesasyouareawareof)
Characteristicsofimplementationsites
(ForeachimplementationsiteofferbasicgeographicIDmdashcityandstateThenyouwillwanttocollectthedetailswhicharerelevanttoyourdecisionmakersrsquoconsiderationsaboutwhatmakesalocationldquosimilartordquoorldquodifferentfromrdquoyoursituationYouwillatleastwanttoincludeforeachimplementationsitePopulationnumberPopulationdensity(ruralurbansuburban)Socio-economiccharacteristicsBeforestartingaskyourdecisionmakersaboutanyparticularcharacteristicsofprogramimplementationsitestheywouldwanttoknowbeforeseekingtoreplicate)
Characteristicsofimplementationpartners
(Anyavailabledetailsaboutorganizationsinstitutionsorpeoplewhowereinvolvedwithimplementingtheintervention-egindependentchildcarecenterpublicelementaryschoolcitypublichealthnurses)
Characteristicsoftargetpopulationeligibleprogramparticipants
(Anyavailabledetailsaboutpeoplewhoarethebeneficiariesoftheprogramorotherwiseincludedoraffectedbytheintervention-egnumberofprogramparticipantscategoriesofpropertyownersaffected)
Policyorprogramduration
(HowlonghastheinterventionbeeninplaceineachimplementationsiteHasitendedandifsowhy)
Whatbenefitscamefromeachimplementation
(Anyobservedpositiveeffectsoftheinterventionincludingeffectsize)
Statisticalestimatesofinterventioneffectiveness
(Anyavailablestatisticalestimatesoftheinterventionrsquoseffects)
Whatdrawbackswereobserved
(Anyavailabledetailsaboutunintendedorunavoidablenegativeeffectsoftheintervention)
Programorpolicycosts
(Anyavailabledetailsaboutprogramcostsperimplementation)
Reviewtheevidenceandcollectdetails
28
Reviewtheevidenceandcollectdetails
29
DecideonapolicytoimplementbasedonlocalfactorsTogetthecriticalbuy-infromleadershiptoimplementanewpolicyorprogramtheprojectleadshouldscheduleameetingwithdecisionmakerstopresenttheevidencesurroundingthepolicyoptionscapturedinthetemplateDependingonyourinsightintoyourcityleadershipswillingnesstomakechangeyoumaywishtosharetheresearchyouhaveconductedontherootcausesoftheproblemandsomeofyouranalysisontheimpactofcurrenteffortstodate
Considertheapplicabilityofanapproach
AsdiscussedinthepreviouschapterpolicymakersoftenlookforevidenceofeffectiveprogramsinothercitiesbutoftenthequestionarisesofhowgeneralizableaprogramisfromonecitytoanotherTheAbdulLatifJameelPovertyActionLab(J-PAL)suggeststhatinreviewingevidencefromanotherlocationitishelpfultoidentifythespecificmechanismthatwasused(deliveryofprogramorserviceinformationsharingetc)identifythelocalconditionsthatwereinplaceobservethebehaviorchangethatresultedfromthemechanismandfinallytounderstandlocalcontextandconditions(legalorcostconstraintsculturalnormsetc)Thissetofpresentationelementscanbedrawnfromthelistofdetailscollectedintheresearchtemplate
EducationpolicymakersinChicagoILusedfindingsfromprogramsinKenyaandIndiainwhichchildrenwhohadfallenbehindtheirpeersweregiveneducationalsupporttargetedtotheirleveloflearningThemechanismusedwastargetedteachingandthereweresimilarconditionspresentinallthreecountriesAlthoughsomestudentshadfallenbehindinthecurriculumteacherswereincentivizedtoteachgrade-levelmaterialsConsiderationsofhowthelocalcontextinChicagomightaffectimplementationofatargetedteachingprogramincludedunderstandinghowtheneighborhoodsthatthestudentswerefrommayhaveaffectedtheireducationaloutcomesandthehighcostofteachinginsmallgroupsInChicagoMatchEducationprovidedtwo-on-onetutoringwhichmatchedstudentswithwell-educatedindividualswhohavenothadformalteachertrainingforalower-costmodelTheresultswerepositiveStudentswhoweretutoredperformedbetterinmathandweremorelikelytopasshighschoolclasses
ThegeneralizabilityframeworkproposedbyJ-PALisbelowUsethisasatemplateasyouconsiderpotentialpolicysolutions
Decideonapolicytoimplementbasedonlocalfactors
30
SourceTheGeneralizabilityPuzzle
OnceyouhavenarrowedthepolicyoptionsdiscussthequestionsbelowforeachoptionsyouareconsideringThisstepiscriticalasunderstandinglocalcontextissoimportantinsuccessfulpolicyimplementation
CommunityContextQuestionnaire
Localcontext
DoesthisoptionmakesenseinourcommunityIsthisaonesizefitsallapproachIsanyparticulargroupabsentfromthisapproachIfsoisthatok
Youranswers
Effectiveness
WhatwouldweexpecttoobserveifthepilotisaseffectivehereasinCityX
Youranswers
Timing
DoesthetimingmakesenseIsthistherighttimetoimplementthispolicychange
Decideonapolicytoimplementbasedonlocalfactors
31
WherearethepotentialroadblocksthatcouldaffectthescheduleIsthereanywaytomitigatethisHowlongwouldittaketoseeanimpact
Youranswers
Implementation
DowehavethestaffcapacitytoimplementthisArethereadditionalpartnersthatyoucanbringtothetabletosupportimplementationDoesthismakesensegivenourformofgovernment
Youranswers
Authority
DoesourcityhavetheauthoritytoenactthisIfsowhatistheappropriatemechanismAretherelegalbarriersArethereanyexistingpreemptionissuesfromthestateorfederalgovernmentCanthosebeaddressedthroughlegislationadvocacyIstherepoliticalsupportoroppositiontothis
Youranswers
Cost
HowmuchwillthiscostannuallyHowwillwepayforitCanthepolicybeimplementedwithexistingfunding
IfsoisthisfundingreliableHowwouldthispolicyoptionaffectourcitysbudgetintheshortandlongtermWhatotherfundingoptionscanweconsiderIsthereanalternativethatwouldcostlessandgiveussimilarresultsWillanyconstituencybeadverselyaffectediffundingforprogramsthatimpactthemisshiftedtoanotherprogrampolicyorinitiative
Youranswers
Decideonapolicytoimplementbasedonlocalfactors
32
EstablishaframeworktomonitorprogressNowthatyouhavereviewedtheevidenceconsideredlocalcontextandchosenyourpolicyitistimetodeterminehowyourcitywillmonitorprogressofthisnewapproachCitystaffandleadersneedtoestablishtheinfrastructureforongoingcontinuousdecisionmakingbasedonwhetherthepolicychangeisleadingtothedesiredoutcomesforresidentsandwhetherthechangeismeaningfulandrationaltofrontlineworkersSettingarbitrarygoalsatthelocallevelthatconflictwithgoalsatotherlevelsisconfusingtotheindividualsresponsibleforachievingthegoalsGoalsthataresettoohightoachievecanbedemoralizingtostaff
ThissectionincludesstepstosetupavenueinwhichCitystaffandleadershiproutinelyvisitkeyindicatorstodeterminewhatisorisnotworkingandtomakedecisionsinreal-timeFormorereadingonperformancemanagementpleasevisittheGovExPerformanceManagementGettingStartedGuide
InperformancemanagementtermssuchasindicatormeasureandmetricmaybeusedinterchangeablyInthisguidewewilluseindicatorInyourworkthemostimportantthingistobeconsistentinwhichevertermyouuseThisvideomaybehelpfulinunderstandingthisconceptandcommunicatingittoothers
Establishthefundamentalsandgatheryourteam
FirstdeterminewhetherthereisanestablishedvenueorregularmeetingscheduleduringwhichyoucandiscussprogressIfyourcityhasanexistingperformancemanagementstructureonarelatedtopicaddingadiscussionoftheprogressofpolicychangetoyouragendacanbeagreatideaIfthiswillbeastandalonemeetingdecidehowfrequentlytheprojectteamanddecisionmakerswillmeetandputthosemeetingsonthecalendarfortheentirelengthoftheproject
DeterminewhowillstaffandattendroutinemeetingsonCityprogressThesemeetingsmayincludetheMayorandorCityManagerdepartmentheadstaffwhoareimplementingthepolicychangeandanalysts
Identifyindicators
ItiscriticalthatindicatorsbeanimportantcityandcommunitypriorityaswellassomethingtheCityplaysamajorroleinaddressingTheremaybeinstancesinwhichtheCitywillneedtoreachouttoorcollaboratewithexternalpartnerstogetamorecompletepictureofprogressontheoverallpolicychange
Establishaframeworktomonitorprogress
33
Anoutcomeindicatordescribesthedesiredeffectofthetargetedpopulation
ExampleBellevuesfamilyfriendlyneighborhoodsindicatormeasuresthepercentofresidentswhobelievetheyliveinaneighborhoodthatsupportsfamiliesparticularlythosewithchildren
WithyourkeypartnersrevisityourproblemstatementcurrentstateandfuturestateexerciseandbegintoidentifytheoutcomeindicatorsthatwillillustratesuccessFornowchooseasingletop-levelkeyperformanceindicator(KPI)ItiscriticalthattheKPI-thepieceyouwillmeasure-beapriorityforthecommunityandsomethingtheCityplaysamajorroleinaddressing
Outputindicatorsmeasuretheprocessofprogramorservicedelivery
ExampleSanDiegoMayorKevinFaulconersgoaltorepair1000milesofstreetby2020
ReviewingtheadministrativedatasetinventorybegintoparedowntherelevantprogramsanddatasetsthatsupportyourKPIWhatarethemeasuresofeffectivenessforeachprogramTheseareyouroutputindicators
Thediagrambelowillustratestherelationshipbetweenoutputandoutcomeindicators
ThinkaboutimplementationindicatorsasmeasuringstepsintheprogramsrolloutthatarenecessarytoobtaindesiredchangesImplementationoutcomescanfocusontheinterventionstaffdeliveringtheinterventionanorganizationorsystemandcanaddressfeasibilityadaptabilitysustainabilityandcostTwokeyquestionstoconsiderwhendevelopingimplementationindicatorsare
1 Istheinterventionacceptableandappropriatetothecommunityand
2 IsitfeasibletoimplementinterventionXatthistimewiththeseresources
Establishaframeworktomonitorprogress
34
Amoreextensivelistwithdefinitionsofpotentialimplementationoutcomesislinkedhere
UsethetablebelowtochoosethedesiredimplementationoutcomesforyourpilotandtoplanyourmeasurementtechniqueThisshouldincludelevelofanalysis(individualproviderorconsumerorganizationorsetting)yourmeasurementtechnique(surveysobservationinterviewfocusgroupsadministrativedataanalysis)andthephasewhenyouwillcollectthisinformation(earlymidlate)Theremaybemorethanonelevelofanalysismeasurementtechniqueandphaseofdatacollection
Implementationoutcome
Definitionofoutcome
Levelofanalysisandmeasurement
Yourmeasurementtechnique
Phaseofpilotwhenyouwillcollectdata
Usethespacebelowtocaptureyourindicatorwishlist
Indicatorwishlist
Outcomes_____________________________________________________________________________
Outputs________________________________________________________________________________
Implementationoutcomes______________________________________________________________
Setapublichigh-levelgoal
Thehigh-levelgoaliscriticalinestablishingdirectiontogalvanizethepeoplewhowillbedoingtheworkWhendiscussingthegoalusetheMayorsbullypulpitLookforkeymilestonesinCitygovernmenttosharethegoalstatementsuchasthebeginningofaMayoraladministrationaStateoftheCityspeechorotherpublicremarks
GoalsshouldbeldquoSMARTrdquo
Specificaspossible(innamingthepopulationwhowillbebetterofftheactionthecitywilltake)
Measurable
Achievable
Establishaframeworktomonitorprogress
35
Results-focused
Time-bound
ThegoalstatementshouldincludeyouroutcomeindicatorlikeChattanoogasgoalldquoWewillendchronicveteranhomelessnessbyDecember2016rdquoAdiagramtohepyoubuildaSMARTgoalaroundyouroutcomeindicatorisbelow
Component Description Yourturn
Specific WhowillbebetteroffWhatactionthecitywilltake
Measurable Howwillyoumeasuresuccess
Achievable Isthegoalattainable
Results-focused Whatistheexpectedresult
Time-bound When
Setrealisticambitioustargetsandtimelinesforeachindicator
ToestablisharealisticandambitioustargetforyourgoalstatementusetheadministrativedatacollectedforyourchosenKPItoreviewactualCityperformanceinthemostrecentyearcomparedwiththethreeorfourmostrecentyearsworthofdataBegintocalculatesummarystatistics(egtotalminimumaveragemedianmaximum)toidentifytrendsCreatecumulativeandmonthlychartstoidentifytrendsandanomaliesBasedonyouranalysisofhistoricaldataignoretheanomaliesandassumetheproductivitytrendswillfollowthesameoverallpatternfrommonth-to-monthFinallyapplyanincreasedproductivityfactortotheperformanceeachmonthtoforecastfutureprogressBasedonthisanalysischooseatargetthatinspiresimprovementwhileaccountingforreality
Manycitiesthataremorematureintheirperformancemanagementpracticesrecognizetheimportanceoftakingacommunity-levelapproachtoimprovingoutcomesforresidentsThesecitiesarecreatinggoaltaxonomiesthatincludemeasuresrelatedtopublicsafetyhousingandevenrecreationCitiesdriveprogressonthesegoalsbyimplementingstrategiesthatengagemultipledepartmentsandcollaboratewithserviceprovidersinthecommunity
EventuallyyouwilldevelopgoalstatementsfortheoutputsinordertocapturetheefforttoachievetheoutcomecontainedinyourgoalstatementYoursetofindicatorsshouldbecomposedofshort-andinterim-termsupportivemeasureswithaccuratetargetsandrelevanttimelinesAnexamplegoaltaxonomyfromtheCityofLasVegassResultsVegasprogramisbelowUsethistaxonomyasamodelforyourownsuiteofindicatorsandgoalstatements
Establishaframeworktomonitorprogress
36
Yourturn
Thetemplatebelowismeanttohelpyoucraftagoalstatementforeachsupportingmeasure
Goalstatement
Departments
Outcome
Activities
KPI
Target
Timeframe
Conductanddiscussrelevantanalysis
ThrougharoutineseriesofmeetingscontinuallymonitorprogressofKPIssupportingindicatorsandanyotherdatacollectedsuchasqualitativedatatoidentifytrendsthatwillindicatewhetherthechosenpolicyishavingthedesiredimpactThesedatapointsshoulddrivethediscussionduringthesemeetingspointingtoopportunitiesforfurtherinquiryinto
Establishaframeworktomonitorprogress
37
whatisdrivingtheobservedtrendsAnalystsshouldbeencouragedandtrainedtoaskquestionsofthedepartmentsproducingthedataandconductsitevisitswhenpossibletodeepentheirunderstandingofthedataPresentationoftheanalysisshouldfocusononeideaatatimetokeeptheconversationfocusedonkeypointsandspecificopportunitiesforimprovementYoumayconsiderdevelopingapublicfacingdashboardtosharethisprogressmorebroadly
AgeneralcomparisonofthestatusquodatawithwhatwasobservedduringthepilotcanprovidedecisionmakerswithaquickunderstandingofthedifferencesbetweentheoldandnewapproachesalthoughthiswillnotshowcausalityOftenthesecomparisonswillincludeoperationalmetricssuchasdaysordollarssavedbutrememberthattheimpactonpeoplewillbemostimportantIfyouareworkingwithanevaluationpartnertheywillsupportthisstep
Usethespacebelowtosummarizetheresultsofyouranalysis
MeasureTargetvalue(seesectiononsettingtargetsabove)
Pilotvalue(whatdidyouobserveduringthepilot)
Historicalaverage(ifavailable)
Results(WhoisbetteroffIsanyonenegativelyimpacted1-2sentencesreassessingyourgoalstatement)
DocumentyourprocessandprogressincludingfeedbackfromtheparticipantsimplementationpartnersandtheprojectteamDidanythingunexpectedoccurduringthepilotArethereinternalandexternalfactorsthatmayaffectyouroutcomesThisiswheretheimplementationoutcomescanbecanhelpfulIftheinterventiondidnotgoasplannedconsiderwhetheritisbecausetheinterventionwasnotacceptableorfeasible
UsethetemplatebelowcreatedbyDenversPeakAcademytocapturethisinformationIfthe306090daytimelinedoesnotmakesenseforyourcityfeelfreetocustomizeit
CurrentState
FutureState
30Day
60Day
90day
Whatwentwell(+)
Whatdidnotgowellshouldbechanged(-)
IfaquestionordiscussionpointarisesthatcannotbeansweredinthemeetinganassignedstaffmembershouldcapturethequestionduringthemeetingAfterwardthefollowupitemsshouldbesenttorelevantpartnersandthenextmeetingshouldbeginwithadiscussionof
Establishaframeworktomonitorprogress
38
thoseitems
Takeaction
DiscussionsaboutwhetheryourindicatorsaremovingintherightdirectionandwhatmightbedrivingtrendsmayleadtoadiscoverythattherearenotenoughresourcesallocatedtothispolicyorissueTheremaybelimitedfundingforstaffprogrammingortechnologythatisdrivingthedatainthewrongdirectionIfCityleadershipisintheroomthisisagreattimetomakethecaseforadditionalfundingduringthebudgetseasonBycollectinganalyzinganddiscussingdataonthispolicyroutinelyyouarebuildingevidencetomakethecasefortherightleveloffundingtoreachtheestablishedtargets
Establishaframeworktomonitorprogress
39
Assessdataaccessibilityqualityandsecurity
Determineaccessibilityandqualityofdata
OnceyouhavecompiledyourwishlistofindicatorsitistimetodeterminetheaccessibilityandqualityofthedataTopreventcomplicationyoumaywishtoprioritizerelevantindicatorsforwhichyouarealreadycollectingdataAvoidthetemptationofadoptingatechnologysolutiontocollectnewdataatthispointsuchasaphotosurveytoolorotherappFornowyouaretryingtodeterminewhichdatasetsarerelevantaccessibleandhighqualityBesuretoworkwithyourkeyinternalpartnerswhileyouareidentifyingyourmeasuresDepartmentstaffwhoworkonprogramsarefamiliarwiththeindicatorsthatwillaccuratelysupporttheoutcomesanddatabaseadministratorswillbevaluablewhendiscussingaccessibilityandqualityofthedata
Abrieflistofconsiderationsfortheaccessibilityofyourdataisbelow
ArewecurrentlycollectingthisdatasetorwillwebecollectingitforthefirsttimeHowfarbackdoestherecordgo(timehorizon)WouldweneednewtechnologytocollectoranalyzethedataCanthedatabecollectedmanuallyorcanitbeautomatedHowfrequentlyisthedataproduced(continuouslyhourlydailyweeklymonthlyquarterlyyearlyetc)IsthedataavailablefortheentireCitypopulationorasubsetofthepopulationHowgranularisthedata(individuallevelCensusblockneighborhoodcitywideetc)DoesthedatacontainsensitiveinformationIsthereaproxymeasurethatwecanusefornowIsthedatamachinereadable
AsnotedbytheWilsonSheehanLabforEconomicOpportunities(LEO)attheUniversityofNotreDameleveragingadministrativedatacansignificantlyreducethecostsandtimelinesofrigorousevaluationCitiescanestablishaninternaldatasharingagreementacrossdepartmentsincludinglanguagethatenablesadministrativedatatobeusedforresearchpurposesCitiescanalsoidentifyexternaladministrativedata-holderssuchasacountyschooldistrictpolicedepartmenthospitaloranotherstatethatmayhaveoutcomedataimportantformeasuringprogramimpactCitiescanleverageexistingrelationshipsandorworkwithresearchinstitutionstodevelopstreamlineddatasharingprocessesandagreementswiththeseexternalentities
Assessdataaccessibilityqualityandsecurity
40
Evaluatethequalityofyourdata
QualityisasimportantasaccessibilityToevaluatedataqualityyoumaywishtocreatearubricliketheonebelow
Value ValueDescription
3 ExcellentDataQualityThedataiscompleteaccurateandupdatedregularly
2 AdequateDataQualityThedataissomewhatcompleteaccurateandupdatedfairlyregularly
1 PoorDataQualityThedataisnotcompleteaccurateorupdatedregularly
ThisisagreattimetobuildonthedatainventorythatyoustartedinthegatherinformationpageSuggestedadditionalfieldstoaddtoyourdatainventorytemplatearebelow
Program DatasetTitle
DatasetDescription DataSource
Departmentorexternalpartner
DoestheCityalreadycollectthisdata
Whatisthenameoftheprogram
Human-readablenameofthedataset
Shouldbeunderstandablebynon-technicalusers
Whatistheoriginalsystemapplicationfilethathousesit
Departmentorexternalpartnerthatmaintainsthedata
Isthisdataalreadycollected
Updatefrequency-ThefileformatforthedatasetTypicallycsvxlsxshapefiletxtetc
Format-Howoftenthedatasetisupdated
Timerange-Theperiodoftimecoveredinthisparticulardataset
Datause-Howisthisdatasetcommonlyused
Datausers-Whotypicallyusesthisdataset
Dataownerstitle-Titleofthedataowner
Dataownersposition-DataOwnersjobtitle
Dataownersemailaddress-DataOwnersemailaddressWhomanagesthedataandorisresponsibleforgrantingpermissiontoaccessthedataWhounderstandswhatthedatasetincludesandcananswerquestionsaboutit
Dataqualityconcerns-Describetheconcernsyoumayhaveaboutthequalityofthedataset
Assessdataaccessibilityqualityandsecurity
41
Division-DivisionwherethedataismaintainedIfmorethanonedivisionisresponsiblelisttheprimaryownerofthedataset
Sensitivedatacomments-DescribetheissuesthatwouldpreventthecityfromsharingthedatapubliclyCommonsensitivityconcernsincludeprivacyviolationssecurityissuesorhighcostorstaffdemandstopostthedata
Publishingstatus-Thistheplacetoputanycommentsregardingthecurrentstatusofthedataset
Publishedlink-Thisisthelinktothepublisheddatasetontheopendataportalorcitywebsite
Reviewdatasecurityandprivacyprotectionprotocols
OftenapolicychangealsoinvolvesachangeinthedatathatisbeingcollectedhowthatdataisbeinghandledwhohasaccesstoitandahostofotheradjustmentsAsyoutakestockofthedataassetsthatareavailableforyourinitiativeitisvitaltoalsothinkthroughwhatwillbedonetoprotectthatdataandtheprivacyofthepeopleitrepresentsAbigpartofgooddatasecurityinvolvesstrongdatagovernancepracticesthatdefineandidentifysensitivedataandconsciouslyworktobalancethevalueandriskofstoringsuchsensitivedataOnewayofevaluatingyourdatacomesfromcaselawrelatedtotheUSFreedomofInformationActwhichwehavetranslatedtothechecklistbelow
1 Woulddisclosureresultinasubstantialinvasionofprivacy2 Haveweconsideredtheextentorvalueofthepublicinterestandthepurposeorobject
oftheindividualsseekingdisclosure3 Istheinformationavailablefromothersources4 Wastheinformationgivenwithanexpectationofconfidentiality5 Isitpossibletomouldreliefsoastolimittheinvasionofindividualprivacy
TheseriskscouldbemitigatedusingavarietyofstrategiesfromaggregatingdatatoobfuscatingindividualstoredactingfieldsthatmayincludePersonallyIdentifiableInformation(PII)orre-evaluatingyourdatacollectionmethodstofindalternativewaysofgettingtheinformationyouneedThesestepsshouldbecodifiedandmanagedbyadatagovernancecommitteeifpossibleForthesensitivedatathatisdeemedessentialcreateaplanaroundpotentialdatabreachesinordertoquicklyandconfidentlyrespondtoemergencysituations
ProactivelypreparingforthesesituationsisoftenthebeststrategyfordatasecuritybutdonotforgetthetechnicalsideInvolveyourITadministratorsinconversationsaboutdatastorageandaccessensuringthattheywillbeprotectedbyenterprise-gradecybersecuritysystems
Assessdataaccessibilityqualityandsecurity
42
Assessdataaccessibilityqualityandsecurity
43
Designimplementationandevaluation
Designtheimplementation
NowthatyouhaveyouraccountabilityplanandkeyindicatorsitistimetogetspecificaboutyourplanforimplementationThetemplatebelowwillhelpyouthinkthroughthespecificmechanismsofimplementingpolicychange
UnderstandingorganizationsandsystemsiscriticaltounderstandingandmakingdecisionsaboutwhereandhowinterventionsshouldbeappliedApolicyimplementedinasinglesettingmayshowpromisingresultsbutimplementingmultiplepoliciesandprogramsatthesametimewithdifferentactorsresponsibleforeachaspectcanhelpensuresustainablebehaviorchange
InSomervilleMAShapeUpSomerville(SUS)isacitywidepublichealthprogramthatisimplementedacrossmultiplelevelsofinfluenceandsystemsthroughinterventionsbeforeduringandafterschooltoincludechangestobuiltenvironmentsuchasbikelanesregionalmasstransportationandphysicaleducationequipmentinschoolsandgymstoincreasephysicalactivitywellnesspoliciesandschoolgardensnutritioneducationduringandafterschoolandhealthyrestaurantprogramstosupporthealthyfoodandhealthyeatinghabits
Usethetemplatebelowtoplanyourimplementation
Policyimplementationplanningtemplate
ListthespecificdesiredoutcomesInthepreviousstepyouworkedwithdepartmentstafftoestablishspecificmeasurableoutcomesandinterimindicatorsPrioritizetheindicatorsthatwilldemonstrateprogressUsethespacebelowtocapturethedesiredoutcomesofthepolicychange
ExampleShapeupSomervillebeganwiththegoalofpreventingobesityin1stthrough3rdgraders
Yourturn
____________________________________________________________________________
____________________________________________________________________________
Designimplementationandevaluation
44
SpecifythetargetpopulationAreyoudesigninganinterventionthattargetsonlycertainsitesAreyouplanningonimplementingaprogramorinitiativeforaspecifictargetaudienceClearlyspecifythepopulationyouplantotarget
IfyouplantoconductarigorousimpactevaluationasnotedbytheWilsonSheehanLabforEconomicOpportunitiesitisimportanttoestablishconcreteobjectiveeligibilitycriteriaandensurethatthesecriteriaareappliedconsistentlybyallstaffduringscreeningandintakeItisalsovitaltoidentifytheindividualswhoareeligibleforaparticularprogramorserviceevenifeveryoneeligiblecannotbeservedbyexistingresourcesThishelpscitiesidentifyanyunmetneedsandguidefutureresourceallocationwhilealsocreatingthepossibilityofarigorousimpactevaluationbyidentifyingapotentialcomparisongroupFinallywhenconductingscreeningandintakeitisimperativetocollectandmaintainbasicinformation(egnamedateofbirthgender)aboutallprogram-eligibleindividualsnotjustthosewhoparticipateintheprogram
ExampleIntheShapeUpSomervilleexampletheprogrambeganbytargeting1stthrough3rdgraders
Yourturn
____________________________________________________________________________
____________________________________________________________________________
DefinethelocationWherewillthepolicychangetakeplaceWillimplementationtakeplaceatonesiteormultiplesites
ExampleShapeUpSomervilleinterventionsweredeliveredinschools
Yourturn
____________________________________________________________________________
____________________________________________________________________________
ClarifyimplementationpartnersandarticulatetheirrolesWhatarethekeyorganizationsinstitutionsorpeopleresponsibleforimplementationTheprojectteamshouldbeasclearandspecificaspossibleinprovidingadescriptionoftheconcreteactivitiesthatimplementationpartnersaremeanttocarryoutPracticeprofilesmaybeahelpfultooltohelpdefinewhatisrequiredoftheactorswhoareresponsibleforimplementingtheintervention
Designimplementationandevaluation
45
ExampleInSomervillemayoralleadershipandengagementofmultiplecityagencies(publichealthprimarycareschoolsCambridgeHealthAllianceetc)streamlinedandconnectedthevariousShapeUpSomervilleinitiativesandgrantswhichhelpedtokeepthetalentandenergyfocusedonhealthyeatingandactivelivingforallsectorsofSomervilleSUSpartnersultimatelyincludedschoolsnonprofitsCityandstateagenciesandnationalfoundations
Yourturn
____________________________________________________________________________
____________________________________________________________________________
DescribetheinterventionWhatarethestepsactivitiesorprocessesRememberthatthemechanismthespecificstepsorprocessthatleadtobehaviorchangeiskeytounderstandingtheeffectivenessofthepolicychangeandsharingtheresultswithothers
ExampleGoalsofShapeUpSomervilleweretoinfluenceeverypartofanearlyelementaryschoolchildrsquosdayndashbeforeduringandafter-schooltopromotehealthyeatingphysicalactivityandtheirweightstatusInterventionincludedfoodserviceenhancementstoimprovebreakfastandlunchandstaffprofessionaldevelopmentwalktoschoolactivitiesSUSclassroomcurriculumandprofessionaldevelopmentschoolwellnesspolicydevelopmentSUSafter-schoolcurriculumandprofessionaldevelopmentoutreachandeducationtohomethroughmaterialsforumseventsldquoSUSapprovedrdquorestaurantscommunityoutreachandcapacitybuildingthroughpolicydevelopmenttrainingsmediaplacements
Yourturn
____________________________________________________________________________
____________________________________________________________________________
ClarifytimelineandmilestonesfordeliveryandevaluationHowfrequentlywilltheservicesinterventionbedeliveredHowlongwilltheprogramrunWhatisthetimetoexpectedimpacts(eg12monthsafterprogramcompletion)
Designimplementationandevaluation
46
ExampleShapeUpSomervilleledbyTuftsUniversityFriedmanSchoolofNutritionScienceandPolicyconductedaquasi-experimental(nonrandomized)interventionstudyamong1stthrough3rdgradestudentsinSomerville(intervention)andtwocomparisoncommunitiesfor9-monthsduringoneschoolyear(fall2002ndashspring2003)ThestudyshowedpositiveassociationoftheinterventiononchangeinBMISUSreleasedreportssummarizingevaluationsprogresstodateplansforadditionalprojectsandimpactstenandfifteenyearsintotheinitiative
Yourturn
____________________________________________________________________________
____________________________________________________________________________
PlanyouranalysisTheprevioussectionaddressedestablishinganaccountabilityframeworktomonitorprogressandimplementationDeterminewhichindicatorswillbediscussthroughthatprocesswhowillsharethedataandanalysisandhowfeedbackfromthoseconversationswillbeincorporatedbytheprojectteam
ExampleTheMayorrsquosofficeimplementedadata-drivendecisionmakingapproachcalledSomerStatwhichwassynergisticwiththeapproachestakenbytheresearchandevaluationteamsdeployedforthespecificgrants
Yourturn
____________________________________________________________________________
____________________________________________________________________________
PlanforsustainabilityIfthepolicyandprogramchangeissuccessfulhowwilltheprojectteamsustainandexpandtoadditionalsitesorpopulations
ExampleAlthoughmostofthepartnershipsforgedintheearlyyearsweremaintainednewgrantsandcollaborationsexpandedattentiontoolderpublicschoolstudentsactivetransportationandtheGreenLinesubwayextensionthebuiltenvironmentfarmerrsquosmarketeffortssuchasldquoGrowninSomervillerdquowithUnionMainStreeteconomicdevelopmentandhealthdisparitiesaswellasdatacollectionmonitoringandsurveillance
Yourturn
Designimplementationandevaluation
47
____________________________________________________________________________
____________________________________________________________________________
Pilotplanningtemplate
IfyouplantoconductapilotfirstdeterminethepurposeofthepilotDoyouwanttoobservewhetherthepolicychangeresultsinthedesiredbehaviorchangeAreyouconcernedaboutfeasibilityscalabilityorcostDoyouwanttotestwhetheryourdesiredresultscanbeachievedthroughthispolicyapproachClarifythegoalofthepilotandthequestionsyouhopetoanswerThiswillhelpyouidentifythekeyindicatorsthatyouwillusetomeasurewhetherthepilotissuccessfulBeforebeginningestablishabaselineofcurrentperformanceandsetyourtargetsaccordinglybasedonwhatyouhopetoaccomplishthroughthepilot
NextestablishthepilotplanforyourprojectteamincludingmilestonesforteamcheckinstodeterminewhethertomoveaheadorchangecourseThisisseparatefromthemeetingsthatwilloccuraspartoftheaccountabilityframeworkUsethespacebelowtocreateyourtimelineanddevelopyourmeetingschedule
Task Personresponsible Startdate Checkindate Deliverable
Planyourevaluation
WhileyouaredesigningtheimplementationitisalsotimetobegintodevelopyourplanforanalysisAsnotedbytheWilsonSheehanLabforEconomicOpportunitiesdecisionsaboutprogramsetup-includingoutreachscreeningintakedatatrackingandenrollment-candeterminewhetherornotarigorousimpactevaluationwillbepossibleAsdiscussedcityleadersmayfindithelpfultoincludeevaluationpartnersinearlyconversationsaboutprogramsetupDuringpreviousstagesyoumayhaveworkedwitharesearchertoplanyourproposedpolicychangeYoushouldbeclearaboutthepurposeoftheevaluationanddevelopanevaluationplanBelowareadditionalconsiderations
Causationvstrendanalysis
IntheEstablishaframeworktomonitorprogressstagetherewasguidanceontrackingtrendsovertimeItisimportanttonoteherethatmonitoringdataisdifferentfromanalyzingacausaleffectTrendanalysiswillillustratethatchangesareoccurringandmayleadtoquestionsthatcouldbeansweredbyyourevaluationbuttrendanalysisalonedoesnotprovideevidenceaboutwhatisdrivingthosechangesYoumayseechangesinyour
Designimplementationandevaluation
48
outcomeindicatorsbasedonvariablesoutsideoftheCityscontrolForinstanceachangeinfederaltradepolicycanhaveanimpactemploymentratesinyourCityoutsideofyourCitysnewjobtrainingandplacementprogram
ToaddresscausalityyoumustcomparetheoutcomesofindividualswhoreceivedtheinterventionwiththosewhodidnotWithoutacomparisongroupofsimilarindividualswhowerenotexposedtotheinterventionitisimpossibletoknowwhichfactorsareresponsiblefortheoutcomesRandomizedControlTrials(RCTs)inwhichindividualsarerandomlyassignedtotreatmentandcontrolgroupsarethegoldstandardinprovingcausalityIfyoucandemonstratethatoutcomesimprovedforthosewhoreceivedtheinterventionanddidnotimproveatallorimprovedlessforacomparisongroupyoucanbesomewhatcertainthattheinterventionwasthecause
Examplesofothernon-causalimpactevaluationsthatarerigorousandreliableincludecertainquasi-experimentalstudiesusingcomparisongroupdesignssmartinterruptedtimeseriesanalysesthatmodeltrendsovertimeinwaysthatallowestimationofinterventioneffectswelldesignedpre-posttestanalysesneedsassessmentsimplementationstudiesetcFormoreinformationondifferenttypesofevaluationsseetheSageResearchMethodswebsite
Lookfordifferencesingroups
BuildintimetoalsoexaminethepilotdatatoseeifgroupsareexperiencingdifferentresultsReviewingperformancebasedonoutcomemeasuresforapopulationinaggregatemayhidethesubtlewaysthatsomepeoplemaybeaffecteddifferentlysoitisnecessarytocarefullyanalyzethedatafordisparateoutcomesForinstancetheMovingtoOpportunity(MTO)studythatexaminedtheimpactofhousingvouchersonfamilieswholivedinhigh-povertyneighborhoodsfoundthatchildrenwhomovedbeforetheageof13hadbettereducationandhigherincomesasadultsHoweveradultsdidnotseeimprovededucationandemploymentoutcomes
Disaggregatingdataintomorespecificcategoriesofraceandethnicitycanhelpunderstandtrendsanddisparitiesinamoregranularwaywhichcanhelpdecisionmakerstargetpoliciesandinterventionsinamoreequitableway
EarlierintheRoadmapweworkedtoidentifystakeholdersandwhomightbepositivelyandnegativelyimpactedbyachangeinpolicyUsetheresultsofthatexercisetoinvolvethecommunityininterpretingdataanalysisandtogatherqualitativefeedbackfromthecommunitytocallattentiontoissuesForinstanceanincreaseinyouthdetentionmaybeexplainedbyinefficienciesorlackofaccesstotransportation
Designimplementationandevaluation
49
Designimplementationandevaluation
50
Shareyourstoryandsuccess
Sharetheresults
Storytelling
AnintegralstepinyourstrategyissharingtheresultsofyourpilotorpolicychangewithyourcommunityThisincludesdevelopingmessagingaroundtheworkofyourorganizationandcreatingaplantodisseminateyourstoriesWhytellstoriesaboutyourwork
Informandorunderstandasubjectorsituation
Giveneededcontextofdataorbitsofinformation
Relatablestoriesarememorableforreadersviewers
ReadersorviewerswillseetheinformationasmorecompellingwhenthereisademonstratedandrelatablechangeorimpactonthemselvesortheirfriendsneighborsandcolleaguesStoriesareaboutpeopleandgovernmentisfundamentallyabouthelpingpeopleSotellthestoriesofthepeopleyouandyourcitycolleagueshelped
Keepthesetipsinmindasyoudevelopyourstorywithyourteam
WhatwereyourfindingsandwhyshouldanyonecareTellyouraudiencehowtheproblemwassolvedWhathastheimpactbeenonresidentsandcitystafftodateWhatwasthesuccessandweretheremultiplesuccessesalongthewayIfyouhavepreliminaryresultsordatashowingimprovementtosharethisisthetimetodoit
WhowasinvolvedandimpactedReviewthelistofstakeholdersHowwereeachofthemimpactedbytheprogramHumanizeyourstoryandimpactbyaddingadditionalvoicesThesevoicescanbeoneofthefollowing
Apersonorfamilywhowaspositivelyaffectedbythechangesuccess
Aninvolvedthird-partydescribingboththeproblemandthesolution(ieanorganizationnonprofitprovidersbusinesstradeassociationsportsteametc)whoispartofthecommunitythatthegovernmentworkedwithonthechangesuccess
Someonefromthegovernmentitselfwhocanspeakaboutthechangesuccess
TellmoreaboutyourcityDescribeyourcityincludingpopulationdemographicsindustriesetcWhatareitschallengesWhatisitknownforWhatmakesituniqueHowisitsimilarordifferentfromothercitiesintheUS
Shareyourstoryandsuccess
51
DescribeyourcityrsquoschallengeWriteoutthechallengeyourcityisfacing(asdefinedinyourproblemstatement)HowdoesitaffectresidentsHowdoesthisproblemaffecttheworkofcitystaffmembers
AddthedetailsProvidedetailsonyourappliedresearchstrategyTalkaboutwhatyourcityhasdonenthepasttoresolvetheissueDiscusshowyourstrategyhaschangedwhenyoubegantoadoptthisapproachWasmoneysavedorinvested
Usethisstorymapexercisetoorganizeyourthoughtsandguaranteethatyouincludealloftheelementsofatraditionalstory
AfteryoudevelopyourmessagingcreateacommunicationsmarketingandoutreachstrategydesignedtospreadyourmessageYourstrategyshouldwalkyouthroughthefollowingkeyactivities
1 IdentifyandprofileyouraudienceWhencreatinganddisseminatingyourmessageitisalwaysgoodtothinkaboutyouraudienceWhoexactlywouldbeinterestedinwhatyouhavetosayThegeneralpopulationofthecityPeoplewhoworkinnonprofitsBusinessownersItmattersbecausenoteveryonewholivesinacitymaybethetargetoftheinformationBusinessownersmaybemoreinterestedindatashowingeffectsonthebusinessdistrictthanothersandsoonWewillcoverthislaterbutthisstepwillalsohelpyoutothinkofthemosteffectivewaytoreachyouraudience
2 Defineyourcalltoaction(ifyouhaveone)SpendsometimeconsideringwhyyouareinterestedintellingyourstoryDoyousimplywanttoupdatethepubliconyourprogressorcelebrateyoursuccessesAreyouinterestedinyouraudiencechangingitsbehaviororsupportingaspecificinitiativebeinglaunchedbyyourorganizationIfyouwanttohaveyouraudienceactorbehaveinacertainwaybespecificandincorporateitinyourmessaging
Shareyourstoryandsuccess
52
3 SelectperformancemeasuresThinkabouthowyouwoulddetermineifyourcampaignissuccessfulandselectappropriateperformancemeasuresYouwillrevisitthesemeasuresattheendofyourcampaigncommunicationsplanimplementationtodeterminewhetheryouneedtomodifyyourcommunicationsandoutreachstrategyforthefuture
4 MethodstocommunicateyourstoryTherearemanywaystocommunicatesuccessofyourprojectHerearesomesuggestionsonreachingaudiencesItisimportanttodeterminewhichmethodsarethemosteffectiveforyourdesiredaudiencetoreceiveinformationDoyourresearch(PewResearchCenterhasexcellentstudiesonthereachofmediadigitalmediaandmarketing)ForexampleifseniorsareyourtargetpopulationandtheyarelessinclinedtouseInstagram(thatistrue)youmaywanttoconsiderFacebookorthenightlylocalnewsbothofwhichareusedmoreoftenbyseniorsBelowisalistofsomemethodsandtoolsyoucanuse
i Pressreleasestolocalnewspapersandtvstations
ii ShortsmartphonevideosAgreatwaytoengageanaudienceisthroughvideoAskonequestionofsomeonewhoispartofthestoryMaybethereisasocialworkerwhocandiscusshungerinyourcityThesevideoscanbereusedonmultipleplatforms
iii Long-formwriting
iv TellingindividualstoriesthroughaseriesofphotosandshortnarrativesforpublicationonFacebookorInstagram
v Op-edsopinionpiecesplacedinthelocalpaper
vi Emailupdatestoyourlist
vii Connectingwithappropriateorganizationsthatareinvestedinthetopicandworkwiththecommunityandaskthemtodisseminateinformationtotheirmembers
viii EmployingldquoWordofMouthrdquo(WOM)tacticsbycreatingtownhallsorfreeeventsforresidentstoattendintheirownneighborhoods
5 DeterminethetypeofcollateralUsuallycampaignsthattargetresidentsrequireseveralavenuestoreachthemPeoplearenotalwaysinthesameplaceorspaceHavingakeenunderstandingofyouraudiencewillassistyouwithyourchoicesonwhatneedstobedevelopedSometimesthatmeansimagesforsocialmediaaonepageflierthatgivespeopleinformationonattendinganeventoranin-depthbrochure
Shareyourstoryandsuccess
53
6 ImplementyourstrategyKeepinmindstrategiescanbefluidSometimessomethingmaynotturnoutasexpectedevenifduringplanningsessionsitwasldquosupposedtordquoItisokYoucananalyzewhathappenedduringadebriefsessionandonceyoursurveysandothermeasurementshavebeenfinalized
Visualizations
Withtherightvisualizationyoucanreviewandshareinformationinsightsanalysisperformanceandtrendsallofwhichisvaluabletoacross-sectionofthepublicButwhichvisualizationisrightforyouTochoosetherighttypeofchartgraphdeterminethetypeofquestionyouwanttoanswerSomecommoncharttypesarebelowandmoreinformationcanbefoundhere
Exploreadditionalanalyticaltechniques
Duringthegoal-settingphaseyouestablishedoutputoutcomeandimplementationmeasuresBasedontheresultsofyourpilotyoumayneedtoadjustyourmeasuresWiththepilotdatainhandyoumayhavesomeideasofhowfutureanalysescouldbedifferentandpotentiallyimprovethequalityofyourprogramTakesometimetobrainstormotherrelevantdatasetsthatyoudidnothavethetimetopursueearlierincludingmessydepartmentaldatathatneedsmoreattentionorsourcesofpublicdatafromthestateandfederalgovernmentnonprofitsandevenprivatecompaniesthatcouldsupplementyourunderstandingofanissueUsingmachinelearningtechniquesyoumightbeabletobetter
Shareyourstoryandsuccess
54
predictdemandforhomelessservicesusingNOAAweatherforecastspreemptwatermainbreaksusingsoilstratigraphyorinvestineconomicdevelopmentbymonitoringrealestatelistingdata
Advancedanalyticaltechniquescanalsohelpyourimplementationrunmoreeffectivelybyintegratingdata-informedtargetinganddecisiontoolsintotheprocessByautomatingroutineworkflowsandsurfacingsubtletrendsthesetoolscanhelpyouiterateinamoreprecisemannerandprovidekeystrategicguidancetofront-lineemployeesthatwouldotherwisebeunavailableThesesortsofdata-science-enhancedprocessescanbeclassifiedasoneofsixprojecttypesasexploredfurtherbytheNewOrleansLANOLAlyticsteamFormoreguidanceonscopingasimilarprojecttheGovExadvancedanalyticsscopingguidegivesyouastep-by-stepoverviewofhowtobesuccessful
Usethespacebelowtolistyourideasforpossibleadvancedanalyticsprojectsbytype
Typeofproject Ideas
Findingtheneedleinthehaystack
Prioritizingforimpact
Earlywarningtools
Betterquickerdecisions
Optimizingresourceallocation
Experimentingforwhatworks
Keepthefeedbackloopgoing
GatherinformationonpublicsentimentaboutpolicychangethroughfocusgroupssurveysandpublicmeetingsledbyadvisorygroupmembersNotewhethernewinformationisavailablethatcanhelpanalyzeorcontextualizeyourprogress(egfederalorstatedatareports)Sharetheresultsofyourpilotandthepolicychangeprocesswithothercitiesandresearchers
Shareyourstoryandsuccess
55
WorkingwitharangeofevidencePresentlythereisalotofattentionandinvestmentinevidence-basedpolicymakingincludingthefederalbipartisanCommissiononEvidence-BasedPolicymakingthePewCharitableTrustsreportsandtoolkitsonHowStatesEngageinEvidence-BasedPolicymakingandtheLauraandJohnArnoldFoundationsupportedEvidence-BasedPolicymakingCollaborativeManyoftheseinitiativesfocusonrigorousevaluation--meaningrandomizedcontrolledtrials(RCT)orotherexperimentalmodels--asthebestkindofevidencetouseHoweverthisbodyofevidenceisofteninsufficientforthespecifickindsofquestionsthatgovernmentscanhavewhenmakingadecisionaboutpolicyInthisguideweproposeanapproachtosupportgovernmentsinincorporatingadditionaltypesofevidenceintotheirdecisionmaking
OnewaytounderstandarangeofevidenceistouseNestarsquosStandardsofEvidencetoframeanddefineevidenceTheBritishnonprofitNestaworkstoinjectpositivechangeintothewayUKgovernmentpoliciesandprogramsfunctionOneoftheirkeyareasofworkhasbeendevelopingWhatWorksCentresorganizationsthatcollectcommissionsynthesizeanddisseminateevidenceaboutinterventioneffectivenessacrossarangeofsocialimpactareasNestadevelopedthefollowingStandardsofEvidencetoprovideausefulwaytodocumenthowdifferentkindsofinformationdemonstratedifferentlevelsofevidence
Workingwitharangeofevidence
56
Theimportanceofpotentiallybiasedinformation
Onechallengepresentedbyanevidenceclassificationthatacknowledgesthevalueofnon-scientificformsofevidenceistofigureouthowtobalanceallofthepiecestogetherwhenyouusethemTounderstandhowtodothiswehavetothinkabouthowtopositionpotentiallybiasedinformationindecisionmaking
WhenwelookatinformationwhichhasnotbeenrigorouslyevaluatedweacceptthatthisinformationmightbebiasedThewholereasonweconductrigorousevaluationistoremovebiasfromourobservationsWeconductstructuredtrialstoproduceresultsthatareconsistentandgeneralizableIfafindingisunbiasedthisaccuracymdashthislackofbiasmdashcanbeprovedbecauseanyindependenttestercanreproduceitwhentheyfollowanidenticalprocess
YoucanlookatpotentiallybiasedinformationasjustinformationthatisearlieroninitspathtorefinementreplicationandverifiedaccuracyApieceofinformationbasedonanyoneindividualrsquosobservationsislikelytobebiasedbecauseitiscomingfromjustonesourceBiasisreducedbygatheringtogetherlotsofindependentsourcesofinformationandfiguringoutitsmostcommonattributes
MaybethebiasedinformationmakesaclaimthatistoolargemdashitsayssomethingistrueunderallconditionswhenitisreallyonlytrueundercertainconditionsorthataneffectofaninterventionisbiggerthanitstrueaverageorthataneffectlastsovertimewhenitdoesnotPerhapschangingtheclaimtobemorespecificmdashcriticallyreviewingtheinformationtoseeifspecificconditionsmustbemettoachievetheoutcomesmdashwouldmaketheclaimmoreaccurate
IfyoutestedyourclaimfounditdidnotholdupinsomecaseschangedittesteditagainuntilyourclaimrepeatedlyheldupthenyoucoulddemonstratethattheclaimwasnolongerbiasedbutaccurateThisprocessoftestingimprovingtestingandvalidatingiswhatwegetfromstructuredscientificstudy
Withindividualpiecesofinformation--crediblehypothesescasestudiesexpertopinion--youcannotassumethatkindofaccuracyAtthesametimetheseearlyobservationsarefoundationaltoanyreplicablescientificobservationsWewillnevergetanywhereifwedonotstartwiththesepiecesofinformationeveniftheymightbebiased
Combininglevelsofevidencetogetthemostdetailedinformationforyourcity
ItisespeciallyimportanttoacceptdifferentformsofevidencewhenyoufacequestionsthathavenotyetbeentestedbyindependentstudiesSomeofthemostimportantquestionsrelatetothingsthatareveryspecifictoyoursituationYoufaceaparticularcombinationof
Workingwitharangeofevidence
57
issuesactorstrendsandresourcesthatisultimatelydifferentfromeveryotherplaceintheworld
ThisisimportantbecauseaspolicypractitionersandnotscientistsyoucaremoreaboutyourspecificcasethanahypotheticalaveragegeneralcaseYoucarelessaboutthetotallikelyrangeofimpactsfromapolicythanhowthissingleimplementationwillworkYouwantinformationaboutpoliciesthatwillprovideyouwiththebestpossiblepredictionabouthowtheywillworkinyourcontext
ByputtingdifferentkindsofevidenceintothesameframeworkNestarsquosStandardsofEvidenceimplythatwhenyoucannotgetrigorouslytestedexperimentallyverifiedinformationitisappropriatetoworkwithwhatwedohaveThisisrelevantforthemunicipalimplementationcontextbecausenostudypulledfromaliteraturereviewwillaccountforallofthevariablesyoufaceinyourowncontext
Intheabsenceofhavingastudythatpredictshowaninterventionworksinyourspecificcontextyouwillneednon-scientificinformationtotrytofillinthosegapsYouwillneedtogatherstoriesfromresidentsstaffmembersandotherstakeholdersthatsitonLevels1or2ofthestandardsofevidence
Someofthekindsofinformationyouwillprobablygetmoreeasilyfrompeoplethanstudiesinclude
howpeopleincitieslikeyoursreactedtoaninterventioninpublicforumshowpeoplewhoimplementedaninterventioninacitylikeyoursfiguredouthowtogetitpassedhowthecitycommunicatedpubliclyabouttheapparentbenefitsanddrawbacksoftheinterventionthepoliticalacceptabilityoftheintervention
Youcanremainsimultaneouslyawarethattheinformationyouaregatheringthroughindividualstoriesisbiased--notgeneralizableinformedbyspecificperspectives--whilealsorecognizingthathavingsomeinformationaboutsomethingimportantisbetterthannoinformationEvidencefromthefirsttwoevidencelevelsareessentialbuildingblocksofknowledgeandperfectlyusefulwhenyoucannotgetdetailedunbiasedknowledge
Youhavetostartsomewhereandyoushouldalwaysstartwithevidenceyouhaveaboutyourveryspecificconditionsthatyouknowareimportantforsuccess
Workingwitharangeofevidence
58
Thankyou
Interviewswithcities
GovExheldsemi-structuredinterviewswithsixstaffmembersinfourcitiestoestablishtheinitialstepsoftheRoadmapWeaskedrespondentstotalkaboutaninitiativeorpolicythattheyhavebeenresponsibleforimplementingintheircityRespondentsfromthreecitiesspokeabouttheireffortstoaddresshomelessnessandrespondentsfromonecityspokeaboutblightAfterrespondentsidentifiedtheproblemtheyspokeabouthowthecityprioritizedthisissuethevariousstepsinvolvedandbarrierspresent
GovExconductedtextanalysisoftheseinterviewswhichrevealedthatpeoplewasthemostcommonlyusedtermduringtheseconversationswhendiscussingthestepstochangeaprogramorpolicyThisisathemethroughoutthisRoadmapThefirstsectionbeginswithidentifyingtherelevantpeopleTheconversationsaroundevidencearefocusedonhumanbehaviorratherthanmethodologyorstrengthofevidence
Thankyou
59
OncewehadadraftoftheRoadmapwefollowedupwithcitiesonanindividualbasistolearnmoreaboutwhatwouldbehelpfulWearegratefultothefollowingcitystafffortheinterviewsandindividualconversations
MichaelBaskinChiefPolicyOfficerChattanoogaTN
MelissaBridgesPerformanceandInnovationCoordinatorLittleRockAR
VictoriaCarreoacutenAdministrativeOfficerLasVegasNV
TyroneGrandstrandCommunityDevelopmentPlannerFargoND
TibLaughlinDirectorofGeneralServicesKansasCityKS
TimMorelandDirectorofPerformanceManagementandOpenDataChattanoogaTN
CharlesRosendahlManagementAnalystDentonTX
Thankyou
60
MelissaSiebenAssistantCountyAdministratorKansasCityKS
GinaThompsonDirectorofStrategicServicesLewisvilleTX
JHUFacultycollaboration
GovExalsoconvenedthecitycontactsandfacultypartnersinaseriesofAdvisoryGroupmeetingsThesemeetingsservedtoconnectcitiesandfacultytounderstandoneanothersperspective
StefanBaralBloombergSchoolofPublicHealth
AshleyBernerJHUSchoolofEducation
MelissaDavey-RothwellBloombergSchoolofPublicHealth
JonLinksBloombergSchoolofPublicHealth
TamarMendelsonBloombergSchoolofPublicHealth
KristinMmariBloombergSchoolofPublicHealth
KeshiaPollackPorterBloombergSchoolofPublicHealth
JoshSharfsteinBloombergSchoolofPublicHealth
LizStuartBloombergSchoolofPublicHealth
LindsayThompsonCareyBusinessSchool
DanielWebsterBloombergSchoolofPublicHealth
Partnersindataandevidence-basedpolicy
Wealsospokewithpartnerswhoareexpertsinthefieldofevidence-basedpolicywhosharedtheirknowledgeandresourcesofcurrentandfuturethinkinginevidence-informedthinking
WendyBarrenoAssociateDirectorWilsonSheehanLabforEconomicOpportunitiesUniversityofNotreDame
TiffanyDavenportDirectorofDataPartnershipsBaltimoresPromise
TomiHiersDirectorBaltimoreCivicSiteAnnieECaseyFoundation
ChrisKingsleySeniorAssociateAnnieECaseyFoundation
KathrynOliverAssociateProfessorLondonSchoolofHygieneandTropicalMedicine
Thankyou
61
KristenWatkinsSeniorPolicyAssociateJ-PAL
GovExParticipants
GovExAnalystMiriamMcKinneyanalyzedtheinterviewtranscriptsandconductedtextanalysisSeniorImplementationAdvisorEmilyShawwillintegrateRoadmapactivitieswithadditionalservicedeliveryactivitiesatGovEx
Thankyou
62
ReferencesAbrahamKGampHaskinsR(2017)Thepromiseofevidence-basedpolicymakingReportofthecommissiononevidence-basedpolicymakingWashingtonDCCommissiononEvidence-basedPolicymakingBigmove2Extendingthegreen(nd)Retrievedfromhttpswwwcityoflewisvillecomdoing-businesslewisville-2025big-move-2-extending-the-greenBatesMAampGlennersterR(2017)ThegeneralizabilitypuzzleRetrievedfromhttpsssirorgarticlesentrythe_generalizability_puzzleBuildwithnotfor(2018January)RetrievedfromhttpsbuildwithnotforwordpresscomCairneyPampOliverK(2017)Evidence-basedpolicymakingisnotlikeevidence-basedmedicinesohowfarshouldyougotobridgethedividebetweenevidenceandpolicyHealthResearchPolicyandSystemsdoi101186s12961-017-0192-xCairneyPOliverKampWellsteadA(2016)TobridgethedividebetweenevidenceandpolicyReduceambiguityasmuchasuncertaintyPublicAdministrationReview76(3)399-402Retrievedfromhttpdxdoiorg101111puar12555Centreforpublicimpact(nd)RetrievedfromhttpswwwcentreforpublicimpactorgCernaL(2013)ThenatureofpolicychangeandimplementationAreviewofdifferenttheoreticalapproachesOrganisationforEconomicCo-OperationandDevelopmentRetrievedfromhttpswwwoecdorgeduceriTheNatureofPolicyChangeandImplementationpdfColoradoSprings(nd)Retrievedfromhttpscoloradospringsgovcommunicationspagewhat-expect-when-you-speakupContinuumofcare(coc)program(nd)RetrievedfromhttpswwwhudexchangeinfoprogramscocCouncilpresidentkicksoff$12millionyouthfund(nd)Retrievedfromhttpwwwbaltimorecitycouncilcomcontentcouncil-president-kicks-12-million-youth-fundDurlakJAampDuPreEP(2008)ImplementationmattersAreviewofresearchontheinfluenceofimplementationonprogramoutcomesandthefactorsaffectingimplementationAmericanJournalofCommunityPsychology41327-350Retrievedfromhttpswwwncbinlmnihgovpubmed18322790Fishbonediagram(ishikawa)(nd)Retrievedfromhttpasqorglearn-about-qualitycause-analysis-toolsoverviewfishbonehtmlFivewhysGettingtotherootofaproblemquickly(nd)RetrievedfromhttpswwwmindtoolscompagesarticlenewTMC_5Whtm
References
63
FligsteinNampMcAdamD(2011)TowardageneraltheoryofstrategicactionfieldsSociologicalTheory29(1)1-26RetrievedfromhttpsociologyberkeleyedusitesdefaultfilesfacultyfligsteinFligstein20McAdam20Soc20Theory20PaperpdHeinkelP(2018January12)CitymullspermitsforbikesharingcompaniesRetrievedfromhttpwwwdentonrccomnewsdenton20180112city-mulls-permits-bike-sharing-companiesHowisareamedianincomecalculated(2012October)Retrievedfromhttpswwwhudexchangeinfofaqs843how-is-area-median-income-calculatedLampkinLMampHatryHP(2003)KeystepsinoutcomemanagementUrbanInstituteRetrievedfromhttpswwwurbanorgresearchpublicationkey-steps-outcome-managementMarshDampSmithM(2000)UnderstandingpolicynetworksTowardsadialecticalapproachPoliticalStudies484-21Retrievedfromhttpjournalssagepubcomdoiabs1011111467-924800247McCannL(nd)RetrievedfromhttplaurenellencomMoultonSampSandfortJR(2016)ThestrategicactionfieldframeworkforpolicyimplementationresearchPolicyStudiesJournal45(1)Retrievedfromhttponlinelibrarywileycomstore101111psj12147assetpsj12147pdfv=1ampt=jdkhhcp4amps=e184c625e1855a6b1615ccb99ee27b1c0b3212d8PaynterB(2018January1)What100citiesarelearningfromeachotherbysharingtheirdataRetrievedfromhttpswwwfastcompanycom40518004what-100-cities-are-learning-from-each-other-by-sharing-their-dataPeakacademy(nd)Retrievedfromhttpswwwdenvergovorgcontentdenvergovenmayors-officeprograms-initiativespeak-performancepeak-academyhtmlPinnockHBarwickMCarpenterCREldridgeSGrandesGGriffithsCJRycroft-MaloneJMeissnerPMurrayEPatelASheikhAampTaylorSJC(2017)Standardsforreportingimplementationstudies(stari)ExplanationandelaborationdocumentBMJOpenRetrievedfromhttpbmjopenbmjcomcontentbmjopen74e013318fullpdfProctorESilmereHRaghavanRHovmandPAaronsGBungerAGriffeyRampHensleyM(2011)OutcomesforimplementationresearchConceptualdistinctionsmeasurementchallengesandresearchagendaAdministrationandPolicyinMentalHealth38(2)65-76doi101007s10488-010-0319-7TandonSD(2018)ProgressincommunityhealthpartnershipsResearcheducationandactionRetrievedfromhttpswwwpressjhuedujournalsprogress-community-health-partnerships-research-education-and-actionProvanKGampKenisP(2007)ModesofnetworkgovernanceStructuremanagementandeffectivenessJournalofPublicAdministrationResearchandTheory
References
64
18(2)229-252Retrievedfromhttpsacademicoupcomjpartarticle182229935895SabatierPA(1988)Anadvocacycoalitionframeworkofpolicychangeandtheroleofpolicy-orientedlearningthereinPolicySciences21(3)129-168Retrievedfromhttpwwwjstororgstable4532139SandstromAampCarlssonL(2008)TheperformanceofpolicynetworksTherelationbetweennetworkstructureandnetworkperformancePolicyStudiesJournal36497-524Retrievedfromhttponlinelibrarywileycomdoi101111j1541-0072200800281xabstractSchwartzMampWilsonE(nd)WhocanaffordtoliveinahomeAlookatdatafromthe2006americancommunitysurveyRetrievedfromhttpswwwcensusgovhousingcensuspublicationswho-can-affordpdfShearerJCAbelsonAKouyateBLavisJNampWaltG(2016)WhydopolicieschangeInstitutionsinterestsideasandnetworksinthreecasesofpolicyreformHealthPolicyandPlanning31(9)1200-1211Retrievedfromhttpsdoiorg101093heapolczw052ThelessonsofadministrativedataHigh-profilepolicy-relevantresearchpoweredbyadministrativedata(nd)Retrievedfromhttpswwwpovertyactionlaborgsitesdefaultfilesresources20161114-The-Lessons-of-Administrative-DatapdfToolsandotherneedsforqualitativeandquantitativemethods(2015February)AgencyforHealthcareResearchandQualityRetrievedfromhttpwwwahrqgovprofessionalsquality-patient-safetytalkingqualityassesstoolshtml
WeaverRCampBagchi-SenS(2013)SpatialanalysisofurbandeclineThegeographyofblightAppliedGeography4061-70
References
65
UniversityPartners
Name Title Affiliation Area(s)ofexpertise
Contactinformation
Examplesofworkingwithnon-academicdecisionmakers
WendyPBarreno
AssociateDirector
WilsonSheehanLabforEconomicOpportunitiesUniversityofNotreDame
LEOsfocusareasincludehealtheducationhousingchildwelfareeconomicopportunitycriminaljustice
wbarrenonduduorleondedu
MelissaDavey-RothwellPhD
AssociateScientist
JohnsHopkinsBloombergSchoolofPublicHealth
Health mdavey1jhuedu410-614-5854
BaltimoreCityHealthDepartmentLeadersMarylandDepartmentofHealthAddictionRecoveryServicesinDCExecutiveDirectorsofcommunityorganizationsacrosstheUSthatfocusonHIVSTIsubstanceabuseandimplementationscience
ElizabethStuart
AssociateDeanforEducationandProfessorDepartmentsofMentalHealthBiostatisticsandHealthPolicyand
JohnsHopkinsBloombergSchoolofPublicHealth
Educationsocialpolicystatisticalmethods
estuartjhuedu
Resources
66
Management
Design-ThinkingResources
EngagementGuideforCivicTech
Journeymap
PerformanceManagementResources
DenverPeakAcademy
Proxymeasuresguide
Talkingaboutperformancewithoutcomplicatingit(video)
SettingPerformanceTargetsguide
Datavisualizationguide
Performancemanagementgettingstartedguide
Exercises
Fishboneexercise
5Whys
AdministrativeData
TheLessonsofAdministrativeData(JPAL)
EvaluationResources
Quantitativeandqualitativemethods(USDepartmentofHealthandHumanServicesAgencyforHealthcareResearchandQuality)
KeyStepsinOutcomeManagement(UrbanInstitute)
RCT-YES
Sageresearchmethods
BestPracticesforMixedMethodsResearchintheHealthSciences
ACCORDChecklist
Pilotandimplementationplanning
Benchmarkingexamplesandproxymeasures
Seattleopendataplaybook
Resources
67
Datasciencescopingguide
Resources
68
AboutGovExGovExhelpsgovernmentsusedataandevidencetoimproveresidentsrsquolivesWesupportgovernmentsinbuildingtheirinternalcapacitytooperatesustainableanalyticspracticeswhichimprovecommunityoutcomesinfieldsfrompublicsafetytoeconomicdevelopmentGovExhasaproventrackrecordofworkinginconcertwithpartnerstoprovidetechnicalassistanceandtrainingatscale
SinceourlaunchinApril2015GovExhas
Trainedover2000governmentemployeesinpersonandonlineintopicsincludingdatamanagementperformancemanagementopendatacommunityengagementandanalytics
Providedtechnicalassistancetomorethan100localgovernmentsacrosstheglobeimprovingtheiruseofdatatoachieveefficienciesinservicedeliveryandbetteroutcomesforresidents
Assessedthecapacityofmorethan130governmentalorganizationstodeterminehowleadersstaffandexternalstakeholdersmanageandusedata
Localgovernmentsareincreasinglyonthefrontlinesofaddressinglargescaleproblemswithglobalimplicationssuchasclimatechangepublicsafetyandeconomicinequality
Whileevidence-basedinterventionstohelpsolvethoseproblemsareavailablelocalgovernmentstaffhavelimitedtimeandcapacitytofindandimplementresearch-basedsolutionsAsaresultcitiescontinuetoworkthewaytheyalwayshave--makingincrementalimprovementstoprogramsbutnotalwayscreatingchangeatasystem-level
ThroughresourcedevelopmentandcollaborationwithJHUfacultyGovExissupportingcityleadersinusingdataandevidencetomakedecisionsandcreatinglastingrelationshipswithresearchers
TheRoadmapwasdevelopedbyGovExaspartofitspartnershipwithBloombergPhilanthropiesWhatWorksCitiesinitiative
AboutGovEx
69
- Title page
- Introduction and purpose of the Roadmap
- Using the Roadmap
-
- Gather information about the problem
- Strategically engage key partners
- Review the evidence and collect details
- Decide on a policy to implement based on local factors
- Establish a framework to monitor progress
- Assess data accessibility quality and security
- Design implementation and evaluation
- Share your story and success
-
- Working with a range of evidence
- Thank you
- References
- Resources
- About GovEx
-
QualitativedatacanalsohelpanswerquestionsaroundtheeffectivenessofpreviouseffortsTrytoobtainavarietyofperspectives--bothmainstreamandoutliers--fromnetworksbeyondyourowntoprovideyouwiththegreatestvantagepointforevaluatingpotentialimpactUnderstandinghowthoseeffortswerereceivedbydifferentpartsofyourcommunityandiftheremaybeelementsofprevioussystemstopreservewillbecrucialtothesuccessofanypolicychangeThisinformationmaybeobtainedthroughfocusgroupsinterviewsandsurveysFormoreinformationonhowtousethesequalitativeresearchtoolspleaseseetheseresourcesdevelopedbytheAgencyforHealthCareandResearchandUrbanInstitute
FinallytakealookatnationaldatatoseewhereelsethisproblemmightexistThisresearchmaybehelpfulinlaterstepswhenyouarelookingforpromisingpoliciesinothercities
Puttingitalltogether
UsethetemplatebelowdevelopedbyDenversPeakAcademytocaptureyourrefinedrightsizedproblemstatementandcurrentandfuturestatesWorkingthroughthisexercisewithpartnersishelpfultogainbuy-inTheresponseswillbeimportantwhengettingotherpartnersonboard
WhyChangeIsNeeded
Yourright-sizedproblem
CurrentState
Analysisoftrendsindata
FutureState
Ideallywhataretheoutcomesyouhopetoachieve
Checklist
Thekeystepstogatheringinformationarebelow
Atthispointfocusontheproblemratherthanjumpingtosolutions
Listentopeoplewhoareaskingforhelpandtoadvocateswhoareraisingissues
Beasspecificaspossibleinidentifyingtherootcauseandhoninginonthepartoftheproblemthatyourcitycanaddress
Identifytrendsbasedonmultiplesourcesofdataincludinghistoricaladministrativedataproxymeasuresqualitativedataandnationalmeasures
Gatherinformationabouttheproblem
14
Captureyourproblemstatementtheresultsofyouranalysisanddesiredoutcomes
Gatherinformationabouttheproblem
15
StrategicallyengagekeypartnersPolicytheoryresearchoftenfocusesonincreasingpolicymakersaccesstoevidenceasawaytoavoidtakingshortcutslikemakingdecisionsbasedonemotionImplementationsciencesuggeststhatincreasingaccesstoevidenceisnotenoughandthatpolicychangeshouldbeexaminedinthewidercontextofdebatecoalitionformingandpersuasionMeaningfulpolicychangerequiresldquoalong-termstrategytoformalliancesanddevelopknowledgeofpolicymakersandpolicymakingratherthanusingevidenceinpolicymakingonanadhocorone-offbasis
UnderstandingnetworksandsystemsisimportanttodevelopsustainablepolicychangeThissectionismeanttohelpreaderstakethestepstobuildengagementstrategiesforpartnersthatrepresentdifferentsystemsandnetworksEngagementshouldbeanongoingprocessofconnectingpeopletooneanotherandtoanissueEngagementshouldnotbeinnameonly-wehaveallexperiencedengagementactivitiesthatgivepeopleaspecificroletoplaythatleadstoaneatpredictableoutcomeAccordingtotheBuildWithmovementundervaluingandunderinvestinginengagementstrategiescanleadtoadisconnectofcertainindividualsorgroupsandmistrustofgovernmentWhendevelopingengagementactivitiesrememberthathowpeopleengageandwhethertheyengagehastodowithhowyoustructureopportunitiesforengagement
Conveneacoreadvisorygroup
AnadvisorygroupcomprisedofpeoplefrommultiplesectorscanbehelpfulasyouconsiderthelocalsocialenvironmentalandpoliticalcontextinwhichyouintendtointroducepolicychangeRevisityourstakeholdermapanddeterminewhowouldbebesttoserveonthisgroupTomakesureyouareconsideringtheproblemsandsolutionsthroughanequitylensincludepeoplewithlivedexperiencewhohaveexpertiseintheeffectivenessofsystemswhetherthosesystemsfunctionasintendedandhowtheymayimpacthumanbehaviorYoumayalsowishtoincludesomeonewithexperienceworkingwithdataandanalysis-thispersoncanhelpdeterminethetypeofdatathatmaybereadilyavailableandusefultocollectandcanalsohelpplanandshapeyouranalysistofigureoutwhichprogramsareeffectiveBesuretoinvitefront-lineworkersaswell-itscriticalthatwhateversolutionscomeoutofthisprocessaremeaningfultothosewhowillberesponsibleforimplementingthem
BringingpeopletogethercanbechallengingbutthereareafewwaystosetandmanageexpectationsFirstmakesurethatthegroupisawareoftheproblemstatementcurrentstateandfuturestate-thiscanhelpstarteveryoneoffonthesamepageNextspend
Strategicallyengagekeypartners
16
sometimegettingtoknoweachotherrsquosworktolearnabouteveryonesstrengthsThissharedunderstandingcanhelpasyouclarifyrolesandresponsibilitiesofeachpartnerinthegroup--thisisimportanttoavoidfinger-pointinginthefutureandtoensurethatnoneofthetasksslipthroughthecracksInsomecasestherearemanyactorswithdifferentresponsibilitiesasinthecaseofBaltimoreCitys38-memberYouthFundTaskForcewhichischairedbynonprofitandcorporatepartnersandincludesrepresentativesfromthenonprofitandfaithcommunitiesstudentsthebusinesssectorlocaluniversitiesCityCouncilandtheMayorsOffice
OncerolesandresponsibilitiesareclearyoumayconsiderdevelopingacharterguidelinesortermsofreferencetosetexpectationsforthegroupBesuretodistinguishthisfromotherboardsandcommissionsthatyourcitymayhaveNextyoumaywishtoestablisharegularongoingmeetingscheduleBesuretosendagendasandmeetingmaterialsaheadoftimeandclarifytheobjectiveofeachmeeting
InSeattletheCitydevelopeditsHousingAffordabilityandLivabilityAgenda(HALA)torespondtorisinghousingcostsandhomelessnessThisisatrendinotherwestcoastcitiesaswellwhichmaybeexplainedbyaninfluxofnewresidentswhoareattractedbytheemergenceoftechjobstheattractionoftheoutdoorlifestyleortherecentpassageofprogressivepolicieslikelegalizedgaymarriagea$15minimumwageandlegalizationofmarijuanaSeattlebroughttogetheramulti-stakeholderadvisorygroupcomprisedoflocalhousingexpertsrentersandhomeownersdevelopersandgovernmentofficialstoshareinformationabouttheirneedsanddeveloprecommendationsbasedontheseneedsTheinterestsandmotivationsofthisgroupwerevariedandittookabouttenmonthstofindcommongroundTheseriesofrecommendationsthattheadvisorygroupcreatedismeanttoserveresidentswithaspectrumofneeds
Buildaprocessforcommunityparticipation
Ofcoursenoteveryonecanbeincludedinanadvisorygroupandadvisorygroupmembersmaynotnecessarilyrepresentthevariedperspectivesoftheirpeers-forinstancenoteveryhomeownerhasthesamesetofneedsTogatherinputfromawiderswathofthepopulationitcanbehelpfultobuildaprocesstoinvitethepublicatlargetosharetheirideasCitiesdothisinavarietyofwaysfromonlineconversationsandsurveystoin-personmeetingsRememberthatbeingintentionalaboutgatheringtheviewpointsofpeopleismoreimportantthanusingthelatesttechnologyandtools
TheinsightgeneratedbycommunityparticipationexercisescanbeincludedinyourreviewofevidencewhichisdiscussedinasubsequentchapterInWestVirginiaastatethathasbeenhithardbytheopioidepidemicstatehealthofficialsdraftedanopioidresponseplanthatwasinformedbypublicengagementaswellaslocalandregionalsubjectmatter
Strategicallyengagekeypartners
17
expertsThepublicwasinvitedtopresentcommentsonideasfordecreasingoverdoseearlyinterventiontreatmentandrecoveryPublichealthandmedicalfacultyfromWestVirginiaUniversityMarshallUniversityandJohnsHopkinsUniversityprovidedresponsestothecommentsTheresultwasanOpioidResponsePlanthathealthcommissionerRahulGuptapresentedtoGovernorJimJustice
AsamplespectrumtoplanyourowncommunityparticipationprocesscreatedbyLaurenellenMcCannisbelow
SourceLaurenellenMcCann
Community-basedparticipatoryresearchisanotherwaytoworkwiththecommunitytoimproveoutcomesCBPRmeansthecommunityisinvolvedinallaspectsoftheresearchprocesswhichisofteniterativeInCBPRtraditionalresearchersandcommunitymembersshareknowledgeaswellasresourcesandcredit
CBPRInActionInSanFranciscoanenvironmentaljusticenonprofitteamedupwiththeCitysDepartmentofPublicHealthandanoutsideevaluatortounderstandtherootcausesofobesityandfoodinsecurityinlow-incomeneighborhoodsandidentifysolutionsThroughresearchmethodssuchasinterviewingresidentsandstoreownersdiagrammingstoreshelvesGISmappinganeconomicanalysisandreviewofsimilarpoliciesinnearbycitiestheteamobservedthatthestoresthatresidentshasveaccesstowerestockedwithlesshealthyfoodsandthatresidentswantedmorehealthyfoodandlessalcoholandtobaccoThisprojectresultedinlocalandstatelegislationtoencouragestorestostockfreshproduce
Strategicallyengagekeypartners
18
Developrelationshipswithsubjectmatterexperts
OnewaytokeepsubjectmatterexpertsengagedistoinvolvetheminthecommunityparticipationprocessasdescribedaboveIntheWestVirginiaexamplesubjectmatterexpertsrespondedtoandsynthesizedthepubliccommentsOtheropportunitiesaredescribedbelow
AdvisoryAcitydepartmentheadMayororstaffmembermaydevelopaone-on-oneadvisoryrelationshipwithuniversityfacultymemberoracollaborativebetweentheCityandUniversitytoaddressaparticularproblemAgreatplacetostartistolookforrelevantpracticecenterswithinyourlocaluniversityandreachingouttosubjectmatterexpertsInadditiontotheirexpertiseontherelevantpolicytopicheorshemayalsobeabletoidentifyotherexpertsandevenprovidesupportonhiringtherighttalenttoleadcertainprogramsAnadvisoryrelationshipwithauniversitysubjectmatterexpertcanalsohelpwhencitiesareconsideringbringinginnewprogramsTheseexpertsareoftenfamiliarwithprogramsthatarebasedonnationalmodelsandcanprovideinsightintoaprogramsoriginandlineageeffectivenessinotherplacesandthepotentialforimplementationinyourcityAcademicscanalsohelpidentifyindicatorsbasedongovernmentindicatorsorliteratureFinallyrelationshipsbetweenpolicymakersandresearchersprovidethemostbroadlyeffectiveroutetopullingresearchintopolicy
ResearchandevaluationAcademicexpertsmaysupportcitiesinevaluation-indeedevaluatingcityinitiativesisatraditionalmodelofcityuniversityrelationshipsSharingtheinformationyoucollectedintheGatherInformationphasecanhelpresearchersunderstandyourworkandgoalswhichinturnwillstrengthenthesupportthattheycanprovideDiscussingdetailsaboutyourinterventionandevaluationplanatthebeginningoftheprocessmeansthataresearchpartnercanhelpwiththeevaluationdesignanddatacollectionForinstancetheWilsonSheehanLabforEconomicOpportunitieswalkscitiesthroughconsideringthefollowingquestionsbeforepartneringonanevaluation
1 Isthereaclearpolicyprogrammodelwithobjectivescreeningcriteria
2 Isitpossibletoevaluatewhathappensbothwiththepolicyprogramandintheabsenceofthepolicyprogram
3 Arethereoneormoremeasurableoutcomesofinterest
4 Arethereenoughpeopleforrigorousquantitativeanalysis
5 Whatdatawillbenecessaryforthestudyandwhataretheprocessesforaccessingthesedatasource(s)
6 Isthereacompellingreasontoevaluate
Strategicallyengagekeypartners
19
InadditiontoassessingreadinesstoconductanevaluationresearcherscanhelpyouhoneinonthepurposeoftheevaluationTheWorldBankdescribesseveralpotentialusesofevaluationwhicharedefinedhereInbriefevaluationcanhelpdecisionmakers
makeresourceallocationdecisionsrethinkrootcausesofproblemsidentifynewproblemsdecideonabestalternativedeterminewhetherprogramsorpoliciesarebeingimplementedwithfidelitydeterminewhetherprogramsorpoliciesarehavingthedesiredimpact
Researcherscanalsoprovidesupportinsharingtheresultswiththescientificcommunitythroughpapersandconferenceswhichisvaluableincreatingapeernetworkofevidence-basedcitypractitioners
TrainingandtechnicalassistanceUniversitypartnersespeciallythoseatpracticecenterscanbehelpfulindevelopingorupdatingtrainingcurriculumtrainingthetraineranddeliveringtrainingtocitystaff
Strategicallyengagekeypartners
20
ReviewtheevidenceandcollectdetailsOnceyouhaveathoroughunderstandingoftheproblemdataandactorsyouwillneedtoconductyourownresearchonavailableevidenceandlearnaboutthelandscapeofpolicyoptionsYourgoalwillbetodevelopanaccessiblelistofinformationaboutpoliciesthathavehelpedsolvesimilarproblemsinotherplacesThischapterwillguideyouthroughthestepsofpreparingyourreviewersunderstandinglevelsofevidenceandcollectingthedetailsonevidencethatyouwillneedfordiscussionwithdecisionmakersinyourcity
Selectandprepareyourreviewers
AssignapersonorteamtoreviewrelevantinterventionsthathavedemonstratedsuccessinotherjurisdictionsTheirgoalwillbetoproduceareviewofalloftherelevantprogramspoliciesorinitiativesthatyourteamcouldconsiderimplementingTheywillbeconductingaliteraturereviewsupplementedbyinterviewswithexpertsandpeerswhohaveinformationaboutpotentialprograms
SincethepeopleyouassigntothistaskareprovidingthebaselineofpotentialoptionstoimplementitisimportantthatyouchoosepeopleyouareconfidentwillbethoughtfulandthoroughintheirworkItisalsocriticalthattheyareawareoftheimportanceofsourcecredibilityEnsureyourreviewershavefamiliarizedthemselveswithrulesforcriticallyanalyzinginformationsourcesbeforetheygetstartedIftheyareincludingnewsarticlesaspartoftheirsearchtheyshouldalsobeawareofmethodsforevaluatingnewscredibility
AskyourreviewerstodevelopaplanandtimelinetocompletethereviewThisstartswithsolidifyingagoverningresearchquestionOnewaytoknowyouarecreatingadetailedresearchquestionistousetheacronymPICOremindingyoutoincludereferencetothepopulationintervention(ifyouhaveoneinmind)comparisonandoutcomeelementsinyourresearchquestionForexampleifyouknowthatyouhaveapopulationofteenagerswhoarehomelessyouhavelearnedfromyourstakeholdersthatmanyoftheseteenagersidentifyasLGBTandyouhaveheardsomethingabouttheapproachofcreatingdrop-incentersfordeliveringsupportyoucouldask
ForLGBTyouthfacinghomelessness(population)howeffectivearedrop-incenters(intervention)relativetostandardsocialservices(comparison)atsettlingtheminpermanenthousing(outcome)
Ifyoudonothaveaninterventioninmindyoucanleaveouttheinterventionandcomparisonandsimplyask
Reviewtheevidenceandcollectdetails
21
HowcanwehelpLGBTyouthfacinghomelessness(population)obtainpermanenthousing(outcome)
OnceyourreviewershavetheirresearchquestionsusethequestionstocreatealistofkeywordstheycanuseintheirliteraturesearchThiswillhelpkeeptheircollectionofarticlesreasonablyfocusedHavethemdescribewhattheirsearchprocesswillbeincludinglistingthedatabasestheywillreviewandtheirplanforstoringtheinformationtheywillcollect(Herearesomeadditionalsuggestionsforstructuringareviewplan)
WhatyourreviewerswillbedoingisalightweightversionofarapidreviewTotheextentthattheywouldliketodevelopamethodologyforspecificsearchprotocolsincludingcriticalappraisalprocessestheycancertainlygoaheadanddoso
Usedifferentlevelsofevidenceforthemostcompletepicture
AsdiscussedintheWorkingwitharangeofevidencechaptertheNestaStandardsofEvidenceframework--reproducedbelow--ishelpfultounderstandthelevelofconfidencethataprogramorinterventionishavingapositiveimpactThisapproachletsyouputinterventionsthatyouhaveheardaboutlocallybutwhichhavenotbeenstudiedtogetherwiththosewithademonstratedcausaleffectThisishelpfulsinceanumberofpoliciesprogramsandpracticesthatyouwillwanttoconsidermaynotyethavemuchofaresearchrecord
Reviewtheevidenceandcollectdetails
22
NearlyanypieceofinformationaboutaninterventioncanbeclassifiedwithinthestandardsofevidenceForexampleifsomeonehasacrediblehypothesisabouthowaninterventioncouldproducebenefitthistypeofevidencewouldbeclassifiedasLevel1ThistypeofevidencecouldincludeinterventionsinothercitiesthatyoureadaboutinthenewsorhearaboutataconferenceForinstancewhenUSDAallowedschoolsandschooldistrictstoprovidefreebreakfastandlunchtoallstudentsthroughtheCommunityEligibilityProvision(CEP)manydistrictsimplementedthisoptionevenbeforetherewasevidenceofeffectivenessThiswouldfitintoLevel1sinceitbuildsonthelogicthatprovidingaccesstomealswilldecreasefoodinsecurity
YoumayalsofindpiecesofinformationthatqualifyasLevel2evidenceForinstanceasmoreschooldistrictsimplementedCEPprincipalstestifiedaboutpositivechangestheyobservedinstudentssuchasamorepositivedemeanorthatcanleadtobetteracademicperformanceDataalsoshowedthatthestudentsdescribedwerereceivingmoremealsthanbeforetheprogramsimplementationHoweveritwastoosoontobuildevidencetoshowacausalrelationshipbetweentheincreaseinfreemealsandincreasedacademicperformance
ldquoLower-levelrdquoevidencecanconsistofnuancedbutvaluableelementsthatrelatetoaninterventionrsquospotentialfitwithacommunityQualitativeinformationsuchastestimonyfrompeoplewhohaveexperiencedaninterventionandthosewhohavefacilitatedorimplementeditwillalwaysbeusefultoconsiderandprovidesdetailedlocalcontext
Experimentalstudies--describedstartingatLevel3withcontrolledexperimentsandgoingthroughLevel5withfullyfleshedoutprotocolsandprocessesforreplication--stilloccupythehigherlevelsofthisdescriptionofevidenceThisprioritizationshowsthevalueofrigorousstudyforevaluatingtheimpactofaparticularpolicyorprogramratherthanotherfactorsthatcouldbeinfluencingoutcomesofinterestRememberthatmultipletypesofevidencecancomplementrigorousimpactevaluationsandgivecityleadersadditionalinsightsandbroaderperspective
Accessinformation
Encourageyourreviewerstobegintheirsearchforpromisingpracticesingovernmentandnon-profitclearinghousesandinacademicpolicyjournalsWorkpublishedinthesesourceshasalreadybeensubjectedtoahighlevelofprofessionalscrutinyandgenerallycontainsagreatdealofdetailwhichmeansthatyourreviewersaremorelikelytobeabletopullouttheinformationyouneedtomakeagooddecision
Clearinghouses
Reviewtheevidenceandcollectdetails
23
Herearesomeoftheclearinghouses(collectionsofprogramevaluations)forevidence-basedpoliciesandprograms
ResultsFirstClearinghouseDatabaseThePewCharitableTrustsCollectstogethereightotherclearinghousestoprovideinformationoninterventionsinthefollowingcategoriesadultcriminaljusticeeducationchildwelfarehealthjuvenilejusticementalhealthsubstanceabuseandyouthdevelopmentIndividualclearinghousesincludedinResultsFirstare
BlueprintsforHealthyYouthDevelopment-foryouthdevelopmentandviolenceprevention
CaliforniaEvidence-BasedClearinghouse-forchildwelfareinterventions
TheLauraandJohnArnoldFoundationrsquosSocialProgramsThatWork-forsocialservicesprograms
TheUSDepartmentofEducationrsquosWhatWorksClearinghouse-foreducationinterventions
TheUSDepartmentofHealthandHumanServicesrsquoResearch-TestedInterventionPrograms(RTIPs)-forcancercontrolinterventions
TheUSDepartmentofHealthandHumanServicesrsquoTeenPregnancyPreventionEvidenceReview(TPPEvidenceReview)-forsexualriskbehaviorandsexualhealthoutcomes
TheUSDepartmentofJusticersquosCrimeSolutionsgov-forcriminaljusticejuvenilejusticeandcrimevictimservices
TheUSSubstanceAbuseandMentalHealthServicesAdministrationrsquosNationalRegistryofEvidence-basedProgramsandPractices(NREPP)-substanceuseandmentalhealthinterventions
TheUniversityofWisconsinPopulationHealthInstituteandRobertWoodJohnsonFoundationrsquosCountyHealthRankingsandRoadmapsWhatWorksforHealth(WhatWorksforHealth)-healthbehaviorsclinicalcaresocialandeconomicfactorsandthephysicalenvironment
UKWhatWorksCentresTheBritishgovernmenthascharteredanetworkofsubject-specificorganizationswhichcollectsynthesizeandreportoutevidenceontheeffectivenessofpolicyprogramsandpractices
NationalInstituteforHealthandCareExcellence(NICE)-healthandsocialcare
Reviewtheevidenceandcollectdetails
24
SuttonTrustEducationalEndowmentFoundation-educationalachievement
CollegeofPolicingWhatWorksCentreforCrimeReduction-crimereduction
EarlyInterventionFoundation-earlyintervention
WhatWorksCentreforLocalEconomicGrowth-localeconomicgrowth
CentreforAgeingBetter-qualityoflifeforolderpeople
WhatWorksCentreforWellbeing-wellbeing
WilsonSheehanLabforEconomicOpportunities(LEO)UniversityofNotreDameProvideslinkstotheirevaluationsofinterventionsincriminaljusticeself-sufficiencyeducationhealthandhousingandhomelessness
ClearinghouseforLaborEvaluationandResearchUSDepartmentofLaborProvidesresearchonworkforceandlabor-relatedpoliciesandprograms
J-PALEvaluationsAbdulLatifJameelPovertyActionLabStudiesonpoverty-reducinginterventionsfromaroundtheworld
CampbellSystematicReviewsCampbellCollaborationSystematicreviewofpolicyinterventionsbyscholarlycoordinatinggroupsinsevenissueareascrimeandjusticeeducationinternationaldevelopmentmethodssocialwelfareknowledgetranslationandimplementationanddisability
Benefit-CostResultsWashingtonStateInstituteforPublicPolicyAWashingtonState-charteredprogramtoestablishthelikelihoodofrelativecostsandbenefitsforprogramsacrossthefollowingareasjuvenilejusticeadultcriminaljusticechildwelfarepre-Kto12educationchildrensmentalhealthhealthcaresubstanceusedisordersadultmentalhealthpublichealthandPreventionworkforcedevelopmentandhighereducation
EffectiveInterventionsCentersforDiseaseControlandPreventionRepositoryforevidence-informedHIVpreventioncareinterventions(withstepsonhowtoimplement)
Journalsandindexes
OnceyouhavereviewedrelevantinformationaccessiblethroughclearinghousesitisworthtakingalookatindividualstudiespublishedasscholarlyarticlesYourreviewerswilldothisbysearchingarticleindexesandacademicjournalsUnfortunatelyanumberofusefulacademicjournalsandindexingplatformsmaynotbegenerallyaccessibletoyourreviewersifyoudonothaveasubscriptiontothemToaddressthisconsiderpartneringwithalocal
Reviewtheevidenceandcollectdetails
25
collegeoruniversitytoprovideyouwithalistofarticlesfromthefollowingsources(Thisisalsoanexcellentopportunitytoseekresearchsupportfromanyacademicstakeholdersyouidentifiedpreviously)
Indexes
GoogleScholarincludesmostpeer-reviewedonlineacademicjournalsandbooksconferencepapersthesesanddissertationspreprintsabstractstechnicalreportsandotherscholarlyliteratureincludingcourtopinionsandpatentsItallowsyoutospecifyapublicationrangeinyoursearchandprovideslinkstolikelysourcesforalllistedarticles
PolicyFileIndexisauniqueresourceforUSpublicpolicyresearchinthatitgrantsusersaccesstotimelyupdatedinformationfromover350publicpolicythinktanksnongovernmentalorganizationsresearchinstitutesuniversitycentersadvocacygroupsandotherentities
PolicyArchiveworkstocollectanddisseminatesummariesandfulltextsvideosreportsbriefsandmultimediamaterialofthinktankuniversitygovernmentandfoundation-fundedpolicyresearch
Anothersourceforhigh-qualityresearchisthereportsoftheCongressionalResearchServiceYoucanfindtheirreportsaggregatedhere
Specificjournals
Oftenthemostusefulacademicjournalsforunearthingpolicyorprogramreviewswillbeineachspecificsubjectarea--thatisineducationjournalsforeducationpoliciesYoucanuseageneralindexlikeGoogleScholartobegintolearnwhichjournalsaremostoftenusefulforyoursearchHoweveryoucanalsosearchthefollowingtwogeneralistpolicypublicationsforissuesandarticlesrelatedtoyourtopic
PublicAdministrationReviewJournalofPolicyAnalysisandManagement
Newssourcessubjectmatterexpertsandinterviews
AlthoughitisvaluabletoknowwhathasbeenexploredthroughascholarlyresearchframeworkitisalsousefultolearnwhatkindofinterventionsarebeingtriedinthefieldrightnowandhowpeoplefeelaboutthesetrialsSearchsubscriptionnewsaggregatorslikeLexisNexisorfreenewsaggregatorslikeGoogleNewsorYahooNewsforyourissueareaplusthewordpolicyorprogramtoseewhatkindsofinterventionsarebeingcoveredinthenews
Reviewtheevidenceandcollectdetails
26
DonotforgettocontacttheadvisorygroupmemberswhowerechosenfortheirsubjectmatterexpertiseInadditiontoprovidinggoodmaterialthroughinterviewsthemselvestheymayhaveideasaboutothersubjectmatterexpertsyoushouldcontacttoaskaboutpotentialinterventions
Peercitiesandnetworks
InadditiontoreadingabouttheworkofothercitiesitisidealifyouareabletospeakwithpeoplewhoareinvolvedwithcurrentimplementationsofinterestinginterventionsInyourcityyoulikelyhaverelationshipswithyourneighborsbutitcanbehelpfultolookoutsideofyourregionforsolutionsaswellTofindcounterpartsinothercitieswhoareworkingonaprogramyouwouldliketolearnmoreaboutyoucancontactstaffinintermediaryorganizations(likeGovExoryourprofessionalassociations)whomightbeinapositiontoputyouintouchimmediatelyorwhomightseeanothergoodrouteforsettingupaconversationHereisalistofnetworksthatmaybehelpful
InternationalCityCountyManagementAssociation(ICMA)NationalAssociationofCounties(NACO)NationalLeagueofCitiesUSConferenceofMayorsWhatWorksCities
Collectthedetails
Onceyourreviewershavefoundevidenceonpossiblepoliciesorprogramstoimplementtheyshouldcapturekeydetailsthatwillhelpdecisionmakersunderstandthepotentialeffectivenessandapplicabilityofeachpotentialintervention
Foreachsourcethathasprovidedinformationyouwanttoconveyusethetemplatebelowtocaptureasmanyofthefollowingdetailsaspossible(Youdonotneedtocompletethisforsourcesthatprovideduplicativeinformationorareotherwisenotthatuseful)
Programorpolicyname (egldquoHousingFirstrdquoorldquoblightedpropertiesregistryrdquo)
Sourcetype (egjournalarticleinterviewsubjectmatterexpertstudyassociationcommunityparticipationprocess)
Sourcedetails (egarticletitleandpublicationnameandtitleofinterviewee)
Evaluationmethodology
(egrandomizedevaluationcomparativeevaluationoftwodifferentgroupsbeforeafteranalysis)
Evidencelevel (egLevel12345)
Programorpolicy
Reviewtheevidenceandcollectdetails
27
description
Programorpolicygoal (1sentencecapturingtheprincipalaimoftheintervention)
Existingimplementations
(WherehastheinterventionbeentriedListasmanysitesasyouareawareof)
Characteristicsofimplementationsites
(ForeachimplementationsiteofferbasicgeographicIDmdashcityandstateThenyouwillwanttocollectthedetailswhicharerelevanttoyourdecisionmakersrsquoconsiderationsaboutwhatmakesalocationldquosimilartordquoorldquodifferentfromrdquoyoursituationYouwillatleastwanttoincludeforeachimplementationsitePopulationnumberPopulationdensity(ruralurbansuburban)Socio-economiccharacteristicsBeforestartingaskyourdecisionmakersaboutanyparticularcharacteristicsofprogramimplementationsitestheywouldwanttoknowbeforeseekingtoreplicate)
Characteristicsofimplementationpartners
(Anyavailabledetailsaboutorganizationsinstitutionsorpeoplewhowereinvolvedwithimplementingtheintervention-egindependentchildcarecenterpublicelementaryschoolcitypublichealthnurses)
Characteristicsoftargetpopulationeligibleprogramparticipants
(Anyavailabledetailsaboutpeoplewhoarethebeneficiariesoftheprogramorotherwiseincludedoraffectedbytheintervention-egnumberofprogramparticipantscategoriesofpropertyownersaffected)
Policyorprogramduration
(HowlonghastheinterventionbeeninplaceineachimplementationsiteHasitendedandifsowhy)
Whatbenefitscamefromeachimplementation
(Anyobservedpositiveeffectsoftheinterventionincludingeffectsize)
Statisticalestimatesofinterventioneffectiveness
(Anyavailablestatisticalestimatesoftheinterventionrsquoseffects)
Whatdrawbackswereobserved
(Anyavailabledetailsaboutunintendedorunavoidablenegativeeffectsoftheintervention)
Programorpolicycosts
(Anyavailabledetailsaboutprogramcostsperimplementation)
Reviewtheevidenceandcollectdetails
28
Reviewtheevidenceandcollectdetails
29
DecideonapolicytoimplementbasedonlocalfactorsTogetthecriticalbuy-infromleadershiptoimplementanewpolicyorprogramtheprojectleadshouldscheduleameetingwithdecisionmakerstopresenttheevidencesurroundingthepolicyoptionscapturedinthetemplateDependingonyourinsightintoyourcityleadershipswillingnesstomakechangeyoumaywishtosharetheresearchyouhaveconductedontherootcausesoftheproblemandsomeofyouranalysisontheimpactofcurrenteffortstodate
Considertheapplicabilityofanapproach
AsdiscussedinthepreviouschapterpolicymakersoftenlookforevidenceofeffectiveprogramsinothercitiesbutoftenthequestionarisesofhowgeneralizableaprogramisfromonecitytoanotherTheAbdulLatifJameelPovertyActionLab(J-PAL)suggeststhatinreviewingevidencefromanotherlocationitishelpfultoidentifythespecificmechanismthatwasused(deliveryofprogramorserviceinformationsharingetc)identifythelocalconditionsthatwereinplaceobservethebehaviorchangethatresultedfromthemechanismandfinallytounderstandlocalcontextandconditions(legalorcostconstraintsculturalnormsetc)Thissetofpresentationelementscanbedrawnfromthelistofdetailscollectedintheresearchtemplate
EducationpolicymakersinChicagoILusedfindingsfromprogramsinKenyaandIndiainwhichchildrenwhohadfallenbehindtheirpeersweregiveneducationalsupporttargetedtotheirleveloflearningThemechanismusedwastargetedteachingandthereweresimilarconditionspresentinallthreecountriesAlthoughsomestudentshadfallenbehindinthecurriculumteacherswereincentivizedtoteachgrade-levelmaterialsConsiderationsofhowthelocalcontextinChicagomightaffectimplementationofatargetedteachingprogramincludedunderstandinghowtheneighborhoodsthatthestudentswerefrommayhaveaffectedtheireducationaloutcomesandthehighcostofteachinginsmallgroupsInChicagoMatchEducationprovidedtwo-on-onetutoringwhichmatchedstudentswithwell-educatedindividualswhohavenothadformalteachertrainingforalower-costmodelTheresultswerepositiveStudentswhoweretutoredperformedbetterinmathandweremorelikelytopasshighschoolclasses
ThegeneralizabilityframeworkproposedbyJ-PALisbelowUsethisasatemplateasyouconsiderpotentialpolicysolutions
Decideonapolicytoimplementbasedonlocalfactors
30
SourceTheGeneralizabilityPuzzle
OnceyouhavenarrowedthepolicyoptionsdiscussthequestionsbelowforeachoptionsyouareconsideringThisstepiscriticalasunderstandinglocalcontextissoimportantinsuccessfulpolicyimplementation
CommunityContextQuestionnaire
Localcontext
DoesthisoptionmakesenseinourcommunityIsthisaonesizefitsallapproachIsanyparticulargroupabsentfromthisapproachIfsoisthatok
Youranswers
Effectiveness
WhatwouldweexpecttoobserveifthepilotisaseffectivehereasinCityX
Youranswers
Timing
DoesthetimingmakesenseIsthistherighttimetoimplementthispolicychange
Decideonapolicytoimplementbasedonlocalfactors
31
WherearethepotentialroadblocksthatcouldaffectthescheduleIsthereanywaytomitigatethisHowlongwouldittaketoseeanimpact
Youranswers
Implementation
DowehavethestaffcapacitytoimplementthisArethereadditionalpartnersthatyoucanbringtothetabletosupportimplementationDoesthismakesensegivenourformofgovernment
Youranswers
Authority
DoesourcityhavetheauthoritytoenactthisIfsowhatistheappropriatemechanismAretherelegalbarriersArethereanyexistingpreemptionissuesfromthestateorfederalgovernmentCanthosebeaddressedthroughlegislationadvocacyIstherepoliticalsupportoroppositiontothis
Youranswers
Cost
HowmuchwillthiscostannuallyHowwillwepayforitCanthepolicybeimplementedwithexistingfunding
IfsoisthisfundingreliableHowwouldthispolicyoptionaffectourcitysbudgetintheshortandlongtermWhatotherfundingoptionscanweconsiderIsthereanalternativethatwouldcostlessandgiveussimilarresultsWillanyconstituencybeadverselyaffectediffundingforprogramsthatimpactthemisshiftedtoanotherprogrampolicyorinitiative
Youranswers
Decideonapolicytoimplementbasedonlocalfactors
32
EstablishaframeworktomonitorprogressNowthatyouhavereviewedtheevidenceconsideredlocalcontextandchosenyourpolicyitistimetodeterminehowyourcitywillmonitorprogressofthisnewapproachCitystaffandleadersneedtoestablishtheinfrastructureforongoingcontinuousdecisionmakingbasedonwhetherthepolicychangeisleadingtothedesiredoutcomesforresidentsandwhetherthechangeismeaningfulandrationaltofrontlineworkersSettingarbitrarygoalsatthelocallevelthatconflictwithgoalsatotherlevelsisconfusingtotheindividualsresponsibleforachievingthegoalsGoalsthataresettoohightoachievecanbedemoralizingtostaff
ThissectionincludesstepstosetupavenueinwhichCitystaffandleadershiproutinelyvisitkeyindicatorstodeterminewhatisorisnotworkingandtomakedecisionsinreal-timeFormorereadingonperformancemanagementpleasevisittheGovExPerformanceManagementGettingStartedGuide
InperformancemanagementtermssuchasindicatormeasureandmetricmaybeusedinterchangeablyInthisguidewewilluseindicatorInyourworkthemostimportantthingistobeconsistentinwhichevertermyouuseThisvideomaybehelpfulinunderstandingthisconceptandcommunicatingittoothers
Establishthefundamentalsandgatheryourteam
FirstdeterminewhetherthereisanestablishedvenueorregularmeetingscheduleduringwhichyoucandiscussprogressIfyourcityhasanexistingperformancemanagementstructureonarelatedtopicaddingadiscussionoftheprogressofpolicychangetoyouragendacanbeagreatideaIfthiswillbeastandalonemeetingdecidehowfrequentlytheprojectteamanddecisionmakerswillmeetandputthosemeetingsonthecalendarfortheentirelengthoftheproject
DeterminewhowillstaffandattendroutinemeetingsonCityprogressThesemeetingsmayincludetheMayorandorCityManagerdepartmentheadstaffwhoareimplementingthepolicychangeandanalysts
Identifyindicators
ItiscriticalthatindicatorsbeanimportantcityandcommunitypriorityaswellassomethingtheCityplaysamajorroleinaddressingTheremaybeinstancesinwhichtheCitywillneedtoreachouttoorcollaboratewithexternalpartnerstogetamorecompletepictureofprogressontheoverallpolicychange
Establishaframeworktomonitorprogress
33
Anoutcomeindicatordescribesthedesiredeffectofthetargetedpopulation
ExampleBellevuesfamilyfriendlyneighborhoodsindicatormeasuresthepercentofresidentswhobelievetheyliveinaneighborhoodthatsupportsfamiliesparticularlythosewithchildren
WithyourkeypartnersrevisityourproblemstatementcurrentstateandfuturestateexerciseandbegintoidentifytheoutcomeindicatorsthatwillillustratesuccessFornowchooseasingletop-levelkeyperformanceindicator(KPI)ItiscriticalthattheKPI-thepieceyouwillmeasure-beapriorityforthecommunityandsomethingtheCityplaysamajorroleinaddressing
Outputindicatorsmeasuretheprocessofprogramorservicedelivery
ExampleSanDiegoMayorKevinFaulconersgoaltorepair1000milesofstreetby2020
ReviewingtheadministrativedatasetinventorybegintoparedowntherelevantprogramsanddatasetsthatsupportyourKPIWhatarethemeasuresofeffectivenessforeachprogramTheseareyouroutputindicators
Thediagrambelowillustratestherelationshipbetweenoutputandoutcomeindicators
ThinkaboutimplementationindicatorsasmeasuringstepsintheprogramsrolloutthatarenecessarytoobtaindesiredchangesImplementationoutcomescanfocusontheinterventionstaffdeliveringtheinterventionanorganizationorsystemandcanaddressfeasibilityadaptabilitysustainabilityandcostTwokeyquestionstoconsiderwhendevelopingimplementationindicatorsare
1 Istheinterventionacceptableandappropriatetothecommunityand
2 IsitfeasibletoimplementinterventionXatthistimewiththeseresources
Establishaframeworktomonitorprogress
34
Amoreextensivelistwithdefinitionsofpotentialimplementationoutcomesislinkedhere
UsethetablebelowtochoosethedesiredimplementationoutcomesforyourpilotandtoplanyourmeasurementtechniqueThisshouldincludelevelofanalysis(individualproviderorconsumerorganizationorsetting)yourmeasurementtechnique(surveysobservationinterviewfocusgroupsadministrativedataanalysis)andthephasewhenyouwillcollectthisinformation(earlymidlate)Theremaybemorethanonelevelofanalysismeasurementtechniqueandphaseofdatacollection
Implementationoutcome
Definitionofoutcome
Levelofanalysisandmeasurement
Yourmeasurementtechnique
Phaseofpilotwhenyouwillcollectdata
Usethespacebelowtocaptureyourindicatorwishlist
Indicatorwishlist
Outcomes_____________________________________________________________________________
Outputs________________________________________________________________________________
Implementationoutcomes______________________________________________________________
Setapublichigh-levelgoal
Thehigh-levelgoaliscriticalinestablishingdirectiontogalvanizethepeoplewhowillbedoingtheworkWhendiscussingthegoalusetheMayorsbullypulpitLookforkeymilestonesinCitygovernmenttosharethegoalstatementsuchasthebeginningofaMayoraladministrationaStateoftheCityspeechorotherpublicremarks
GoalsshouldbeldquoSMARTrdquo
Specificaspossible(innamingthepopulationwhowillbebetterofftheactionthecitywilltake)
Measurable
Achievable
Establishaframeworktomonitorprogress
35
Results-focused
Time-bound
ThegoalstatementshouldincludeyouroutcomeindicatorlikeChattanoogasgoalldquoWewillendchronicveteranhomelessnessbyDecember2016rdquoAdiagramtohepyoubuildaSMARTgoalaroundyouroutcomeindicatorisbelow
Component Description Yourturn
Specific WhowillbebetteroffWhatactionthecitywilltake
Measurable Howwillyoumeasuresuccess
Achievable Isthegoalattainable
Results-focused Whatistheexpectedresult
Time-bound When
Setrealisticambitioustargetsandtimelinesforeachindicator
ToestablisharealisticandambitioustargetforyourgoalstatementusetheadministrativedatacollectedforyourchosenKPItoreviewactualCityperformanceinthemostrecentyearcomparedwiththethreeorfourmostrecentyearsworthofdataBegintocalculatesummarystatistics(egtotalminimumaveragemedianmaximum)toidentifytrendsCreatecumulativeandmonthlychartstoidentifytrendsandanomaliesBasedonyouranalysisofhistoricaldataignoretheanomaliesandassumetheproductivitytrendswillfollowthesameoverallpatternfrommonth-to-monthFinallyapplyanincreasedproductivityfactortotheperformanceeachmonthtoforecastfutureprogressBasedonthisanalysischooseatargetthatinspiresimprovementwhileaccountingforreality
Manycitiesthataremorematureintheirperformancemanagementpracticesrecognizetheimportanceoftakingacommunity-levelapproachtoimprovingoutcomesforresidentsThesecitiesarecreatinggoaltaxonomiesthatincludemeasuresrelatedtopublicsafetyhousingandevenrecreationCitiesdriveprogressonthesegoalsbyimplementingstrategiesthatengagemultipledepartmentsandcollaboratewithserviceprovidersinthecommunity
EventuallyyouwilldevelopgoalstatementsfortheoutputsinordertocapturetheefforttoachievetheoutcomecontainedinyourgoalstatementYoursetofindicatorsshouldbecomposedofshort-andinterim-termsupportivemeasureswithaccuratetargetsandrelevanttimelinesAnexamplegoaltaxonomyfromtheCityofLasVegassResultsVegasprogramisbelowUsethistaxonomyasamodelforyourownsuiteofindicatorsandgoalstatements
Establishaframeworktomonitorprogress
36
Yourturn
Thetemplatebelowismeanttohelpyoucraftagoalstatementforeachsupportingmeasure
Goalstatement
Departments
Outcome
Activities
KPI
Target
Timeframe
Conductanddiscussrelevantanalysis
ThrougharoutineseriesofmeetingscontinuallymonitorprogressofKPIssupportingindicatorsandanyotherdatacollectedsuchasqualitativedatatoidentifytrendsthatwillindicatewhetherthechosenpolicyishavingthedesiredimpactThesedatapointsshoulddrivethediscussionduringthesemeetingspointingtoopportunitiesforfurtherinquiryinto
Establishaframeworktomonitorprogress
37
whatisdrivingtheobservedtrendsAnalystsshouldbeencouragedandtrainedtoaskquestionsofthedepartmentsproducingthedataandconductsitevisitswhenpossibletodeepentheirunderstandingofthedataPresentationoftheanalysisshouldfocusononeideaatatimetokeeptheconversationfocusedonkeypointsandspecificopportunitiesforimprovementYoumayconsiderdevelopingapublicfacingdashboardtosharethisprogressmorebroadly
AgeneralcomparisonofthestatusquodatawithwhatwasobservedduringthepilotcanprovidedecisionmakerswithaquickunderstandingofthedifferencesbetweentheoldandnewapproachesalthoughthiswillnotshowcausalityOftenthesecomparisonswillincludeoperationalmetricssuchasdaysordollarssavedbutrememberthattheimpactonpeoplewillbemostimportantIfyouareworkingwithanevaluationpartnertheywillsupportthisstep
Usethespacebelowtosummarizetheresultsofyouranalysis
MeasureTargetvalue(seesectiononsettingtargetsabove)
Pilotvalue(whatdidyouobserveduringthepilot)
Historicalaverage(ifavailable)
Results(WhoisbetteroffIsanyonenegativelyimpacted1-2sentencesreassessingyourgoalstatement)
DocumentyourprocessandprogressincludingfeedbackfromtheparticipantsimplementationpartnersandtheprojectteamDidanythingunexpectedoccurduringthepilotArethereinternalandexternalfactorsthatmayaffectyouroutcomesThisiswheretheimplementationoutcomescanbecanhelpfulIftheinterventiondidnotgoasplannedconsiderwhetheritisbecausetheinterventionwasnotacceptableorfeasible
UsethetemplatebelowcreatedbyDenversPeakAcademytocapturethisinformationIfthe306090daytimelinedoesnotmakesenseforyourcityfeelfreetocustomizeit
CurrentState
FutureState
30Day
60Day
90day
Whatwentwell(+)
Whatdidnotgowellshouldbechanged(-)
IfaquestionordiscussionpointarisesthatcannotbeansweredinthemeetinganassignedstaffmembershouldcapturethequestionduringthemeetingAfterwardthefollowupitemsshouldbesenttorelevantpartnersandthenextmeetingshouldbeginwithadiscussionof
Establishaframeworktomonitorprogress
38
thoseitems
Takeaction
DiscussionsaboutwhetheryourindicatorsaremovingintherightdirectionandwhatmightbedrivingtrendsmayleadtoadiscoverythattherearenotenoughresourcesallocatedtothispolicyorissueTheremaybelimitedfundingforstaffprogrammingortechnologythatisdrivingthedatainthewrongdirectionIfCityleadershipisintheroomthisisagreattimetomakethecaseforadditionalfundingduringthebudgetseasonBycollectinganalyzinganddiscussingdataonthispolicyroutinelyyouarebuildingevidencetomakethecasefortherightleveloffundingtoreachtheestablishedtargets
Establishaframeworktomonitorprogress
39
Assessdataaccessibilityqualityandsecurity
Determineaccessibilityandqualityofdata
OnceyouhavecompiledyourwishlistofindicatorsitistimetodeterminetheaccessibilityandqualityofthedataTopreventcomplicationyoumaywishtoprioritizerelevantindicatorsforwhichyouarealreadycollectingdataAvoidthetemptationofadoptingatechnologysolutiontocollectnewdataatthispointsuchasaphotosurveytoolorotherappFornowyouaretryingtodeterminewhichdatasetsarerelevantaccessibleandhighqualityBesuretoworkwithyourkeyinternalpartnerswhileyouareidentifyingyourmeasuresDepartmentstaffwhoworkonprogramsarefamiliarwiththeindicatorsthatwillaccuratelysupporttheoutcomesanddatabaseadministratorswillbevaluablewhendiscussingaccessibilityandqualityofthedata
Abrieflistofconsiderationsfortheaccessibilityofyourdataisbelow
ArewecurrentlycollectingthisdatasetorwillwebecollectingitforthefirsttimeHowfarbackdoestherecordgo(timehorizon)WouldweneednewtechnologytocollectoranalyzethedataCanthedatabecollectedmanuallyorcanitbeautomatedHowfrequentlyisthedataproduced(continuouslyhourlydailyweeklymonthlyquarterlyyearlyetc)IsthedataavailablefortheentireCitypopulationorasubsetofthepopulationHowgranularisthedata(individuallevelCensusblockneighborhoodcitywideetc)DoesthedatacontainsensitiveinformationIsthereaproxymeasurethatwecanusefornowIsthedatamachinereadable
AsnotedbytheWilsonSheehanLabforEconomicOpportunities(LEO)attheUniversityofNotreDameleveragingadministrativedatacansignificantlyreducethecostsandtimelinesofrigorousevaluationCitiescanestablishaninternaldatasharingagreementacrossdepartmentsincludinglanguagethatenablesadministrativedatatobeusedforresearchpurposesCitiescanalsoidentifyexternaladministrativedata-holderssuchasacountyschooldistrictpolicedepartmenthospitaloranotherstatethatmayhaveoutcomedataimportantformeasuringprogramimpactCitiescanleverageexistingrelationshipsandorworkwithresearchinstitutionstodevelopstreamlineddatasharingprocessesandagreementswiththeseexternalentities
Assessdataaccessibilityqualityandsecurity
40
Evaluatethequalityofyourdata
QualityisasimportantasaccessibilityToevaluatedataqualityyoumaywishtocreatearubricliketheonebelow
Value ValueDescription
3 ExcellentDataQualityThedataiscompleteaccurateandupdatedregularly
2 AdequateDataQualityThedataissomewhatcompleteaccurateandupdatedfairlyregularly
1 PoorDataQualityThedataisnotcompleteaccurateorupdatedregularly
ThisisagreattimetobuildonthedatainventorythatyoustartedinthegatherinformationpageSuggestedadditionalfieldstoaddtoyourdatainventorytemplatearebelow
Program DatasetTitle
DatasetDescription DataSource
Departmentorexternalpartner
DoestheCityalreadycollectthisdata
Whatisthenameoftheprogram
Human-readablenameofthedataset
Shouldbeunderstandablebynon-technicalusers
Whatistheoriginalsystemapplicationfilethathousesit
Departmentorexternalpartnerthatmaintainsthedata
Isthisdataalreadycollected
Updatefrequency-ThefileformatforthedatasetTypicallycsvxlsxshapefiletxtetc
Format-Howoftenthedatasetisupdated
Timerange-Theperiodoftimecoveredinthisparticulardataset
Datause-Howisthisdatasetcommonlyused
Datausers-Whotypicallyusesthisdataset
Dataownerstitle-Titleofthedataowner
Dataownersposition-DataOwnersjobtitle
Dataownersemailaddress-DataOwnersemailaddressWhomanagesthedataandorisresponsibleforgrantingpermissiontoaccessthedataWhounderstandswhatthedatasetincludesandcananswerquestionsaboutit
Dataqualityconcerns-Describetheconcernsyoumayhaveaboutthequalityofthedataset
Assessdataaccessibilityqualityandsecurity
41
Division-DivisionwherethedataismaintainedIfmorethanonedivisionisresponsiblelisttheprimaryownerofthedataset
Sensitivedatacomments-DescribetheissuesthatwouldpreventthecityfromsharingthedatapubliclyCommonsensitivityconcernsincludeprivacyviolationssecurityissuesorhighcostorstaffdemandstopostthedata
Publishingstatus-Thistheplacetoputanycommentsregardingthecurrentstatusofthedataset
Publishedlink-Thisisthelinktothepublisheddatasetontheopendataportalorcitywebsite
Reviewdatasecurityandprivacyprotectionprotocols
OftenapolicychangealsoinvolvesachangeinthedatathatisbeingcollectedhowthatdataisbeinghandledwhohasaccesstoitandahostofotheradjustmentsAsyoutakestockofthedataassetsthatareavailableforyourinitiativeitisvitaltoalsothinkthroughwhatwillbedonetoprotectthatdataandtheprivacyofthepeopleitrepresentsAbigpartofgooddatasecurityinvolvesstrongdatagovernancepracticesthatdefineandidentifysensitivedataandconsciouslyworktobalancethevalueandriskofstoringsuchsensitivedataOnewayofevaluatingyourdatacomesfromcaselawrelatedtotheUSFreedomofInformationActwhichwehavetranslatedtothechecklistbelow
1 Woulddisclosureresultinasubstantialinvasionofprivacy2 Haveweconsideredtheextentorvalueofthepublicinterestandthepurposeorobject
oftheindividualsseekingdisclosure3 Istheinformationavailablefromothersources4 Wastheinformationgivenwithanexpectationofconfidentiality5 Isitpossibletomouldreliefsoastolimittheinvasionofindividualprivacy
TheseriskscouldbemitigatedusingavarietyofstrategiesfromaggregatingdatatoobfuscatingindividualstoredactingfieldsthatmayincludePersonallyIdentifiableInformation(PII)orre-evaluatingyourdatacollectionmethodstofindalternativewaysofgettingtheinformationyouneedThesestepsshouldbecodifiedandmanagedbyadatagovernancecommitteeifpossibleForthesensitivedatathatisdeemedessentialcreateaplanaroundpotentialdatabreachesinordertoquicklyandconfidentlyrespondtoemergencysituations
ProactivelypreparingforthesesituationsisoftenthebeststrategyfordatasecuritybutdonotforgetthetechnicalsideInvolveyourITadministratorsinconversationsaboutdatastorageandaccessensuringthattheywillbeprotectedbyenterprise-gradecybersecuritysystems
Assessdataaccessibilityqualityandsecurity
42
Assessdataaccessibilityqualityandsecurity
43
Designimplementationandevaluation
Designtheimplementation
NowthatyouhaveyouraccountabilityplanandkeyindicatorsitistimetogetspecificaboutyourplanforimplementationThetemplatebelowwillhelpyouthinkthroughthespecificmechanismsofimplementingpolicychange
UnderstandingorganizationsandsystemsiscriticaltounderstandingandmakingdecisionsaboutwhereandhowinterventionsshouldbeappliedApolicyimplementedinasinglesettingmayshowpromisingresultsbutimplementingmultiplepoliciesandprogramsatthesametimewithdifferentactorsresponsibleforeachaspectcanhelpensuresustainablebehaviorchange
InSomervilleMAShapeUpSomerville(SUS)isacitywidepublichealthprogramthatisimplementedacrossmultiplelevelsofinfluenceandsystemsthroughinterventionsbeforeduringandafterschooltoincludechangestobuiltenvironmentsuchasbikelanesregionalmasstransportationandphysicaleducationequipmentinschoolsandgymstoincreasephysicalactivitywellnesspoliciesandschoolgardensnutritioneducationduringandafterschoolandhealthyrestaurantprogramstosupporthealthyfoodandhealthyeatinghabits
Usethetemplatebelowtoplanyourimplementation
Policyimplementationplanningtemplate
ListthespecificdesiredoutcomesInthepreviousstepyouworkedwithdepartmentstafftoestablishspecificmeasurableoutcomesandinterimindicatorsPrioritizetheindicatorsthatwilldemonstrateprogressUsethespacebelowtocapturethedesiredoutcomesofthepolicychange
ExampleShapeupSomervillebeganwiththegoalofpreventingobesityin1stthrough3rdgraders
Yourturn
____________________________________________________________________________
____________________________________________________________________________
Designimplementationandevaluation
44
SpecifythetargetpopulationAreyoudesigninganinterventionthattargetsonlycertainsitesAreyouplanningonimplementingaprogramorinitiativeforaspecifictargetaudienceClearlyspecifythepopulationyouplantotarget
IfyouplantoconductarigorousimpactevaluationasnotedbytheWilsonSheehanLabforEconomicOpportunitiesitisimportanttoestablishconcreteobjectiveeligibilitycriteriaandensurethatthesecriteriaareappliedconsistentlybyallstaffduringscreeningandintakeItisalsovitaltoidentifytheindividualswhoareeligibleforaparticularprogramorserviceevenifeveryoneeligiblecannotbeservedbyexistingresourcesThishelpscitiesidentifyanyunmetneedsandguidefutureresourceallocationwhilealsocreatingthepossibilityofarigorousimpactevaluationbyidentifyingapotentialcomparisongroupFinallywhenconductingscreeningandintakeitisimperativetocollectandmaintainbasicinformation(egnamedateofbirthgender)aboutallprogram-eligibleindividualsnotjustthosewhoparticipateintheprogram
ExampleIntheShapeUpSomervilleexampletheprogrambeganbytargeting1stthrough3rdgraders
Yourturn
____________________________________________________________________________
____________________________________________________________________________
DefinethelocationWherewillthepolicychangetakeplaceWillimplementationtakeplaceatonesiteormultiplesites
ExampleShapeUpSomervilleinterventionsweredeliveredinschools
Yourturn
____________________________________________________________________________
____________________________________________________________________________
ClarifyimplementationpartnersandarticulatetheirrolesWhatarethekeyorganizationsinstitutionsorpeopleresponsibleforimplementationTheprojectteamshouldbeasclearandspecificaspossibleinprovidingadescriptionoftheconcreteactivitiesthatimplementationpartnersaremeanttocarryoutPracticeprofilesmaybeahelpfultooltohelpdefinewhatisrequiredoftheactorswhoareresponsibleforimplementingtheintervention
Designimplementationandevaluation
45
ExampleInSomervillemayoralleadershipandengagementofmultiplecityagencies(publichealthprimarycareschoolsCambridgeHealthAllianceetc)streamlinedandconnectedthevariousShapeUpSomervilleinitiativesandgrantswhichhelpedtokeepthetalentandenergyfocusedonhealthyeatingandactivelivingforallsectorsofSomervilleSUSpartnersultimatelyincludedschoolsnonprofitsCityandstateagenciesandnationalfoundations
Yourturn
____________________________________________________________________________
____________________________________________________________________________
DescribetheinterventionWhatarethestepsactivitiesorprocessesRememberthatthemechanismthespecificstepsorprocessthatleadtobehaviorchangeiskeytounderstandingtheeffectivenessofthepolicychangeandsharingtheresultswithothers
ExampleGoalsofShapeUpSomervilleweretoinfluenceeverypartofanearlyelementaryschoolchildrsquosdayndashbeforeduringandafter-schooltopromotehealthyeatingphysicalactivityandtheirweightstatusInterventionincludedfoodserviceenhancementstoimprovebreakfastandlunchandstaffprofessionaldevelopmentwalktoschoolactivitiesSUSclassroomcurriculumandprofessionaldevelopmentschoolwellnesspolicydevelopmentSUSafter-schoolcurriculumandprofessionaldevelopmentoutreachandeducationtohomethroughmaterialsforumseventsldquoSUSapprovedrdquorestaurantscommunityoutreachandcapacitybuildingthroughpolicydevelopmenttrainingsmediaplacements
Yourturn
____________________________________________________________________________
____________________________________________________________________________
ClarifytimelineandmilestonesfordeliveryandevaluationHowfrequentlywilltheservicesinterventionbedeliveredHowlongwilltheprogramrunWhatisthetimetoexpectedimpacts(eg12monthsafterprogramcompletion)
Designimplementationandevaluation
46
ExampleShapeUpSomervilleledbyTuftsUniversityFriedmanSchoolofNutritionScienceandPolicyconductedaquasi-experimental(nonrandomized)interventionstudyamong1stthrough3rdgradestudentsinSomerville(intervention)andtwocomparisoncommunitiesfor9-monthsduringoneschoolyear(fall2002ndashspring2003)ThestudyshowedpositiveassociationoftheinterventiononchangeinBMISUSreleasedreportssummarizingevaluationsprogresstodateplansforadditionalprojectsandimpactstenandfifteenyearsintotheinitiative
Yourturn
____________________________________________________________________________
____________________________________________________________________________
PlanyouranalysisTheprevioussectionaddressedestablishinganaccountabilityframeworktomonitorprogressandimplementationDeterminewhichindicatorswillbediscussthroughthatprocesswhowillsharethedataandanalysisandhowfeedbackfromthoseconversationswillbeincorporatedbytheprojectteam
ExampleTheMayorrsquosofficeimplementedadata-drivendecisionmakingapproachcalledSomerStatwhichwassynergisticwiththeapproachestakenbytheresearchandevaluationteamsdeployedforthespecificgrants
Yourturn
____________________________________________________________________________
____________________________________________________________________________
PlanforsustainabilityIfthepolicyandprogramchangeissuccessfulhowwilltheprojectteamsustainandexpandtoadditionalsitesorpopulations
ExampleAlthoughmostofthepartnershipsforgedintheearlyyearsweremaintainednewgrantsandcollaborationsexpandedattentiontoolderpublicschoolstudentsactivetransportationandtheGreenLinesubwayextensionthebuiltenvironmentfarmerrsquosmarketeffortssuchasldquoGrowninSomervillerdquowithUnionMainStreeteconomicdevelopmentandhealthdisparitiesaswellasdatacollectionmonitoringandsurveillance
Yourturn
Designimplementationandevaluation
47
____________________________________________________________________________
____________________________________________________________________________
Pilotplanningtemplate
IfyouplantoconductapilotfirstdeterminethepurposeofthepilotDoyouwanttoobservewhetherthepolicychangeresultsinthedesiredbehaviorchangeAreyouconcernedaboutfeasibilityscalabilityorcostDoyouwanttotestwhetheryourdesiredresultscanbeachievedthroughthispolicyapproachClarifythegoalofthepilotandthequestionsyouhopetoanswerThiswillhelpyouidentifythekeyindicatorsthatyouwillusetomeasurewhetherthepilotissuccessfulBeforebeginningestablishabaselineofcurrentperformanceandsetyourtargetsaccordinglybasedonwhatyouhopetoaccomplishthroughthepilot
NextestablishthepilotplanforyourprojectteamincludingmilestonesforteamcheckinstodeterminewhethertomoveaheadorchangecourseThisisseparatefromthemeetingsthatwilloccuraspartoftheaccountabilityframeworkUsethespacebelowtocreateyourtimelineanddevelopyourmeetingschedule
Task Personresponsible Startdate Checkindate Deliverable
Planyourevaluation
WhileyouaredesigningtheimplementationitisalsotimetobegintodevelopyourplanforanalysisAsnotedbytheWilsonSheehanLabforEconomicOpportunitiesdecisionsaboutprogramsetup-includingoutreachscreeningintakedatatrackingandenrollment-candeterminewhetherornotarigorousimpactevaluationwillbepossibleAsdiscussedcityleadersmayfindithelpfultoincludeevaluationpartnersinearlyconversationsaboutprogramsetupDuringpreviousstagesyoumayhaveworkedwitharesearchertoplanyourproposedpolicychangeYoushouldbeclearaboutthepurposeoftheevaluationanddevelopanevaluationplanBelowareadditionalconsiderations
Causationvstrendanalysis
IntheEstablishaframeworktomonitorprogressstagetherewasguidanceontrackingtrendsovertimeItisimportanttonoteherethatmonitoringdataisdifferentfromanalyzingacausaleffectTrendanalysiswillillustratethatchangesareoccurringandmayleadtoquestionsthatcouldbeansweredbyyourevaluationbuttrendanalysisalonedoesnotprovideevidenceaboutwhatisdrivingthosechangesYoumayseechangesinyour
Designimplementationandevaluation
48
outcomeindicatorsbasedonvariablesoutsideoftheCityscontrolForinstanceachangeinfederaltradepolicycanhaveanimpactemploymentratesinyourCityoutsideofyourCitysnewjobtrainingandplacementprogram
ToaddresscausalityyoumustcomparetheoutcomesofindividualswhoreceivedtheinterventionwiththosewhodidnotWithoutacomparisongroupofsimilarindividualswhowerenotexposedtotheinterventionitisimpossibletoknowwhichfactorsareresponsiblefortheoutcomesRandomizedControlTrials(RCTs)inwhichindividualsarerandomlyassignedtotreatmentandcontrolgroupsarethegoldstandardinprovingcausalityIfyoucandemonstratethatoutcomesimprovedforthosewhoreceivedtheinterventionanddidnotimproveatallorimprovedlessforacomparisongroupyoucanbesomewhatcertainthattheinterventionwasthecause
Examplesofothernon-causalimpactevaluationsthatarerigorousandreliableincludecertainquasi-experimentalstudiesusingcomparisongroupdesignssmartinterruptedtimeseriesanalysesthatmodeltrendsovertimeinwaysthatallowestimationofinterventioneffectswelldesignedpre-posttestanalysesneedsassessmentsimplementationstudiesetcFormoreinformationondifferenttypesofevaluationsseetheSageResearchMethodswebsite
Lookfordifferencesingroups
BuildintimetoalsoexaminethepilotdatatoseeifgroupsareexperiencingdifferentresultsReviewingperformancebasedonoutcomemeasuresforapopulationinaggregatemayhidethesubtlewaysthatsomepeoplemaybeaffecteddifferentlysoitisnecessarytocarefullyanalyzethedatafordisparateoutcomesForinstancetheMovingtoOpportunity(MTO)studythatexaminedtheimpactofhousingvouchersonfamilieswholivedinhigh-povertyneighborhoodsfoundthatchildrenwhomovedbeforetheageof13hadbettereducationandhigherincomesasadultsHoweveradultsdidnotseeimprovededucationandemploymentoutcomes
Disaggregatingdataintomorespecificcategoriesofraceandethnicitycanhelpunderstandtrendsanddisparitiesinamoregranularwaywhichcanhelpdecisionmakerstargetpoliciesandinterventionsinamoreequitableway
EarlierintheRoadmapweworkedtoidentifystakeholdersandwhomightbepositivelyandnegativelyimpactedbyachangeinpolicyUsetheresultsofthatexercisetoinvolvethecommunityininterpretingdataanalysisandtogatherqualitativefeedbackfromthecommunitytocallattentiontoissuesForinstanceanincreaseinyouthdetentionmaybeexplainedbyinefficienciesorlackofaccesstotransportation
Designimplementationandevaluation
49
Designimplementationandevaluation
50
Shareyourstoryandsuccess
Sharetheresults
Storytelling
AnintegralstepinyourstrategyissharingtheresultsofyourpilotorpolicychangewithyourcommunityThisincludesdevelopingmessagingaroundtheworkofyourorganizationandcreatingaplantodisseminateyourstoriesWhytellstoriesaboutyourwork
Informandorunderstandasubjectorsituation
Giveneededcontextofdataorbitsofinformation
Relatablestoriesarememorableforreadersviewers
ReadersorviewerswillseetheinformationasmorecompellingwhenthereisademonstratedandrelatablechangeorimpactonthemselvesortheirfriendsneighborsandcolleaguesStoriesareaboutpeopleandgovernmentisfundamentallyabouthelpingpeopleSotellthestoriesofthepeopleyouandyourcitycolleagueshelped
Keepthesetipsinmindasyoudevelopyourstorywithyourteam
WhatwereyourfindingsandwhyshouldanyonecareTellyouraudiencehowtheproblemwassolvedWhathastheimpactbeenonresidentsandcitystafftodateWhatwasthesuccessandweretheremultiplesuccessesalongthewayIfyouhavepreliminaryresultsordatashowingimprovementtosharethisisthetimetodoit
WhowasinvolvedandimpactedReviewthelistofstakeholdersHowwereeachofthemimpactedbytheprogramHumanizeyourstoryandimpactbyaddingadditionalvoicesThesevoicescanbeoneofthefollowing
Apersonorfamilywhowaspositivelyaffectedbythechangesuccess
Aninvolvedthird-partydescribingboththeproblemandthesolution(ieanorganizationnonprofitprovidersbusinesstradeassociationsportsteametc)whoispartofthecommunitythatthegovernmentworkedwithonthechangesuccess
Someonefromthegovernmentitselfwhocanspeakaboutthechangesuccess
TellmoreaboutyourcityDescribeyourcityincludingpopulationdemographicsindustriesetcWhatareitschallengesWhatisitknownforWhatmakesituniqueHowisitsimilarordifferentfromothercitiesintheUS
Shareyourstoryandsuccess
51
DescribeyourcityrsquoschallengeWriteoutthechallengeyourcityisfacing(asdefinedinyourproblemstatement)HowdoesitaffectresidentsHowdoesthisproblemaffecttheworkofcitystaffmembers
AddthedetailsProvidedetailsonyourappliedresearchstrategyTalkaboutwhatyourcityhasdonenthepasttoresolvetheissueDiscusshowyourstrategyhaschangedwhenyoubegantoadoptthisapproachWasmoneysavedorinvested
Usethisstorymapexercisetoorganizeyourthoughtsandguaranteethatyouincludealloftheelementsofatraditionalstory
AfteryoudevelopyourmessagingcreateacommunicationsmarketingandoutreachstrategydesignedtospreadyourmessageYourstrategyshouldwalkyouthroughthefollowingkeyactivities
1 IdentifyandprofileyouraudienceWhencreatinganddisseminatingyourmessageitisalwaysgoodtothinkaboutyouraudienceWhoexactlywouldbeinterestedinwhatyouhavetosayThegeneralpopulationofthecityPeoplewhoworkinnonprofitsBusinessownersItmattersbecausenoteveryonewholivesinacitymaybethetargetoftheinformationBusinessownersmaybemoreinterestedindatashowingeffectsonthebusinessdistrictthanothersandsoonWewillcoverthislaterbutthisstepwillalsohelpyoutothinkofthemosteffectivewaytoreachyouraudience
2 Defineyourcalltoaction(ifyouhaveone)SpendsometimeconsideringwhyyouareinterestedintellingyourstoryDoyousimplywanttoupdatethepubliconyourprogressorcelebrateyoursuccessesAreyouinterestedinyouraudiencechangingitsbehaviororsupportingaspecificinitiativebeinglaunchedbyyourorganizationIfyouwanttohaveyouraudienceactorbehaveinacertainwaybespecificandincorporateitinyourmessaging
Shareyourstoryandsuccess
52
3 SelectperformancemeasuresThinkabouthowyouwoulddetermineifyourcampaignissuccessfulandselectappropriateperformancemeasuresYouwillrevisitthesemeasuresattheendofyourcampaigncommunicationsplanimplementationtodeterminewhetheryouneedtomodifyyourcommunicationsandoutreachstrategyforthefuture
4 MethodstocommunicateyourstoryTherearemanywaystocommunicatesuccessofyourprojectHerearesomesuggestionsonreachingaudiencesItisimportanttodeterminewhichmethodsarethemosteffectiveforyourdesiredaudiencetoreceiveinformationDoyourresearch(PewResearchCenterhasexcellentstudiesonthereachofmediadigitalmediaandmarketing)ForexampleifseniorsareyourtargetpopulationandtheyarelessinclinedtouseInstagram(thatistrue)youmaywanttoconsiderFacebookorthenightlylocalnewsbothofwhichareusedmoreoftenbyseniorsBelowisalistofsomemethodsandtoolsyoucanuse
i Pressreleasestolocalnewspapersandtvstations
ii ShortsmartphonevideosAgreatwaytoengageanaudienceisthroughvideoAskonequestionofsomeonewhoispartofthestoryMaybethereisasocialworkerwhocandiscusshungerinyourcityThesevideoscanbereusedonmultipleplatforms
iii Long-formwriting
iv TellingindividualstoriesthroughaseriesofphotosandshortnarrativesforpublicationonFacebookorInstagram
v Op-edsopinionpiecesplacedinthelocalpaper
vi Emailupdatestoyourlist
vii Connectingwithappropriateorganizationsthatareinvestedinthetopicandworkwiththecommunityandaskthemtodisseminateinformationtotheirmembers
viii EmployingldquoWordofMouthrdquo(WOM)tacticsbycreatingtownhallsorfreeeventsforresidentstoattendintheirownneighborhoods
5 DeterminethetypeofcollateralUsuallycampaignsthattargetresidentsrequireseveralavenuestoreachthemPeoplearenotalwaysinthesameplaceorspaceHavingakeenunderstandingofyouraudiencewillassistyouwithyourchoicesonwhatneedstobedevelopedSometimesthatmeansimagesforsocialmediaaonepageflierthatgivespeopleinformationonattendinganeventoranin-depthbrochure
Shareyourstoryandsuccess
53
6 ImplementyourstrategyKeepinmindstrategiescanbefluidSometimessomethingmaynotturnoutasexpectedevenifduringplanningsessionsitwasldquosupposedtordquoItisokYoucananalyzewhathappenedduringadebriefsessionandonceyoursurveysandothermeasurementshavebeenfinalized
Visualizations
Withtherightvisualizationyoucanreviewandshareinformationinsightsanalysisperformanceandtrendsallofwhichisvaluabletoacross-sectionofthepublicButwhichvisualizationisrightforyouTochoosetherighttypeofchartgraphdeterminethetypeofquestionyouwanttoanswerSomecommoncharttypesarebelowandmoreinformationcanbefoundhere
Exploreadditionalanalyticaltechniques
Duringthegoal-settingphaseyouestablishedoutputoutcomeandimplementationmeasuresBasedontheresultsofyourpilotyoumayneedtoadjustyourmeasuresWiththepilotdatainhandyoumayhavesomeideasofhowfutureanalysescouldbedifferentandpotentiallyimprovethequalityofyourprogramTakesometimetobrainstormotherrelevantdatasetsthatyoudidnothavethetimetopursueearlierincludingmessydepartmentaldatathatneedsmoreattentionorsourcesofpublicdatafromthestateandfederalgovernmentnonprofitsandevenprivatecompaniesthatcouldsupplementyourunderstandingofanissueUsingmachinelearningtechniquesyoumightbeabletobetter
Shareyourstoryandsuccess
54
predictdemandforhomelessservicesusingNOAAweatherforecastspreemptwatermainbreaksusingsoilstratigraphyorinvestineconomicdevelopmentbymonitoringrealestatelistingdata
Advancedanalyticaltechniquescanalsohelpyourimplementationrunmoreeffectivelybyintegratingdata-informedtargetinganddecisiontoolsintotheprocessByautomatingroutineworkflowsandsurfacingsubtletrendsthesetoolscanhelpyouiterateinamoreprecisemannerandprovidekeystrategicguidancetofront-lineemployeesthatwouldotherwisebeunavailableThesesortsofdata-science-enhancedprocessescanbeclassifiedasoneofsixprojecttypesasexploredfurtherbytheNewOrleansLANOLAlyticsteamFormoreguidanceonscopingasimilarprojecttheGovExadvancedanalyticsscopingguidegivesyouastep-by-stepoverviewofhowtobesuccessful
Usethespacebelowtolistyourideasforpossibleadvancedanalyticsprojectsbytype
Typeofproject Ideas
Findingtheneedleinthehaystack
Prioritizingforimpact
Earlywarningtools
Betterquickerdecisions
Optimizingresourceallocation
Experimentingforwhatworks
Keepthefeedbackloopgoing
GatherinformationonpublicsentimentaboutpolicychangethroughfocusgroupssurveysandpublicmeetingsledbyadvisorygroupmembersNotewhethernewinformationisavailablethatcanhelpanalyzeorcontextualizeyourprogress(egfederalorstatedatareports)Sharetheresultsofyourpilotandthepolicychangeprocesswithothercitiesandresearchers
Shareyourstoryandsuccess
55
WorkingwitharangeofevidencePresentlythereisalotofattentionandinvestmentinevidence-basedpolicymakingincludingthefederalbipartisanCommissiononEvidence-BasedPolicymakingthePewCharitableTrustsreportsandtoolkitsonHowStatesEngageinEvidence-BasedPolicymakingandtheLauraandJohnArnoldFoundationsupportedEvidence-BasedPolicymakingCollaborativeManyoftheseinitiativesfocusonrigorousevaluation--meaningrandomizedcontrolledtrials(RCT)orotherexperimentalmodels--asthebestkindofevidencetouseHoweverthisbodyofevidenceisofteninsufficientforthespecifickindsofquestionsthatgovernmentscanhavewhenmakingadecisionaboutpolicyInthisguideweproposeanapproachtosupportgovernmentsinincorporatingadditionaltypesofevidenceintotheirdecisionmaking
OnewaytounderstandarangeofevidenceistouseNestarsquosStandardsofEvidencetoframeanddefineevidenceTheBritishnonprofitNestaworkstoinjectpositivechangeintothewayUKgovernmentpoliciesandprogramsfunctionOneoftheirkeyareasofworkhasbeendevelopingWhatWorksCentresorganizationsthatcollectcommissionsynthesizeanddisseminateevidenceaboutinterventioneffectivenessacrossarangeofsocialimpactareasNestadevelopedthefollowingStandardsofEvidencetoprovideausefulwaytodocumenthowdifferentkindsofinformationdemonstratedifferentlevelsofevidence
Workingwitharangeofevidence
56
Theimportanceofpotentiallybiasedinformation
Onechallengepresentedbyanevidenceclassificationthatacknowledgesthevalueofnon-scientificformsofevidenceistofigureouthowtobalanceallofthepiecestogetherwhenyouusethemTounderstandhowtodothiswehavetothinkabouthowtopositionpotentiallybiasedinformationindecisionmaking
WhenwelookatinformationwhichhasnotbeenrigorouslyevaluatedweacceptthatthisinformationmightbebiasedThewholereasonweconductrigorousevaluationistoremovebiasfromourobservationsWeconductstructuredtrialstoproduceresultsthatareconsistentandgeneralizableIfafindingisunbiasedthisaccuracymdashthislackofbiasmdashcanbeprovedbecauseanyindependenttestercanreproduceitwhentheyfollowanidenticalprocess
YoucanlookatpotentiallybiasedinformationasjustinformationthatisearlieroninitspathtorefinementreplicationandverifiedaccuracyApieceofinformationbasedonanyoneindividualrsquosobservationsislikelytobebiasedbecauseitiscomingfromjustonesourceBiasisreducedbygatheringtogetherlotsofindependentsourcesofinformationandfiguringoutitsmostcommonattributes
MaybethebiasedinformationmakesaclaimthatistoolargemdashitsayssomethingistrueunderallconditionswhenitisreallyonlytrueundercertainconditionsorthataneffectofaninterventionisbiggerthanitstrueaverageorthataneffectlastsovertimewhenitdoesnotPerhapschangingtheclaimtobemorespecificmdashcriticallyreviewingtheinformationtoseeifspecificconditionsmustbemettoachievetheoutcomesmdashwouldmaketheclaimmoreaccurate
IfyoutestedyourclaimfounditdidnotholdupinsomecaseschangedittesteditagainuntilyourclaimrepeatedlyheldupthenyoucoulddemonstratethattheclaimwasnolongerbiasedbutaccurateThisprocessoftestingimprovingtestingandvalidatingiswhatwegetfromstructuredscientificstudy
Withindividualpiecesofinformation--crediblehypothesescasestudiesexpertopinion--youcannotassumethatkindofaccuracyAtthesametimetheseearlyobservationsarefoundationaltoanyreplicablescientificobservationsWewillnevergetanywhereifwedonotstartwiththesepiecesofinformationeveniftheymightbebiased
Combininglevelsofevidencetogetthemostdetailedinformationforyourcity
ItisespeciallyimportanttoacceptdifferentformsofevidencewhenyoufacequestionsthathavenotyetbeentestedbyindependentstudiesSomeofthemostimportantquestionsrelatetothingsthatareveryspecifictoyoursituationYoufaceaparticularcombinationof
Workingwitharangeofevidence
57
issuesactorstrendsandresourcesthatisultimatelydifferentfromeveryotherplaceintheworld
ThisisimportantbecauseaspolicypractitionersandnotscientistsyoucaremoreaboutyourspecificcasethanahypotheticalaveragegeneralcaseYoucarelessaboutthetotallikelyrangeofimpactsfromapolicythanhowthissingleimplementationwillworkYouwantinformationaboutpoliciesthatwillprovideyouwiththebestpossiblepredictionabouthowtheywillworkinyourcontext
ByputtingdifferentkindsofevidenceintothesameframeworkNestarsquosStandardsofEvidenceimplythatwhenyoucannotgetrigorouslytestedexperimentallyverifiedinformationitisappropriatetoworkwithwhatwedohaveThisisrelevantforthemunicipalimplementationcontextbecausenostudypulledfromaliteraturereviewwillaccountforallofthevariablesyoufaceinyourowncontext
Intheabsenceofhavingastudythatpredictshowaninterventionworksinyourspecificcontextyouwillneednon-scientificinformationtotrytofillinthosegapsYouwillneedtogatherstoriesfromresidentsstaffmembersandotherstakeholdersthatsitonLevels1or2ofthestandardsofevidence
Someofthekindsofinformationyouwillprobablygetmoreeasilyfrompeoplethanstudiesinclude
howpeopleincitieslikeyoursreactedtoaninterventioninpublicforumshowpeoplewhoimplementedaninterventioninacitylikeyoursfiguredouthowtogetitpassedhowthecitycommunicatedpubliclyabouttheapparentbenefitsanddrawbacksoftheinterventionthepoliticalacceptabilityoftheintervention
Youcanremainsimultaneouslyawarethattheinformationyouaregatheringthroughindividualstoriesisbiased--notgeneralizableinformedbyspecificperspectives--whilealsorecognizingthathavingsomeinformationaboutsomethingimportantisbetterthannoinformationEvidencefromthefirsttwoevidencelevelsareessentialbuildingblocksofknowledgeandperfectlyusefulwhenyoucannotgetdetailedunbiasedknowledge
Youhavetostartsomewhereandyoushouldalwaysstartwithevidenceyouhaveaboutyourveryspecificconditionsthatyouknowareimportantforsuccess
Workingwitharangeofevidence
58
Thankyou
Interviewswithcities
GovExheldsemi-structuredinterviewswithsixstaffmembersinfourcitiestoestablishtheinitialstepsoftheRoadmapWeaskedrespondentstotalkaboutaninitiativeorpolicythattheyhavebeenresponsibleforimplementingintheircityRespondentsfromthreecitiesspokeabouttheireffortstoaddresshomelessnessandrespondentsfromonecityspokeaboutblightAfterrespondentsidentifiedtheproblemtheyspokeabouthowthecityprioritizedthisissuethevariousstepsinvolvedandbarrierspresent
GovExconductedtextanalysisoftheseinterviewswhichrevealedthatpeoplewasthemostcommonlyusedtermduringtheseconversationswhendiscussingthestepstochangeaprogramorpolicyThisisathemethroughoutthisRoadmapThefirstsectionbeginswithidentifyingtherelevantpeopleTheconversationsaroundevidencearefocusedonhumanbehaviorratherthanmethodologyorstrengthofevidence
Thankyou
59
OncewehadadraftoftheRoadmapwefollowedupwithcitiesonanindividualbasistolearnmoreaboutwhatwouldbehelpfulWearegratefultothefollowingcitystafffortheinterviewsandindividualconversations
MichaelBaskinChiefPolicyOfficerChattanoogaTN
MelissaBridgesPerformanceandInnovationCoordinatorLittleRockAR
VictoriaCarreoacutenAdministrativeOfficerLasVegasNV
TyroneGrandstrandCommunityDevelopmentPlannerFargoND
TibLaughlinDirectorofGeneralServicesKansasCityKS
TimMorelandDirectorofPerformanceManagementandOpenDataChattanoogaTN
CharlesRosendahlManagementAnalystDentonTX
Thankyou
60
MelissaSiebenAssistantCountyAdministratorKansasCityKS
GinaThompsonDirectorofStrategicServicesLewisvilleTX
JHUFacultycollaboration
GovExalsoconvenedthecitycontactsandfacultypartnersinaseriesofAdvisoryGroupmeetingsThesemeetingsservedtoconnectcitiesandfacultytounderstandoneanothersperspective
StefanBaralBloombergSchoolofPublicHealth
AshleyBernerJHUSchoolofEducation
MelissaDavey-RothwellBloombergSchoolofPublicHealth
JonLinksBloombergSchoolofPublicHealth
TamarMendelsonBloombergSchoolofPublicHealth
KristinMmariBloombergSchoolofPublicHealth
KeshiaPollackPorterBloombergSchoolofPublicHealth
JoshSharfsteinBloombergSchoolofPublicHealth
LizStuartBloombergSchoolofPublicHealth
LindsayThompsonCareyBusinessSchool
DanielWebsterBloombergSchoolofPublicHealth
Partnersindataandevidence-basedpolicy
Wealsospokewithpartnerswhoareexpertsinthefieldofevidence-basedpolicywhosharedtheirknowledgeandresourcesofcurrentandfuturethinkinginevidence-informedthinking
WendyBarrenoAssociateDirectorWilsonSheehanLabforEconomicOpportunitiesUniversityofNotreDame
TiffanyDavenportDirectorofDataPartnershipsBaltimoresPromise
TomiHiersDirectorBaltimoreCivicSiteAnnieECaseyFoundation
ChrisKingsleySeniorAssociateAnnieECaseyFoundation
KathrynOliverAssociateProfessorLondonSchoolofHygieneandTropicalMedicine
Thankyou
61
KristenWatkinsSeniorPolicyAssociateJ-PAL
GovExParticipants
GovExAnalystMiriamMcKinneyanalyzedtheinterviewtranscriptsandconductedtextanalysisSeniorImplementationAdvisorEmilyShawwillintegrateRoadmapactivitieswithadditionalservicedeliveryactivitiesatGovEx
Thankyou
62
ReferencesAbrahamKGampHaskinsR(2017)Thepromiseofevidence-basedpolicymakingReportofthecommissiononevidence-basedpolicymakingWashingtonDCCommissiononEvidence-basedPolicymakingBigmove2Extendingthegreen(nd)Retrievedfromhttpswwwcityoflewisvillecomdoing-businesslewisville-2025big-move-2-extending-the-greenBatesMAampGlennersterR(2017)ThegeneralizabilitypuzzleRetrievedfromhttpsssirorgarticlesentrythe_generalizability_puzzleBuildwithnotfor(2018January)RetrievedfromhttpsbuildwithnotforwordpresscomCairneyPampOliverK(2017)Evidence-basedpolicymakingisnotlikeevidence-basedmedicinesohowfarshouldyougotobridgethedividebetweenevidenceandpolicyHealthResearchPolicyandSystemsdoi101186s12961-017-0192-xCairneyPOliverKampWellsteadA(2016)TobridgethedividebetweenevidenceandpolicyReduceambiguityasmuchasuncertaintyPublicAdministrationReview76(3)399-402Retrievedfromhttpdxdoiorg101111puar12555Centreforpublicimpact(nd)RetrievedfromhttpswwwcentreforpublicimpactorgCernaL(2013)ThenatureofpolicychangeandimplementationAreviewofdifferenttheoreticalapproachesOrganisationforEconomicCo-OperationandDevelopmentRetrievedfromhttpswwwoecdorgeduceriTheNatureofPolicyChangeandImplementationpdfColoradoSprings(nd)Retrievedfromhttpscoloradospringsgovcommunicationspagewhat-expect-when-you-speakupContinuumofcare(coc)program(nd)RetrievedfromhttpswwwhudexchangeinfoprogramscocCouncilpresidentkicksoff$12millionyouthfund(nd)Retrievedfromhttpwwwbaltimorecitycouncilcomcontentcouncil-president-kicks-12-million-youth-fundDurlakJAampDuPreEP(2008)ImplementationmattersAreviewofresearchontheinfluenceofimplementationonprogramoutcomesandthefactorsaffectingimplementationAmericanJournalofCommunityPsychology41327-350Retrievedfromhttpswwwncbinlmnihgovpubmed18322790Fishbonediagram(ishikawa)(nd)Retrievedfromhttpasqorglearn-about-qualitycause-analysis-toolsoverviewfishbonehtmlFivewhysGettingtotherootofaproblemquickly(nd)RetrievedfromhttpswwwmindtoolscompagesarticlenewTMC_5Whtm
References
63
FligsteinNampMcAdamD(2011)TowardageneraltheoryofstrategicactionfieldsSociologicalTheory29(1)1-26RetrievedfromhttpsociologyberkeleyedusitesdefaultfilesfacultyfligsteinFligstein20McAdam20Soc20Theory20PaperpdHeinkelP(2018January12)CitymullspermitsforbikesharingcompaniesRetrievedfromhttpwwwdentonrccomnewsdenton20180112city-mulls-permits-bike-sharing-companiesHowisareamedianincomecalculated(2012October)Retrievedfromhttpswwwhudexchangeinfofaqs843how-is-area-median-income-calculatedLampkinLMampHatryHP(2003)KeystepsinoutcomemanagementUrbanInstituteRetrievedfromhttpswwwurbanorgresearchpublicationkey-steps-outcome-managementMarshDampSmithM(2000)UnderstandingpolicynetworksTowardsadialecticalapproachPoliticalStudies484-21Retrievedfromhttpjournalssagepubcomdoiabs1011111467-924800247McCannL(nd)RetrievedfromhttplaurenellencomMoultonSampSandfortJR(2016)ThestrategicactionfieldframeworkforpolicyimplementationresearchPolicyStudiesJournal45(1)Retrievedfromhttponlinelibrarywileycomstore101111psj12147assetpsj12147pdfv=1ampt=jdkhhcp4amps=e184c625e1855a6b1615ccb99ee27b1c0b3212d8PaynterB(2018January1)What100citiesarelearningfromeachotherbysharingtheirdataRetrievedfromhttpswwwfastcompanycom40518004what-100-cities-are-learning-from-each-other-by-sharing-their-dataPeakacademy(nd)Retrievedfromhttpswwwdenvergovorgcontentdenvergovenmayors-officeprograms-initiativespeak-performancepeak-academyhtmlPinnockHBarwickMCarpenterCREldridgeSGrandesGGriffithsCJRycroft-MaloneJMeissnerPMurrayEPatelASheikhAampTaylorSJC(2017)Standardsforreportingimplementationstudies(stari)ExplanationandelaborationdocumentBMJOpenRetrievedfromhttpbmjopenbmjcomcontentbmjopen74e013318fullpdfProctorESilmereHRaghavanRHovmandPAaronsGBungerAGriffeyRampHensleyM(2011)OutcomesforimplementationresearchConceptualdistinctionsmeasurementchallengesandresearchagendaAdministrationandPolicyinMentalHealth38(2)65-76doi101007s10488-010-0319-7TandonSD(2018)ProgressincommunityhealthpartnershipsResearcheducationandactionRetrievedfromhttpswwwpressjhuedujournalsprogress-community-health-partnerships-research-education-and-actionProvanKGampKenisP(2007)ModesofnetworkgovernanceStructuremanagementandeffectivenessJournalofPublicAdministrationResearchandTheory
References
64
18(2)229-252Retrievedfromhttpsacademicoupcomjpartarticle182229935895SabatierPA(1988)Anadvocacycoalitionframeworkofpolicychangeandtheroleofpolicy-orientedlearningthereinPolicySciences21(3)129-168Retrievedfromhttpwwwjstororgstable4532139SandstromAampCarlssonL(2008)TheperformanceofpolicynetworksTherelationbetweennetworkstructureandnetworkperformancePolicyStudiesJournal36497-524Retrievedfromhttponlinelibrarywileycomdoi101111j1541-0072200800281xabstractSchwartzMampWilsonE(nd)WhocanaffordtoliveinahomeAlookatdatafromthe2006americancommunitysurveyRetrievedfromhttpswwwcensusgovhousingcensuspublicationswho-can-affordpdfShearerJCAbelsonAKouyateBLavisJNampWaltG(2016)WhydopolicieschangeInstitutionsinterestsideasandnetworksinthreecasesofpolicyreformHealthPolicyandPlanning31(9)1200-1211Retrievedfromhttpsdoiorg101093heapolczw052ThelessonsofadministrativedataHigh-profilepolicy-relevantresearchpoweredbyadministrativedata(nd)Retrievedfromhttpswwwpovertyactionlaborgsitesdefaultfilesresources20161114-The-Lessons-of-Administrative-DatapdfToolsandotherneedsforqualitativeandquantitativemethods(2015February)AgencyforHealthcareResearchandQualityRetrievedfromhttpwwwahrqgovprofessionalsquality-patient-safetytalkingqualityassesstoolshtml
WeaverRCampBagchi-SenS(2013)SpatialanalysisofurbandeclineThegeographyofblightAppliedGeography4061-70
References
65
UniversityPartners
Name Title Affiliation Area(s)ofexpertise
Contactinformation
Examplesofworkingwithnon-academicdecisionmakers
WendyPBarreno
AssociateDirector
WilsonSheehanLabforEconomicOpportunitiesUniversityofNotreDame
LEOsfocusareasincludehealtheducationhousingchildwelfareeconomicopportunitycriminaljustice
wbarrenonduduorleondedu
MelissaDavey-RothwellPhD
AssociateScientist
JohnsHopkinsBloombergSchoolofPublicHealth
Health mdavey1jhuedu410-614-5854
BaltimoreCityHealthDepartmentLeadersMarylandDepartmentofHealthAddictionRecoveryServicesinDCExecutiveDirectorsofcommunityorganizationsacrosstheUSthatfocusonHIVSTIsubstanceabuseandimplementationscience
ElizabethStuart
AssociateDeanforEducationandProfessorDepartmentsofMentalHealthBiostatisticsandHealthPolicyand
JohnsHopkinsBloombergSchoolofPublicHealth
Educationsocialpolicystatisticalmethods
estuartjhuedu
Resources
66
Management
Design-ThinkingResources
EngagementGuideforCivicTech
Journeymap
PerformanceManagementResources
DenverPeakAcademy
Proxymeasuresguide
Talkingaboutperformancewithoutcomplicatingit(video)
SettingPerformanceTargetsguide
Datavisualizationguide
Performancemanagementgettingstartedguide
Exercises
Fishboneexercise
5Whys
AdministrativeData
TheLessonsofAdministrativeData(JPAL)
EvaluationResources
Quantitativeandqualitativemethods(USDepartmentofHealthandHumanServicesAgencyforHealthcareResearchandQuality)
KeyStepsinOutcomeManagement(UrbanInstitute)
RCT-YES
Sageresearchmethods
BestPracticesforMixedMethodsResearchintheHealthSciences
ACCORDChecklist
Pilotandimplementationplanning
Benchmarkingexamplesandproxymeasures
Seattleopendataplaybook
Resources
67
Datasciencescopingguide
Resources
68
AboutGovExGovExhelpsgovernmentsusedataandevidencetoimproveresidentsrsquolivesWesupportgovernmentsinbuildingtheirinternalcapacitytooperatesustainableanalyticspracticeswhichimprovecommunityoutcomesinfieldsfrompublicsafetytoeconomicdevelopmentGovExhasaproventrackrecordofworkinginconcertwithpartnerstoprovidetechnicalassistanceandtrainingatscale
SinceourlaunchinApril2015GovExhas
Trainedover2000governmentemployeesinpersonandonlineintopicsincludingdatamanagementperformancemanagementopendatacommunityengagementandanalytics
Providedtechnicalassistancetomorethan100localgovernmentsacrosstheglobeimprovingtheiruseofdatatoachieveefficienciesinservicedeliveryandbetteroutcomesforresidents
Assessedthecapacityofmorethan130governmentalorganizationstodeterminehowleadersstaffandexternalstakeholdersmanageandusedata
Localgovernmentsareincreasinglyonthefrontlinesofaddressinglargescaleproblemswithglobalimplicationssuchasclimatechangepublicsafetyandeconomicinequality
Whileevidence-basedinterventionstohelpsolvethoseproblemsareavailablelocalgovernmentstaffhavelimitedtimeandcapacitytofindandimplementresearch-basedsolutionsAsaresultcitiescontinuetoworkthewaytheyalwayshave--makingincrementalimprovementstoprogramsbutnotalwayscreatingchangeatasystem-level
ThroughresourcedevelopmentandcollaborationwithJHUfacultyGovExissupportingcityleadersinusingdataandevidencetomakedecisionsandcreatinglastingrelationshipswithresearchers
TheRoadmapwasdevelopedbyGovExaspartofitspartnershipwithBloombergPhilanthropiesWhatWorksCitiesinitiative
AboutGovEx
69
- Title page
- Introduction and purpose of the Roadmap
- Using the Roadmap
-
- Gather information about the problem
- Strategically engage key partners
- Review the evidence and collect details
- Decide on a policy to implement based on local factors
- Establish a framework to monitor progress
- Assess data accessibility quality and security
- Design implementation and evaluation
- Share your story and success
-
- Working with a range of evidence
- Thank you
- References
- Resources
- About GovEx
-
Captureyourproblemstatementtheresultsofyouranalysisanddesiredoutcomes
Gatherinformationabouttheproblem
15
StrategicallyengagekeypartnersPolicytheoryresearchoftenfocusesonincreasingpolicymakersaccesstoevidenceasawaytoavoidtakingshortcutslikemakingdecisionsbasedonemotionImplementationsciencesuggeststhatincreasingaccesstoevidenceisnotenoughandthatpolicychangeshouldbeexaminedinthewidercontextofdebatecoalitionformingandpersuasionMeaningfulpolicychangerequiresldquoalong-termstrategytoformalliancesanddevelopknowledgeofpolicymakersandpolicymakingratherthanusingevidenceinpolicymakingonanadhocorone-offbasis
UnderstandingnetworksandsystemsisimportanttodevelopsustainablepolicychangeThissectionismeanttohelpreaderstakethestepstobuildengagementstrategiesforpartnersthatrepresentdifferentsystemsandnetworksEngagementshouldbeanongoingprocessofconnectingpeopletooneanotherandtoanissueEngagementshouldnotbeinnameonly-wehaveallexperiencedengagementactivitiesthatgivepeopleaspecificroletoplaythatleadstoaneatpredictableoutcomeAccordingtotheBuildWithmovementundervaluingandunderinvestinginengagementstrategiescanleadtoadisconnectofcertainindividualsorgroupsandmistrustofgovernmentWhendevelopingengagementactivitiesrememberthathowpeopleengageandwhethertheyengagehastodowithhowyoustructureopportunitiesforengagement
Conveneacoreadvisorygroup
AnadvisorygroupcomprisedofpeoplefrommultiplesectorscanbehelpfulasyouconsiderthelocalsocialenvironmentalandpoliticalcontextinwhichyouintendtointroducepolicychangeRevisityourstakeholdermapanddeterminewhowouldbebesttoserveonthisgroupTomakesureyouareconsideringtheproblemsandsolutionsthroughanequitylensincludepeoplewithlivedexperiencewhohaveexpertiseintheeffectivenessofsystemswhetherthosesystemsfunctionasintendedandhowtheymayimpacthumanbehaviorYoumayalsowishtoincludesomeonewithexperienceworkingwithdataandanalysis-thispersoncanhelpdeterminethetypeofdatathatmaybereadilyavailableandusefultocollectandcanalsohelpplanandshapeyouranalysistofigureoutwhichprogramsareeffectiveBesuretoinvitefront-lineworkersaswell-itscriticalthatwhateversolutionscomeoutofthisprocessaremeaningfultothosewhowillberesponsibleforimplementingthem
BringingpeopletogethercanbechallengingbutthereareafewwaystosetandmanageexpectationsFirstmakesurethatthegroupisawareoftheproblemstatementcurrentstateandfuturestate-thiscanhelpstarteveryoneoffonthesamepageNextspend
Strategicallyengagekeypartners
16
sometimegettingtoknoweachotherrsquosworktolearnabouteveryonesstrengthsThissharedunderstandingcanhelpasyouclarifyrolesandresponsibilitiesofeachpartnerinthegroup--thisisimportanttoavoidfinger-pointinginthefutureandtoensurethatnoneofthetasksslipthroughthecracksInsomecasestherearemanyactorswithdifferentresponsibilitiesasinthecaseofBaltimoreCitys38-memberYouthFundTaskForcewhichischairedbynonprofitandcorporatepartnersandincludesrepresentativesfromthenonprofitandfaithcommunitiesstudentsthebusinesssectorlocaluniversitiesCityCouncilandtheMayorsOffice
OncerolesandresponsibilitiesareclearyoumayconsiderdevelopingacharterguidelinesortermsofreferencetosetexpectationsforthegroupBesuretodistinguishthisfromotherboardsandcommissionsthatyourcitymayhaveNextyoumaywishtoestablisharegularongoingmeetingscheduleBesuretosendagendasandmeetingmaterialsaheadoftimeandclarifytheobjectiveofeachmeeting
InSeattletheCitydevelopeditsHousingAffordabilityandLivabilityAgenda(HALA)torespondtorisinghousingcostsandhomelessnessThisisatrendinotherwestcoastcitiesaswellwhichmaybeexplainedbyaninfluxofnewresidentswhoareattractedbytheemergenceoftechjobstheattractionoftheoutdoorlifestyleortherecentpassageofprogressivepolicieslikelegalizedgaymarriagea$15minimumwageandlegalizationofmarijuanaSeattlebroughttogetheramulti-stakeholderadvisorygroupcomprisedoflocalhousingexpertsrentersandhomeownersdevelopersandgovernmentofficialstoshareinformationabouttheirneedsanddeveloprecommendationsbasedontheseneedsTheinterestsandmotivationsofthisgroupwerevariedandittookabouttenmonthstofindcommongroundTheseriesofrecommendationsthattheadvisorygroupcreatedismeanttoserveresidentswithaspectrumofneeds
Buildaprocessforcommunityparticipation
Ofcoursenoteveryonecanbeincludedinanadvisorygroupandadvisorygroupmembersmaynotnecessarilyrepresentthevariedperspectivesoftheirpeers-forinstancenoteveryhomeownerhasthesamesetofneedsTogatherinputfromawiderswathofthepopulationitcanbehelpfultobuildaprocesstoinvitethepublicatlargetosharetheirideasCitiesdothisinavarietyofwaysfromonlineconversationsandsurveystoin-personmeetingsRememberthatbeingintentionalaboutgatheringtheviewpointsofpeopleismoreimportantthanusingthelatesttechnologyandtools
TheinsightgeneratedbycommunityparticipationexercisescanbeincludedinyourreviewofevidencewhichisdiscussedinasubsequentchapterInWestVirginiaastatethathasbeenhithardbytheopioidepidemicstatehealthofficialsdraftedanopioidresponseplanthatwasinformedbypublicengagementaswellaslocalandregionalsubjectmatter
Strategicallyengagekeypartners
17
expertsThepublicwasinvitedtopresentcommentsonideasfordecreasingoverdoseearlyinterventiontreatmentandrecoveryPublichealthandmedicalfacultyfromWestVirginiaUniversityMarshallUniversityandJohnsHopkinsUniversityprovidedresponsestothecommentsTheresultwasanOpioidResponsePlanthathealthcommissionerRahulGuptapresentedtoGovernorJimJustice
AsamplespectrumtoplanyourowncommunityparticipationprocesscreatedbyLaurenellenMcCannisbelow
SourceLaurenellenMcCann
Community-basedparticipatoryresearchisanotherwaytoworkwiththecommunitytoimproveoutcomesCBPRmeansthecommunityisinvolvedinallaspectsoftheresearchprocesswhichisofteniterativeInCBPRtraditionalresearchersandcommunitymembersshareknowledgeaswellasresourcesandcredit
CBPRInActionInSanFranciscoanenvironmentaljusticenonprofitteamedupwiththeCitysDepartmentofPublicHealthandanoutsideevaluatortounderstandtherootcausesofobesityandfoodinsecurityinlow-incomeneighborhoodsandidentifysolutionsThroughresearchmethodssuchasinterviewingresidentsandstoreownersdiagrammingstoreshelvesGISmappinganeconomicanalysisandreviewofsimilarpoliciesinnearbycitiestheteamobservedthatthestoresthatresidentshasveaccesstowerestockedwithlesshealthyfoodsandthatresidentswantedmorehealthyfoodandlessalcoholandtobaccoThisprojectresultedinlocalandstatelegislationtoencouragestorestostockfreshproduce
Strategicallyengagekeypartners
18
Developrelationshipswithsubjectmatterexperts
OnewaytokeepsubjectmatterexpertsengagedistoinvolvetheminthecommunityparticipationprocessasdescribedaboveIntheWestVirginiaexamplesubjectmatterexpertsrespondedtoandsynthesizedthepubliccommentsOtheropportunitiesaredescribedbelow
AdvisoryAcitydepartmentheadMayororstaffmembermaydevelopaone-on-oneadvisoryrelationshipwithuniversityfacultymemberoracollaborativebetweentheCityandUniversitytoaddressaparticularproblemAgreatplacetostartistolookforrelevantpracticecenterswithinyourlocaluniversityandreachingouttosubjectmatterexpertsInadditiontotheirexpertiseontherelevantpolicytopicheorshemayalsobeabletoidentifyotherexpertsandevenprovidesupportonhiringtherighttalenttoleadcertainprogramsAnadvisoryrelationshipwithauniversitysubjectmatterexpertcanalsohelpwhencitiesareconsideringbringinginnewprogramsTheseexpertsareoftenfamiliarwithprogramsthatarebasedonnationalmodelsandcanprovideinsightintoaprogramsoriginandlineageeffectivenessinotherplacesandthepotentialforimplementationinyourcityAcademicscanalsohelpidentifyindicatorsbasedongovernmentindicatorsorliteratureFinallyrelationshipsbetweenpolicymakersandresearchersprovidethemostbroadlyeffectiveroutetopullingresearchintopolicy
ResearchandevaluationAcademicexpertsmaysupportcitiesinevaluation-indeedevaluatingcityinitiativesisatraditionalmodelofcityuniversityrelationshipsSharingtheinformationyoucollectedintheGatherInformationphasecanhelpresearchersunderstandyourworkandgoalswhichinturnwillstrengthenthesupportthattheycanprovideDiscussingdetailsaboutyourinterventionandevaluationplanatthebeginningoftheprocessmeansthataresearchpartnercanhelpwiththeevaluationdesignanddatacollectionForinstancetheWilsonSheehanLabforEconomicOpportunitieswalkscitiesthroughconsideringthefollowingquestionsbeforepartneringonanevaluation
1 Isthereaclearpolicyprogrammodelwithobjectivescreeningcriteria
2 Isitpossibletoevaluatewhathappensbothwiththepolicyprogramandintheabsenceofthepolicyprogram
3 Arethereoneormoremeasurableoutcomesofinterest
4 Arethereenoughpeopleforrigorousquantitativeanalysis
5 Whatdatawillbenecessaryforthestudyandwhataretheprocessesforaccessingthesedatasource(s)
6 Isthereacompellingreasontoevaluate
Strategicallyengagekeypartners
19
InadditiontoassessingreadinesstoconductanevaluationresearcherscanhelpyouhoneinonthepurposeoftheevaluationTheWorldBankdescribesseveralpotentialusesofevaluationwhicharedefinedhereInbriefevaluationcanhelpdecisionmakers
makeresourceallocationdecisionsrethinkrootcausesofproblemsidentifynewproblemsdecideonabestalternativedeterminewhetherprogramsorpoliciesarebeingimplementedwithfidelitydeterminewhetherprogramsorpoliciesarehavingthedesiredimpact
Researcherscanalsoprovidesupportinsharingtheresultswiththescientificcommunitythroughpapersandconferenceswhichisvaluableincreatingapeernetworkofevidence-basedcitypractitioners
TrainingandtechnicalassistanceUniversitypartnersespeciallythoseatpracticecenterscanbehelpfulindevelopingorupdatingtrainingcurriculumtrainingthetraineranddeliveringtrainingtocitystaff
Strategicallyengagekeypartners
20
ReviewtheevidenceandcollectdetailsOnceyouhaveathoroughunderstandingoftheproblemdataandactorsyouwillneedtoconductyourownresearchonavailableevidenceandlearnaboutthelandscapeofpolicyoptionsYourgoalwillbetodevelopanaccessiblelistofinformationaboutpoliciesthathavehelpedsolvesimilarproblemsinotherplacesThischapterwillguideyouthroughthestepsofpreparingyourreviewersunderstandinglevelsofevidenceandcollectingthedetailsonevidencethatyouwillneedfordiscussionwithdecisionmakersinyourcity
Selectandprepareyourreviewers
AssignapersonorteamtoreviewrelevantinterventionsthathavedemonstratedsuccessinotherjurisdictionsTheirgoalwillbetoproduceareviewofalloftherelevantprogramspoliciesorinitiativesthatyourteamcouldconsiderimplementingTheywillbeconductingaliteraturereviewsupplementedbyinterviewswithexpertsandpeerswhohaveinformationaboutpotentialprograms
SincethepeopleyouassigntothistaskareprovidingthebaselineofpotentialoptionstoimplementitisimportantthatyouchoosepeopleyouareconfidentwillbethoughtfulandthoroughintheirworkItisalsocriticalthattheyareawareoftheimportanceofsourcecredibilityEnsureyourreviewershavefamiliarizedthemselveswithrulesforcriticallyanalyzinginformationsourcesbeforetheygetstartedIftheyareincludingnewsarticlesaspartoftheirsearchtheyshouldalsobeawareofmethodsforevaluatingnewscredibility
AskyourreviewerstodevelopaplanandtimelinetocompletethereviewThisstartswithsolidifyingagoverningresearchquestionOnewaytoknowyouarecreatingadetailedresearchquestionistousetheacronymPICOremindingyoutoincludereferencetothepopulationintervention(ifyouhaveoneinmind)comparisonandoutcomeelementsinyourresearchquestionForexampleifyouknowthatyouhaveapopulationofteenagerswhoarehomelessyouhavelearnedfromyourstakeholdersthatmanyoftheseteenagersidentifyasLGBTandyouhaveheardsomethingabouttheapproachofcreatingdrop-incentersfordeliveringsupportyoucouldask
ForLGBTyouthfacinghomelessness(population)howeffectivearedrop-incenters(intervention)relativetostandardsocialservices(comparison)atsettlingtheminpermanenthousing(outcome)
Ifyoudonothaveaninterventioninmindyoucanleaveouttheinterventionandcomparisonandsimplyask
Reviewtheevidenceandcollectdetails
21
HowcanwehelpLGBTyouthfacinghomelessness(population)obtainpermanenthousing(outcome)
OnceyourreviewershavetheirresearchquestionsusethequestionstocreatealistofkeywordstheycanuseintheirliteraturesearchThiswillhelpkeeptheircollectionofarticlesreasonablyfocusedHavethemdescribewhattheirsearchprocesswillbeincludinglistingthedatabasestheywillreviewandtheirplanforstoringtheinformationtheywillcollect(Herearesomeadditionalsuggestionsforstructuringareviewplan)
WhatyourreviewerswillbedoingisalightweightversionofarapidreviewTotheextentthattheywouldliketodevelopamethodologyforspecificsearchprotocolsincludingcriticalappraisalprocessestheycancertainlygoaheadanddoso
Usedifferentlevelsofevidenceforthemostcompletepicture
AsdiscussedintheWorkingwitharangeofevidencechaptertheNestaStandardsofEvidenceframework--reproducedbelow--ishelpfultounderstandthelevelofconfidencethataprogramorinterventionishavingapositiveimpactThisapproachletsyouputinterventionsthatyouhaveheardaboutlocallybutwhichhavenotbeenstudiedtogetherwiththosewithademonstratedcausaleffectThisishelpfulsinceanumberofpoliciesprogramsandpracticesthatyouwillwanttoconsidermaynotyethavemuchofaresearchrecord
Reviewtheevidenceandcollectdetails
22
NearlyanypieceofinformationaboutaninterventioncanbeclassifiedwithinthestandardsofevidenceForexampleifsomeonehasacrediblehypothesisabouthowaninterventioncouldproducebenefitthistypeofevidencewouldbeclassifiedasLevel1ThistypeofevidencecouldincludeinterventionsinothercitiesthatyoureadaboutinthenewsorhearaboutataconferenceForinstancewhenUSDAallowedschoolsandschooldistrictstoprovidefreebreakfastandlunchtoallstudentsthroughtheCommunityEligibilityProvision(CEP)manydistrictsimplementedthisoptionevenbeforetherewasevidenceofeffectivenessThiswouldfitintoLevel1sinceitbuildsonthelogicthatprovidingaccesstomealswilldecreasefoodinsecurity
YoumayalsofindpiecesofinformationthatqualifyasLevel2evidenceForinstanceasmoreschooldistrictsimplementedCEPprincipalstestifiedaboutpositivechangestheyobservedinstudentssuchasamorepositivedemeanorthatcanleadtobetteracademicperformanceDataalsoshowedthatthestudentsdescribedwerereceivingmoremealsthanbeforetheprogramsimplementationHoweveritwastoosoontobuildevidencetoshowacausalrelationshipbetweentheincreaseinfreemealsandincreasedacademicperformance
ldquoLower-levelrdquoevidencecanconsistofnuancedbutvaluableelementsthatrelatetoaninterventionrsquospotentialfitwithacommunityQualitativeinformationsuchastestimonyfrompeoplewhohaveexperiencedaninterventionandthosewhohavefacilitatedorimplementeditwillalwaysbeusefultoconsiderandprovidesdetailedlocalcontext
Experimentalstudies--describedstartingatLevel3withcontrolledexperimentsandgoingthroughLevel5withfullyfleshedoutprotocolsandprocessesforreplication--stilloccupythehigherlevelsofthisdescriptionofevidenceThisprioritizationshowsthevalueofrigorousstudyforevaluatingtheimpactofaparticularpolicyorprogramratherthanotherfactorsthatcouldbeinfluencingoutcomesofinterestRememberthatmultipletypesofevidencecancomplementrigorousimpactevaluationsandgivecityleadersadditionalinsightsandbroaderperspective
Accessinformation
Encourageyourreviewerstobegintheirsearchforpromisingpracticesingovernmentandnon-profitclearinghousesandinacademicpolicyjournalsWorkpublishedinthesesourceshasalreadybeensubjectedtoahighlevelofprofessionalscrutinyandgenerallycontainsagreatdealofdetailwhichmeansthatyourreviewersaremorelikelytobeabletopullouttheinformationyouneedtomakeagooddecision
Clearinghouses
Reviewtheevidenceandcollectdetails
23
Herearesomeoftheclearinghouses(collectionsofprogramevaluations)forevidence-basedpoliciesandprograms
ResultsFirstClearinghouseDatabaseThePewCharitableTrustsCollectstogethereightotherclearinghousestoprovideinformationoninterventionsinthefollowingcategoriesadultcriminaljusticeeducationchildwelfarehealthjuvenilejusticementalhealthsubstanceabuseandyouthdevelopmentIndividualclearinghousesincludedinResultsFirstare
BlueprintsforHealthyYouthDevelopment-foryouthdevelopmentandviolenceprevention
CaliforniaEvidence-BasedClearinghouse-forchildwelfareinterventions
TheLauraandJohnArnoldFoundationrsquosSocialProgramsThatWork-forsocialservicesprograms
TheUSDepartmentofEducationrsquosWhatWorksClearinghouse-foreducationinterventions
TheUSDepartmentofHealthandHumanServicesrsquoResearch-TestedInterventionPrograms(RTIPs)-forcancercontrolinterventions
TheUSDepartmentofHealthandHumanServicesrsquoTeenPregnancyPreventionEvidenceReview(TPPEvidenceReview)-forsexualriskbehaviorandsexualhealthoutcomes
TheUSDepartmentofJusticersquosCrimeSolutionsgov-forcriminaljusticejuvenilejusticeandcrimevictimservices
TheUSSubstanceAbuseandMentalHealthServicesAdministrationrsquosNationalRegistryofEvidence-basedProgramsandPractices(NREPP)-substanceuseandmentalhealthinterventions
TheUniversityofWisconsinPopulationHealthInstituteandRobertWoodJohnsonFoundationrsquosCountyHealthRankingsandRoadmapsWhatWorksforHealth(WhatWorksforHealth)-healthbehaviorsclinicalcaresocialandeconomicfactorsandthephysicalenvironment
UKWhatWorksCentresTheBritishgovernmenthascharteredanetworkofsubject-specificorganizationswhichcollectsynthesizeandreportoutevidenceontheeffectivenessofpolicyprogramsandpractices
NationalInstituteforHealthandCareExcellence(NICE)-healthandsocialcare
Reviewtheevidenceandcollectdetails
24
SuttonTrustEducationalEndowmentFoundation-educationalachievement
CollegeofPolicingWhatWorksCentreforCrimeReduction-crimereduction
EarlyInterventionFoundation-earlyintervention
WhatWorksCentreforLocalEconomicGrowth-localeconomicgrowth
CentreforAgeingBetter-qualityoflifeforolderpeople
WhatWorksCentreforWellbeing-wellbeing
WilsonSheehanLabforEconomicOpportunities(LEO)UniversityofNotreDameProvideslinkstotheirevaluationsofinterventionsincriminaljusticeself-sufficiencyeducationhealthandhousingandhomelessness
ClearinghouseforLaborEvaluationandResearchUSDepartmentofLaborProvidesresearchonworkforceandlabor-relatedpoliciesandprograms
J-PALEvaluationsAbdulLatifJameelPovertyActionLabStudiesonpoverty-reducinginterventionsfromaroundtheworld
CampbellSystematicReviewsCampbellCollaborationSystematicreviewofpolicyinterventionsbyscholarlycoordinatinggroupsinsevenissueareascrimeandjusticeeducationinternationaldevelopmentmethodssocialwelfareknowledgetranslationandimplementationanddisability
Benefit-CostResultsWashingtonStateInstituteforPublicPolicyAWashingtonState-charteredprogramtoestablishthelikelihoodofrelativecostsandbenefitsforprogramsacrossthefollowingareasjuvenilejusticeadultcriminaljusticechildwelfarepre-Kto12educationchildrensmentalhealthhealthcaresubstanceusedisordersadultmentalhealthpublichealthandPreventionworkforcedevelopmentandhighereducation
EffectiveInterventionsCentersforDiseaseControlandPreventionRepositoryforevidence-informedHIVpreventioncareinterventions(withstepsonhowtoimplement)
Journalsandindexes
OnceyouhavereviewedrelevantinformationaccessiblethroughclearinghousesitisworthtakingalookatindividualstudiespublishedasscholarlyarticlesYourreviewerswilldothisbysearchingarticleindexesandacademicjournalsUnfortunatelyanumberofusefulacademicjournalsandindexingplatformsmaynotbegenerallyaccessibletoyourreviewersifyoudonothaveasubscriptiontothemToaddressthisconsiderpartneringwithalocal
Reviewtheevidenceandcollectdetails
25
collegeoruniversitytoprovideyouwithalistofarticlesfromthefollowingsources(Thisisalsoanexcellentopportunitytoseekresearchsupportfromanyacademicstakeholdersyouidentifiedpreviously)
Indexes
GoogleScholarincludesmostpeer-reviewedonlineacademicjournalsandbooksconferencepapersthesesanddissertationspreprintsabstractstechnicalreportsandotherscholarlyliteratureincludingcourtopinionsandpatentsItallowsyoutospecifyapublicationrangeinyoursearchandprovideslinkstolikelysourcesforalllistedarticles
PolicyFileIndexisauniqueresourceforUSpublicpolicyresearchinthatitgrantsusersaccesstotimelyupdatedinformationfromover350publicpolicythinktanksnongovernmentalorganizationsresearchinstitutesuniversitycentersadvocacygroupsandotherentities
PolicyArchiveworkstocollectanddisseminatesummariesandfulltextsvideosreportsbriefsandmultimediamaterialofthinktankuniversitygovernmentandfoundation-fundedpolicyresearch
Anothersourceforhigh-qualityresearchisthereportsoftheCongressionalResearchServiceYoucanfindtheirreportsaggregatedhere
Specificjournals
Oftenthemostusefulacademicjournalsforunearthingpolicyorprogramreviewswillbeineachspecificsubjectarea--thatisineducationjournalsforeducationpoliciesYoucanuseageneralindexlikeGoogleScholartobegintolearnwhichjournalsaremostoftenusefulforyoursearchHoweveryoucanalsosearchthefollowingtwogeneralistpolicypublicationsforissuesandarticlesrelatedtoyourtopic
PublicAdministrationReviewJournalofPolicyAnalysisandManagement
Newssourcessubjectmatterexpertsandinterviews
AlthoughitisvaluabletoknowwhathasbeenexploredthroughascholarlyresearchframeworkitisalsousefultolearnwhatkindofinterventionsarebeingtriedinthefieldrightnowandhowpeoplefeelaboutthesetrialsSearchsubscriptionnewsaggregatorslikeLexisNexisorfreenewsaggregatorslikeGoogleNewsorYahooNewsforyourissueareaplusthewordpolicyorprogramtoseewhatkindsofinterventionsarebeingcoveredinthenews
Reviewtheevidenceandcollectdetails
26
DonotforgettocontacttheadvisorygroupmemberswhowerechosenfortheirsubjectmatterexpertiseInadditiontoprovidinggoodmaterialthroughinterviewsthemselvestheymayhaveideasaboutothersubjectmatterexpertsyoushouldcontacttoaskaboutpotentialinterventions
Peercitiesandnetworks
InadditiontoreadingabouttheworkofothercitiesitisidealifyouareabletospeakwithpeoplewhoareinvolvedwithcurrentimplementationsofinterestinginterventionsInyourcityyoulikelyhaverelationshipswithyourneighborsbutitcanbehelpfultolookoutsideofyourregionforsolutionsaswellTofindcounterpartsinothercitieswhoareworkingonaprogramyouwouldliketolearnmoreaboutyoucancontactstaffinintermediaryorganizations(likeGovExoryourprofessionalassociations)whomightbeinapositiontoputyouintouchimmediatelyorwhomightseeanothergoodrouteforsettingupaconversationHereisalistofnetworksthatmaybehelpful
InternationalCityCountyManagementAssociation(ICMA)NationalAssociationofCounties(NACO)NationalLeagueofCitiesUSConferenceofMayorsWhatWorksCities
Collectthedetails
Onceyourreviewershavefoundevidenceonpossiblepoliciesorprogramstoimplementtheyshouldcapturekeydetailsthatwillhelpdecisionmakersunderstandthepotentialeffectivenessandapplicabilityofeachpotentialintervention
Foreachsourcethathasprovidedinformationyouwanttoconveyusethetemplatebelowtocaptureasmanyofthefollowingdetailsaspossible(Youdonotneedtocompletethisforsourcesthatprovideduplicativeinformationorareotherwisenotthatuseful)
Programorpolicyname (egldquoHousingFirstrdquoorldquoblightedpropertiesregistryrdquo)
Sourcetype (egjournalarticleinterviewsubjectmatterexpertstudyassociationcommunityparticipationprocess)
Sourcedetails (egarticletitleandpublicationnameandtitleofinterviewee)
Evaluationmethodology
(egrandomizedevaluationcomparativeevaluationoftwodifferentgroupsbeforeafteranalysis)
Evidencelevel (egLevel12345)
Programorpolicy
Reviewtheevidenceandcollectdetails
27
description
Programorpolicygoal (1sentencecapturingtheprincipalaimoftheintervention)
Existingimplementations
(WherehastheinterventionbeentriedListasmanysitesasyouareawareof)
Characteristicsofimplementationsites
(ForeachimplementationsiteofferbasicgeographicIDmdashcityandstateThenyouwillwanttocollectthedetailswhicharerelevanttoyourdecisionmakersrsquoconsiderationsaboutwhatmakesalocationldquosimilartordquoorldquodifferentfromrdquoyoursituationYouwillatleastwanttoincludeforeachimplementationsitePopulationnumberPopulationdensity(ruralurbansuburban)Socio-economiccharacteristicsBeforestartingaskyourdecisionmakersaboutanyparticularcharacteristicsofprogramimplementationsitestheywouldwanttoknowbeforeseekingtoreplicate)
Characteristicsofimplementationpartners
(Anyavailabledetailsaboutorganizationsinstitutionsorpeoplewhowereinvolvedwithimplementingtheintervention-egindependentchildcarecenterpublicelementaryschoolcitypublichealthnurses)
Characteristicsoftargetpopulationeligibleprogramparticipants
(Anyavailabledetailsaboutpeoplewhoarethebeneficiariesoftheprogramorotherwiseincludedoraffectedbytheintervention-egnumberofprogramparticipantscategoriesofpropertyownersaffected)
Policyorprogramduration
(HowlonghastheinterventionbeeninplaceineachimplementationsiteHasitendedandifsowhy)
Whatbenefitscamefromeachimplementation
(Anyobservedpositiveeffectsoftheinterventionincludingeffectsize)
Statisticalestimatesofinterventioneffectiveness
(Anyavailablestatisticalestimatesoftheinterventionrsquoseffects)
Whatdrawbackswereobserved
(Anyavailabledetailsaboutunintendedorunavoidablenegativeeffectsoftheintervention)
Programorpolicycosts
(Anyavailabledetailsaboutprogramcostsperimplementation)
Reviewtheevidenceandcollectdetails
28
Reviewtheevidenceandcollectdetails
29
DecideonapolicytoimplementbasedonlocalfactorsTogetthecriticalbuy-infromleadershiptoimplementanewpolicyorprogramtheprojectleadshouldscheduleameetingwithdecisionmakerstopresenttheevidencesurroundingthepolicyoptionscapturedinthetemplateDependingonyourinsightintoyourcityleadershipswillingnesstomakechangeyoumaywishtosharetheresearchyouhaveconductedontherootcausesoftheproblemandsomeofyouranalysisontheimpactofcurrenteffortstodate
Considertheapplicabilityofanapproach
AsdiscussedinthepreviouschapterpolicymakersoftenlookforevidenceofeffectiveprogramsinothercitiesbutoftenthequestionarisesofhowgeneralizableaprogramisfromonecitytoanotherTheAbdulLatifJameelPovertyActionLab(J-PAL)suggeststhatinreviewingevidencefromanotherlocationitishelpfultoidentifythespecificmechanismthatwasused(deliveryofprogramorserviceinformationsharingetc)identifythelocalconditionsthatwereinplaceobservethebehaviorchangethatresultedfromthemechanismandfinallytounderstandlocalcontextandconditions(legalorcostconstraintsculturalnormsetc)Thissetofpresentationelementscanbedrawnfromthelistofdetailscollectedintheresearchtemplate
EducationpolicymakersinChicagoILusedfindingsfromprogramsinKenyaandIndiainwhichchildrenwhohadfallenbehindtheirpeersweregiveneducationalsupporttargetedtotheirleveloflearningThemechanismusedwastargetedteachingandthereweresimilarconditionspresentinallthreecountriesAlthoughsomestudentshadfallenbehindinthecurriculumteacherswereincentivizedtoteachgrade-levelmaterialsConsiderationsofhowthelocalcontextinChicagomightaffectimplementationofatargetedteachingprogramincludedunderstandinghowtheneighborhoodsthatthestudentswerefrommayhaveaffectedtheireducationaloutcomesandthehighcostofteachinginsmallgroupsInChicagoMatchEducationprovidedtwo-on-onetutoringwhichmatchedstudentswithwell-educatedindividualswhohavenothadformalteachertrainingforalower-costmodelTheresultswerepositiveStudentswhoweretutoredperformedbetterinmathandweremorelikelytopasshighschoolclasses
ThegeneralizabilityframeworkproposedbyJ-PALisbelowUsethisasatemplateasyouconsiderpotentialpolicysolutions
Decideonapolicytoimplementbasedonlocalfactors
30
SourceTheGeneralizabilityPuzzle
OnceyouhavenarrowedthepolicyoptionsdiscussthequestionsbelowforeachoptionsyouareconsideringThisstepiscriticalasunderstandinglocalcontextissoimportantinsuccessfulpolicyimplementation
CommunityContextQuestionnaire
Localcontext
DoesthisoptionmakesenseinourcommunityIsthisaonesizefitsallapproachIsanyparticulargroupabsentfromthisapproachIfsoisthatok
Youranswers
Effectiveness
WhatwouldweexpecttoobserveifthepilotisaseffectivehereasinCityX
Youranswers
Timing
DoesthetimingmakesenseIsthistherighttimetoimplementthispolicychange
Decideonapolicytoimplementbasedonlocalfactors
31
WherearethepotentialroadblocksthatcouldaffectthescheduleIsthereanywaytomitigatethisHowlongwouldittaketoseeanimpact
Youranswers
Implementation
DowehavethestaffcapacitytoimplementthisArethereadditionalpartnersthatyoucanbringtothetabletosupportimplementationDoesthismakesensegivenourformofgovernment
Youranswers
Authority
DoesourcityhavetheauthoritytoenactthisIfsowhatistheappropriatemechanismAretherelegalbarriersArethereanyexistingpreemptionissuesfromthestateorfederalgovernmentCanthosebeaddressedthroughlegislationadvocacyIstherepoliticalsupportoroppositiontothis
Youranswers
Cost
HowmuchwillthiscostannuallyHowwillwepayforitCanthepolicybeimplementedwithexistingfunding
IfsoisthisfundingreliableHowwouldthispolicyoptionaffectourcitysbudgetintheshortandlongtermWhatotherfundingoptionscanweconsiderIsthereanalternativethatwouldcostlessandgiveussimilarresultsWillanyconstituencybeadverselyaffectediffundingforprogramsthatimpactthemisshiftedtoanotherprogrampolicyorinitiative
Youranswers
Decideonapolicytoimplementbasedonlocalfactors
32
EstablishaframeworktomonitorprogressNowthatyouhavereviewedtheevidenceconsideredlocalcontextandchosenyourpolicyitistimetodeterminehowyourcitywillmonitorprogressofthisnewapproachCitystaffandleadersneedtoestablishtheinfrastructureforongoingcontinuousdecisionmakingbasedonwhetherthepolicychangeisleadingtothedesiredoutcomesforresidentsandwhetherthechangeismeaningfulandrationaltofrontlineworkersSettingarbitrarygoalsatthelocallevelthatconflictwithgoalsatotherlevelsisconfusingtotheindividualsresponsibleforachievingthegoalsGoalsthataresettoohightoachievecanbedemoralizingtostaff
ThissectionincludesstepstosetupavenueinwhichCitystaffandleadershiproutinelyvisitkeyindicatorstodeterminewhatisorisnotworkingandtomakedecisionsinreal-timeFormorereadingonperformancemanagementpleasevisittheGovExPerformanceManagementGettingStartedGuide
InperformancemanagementtermssuchasindicatormeasureandmetricmaybeusedinterchangeablyInthisguidewewilluseindicatorInyourworkthemostimportantthingistobeconsistentinwhichevertermyouuseThisvideomaybehelpfulinunderstandingthisconceptandcommunicatingittoothers
Establishthefundamentalsandgatheryourteam
FirstdeterminewhetherthereisanestablishedvenueorregularmeetingscheduleduringwhichyoucandiscussprogressIfyourcityhasanexistingperformancemanagementstructureonarelatedtopicaddingadiscussionoftheprogressofpolicychangetoyouragendacanbeagreatideaIfthiswillbeastandalonemeetingdecidehowfrequentlytheprojectteamanddecisionmakerswillmeetandputthosemeetingsonthecalendarfortheentirelengthoftheproject
DeterminewhowillstaffandattendroutinemeetingsonCityprogressThesemeetingsmayincludetheMayorandorCityManagerdepartmentheadstaffwhoareimplementingthepolicychangeandanalysts
Identifyindicators
ItiscriticalthatindicatorsbeanimportantcityandcommunitypriorityaswellassomethingtheCityplaysamajorroleinaddressingTheremaybeinstancesinwhichtheCitywillneedtoreachouttoorcollaboratewithexternalpartnerstogetamorecompletepictureofprogressontheoverallpolicychange
Establishaframeworktomonitorprogress
33
Anoutcomeindicatordescribesthedesiredeffectofthetargetedpopulation
ExampleBellevuesfamilyfriendlyneighborhoodsindicatormeasuresthepercentofresidentswhobelievetheyliveinaneighborhoodthatsupportsfamiliesparticularlythosewithchildren
WithyourkeypartnersrevisityourproblemstatementcurrentstateandfuturestateexerciseandbegintoidentifytheoutcomeindicatorsthatwillillustratesuccessFornowchooseasingletop-levelkeyperformanceindicator(KPI)ItiscriticalthattheKPI-thepieceyouwillmeasure-beapriorityforthecommunityandsomethingtheCityplaysamajorroleinaddressing
Outputindicatorsmeasuretheprocessofprogramorservicedelivery
ExampleSanDiegoMayorKevinFaulconersgoaltorepair1000milesofstreetby2020
ReviewingtheadministrativedatasetinventorybegintoparedowntherelevantprogramsanddatasetsthatsupportyourKPIWhatarethemeasuresofeffectivenessforeachprogramTheseareyouroutputindicators
Thediagrambelowillustratestherelationshipbetweenoutputandoutcomeindicators
ThinkaboutimplementationindicatorsasmeasuringstepsintheprogramsrolloutthatarenecessarytoobtaindesiredchangesImplementationoutcomescanfocusontheinterventionstaffdeliveringtheinterventionanorganizationorsystemandcanaddressfeasibilityadaptabilitysustainabilityandcostTwokeyquestionstoconsiderwhendevelopingimplementationindicatorsare
1 Istheinterventionacceptableandappropriatetothecommunityand
2 IsitfeasibletoimplementinterventionXatthistimewiththeseresources
Establishaframeworktomonitorprogress
34
Amoreextensivelistwithdefinitionsofpotentialimplementationoutcomesislinkedhere
UsethetablebelowtochoosethedesiredimplementationoutcomesforyourpilotandtoplanyourmeasurementtechniqueThisshouldincludelevelofanalysis(individualproviderorconsumerorganizationorsetting)yourmeasurementtechnique(surveysobservationinterviewfocusgroupsadministrativedataanalysis)andthephasewhenyouwillcollectthisinformation(earlymidlate)Theremaybemorethanonelevelofanalysismeasurementtechniqueandphaseofdatacollection
Implementationoutcome
Definitionofoutcome
Levelofanalysisandmeasurement
Yourmeasurementtechnique
Phaseofpilotwhenyouwillcollectdata
Usethespacebelowtocaptureyourindicatorwishlist
Indicatorwishlist
Outcomes_____________________________________________________________________________
Outputs________________________________________________________________________________
Implementationoutcomes______________________________________________________________
Setapublichigh-levelgoal
Thehigh-levelgoaliscriticalinestablishingdirectiontogalvanizethepeoplewhowillbedoingtheworkWhendiscussingthegoalusetheMayorsbullypulpitLookforkeymilestonesinCitygovernmenttosharethegoalstatementsuchasthebeginningofaMayoraladministrationaStateoftheCityspeechorotherpublicremarks
GoalsshouldbeldquoSMARTrdquo
Specificaspossible(innamingthepopulationwhowillbebetterofftheactionthecitywilltake)
Measurable
Achievable
Establishaframeworktomonitorprogress
35
Results-focused
Time-bound
ThegoalstatementshouldincludeyouroutcomeindicatorlikeChattanoogasgoalldquoWewillendchronicveteranhomelessnessbyDecember2016rdquoAdiagramtohepyoubuildaSMARTgoalaroundyouroutcomeindicatorisbelow
Component Description Yourturn
Specific WhowillbebetteroffWhatactionthecitywilltake
Measurable Howwillyoumeasuresuccess
Achievable Isthegoalattainable
Results-focused Whatistheexpectedresult
Time-bound When
Setrealisticambitioustargetsandtimelinesforeachindicator
ToestablisharealisticandambitioustargetforyourgoalstatementusetheadministrativedatacollectedforyourchosenKPItoreviewactualCityperformanceinthemostrecentyearcomparedwiththethreeorfourmostrecentyearsworthofdataBegintocalculatesummarystatistics(egtotalminimumaveragemedianmaximum)toidentifytrendsCreatecumulativeandmonthlychartstoidentifytrendsandanomaliesBasedonyouranalysisofhistoricaldataignoretheanomaliesandassumetheproductivitytrendswillfollowthesameoverallpatternfrommonth-to-monthFinallyapplyanincreasedproductivityfactortotheperformanceeachmonthtoforecastfutureprogressBasedonthisanalysischooseatargetthatinspiresimprovementwhileaccountingforreality
Manycitiesthataremorematureintheirperformancemanagementpracticesrecognizetheimportanceoftakingacommunity-levelapproachtoimprovingoutcomesforresidentsThesecitiesarecreatinggoaltaxonomiesthatincludemeasuresrelatedtopublicsafetyhousingandevenrecreationCitiesdriveprogressonthesegoalsbyimplementingstrategiesthatengagemultipledepartmentsandcollaboratewithserviceprovidersinthecommunity
EventuallyyouwilldevelopgoalstatementsfortheoutputsinordertocapturetheefforttoachievetheoutcomecontainedinyourgoalstatementYoursetofindicatorsshouldbecomposedofshort-andinterim-termsupportivemeasureswithaccuratetargetsandrelevanttimelinesAnexamplegoaltaxonomyfromtheCityofLasVegassResultsVegasprogramisbelowUsethistaxonomyasamodelforyourownsuiteofindicatorsandgoalstatements
Establishaframeworktomonitorprogress
36
Yourturn
Thetemplatebelowismeanttohelpyoucraftagoalstatementforeachsupportingmeasure
Goalstatement
Departments
Outcome
Activities
KPI
Target
Timeframe
Conductanddiscussrelevantanalysis
ThrougharoutineseriesofmeetingscontinuallymonitorprogressofKPIssupportingindicatorsandanyotherdatacollectedsuchasqualitativedatatoidentifytrendsthatwillindicatewhetherthechosenpolicyishavingthedesiredimpactThesedatapointsshoulddrivethediscussionduringthesemeetingspointingtoopportunitiesforfurtherinquiryinto
Establishaframeworktomonitorprogress
37
whatisdrivingtheobservedtrendsAnalystsshouldbeencouragedandtrainedtoaskquestionsofthedepartmentsproducingthedataandconductsitevisitswhenpossibletodeepentheirunderstandingofthedataPresentationoftheanalysisshouldfocusononeideaatatimetokeeptheconversationfocusedonkeypointsandspecificopportunitiesforimprovementYoumayconsiderdevelopingapublicfacingdashboardtosharethisprogressmorebroadly
AgeneralcomparisonofthestatusquodatawithwhatwasobservedduringthepilotcanprovidedecisionmakerswithaquickunderstandingofthedifferencesbetweentheoldandnewapproachesalthoughthiswillnotshowcausalityOftenthesecomparisonswillincludeoperationalmetricssuchasdaysordollarssavedbutrememberthattheimpactonpeoplewillbemostimportantIfyouareworkingwithanevaluationpartnertheywillsupportthisstep
Usethespacebelowtosummarizetheresultsofyouranalysis
MeasureTargetvalue(seesectiononsettingtargetsabove)
Pilotvalue(whatdidyouobserveduringthepilot)
Historicalaverage(ifavailable)
Results(WhoisbetteroffIsanyonenegativelyimpacted1-2sentencesreassessingyourgoalstatement)
DocumentyourprocessandprogressincludingfeedbackfromtheparticipantsimplementationpartnersandtheprojectteamDidanythingunexpectedoccurduringthepilotArethereinternalandexternalfactorsthatmayaffectyouroutcomesThisiswheretheimplementationoutcomescanbecanhelpfulIftheinterventiondidnotgoasplannedconsiderwhetheritisbecausetheinterventionwasnotacceptableorfeasible
UsethetemplatebelowcreatedbyDenversPeakAcademytocapturethisinformationIfthe306090daytimelinedoesnotmakesenseforyourcityfeelfreetocustomizeit
CurrentState
FutureState
30Day
60Day
90day
Whatwentwell(+)
Whatdidnotgowellshouldbechanged(-)
IfaquestionordiscussionpointarisesthatcannotbeansweredinthemeetinganassignedstaffmembershouldcapturethequestionduringthemeetingAfterwardthefollowupitemsshouldbesenttorelevantpartnersandthenextmeetingshouldbeginwithadiscussionof
Establishaframeworktomonitorprogress
38
thoseitems
Takeaction
DiscussionsaboutwhetheryourindicatorsaremovingintherightdirectionandwhatmightbedrivingtrendsmayleadtoadiscoverythattherearenotenoughresourcesallocatedtothispolicyorissueTheremaybelimitedfundingforstaffprogrammingortechnologythatisdrivingthedatainthewrongdirectionIfCityleadershipisintheroomthisisagreattimetomakethecaseforadditionalfundingduringthebudgetseasonBycollectinganalyzinganddiscussingdataonthispolicyroutinelyyouarebuildingevidencetomakethecasefortherightleveloffundingtoreachtheestablishedtargets
Establishaframeworktomonitorprogress
39
Assessdataaccessibilityqualityandsecurity
Determineaccessibilityandqualityofdata
OnceyouhavecompiledyourwishlistofindicatorsitistimetodeterminetheaccessibilityandqualityofthedataTopreventcomplicationyoumaywishtoprioritizerelevantindicatorsforwhichyouarealreadycollectingdataAvoidthetemptationofadoptingatechnologysolutiontocollectnewdataatthispointsuchasaphotosurveytoolorotherappFornowyouaretryingtodeterminewhichdatasetsarerelevantaccessibleandhighqualityBesuretoworkwithyourkeyinternalpartnerswhileyouareidentifyingyourmeasuresDepartmentstaffwhoworkonprogramsarefamiliarwiththeindicatorsthatwillaccuratelysupporttheoutcomesanddatabaseadministratorswillbevaluablewhendiscussingaccessibilityandqualityofthedata
Abrieflistofconsiderationsfortheaccessibilityofyourdataisbelow
ArewecurrentlycollectingthisdatasetorwillwebecollectingitforthefirsttimeHowfarbackdoestherecordgo(timehorizon)WouldweneednewtechnologytocollectoranalyzethedataCanthedatabecollectedmanuallyorcanitbeautomatedHowfrequentlyisthedataproduced(continuouslyhourlydailyweeklymonthlyquarterlyyearlyetc)IsthedataavailablefortheentireCitypopulationorasubsetofthepopulationHowgranularisthedata(individuallevelCensusblockneighborhoodcitywideetc)DoesthedatacontainsensitiveinformationIsthereaproxymeasurethatwecanusefornowIsthedatamachinereadable
AsnotedbytheWilsonSheehanLabforEconomicOpportunities(LEO)attheUniversityofNotreDameleveragingadministrativedatacansignificantlyreducethecostsandtimelinesofrigorousevaluationCitiescanestablishaninternaldatasharingagreementacrossdepartmentsincludinglanguagethatenablesadministrativedatatobeusedforresearchpurposesCitiescanalsoidentifyexternaladministrativedata-holderssuchasacountyschooldistrictpolicedepartmenthospitaloranotherstatethatmayhaveoutcomedataimportantformeasuringprogramimpactCitiescanleverageexistingrelationshipsandorworkwithresearchinstitutionstodevelopstreamlineddatasharingprocessesandagreementswiththeseexternalentities
Assessdataaccessibilityqualityandsecurity
40
Evaluatethequalityofyourdata
QualityisasimportantasaccessibilityToevaluatedataqualityyoumaywishtocreatearubricliketheonebelow
Value ValueDescription
3 ExcellentDataQualityThedataiscompleteaccurateandupdatedregularly
2 AdequateDataQualityThedataissomewhatcompleteaccurateandupdatedfairlyregularly
1 PoorDataQualityThedataisnotcompleteaccurateorupdatedregularly
ThisisagreattimetobuildonthedatainventorythatyoustartedinthegatherinformationpageSuggestedadditionalfieldstoaddtoyourdatainventorytemplatearebelow
Program DatasetTitle
DatasetDescription DataSource
Departmentorexternalpartner
DoestheCityalreadycollectthisdata
Whatisthenameoftheprogram
Human-readablenameofthedataset
Shouldbeunderstandablebynon-technicalusers
Whatistheoriginalsystemapplicationfilethathousesit
Departmentorexternalpartnerthatmaintainsthedata
Isthisdataalreadycollected
Updatefrequency-ThefileformatforthedatasetTypicallycsvxlsxshapefiletxtetc
Format-Howoftenthedatasetisupdated
Timerange-Theperiodoftimecoveredinthisparticulardataset
Datause-Howisthisdatasetcommonlyused
Datausers-Whotypicallyusesthisdataset
Dataownerstitle-Titleofthedataowner
Dataownersposition-DataOwnersjobtitle
Dataownersemailaddress-DataOwnersemailaddressWhomanagesthedataandorisresponsibleforgrantingpermissiontoaccessthedataWhounderstandswhatthedatasetincludesandcananswerquestionsaboutit
Dataqualityconcerns-Describetheconcernsyoumayhaveaboutthequalityofthedataset
Assessdataaccessibilityqualityandsecurity
41
Division-DivisionwherethedataismaintainedIfmorethanonedivisionisresponsiblelisttheprimaryownerofthedataset
Sensitivedatacomments-DescribetheissuesthatwouldpreventthecityfromsharingthedatapubliclyCommonsensitivityconcernsincludeprivacyviolationssecurityissuesorhighcostorstaffdemandstopostthedata
Publishingstatus-Thistheplacetoputanycommentsregardingthecurrentstatusofthedataset
Publishedlink-Thisisthelinktothepublisheddatasetontheopendataportalorcitywebsite
Reviewdatasecurityandprivacyprotectionprotocols
OftenapolicychangealsoinvolvesachangeinthedatathatisbeingcollectedhowthatdataisbeinghandledwhohasaccesstoitandahostofotheradjustmentsAsyoutakestockofthedataassetsthatareavailableforyourinitiativeitisvitaltoalsothinkthroughwhatwillbedonetoprotectthatdataandtheprivacyofthepeopleitrepresentsAbigpartofgooddatasecurityinvolvesstrongdatagovernancepracticesthatdefineandidentifysensitivedataandconsciouslyworktobalancethevalueandriskofstoringsuchsensitivedataOnewayofevaluatingyourdatacomesfromcaselawrelatedtotheUSFreedomofInformationActwhichwehavetranslatedtothechecklistbelow
1 Woulddisclosureresultinasubstantialinvasionofprivacy2 Haveweconsideredtheextentorvalueofthepublicinterestandthepurposeorobject
oftheindividualsseekingdisclosure3 Istheinformationavailablefromothersources4 Wastheinformationgivenwithanexpectationofconfidentiality5 Isitpossibletomouldreliefsoastolimittheinvasionofindividualprivacy
TheseriskscouldbemitigatedusingavarietyofstrategiesfromaggregatingdatatoobfuscatingindividualstoredactingfieldsthatmayincludePersonallyIdentifiableInformation(PII)orre-evaluatingyourdatacollectionmethodstofindalternativewaysofgettingtheinformationyouneedThesestepsshouldbecodifiedandmanagedbyadatagovernancecommitteeifpossibleForthesensitivedatathatisdeemedessentialcreateaplanaroundpotentialdatabreachesinordertoquicklyandconfidentlyrespondtoemergencysituations
ProactivelypreparingforthesesituationsisoftenthebeststrategyfordatasecuritybutdonotforgetthetechnicalsideInvolveyourITadministratorsinconversationsaboutdatastorageandaccessensuringthattheywillbeprotectedbyenterprise-gradecybersecuritysystems
Assessdataaccessibilityqualityandsecurity
42
Assessdataaccessibilityqualityandsecurity
43
Designimplementationandevaluation
Designtheimplementation
NowthatyouhaveyouraccountabilityplanandkeyindicatorsitistimetogetspecificaboutyourplanforimplementationThetemplatebelowwillhelpyouthinkthroughthespecificmechanismsofimplementingpolicychange
UnderstandingorganizationsandsystemsiscriticaltounderstandingandmakingdecisionsaboutwhereandhowinterventionsshouldbeappliedApolicyimplementedinasinglesettingmayshowpromisingresultsbutimplementingmultiplepoliciesandprogramsatthesametimewithdifferentactorsresponsibleforeachaspectcanhelpensuresustainablebehaviorchange
InSomervilleMAShapeUpSomerville(SUS)isacitywidepublichealthprogramthatisimplementedacrossmultiplelevelsofinfluenceandsystemsthroughinterventionsbeforeduringandafterschooltoincludechangestobuiltenvironmentsuchasbikelanesregionalmasstransportationandphysicaleducationequipmentinschoolsandgymstoincreasephysicalactivitywellnesspoliciesandschoolgardensnutritioneducationduringandafterschoolandhealthyrestaurantprogramstosupporthealthyfoodandhealthyeatinghabits
Usethetemplatebelowtoplanyourimplementation
Policyimplementationplanningtemplate
ListthespecificdesiredoutcomesInthepreviousstepyouworkedwithdepartmentstafftoestablishspecificmeasurableoutcomesandinterimindicatorsPrioritizetheindicatorsthatwilldemonstrateprogressUsethespacebelowtocapturethedesiredoutcomesofthepolicychange
ExampleShapeupSomervillebeganwiththegoalofpreventingobesityin1stthrough3rdgraders
Yourturn
____________________________________________________________________________
____________________________________________________________________________
Designimplementationandevaluation
44
SpecifythetargetpopulationAreyoudesigninganinterventionthattargetsonlycertainsitesAreyouplanningonimplementingaprogramorinitiativeforaspecifictargetaudienceClearlyspecifythepopulationyouplantotarget
IfyouplantoconductarigorousimpactevaluationasnotedbytheWilsonSheehanLabforEconomicOpportunitiesitisimportanttoestablishconcreteobjectiveeligibilitycriteriaandensurethatthesecriteriaareappliedconsistentlybyallstaffduringscreeningandintakeItisalsovitaltoidentifytheindividualswhoareeligibleforaparticularprogramorserviceevenifeveryoneeligiblecannotbeservedbyexistingresourcesThishelpscitiesidentifyanyunmetneedsandguidefutureresourceallocationwhilealsocreatingthepossibilityofarigorousimpactevaluationbyidentifyingapotentialcomparisongroupFinallywhenconductingscreeningandintakeitisimperativetocollectandmaintainbasicinformation(egnamedateofbirthgender)aboutallprogram-eligibleindividualsnotjustthosewhoparticipateintheprogram
ExampleIntheShapeUpSomervilleexampletheprogrambeganbytargeting1stthrough3rdgraders
Yourturn
____________________________________________________________________________
____________________________________________________________________________
DefinethelocationWherewillthepolicychangetakeplaceWillimplementationtakeplaceatonesiteormultiplesites
ExampleShapeUpSomervilleinterventionsweredeliveredinschools
Yourturn
____________________________________________________________________________
____________________________________________________________________________
ClarifyimplementationpartnersandarticulatetheirrolesWhatarethekeyorganizationsinstitutionsorpeopleresponsibleforimplementationTheprojectteamshouldbeasclearandspecificaspossibleinprovidingadescriptionoftheconcreteactivitiesthatimplementationpartnersaremeanttocarryoutPracticeprofilesmaybeahelpfultooltohelpdefinewhatisrequiredoftheactorswhoareresponsibleforimplementingtheintervention
Designimplementationandevaluation
45
ExampleInSomervillemayoralleadershipandengagementofmultiplecityagencies(publichealthprimarycareschoolsCambridgeHealthAllianceetc)streamlinedandconnectedthevariousShapeUpSomervilleinitiativesandgrantswhichhelpedtokeepthetalentandenergyfocusedonhealthyeatingandactivelivingforallsectorsofSomervilleSUSpartnersultimatelyincludedschoolsnonprofitsCityandstateagenciesandnationalfoundations
Yourturn
____________________________________________________________________________
____________________________________________________________________________
DescribetheinterventionWhatarethestepsactivitiesorprocessesRememberthatthemechanismthespecificstepsorprocessthatleadtobehaviorchangeiskeytounderstandingtheeffectivenessofthepolicychangeandsharingtheresultswithothers
ExampleGoalsofShapeUpSomervilleweretoinfluenceeverypartofanearlyelementaryschoolchildrsquosdayndashbeforeduringandafter-schooltopromotehealthyeatingphysicalactivityandtheirweightstatusInterventionincludedfoodserviceenhancementstoimprovebreakfastandlunchandstaffprofessionaldevelopmentwalktoschoolactivitiesSUSclassroomcurriculumandprofessionaldevelopmentschoolwellnesspolicydevelopmentSUSafter-schoolcurriculumandprofessionaldevelopmentoutreachandeducationtohomethroughmaterialsforumseventsldquoSUSapprovedrdquorestaurantscommunityoutreachandcapacitybuildingthroughpolicydevelopmenttrainingsmediaplacements
Yourturn
____________________________________________________________________________
____________________________________________________________________________
ClarifytimelineandmilestonesfordeliveryandevaluationHowfrequentlywilltheservicesinterventionbedeliveredHowlongwilltheprogramrunWhatisthetimetoexpectedimpacts(eg12monthsafterprogramcompletion)
Designimplementationandevaluation
46
ExampleShapeUpSomervilleledbyTuftsUniversityFriedmanSchoolofNutritionScienceandPolicyconductedaquasi-experimental(nonrandomized)interventionstudyamong1stthrough3rdgradestudentsinSomerville(intervention)andtwocomparisoncommunitiesfor9-monthsduringoneschoolyear(fall2002ndashspring2003)ThestudyshowedpositiveassociationoftheinterventiononchangeinBMISUSreleasedreportssummarizingevaluationsprogresstodateplansforadditionalprojectsandimpactstenandfifteenyearsintotheinitiative
Yourturn
____________________________________________________________________________
____________________________________________________________________________
PlanyouranalysisTheprevioussectionaddressedestablishinganaccountabilityframeworktomonitorprogressandimplementationDeterminewhichindicatorswillbediscussthroughthatprocesswhowillsharethedataandanalysisandhowfeedbackfromthoseconversationswillbeincorporatedbytheprojectteam
ExampleTheMayorrsquosofficeimplementedadata-drivendecisionmakingapproachcalledSomerStatwhichwassynergisticwiththeapproachestakenbytheresearchandevaluationteamsdeployedforthespecificgrants
Yourturn
____________________________________________________________________________
____________________________________________________________________________
PlanforsustainabilityIfthepolicyandprogramchangeissuccessfulhowwilltheprojectteamsustainandexpandtoadditionalsitesorpopulations
ExampleAlthoughmostofthepartnershipsforgedintheearlyyearsweremaintainednewgrantsandcollaborationsexpandedattentiontoolderpublicschoolstudentsactivetransportationandtheGreenLinesubwayextensionthebuiltenvironmentfarmerrsquosmarketeffortssuchasldquoGrowninSomervillerdquowithUnionMainStreeteconomicdevelopmentandhealthdisparitiesaswellasdatacollectionmonitoringandsurveillance
Yourturn
Designimplementationandevaluation
47
____________________________________________________________________________
____________________________________________________________________________
Pilotplanningtemplate
IfyouplantoconductapilotfirstdeterminethepurposeofthepilotDoyouwanttoobservewhetherthepolicychangeresultsinthedesiredbehaviorchangeAreyouconcernedaboutfeasibilityscalabilityorcostDoyouwanttotestwhetheryourdesiredresultscanbeachievedthroughthispolicyapproachClarifythegoalofthepilotandthequestionsyouhopetoanswerThiswillhelpyouidentifythekeyindicatorsthatyouwillusetomeasurewhetherthepilotissuccessfulBeforebeginningestablishabaselineofcurrentperformanceandsetyourtargetsaccordinglybasedonwhatyouhopetoaccomplishthroughthepilot
NextestablishthepilotplanforyourprojectteamincludingmilestonesforteamcheckinstodeterminewhethertomoveaheadorchangecourseThisisseparatefromthemeetingsthatwilloccuraspartoftheaccountabilityframeworkUsethespacebelowtocreateyourtimelineanddevelopyourmeetingschedule
Task Personresponsible Startdate Checkindate Deliverable
Planyourevaluation
WhileyouaredesigningtheimplementationitisalsotimetobegintodevelopyourplanforanalysisAsnotedbytheWilsonSheehanLabforEconomicOpportunitiesdecisionsaboutprogramsetup-includingoutreachscreeningintakedatatrackingandenrollment-candeterminewhetherornotarigorousimpactevaluationwillbepossibleAsdiscussedcityleadersmayfindithelpfultoincludeevaluationpartnersinearlyconversationsaboutprogramsetupDuringpreviousstagesyoumayhaveworkedwitharesearchertoplanyourproposedpolicychangeYoushouldbeclearaboutthepurposeoftheevaluationanddevelopanevaluationplanBelowareadditionalconsiderations
Causationvstrendanalysis
IntheEstablishaframeworktomonitorprogressstagetherewasguidanceontrackingtrendsovertimeItisimportanttonoteherethatmonitoringdataisdifferentfromanalyzingacausaleffectTrendanalysiswillillustratethatchangesareoccurringandmayleadtoquestionsthatcouldbeansweredbyyourevaluationbuttrendanalysisalonedoesnotprovideevidenceaboutwhatisdrivingthosechangesYoumayseechangesinyour
Designimplementationandevaluation
48
outcomeindicatorsbasedonvariablesoutsideoftheCityscontrolForinstanceachangeinfederaltradepolicycanhaveanimpactemploymentratesinyourCityoutsideofyourCitysnewjobtrainingandplacementprogram
ToaddresscausalityyoumustcomparetheoutcomesofindividualswhoreceivedtheinterventionwiththosewhodidnotWithoutacomparisongroupofsimilarindividualswhowerenotexposedtotheinterventionitisimpossibletoknowwhichfactorsareresponsiblefortheoutcomesRandomizedControlTrials(RCTs)inwhichindividualsarerandomlyassignedtotreatmentandcontrolgroupsarethegoldstandardinprovingcausalityIfyoucandemonstratethatoutcomesimprovedforthosewhoreceivedtheinterventionanddidnotimproveatallorimprovedlessforacomparisongroupyoucanbesomewhatcertainthattheinterventionwasthecause
Examplesofothernon-causalimpactevaluationsthatarerigorousandreliableincludecertainquasi-experimentalstudiesusingcomparisongroupdesignssmartinterruptedtimeseriesanalysesthatmodeltrendsovertimeinwaysthatallowestimationofinterventioneffectswelldesignedpre-posttestanalysesneedsassessmentsimplementationstudiesetcFormoreinformationondifferenttypesofevaluationsseetheSageResearchMethodswebsite
Lookfordifferencesingroups
BuildintimetoalsoexaminethepilotdatatoseeifgroupsareexperiencingdifferentresultsReviewingperformancebasedonoutcomemeasuresforapopulationinaggregatemayhidethesubtlewaysthatsomepeoplemaybeaffecteddifferentlysoitisnecessarytocarefullyanalyzethedatafordisparateoutcomesForinstancetheMovingtoOpportunity(MTO)studythatexaminedtheimpactofhousingvouchersonfamilieswholivedinhigh-povertyneighborhoodsfoundthatchildrenwhomovedbeforetheageof13hadbettereducationandhigherincomesasadultsHoweveradultsdidnotseeimprovededucationandemploymentoutcomes
Disaggregatingdataintomorespecificcategoriesofraceandethnicitycanhelpunderstandtrendsanddisparitiesinamoregranularwaywhichcanhelpdecisionmakerstargetpoliciesandinterventionsinamoreequitableway
EarlierintheRoadmapweworkedtoidentifystakeholdersandwhomightbepositivelyandnegativelyimpactedbyachangeinpolicyUsetheresultsofthatexercisetoinvolvethecommunityininterpretingdataanalysisandtogatherqualitativefeedbackfromthecommunitytocallattentiontoissuesForinstanceanincreaseinyouthdetentionmaybeexplainedbyinefficienciesorlackofaccesstotransportation
Designimplementationandevaluation
49
Designimplementationandevaluation
50
Shareyourstoryandsuccess
Sharetheresults
Storytelling
AnintegralstepinyourstrategyissharingtheresultsofyourpilotorpolicychangewithyourcommunityThisincludesdevelopingmessagingaroundtheworkofyourorganizationandcreatingaplantodisseminateyourstoriesWhytellstoriesaboutyourwork
Informandorunderstandasubjectorsituation
Giveneededcontextofdataorbitsofinformation
Relatablestoriesarememorableforreadersviewers
ReadersorviewerswillseetheinformationasmorecompellingwhenthereisademonstratedandrelatablechangeorimpactonthemselvesortheirfriendsneighborsandcolleaguesStoriesareaboutpeopleandgovernmentisfundamentallyabouthelpingpeopleSotellthestoriesofthepeopleyouandyourcitycolleagueshelped
Keepthesetipsinmindasyoudevelopyourstorywithyourteam
WhatwereyourfindingsandwhyshouldanyonecareTellyouraudiencehowtheproblemwassolvedWhathastheimpactbeenonresidentsandcitystafftodateWhatwasthesuccessandweretheremultiplesuccessesalongthewayIfyouhavepreliminaryresultsordatashowingimprovementtosharethisisthetimetodoit
WhowasinvolvedandimpactedReviewthelistofstakeholdersHowwereeachofthemimpactedbytheprogramHumanizeyourstoryandimpactbyaddingadditionalvoicesThesevoicescanbeoneofthefollowing
Apersonorfamilywhowaspositivelyaffectedbythechangesuccess
Aninvolvedthird-partydescribingboththeproblemandthesolution(ieanorganizationnonprofitprovidersbusinesstradeassociationsportsteametc)whoispartofthecommunitythatthegovernmentworkedwithonthechangesuccess
Someonefromthegovernmentitselfwhocanspeakaboutthechangesuccess
TellmoreaboutyourcityDescribeyourcityincludingpopulationdemographicsindustriesetcWhatareitschallengesWhatisitknownforWhatmakesituniqueHowisitsimilarordifferentfromothercitiesintheUS
Shareyourstoryandsuccess
51
DescribeyourcityrsquoschallengeWriteoutthechallengeyourcityisfacing(asdefinedinyourproblemstatement)HowdoesitaffectresidentsHowdoesthisproblemaffecttheworkofcitystaffmembers
AddthedetailsProvidedetailsonyourappliedresearchstrategyTalkaboutwhatyourcityhasdonenthepasttoresolvetheissueDiscusshowyourstrategyhaschangedwhenyoubegantoadoptthisapproachWasmoneysavedorinvested
Usethisstorymapexercisetoorganizeyourthoughtsandguaranteethatyouincludealloftheelementsofatraditionalstory
AfteryoudevelopyourmessagingcreateacommunicationsmarketingandoutreachstrategydesignedtospreadyourmessageYourstrategyshouldwalkyouthroughthefollowingkeyactivities
1 IdentifyandprofileyouraudienceWhencreatinganddisseminatingyourmessageitisalwaysgoodtothinkaboutyouraudienceWhoexactlywouldbeinterestedinwhatyouhavetosayThegeneralpopulationofthecityPeoplewhoworkinnonprofitsBusinessownersItmattersbecausenoteveryonewholivesinacitymaybethetargetoftheinformationBusinessownersmaybemoreinterestedindatashowingeffectsonthebusinessdistrictthanothersandsoonWewillcoverthislaterbutthisstepwillalsohelpyoutothinkofthemosteffectivewaytoreachyouraudience
2 Defineyourcalltoaction(ifyouhaveone)SpendsometimeconsideringwhyyouareinterestedintellingyourstoryDoyousimplywanttoupdatethepubliconyourprogressorcelebrateyoursuccessesAreyouinterestedinyouraudiencechangingitsbehaviororsupportingaspecificinitiativebeinglaunchedbyyourorganizationIfyouwanttohaveyouraudienceactorbehaveinacertainwaybespecificandincorporateitinyourmessaging
Shareyourstoryandsuccess
52
3 SelectperformancemeasuresThinkabouthowyouwoulddetermineifyourcampaignissuccessfulandselectappropriateperformancemeasuresYouwillrevisitthesemeasuresattheendofyourcampaigncommunicationsplanimplementationtodeterminewhetheryouneedtomodifyyourcommunicationsandoutreachstrategyforthefuture
4 MethodstocommunicateyourstoryTherearemanywaystocommunicatesuccessofyourprojectHerearesomesuggestionsonreachingaudiencesItisimportanttodeterminewhichmethodsarethemosteffectiveforyourdesiredaudiencetoreceiveinformationDoyourresearch(PewResearchCenterhasexcellentstudiesonthereachofmediadigitalmediaandmarketing)ForexampleifseniorsareyourtargetpopulationandtheyarelessinclinedtouseInstagram(thatistrue)youmaywanttoconsiderFacebookorthenightlylocalnewsbothofwhichareusedmoreoftenbyseniorsBelowisalistofsomemethodsandtoolsyoucanuse
i Pressreleasestolocalnewspapersandtvstations
ii ShortsmartphonevideosAgreatwaytoengageanaudienceisthroughvideoAskonequestionofsomeonewhoispartofthestoryMaybethereisasocialworkerwhocandiscusshungerinyourcityThesevideoscanbereusedonmultipleplatforms
iii Long-formwriting
iv TellingindividualstoriesthroughaseriesofphotosandshortnarrativesforpublicationonFacebookorInstagram
v Op-edsopinionpiecesplacedinthelocalpaper
vi Emailupdatestoyourlist
vii Connectingwithappropriateorganizationsthatareinvestedinthetopicandworkwiththecommunityandaskthemtodisseminateinformationtotheirmembers
viii EmployingldquoWordofMouthrdquo(WOM)tacticsbycreatingtownhallsorfreeeventsforresidentstoattendintheirownneighborhoods
5 DeterminethetypeofcollateralUsuallycampaignsthattargetresidentsrequireseveralavenuestoreachthemPeoplearenotalwaysinthesameplaceorspaceHavingakeenunderstandingofyouraudiencewillassistyouwithyourchoicesonwhatneedstobedevelopedSometimesthatmeansimagesforsocialmediaaonepageflierthatgivespeopleinformationonattendinganeventoranin-depthbrochure
Shareyourstoryandsuccess
53
6 ImplementyourstrategyKeepinmindstrategiescanbefluidSometimessomethingmaynotturnoutasexpectedevenifduringplanningsessionsitwasldquosupposedtordquoItisokYoucananalyzewhathappenedduringadebriefsessionandonceyoursurveysandothermeasurementshavebeenfinalized
Visualizations
Withtherightvisualizationyoucanreviewandshareinformationinsightsanalysisperformanceandtrendsallofwhichisvaluabletoacross-sectionofthepublicButwhichvisualizationisrightforyouTochoosetherighttypeofchartgraphdeterminethetypeofquestionyouwanttoanswerSomecommoncharttypesarebelowandmoreinformationcanbefoundhere
Exploreadditionalanalyticaltechniques
Duringthegoal-settingphaseyouestablishedoutputoutcomeandimplementationmeasuresBasedontheresultsofyourpilotyoumayneedtoadjustyourmeasuresWiththepilotdatainhandyoumayhavesomeideasofhowfutureanalysescouldbedifferentandpotentiallyimprovethequalityofyourprogramTakesometimetobrainstormotherrelevantdatasetsthatyoudidnothavethetimetopursueearlierincludingmessydepartmentaldatathatneedsmoreattentionorsourcesofpublicdatafromthestateandfederalgovernmentnonprofitsandevenprivatecompaniesthatcouldsupplementyourunderstandingofanissueUsingmachinelearningtechniquesyoumightbeabletobetter
Shareyourstoryandsuccess
54
predictdemandforhomelessservicesusingNOAAweatherforecastspreemptwatermainbreaksusingsoilstratigraphyorinvestineconomicdevelopmentbymonitoringrealestatelistingdata
Advancedanalyticaltechniquescanalsohelpyourimplementationrunmoreeffectivelybyintegratingdata-informedtargetinganddecisiontoolsintotheprocessByautomatingroutineworkflowsandsurfacingsubtletrendsthesetoolscanhelpyouiterateinamoreprecisemannerandprovidekeystrategicguidancetofront-lineemployeesthatwouldotherwisebeunavailableThesesortsofdata-science-enhancedprocessescanbeclassifiedasoneofsixprojecttypesasexploredfurtherbytheNewOrleansLANOLAlyticsteamFormoreguidanceonscopingasimilarprojecttheGovExadvancedanalyticsscopingguidegivesyouastep-by-stepoverviewofhowtobesuccessful
Usethespacebelowtolistyourideasforpossibleadvancedanalyticsprojectsbytype
Typeofproject Ideas
Findingtheneedleinthehaystack
Prioritizingforimpact
Earlywarningtools
Betterquickerdecisions
Optimizingresourceallocation
Experimentingforwhatworks
Keepthefeedbackloopgoing
GatherinformationonpublicsentimentaboutpolicychangethroughfocusgroupssurveysandpublicmeetingsledbyadvisorygroupmembersNotewhethernewinformationisavailablethatcanhelpanalyzeorcontextualizeyourprogress(egfederalorstatedatareports)Sharetheresultsofyourpilotandthepolicychangeprocesswithothercitiesandresearchers
Shareyourstoryandsuccess
55
WorkingwitharangeofevidencePresentlythereisalotofattentionandinvestmentinevidence-basedpolicymakingincludingthefederalbipartisanCommissiononEvidence-BasedPolicymakingthePewCharitableTrustsreportsandtoolkitsonHowStatesEngageinEvidence-BasedPolicymakingandtheLauraandJohnArnoldFoundationsupportedEvidence-BasedPolicymakingCollaborativeManyoftheseinitiativesfocusonrigorousevaluation--meaningrandomizedcontrolledtrials(RCT)orotherexperimentalmodels--asthebestkindofevidencetouseHoweverthisbodyofevidenceisofteninsufficientforthespecifickindsofquestionsthatgovernmentscanhavewhenmakingadecisionaboutpolicyInthisguideweproposeanapproachtosupportgovernmentsinincorporatingadditionaltypesofevidenceintotheirdecisionmaking
OnewaytounderstandarangeofevidenceistouseNestarsquosStandardsofEvidencetoframeanddefineevidenceTheBritishnonprofitNestaworkstoinjectpositivechangeintothewayUKgovernmentpoliciesandprogramsfunctionOneoftheirkeyareasofworkhasbeendevelopingWhatWorksCentresorganizationsthatcollectcommissionsynthesizeanddisseminateevidenceaboutinterventioneffectivenessacrossarangeofsocialimpactareasNestadevelopedthefollowingStandardsofEvidencetoprovideausefulwaytodocumenthowdifferentkindsofinformationdemonstratedifferentlevelsofevidence
Workingwitharangeofevidence
56
Theimportanceofpotentiallybiasedinformation
Onechallengepresentedbyanevidenceclassificationthatacknowledgesthevalueofnon-scientificformsofevidenceistofigureouthowtobalanceallofthepiecestogetherwhenyouusethemTounderstandhowtodothiswehavetothinkabouthowtopositionpotentiallybiasedinformationindecisionmaking
WhenwelookatinformationwhichhasnotbeenrigorouslyevaluatedweacceptthatthisinformationmightbebiasedThewholereasonweconductrigorousevaluationistoremovebiasfromourobservationsWeconductstructuredtrialstoproduceresultsthatareconsistentandgeneralizableIfafindingisunbiasedthisaccuracymdashthislackofbiasmdashcanbeprovedbecauseanyindependenttestercanreproduceitwhentheyfollowanidenticalprocess
YoucanlookatpotentiallybiasedinformationasjustinformationthatisearlieroninitspathtorefinementreplicationandverifiedaccuracyApieceofinformationbasedonanyoneindividualrsquosobservationsislikelytobebiasedbecauseitiscomingfromjustonesourceBiasisreducedbygatheringtogetherlotsofindependentsourcesofinformationandfiguringoutitsmostcommonattributes
MaybethebiasedinformationmakesaclaimthatistoolargemdashitsayssomethingistrueunderallconditionswhenitisreallyonlytrueundercertainconditionsorthataneffectofaninterventionisbiggerthanitstrueaverageorthataneffectlastsovertimewhenitdoesnotPerhapschangingtheclaimtobemorespecificmdashcriticallyreviewingtheinformationtoseeifspecificconditionsmustbemettoachievetheoutcomesmdashwouldmaketheclaimmoreaccurate
IfyoutestedyourclaimfounditdidnotholdupinsomecaseschangedittesteditagainuntilyourclaimrepeatedlyheldupthenyoucoulddemonstratethattheclaimwasnolongerbiasedbutaccurateThisprocessoftestingimprovingtestingandvalidatingiswhatwegetfromstructuredscientificstudy
Withindividualpiecesofinformation--crediblehypothesescasestudiesexpertopinion--youcannotassumethatkindofaccuracyAtthesametimetheseearlyobservationsarefoundationaltoanyreplicablescientificobservationsWewillnevergetanywhereifwedonotstartwiththesepiecesofinformationeveniftheymightbebiased
Combininglevelsofevidencetogetthemostdetailedinformationforyourcity
ItisespeciallyimportanttoacceptdifferentformsofevidencewhenyoufacequestionsthathavenotyetbeentestedbyindependentstudiesSomeofthemostimportantquestionsrelatetothingsthatareveryspecifictoyoursituationYoufaceaparticularcombinationof
Workingwitharangeofevidence
57
issuesactorstrendsandresourcesthatisultimatelydifferentfromeveryotherplaceintheworld
ThisisimportantbecauseaspolicypractitionersandnotscientistsyoucaremoreaboutyourspecificcasethanahypotheticalaveragegeneralcaseYoucarelessaboutthetotallikelyrangeofimpactsfromapolicythanhowthissingleimplementationwillworkYouwantinformationaboutpoliciesthatwillprovideyouwiththebestpossiblepredictionabouthowtheywillworkinyourcontext
ByputtingdifferentkindsofevidenceintothesameframeworkNestarsquosStandardsofEvidenceimplythatwhenyoucannotgetrigorouslytestedexperimentallyverifiedinformationitisappropriatetoworkwithwhatwedohaveThisisrelevantforthemunicipalimplementationcontextbecausenostudypulledfromaliteraturereviewwillaccountforallofthevariablesyoufaceinyourowncontext
Intheabsenceofhavingastudythatpredictshowaninterventionworksinyourspecificcontextyouwillneednon-scientificinformationtotrytofillinthosegapsYouwillneedtogatherstoriesfromresidentsstaffmembersandotherstakeholdersthatsitonLevels1or2ofthestandardsofevidence
Someofthekindsofinformationyouwillprobablygetmoreeasilyfrompeoplethanstudiesinclude
howpeopleincitieslikeyoursreactedtoaninterventioninpublicforumshowpeoplewhoimplementedaninterventioninacitylikeyoursfiguredouthowtogetitpassedhowthecitycommunicatedpubliclyabouttheapparentbenefitsanddrawbacksoftheinterventionthepoliticalacceptabilityoftheintervention
Youcanremainsimultaneouslyawarethattheinformationyouaregatheringthroughindividualstoriesisbiased--notgeneralizableinformedbyspecificperspectives--whilealsorecognizingthathavingsomeinformationaboutsomethingimportantisbetterthannoinformationEvidencefromthefirsttwoevidencelevelsareessentialbuildingblocksofknowledgeandperfectlyusefulwhenyoucannotgetdetailedunbiasedknowledge
Youhavetostartsomewhereandyoushouldalwaysstartwithevidenceyouhaveaboutyourveryspecificconditionsthatyouknowareimportantforsuccess
Workingwitharangeofevidence
58
Thankyou
Interviewswithcities
GovExheldsemi-structuredinterviewswithsixstaffmembersinfourcitiestoestablishtheinitialstepsoftheRoadmapWeaskedrespondentstotalkaboutaninitiativeorpolicythattheyhavebeenresponsibleforimplementingintheircityRespondentsfromthreecitiesspokeabouttheireffortstoaddresshomelessnessandrespondentsfromonecityspokeaboutblightAfterrespondentsidentifiedtheproblemtheyspokeabouthowthecityprioritizedthisissuethevariousstepsinvolvedandbarrierspresent
GovExconductedtextanalysisoftheseinterviewswhichrevealedthatpeoplewasthemostcommonlyusedtermduringtheseconversationswhendiscussingthestepstochangeaprogramorpolicyThisisathemethroughoutthisRoadmapThefirstsectionbeginswithidentifyingtherelevantpeopleTheconversationsaroundevidencearefocusedonhumanbehaviorratherthanmethodologyorstrengthofevidence
Thankyou
59
OncewehadadraftoftheRoadmapwefollowedupwithcitiesonanindividualbasistolearnmoreaboutwhatwouldbehelpfulWearegratefultothefollowingcitystafffortheinterviewsandindividualconversations
MichaelBaskinChiefPolicyOfficerChattanoogaTN
MelissaBridgesPerformanceandInnovationCoordinatorLittleRockAR
VictoriaCarreoacutenAdministrativeOfficerLasVegasNV
TyroneGrandstrandCommunityDevelopmentPlannerFargoND
TibLaughlinDirectorofGeneralServicesKansasCityKS
TimMorelandDirectorofPerformanceManagementandOpenDataChattanoogaTN
CharlesRosendahlManagementAnalystDentonTX
Thankyou
60
MelissaSiebenAssistantCountyAdministratorKansasCityKS
GinaThompsonDirectorofStrategicServicesLewisvilleTX
JHUFacultycollaboration
GovExalsoconvenedthecitycontactsandfacultypartnersinaseriesofAdvisoryGroupmeetingsThesemeetingsservedtoconnectcitiesandfacultytounderstandoneanothersperspective
StefanBaralBloombergSchoolofPublicHealth
AshleyBernerJHUSchoolofEducation
MelissaDavey-RothwellBloombergSchoolofPublicHealth
JonLinksBloombergSchoolofPublicHealth
TamarMendelsonBloombergSchoolofPublicHealth
KristinMmariBloombergSchoolofPublicHealth
KeshiaPollackPorterBloombergSchoolofPublicHealth
JoshSharfsteinBloombergSchoolofPublicHealth
LizStuartBloombergSchoolofPublicHealth
LindsayThompsonCareyBusinessSchool
DanielWebsterBloombergSchoolofPublicHealth
Partnersindataandevidence-basedpolicy
Wealsospokewithpartnerswhoareexpertsinthefieldofevidence-basedpolicywhosharedtheirknowledgeandresourcesofcurrentandfuturethinkinginevidence-informedthinking
WendyBarrenoAssociateDirectorWilsonSheehanLabforEconomicOpportunitiesUniversityofNotreDame
TiffanyDavenportDirectorofDataPartnershipsBaltimoresPromise
TomiHiersDirectorBaltimoreCivicSiteAnnieECaseyFoundation
ChrisKingsleySeniorAssociateAnnieECaseyFoundation
KathrynOliverAssociateProfessorLondonSchoolofHygieneandTropicalMedicine
Thankyou
61
KristenWatkinsSeniorPolicyAssociateJ-PAL
GovExParticipants
GovExAnalystMiriamMcKinneyanalyzedtheinterviewtranscriptsandconductedtextanalysisSeniorImplementationAdvisorEmilyShawwillintegrateRoadmapactivitieswithadditionalservicedeliveryactivitiesatGovEx
Thankyou
62
ReferencesAbrahamKGampHaskinsR(2017)Thepromiseofevidence-basedpolicymakingReportofthecommissiononevidence-basedpolicymakingWashingtonDCCommissiononEvidence-basedPolicymakingBigmove2Extendingthegreen(nd)Retrievedfromhttpswwwcityoflewisvillecomdoing-businesslewisville-2025big-move-2-extending-the-greenBatesMAampGlennersterR(2017)ThegeneralizabilitypuzzleRetrievedfromhttpsssirorgarticlesentrythe_generalizability_puzzleBuildwithnotfor(2018January)RetrievedfromhttpsbuildwithnotforwordpresscomCairneyPampOliverK(2017)Evidence-basedpolicymakingisnotlikeevidence-basedmedicinesohowfarshouldyougotobridgethedividebetweenevidenceandpolicyHealthResearchPolicyandSystemsdoi101186s12961-017-0192-xCairneyPOliverKampWellsteadA(2016)TobridgethedividebetweenevidenceandpolicyReduceambiguityasmuchasuncertaintyPublicAdministrationReview76(3)399-402Retrievedfromhttpdxdoiorg101111puar12555Centreforpublicimpact(nd)RetrievedfromhttpswwwcentreforpublicimpactorgCernaL(2013)ThenatureofpolicychangeandimplementationAreviewofdifferenttheoreticalapproachesOrganisationforEconomicCo-OperationandDevelopmentRetrievedfromhttpswwwoecdorgeduceriTheNatureofPolicyChangeandImplementationpdfColoradoSprings(nd)Retrievedfromhttpscoloradospringsgovcommunicationspagewhat-expect-when-you-speakupContinuumofcare(coc)program(nd)RetrievedfromhttpswwwhudexchangeinfoprogramscocCouncilpresidentkicksoff$12millionyouthfund(nd)Retrievedfromhttpwwwbaltimorecitycouncilcomcontentcouncil-president-kicks-12-million-youth-fundDurlakJAampDuPreEP(2008)ImplementationmattersAreviewofresearchontheinfluenceofimplementationonprogramoutcomesandthefactorsaffectingimplementationAmericanJournalofCommunityPsychology41327-350Retrievedfromhttpswwwncbinlmnihgovpubmed18322790Fishbonediagram(ishikawa)(nd)Retrievedfromhttpasqorglearn-about-qualitycause-analysis-toolsoverviewfishbonehtmlFivewhysGettingtotherootofaproblemquickly(nd)RetrievedfromhttpswwwmindtoolscompagesarticlenewTMC_5Whtm
References
63
FligsteinNampMcAdamD(2011)TowardageneraltheoryofstrategicactionfieldsSociologicalTheory29(1)1-26RetrievedfromhttpsociologyberkeleyedusitesdefaultfilesfacultyfligsteinFligstein20McAdam20Soc20Theory20PaperpdHeinkelP(2018January12)CitymullspermitsforbikesharingcompaniesRetrievedfromhttpwwwdentonrccomnewsdenton20180112city-mulls-permits-bike-sharing-companiesHowisareamedianincomecalculated(2012October)Retrievedfromhttpswwwhudexchangeinfofaqs843how-is-area-median-income-calculatedLampkinLMampHatryHP(2003)KeystepsinoutcomemanagementUrbanInstituteRetrievedfromhttpswwwurbanorgresearchpublicationkey-steps-outcome-managementMarshDampSmithM(2000)UnderstandingpolicynetworksTowardsadialecticalapproachPoliticalStudies484-21Retrievedfromhttpjournalssagepubcomdoiabs1011111467-924800247McCannL(nd)RetrievedfromhttplaurenellencomMoultonSampSandfortJR(2016)ThestrategicactionfieldframeworkforpolicyimplementationresearchPolicyStudiesJournal45(1)Retrievedfromhttponlinelibrarywileycomstore101111psj12147assetpsj12147pdfv=1ampt=jdkhhcp4amps=e184c625e1855a6b1615ccb99ee27b1c0b3212d8PaynterB(2018January1)What100citiesarelearningfromeachotherbysharingtheirdataRetrievedfromhttpswwwfastcompanycom40518004what-100-cities-are-learning-from-each-other-by-sharing-their-dataPeakacademy(nd)Retrievedfromhttpswwwdenvergovorgcontentdenvergovenmayors-officeprograms-initiativespeak-performancepeak-academyhtmlPinnockHBarwickMCarpenterCREldridgeSGrandesGGriffithsCJRycroft-MaloneJMeissnerPMurrayEPatelASheikhAampTaylorSJC(2017)Standardsforreportingimplementationstudies(stari)ExplanationandelaborationdocumentBMJOpenRetrievedfromhttpbmjopenbmjcomcontentbmjopen74e013318fullpdfProctorESilmereHRaghavanRHovmandPAaronsGBungerAGriffeyRampHensleyM(2011)OutcomesforimplementationresearchConceptualdistinctionsmeasurementchallengesandresearchagendaAdministrationandPolicyinMentalHealth38(2)65-76doi101007s10488-010-0319-7TandonSD(2018)ProgressincommunityhealthpartnershipsResearcheducationandactionRetrievedfromhttpswwwpressjhuedujournalsprogress-community-health-partnerships-research-education-and-actionProvanKGampKenisP(2007)ModesofnetworkgovernanceStructuremanagementandeffectivenessJournalofPublicAdministrationResearchandTheory
References
64
18(2)229-252Retrievedfromhttpsacademicoupcomjpartarticle182229935895SabatierPA(1988)Anadvocacycoalitionframeworkofpolicychangeandtheroleofpolicy-orientedlearningthereinPolicySciences21(3)129-168Retrievedfromhttpwwwjstororgstable4532139SandstromAampCarlssonL(2008)TheperformanceofpolicynetworksTherelationbetweennetworkstructureandnetworkperformancePolicyStudiesJournal36497-524Retrievedfromhttponlinelibrarywileycomdoi101111j1541-0072200800281xabstractSchwartzMampWilsonE(nd)WhocanaffordtoliveinahomeAlookatdatafromthe2006americancommunitysurveyRetrievedfromhttpswwwcensusgovhousingcensuspublicationswho-can-affordpdfShearerJCAbelsonAKouyateBLavisJNampWaltG(2016)WhydopolicieschangeInstitutionsinterestsideasandnetworksinthreecasesofpolicyreformHealthPolicyandPlanning31(9)1200-1211Retrievedfromhttpsdoiorg101093heapolczw052ThelessonsofadministrativedataHigh-profilepolicy-relevantresearchpoweredbyadministrativedata(nd)Retrievedfromhttpswwwpovertyactionlaborgsitesdefaultfilesresources20161114-The-Lessons-of-Administrative-DatapdfToolsandotherneedsforqualitativeandquantitativemethods(2015February)AgencyforHealthcareResearchandQualityRetrievedfromhttpwwwahrqgovprofessionalsquality-patient-safetytalkingqualityassesstoolshtml
WeaverRCampBagchi-SenS(2013)SpatialanalysisofurbandeclineThegeographyofblightAppliedGeography4061-70
References
65
UniversityPartners
Name Title Affiliation Area(s)ofexpertise
Contactinformation
Examplesofworkingwithnon-academicdecisionmakers
WendyPBarreno
AssociateDirector
WilsonSheehanLabforEconomicOpportunitiesUniversityofNotreDame
LEOsfocusareasincludehealtheducationhousingchildwelfareeconomicopportunitycriminaljustice
wbarrenonduduorleondedu
MelissaDavey-RothwellPhD
AssociateScientist
JohnsHopkinsBloombergSchoolofPublicHealth
Health mdavey1jhuedu410-614-5854
BaltimoreCityHealthDepartmentLeadersMarylandDepartmentofHealthAddictionRecoveryServicesinDCExecutiveDirectorsofcommunityorganizationsacrosstheUSthatfocusonHIVSTIsubstanceabuseandimplementationscience
ElizabethStuart
AssociateDeanforEducationandProfessorDepartmentsofMentalHealthBiostatisticsandHealthPolicyand
JohnsHopkinsBloombergSchoolofPublicHealth
Educationsocialpolicystatisticalmethods
estuartjhuedu
Resources
66
Management
Design-ThinkingResources
EngagementGuideforCivicTech
Journeymap
PerformanceManagementResources
DenverPeakAcademy
Proxymeasuresguide
Talkingaboutperformancewithoutcomplicatingit(video)
SettingPerformanceTargetsguide
Datavisualizationguide
Performancemanagementgettingstartedguide
Exercises
Fishboneexercise
5Whys
AdministrativeData
TheLessonsofAdministrativeData(JPAL)
EvaluationResources
Quantitativeandqualitativemethods(USDepartmentofHealthandHumanServicesAgencyforHealthcareResearchandQuality)
KeyStepsinOutcomeManagement(UrbanInstitute)
RCT-YES
Sageresearchmethods
BestPracticesforMixedMethodsResearchintheHealthSciences
ACCORDChecklist
Pilotandimplementationplanning
Benchmarkingexamplesandproxymeasures
Seattleopendataplaybook
Resources
67
Datasciencescopingguide
Resources
68
AboutGovExGovExhelpsgovernmentsusedataandevidencetoimproveresidentsrsquolivesWesupportgovernmentsinbuildingtheirinternalcapacitytooperatesustainableanalyticspracticeswhichimprovecommunityoutcomesinfieldsfrompublicsafetytoeconomicdevelopmentGovExhasaproventrackrecordofworkinginconcertwithpartnerstoprovidetechnicalassistanceandtrainingatscale
SinceourlaunchinApril2015GovExhas
Trainedover2000governmentemployeesinpersonandonlineintopicsincludingdatamanagementperformancemanagementopendatacommunityengagementandanalytics
Providedtechnicalassistancetomorethan100localgovernmentsacrosstheglobeimprovingtheiruseofdatatoachieveefficienciesinservicedeliveryandbetteroutcomesforresidents
Assessedthecapacityofmorethan130governmentalorganizationstodeterminehowleadersstaffandexternalstakeholdersmanageandusedata
Localgovernmentsareincreasinglyonthefrontlinesofaddressinglargescaleproblemswithglobalimplicationssuchasclimatechangepublicsafetyandeconomicinequality
Whileevidence-basedinterventionstohelpsolvethoseproblemsareavailablelocalgovernmentstaffhavelimitedtimeandcapacitytofindandimplementresearch-basedsolutionsAsaresultcitiescontinuetoworkthewaytheyalwayshave--makingincrementalimprovementstoprogramsbutnotalwayscreatingchangeatasystem-level
ThroughresourcedevelopmentandcollaborationwithJHUfacultyGovExissupportingcityleadersinusingdataandevidencetomakedecisionsandcreatinglastingrelationshipswithresearchers
TheRoadmapwasdevelopedbyGovExaspartofitspartnershipwithBloombergPhilanthropiesWhatWorksCitiesinitiative
AboutGovEx
69
- Title page
- Introduction and purpose of the Roadmap
- Using the Roadmap
-
- Gather information about the problem
- Strategically engage key partners
- Review the evidence and collect details
- Decide on a policy to implement based on local factors
- Establish a framework to monitor progress
- Assess data accessibility quality and security
- Design implementation and evaluation
- Share your story and success
-
- Working with a range of evidence
- Thank you
- References
- Resources
- About GovEx
-
StrategicallyengagekeypartnersPolicytheoryresearchoftenfocusesonincreasingpolicymakersaccesstoevidenceasawaytoavoidtakingshortcutslikemakingdecisionsbasedonemotionImplementationsciencesuggeststhatincreasingaccesstoevidenceisnotenoughandthatpolicychangeshouldbeexaminedinthewidercontextofdebatecoalitionformingandpersuasionMeaningfulpolicychangerequiresldquoalong-termstrategytoformalliancesanddevelopknowledgeofpolicymakersandpolicymakingratherthanusingevidenceinpolicymakingonanadhocorone-offbasis
UnderstandingnetworksandsystemsisimportanttodevelopsustainablepolicychangeThissectionismeanttohelpreaderstakethestepstobuildengagementstrategiesforpartnersthatrepresentdifferentsystemsandnetworksEngagementshouldbeanongoingprocessofconnectingpeopletooneanotherandtoanissueEngagementshouldnotbeinnameonly-wehaveallexperiencedengagementactivitiesthatgivepeopleaspecificroletoplaythatleadstoaneatpredictableoutcomeAccordingtotheBuildWithmovementundervaluingandunderinvestinginengagementstrategiescanleadtoadisconnectofcertainindividualsorgroupsandmistrustofgovernmentWhendevelopingengagementactivitiesrememberthathowpeopleengageandwhethertheyengagehastodowithhowyoustructureopportunitiesforengagement
Conveneacoreadvisorygroup
AnadvisorygroupcomprisedofpeoplefrommultiplesectorscanbehelpfulasyouconsiderthelocalsocialenvironmentalandpoliticalcontextinwhichyouintendtointroducepolicychangeRevisityourstakeholdermapanddeterminewhowouldbebesttoserveonthisgroupTomakesureyouareconsideringtheproblemsandsolutionsthroughanequitylensincludepeoplewithlivedexperiencewhohaveexpertiseintheeffectivenessofsystemswhetherthosesystemsfunctionasintendedandhowtheymayimpacthumanbehaviorYoumayalsowishtoincludesomeonewithexperienceworkingwithdataandanalysis-thispersoncanhelpdeterminethetypeofdatathatmaybereadilyavailableandusefultocollectandcanalsohelpplanandshapeyouranalysistofigureoutwhichprogramsareeffectiveBesuretoinvitefront-lineworkersaswell-itscriticalthatwhateversolutionscomeoutofthisprocessaremeaningfultothosewhowillberesponsibleforimplementingthem
BringingpeopletogethercanbechallengingbutthereareafewwaystosetandmanageexpectationsFirstmakesurethatthegroupisawareoftheproblemstatementcurrentstateandfuturestate-thiscanhelpstarteveryoneoffonthesamepageNextspend
Strategicallyengagekeypartners
16
sometimegettingtoknoweachotherrsquosworktolearnabouteveryonesstrengthsThissharedunderstandingcanhelpasyouclarifyrolesandresponsibilitiesofeachpartnerinthegroup--thisisimportanttoavoidfinger-pointinginthefutureandtoensurethatnoneofthetasksslipthroughthecracksInsomecasestherearemanyactorswithdifferentresponsibilitiesasinthecaseofBaltimoreCitys38-memberYouthFundTaskForcewhichischairedbynonprofitandcorporatepartnersandincludesrepresentativesfromthenonprofitandfaithcommunitiesstudentsthebusinesssectorlocaluniversitiesCityCouncilandtheMayorsOffice
OncerolesandresponsibilitiesareclearyoumayconsiderdevelopingacharterguidelinesortermsofreferencetosetexpectationsforthegroupBesuretodistinguishthisfromotherboardsandcommissionsthatyourcitymayhaveNextyoumaywishtoestablisharegularongoingmeetingscheduleBesuretosendagendasandmeetingmaterialsaheadoftimeandclarifytheobjectiveofeachmeeting
InSeattletheCitydevelopeditsHousingAffordabilityandLivabilityAgenda(HALA)torespondtorisinghousingcostsandhomelessnessThisisatrendinotherwestcoastcitiesaswellwhichmaybeexplainedbyaninfluxofnewresidentswhoareattractedbytheemergenceoftechjobstheattractionoftheoutdoorlifestyleortherecentpassageofprogressivepolicieslikelegalizedgaymarriagea$15minimumwageandlegalizationofmarijuanaSeattlebroughttogetheramulti-stakeholderadvisorygroupcomprisedoflocalhousingexpertsrentersandhomeownersdevelopersandgovernmentofficialstoshareinformationabouttheirneedsanddeveloprecommendationsbasedontheseneedsTheinterestsandmotivationsofthisgroupwerevariedandittookabouttenmonthstofindcommongroundTheseriesofrecommendationsthattheadvisorygroupcreatedismeanttoserveresidentswithaspectrumofneeds
Buildaprocessforcommunityparticipation
Ofcoursenoteveryonecanbeincludedinanadvisorygroupandadvisorygroupmembersmaynotnecessarilyrepresentthevariedperspectivesoftheirpeers-forinstancenoteveryhomeownerhasthesamesetofneedsTogatherinputfromawiderswathofthepopulationitcanbehelpfultobuildaprocesstoinvitethepublicatlargetosharetheirideasCitiesdothisinavarietyofwaysfromonlineconversationsandsurveystoin-personmeetingsRememberthatbeingintentionalaboutgatheringtheviewpointsofpeopleismoreimportantthanusingthelatesttechnologyandtools
TheinsightgeneratedbycommunityparticipationexercisescanbeincludedinyourreviewofevidencewhichisdiscussedinasubsequentchapterInWestVirginiaastatethathasbeenhithardbytheopioidepidemicstatehealthofficialsdraftedanopioidresponseplanthatwasinformedbypublicengagementaswellaslocalandregionalsubjectmatter
Strategicallyengagekeypartners
17
expertsThepublicwasinvitedtopresentcommentsonideasfordecreasingoverdoseearlyinterventiontreatmentandrecoveryPublichealthandmedicalfacultyfromWestVirginiaUniversityMarshallUniversityandJohnsHopkinsUniversityprovidedresponsestothecommentsTheresultwasanOpioidResponsePlanthathealthcommissionerRahulGuptapresentedtoGovernorJimJustice
AsamplespectrumtoplanyourowncommunityparticipationprocesscreatedbyLaurenellenMcCannisbelow
SourceLaurenellenMcCann
Community-basedparticipatoryresearchisanotherwaytoworkwiththecommunitytoimproveoutcomesCBPRmeansthecommunityisinvolvedinallaspectsoftheresearchprocesswhichisofteniterativeInCBPRtraditionalresearchersandcommunitymembersshareknowledgeaswellasresourcesandcredit
CBPRInActionInSanFranciscoanenvironmentaljusticenonprofitteamedupwiththeCitysDepartmentofPublicHealthandanoutsideevaluatortounderstandtherootcausesofobesityandfoodinsecurityinlow-incomeneighborhoodsandidentifysolutionsThroughresearchmethodssuchasinterviewingresidentsandstoreownersdiagrammingstoreshelvesGISmappinganeconomicanalysisandreviewofsimilarpoliciesinnearbycitiestheteamobservedthatthestoresthatresidentshasveaccesstowerestockedwithlesshealthyfoodsandthatresidentswantedmorehealthyfoodandlessalcoholandtobaccoThisprojectresultedinlocalandstatelegislationtoencouragestorestostockfreshproduce
Strategicallyengagekeypartners
18
Developrelationshipswithsubjectmatterexperts
OnewaytokeepsubjectmatterexpertsengagedistoinvolvetheminthecommunityparticipationprocessasdescribedaboveIntheWestVirginiaexamplesubjectmatterexpertsrespondedtoandsynthesizedthepubliccommentsOtheropportunitiesaredescribedbelow
AdvisoryAcitydepartmentheadMayororstaffmembermaydevelopaone-on-oneadvisoryrelationshipwithuniversityfacultymemberoracollaborativebetweentheCityandUniversitytoaddressaparticularproblemAgreatplacetostartistolookforrelevantpracticecenterswithinyourlocaluniversityandreachingouttosubjectmatterexpertsInadditiontotheirexpertiseontherelevantpolicytopicheorshemayalsobeabletoidentifyotherexpertsandevenprovidesupportonhiringtherighttalenttoleadcertainprogramsAnadvisoryrelationshipwithauniversitysubjectmatterexpertcanalsohelpwhencitiesareconsideringbringinginnewprogramsTheseexpertsareoftenfamiliarwithprogramsthatarebasedonnationalmodelsandcanprovideinsightintoaprogramsoriginandlineageeffectivenessinotherplacesandthepotentialforimplementationinyourcityAcademicscanalsohelpidentifyindicatorsbasedongovernmentindicatorsorliteratureFinallyrelationshipsbetweenpolicymakersandresearchersprovidethemostbroadlyeffectiveroutetopullingresearchintopolicy
ResearchandevaluationAcademicexpertsmaysupportcitiesinevaluation-indeedevaluatingcityinitiativesisatraditionalmodelofcityuniversityrelationshipsSharingtheinformationyoucollectedintheGatherInformationphasecanhelpresearchersunderstandyourworkandgoalswhichinturnwillstrengthenthesupportthattheycanprovideDiscussingdetailsaboutyourinterventionandevaluationplanatthebeginningoftheprocessmeansthataresearchpartnercanhelpwiththeevaluationdesignanddatacollectionForinstancetheWilsonSheehanLabforEconomicOpportunitieswalkscitiesthroughconsideringthefollowingquestionsbeforepartneringonanevaluation
1 Isthereaclearpolicyprogrammodelwithobjectivescreeningcriteria
2 Isitpossibletoevaluatewhathappensbothwiththepolicyprogramandintheabsenceofthepolicyprogram
3 Arethereoneormoremeasurableoutcomesofinterest
4 Arethereenoughpeopleforrigorousquantitativeanalysis
5 Whatdatawillbenecessaryforthestudyandwhataretheprocessesforaccessingthesedatasource(s)
6 Isthereacompellingreasontoevaluate
Strategicallyengagekeypartners
19
InadditiontoassessingreadinesstoconductanevaluationresearcherscanhelpyouhoneinonthepurposeoftheevaluationTheWorldBankdescribesseveralpotentialusesofevaluationwhicharedefinedhereInbriefevaluationcanhelpdecisionmakers
makeresourceallocationdecisionsrethinkrootcausesofproblemsidentifynewproblemsdecideonabestalternativedeterminewhetherprogramsorpoliciesarebeingimplementedwithfidelitydeterminewhetherprogramsorpoliciesarehavingthedesiredimpact
Researcherscanalsoprovidesupportinsharingtheresultswiththescientificcommunitythroughpapersandconferenceswhichisvaluableincreatingapeernetworkofevidence-basedcitypractitioners
TrainingandtechnicalassistanceUniversitypartnersespeciallythoseatpracticecenterscanbehelpfulindevelopingorupdatingtrainingcurriculumtrainingthetraineranddeliveringtrainingtocitystaff
Strategicallyengagekeypartners
20
ReviewtheevidenceandcollectdetailsOnceyouhaveathoroughunderstandingoftheproblemdataandactorsyouwillneedtoconductyourownresearchonavailableevidenceandlearnaboutthelandscapeofpolicyoptionsYourgoalwillbetodevelopanaccessiblelistofinformationaboutpoliciesthathavehelpedsolvesimilarproblemsinotherplacesThischapterwillguideyouthroughthestepsofpreparingyourreviewersunderstandinglevelsofevidenceandcollectingthedetailsonevidencethatyouwillneedfordiscussionwithdecisionmakersinyourcity
Selectandprepareyourreviewers
AssignapersonorteamtoreviewrelevantinterventionsthathavedemonstratedsuccessinotherjurisdictionsTheirgoalwillbetoproduceareviewofalloftherelevantprogramspoliciesorinitiativesthatyourteamcouldconsiderimplementingTheywillbeconductingaliteraturereviewsupplementedbyinterviewswithexpertsandpeerswhohaveinformationaboutpotentialprograms
SincethepeopleyouassigntothistaskareprovidingthebaselineofpotentialoptionstoimplementitisimportantthatyouchoosepeopleyouareconfidentwillbethoughtfulandthoroughintheirworkItisalsocriticalthattheyareawareoftheimportanceofsourcecredibilityEnsureyourreviewershavefamiliarizedthemselveswithrulesforcriticallyanalyzinginformationsourcesbeforetheygetstartedIftheyareincludingnewsarticlesaspartoftheirsearchtheyshouldalsobeawareofmethodsforevaluatingnewscredibility
AskyourreviewerstodevelopaplanandtimelinetocompletethereviewThisstartswithsolidifyingagoverningresearchquestionOnewaytoknowyouarecreatingadetailedresearchquestionistousetheacronymPICOremindingyoutoincludereferencetothepopulationintervention(ifyouhaveoneinmind)comparisonandoutcomeelementsinyourresearchquestionForexampleifyouknowthatyouhaveapopulationofteenagerswhoarehomelessyouhavelearnedfromyourstakeholdersthatmanyoftheseteenagersidentifyasLGBTandyouhaveheardsomethingabouttheapproachofcreatingdrop-incentersfordeliveringsupportyoucouldask
ForLGBTyouthfacinghomelessness(population)howeffectivearedrop-incenters(intervention)relativetostandardsocialservices(comparison)atsettlingtheminpermanenthousing(outcome)
Ifyoudonothaveaninterventioninmindyoucanleaveouttheinterventionandcomparisonandsimplyask
Reviewtheevidenceandcollectdetails
21
HowcanwehelpLGBTyouthfacinghomelessness(population)obtainpermanenthousing(outcome)
OnceyourreviewershavetheirresearchquestionsusethequestionstocreatealistofkeywordstheycanuseintheirliteraturesearchThiswillhelpkeeptheircollectionofarticlesreasonablyfocusedHavethemdescribewhattheirsearchprocesswillbeincludinglistingthedatabasestheywillreviewandtheirplanforstoringtheinformationtheywillcollect(Herearesomeadditionalsuggestionsforstructuringareviewplan)
WhatyourreviewerswillbedoingisalightweightversionofarapidreviewTotheextentthattheywouldliketodevelopamethodologyforspecificsearchprotocolsincludingcriticalappraisalprocessestheycancertainlygoaheadanddoso
Usedifferentlevelsofevidenceforthemostcompletepicture
AsdiscussedintheWorkingwitharangeofevidencechaptertheNestaStandardsofEvidenceframework--reproducedbelow--ishelpfultounderstandthelevelofconfidencethataprogramorinterventionishavingapositiveimpactThisapproachletsyouputinterventionsthatyouhaveheardaboutlocallybutwhichhavenotbeenstudiedtogetherwiththosewithademonstratedcausaleffectThisishelpfulsinceanumberofpoliciesprogramsandpracticesthatyouwillwanttoconsidermaynotyethavemuchofaresearchrecord
Reviewtheevidenceandcollectdetails
22
NearlyanypieceofinformationaboutaninterventioncanbeclassifiedwithinthestandardsofevidenceForexampleifsomeonehasacrediblehypothesisabouthowaninterventioncouldproducebenefitthistypeofevidencewouldbeclassifiedasLevel1ThistypeofevidencecouldincludeinterventionsinothercitiesthatyoureadaboutinthenewsorhearaboutataconferenceForinstancewhenUSDAallowedschoolsandschooldistrictstoprovidefreebreakfastandlunchtoallstudentsthroughtheCommunityEligibilityProvision(CEP)manydistrictsimplementedthisoptionevenbeforetherewasevidenceofeffectivenessThiswouldfitintoLevel1sinceitbuildsonthelogicthatprovidingaccesstomealswilldecreasefoodinsecurity
YoumayalsofindpiecesofinformationthatqualifyasLevel2evidenceForinstanceasmoreschooldistrictsimplementedCEPprincipalstestifiedaboutpositivechangestheyobservedinstudentssuchasamorepositivedemeanorthatcanleadtobetteracademicperformanceDataalsoshowedthatthestudentsdescribedwerereceivingmoremealsthanbeforetheprogramsimplementationHoweveritwastoosoontobuildevidencetoshowacausalrelationshipbetweentheincreaseinfreemealsandincreasedacademicperformance
ldquoLower-levelrdquoevidencecanconsistofnuancedbutvaluableelementsthatrelatetoaninterventionrsquospotentialfitwithacommunityQualitativeinformationsuchastestimonyfrompeoplewhohaveexperiencedaninterventionandthosewhohavefacilitatedorimplementeditwillalwaysbeusefultoconsiderandprovidesdetailedlocalcontext
Experimentalstudies--describedstartingatLevel3withcontrolledexperimentsandgoingthroughLevel5withfullyfleshedoutprotocolsandprocessesforreplication--stilloccupythehigherlevelsofthisdescriptionofevidenceThisprioritizationshowsthevalueofrigorousstudyforevaluatingtheimpactofaparticularpolicyorprogramratherthanotherfactorsthatcouldbeinfluencingoutcomesofinterestRememberthatmultipletypesofevidencecancomplementrigorousimpactevaluationsandgivecityleadersadditionalinsightsandbroaderperspective
Accessinformation
Encourageyourreviewerstobegintheirsearchforpromisingpracticesingovernmentandnon-profitclearinghousesandinacademicpolicyjournalsWorkpublishedinthesesourceshasalreadybeensubjectedtoahighlevelofprofessionalscrutinyandgenerallycontainsagreatdealofdetailwhichmeansthatyourreviewersaremorelikelytobeabletopullouttheinformationyouneedtomakeagooddecision
Clearinghouses
Reviewtheevidenceandcollectdetails
23
Herearesomeoftheclearinghouses(collectionsofprogramevaluations)forevidence-basedpoliciesandprograms
ResultsFirstClearinghouseDatabaseThePewCharitableTrustsCollectstogethereightotherclearinghousestoprovideinformationoninterventionsinthefollowingcategoriesadultcriminaljusticeeducationchildwelfarehealthjuvenilejusticementalhealthsubstanceabuseandyouthdevelopmentIndividualclearinghousesincludedinResultsFirstare
BlueprintsforHealthyYouthDevelopment-foryouthdevelopmentandviolenceprevention
CaliforniaEvidence-BasedClearinghouse-forchildwelfareinterventions
TheLauraandJohnArnoldFoundationrsquosSocialProgramsThatWork-forsocialservicesprograms
TheUSDepartmentofEducationrsquosWhatWorksClearinghouse-foreducationinterventions
TheUSDepartmentofHealthandHumanServicesrsquoResearch-TestedInterventionPrograms(RTIPs)-forcancercontrolinterventions
TheUSDepartmentofHealthandHumanServicesrsquoTeenPregnancyPreventionEvidenceReview(TPPEvidenceReview)-forsexualriskbehaviorandsexualhealthoutcomes
TheUSDepartmentofJusticersquosCrimeSolutionsgov-forcriminaljusticejuvenilejusticeandcrimevictimservices
TheUSSubstanceAbuseandMentalHealthServicesAdministrationrsquosNationalRegistryofEvidence-basedProgramsandPractices(NREPP)-substanceuseandmentalhealthinterventions
TheUniversityofWisconsinPopulationHealthInstituteandRobertWoodJohnsonFoundationrsquosCountyHealthRankingsandRoadmapsWhatWorksforHealth(WhatWorksforHealth)-healthbehaviorsclinicalcaresocialandeconomicfactorsandthephysicalenvironment
UKWhatWorksCentresTheBritishgovernmenthascharteredanetworkofsubject-specificorganizationswhichcollectsynthesizeandreportoutevidenceontheeffectivenessofpolicyprogramsandpractices
NationalInstituteforHealthandCareExcellence(NICE)-healthandsocialcare
Reviewtheevidenceandcollectdetails
24
SuttonTrustEducationalEndowmentFoundation-educationalachievement
CollegeofPolicingWhatWorksCentreforCrimeReduction-crimereduction
EarlyInterventionFoundation-earlyintervention
WhatWorksCentreforLocalEconomicGrowth-localeconomicgrowth
CentreforAgeingBetter-qualityoflifeforolderpeople
WhatWorksCentreforWellbeing-wellbeing
WilsonSheehanLabforEconomicOpportunities(LEO)UniversityofNotreDameProvideslinkstotheirevaluationsofinterventionsincriminaljusticeself-sufficiencyeducationhealthandhousingandhomelessness
ClearinghouseforLaborEvaluationandResearchUSDepartmentofLaborProvidesresearchonworkforceandlabor-relatedpoliciesandprograms
J-PALEvaluationsAbdulLatifJameelPovertyActionLabStudiesonpoverty-reducinginterventionsfromaroundtheworld
CampbellSystematicReviewsCampbellCollaborationSystematicreviewofpolicyinterventionsbyscholarlycoordinatinggroupsinsevenissueareascrimeandjusticeeducationinternationaldevelopmentmethodssocialwelfareknowledgetranslationandimplementationanddisability
Benefit-CostResultsWashingtonStateInstituteforPublicPolicyAWashingtonState-charteredprogramtoestablishthelikelihoodofrelativecostsandbenefitsforprogramsacrossthefollowingareasjuvenilejusticeadultcriminaljusticechildwelfarepre-Kto12educationchildrensmentalhealthhealthcaresubstanceusedisordersadultmentalhealthpublichealthandPreventionworkforcedevelopmentandhighereducation
EffectiveInterventionsCentersforDiseaseControlandPreventionRepositoryforevidence-informedHIVpreventioncareinterventions(withstepsonhowtoimplement)
Journalsandindexes
OnceyouhavereviewedrelevantinformationaccessiblethroughclearinghousesitisworthtakingalookatindividualstudiespublishedasscholarlyarticlesYourreviewerswilldothisbysearchingarticleindexesandacademicjournalsUnfortunatelyanumberofusefulacademicjournalsandindexingplatformsmaynotbegenerallyaccessibletoyourreviewersifyoudonothaveasubscriptiontothemToaddressthisconsiderpartneringwithalocal
Reviewtheevidenceandcollectdetails
25
collegeoruniversitytoprovideyouwithalistofarticlesfromthefollowingsources(Thisisalsoanexcellentopportunitytoseekresearchsupportfromanyacademicstakeholdersyouidentifiedpreviously)
Indexes
GoogleScholarincludesmostpeer-reviewedonlineacademicjournalsandbooksconferencepapersthesesanddissertationspreprintsabstractstechnicalreportsandotherscholarlyliteratureincludingcourtopinionsandpatentsItallowsyoutospecifyapublicationrangeinyoursearchandprovideslinkstolikelysourcesforalllistedarticles
PolicyFileIndexisauniqueresourceforUSpublicpolicyresearchinthatitgrantsusersaccesstotimelyupdatedinformationfromover350publicpolicythinktanksnongovernmentalorganizationsresearchinstitutesuniversitycentersadvocacygroupsandotherentities
PolicyArchiveworkstocollectanddisseminatesummariesandfulltextsvideosreportsbriefsandmultimediamaterialofthinktankuniversitygovernmentandfoundation-fundedpolicyresearch
Anothersourceforhigh-qualityresearchisthereportsoftheCongressionalResearchServiceYoucanfindtheirreportsaggregatedhere
Specificjournals
Oftenthemostusefulacademicjournalsforunearthingpolicyorprogramreviewswillbeineachspecificsubjectarea--thatisineducationjournalsforeducationpoliciesYoucanuseageneralindexlikeGoogleScholartobegintolearnwhichjournalsaremostoftenusefulforyoursearchHoweveryoucanalsosearchthefollowingtwogeneralistpolicypublicationsforissuesandarticlesrelatedtoyourtopic
PublicAdministrationReviewJournalofPolicyAnalysisandManagement
Newssourcessubjectmatterexpertsandinterviews
AlthoughitisvaluabletoknowwhathasbeenexploredthroughascholarlyresearchframeworkitisalsousefultolearnwhatkindofinterventionsarebeingtriedinthefieldrightnowandhowpeoplefeelaboutthesetrialsSearchsubscriptionnewsaggregatorslikeLexisNexisorfreenewsaggregatorslikeGoogleNewsorYahooNewsforyourissueareaplusthewordpolicyorprogramtoseewhatkindsofinterventionsarebeingcoveredinthenews
Reviewtheevidenceandcollectdetails
26
DonotforgettocontacttheadvisorygroupmemberswhowerechosenfortheirsubjectmatterexpertiseInadditiontoprovidinggoodmaterialthroughinterviewsthemselvestheymayhaveideasaboutothersubjectmatterexpertsyoushouldcontacttoaskaboutpotentialinterventions
Peercitiesandnetworks
InadditiontoreadingabouttheworkofothercitiesitisidealifyouareabletospeakwithpeoplewhoareinvolvedwithcurrentimplementationsofinterestinginterventionsInyourcityyoulikelyhaverelationshipswithyourneighborsbutitcanbehelpfultolookoutsideofyourregionforsolutionsaswellTofindcounterpartsinothercitieswhoareworkingonaprogramyouwouldliketolearnmoreaboutyoucancontactstaffinintermediaryorganizations(likeGovExoryourprofessionalassociations)whomightbeinapositiontoputyouintouchimmediatelyorwhomightseeanothergoodrouteforsettingupaconversationHereisalistofnetworksthatmaybehelpful
InternationalCityCountyManagementAssociation(ICMA)NationalAssociationofCounties(NACO)NationalLeagueofCitiesUSConferenceofMayorsWhatWorksCities
Collectthedetails
Onceyourreviewershavefoundevidenceonpossiblepoliciesorprogramstoimplementtheyshouldcapturekeydetailsthatwillhelpdecisionmakersunderstandthepotentialeffectivenessandapplicabilityofeachpotentialintervention
Foreachsourcethathasprovidedinformationyouwanttoconveyusethetemplatebelowtocaptureasmanyofthefollowingdetailsaspossible(Youdonotneedtocompletethisforsourcesthatprovideduplicativeinformationorareotherwisenotthatuseful)
Programorpolicyname (egldquoHousingFirstrdquoorldquoblightedpropertiesregistryrdquo)
Sourcetype (egjournalarticleinterviewsubjectmatterexpertstudyassociationcommunityparticipationprocess)
Sourcedetails (egarticletitleandpublicationnameandtitleofinterviewee)
Evaluationmethodology
(egrandomizedevaluationcomparativeevaluationoftwodifferentgroupsbeforeafteranalysis)
Evidencelevel (egLevel12345)
Programorpolicy
Reviewtheevidenceandcollectdetails
27
description
Programorpolicygoal (1sentencecapturingtheprincipalaimoftheintervention)
Existingimplementations
(WherehastheinterventionbeentriedListasmanysitesasyouareawareof)
Characteristicsofimplementationsites
(ForeachimplementationsiteofferbasicgeographicIDmdashcityandstateThenyouwillwanttocollectthedetailswhicharerelevanttoyourdecisionmakersrsquoconsiderationsaboutwhatmakesalocationldquosimilartordquoorldquodifferentfromrdquoyoursituationYouwillatleastwanttoincludeforeachimplementationsitePopulationnumberPopulationdensity(ruralurbansuburban)Socio-economiccharacteristicsBeforestartingaskyourdecisionmakersaboutanyparticularcharacteristicsofprogramimplementationsitestheywouldwanttoknowbeforeseekingtoreplicate)
Characteristicsofimplementationpartners
(Anyavailabledetailsaboutorganizationsinstitutionsorpeoplewhowereinvolvedwithimplementingtheintervention-egindependentchildcarecenterpublicelementaryschoolcitypublichealthnurses)
Characteristicsoftargetpopulationeligibleprogramparticipants
(Anyavailabledetailsaboutpeoplewhoarethebeneficiariesoftheprogramorotherwiseincludedoraffectedbytheintervention-egnumberofprogramparticipantscategoriesofpropertyownersaffected)
Policyorprogramduration
(HowlonghastheinterventionbeeninplaceineachimplementationsiteHasitendedandifsowhy)
Whatbenefitscamefromeachimplementation
(Anyobservedpositiveeffectsoftheinterventionincludingeffectsize)
Statisticalestimatesofinterventioneffectiveness
(Anyavailablestatisticalestimatesoftheinterventionrsquoseffects)
Whatdrawbackswereobserved
(Anyavailabledetailsaboutunintendedorunavoidablenegativeeffectsoftheintervention)
Programorpolicycosts
(Anyavailabledetailsaboutprogramcostsperimplementation)
Reviewtheevidenceandcollectdetails
28
Reviewtheevidenceandcollectdetails
29
DecideonapolicytoimplementbasedonlocalfactorsTogetthecriticalbuy-infromleadershiptoimplementanewpolicyorprogramtheprojectleadshouldscheduleameetingwithdecisionmakerstopresenttheevidencesurroundingthepolicyoptionscapturedinthetemplateDependingonyourinsightintoyourcityleadershipswillingnesstomakechangeyoumaywishtosharetheresearchyouhaveconductedontherootcausesoftheproblemandsomeofyouranalysisontheimpactofcurrenteffortstodate
Considertheapplicabilityofanapproach
AsdiscussedinthepreviouschapterpolicymakersoftenlookforevidenceofeffectiveprogramsinothercitiesbutoftenthequestionarisesofhowgeneralizableaprogramisfromonecitytoanotherTheAbdulLatifJameelPovertyActionLab(J-PAL)suggeststhatinreviewingevidencefromanotherlocationitishelpfultoidentifythespecificmechanismthatwasused(deliveryofprogramorserviceinformationsharingetc)identifythelocalconditionsthatwereinplaceobservethebehaviorchangethatresultedfromthemechanismandfinallytounderstandlocalcontextandconditions(legalorcostconstraintsculturalnormsetc)Thissetofpresentationelementscanbedrawnfromthelistofdetailscollectedintheresearchtemplate
EducationpolicymakersinChicagoILusedfindingsfromprogramsinKenyaandIndiainwhichchildrenwhohadfallenbehindtheirpeersweregiveneducationalsupporttargetedtotheirleveloflearningThemechanismusedwastargetedteachingandthereweresimilarconditionspresentinallthreecountriesAlthoughsomestudentshadfallenbehindinthecurriculumteacherswereincentivizedtoteachgrade-levelmaterialsConsiderationsofhowthelocalcontextinChicagomightaffectimplementationofatargetedteachingprogramincludedunderstandinghowtheneighborhoodsthatthestudentswerefrommayhaveaffectedtheireducationaloutcomesandthehighcostofteachinginsmallgroupsInChicagoMatchEducationprovidedtwo-on-onetutoringwhichmatchedstudentswithwell-educatedindividualswhohavenothadformalteachertrainingforalower-costmodelTheresultswerepositiveStudentswhoweretutoredperformedbetterinmathandweremorelikelytopasshighschoolclasses
ThegeneralizabilityframeworkproposedbyJ-PALisbelowUsethisasatemplateasyouconsiderpotentialpolicysolutions
Decideonapolicytoimplementbasedonlocalfactors
30
SourceTheGeneralizabilityPuzzle
OnceyouhavenarrowedthepolicyoptionsdiscussthequestionsbelowforeachoptionsyouareconsideringThisstepiscriticalasunderstandinglocalcontextissoimportantinsuccessfulpolicyimplementation
CommunityContextQuestionnaire
Localcontext
DoesthisoptionmakesenseinourcommunityIsthisaonesizefitsallapproachIsanyparticulargroupabsentfromthisapproachIfsoisthatok
Youranswers
Effectiveness
WhatwouldweexpecttoobserveifthepilotisaseffectivehereasinCityX
Youranswers
Timing
DoesthetimingmakesenseIsthistherighttimetoimplementthispolicychange
Decideonapolicytoimplementbasedonlocalfactors
31
WherearethepotentialroadblocksthatcouldaffectthescheduleIsthereanywaytomitigatethisHowlongwouldittaketoseeanimpact
Youranswers
Implementation
DowehavethestaffcapacitytoimplementthisArethereadditionalpartnersthatyoucanbringtothetabletosupportimplementationDoesthismakesensegivenourformofgovernment
Youranswers
Authority
DoesourcityhavetheauthoritytoenactthisIfsowhatistheappropriatemechanismAretherelegalbarriersArethereanyexistingpreemptionissuesfromthestateorfederalgovernmentCanthosebeaddressedthroughlegislationadvocacyIstherepoliticalsupportoroppositiontothis
Youranswers
Cost
HowmuchwillthiscostannuallyHowwillwepayforitCanthepolicybeimplementedwithexistingfunding
IfsoisthisfundingreliableHowwouldthispolicyoptionaffectourcitysbudgetintheshortandlongtermWhatotherfundingoptionscanweconsiderIsthereanalternativethatwouldcostlessandgiveussimilarresultsWillanyconstituencybeadverselyaffectediffundingforprogramsthatimpactthemisshiftedtoanotherprogrampolicyorinitiative
Youranswers
Decideonapolicytoimplementbasedonlocalfactors
32
EstablishaframeworktomonitorprogressNowthatyouhavereviewedtheevidenceconsideredlocalcontextandchosenyourpolicyitistimetodeterminehowyourcitywillmonitorprogressofthisnewapproachCitystaffandleadersneedtoestablishtheinfrastructureforongoingcontinuousdecisionmakingbasedonwhetherthepolicychangeisleadingtothedesiredoutcomesforresidentsandwhetherthechangeismeaningfulandrationaltofrontlineworkersSettingarbitrarygoalsatthelocallevelthatconflictwithgoalsatotherlevelsisconfusingtotheindividualsresponsibleforachievingthegoalsGoalsthataresettoohightoachievecanbedemoralizingtostaff
ThissectionincludesstepstosetupavenueinwhichCitystaffandleadershiproutinelyvisitkeyindicatorstodeterminewhatisorisnotworkingandtomakedecisionsinreal-timeFormorereadingonperformancemanagementpleasevisittheGovExPerformanceManagementGettingStartedGuide
InperformancemanagementtermssuchasindicatormeasureandmetricmaybeusedinterchangeablyInthisguidewewilluseindicatorInyourworkthemostimportantthingistobeconsistentinwhichevertermyouuseThisvideomaybehelpfulinunderstandingthisconceptandcommunicatingittoothers
Establishthefundamentalsandgatheryourteam
FirstdeterminewhetherthereisanestablishedvenueorregularmeetingscheduleduringwhichyoucandiscussprogressIfyourcityhasanexistingperformancemanagementstructureonarelatedtopicaddingadiscussionoftheprogressofpolicychangetoyouragendacanbeagreatideaIfthiswillbeastandalonemeetingdecidehowfrequentlytheprojectteamanddecisionmakerswillmeetandputthosemeetingsonthecalendarfortheentirelengthoftheproject
DeterminewhowillstaffandattendroutinemeetingsonCityprogressThesemeetingsmayincludetheMayorandorCityManagerdepartmentheadstaffwhoareimplementingthepolicychangeandanalysts
Identifyindicators
ItiscriticalthatindicatorsbeanimportantcityandcommunitypriorityaswellassomethingtheCityplaysamajorroleinaddressingTheremaybeinstancesinwhichtheCitywillneedtoreachouttoorcollaboratewithexternalpartnerstogetamorecompletepictureofprogressontheoverallpolicychange
Establishaframeworktomonitorprogress
33
Anoutcomeindicatordescribesthedesiredeffectofthetargetedpopulation
ExampleBellevuesfamilyfriendlyneighborhoodsindicatormeasuresthepercentofresidentswhobelievetheyliveinaneighborhoodthatsupportsfamiliesparticularlythosewithchildren
WithyourkeypartnersrevisityourproblemstatementcurrentstateandfuturestateexerciseandbegintoidentifytheoutcomeindicatorsthatwillillustratesuccessFornowchooseasingletop-levelkeyperformanceindicator(KPI)ItiscriticalthattheKPI-thepieceyouwillmeasure-beapriorityforthecommunityandsomethingtheCityplaysamajorroleinaddressing
Outputindicatorsmeasuretheprocessofprogramorservicedelivery
ExampleSanDiegoMayorKevinFaulconersgoaltorepair1000milesofstreetby2020
ReviewingtheadministrativedatasetinventorybegintoparedowntherelevantprogramsanddatasetsthatsupportyourKPIWhatarethemeasuresofeffectivenessforeachprogramTheseareyouroutputindicators
Thediagrambelowillustratestherelationshipbetweenoutputandoutcomeindicators
ThinkaboutimplementationindicatorsasmeasuringstepsintheprogramsrolloutthatarenecessarytoobtaindesiredchangesImplementationoutcomescanfocusontheinterventionstaffdeliveringtheinterventionanorganizationorsystemandcanaddressfeasibilityadaptabilitysustainabilityandcostTwokeyquestionstoconsiderwhendevelopingimplementationindicatorsare
1 Istheinterventionacceptableandappropriatetothecommunityand
2 IsitfeasibletoimplementinterventionXatthistimewiththeseresources
Establishaframeworktomonitorprogress
34
Amoreextensivelistwithdefinitionsofpotentialimplementationoutcomesislinkedhere
UsethetablebelowtochoosethedesiredimplementationoutcomesforyourpilotandtoplanyourmeasurementtechniqueThisshouldincludelevelofanalysis(individualproviderorconsumerorganizationorsetting)yourmeasurementtechnique(surveysobservationinterviewfocusgroupsadministrativedataanalysis)andthephasewhenyouwillcollectthisinformation(earlymidlate)Theremaybemorethanonelevelofanalysismeasurementtechniqueandphaseofdatacollection
Implementationoutcome
Definitionofoutcome
Levelofanalysisandmeasurement
Yourmeasurementtechnique
Phaseofpilotwhenyouwillcollectdata
Usethespacebelowtocaptureyourindicatorwishlist
Indicatorwishlist
Outcomes_____________________________________________________________________________
Outputs________________________________________________________________________________
Implementationoutcomes______________________________________________________________
Setapublichigh-levelgoal
Thehigh-levelgoaliscriticalinestablishingdirectiontogalvanizethepeoplewhowillbedoingtheworkWhendiscussingthegoalusetheMayorsbullypulpitLookforkeymilestonesinCitygovernmenttosharethegoalstatementsuchasthebeginningofaMayoraladministrationaStateoftheCityspeechorotherpublicremarks
GoalsshouldbeldquoSMARTrdquo
Specificaspossible(innamingthepopulationwhowillbebetterofftheactionthecitywilltake)
Measurable
Achievable
Establishaframeworktomonitorprogress
35
Results-focused
Time-bound
ThegoalstatementshouldincludeyouroutcomeindicatorlikeChattanoogasgoalldquoWewillendchronicveteranhomelessnessbyDecember2016rdquoAdiagramtohepyoubuildaSMARTgoalaroundyouroutcomeindicatorisbelow
Component Description Yourturn
Specific WhowillbebetteroffWhatactionthecitywilltake
Measurable Howwillyoumeasuresuccess
Achievable Isthegoalattainable
Results-focused Whatistheexpectedresult
Time-bound When
Setrealisticambitioustargetsandtimelinesforeachindicator
ToestablisharealisticandambitioustargetforyourgoalstatementusetheadministrativedatacollectedforyourchosenKPItoreviewactualCityperformanceinthemostrecentyearcomparedwiththethreeorfourmostrecentyearsworthofdataBegintocalculatesummarystatistics(egtotalminimumaveragemedianmaximum)toidentifytrendsCreatecumulativeandmonthlychartstoidentifytrendsandanomaliesBasedonyouranalysisofhistoricaldataignoretheanomaliesandassumetheproductivitytrendswillfollowthesameoverallpatternfrommonth-to-monthFinallyapplyanincreasedproductivityfactortotheperformanceeachmonthtoforecastfutureprogressBasedonthisanalysischooseatargetthatinspiresimprovementwhileaccountingforreality
Manycitiesthataremorematureintheirperformancemanagementpracticesrecognizetheimportanceoftakingacommunity-levelapproachtoimprovingoutcomesforresidentsThesecitiesarecreatinggoaltaxonomiesthatincludemeasuresrelatedtopublicsafetyhousingandevenrecreationCitiesdriveprogressonthesegoalsbyimplementingstrategiesthatengagemultipledepartmentsandcollaboratewithserviceprovidersinthecommunity
EventuallyyouwilldevelopgoalstatementsfortheoutputsinordertocapturetheefforttoachievetheoutcomecontainedinyourgoalstatementYoursetofindicatorsshouldbecomposedofshort-andinterim-termsupportivemeasureswithaccuratetargetsandrelevanttimelinesAnexamplegoaltaxonomyfromtheCityofLasVegassResultsVegasprogramisbelowUsethistaxonomyasamodelforyourownsuiteofindicatorsandgoalstatements
Establishaframeworktomonitorprogress
36
Yourturn
Thetemplatebelowismeanttohelpyoucraftagoalstatementforeachsupportingmeasure
Goalstatement
Departments
Outcome
Activities
KPI
Target
Timeframe
Conductanddiscussrelevantanalysis
ThrougharoutineseriesofmeetingscontinuallymonitorprogressofKPIssupportingindicatorsandanyotherdatacollectedsuchasqualitativedatatoidentifytrendsthatwillindicatewhetherthechosenpolicyishavingthedesiredimpactThesedatapointsshoulddrivethediscussionduringthesemeetingspointingtoopportunitiesforfurtherinquiryinto
Establishaframeworktomonitorprogress
37
whatisdrivingtheobservedtrendsAnalystsshouldbeencouragedandtrainedtoaskquestionsofthedepartmentsproducingthedataandconductsitevisitswhenpossibletodeepentheirunderstandingofthedataPresentationoftheanalysisshouldfocusononeideaatatimetokeeptheconversationfocusedonkeypointsandspecificopportunitiesforimprovementYoumayconsiderdevelopingapublicfacingdashboardtosharethisprogressmorebroadly
AgeneralcomparisonofthestatusquodatawithwhatwasobservedduringthepilotcanprovidedecisionmakerswithaquickunderstandingofthedifferencesbetweentheoldandnewapproachesalthoughthiswillnotshowcausalityOftenthesecomparisonswillincludeoperationalmetricssuchasdaysordollarssavedbutrememberthattheimpactonpeoplewillbemostimportantIfyouareworkingwithanevaluationpartnertheywillsupportthisstep
Usethespacebelowtosummarizetheresultsofyouranalysis
MeasureTargetvalue(seesectiononsettingtargetsabove)
Pilotvalue(whatdidyouobserveduringthepilot)
Historicalaverage(ifavailable)
Results(WhoisbetteroffIsanyonenegativelyimpacted1-2sentencesreassessingyourgoalstatement)
DocumentyourprocessandprogressincludingfeedbackfromtheparticipantsimplementationpartnersandtheprojectteamDidanythingunexpectedoccurduringthepilotArethereinternalandexternalfactorsthatmayaffectyouroutcomesThisiswheretheimplementationoutcomescanbecanhelpfulIftheinterventiondidnotgoasplannedconsiderwhetheritisbecausetheinterventionwasnotacceptableorfeasible
UsethetemplatebelowcreatedbyDenversPeakAcademytocapturethisinformationIfthe306090daytimelinedoesnotmakesenseforyourcityfeelfreetocustomizeit
CurrentState
FutureState
30Day
60Day
90day
Whatwentwell(+)
Whatdidnotgowellshouldbechanged(-)
IfaquestionordiscussionpointarisesthatcannotbeansweredinthemeetinganassignedstaffmembershouldcapturethequestionduringthemeetingAfterwardthefollowupitemsshouldbesenttorelevantpartnersandthenextmeetingshouldbeginwithadiscussionof
Establishaframeworktomonitorprogress
38
thoseitems
Takeaction
DiscussionsaboutwhetheryourindicatorsaremovingintherightdirectionandwhatmightbedrivingtrendsmayleadtoadiscoverythattherearenotenoughresourcesallocatedtothispolicyorissueTheremaybelimitedfundingforstaffprogrammingortechnologythatisdrivingthedatainthewrongdirectionIfCityleadershipisintheroomthisisagreattimetomakethecaseforadditionalfundingduringthebudgetseasonBycollectinganalyzinganddiscussingdataonthispolicyroutinelyyouarebuildingevidencetomakethecasefortherightleveloffundingtoreachtheestablishedtargets
Establishaframeworktomonitorprogress
39
Assessdataaccessibilityqualityandsecurity
Determineaccessibilityandqualityofdata
OnceyouhavecompiledyourwishlistofindicatorsitistimetodeterminetheaccessibilityandqualityofthedataTopreventcomplicationyoumaywishtoprioritizerelevantindicatorsforwhichyouarealreadycollectingdataAvoidthetemptationofadoptingatechnologysolutiontocollectnewdataatthispointsuchasaphotosurveytoolorotherappFornowyouaretryingtodeterminewhichdatasetsarerelevantaccessibleandhighqualityBesuretoworkwithyourkeyinternalpartnerswhileyouareidentifyingyourmeasuresDepartmentstaffwhoworkonprogramsarefamiliarwiththeindicatorsthatwillaccuratelysupporttheoutcomesanddatabaseadministratorswillbevaluablewhendiscussingaccessibilityandqualityofthedata
Abrieflistofconsiderationsfortheaccessibilityofyourdataisbelow
ArewecurrentlycollectingthisdatasetorwillwebecollectingitforthefirsttimeHowfarbackdoestherecordgo(timehorizon)WouldweneednewtechnologytocollectoranalyzethedataCanthedatabecollectedmanuallyorcanitbeautomatedHowfrequentlyisthedataproduced(continuouslyhourlydailyweeklymonthlyquarterlyyearlyetc)IsthedataavailablefortheentireCitypopulationorasubsetofthepopulationHowgranularisthedata(individuallevelCensusblockneighborhoodcitywideetc)DoesthedatacontainsensitiveinformationIsthereaproxymeasurethatwecanusefornowIsthedatamachinereadable
AsnotedbytheWilsonSheehanLabforEconomicOpportunities(LEO)attheUniversityofNotreDameleveragingadministrativedatacansignificantlyreducethecostsandtimelinesofrigorousevaluationCitiescanestablishaninternaldatasharingagreementacrossdepartmentsincludinglanguagethatenablesadministrativedatatobeusedforresearchpurposesCitiescanalsoidentifyexternaladministrativedata-holderssuchasacountyschooldistrictpolicedepartmenthospitaloranotherstatethatmayhaveoutcomedataimportantformeasuringprogramimpactCitiescanleverageexistingrelationshipsandorworkwithresearchinstitutionstodevelopstreamlineddatasharingprocessesandagreementswiththeseexternalentities
Assessdataaccessibilityqualityandsecurity
40
Evaluatethequalityofyourdata
QualityisasimportantasaccessibilityToevaluatedataqualityyoumaywishtocreatearubricliketheonebelow
Value ValueDescription
3 ExcellentDataQualityThedataiscompleteaccurateandupdatedregularly
2 AdequateDataQualityThedataissomewhatcompleteaccurateandupdatedfairlyregularly
1 PoorDataQualityThedataisnotcompleteaccurateorupdatedregularly
ThisisagreattimetobuildonthedatainventorythatyoustartedinthegatherinformationpageSuggestedadditionalfieldstoaddtoyourdatainventorytemplatearebelow
Program DatasetTitle
DatasetDescription DataSource
Departmentorexternalpartner
DoestheCityalreadycollectthisdata
Whatisthenameoftheprogram
Human-readablenameofthedataset
Shouldbeunderstandablebynon-technicalusers
Whatistheoriginalsystemapplicationfilethathousesit
Departmentorexternalpartnerthatmaintainsthedata
Isthisdataalreadycollected
Updatefrequency-ThefileformatforthedatasetTypicallycsvxlsxshapefiletxtetc
Format-Howoftenthedatasetisupdated
Timerange-Theperiodoftimecoveredinthisparticulardataset
Datause-Howisthisdatasetcommonlyused
Datausers-Whotypicallyusesthisdataset
Dataownerstitle-Titleofthedataowner
Dataownersposition-DataOwnersjobtitle
Dataownersemailaddress-DataOwnersemailaddressWhomanagesthedataandorisresponsibleforgrantingpermissiontoaccessthedataWhounderstandswhatthedatasetincludesandcananswerquestionsaboutit
Dataqualityconcerns-Describetheconcernsyoumayhaveaboutthequalityofthedataset
Assessdataaccessibilityqualityandsecurity
41
Division-DivisionwherethedataismaintainedIfmorethanonedivisionisresponsiblelisttheprimaryownerofthedataset
Sensitivedatacomments-DescribetheissuesthatwouldpreventthecityfromsharingthedatapubliclyCommonsensitivityconcernsincludeprivacyviolationssecurityissuesorhighcostorstaffdemandstopostthedata
Publishingstatus-Thistheplacetoputanycommentsregardingthecurrentstatusofthedataset
Publishedlink-Thisisthelinktothepublisheddatasetontheopendataportalorcitywebsite
Reviewdatasecurityandprivacyprotectionprotocols
OftenapolicychangealsoinvolvesachangeinthedatathatisbeingcollectedhowthatdataisbeinghandledwhohasaccesstoitandahostofotheradjustmentsAsyoutakestockofthedataassetsthatareavailableforyourinitiativeitisvitaltoalsothinkthroughwhatwillbedonetoprotectthatdataandtheprivacyofthepeopleitrepresentsAbigpartofgooddatasecurityinvolvesstrongdatagovernancepracticesthatdefineandidentifysensitivedataandconsciouslyworktobalancethevalueandriskofstoringsuchsensitivedataOnewayofevaluatingyourdatacomesfromcaselawrelatedtotheUSFreedomofInformationActwhichwehavetranslatedtothechecklistbelow
1 Woulddisclosureresultinasubstantialinvasionofprivacy2 Haveweconsideredtheextentorvalueofthepublicinterestandthepurposeorobject
oftheindividualsseekingdisclosure3 Istheinformationavailablefromothersources4 Wastheinformationgivenwithanexpectationofconfidentiality5 Isitpossibletomouldreliefsoastolimittheinvasionofindividualprivacy
TheseriskscouldbemitigatedusingavarietyofstrategiesfromaggregatingdatatoobfuscatingindividualstoredactingfieldsthatmayincludePersonallyIdentifiableInformation(PII)orre-evaluatingyourdatacollectionmethodstofindalternativewaysofgettingtheinformationyouneedThesestepsshouldbecodifiedandmanagedbyadatagovernancecommitteeifpossibleForthesensitivedatathatisdeemedessentialcreateaplanaroundpotentialdatabreachesinordertoquicklyandconfidentlyrespondtoemergencysituations
ProactivelypreparingforthesesituationsisoftenthebeststrategyfordatasecuritybutdonotforgetthetechnicalsideInvolveyourITadministratorsinconversationsaboutdatastorageandaccessensuringthattheywillbeprotectedbyenterprise-gradecybersecuritysystems
Assessdataaccessibilityqualityandsecurity
42
Assessdataaccessibilityqualityandsecurity
43
Designimplementationandevaluation
Designtheimplementation
NowthatyouhaveyouraccountabilityplanandkeyindicatorsitistimetogetspecificaboutyourplanforimplementationThetemplatebelowwillhelpyouthinkthroughthespecificmechanismsofimplementingpolicychange
UnderstandingorganizationsandsystemsiscriticaltounderstandingandmakingdecisionsaboutwhereandhowinterventionsshouldbeappliedApolicyimplementedinasinglesettingmayshowpromisingresultsbutimplementingmultiplepoliciesandprogramsatthesametimewithdifferentactorsresponsibleforeachaspectcanhelpensuresustainablebehaviorchange
InSomervilleMAShapeUpSomerville(SUS)isacitywidepublichealthprogramthatisimplementedacrossmultiplelevelsofinfluenceandsystemsthroughinterventionsbeforeduringandafterschooltoincludechangestobuiltenvironmentsuchasbikelanesregionalmasstransportationandphysicaleducationequipmentinschoolsandgymstoincreasephysicalactivitywellnesspoliciesandschoolgardensnutritioneducationduringandafterschoolandhealthyrestaurantprogramstosupporthealthyfoodandhealthyeatinghabits
Usethetemplatebelowtoplanyourimplementation
Policyimplementationplanningtemplate
ListthespecificdesiredoutcomesInthepreviousstepyouworkedwithdepartmentstafftoestablishspecificmeasurableoutcomesandinterimindicatorsPrioritizetheindicatorsthatwilldemonstrateprogressUsethespacebelowtocapturethedesiredoutcomesofthepolicychange
ExampleShapeupSomervillebeganwiththegoalofpreventingobesityin1stthrough3rdgraders
Yourturn
____________________________________________________________________________
____________________________________________________________________________
Designimplementationandevaluation
44
SpecifythetargetpopulationAreyoudesigninganinterventionthattargetsonlycertainsitesAreyouplanningonimplementingaprogramorinitiativeforaspecifictargetaudienceClearlyspecifythepopulationyouplantotarget
IfyouplantoconductarigorousimpactevaluationasnotedbytheWilsonSheehanLabforEconomicOpportunitiesitisimportanttoestablishconcreteobjectiveeligibilitycriteriaandensurethatthesecriteriaareappliedconsistentlybyallstaffduringscreeningandintakeItisalsovitaltoidentifytheindividualswhoareeligibleforaparticularprogramorserviceevenifeveryoneeligiblecannotbeservedbyexistingresourcesThishelpscitiesidentifyanyunmetneedsandguidefutureresourceallocationwhilealsocreatingthepossibilityofarigorousimpactevaluationbyidentifyingapotentialcomparisongroupFinallywhenconductingscreeningandintakeitisimperativetocollectandmaintainbasicinformation(egnamedateofbirthgender)aboutallprogram-eligibleindividualsnotjustthosewhoparticipateintheprogram
ExampleIntheShapeUpSomervilleexampletheprogrambeganbytargeting1stthrough3rdgraders
Yourturn
____________________________________________________________________________
____________________________________________________________________________
DefinethelocationWherewillthepolicychangetakeplaceWillimplementationtakeplaceatonesiteormultiplesites
ExampleShapeUpSomervilleinterventionsweredeliveredinschools
Yourturn
____________________________________________________________________________
____________________________________________________________________________
ClarifyimplementationpartnersandarticulatetheirrolesWhatarethekeyorganizationsinstitutionsorpeopleresponsibleforimplementationTheprojectteamshouldbeasclearandspecificaspossibleinprovidingadescriptionoftheconcreteactivitiesthatimplementationpartnersaremeanttocarryoutPracticeprofilesmaybeahelpfultooltohelpdefinewhatisrequiredoftheactorswhoareresponsibleforimplementingtheintervention
Designimplementationandevaluation
45
ExampleInSomervillemayoralleadershipandengagementofmultiplecityagencies(publichealthprimarycareschoolsCambridgeHealthAllianceetc)streamlinedandconnectedthevariousShapeUpSomervilleinitiativesandgrantswhichhelpedtokeepthetalentandenergyfocusedonhealthyeatingandactivelivingforallsectorsofSomervilleSUSpartnersultimatelyincludedschoolsnonprofitsCityandstateagenciesandnationalfoundations
Yourturn
____________________________________________________________________________
____________________________________________________________________________
DescribetheinterventionWhatarethestepsactivitiesorprocessesRememberthatthemechanismthespecificstepsorprocessthatleadtobehaviorchangeiskeytounderstandingtheeffectivenessofthepolicychangeandsharingtheresultswithothers
ExampleGoalsofShapeUpSomervilleweretoinfluenceeverypartofanearlyelementaryschoolchildrsquosdayndashbeforeduringandafter-schooltopromotehealthyeatingphysicalactivityandtheirweightstatusInterventionincludedfoodserviceenhancementstoimprovebreakfastandlunchandstaffprofessionaldevelopmentwalktoschoolactivitiesSUSclassroomcurriculumandprofessionaldevelopmentschoolwellnesspolicydevelopmentSUSafter-schoolcurriculumandprofessionaldevelopmentoutreachandeducationtohomethroughmaterialsforumseventsldquoSUSapprovedrdquorestaurantscommunityoutreachandcapacitybuildingthroughpolicydevelopmenttrainingsmediaplacements
Yourturn
____________________________________________________________________________
____________________________________________________________________________
ClarifytimelineandmilestonesfordeliveryandevaluationHowfrequentlywilltheservicesinterventionbedeliveredHowlongwilltheprogramrunWhatisthetimetoexpectedimpacts(eg12monthsafterprogramcompletion)
Designimplementationandevaluation
46
ExampleShapeUpSomervilleledbyTuftsUniversityFriedmanSchoolofNutritionScienceandPolicyconductedaquasi-experimental(nonrandomized)interventionstudyamong1stthrough3rdgradestudentsinSomerville(intervention)andtwocomparisoncommunitiesfor9-monthsduringoneschoolyear(fall2002ndashspring2003)ThestudyshowedpositiveassociationoftheinterventiononchangeinBMISUSreleasedreportssummarizingevaluationsprogresstodateplansforadditionalprojectsandimpactstenandfifteenyearsintotheinitiative
Yourturn
____________________________________________________________________________
____________________________________________________________________________
PlanyouranalysisTheprevioussectionaddressedestablishinganaccountabilityframeworktomonitorprogressandimplementationDeterminewhichindicatorswillbediscussthroughthatprocesswhowillsharethedataandanalysisandhowfeedbackfromthoseconversationswillbeincorporatedbytheprojectteam
ExampleTheMayorrsquosofficeimplementedadata-drivendecisionmakingapproachcalledSomerStatwhichwassynergisticwiththeapproachestakenbytheresearchandevaluationteamsdeployedforthespecificgrants
Yourturn
____________________________________________________________________________
____________________________________________________________________________
PlanforsustainabilityIfthepolicyandprogramchangeissuccessfulhowwilltheprojectteamsustainandexpandtoadditionalsitesorpopulations
ExampleAlthoughmostofthepartnershipsforgedintheearlyyearsweremaintainednewgrantsandcollaborationsexpandedattentiontoolderpublicschoolstudentsactivetransportationandtheGreenLinesubwayextensionthebuiltenvironmentfarmerrsquosmarketeffortssuchasldquoGrowninSomervillerdquowithUnionMainStreeteconomicdevelopmentandhealthdisparitiesaswellasdatacollectionmonitoringandsurveillance
Yourturn
Designimplementationandevaluation
47
____________________________________________________________________________
____________________________________________________________________________
Pilotplanningtemplate
IfyouplantoconductapilotfirstdeterminethepurposeofthepilotDoyouwanttoobservewhetherthepolicychangeresultsinthedesiredbehaviorchangeAreyouconcernedaboutfeasibilityscalabilityorcostDoyouwanttotestwhetheryourdesiredresultscanbeachievedthroughthispolicyapproachClarifythegoalofthepilotandthequestionsyouhopetoanswerThiswillhelpyouidentifythekeyindicatorsthatyouwillusetomeasurewhetherthepilotissuccessfulBeforebeginningestablishabaselineofcurrentperformanceandsetyourtargetsaccordinglybasedonwhatyouhopetoaccomplishthroughthepilot
NextestablishthepilotplanforyourprojectteamincludingmilestonesforteamcheckinstodeterminewhethertomoveaheadorchangecourseThisisseparatefromthemeetingsthatwilloccuraspartoftheaccountabilityframeworkUsethespacebelowtocreateyourtimelineanddevelopyourmeetingschedule
Task Personresponsible Startdate Checkindate Deliverable
Planyourevaluation
WhileyouaredesigningtheimplementationitisalsotimetobegintodevelopyourplanforanalysisAsnotedbytheWilsonSheehanLabforEconomicOpportunitiesdecisionsaboutprogramsetup-includingoutreachscreeningintakedatatrackingandenrollment-candeterminewhetherornotarigorousimpactevaluationwillbepossibleAsdiscussedcityleadersmayfindithelpfultoincludeevaluationpartnersinearlyconversationsaboutprogramsetupDuringpreviousstagesyoumayhaveworkedwitharesearchertoplanyourproposedpolicychangeYoushouldbeclearaboutthepurposeoftheevaluationanddevelopanevaluationplanBelowareadditionalconsiderations
Causationvstrendanalysis
IntheEstablishaframeworktomonitorprogressstagetherewasguidanceontrackingtrendsovertimeItisimportanttonoteherethatmonitoringdataisdifferentfromanalyzingacausaleffectTrendanalysiswillillustratethatchangesareoccurringandmayleadtoquestionsthatcouldbeansweredbyyourevaluationbuttrendanalysisalonedoesnotprovideevidenceaboutwhatisdrivingthosechangesYoumayseechangesinyour
Designimplementationandevaluation
48
outcomeindicatorsbasedonvariablesoutsideoftheCityscontrolForinstanceachangeinfederaltradepolicycanhaveanimpactemploymentratesinyourCityoutsideofyourCitysnewjobtrainingandplacementprogram
ToaddresscausalityyoumustcomparetheoutcomesofindividualswhoreceivedtheinterventionwiththosewhodidnotWithoutacomparisongroupofsimilarindividualswhowerenotexposedtotheinterventionitisimpossibletoknowwhichfactorsareresponsiblefortheoutcomesRandomizedControlTrials(RCTs)inwhichindividualsarerandomlyassignedtotreatmentandcontrolgroupsarethegoldstandardinprovingcausalityIfyoucandemonstratethatoutcomesimprovedforthosewhoreceivedtheinterventionanddidnotimproveatallorimprovedlessforacomparisongroupyoucanbesomewhatcertainthattheinterventionwasthecause
Examplesofothernon-causalimpactevaluationsthatarerigorousandreliableincludecertainquasi-experimentalstudiesusingcomparisongroupdesignssmartinterruptedtimeseriesanalysesthatmodeltrendsovertimeinwaysthatallowestimationofinterventioneffectswelldesignedpre-posttestanalysesneedsassessmentsimplementationstudiesetcFormoreinformationondifferenttypesofevaluationsseetheSageResearchMethodswebsite
Lookfordifferencesingroups
BuildintimetoalsoexaminethepilotdatatoseeifgroupsareexperiencingdifferentresultsReviewingperformancebasedonoutcomemeasuresforapopulationinaggregatemayhidethesubtlewaysthatsomepeoplemaybeaffecteddifferentlysoitisnecessarytocarefullyanalyzethedatafordisparateoutcomesForinstancetheMovingtoOpportunity(MTO)studythatexaminedtheimpactofhousingvouchersonfamilieswholivedinhigh-povertyneighborhoodsfoundthatchildrenwhomovedbeforetheageof13hadbettereducationandhigherincomesasadultsHoweveradultsdidnotseeimprovededucationandemploymentoutcomes
Disaggregatingdataintomorespecificcategoriesofraceandethnicitycanhelpunderstandtrendsanddisparitiesinamoregranularwaywhichcanhelpdecisionmakerstargetpoliciesandinterventionsinamoreequitableway
EarlierintheRoadmapweworkedtoidentifystakeholdersandwhomightbepositivelyandnegativelyimpactedbyachangeinpolicyUsetheresultsofthatexercisetoinvolvethecommunityininterpretingdataanalysisandtogatherqualitativefeedbackfromthecommunitytocallattentiontoissuesForinstanceanincreaseinyouthdetentionmaybeexplainedbyinefficienciesorlackofaccesstotransportation
Designimplementationandevaluation
49
Designimplementationandevaluation
50
Shareyourstoryandsuccess
Sharetheresults
Storytelling
AnintegralstepinyourstrategyissharingtheresultsofyourpilotorpolicychangewithyourcommunityThisincludesdevelopingmessagingaroundtheworkofyourorganizationandcreatingaplantodisseminateyourstoriesWhytellstoriesaboutyourwork
Informandorunderstandasubjectorsituation
Giveneededcontextofdataorbitsofinformation
Relatablestoriesarememorableforreadersviewers
ReadersorviewerswillseetheinformationasmorecompellingwhenthereisademonstratedandrelatablechangeorimpactonthemselvesortheirfriendsneighborsandcolleaguesStoriesareaboutpeopleandgovernmentisfundamentallyabouthelpingpeopleSotellthestoriesofthepeopleyouandyourcitycolleagueshelped
Keepthesetipsinmindasyoudevelopyourstorywithyourteam
WhatwereyourfindingsandwhyshouldanyonecareTellyouraudiencehowtheproblemwassolvedWhathastheimpactbeenonresidentsandcitystafftodateWhatwasthesuccessandweretheremultiplesuccessesalongthewayIfyouhavepreliminaryresultsordatashowingimprovementtosharethisisthetimetodoit
WhowasinvolvedandimpactedReviewthelistofstakeholdersHowwereeachofthemimpactedbytheprogramHumanizeyourstoryandimpactbyaddingadditionalvoicesThesevoicescanbeoneofthefollowing
Apersonorfamilywhowaspositivelyaffectedbythechangesuccess
Aninvolvedthird-partydescribingboththeproblemandthesolution(ieanorganizationnonprofitprovidersbusinesstradeassociationsportsteametc)whoispartofthecommunitythatthegovernmentworkedwithonthechangesuccess
Someonefromthegovernmentitselfwhocanspeakaboutthechangesuccess
TellmoreaboutyourcityDescribeyourcityincludingpopulationdemographicsindustriesetcWhatareitschallengesWhatisitknownforWhatmakesituniqueHowisitsimilarordifferentfromothercitiesintheUS
Shareyourstoryandsuccess
51
DescribeyourcityrsquoschallengeWriteoutthechallengeyourcityisfacing(asdefinedinyourproblemstatement)HowdoesitaffectresidentsHowdoesthisproblemaffecttheworkofcitystaffmembers
AddthedetailsProvidedetailsonyourappliedresearchstrategyTalkaboutwhatyourcityhasdonenthepasttoresolvetheissueDiscusshowyourstrategyhaschangedwhenyoubegantoadoptthisapproachWasmoneysavedorinvested
Usethisstorymapexercisetoorganizeyourthoughtsandguaranteethatyouincludealloftheelementsofatraditionalstory
AfteryoudevelopyourmessagingcreateacommunicationsmarketingandoutreachstrategydesignedtospreadyourmessageYourstrategyshouldwalkyouthroughthefollowingkeyactivities
1 IdentifyandprofileyouraudienceWhencreatinganddisseminatingyourmessageitisalwaysgoodtothinkaboutyouraudienceWhoexactlywouldbeinterestedinwhatyouhavetosayThegeneralpopulationofthecityPeoplewhoworkinnonprofitsBusinessownersItmattersbecausenoteveryonewholivesinacitymaybethetargetoftheinformationBusinessownersmaybemoreinterestedindatashowingeffectsonthebusinessdistrictthanothersandsoonWewillcoverthislaterbutthisstepwillalsohelpyoutothinkofthemosteffectivewaytoreachyouraudience
2 Defineyourcalltoaction(ifyouhaveone)SpendsometimeconsideringwhyyouareinterestedintellingyourstoryDoyousimplywanttoupdatethepubliconyourprogressorcelebrateyoursuccessesAreyouinterestedinyouraudiencechangingitsbehaviororsupportingaspecificinitiativebeinglaunchedbyyourorganizationIfyouwanttohaveyouraudienceactorbehaveinacertainwaybespecificandincorporateitinyourmessaging
Shareyourstoryandsuccess
52
3 SelectperformancemeasuresThinkabouthowyouwoulddetermineifyourcampaignissuccessfulandselectappropriateperformancemeasuresYouwillrevisitthesemeasuresattheendofyourcampaigncommunicationsplanimplementationtodeterminewhetheryouneedtomodifyyourcommunicationsandoutreachstrategyforthefuture
4 MethodstocommunicateyourstoryTherearemanywaystocommunicatesuccessofyourprojectHerearesomesuggestionsonreachingaudiencesItisimportanttodeterminewhichmethodsarethemosteffectiveforyourdesiredaudiencetoreceiveinformationDoyourresearch(PewResearchCenterhasexcellentstudiesonthereachofmediadigitalmediaandmarketing)ForexampleifseniorsareyourtargetpopulationandtheyarelessinclinedtouseInstagram(thatistrue)youmaywanttoconsiderFacebookorthenightlylocalnewsbothofwhichareusedmoreoftenbyseniorsBelowisalistofsomemethodsandtoolsyoucanuse
i Pressreleasestolocalnewspapersandtvstations
ii ShortsmartphonevideosAgreatwaytoengageanaudienceisthroughvideoAskonequestionofsomeonewhoispartofthestoryMaybethereisasocialworkerwhocandiscusshungerinyourcityThesevideoscanbereusedonmultipleplatforms
iii Long-formwriting
iv TellingindividualstoriesthroughaseriesofphotosandshortnarrativesforpublicationonFacebookorInstagram
v Op-edsopinionpiecesplacedinthelocalpaper
vi Emailupdatestoyourlist
vii Connectingwithappropriateorganizationsthatareinvestedinthetopicandworkwiththecommunityandaskthemtodisseminateinformationtotheirmembers
viii EmployingldquoWordofMouthrdquo(WOM)tacticsbycreatingtownhallsorfreeeventsforresidentstoattendintheirownneighborhoods
5 DeterminethetypeofcollateralUsuallycampaignsthattargetresidentsrequireseveralavenuestoreachthemPeoplearenotalwaysinthesameplaceorspaceHavingakeenunderstandingofyouraudiencewillassistyouwithyourchoicesonwhatneedstobedevelopedSometimesthatmeansimagesforsocialmediaaonepageflierthatgivespeopleinformationonattendinganeventoranin-depthbrochure
Shareyourstoryandsuccess
53
6 ImplementyourstrategyKeepinmindstrategiescanbefluidSometimessomethingmaynotturnoutasexpectedevenifduringplanningsessionsitwasldquosupposedtordquoItisokYoucananalyzewhathappenedduringadebriefsessionandonceyoursurveysandothermeasurementshavebeenfinalized
Visualizations
Withtherightvisualizationyoucanreviewandshareinformationinsightsanalysisperformanceandtrendsallofwhichisvaluabletoacross-sectionofthepublicButwhichvisualizationisrightforyouTochoosetherighttypeofchartgraphdeterminethetypeofquestionyouwanttoanswerSomecommoncharttypesarebelowandmoreinformationcanbefoundhere
Exploreadditionalanalyticaltechniques
Duringthegoal-settingphaseyouestablishedoutputoutcomeandimplementationmeasuresBasedontheresultsofyourpilotyoumayneedtoadjustyourmeasuresWiththepilotdatainhandyoumayhavesomeideasofhowfutureanalysescouldbedifferentandpotentiallyimprovethequalityofyourprogramTakesometimetobrainstormotherrelevantdatasetsthatyoudidnothavethetimetopursueearlierincludingmessydepartmentaldatathatneedsmoreattentionorsourcesofpublicdatafromthestateandfederalgovernmentnonprofitsandevenprivatecompaniesthatcouldsupplementyourunderstandingofanissueUsingmachinelearningtechniquesyoumightbeabletobetter
Shareyourstoryandsuccess
54
predictdemandforhomelessservicesusingNOAAweatherforecastspreemptwatermainbreaksusingsoilstratigraphyorinvestineconomicdevelopmentbymonitoringrealestatelistingdata
Advancedanalyticaltechniquescanalsohelpyourimplementationrunmoreeffectivelybyintegratingdata-informedtargetinganddecisiontoolsintotheprocessByautomatingroutineworkflowsandsurfacingsubtletrendsthesetoolscanhelpyouiterateinamoreprecisemannerandprovidekeystrategicguidancetofront-lineemployeesthatwouldotherwisebeunavailableThesesortsofdata-science-enhancedprocessescanbeclassifiedasoneofsixprojecttypesasexploredfurtherbytheNewOrleansLANOLAlyticsteamFormoreguidanceonscopingasimilarprojecttheGovExadvancedanalyticsscopingguidegivesyouastep-by-stepoverviewofhowtobesuccessful
Usethespacebelowtolistyourideasforpossibleadvancedanalyticsprojectsbytype
Typeofproject Ideas
Findingtheneedleinthehaystack
Prioritizingforimpact
Earlywarningtools
Betterquickerdecisions
Optimizingresourceallocation
Experimentingforwhatworks
Keepthefeedbackloopgoing
GatherinformationonpublicsentimentaboutpolicychangethroughfocusgroupssurveysandpublicmeetingsledbyadvisorygroupmembersNotewhethernewinformationisavailablethatcanhelpanalyzeorcontextualizeyourprogress(egfederalorstatedatareports)Sharetheresultsofyourpilotandthepolicychangeprocesswithothercitiesandresearchers
Shareyourstoryandsuccess
55
WorkingwitharangeofevidencePresentlythereisalotofattentionandinvestmentinevidence-basedpolicymakingincludingthefederalbipartisanCommissiononEvidence-BasedPolicymakingthePewCharitableTrustsreportsandtoolkitsonHowStatesEngageinEvidence-BasedPolicymakingandtheLauraandJohnArnoldFoundationsupportedEvidence-BasedPolicymakingCollaborativeManyoftheseinitiativesfocusonrigorousevaluation--meaningrandomizedcontrolledtrials(RCT)orotherexperimentalmodels--asthebestkindofevidencetouseHoweverthisbodyofevidenceisofteninsufficientforthespecifickindsofquestionsthatgovernmentscanhavewhenmakingadecisionaboutpolicyInthisguideweproposeanapproachtosupportgovernmentsinincorporatingadditionaltypesofevidenceintotheirdecisionmaking
OnewaytounderstandarangeofevidenceistouseNestarsquosStandardsofEvidencetoframeanddefineevidenceTheBritishnonprofitNestaworkstoinjectpositivechangeintothewayUKgovernmentpoliciesandprogramsfunctionOneoftheirkeyareasofworkhasbeendevelopingWhatWorksCentresorganizationsthatcollectcommissionsynthesizeanddisseminateevidenceaboutinterventioneffectivenessacrossarangeofsocialimpactareasNestadevelopedthefollowingStandardsofEvidencetoprovideausefulwaytodocumenthowdifferentkindsofinformationdemonstratedifferentlevelsofevidence
Workingwitharangeofevidence
56
Theimportanceofpotentiallybiasedinformation
Onechallengepresentedbyanevidenceclassificationthatacknowledgesthevalueofnon-scientificformsofevidenceistofigureouthowtobalanceallofthepiecestogetherwhenyouusethemTounderstandhowtodothiswehavetothinkabouthowtopositionpotentiallybiasedinformationindecisionmaking
WhenwelookatinformationwhichhasnotbeenrigorouslyevaluatedweacceptthatthisinformationmightbebiasedThewholereasonweconductrigorousevaluationistoremovebiasfromourobservationsWeconductstructuredtrialstoproduceresultsthatareconsistentandgeneralizableIfafindingisunbiasedthisaccuracymdashthislackofbiasmdashcanbeprovedbecauseanyindependenttestercanreproduceitwhentheyfollowanidenticalprocess
YoucanlookatpotentiallybiasedinformationasjustinformationthatisearlieroninitspathtorefinementreplicationandverifiedaccuracyApieceofinformationbasedonanyoneindividualrsquosobservationsislikelytobebiasedbecauseitiscomingfromjustonesourceBiasisreducedbygatheringtogetherlotsofindependentsourcesofinformationandfiguringoutitsmostcommonattributes
MaybethebiasedinformationmakesaclaimthatistoolargemdashitsayssomethingistrueunderallconditionswhenitisreallyonlytrueundercertainconditionsorthataneffectofaninterventionisbiggerthanitstrueaverageorthataneffectlastsovertimewhenitdoesnotPerhapschangingtheclaimtobemorespecificmdashcriticallyreviewingtheinformationtoseeifspecificconditionsmustbemettoachievetheoutcomesmdashwouldmaketheclaimmoreaccurate
IfyoutestedyourclaimfounditdidnotholdupinsomecaseschangedittesteditagainuntilyourclaimrepeatedlyheldupthenyoucoulddemonstratethattheclaimwasnolongerbiasedbutaccurateThisprocessoftestingimprovingtestingandvalidatingiswhatwegetfromstructuredscientificstudy
Withindividualpiecesofinformation--crediblehypothesescasestudiesexpertopinion--youcannotassumethatkindofaccuracyAtthesametimetheseearlyobservationsarefoundationaltoanyreplicablescientificobservationsWewillnevergetanywhereifwedonotstartwiththesepiecesofinformationeveniftheymightbebiased
Combininglevelsofevidencetogetthemostdetailedinformationforyourcity
ItisespeciallyimportanttoacceptdifferentformsofevidencewhenyoufacequestionsthathavenotyetbeentestedbyindependentstudiesSomeofthemostimportantquestionsrelatetothingsthatareveryspecifictoyoursituationYoufaceaparticularcombinationof
Workingwitharangeofevidence
57
issuesactorstrendsandresourcesthatisultimatelydifferentfromeveryotherplaceintheworld
ThisisimportantbecauseaspolicypractitionersandnotscientistsyoucaremoreaboutyourspecificcasethanahypotheticalaveragegeneralcaseYoucarelessaboutthetotallikelyrangeofimpactsfromapolicythanhowthissingleimplementationwillworkYouwantinformationaboutpoliciesthatwillprovideyouwiththebestpossiblepredictionabouthowtheywillworkinyourcontext
ByputtingdifferentkindsofevidenceintothesameframeworkNestarsquosStandardsofEvidenceimplythatwhenyoucannotgetrigorouslytestedexperimentallyverifiedinformationitisappropriatetoworkwithwhatwedohaveThisisrelevantforthemunicipalimplementationcontextbecausenostudypulledfromaliteraturereviewwillaccountforallofthevariablesyoufaceinyourowncontext
Intheabsenceofhavingastudythatpredictshowaninterventionworksinyourspecificcontextyouwillneednon-scientificinformationtotrytofillinthosegapsYouwillneedtogatherstoriesfromresidentsstaffmembersandotherstakeholdersthatsitonLevels1or2ofthestandardsofevidence
Someofthekindsofinformationyouwillprobablygetmoreeasilyfrompeoplethanstudiesinclude
howpeopleincitieslikeyoursreactedtoaninterventioninpublicforumshowpeoplewhoimplementedaninterventioninacitylikeyoursfiguredouthowtogetitpassedhowthecitycommunicatedpubliclyabouttheapparentbenefitsanddrawbacksoftheinterventionthepoliticalacceptabilityoftheintervention
Youcanremainsimultaneouslyawarethattheinformationyouaregatheringthroughindividualstoriesisbiased--notgeneralizableinformedbyspecificperspectives--whilealsorecognizingthathavingsomeinformationaboutsomethingimportantisbetterthannoinformationEvidencefromthefirsttwoevidencelevelsareessentialbuildingblocksofknowledgeandperfectlyusefulwhenyoucannotgetdetailedunbiasedknowledge
Youhavetostartsomewhereandyoushouldalwaysstartwithevidenceyouhaveaboutyourveryspecificconditionsthatyouknowareimportantforsuccess
Workingwitharangeofevidence
58
Thankyou
Interviewswithcities
GovExheldsemi-structuredinterviewswithsixstaffmembersinfourcitiestoestablishtheinitialstepsoftheRoadmapWeaskedrespondentstotalkaboutaninitiativeorpolicythattheyhavebeenresponsibleforimplementingintheircityRespondentsfromthreecitiesspokeabouttheireffortstoaddresshomelessnessandrespondentsfromonecityspokeaboutblightAfterrespondentsidentifiedtheproblemtheyspokeabouthowthecityprioritizedthisissuethevariousstepsinvolvedandbarrierspresent
GovExconductedtextanalysisoftheseinterviewswhichrevealedthatpeoplewasthemostcommonlyusedtermduringtheseconversationswhendiscussingthestepstochangeaprogramorpolicyThisisathemethroughoutthisRoadmapThefirstsectionbeginswithidentifyingtherelevantpeopleTheconversationsaroundevidencearefocusedonhumanbehaviorratherthanmethodologyorstrengthofevidence
Thankyou
59
OncewehadadraftoftheRoadmapwefollowedupwithcitiesonanindividualbasistolearnmoreaboutwhatwouldbehelpfulWearegratefultothefollowingcitystafffortheinterviewsandindividualconversations
MichaelBaskinChiefPolicyOfficerChattanoogaTN
MelissaBridgesPerformanceandInnovationCoordinatorLittleRockAR
VictoriaCarreoacutenAdministrativeOfficerLasVegasNV
TyroneGrandstrandCommunityDevelopmentPlannerFargoND
TibLaughlinDirectorofGeneralServicesKansasCityKS
TimMorelandDirectorofPerformanceManagementandOpenDataChattanoogaTN
CharlesRosendahlManagementAnalystDentonTX
Thankyou
60
MelissaSiebenAssistantCountyAdministratorKansasCityKS
GinaThompsonDirectorofStrategicServicesLewisvilleTX
JHUFacultycollaboration
GovExalsoconvenedthecitycontactsandfacultypartnersinaseriesofAdvisoryGroupmeetingsThesemeetingsservedtoconnectcitiesandfacultytounderstandoneanothersperspective
StefanBaralBloombergSchoolofPublicHealth
AshleyBernerJHUSchoolofEducation
MelissaDavey-RothwellBloombergSchoolofPublicHealth
JonLinksBloombergSchoolofPublicHealth
TamarMendelsonBloombergSchoolofPublicHealth
KristinMmariBloombergSchoolofPublicHealth
KeshiaPollackPorterBloombergSchoolofPublicHealth
JoshSharfsteinBloombergSchoolofPublicHealth
LizStuartBloombergSchoolofPublicHealth
LindsayThompsonCareyBusinessSchool
DanielWebsterBloombergSchoolofPublicHealth
Partnersindataandevidence-basedpolicy
Wealsospokewithpartnerswhoareexpertsinthefieldofevidence-basedpolicywhosharedtheirknowledgeandresourcesofcurrentandfuturethinkinginevidence-informedthinking
WendyBarrenoAssociateDirectorWilsonSheehanLabforEconomicOpportunitiesUniversityofNotreDame
TiffanyDavenportDirectorofDataPartnershipsBaltimoresPromise
TomiHiersDirectorBaltimoreCivicSiteAnnieECaseyFoundation
ChrisKingsleySeniorAssociateAnnieECaseyFoundation
KathrynOliverAssociateProfessorLondonSchoolofHygieneandTropicalMedicine
Thankyou
61
KristenWatkinsSeniorPolicyAssociateJ-PAL
GovExParticipants
GovExAnalystMiriamMcKinneyanalyzedtheinterviewtranscriptsandconductedtextanalysisSeniorImplementationAdvisorEmilyShawwillintegrateRoadmapactivitieswithadditionalservicedeliveryactivitiesatGovEx
Thankyou
62
ReferencesAbrahamKGampHaskinsR(2017)Thepromiseofevidence-basedpolicymakingReportofthecommissiononevidence-basedpolicymakingWashingtonDCCommissiononEvidence-basedPolicymakingBigmove2Extendingthegreen(nd)Retrievedfromhttpswwwcityoflewisvillecomdoing-businesslewisville-2025big-move-2-extending-the-greenBatesMAampGlennersterR(2017)ThegeneralizabilitypuzzleRetrievedfromhttpsssirorgarticlesentrythe_generalizability_puzzleBuildwithnotfor(2018January)RetrievedfromhttpsbuildwithnotforwordpresscomCairneyPampOliverK(2017)Evidence-basedpolicymakingisnotlikeevidence-basedmedicinesohowfarshouldyougotobridgethedividebetweenevidenceandpolicyHealthResearchPolicyandSystemsdoi101186s12961-017-0192-xCairneyPOliverKampWellsteadA(2016)TobridgethedividebetweenevidenceandpolicyReduceambiguityasmuchasuncertaintyPublicAdministrationReview76(3)399-402Retrievedfromhttpdxdoiorg101111puar12555Centreforpublicimpact(nd)RetrievedfromhttpswwwcentreforpublicimpactorgCernaL(2013)ThenatureofpolicychangeandimplementationAreviewofdifferenttheoreticalapproachesOrganisationforEconomicCo-OperationandDevelopmentRetrievedfromhttpswwwoecdorgeduceriTheNatureofPolicyChangeandImplementationpdfColoradoSprings(nd)Retrievedfromhttpscoloradospringsgovcommunicationspagewhat-expect-when-you-speakupContinuumofcare(coc)program(nd)RetrievedfromhttpswwwhudexchangeinfoprogramscocCouncilpresidentkicksoff$12millionyouthfund(nd)Retrievedfromhttpwwwbaltimorecitycouncilcomcontentcouncil-president-kicks-12-million-youth-fundDurlakJAampDuPreEP(2008)ImplementationmattersAreviewofresearchontheinfluenceofimplementationonprogramoutcomesandthefactorsaffectingimplementationAmericanJournalofCommunityPsychology41327-350Retrievedfromhttpswwwncbinlmnihgovpubmed18322790Fishbonediagram(ishikawa)(nd)Retrievedfromhttpasqorglearn-about-qualitycause-analysis-toolsoverviewfishbonehtmlFivewhysGettingtotherootofaproblemquickly(nd)RetrievedfromhttpswwwmindtoolscompagesarticlenewTMC_5Whtm
References
63
FligsteinNampMcAdamD(2011)TowardageneraltheoryofstrategicactionfieldsSociologicalTheory29(1)1-26RetrievedfromhttpsociologyberkeleyedusitesdefaultfilesfacultyfligsteinFligstein20McAdam20Soc20Theory20PaperpdHeinkelP(2018January12)CitymullspermitsforbikesharingcompaniesRetrievedfromhttpwwwdentonrccomnewsdenton20180112city-mulls-permits-bike-sharing-companiesHowisareamedianincomecalculated(2012October)Retrievedfromhttpswwwhudexchangeinfofaqs843how-is-area-median-income-calculatedLampkinLMampHatryHP(2003)KeystepsinoutcomemanagementUrbanInstituteRetrievedfromhttpswwwurbanorgresearchpublicationkey-steps-outcome-managementMarshDampSmithM(2000)UnderstandingpolicynetworksTowardsadialecticalapproachPoliticalStudies484-21Retrievedfromhttpjournalssagepubcomdoiabs1011111467-924800247McCannL(nd)RetrievedfromhttplaurenellencomMoultonSampSandfortJR(2016)ThestrategicactionfieldframeworkforpolicyimplementationresearchPolicyStudiesJournal45(1)Retrievedfromhttponlinelibrarywileycomstore101111psj12147assetpsj12147pdfv=1ampt=jdkhhcp4amps=e184c625e1855a6b1615ccb99ee27b1c0b3212d8PaynterB(2018January1)What100citiesarelearningfromeachotherbysharingtheirdataRetrievedfromhttpswwwfastcompanycom40518004what-100-cities-are-learning-from-each-other-by-sharing-their-dataPeakacademy(nd)Retrievedfromhttpswwwdenvergovorgcontentdenvergovenmayors-officeprograms-initiativespeak-performancepeak-academyhtmlPinnockHBarwickMCarpenterCREldridgeSGrandesGGriffithsCJRycroft-MaloneJMeissnerPMurrayEPatelASheikhAampTaylorSJC(2017)Standardsforreportingimplementationstudies(stari)ExplanationandelaborationdocumentBMJOpenRetrievedfromhttpbmjopenbmjcomcontentbmjopen74e013318fullpdfProctorESilmereHRaghavanRHovmandPAaronsGBungerAGriffeyRampHensleyM(2011)OutcomesforimplementationresearchConceptualdistinctionsmeasurementchallengesandresearchagendaAdministrationandPolicyinMentalHealth38(2)65-76doi101007s10488-010-0319-7TandonSD(2018)ProgressincommunityhealthpartnershipsResearcheducationandactionRetrievedfromhttpswwwpressjhuedujournalsprogress-community-health-partnerships-research-education-and-actionProvanKGampKenisP(2007)ModesofnetworkgovernanceStructuremanagementandeffectivenessJournalofPublicAdministrationResearchandTheory
References
64
18(2)229-252Retrievedfromhttpsacademicoupcomjpartarticle182229935895SabatierPA(1988)Anadvocacycoalitionframeworkofpolicychangeandtheroleofpolicy-orientedlearningthereinPolicySciences21(3)129-168Retrievedfromhttpwwwjstororgstable4532139SandstromAampCarlssonL(2008)TheperformanceofpolicynetworksTherelationbetweennetworkstructureandnetworkperformancePolicyStudiesJournal36497-524Retrievedfromhttponlinelibrarywileycomdoi101111j1541-0072200800281xabstractSchwartzMampWilsonE(nd)WhocanaffordtoliveinahomeAlookatdatafromthe2006americancommunitysurveyRetrievedfromhttpswwwcensusgovhousingcensuspublicationswho-can-affordpdfShearerJCAbelsonAKouyateBLavisJNampWaltG(2016)WhydopolicieschangeInstitutionsinterestsideasandnetworksinthreecasesofpolicyreformHealthPolicyandPlanning31(9)1200-1211Retrievedfromhttpsdoiorg101093heapolczw052ThelessonsofadministrativedataHigh-profilepolicy-relevantresearchpoweredbyadministrativedata(nd)Retrievedfromhttpswwwpovertyactionlaborgsitesdefaultfilesresources20161114-The-Lessons-of-Administrative-DatapdfToolsandotherneedsforqualitativeandquantitativemethods(2015February)AgencyforHealthcareResearchandQualityRetrievedfromhttpwwwahrqgovprofessionalsquality-patient-safetytalkingqualityassesstoolshtml
WeaverRCampBagchi-SenS(2013)SpatialanalysisofurbandeclineThegeographyofblightAppliedGeography4061-70
References
65
UniversityPartners
Name Title Affiliation Area(s)ofexpertise
Contactinformation
Examplesofworkingwithnon-academicdecisionmakers
WendyPBarreno
AssociateDirector
WilsonSheehanLabforEconomicOpportunitiesUniversityofNotreDame
LEOsfocusareasincludehealtheducationhousingchildwelfareeconomicopportunitycriminaljustice
wbarrenonduduorleondedu
MelissaDavey-RothwellPhD
AssociateScientist
JohnsHopkinsBloombergSchoolofPublicHealth
Health mdavey1jhuedu410-614-5854
BaltimoreCityHealthDepartmentLeadersMarylandDepartmentofHealthAddictionRecoveryServicesinDCExecutiveDirectorsofcommunityorganizationsacrosstheUSthatfocusonHIVSTIsubstanceabuseandimplementationscience
ElizabethStuart
AssociateDeanforEducationandProfessorDepartmentsofMentalHealthBiostatisticsandHealthPolicyand
JohnsHopkinsBloombergSchoolofPublicHealth
Educationsocialpolicystatisticalmethods
estuartjhuedu
Resources
66
Management
Design-ThinkingResources
EngagementGuideforCivicTech
Journeymap
PerformanceManagementResources
DenverPeakAcademy
Proxymeasuresguide
Talkingaboutperformancewithoutcomplicatingit(video)
SettingPerformanceTargetsguide
Datavisualizationguide
Performancemanagementgettingstartedguide
Exercises
Fishboneexercise
5Whys
AdministrativeData
TheLessonsofAdministrativeData(JPAL)
EvaluationResources
Quantitativeandqualitativemethods(USDepartmentofHealthandHumanServicesAgencyforHealthcareResearchandQuality)
KeyStepsinOutcomeManagement(UrbanInstitute)
RCT-YES
Sageresearchmethods
BestPracticesforMixedMethodsResearchintheHealthSciences
ACCORDChecklist
Pilotandimplementationplanning
Benchmarkingexamplesandproxymeasures
Seattleopendataplaybook
Resources
67
Datasciencescopingguide
Resources
68
AboutGovExGovExhelpsgovernmentsusedataandevidencetoimproveresidentsrsquolivesWesupportgovernmentsinbuildingtheirinternalcapacitytooperatesustainableanalyticspracticeswhichimprovecommunityoutcomesinfieldsfrompublicsafetytoeconomicdevelopmentGovExhasaproventrackrecordofworkinginconcertwithpartnerstoprovidetechnicalassistanceandtrainingatscale
SinceourlaunchinApril2015GovExhas
Trainedover2000governmentemployeesinpersonandonlineintopicsincludingdatamanagementperformancemanagementopendatacommunityengagementandanalytics
Providedtechnicalassistancetomorethan100localgovernmentsacrosstheglobeimprovingtheiruseofdatatoachieveefficienciesinservicedeliveryandbetteroutcomesforresidents
Assessedthecapacityofmorethan130governmentalorganizationstodeterminehowleadersstaffandexternalstakeholdersmanageandusedata
Localgovernmentsareincreasinglyonthefrontlinesofaddressinglargescaleproblemswithglobalimplicationssuchasclimatechangepublicsafetyandeconomicinequality
Whileevidence-basedinterventionstohelpsolvethoseproblemsareavailablelocalgovernmentstaffhavelimitedtimeandcapacitytofindandimplementresearch-basedsolutionsAsaresultcitiescontinuetoworkthewaytheyalwayshave--makingincrementalimprovementstoprogramsbutnotalwayscreatingchangeatasystem-level
ThroughresourcedevelopmentandcollaborationwithJHUfacultyGovExissupportingcityleadersinusingdataandevidencetomakedecisionsandcreatinglastingrelationshipswithresearchers
TheRoadmapwasdevelopedbyGovExaspartofitspartnershipwithBloombergPhilanthropiesWhatWorksCitiesinitiative
AboutGovEx
69
- Title page
- Introduction and purpose of the Roadmap
- Using the Roadmap
-
- Gather information about the problem
- Strategically engage key partners
- Review the evidence and collect details
- Decide on a policy to implement based on local factors
- Establish a framework to monitor progress
- Assess data accessibility quality and security
- Design implementation and evaluation
- Share your story and success
-
- Working with a range of evidence
- Thank you
- References
- Resources
- About GovEx
-
sometimegettingtoknoweachotherrsquosworktolearnabouteveryonesstrengthsThissharedunderstandingcanhelpasyouclarifyrolesandresponsibilitiesofeachpartnerinthegroup--thisisimportanttoavoidfinger-pointinginthefutureandtoensurethatnoneofthetasksslipthroughthecracksInsomecasestherearemanyactorswithdifferentresponsibilitiesasinthecaseofBaltimoreCitys38-memberYouthFundTaskForcewhichischairedbynonprofitandcorporatepartnersandincludesrepresentativesfromthenonprofitandfaithcommunitiesstudentsthebusinesssectorlocaluniversitiesCityCouncilandtheMayorsOffice
OncerolesandresponsibilitiesareclearyoumayconsiderdevelopingacharterguidelinesortermsofreferencetosetexpectationsforthegroupBesuretodistinguishthisfromotherboardsandcommissionsthatyourcitymayhaveNextyoumaywishtoestablisharegularongoingmeetingscheduleBesuretosendagendasandmeetingmaterialsaheadoftimeandclarifytheobjectiveofeachmeeting
InSeattletheCitydevelopeditsHousingAffordabilityandLivabilityAgenda(HALA)torespondtorisinghousingcostsandhomelessnessThisisatrendinotherwestcoastcitiesaswellwhichmaybeexplainedbyaninfluxofnewresidentswhoareattractedbytheemergenceoftechjobstheattractionoftheoutdoorlifestyleortherecentpassageofprogressivepolicieslikelegalizedgaymarriagea$15minimumwageandlegalizationofmarijuanaSeattlebroughttogetheramulti-stakeholderadvisorygroupcomprisedoflocalhousingexpertsrentersandhomeownersdevelopersandgovernmentofficialstoshareinformationabouttheirneedsanddeveloprecommendationsbasedontheseneedsTheinterestsandmotivationsofthisgroupwerevariedandittookabouttenmonthstofindcommongroundTheseriesofrecommendationsthattheadvisorygroupcreatedismeanttoserveresidentswithaspectrumofneeds
Buildaprocessforcommunityparticipation
Ofcoursenoteveryonecanbeincludedinanadvisorygroupandadvisorygroupmembersmaynotnecessarilyrepresentthevariedperspectivesoftheirpeers-forinstancenoteveryhomeownerhasthesamesetofneedsTogatherinputfromawiderswathofthepopulationitcanbehelpfultobuildaprocesstoinvitethepublicatlargetosharetheirideasCitiesdothisinavarietyofwaysfromonlineconversationsandsurveystoin-personmeetingsRememberthatbeingintentionalaboutgatheringtheviewpointsofpeopleismoreimportantthanusingthelatesttechnologyandtools
TheinsightgeneratedbycommunityparticipationexercisescanbeincludedinyourreviewofevidencewhichisdiscussedinasubsequentchapterInWestVirginiaastatethathasbeenhithardbytheopioidepidemicstatehealthofficialsdraftedanopioidresponseplanthatwasinformedbypublicengagementaswellaslocalandregionalsubjectmatter
Strategicallyengagekeypartners
17
expertsThepublicwasinvitedtopresentcommentsonideasfordecreasingoverdoseearlyinterventiontreatmentandrecoveryPublichealthandmedicalfacultyfromWestVirginiaUniversityMarshallUniversityandJohnsHopkinsUniversityprovidedresponsestothecommentsTheresultwasanOpioidResponsePlanthathealthcommissionerRahulGuptapresentedtoGovernorJimJustice
AsamplespectrumtoplanyourowncommunityparticipationprocesscreatedbyLaurenellenMcCannisbelow
SourceLaurenellenMcCann
Community-basedparticipatoryresearchisanotherwaytoworkwiththecommunitytoimproveoutcomesCBPRmeansthecommunityisinvolvedinallaspectsoftheresearchprocesswhichisofteniterativeInCBPRtraditionalresearchersandcommunitymembersshareknowledgeaswellasresourcesandcredit
CBPRInActionInSanFranciscoanenvironmentaljusticenonprofitteamedupwiththeCitysDepartmentofPublicHealthandanoutsideevaluatortounderstandtherootcausesofobesityandfoodinsecurityinlow-incomeneighborhoodsandidentifysolutionsThroughresearchmethodssuchasinterviewingresidentsandstoreownersdiagrammingstoreshelvesGISmappinganeconomicanalysisandreviewofsimilarpoliciesinnearbycitiestheteamobservedthatthestoresthatresidentshasveaccesstowerestockedwithlesshealthyfoodsandthatresidentswantedmorehealthyfoodandlessalcoholandtobaccoThisprojectresultedinlocalandstatelegislationtoencouragestorestostockfreshproduce
Strategicallyengagekeypartners
18
Developrelationshipswithsubjectmatterexperts
OnewaytokeepsubjectmatterexpertsengagedistoinvolvetheminthecommunityparticipationprocessasdescribedaboveIntheWestVirginiaexamplesubjectmatterexpertsrespondedtoandsynthesizedthepubliccommentsOtheropportunitiesaredescribedbelow
AdvisoryAcitydepartmentheadMayororstaffmembermaydevelopaone-on-oneadvisoryrelationshipwithuniversityfacultymemberoracollaborativebetweentheCityandUniversitytoaddressaparticularproblemAgreatplacetostartistolookforrelevantpracticecenterswithinyourlocaluniversityandreachingouttosubjectmatterexpertsInadditiontotheirexpertiseontherelevantpolicytopicheorshemayalsobeabletoidentifyotherexpertsandevenprovidesupportonhiringtherighttalenttoleadcertainprogramsAnadvisoryrelationshipwithauniversitysubjectmatterexpertcanalsohelpwhencitiesareconsideringbringinginnewprogramsTheseexpertsareoftenfamiliarwithprogramsthatarebasedonnationalmodelsandcanprovideinsightintoaprogramsoriginandlineageeffectivenessinotherplacesandthepotentialforimplementationinyourcityAcademicscanalsohelpidentifyindicatorsbasedongovernmentindicatorsorliteratureFinallyrelationshipsbetweenpolicymakersandresearchersprovidethemostbroadlyeffectiveroutetopullingresearchintopolicy
ResearchandevaluationAcademicexpertsmaysupportcitiesinevaluation-indeedevaluatingcityinitiativesisatraditionalmodelofcityuniversityrelationshipsSharingtheinformationyoucollectedintheGatherInformationphasecanhelpresearchersunderstandyourworkandgoalswhichinturnwillstrengthenthesupportthattheycanprovideDiscussingdetailsaboutyourinterventionandevaluationplanatthebeginningoftheprocessmeansthataresearchpartnercanhelpwiththeevaluationdesignanddatacollectionForinstancetheWilsonSheehanLabforEconomicOpportunitieswalkscitiesthroughconsideringthefollowingquestionsbeforepartneringonanevaluation
1 Isthereaclearpolicyprogrammodelwithobjectivescreeningcriteria
2 Isitpossibletoevaluatewhathappensbothwiththepolicyprogramandintheabsenceofthepolicyprogram
3 Arethereoneormoremeasurableoutcomesofinterest
4 Arethereenoughpeopleforrigorousquantitativeanalysis
5 Whatdatawillbenecessaryforthestudyandwhataretheprocessesforaccessingthesedatasource(s)
6 Isthereacompellingreasontoevaluate
Strategicallyengagekeypartners
19
InadditiontoassessingreadinesstoconductanevaluationresearcherscanhelpyouhoneinonthepurposeoftheevaluationTheWorldBankdescribesseveralpotentialusesofevaluationwhicharedefinedhereInbriefevaluationcanhelpdecisionmakers
makeresourceallocationdecisionsrethinkrootcausesofproblemsidentifynewproblemsdecideonabestalternativedeterminewhetherprogramsorpoliciesarebeingimplementedwithfidelitydeterminewhetherprogramsorpoliciesarehavingthedesiredimpact
Researcherscanalsoprovidesupportinsharingtheresultswiththescientificcommunitythroughpapersandconferenceswhichisvaluableincreatingapeernetworkofevidence-basedcitypractitioners
TrainingandtechnicalassistanceUniversitypartnersespeciallythoseatpracticecenterscanbehelpfulindevelopingorupdatingtrainingcurriculumtrainingthetraineranddeliveringtrainingtocitystaff
Strategicallyengagekeypartners
20
ReviewtheevidenceandcollectdetailsOnceyouhaveathoroughunderstandingoftheproblemdataandactorsyouwillneedtoconductyourownresearchonavailableevidenceandlearnaboutthelandscapeofpolicyoptionsYourgoalwillbetodevelopanaccessiblelistofinformationaboutpoliciesthathavehelpedsolvesimilarproblemsinotherplacesThischapterwillguideyouthroughthestepsofpreparingyourreviewersunderstandinglevelsofevidenceandcollectingthedetailsonevidencethatyouwillneedfordiscussionwithdecisionmakersinyourcity
Selectandprepareyourreviewers
AssignapersonorteamtoreviewrelevantinterventionsthathavedemonstratedsuccessinotherjurisdictionsTheirgoalwillbetoproduceareviewofalloftherelevantprogramspoliciesorinitiativesthatyourteamcouldconsiderimplementingTheywillbeconductingaliteraturereviewsupplementedbyinterviewswithexpertsandpeerswhohaveinformationaboutpotentialprograms
SincethepeopleyouassigntothistaskareprovidingthebaselineofpotentialoptionstoimplementitisimportantthatyouchoosepeopleyouareconfidentwillbethoughtfulandthoroughintheirworkItisalsocriticalthattheyareawareoftheimportanceofsourcecredibilityEnsureyourreviewershavefamiliarizedthemselveswithrulesforcriticallyanalyzinginformationsourcesbeforetheygetstartedIftheyareincludingnewsarticlesaspartoftheirsearchtheyshouldalsobeawareofmethodsforevaluatingnewscredibility
AskyourreviewerstodevelopaplanandtimelinetocompletethereviewThisstartswithsolidifyingagoverningresearchquestionOnewaytoknowyouarecreatingadetailedresearchquestionistousetheacronymPICOremindingyoutoincludereferencetothepopulationintervention(ifyouhaveoneinmind)comparisonandoutcomeelementsinyourresearchquestionForexampleifyouknowthatyouhaveapopulationofteenagerswhoarehomelessyouhavelearnedfromyourstakeholdersthatmanyoftheseteenagersidentifyasLGBTandyouhaveheardsomethingabouttheapproachofcreatingdrop-incentersfordeliveringsupportyoucouldask
ForLGBTyouthfacinghomelessness(population)howeffectivearedrop-incenters(intervention)relativetostandardsocialservices(comparison)atsettlingtheminpermanenthousing(outcome)
Ifyoudonothaveaninterventioninmindyoucanleaveouttheinterventionandcomparisonandsimplyask
Reviewtheevidenceandcollectdetails
21
HowcanwehelpLGBTyouthfacinghomelessness(population)obtainpermanenthousing(outcome)
OnceyourreviewershavetheirresearchquestionsusethequestionstocreatealistofkeywordstheycanuseintheirliteraturesearchThiswillhelpkeeptheircollectionofarticlesreasonablyfocusedHavethemdescribewhattheirsearchprocesswillbeincludinglistingthedatabasestheywillreviewandtheirplanforstoringtheinformationtheywillcollect(Herearesomeadditionalsuggestionsforstructuringareviewplan)
WhatyourreviewerswillbedoingisalightweightversionofarapidreviewTotheextentthattheywouldliketodevelopamethodologyforspecificsearchprotocolsincludingcriticalappraisalprocessestheycancertainlygoaheadanddoso
Usedifferentlevelsofevidenceforthemostcompletepicture
AsdiscussedintheWorkingwitharangeofevidencechaptertheNestaStandardsofEvidenceframework--reproducedbelow--ishelpfultounderstandthelevelofconfidencethataprogramorinterventionishavingapositiveimpactThisapproachletsyouputinterventionsthatyouhaveheardaboutlocallybutwhichhavenotbeenstudiedtogetherwiththosewithademonstratedcausaleffectThisishelpfulsinceanumberofpoliciesprogramsandpracticesthatyouwillwanttoconsidermaynotyethavemuchofaresearchrecord
Reviewtheevidenceandcollectdetails
22
NearlyanypieceofinformationaboutaninterventioncanbeclassifiedwithinthestandardsofevidenceForexampleifsomeonehasacrediblehypothesisabouthowaninterventioncouldproducebenefitthistypeofevidencewouldbeclassifiedasLevel1ThistypeofevidencecouldincludeinterventionsinothercitiesthatyoureadaboutinthenewsorhearaboutataconferenceForinstancewhenUSDAallowedschoolsandschooldistrictstoprovidefreebreakfastandlunchtoallstudentsthroughtheCommunityEligibilityProvision(CEP)manydistrictsimplementedthisoptionevenbeforetherewasevidenceofeffectivenessThiswouldfitintoLevel1sinceitbuildsonthelogicthatprovidingaccesstomealswilldecreasefoodinsecurity
YoumayalsofindpiecesofinformationthatqualifyasLevel2evidenceForinstanceasmoreschooldistrictsimplementedCEPprincipalstestifiedaboutpositivechangestheyobservedinstudentssuchasamorepositivedemeanorthatcanleadtobetteracademicperformanceDataalsoshowedthatthestudentsdescribedwerereceivingmoremealsthanbeforetheprogramsimplementationHoweveritwastoosoontobuildevidencetoshowacausalrelationshipbetweentheincreaseinfreemealsandincreasedacademicperformance
ldquoLower-levelrdquoevidencecanconsistofnuancedbutvaluableelementsthatrelatetoaninterventionrsquospotentialfitwithacommunityQualitativeinformationsuchastestimonyfrompeoplewhohaveexperiencedaninterventionandthosewhohavefacilitatedorimplementeditwillalwaysbeusefultoconsiderandprovidesdetailedlocalcontext
Experimentalstudies--describedstartingatLevel3withcontrolledexperimentsandgoingthroughLevel5withfullyfleshedoutprotocolsandprocessesforreplication--stilloccupythehigherlevelsofthisdescriptionofevidenceThisprioritizationshowsthevalueofrigorousstudyforevaluatingtheimpactofaparticularpolicyorprogramratherthanotherfactorsthatcouldbeinfluencingoutcomesofinterestRememberthatmultipletypesofevidencecancomplementrigorousimpactevaluationsandgivecityleadersadditionalinsightsandbroaderperspective
Accessinformation
Encourageyourreviewerstobegintheirsearchforpromisingpracticesingovernmentandnon-profitclearinghousesandinacademicpolicyjournalsWorkpublishedinthesesourceshasalreadybeensubjectedtoahighlevelofprofessionalscrutinyandgenerallycontainsagreatdealofdetailwhichmeansthatyourreviewersaremorelikelytobeabletopullouttheinformationyouneedtomakeagooddecision
Clearinghouses
Reviewtheevidenceandcollectdetails
23
Herearesomeoftheclearinghouses(collectionsofprogramevaluations)forevidence-basedpoliciesandprograms
ResultsFirstClearinghouseDatabaseThePewCharitableTrustsCollectstogethereightotherclearinghousestoprovideinformationoninterventionsinthefollowingcategoriesadultcriminaljusticeeducationchildwelfarehealthjuvenilejusticementalhealthsubstanceabuseandyouthdevelopmentIndividualclearinghousesincludedinResultsFirstare
BlueprintsforHealthyYouthDevelopment-foryouthdevelopmentandviolenceprevention
CaliforniaEvidence-BasedClearinghouse-forchildwelfareinterventions
TheLauraandJohnArnoldFoundationrsquosSocialProgramsThatWork-forsocialservicesprograms
TheUSDepartmentofEducationrsquosWhatWorksClearinghouse-foreducationinterventions
TheUSDepartmentofHealthandHumanServicesrsquoResearch-TestedInterventionPrograms(RTIPs)-forcancercontrolinterventions
TheUSDepartmentofHealthandHumanServicesrsquoTeenPregnancyPreventionEvidenceReview(TPPEvidenceReview)-forsexualriskbehaviorandsexualhealthoutcomes
TheUSDepartmentofJusticersquosCrimeSolutionsgov-forcriminaljusticejuvenilejusticeandcrimevictimservices
TheUSSubstanceAbuseandMentalHealthServicesAdministrationrsquosNationalRegistryofEvidence-basedProgramsandPractices(NREPP)-substanceuseandmentalhealthinterventions
TheUniversityofWisconsinPopulationHealthInstituteandRobertWoodJohnsonFoundationrsquosCountyHealthRankingsandRoadmapsWhatWorksforHealth(WhatWorksforHealth)-healthbehaviorsclinicalcaresocialandeconomicfactorsandthephysicalenvironment
UKWhatWorksCentresTheBritishgovernmenthascharteredanetworkofsubject-specificorganizationswhichcollectsynthesizeandreportoutevidenceontheeffectivenessofpolicyprogramsandpractices
NationalInstituteforHealthandCareExcellence(NICE)-healthandsocialcare
Reviewtheevidenceandcollectdetails
24
SuttonTrustEducationalEndowmentFoundation-educationalachievement
CollegeofPolicingWhatWorksCentreforCrimeReduction-crimereduction
EarlyInterventionFoundation-earlyintervention
WhatWorksCentreforLocalEconomicGrowth-localeconomicgrowth
CentreforAgeingBetter-qualityoflifeforolderpeople
WhatWorksCentreforWellbeing-wellbeing
WilsonSheehanLabforEconomicOpportunities(LEO)UniversityofNotreDameProvideslinkstotheirevaluationsofinterventionsincriminaljusticeself-sufficiencyeducationhealthandhousingandhomelessness
ClearinghouseforLaborEvaluationandResearchUSDepartmentofLaborProvidesresearchonworkforceandlabor-relatedpoliciesandprograms
J-PALEvaluationsAbdulLatifJameelPovertyActionLabStudiesonpoverty-reducinginterventionsfromaroundtheworld
CampbellSystematicReviewsCampbellCollaborationSystematicreviewofpolicyinterventionsbyscholarlycoordinatinggroupsinsevenissueareascrimeandjusticeeducationinternationaldevelopmentmethodssocialwelfareknowledgetranslationandimplementationanddisability
Benefit-CostResultsWashingtonStateInstituteforPublicPolicyAWashingtonState-charteredprogramtoestablishthelikelihoodofrelativecostsandbenefitsforprogramsacrossthefollowingareasjuvenilejusticeadultcriminaljusticechildwelfarepre-Kto12educationchildrensmentalhealthhealthcaresubstanceusedisordersadultmentalhealthpublichealthandPreventionworkforcedevelopmentandhighereducation
EffectiveInterventionsCentersforDiseaseControlandPreventionRepositoryforevidence-informedHIVpreventioncareinterventions(withstepsonhowtoimplement)
Journalsandindexes
OnceyouhavereviewedrelevantinformationaccessiblethroughclearinghousesitisworthtakingalookatindividualstudiespublishedasscholarlyarticlesYourreviewerswilldothisbysearchingarticleindexesandacademicjournalsUnfortunatelyanumberofusefulacademicjournalsandindexingplatformsmaynotbegenerallyaccessibletoyourreviewersifyoudonothaveasubscriptiontothemToaddressthisconsiderpartneringwithalocal
Reviewtheevidenceandcollectdetails
25
collegeoruniversitytoprovideyouwithalistofarticlesfromthefollowingsources(Thisisalsoanexcellentopportunitytoseekresearchsupportfromanyacademicstakeholdersyouidentifiedpreviously)
Indexes
GoogleScholarincludesmostpeer-reviewedonlineacademicjournalsandbooksconferencepapersthesesanddissertationspreprintsabstractstechnicalreportsandotherscholarlyliteratureincludingcourtopinionsandpatentsItallowsyoutospecifyapublicationrangeinyoursearchandprovideslinkstolikelysourcesforalllistedarticles
PolicyFileIndexisauniqueresourceforUSpublicpolicyresearchinthatitgrantsusersaccesstotimelyupdatedinformationfromover350publicpolicythinktanksnongovernmentalorganizationsresearchinstitutesuniversitycentersadvocacygroupsandotherentities
PolicyArchiveworkstocollectanddisseminatesummariesandfulltextsvideosreportsbriefsandmultimediamaterialofthinktankuniversitygovernmentandfoundation-fundedpolicyresearch
Anothersourceforhigh-qualityresearchisthereportsoftheCongressionalResearchServiceYoucanfindtheirreportsaggregatedhere
Specificjournals
Oftenthemostusefulacademicjournalsforunearthingpolicyorprogramreviewswillbeineachspecificsubjectarea--thatisineducationjournalsforeducationpoliciesYoucanuseageneralindexlikeGoogleScholartobegintolearnwhichjournalsaremostoftenusefulforyoursearchHoweveryoucanalsosearchthefollowingtwogeneralistpolicypublicationsforissuesandarticlesrelatedtoyourtopic
PublicAdministrationReviewJournalofPolicyAnalysisandManagement
Newssourcessubjectmatterexpertsandinterviews
AlthoughitisvaluabletoknowwhathasbeenexploredthroughascholarlyresearchframeworkitisalsousefultolearnwhatkindofinterventionsarebeingtriedinthefieldrightnowandhowpeoplefeelaboutthesetrialsSearchsubscriptionnewsaggregatorslikeLexisNexisorfreenewsaggregatorslikeGoogleNewsorYahooNewsforyourissueareaplusthewordpolicyorprogramtoseewhatkindsofinterventionsarebeingcoveredinthenews
Reviewtheevidenceandcollectdetails
26
DonotforgettocontacttheadvisorygroupmemberswhowerechosenfortheirsubjectmatterexpertiseInadditiontoprovidinggoodmaterialthroughinterviewsthemselvestheymayhaveideasaboutothersubjectmatterexpertsyoushouldcontacttoaskaboutpotentialinterventions
Peercitiesandnetworks
InadditiontoreadingabouttheworkofothercitiesitisidealifyouareabletospeakwithpeoplewhoareinvolvedwithcurrentimplementationsofinterestinginterventionsInyourcityyoulikelyhaverelationshipswithyourneighborsbutitcanbehelpfultolookoutsideofyourregionforsolutionsaswellTofindcounterpartsinothercitieswhoareworkingonaprogramyouwouldliketolearnmoreaboutyoucancontactstaffinintermediaryorganizations(likeGovExoryourprofessionalassociations)whomightbeinapositiontoputyouintouchimmediatelyorwhomightseeanothergoodrouteforsettingupaconversationHereisalistofnetworksthatmaybehelpful
InternationalCityCountyManagementAssociation(ICMA)NationalAssociationofCounties(NACO)NationalLeagueofCitiesUSConferenceofMayorsWhatWorksCities
Collectthedetails
Onceyourreviewershavefoundevidenceonpossiblepoliciesorprogramstoimplementtheyshouldcapturekeydetailsthatwillhelpdecisionmakersunderstandthepotentialeffectivenessandapplicabilityofeachpotentialintervention
Foreachsourcethathasprovidedinformationyouwanttoconveyusethetemplatebelowtocaptureasmanyofthefollowingdetailsaspossible(Youdonotneedtocompletethisforsourcesthatprovideduplicativeinformationorareotherwisenotthatuseful)
Programorpolicyname (egldquoHousingFirstrdquoorldquoblightedpropertiesregistryrdquo)
Sourcetype (egjournalarticleinterviewsubjectmatterexpertstudyassociationcommunityparticipationprocess)
Sourcedetails (egarticletitleandpublicationnameandtitleofinterviewee)
Evaluationmethodology
(egrandomizedevaluationcomparativeevaluationoftwodifferentgroupsbeforeafteranalysis)
Evidencelevel (egLevel12345)
Programorpolicy
Reviewtheevidenceandcollectdetails
27
description
Programorpolicygoal (1sentencecapturingtheprincipalaimoftheintervention)
Existingimplementations
(WherehastheinterventionbeentriedListasmanysitesasyouareawareof)
Characteristicsofimplementationsites
(ForeachimplementationsiteofferbasicgeographicIDmdashcityandstateThenyouwillwanttocollectthedetailswhicharerelevanttoyourdecisionmakersrsquoconsiderationsaboutwhatmakesalocationldquosimilartordquoorldquodifferentfromrdquoyoursituationYouwillatleastwanttoincludeforeachimplementationsitePopulationnumberPopulationdensity(ruralurbansuburban)Socio-economiccharacteristicsBeforestartingaskyourdecisionmakersaboutanyparticularcharacteristicsofprogramimplementationsitestheywouldwanttoknowbeforeseekingtoreplicate)
Characteristicsofimplementationpartners
(Anyavailabledetailsaboutorganizationsinstitutionsorpeoplewhowereinvolvedwithimplementingtheintervention-egindependentchildcarecenterpublicelementaryschoolcitypublichealthnurses)
Characteristicsoftargetpopulationeligibleprogramparticipants
(Anyavailabledetailsaboutpeoplewhoarethebeneficiariesoftheprogramorotherwiseincludedoraffectedbytheintervention-egnumberofprogramparticipantscategoriesofpropertyownersaffected)
Policyorprogramduration
(HowlonghastheinterventionbeeninplaceineachimplementationsiteHasitendedandifsowhy)
Whatbenefitscamefromeachimplementation
(Anyobservedpositiveeffectsoftheinterventionincludingeffectsize)
Statisticalestimatesofinterventioneffectiveness
(Anyavailablestatisticalestimatesoftheinterventionrsquoseffects)
Whatdrawbackswereobserved
(Anyavailabledetailsaboutunintendedorunavoidablenegativeeffectsoftheintervention)
Programorpolicycosts
(Anyavailabledetailsaboutprogramcostsperimplementation)
Reviewtheevidenceandcollectdetails
28
Reviewtheevidenceandcollectdetails
29
DecideonapolicytoimplementbasedonlocalfactorsTogetthecriticalbuy-infromleadershiptoimplementanewpolicyorprogramtheprojectleadshouldscheduleameetingwithdecisionmakerstopresenttheevidencesurroundingthepolicyoptionscapturedinthetemplateDependingonyourinsightintoyourcityleadershipswillingnesstomakechangeyoumaywishtosharetheresearchyouhaveconductedontherootcausesoftheproblemandsomeofyouranalysisontheimpactofcurrenteffortstodate
Considertheapplicabilityofanapproach
AsdiscussedinthepreviouschapterpolicymakersoftenlookforevidenceofeffectiveprogramsinothercitiesbutoftenthequestionarisesofhowgeneralizableaprogramisfromonecitytoanotherTheAbdulLatifJameelPovertyActionLab(J-PAL)suggeststhatinreviewingevidencefromanotherlocationitishelpfultoidentifythespecificmechanismthatwasused(deliveryofprogramorserviceinformationsharingetc)identifythelocalconditionsthatwereinplaceobservethebehaviorchangethatresultedfromthemechanismandfinallytounderstandlocalcontextandconditions(legalorcostconstraintsculturalnormsetc)Thissetofpresentationelementscanbedrawnfromthelistofdetailscollectedintheresearchtemplate
EducationpolicymakersinChicagoILusedfindingsfromprogramsinKenyaandIndiainwhichchildrenwhohadfallenbehindtheirpeersweregiveneducationalsupporttargetedtotheirleveloflearningThemechanismusedwastargetedteachingandthereweresimilarconditionspresentinallthreecountriesAlthoughsomestudentshadfallenbehindinthecurriculumteacherswereincentivizedtoteachgrade-levelmaterialsConsiderationsofhowthelocalcontextinChicagomightaffectimplementationofatargetedteachingprogramincludedunderstandinghowtheneighborhoodsthatthestudentswerefrommayhaveaffectedtheireducationaloutcomesandthehighcostofteachinginsmallgroupsInChicagoMatchEducationprovidedtwo-on-onetutoringwhichmatchedstudentswithwell-educatedindividualswhohavenothadformalteachertrainingforalower-costmodelTheresultswerepositiveStudentswhoweretutoredperformedbetterinmathandweremorelikelytopasshighschoolclasses
ThegeneralizabilityframeworkproposedbyJ-PALisbelowUsethisasatemplateasyouconsiderpotentialpolicysolutions
Decideonapolicytoimplementbasedonlocalfactors
30
SourceTheGeneralizabilityPuzzle
OnceyouhavenarrowedthepolicyoptionsdiscussthequestionsbelowforeachoptionsyouareconsideringThisstepiscriticalasunderstandinglocalcontextissoimportantinsuccessfulpolicyimplementation
CommunityContextQuestionnaire
Localcontext
DoesthisoptionmakesenseinourcommunityIsthisaonesizefitsallapproachIsanyparticulargroupabsentfromthisapproachIfsoisthatok
Youranswers
Effectiveness
WhatwouldweexpecttoobserveifthepilotisaseffectivehereasinCityX
Youranswers
Timing
DoesthetimingmakesenseIsthistherighttimetoimplementthispolicychange
Decideonapolicytoimplementbasedonlocalfactors
31
WherearethepotentialroadblocksthatcouldaffectthescheduleIsthereanywaytomitigatethisHowlongwouldittaketoseeanimpact
Youranswers
Implementation
DowehavethestaffcapacitytoimplementthisArethereadditionalpartnersthatyoucanbringtothetabletosupportimplementationDoesthismakesensegivenourformofgovernment
Youranswers
Authority
DoesourcityhavetheauthoritytoenactthisIfsowhatistheappropriatemechanismAretherelegalbarriersArethereanyexistingpreemptionissuesfromthestateorfederalgovernmentCanthosebeaddressedthroughlegislationadvocacyIstherepoliticalsupportoroppositiontothis
Youranswers
Cost
HowmuchwillthiscostannuallyHowwillwepayforitCanthepolicybeimplementedwithexistingfunding
IfsoisthisfundingreliableHowwouldthispolicyoptionaffectourcitysbudgetintheshortandlongtermWhatotherfundingoptionscanweconsiderIsthereanalternativethatwouldcostlessandgiveussimilarresultsWillanyconstituencybeadverselyaffectediffundingforprogramsthatimpactthemisshiftedtoanotherprogrampolicyorinitiative
Youranswers
Decideonapolicytoimplementbasedonlocalfactors
32
EstablishaframeworktomonitorprogressNowthatyouhavereviewedtheevidenceconsideredlocalcontextandchosenyourpolicyitistimetodeterminehowyourcitywillmonitorprogressofthisnewapproachCitystaffandleadersneedtoestablishtheinfrastructureforongoingcontinuousdecisionmakingbasedonwhetherthepolicychangeisleadingtothedesiredoutcomesforresidentsandwhetherthechangeismeaningfulandrationaltofrontlineworkersSettingarbitrarygoalsatthelocallevelthatconflictwithgoalsatotherlevelsisconfusingtotheindividualsresponsibleforachievingthegoalsGoalsthataresettoohightoachievecanbedemoralizingtostaff
ThissectionincludesstepstosetupavenueinwhichCitystaffandleadershiproutinelyvisitkeyindicatorstodeterminewhatisorisnotworkingandtomakedecisionsinreal-timeFormorereadingonperformancemanagementpleasevisittheGovExPerformanceManagementGettingStartedGuide
InperformancemanagementtermssuchasindicatormeasureandmetricmaybeusedinterchangeablyInthisguidewewilluseindicatorInyourworkthemostimportantthingistobeconsistentinwhichevertermyouuseThisvideomaybehelpfulinunderstandingthisconceptandcommunicatingittoothers
Establishthefundamentalsandgatheryourteam
FirstdeterminewhetherthereisanestablishedvenueorregularmeetingscheduleduringwhichyoucandiscussprogressIfyourcityhasanexistingperformancemanagementstructureonarelatedtopicaddingadiscussionoftheprogressofpolicychangetoyouragendacanbeagreatideaIfthiswillbeastandalonemeetingdecidehowfrequentlytheprojectteamanddecisionmakerswillmeetandputthosemeetingsonthecalendarfortheentirelengthoftheproject
DeterminewhowillstaffandattendroutinemeetingsonCityprogressThesemeetingsmayincludetheMayorandorCityManagerdepartmentheadstaffwhoareimplementingthepolicychangeandanalysts
Identifyindicators
ItiscriticalthatindicatorsbeanimportantcityandcommunitypriorityaswellassomethingtheCityplaysamajorroleinaddressingTheremaybeinstancesinwhichtheCitywillneedtoreachouttoorcollaboratewithexternalpartnerstogetamorecompletepictureofprogressontheoverallpolicychange
Establishaframeworktomonitorprogress
33
Anoutcomeindicatordescribesthedesiredeffectofthetargetedpopulation
ExampleBellevuesfamilyfriendlyneighborhoodsindicatormeasuresthepercentofresidentswhobelievetheyliveinaneighborhoodthatsupportsfamiliesparticularlythosewithchildren
WithyourkeypartnersrevisityourproblemstatementcurrentstateandfuturestateexerciseandbegintoidentifytheoutcomeindicatorsthatwillillustratesuccessFornowchooseasingletop-levelkeyperformanceindicator(KPI)ItiscriticalthattheKPI-thepieceyouwillmeasure-beapriorityforthecommunityandsomethingtheCityplaysamajorroleinaddressing
Outputindicatorsmeasuretheprocessofprogramorservicedelivery
ExampleSanDiegoMayorKevinFaulconersgoaltorepair1000milesofstreetby2020
ReviewingtheadministrativedatasetinventorybegintoparedowntherelevantprogramsanddatasetsthatsupportyourKPIWhatarethemeasuresofeffectivenessforeachprogramTheseareyouroutputindicators
Thediagrambelowillustratestherelationshipbetweenoutputandoutcomeindicators
ThinkaboutimplementationindicatorsasmeasuringstepsintheprogramsrolloutthatarenecessarytoobtaindesiredchangesImplementationoutcomescanfocusontheinterventionstaffdeliveringtheinterventionanorganizationorsystemandcanaddressfeasibilityadaptabilitysustainabilityandcostTwokeyquestionstoconsiderwhendevelopingimplementationindicatorsare
1 Istheinterventionacceptableandappropriatetothecommunityand
2 IsitfeasibletoimplementinterventionXatthistimewiththeseresources
Establishaframeworktomonitorprogress
34
Amoreextensivelistwithdefinitionsofpotentialimplementationoutcomesislinkedhere
UsethetablebelowtochoosethedesiredimplementationoutcomesforyourpilotandtoplanyourmeasurementtechniqueThisshouldincludelevelofanalysis(individualproviderorconsumerorganizationorsetting)yourmeasurementtechnique(surveysobservationinterviewfocusgroupsadministrativedataanalysis)andthephasewhenyouwillcollectthisinformation(earlymidlate)Theremaybemorethanonelevelofanalysismeasurementtechniqueandphaseofdatacollection
Implementationoutcome
Definitionofoutcome
Levelofanalysisandmeasurement
Yourmeasurementtechnique
Phaseofpilotwhenyouwillcollectdata
Usethespacebelowtocaptureyourindicatorwishlist
Indicatorwishlist
Outcomes_____________________________________________________________________________
Outputs________________________________________________________________________________
Implementationoutcomes______________________________________________________________
Setapublichigh-levelgoal
Thehigh-levelgoaliscriticalinestablishingdirectiontogalvanizethepeoplewhowillbedoingtheworkWhendiscussingthegoalusetheMayorsbullypulpitLookforkeymilestonesinCitygovernmenttosharethegoalstatementsuchasthebeginningofaMayoraladministrationaStateoftheCityspeechorotherpublicremarks
GoalsshouldbeldquoSMARTrdquo
Specificaspossible(innamingthepopulationwhowillbebetterofftheactionthecitywilltake)
Measurable
Achievable
Establishaframeworktomonitorprogress
35
Results-focused
Time-bound
ThegoalstatementshouldincludeyouroutcomeindicatorlikeChattanoogasgoalldquoWewillendchronicveteranhomelessnessbyDecember2016rdquoAdiagramtohepyoubuildaSMARTgoalaroundyouroutcomeindicatorisbelow
Component Description Yourturn
Specific WhowillbebetteroffWhatactionthecitywilltake
Measurable Howwillyoumeasuresuccess
Achievable Isthegoalattainable
Results-focused Whatistheexpectedresult
Time-bound When
Setrealisticambitioustargetsandtimelinesforeachindicator
ToestablisharealisticandambitioustargetforyourgoalstatementusetheadministrativedatacollectedforyourchosenKPItoreviewactualCityperformanceinthemostrecentyearcomparedwiththethreeorfourmostrecentyearsworthofdataBegintocalculatesummarystatistics(egtotalminimumaveragemedianmaximum)toidentifytrendsCreatecumulativeandmonthlychartstoidentifytrendsandanomaliesBasedonyouranalysisofhistoricaldataignoretheanomaliesandassumetheproductivitytrendswillfollowthesameoverallpatternfrommonth-to-monthFinallyapplyanincreasedproductivityfactortotheperformanceeachmonthtoforecastfutureprogressBasedonthisanalysischooseatargetthatinspiresimprovementwhileaccountingforreality
Manycitiesthataremorematureintheirperformancemanagementpracticesrecognizetheimportanceoftakingacommunity-levelapproachtoimprovingoutcomesforresidentsThesecitiesarecreatinggoaltaxonomiesthatincludemeasuresrelatedtopublicsafetyhousingandevenrecreationCitiesdriveprogressonthesegoalsbyimplementingstrategiesthatengagemultipledepartmentsandcollaboratewithserviceprovidersinthecommunity
EventuallyyouwilldevelopgoalstatementsfortheoutputsinordertocapturetheefforttoachievetheoutcomecontainedinyourgoalstatementYoursetofindicatorsshouldbecomposedofshort-andinterim-termsupportivemeasureswithaccuratetargetsandrelevanttimelinesAnexamplegoaltaxonomyfromtheCityofLasVegassResultsVegasprogramisbelowUsethistaxonomyasamodelforyourownsuiteofindicatorsandgoalstatements
Establishaframeworktomonitorprogress
36
Yourturn
Thetemplatebelowismeanttohelpyoucraftagoalstatementforeachsupportingmeasure
Goalstatement
Departments
Outcome
Activities
KPI
Target
Timeframe
Conductanddiscussrelevantanalysis
ThrougharoutineseriesofmeetingscontinuallymonitorprogressofKPIssupportingindicatorsandanyotherdatacollectedsuchasqualitativedatatoidentifytrendsthatwillindicatewhetherthechosenpolicyishavingthedesiredimpactThesedatapointsshoulddrivethediscussionduringthesemeetingspointingtoopportunitiesforfurtherinquiryinto
Establishaframeworktomonitorprogress
37
whatisdrivingtheobservedtrendsAnalystsshouldbeencouragedandtrainedtoaskquestionsofthedepartmentsproducingthedataandconductsitevisitswhenpossibletodeepentheirunderstandingofthedataPresentationoftheanalysisshouldfocusononeideaatatimetokeeptheconversationfocusedonkeypointsandspecificopportunitiesforimprovementYoumayconsiderdevelopingapublicfacingdashboardtosharethisprogressmorebroadly
AgeneralcomparisonofthestatusquodatawithwhatwasobservedduringthepilotcanprovidedecisionmakerswithaquickunderstandingofthedifferencesbetweentheoldandnewapproachesalthoughthiswillnotshowcausalityOftenthesecomparisonswillincludeoperationalmetricssuchasdaysordollarssavedbutrememberthattheimpactonpeoplewillbemostimportantIfyouareworkingwithanevaluationpartnertheywillsupportthisstep
Usethespacebelowtosummarizetheresultsofyouranalysis
MeasureTargetvalue(seesectiononsettingtargetsabove)
Pilotvalue(whatdidyouobserveduringthepilot)
Historicalaverage(ifavailable)
Results(WhoisbetteroffIsanyonenegativelyimpacted1-2sentencesreassessingyourgoalstatement)
DocumentyourprocessandprogressincludingfeedbackfromtheparticipantsimplementationpartnersandtheprojectteamDidanythingunexpectedoccurduringthepilotArethereinternalandexternalfactorsthatmayaffectyouroutcomesThisiswheretheimplementationoutcomescanbecanhelpfulIftheinterventiondidnotgoasplannedconsiderwhetheritisbecausetheinterventionwasnotacceptableorfeasible
UsethetemplatebelowcreatedbyDenversPeakAcademytocapturethisinformationIfthe306090daytimelinedoesnotmakesenseforyourcityfeelfreetocustomizeit
CurrentState
FutureState
30Day
60Day
90day
Whatwentwell(+)
Whatdidnotgowellshouldbechanged(-)
IfaquestionordiscussionpointarisesthatcannotbeansweredinthemeetinganassignedstaffmembershouldcapturethequestionduringthemeetingAfterwardthefollowupitemsshouldbesenttorelevantpartnersandthenextmeetingshouldbeginwithadiscussionof
Establishaframeworktomonitorprogress
38
thoseitems
Takeaction
DiscussionsaboutwhetheryourindicatorsaremovingintherightdirectionandwhatmightbedrivingtrendsmayleadtoadiscoverythattherearenotenoughresourcesallocatedtothispolicyorissueTheremaybelimitedfundingforstaffprogrammingortechnologythatisdrivingthedatainthewrongdirectionIfCityleadershipisintheroomthisisagreattimetomakethecaseforadditionalfundingduringthebudgetseasonBycollectinganalyzinganddiscussingdataonthispolicyroutinelyyouarebuildingevidencetomakethecasefortherightleveloffundingtoreachtheestablishedtargets
Establishaframeworktomonitorprogress
39
Assessdataaccessibilityqualityandsecurity
Determineaccessibilityandqualityofdata
OnceyouhavecompiledyourwishlistofindicatorsitistimetodeterminetheaccessibilityandqualityofthedataTopreventcomplicationyoumaywishtoprioritizerelevantindicatorsforwhichyouarealreadycollectingdataAvoidthetemptationofadoptingatechnologysolutiontocollectnewdataatthispointsuchasaphotosurveytoolorotherappFornowyouaretryingtodeterminewhichdatasetsarerelevantaccessibleandhighqualityBesuretoworkwithyourkeyinternalpartnerswhileyouareidentifyingyourmeasuresDepartmentstaffwhoworkonprogramsarefamiliarwiththeindicatorsthatwillaccuratelysupporttheoutcomesanddatabaseadministratorswillbevaluablewhendiscussingaccessibilityandqualityofthedata
Abrieflistofconsiderationsfortheaccessibilityofyourdataisbelow
ArewecurrentlycollectingthisdatasetorwillwebecollectingitforthefirsttimeHowfarbackdoestherecordgo(timehorizon)WouldweneednewtechnologytocollectoranalyzethedataCanthedatabecollectedmanuallyorcanitbeautomatedHowfrequentlyisthedataproduced(continuouslyhourlydailyweeklymonthlyquarterlyyearlyetc)IsthedataavailablefortheentireCitypopulationorasubsetofthepopulationHowgranularisthedata(individuallevelCensusblockneighborhoodcitywideetc)DoesthedatacontainsensitiveinformationIsthereaproxymeasurethatwecanusefornowIsthedatamachinereadable
AsnotedbytheWilsonSheehanLabforEconomicOpportunities(LEO)attheUniversityofNotreDameleveragingadministrativedatacansignificantlyreducethecostsandtimelinesofrigorousevaluationCitiescanestablishaninternaldatasharingagreementacrossdepartmentsincludinglanguagethatenablesadministrativedatatobeusedforresearchpurposesCitiescanalsoidentifyexternaladministrativedata-holderssuchasacountyschooldistrictpolicedepartmenthospitaloranotherstatethatmayhaveoutcomedataimportantformeasuringprogramimpactCitiescanleverageexistingrelationshipsandorworkwithresearchinstitutionstodevelopstreamlineddatasharingprocessesandagreementswiththeseexternalentities
Assessdataaccessibilityqualityandsecurity
40
Evaluatethequalityofyourdata
QualityisasimportantasaccessibilityToevaluatedataqualityyoumaywishtocreatearubricliketheonebelow
Value ValueDescription
3 ExcellentDataQualityThedataiscompleteaccurateandupdatedregularly
2 AdequateDataQualityThedataissomewhatcompleteaccurateandupdatedfairlyregularly
1 PoorDataQualityThedataisnotcompleteaccurateorupdatedregularly
ThisisagreattimetobuildonthedatainventorythatyoustartedinthegatherinformationpageSuggestedadditionalfieldstoaddtoyourdatainventorytemplatearebelow
Program DatasetTitle
DatasetDescription DataSource
Departmentorexternalpartner
DoestheCityalreadycollectthisdata
Whatisthenameoftheprogram
Human-readablenameofthedataset
Shouldbeunderstandablebynon-technicalusers
Whatistheoriginalsystemapplicationfilethathousesit
Departmentorexternalpartnerthatmaintainsthedata
Isthisdataalreadycollected
Updatefrequency-ThefileformatforthedatasetTypicallycsvxlsxshapefiletxtetc
Format-Howoftenthedatasetisupdated
Timerange-Theperiodoftimecoveredinthisparticulardataset
Datause-Howisthisdatasetcommonlyused
Datausers-Whotypicallyusesthisdataset
Dataownerstitle-Titleofthedataowner
Dataownersposition-DataOwnersjobtitle
Dataownersemailaddress-DataOwnersemailaddressWhomanagesthedataandorisresponsibleforgrantingpermissiontoaccessthedataWhounderstandswhatthedatasetincludesandcananswerquestionsaboutit
Dataqualityconcerns-Describetheconcernsyoumayhaveaboutthequalityofthedataset
Assessdataaccessibilityqualityandsecurity
41
Division-DivisionwherethedataismaintainedIfmorethanonedivisionisresponsiblelisttheprimaryownerofthedataset
Sensitivedatacomments-DescribetheissuesthatwouldpreventthecityfromsharingthedatapubliclyCommonsensitivityconcernsincludeprivacyviolationssecurityissuesorhighcostorstaffdemandstopostthedata
Publishingstatus-Thistheplacetoputanycommentsregardingthecurrentstatusofthedataset
Publishedlink-Thisisthelinktothepublisheddatasetontheopendataportalorcitywebsite
Reviewdatasecurityandprivacyprotectionprotocols
OftenapolicychangealsoinvolvesachangeinthedatathatisbeingcollectedhowthatdataisbeinghandledwhohasaccesstoitandahostofotheradjustmentsAsyoutakestockofthedataassetsthatareavailableforyourinitiativeitisvitaltoalsothinkthroughwhatwillbedonetoprotectthatdataandtheprivacyofthepeopleitrepresentsAbigpartofgooddatasecurityinvolvesstrongdatagovernancepracticesthatdefineandidentifysensitivedataandconsciouslyworktobalancethevalueandriskofstoringsuchsensitivedataOnewayofevaluatingyourdatacomesfromcaselawrelatedtotheUSFreedomofInformationActwhichwehavetranslatedtothechecklistbelow
1 Woulddisclosureresultinasubstantialinvasionofprivacy2 Haveweconsideredtheextentorvalueofthepublicinterestandthepurposeorobject
oftheindividualsseekingdisclosure3 Istheinformationavailablefromothersources4 Wastheinformationgivenwithanexpectationofconfidentiality5 Isitpossibletomouldreliefsoastolimittheinvasionofindividualprivacy
TheseriskscouldbemitigatedusingavarietyofstrategiesfromaggregatingdatatoobfuscatingindividualstoredactingfieldsthatmayincludePersonallyIdentifiableInformation(PII)orre-evaluatingyourdatacollectionmethodstofindalternativewaysofgettingtheinformationyouneedThesestepsshouldbecodifiedandmanagedbyadatagovernancecommitteeifpossibleForthesensitivedatathatisdeemedessentialcreateaplanaroundpotentialdatabreachesinordertoquicklyandconfidentlyrespondtoemergencysituations
ProactivelypreparingforthesesituationsisoftenthebeststrategyfordatasecuritybutdonotforgetthetechnicalsideInvolveyourITadministratorsinconversationsaboutdatastorageandaccessensuringthattheywillbeprotectedbyenterprise-gradecybersecuritysystems
Assessdataaccessibilityqualityandsecurity
42
Assessdataaccessibilityqualityandsecurity
43
Designimplementationandevaluation
Designtheimplementation
NowthatyouhaveyouraccountabilityplanandkeyindicatorsitistimetogetspecificaboutyourplanforimplementationThetemplatebelowwillhelpyouthinkthroughthespecificmechanismsofimplementingpolicychange
UnderstandingorganizationsandsystemsiscriticaltounderstandingandmakingdecisionsaboutwhereandhowinterventionsshouldbeappliedApolicyimplementedinasinglesettingmayshowpromisingresultsbutimplementingmultiplepoliciesandprogramsatthesametimewithdifferentactorsresponsibleforeachaspectcanhelpensuresustainablebehaviorchange
InSomervilleMAShapeUpSomerville(SUS)isacitywidepublichealthprogramthatisimplementedacrossmultiplelevelsofinfluenceandsystemsthroughinterventionsbeforeduringandafterschooltoincludechangestobuiltenvironmentsuchasbikelanesregionalmasstransportationandphysicaleducationequipmentinschoolsandgymstoincreasephysicalactivitywellnesspoliciesandschoolgardensnutritioneducationduringandafterschoolandhealthyrestaurantprogramstosupporthealthyfoodandhealthyeatinghabits
Usethetemplatebelowtoplanyourimplementation
Policyimplementationplanningtemplate
ListthespecificdesiredoutcomesInthepreviousstepyouworkedwithdepartmentstafftoestablishspecificmeasurableoutcomesandinterimindicatorsPrioritizetheindicatorsthatwilldemonstrateprogressUsethespacebelowtocapturethedesiredoutcomesofthepolicychange
ExampleShapeupSomervillebeganwiththegoalofpreventingobesityin1stthrough3rdgraders
Yourturn
____________________________________________________________________________
____________________________________________________________________________
Designimplementationandevaluation
44
SpecifythetargetpopulationAreyoudesigninganinterventionthattargetsonlycertainsitesAreyouplanningonimplementingaprogramorinitiativeforaspecifictargetaudienceClearlyspecifythepopulationyouplantotarget
IfyouplantoconductarigorousimpactevaluationasnotedbytheWilsonSheehanLabforEconomicOpportunitiesitisimportanttoestablishconcreteobjectiveeligibilitycriteriaandensurethatthesecriteriaareappliedconsistentlybyallstaffduringscreeningandintakeItisalsovitaltoidentifytheindividualswhoareeligibleforaparticularprogramorserviceevenifeveryoneeligiblecannotbeservedbyexistingresourcesThishelpscitiesidentifyanyunmetneedsandguidefutureresourceallocationwhilealsocreatingthepossibilityofarigorousimpactevaluationbyidentifyingapotentialcomparisongroupFinallywhenconductingscreeningandintakeitisimperativetocollectandmaintainbasicinformation(egnamedateofbirthgender)aboutallprogram-eligibleindividualsnotjustthosewhoparticipateintheprogram
ExampleIntheShapeUpSomervilleexampletheprogrambeganbytargeting1stthrough3rdgraders
Yourturn
____________________________________________________________________________
____________________________________________________________________________
DefinethelocationWherewillthepolicychangetakeplaceWillimplementationtakeplaceatonesiteormultiplesites
ExampleShapeUpSomervilleinterventionsweredeliveredinschools
Yourturn
____________________________________________________________________________
____________________________________________________________________________
ClarifyimplementationpartnersandarticulatetheirrolesWhatarethekeyorganizationsinstitutionsorpeopleresponsibleforimplementationTheprojectteamshouldbeasclearandspecificaspossibleinprovidingadescriptionoftheconcreteactivitiesthatimplementationpartnersaremeanttocarryoutPracticeprofilesmaybeahelpfultooltohelpdefinewhatisrequiredoftheactorswhoareresponsibleforimplementingtheintervention
Designimplementationandevaluation
45
ExampleInSomervillemayoralleadershipandengagementofmultiplecityagencies(publichealthprimarycareschoolsCambridgeHealthAllianceetc)streamlinedandconnectedthevariousShapeUpSomervilleinitiativesandgrantswhichhelpedtokeepthetalentandenergyfocusedonhealthyeatingandactivelivingforallsectorsofSomervilleSUSpartnersultimatelyincludedschoolsnonprofitsCityandstateagenciesandnationalfoundations
Yourturn
____________________________________________________________________________
____________________________________________________________________________
DescribetheinterventionWhatarethestepsactivitiesorprocessesRememberthatthemechanismthespecificstepsorprocessthatleadtobehaviorchangeiskeytounderstandingtheeffectivenessofthepolicychangeandsharingtheresultswithothers
ExampleGoalsofShapeUpSomervilleweretoinfluenceeverypartofanearlyelementaryschoolchildrsquosdayndashbeforeduringandafter-schooltopromotehealthyeatingphysicalactivityandtheirweightstatusInterventionincludedfoodserviceenhancementstoimprovebreakfastandlunchandstaffprofessionaldevelopmentwalktoschoolactivitiesSUSclassroomcurriculumandprofessionaldevelopmentschoolwellnesspolicydevelopmentSUSafter-schoolcurriculumandprofessionaldevelopmentoutreachandeducationtohomethroughmaterialsforumseventsldquoSUSapprovedrdquorestaurantscommunityoutreachandcapacitybuildingthroughpolicydevelopmenttrainingsmediaplacements
Yourturn
____________________________________________________________________________
____________________________________________________________________________
ClarifytimelineandmilestonesfordeliveryandevaluationHowfrequentlywilltheservicesinterventionbedeliveredHowlongwilltheprogramrunWhatisthetimetoexpectedimpacts(eg12monthsafterprogramcompletion)
Designimplementationandevaluation
46
ExampleShapeUpSomervilleledbyTuftsUniversityFriedmanSchoolofNutritionScienceandPolicyconductedaquasi-experimental(nonrandomized)interventionstudyamong1stthrough3rdgradestudentsinSomerville(intervention)andtwocomparisoncommunitiesfor9-monthsduringoneschoolyear(fall2002ndashspring2003)ThestudyshowedpositiveassociationoftheinterventiononchangeinBMISUSreleasedreportssummarizingevaluationsprogresstodateplansforadditionalprojectsandimpactstenandfifteenyearsintotheinitiative
Yourturn
____________________________________________________________________________
____________________________________________________________________________
PlanyouranalysisTheprevioussectionaddressedestablishinganaccountabilityframeworktomonitorprogressandimplementationDeterminewhichindicatorswillbediscussthroughthatprocesswhowillsharethedataandanalysisandhowfeedbackfromthoseconversationswillbeincorporatedbytheprojectteam
ExampleTheMayorrsquosofficeimplementedadata-drivendecisionmakingapproachcalledSomerStatwhichwassynergisticwiththeapproachestakenbytheresearchandevaluationteamsdeployedforthespecificgrants
Yourturn
____________________________________________________________________________
____________________________________________________________________________
PlanforsustainabilityIfthepolicyandprogramchangeissuccessfulhowwilltheprojectteamsustainandexpandtoadditionalsitesorpopulations
ExampleAlthoughmostofthepartnershipsforgedintheearlyyearsweremaintainednewgrantsandcollaborationsexpandedattentiontoolderpublicschoolstudentsactivetransportationandtheGreenLinesubwayextensionthebuiltenvironmentfarmerrsquosmarketeffortssuchasldquoGrowninSomervillerdquowithUnionMainStreeteconomicdevelopmentandhealthdisparitiesaswellasdatacollectionmonitoringandsurveillance
Yourturn
Designimplementationandevaluation
47
____________________________________________________________________________
____________________________________________________________________________
Pilotplanningtemplate
IfyouplantoconductapilotfirstdeterminethepurposeofthepilotDoyouwanttoobservewhetherthepolicychangeresultsinthedesiredbehaviorchangeAreyouconcernedaboutfeasibilityscalabilityorcostDoyouwanttotestwhetheryourdesiredresultscanbeachievedthroughthispolicyapproachClarifythegoalofthepilotandthequestionsyouhopetoanswerThiswillhelpyouidentifythekeyindicatorsthatyouwillusetomeasurewhetherthepilotissuccessfulBeforebeginningestablishabaselineofcurrentperformanceandsetyourtargetsaccordinglybasedonwhatyouhopetoaccomplishthroughthepilot
NextestablishthepilotplanforyourprojectteamincludingmilestonesforteamcheckinstodeterminewhethertomoveaheadorchangecourseThisisseparatefromthemeetingsthatwilloccuraspartoftheaccountabilityframeworkUsethespacebelowtocreateyourtimelineanddevelopyourmeetingschedule
Task Personresponsible Startdate Checkindate Deliverable
Planyourevaluation
WhileyouaredesigningtheimplementationitisalsotimetobegintodevelopyourplanforanalysisAsnotedbytheWilsonSheehanLabforEconomicOpportunitiesdecisionsaboutprogramsetup-includingoutreachscreeningintakedatatrackingandenrollment-candeterminewhetherornotarigorousimpactevaluationwillbepossibleAsdiscussedcityleadersmayfindithelpfultoincludeevaluationpartnersinearlyconversationsaboutprogramsetupDuringpreviousstagesyoumayhaveworkedwitharesearchertoplanyourproposedpolicychangeYoushouldbeclearaboutthepurposeoftheevaluationanddevelopanevaluationplanBelowareadditionalconsiderations
Causationvstrendanalysis
IntheEstablishaframeworktomonitorprogressstagetherewasguidanceontrackingtrendsovertimeItisimportanttonoteherethatmonitoringdataisdifferentfromanalyzingacausaleffectTrendanalysiswillillustratethatchangesareoccurringandmayleadtoquestionsthatcouldbeansweredbyyourevaluationbuttrendanalysisalonedoesnotprovideevidenceaboutwhatisdrivingthosechangesYoumayseechangesinyour
Designimplementationandevaluation
48
outcomeindicatorsbasedonvariablesoutsideoftheCityscontrolForinstanceachangeinfederaltradepolicycanhaveanimpactemploymentratesinyourCityoutsideofyourCitysnewjobtrainingandplacementprogram
ToaddresscausalityyoumustcomparetheoutcomesofindividualswhoreceivedtheinterventionwiththosewhodidnotWithoutacomparisongroupofsimilarindividualswhowerenotexposedtotheinterventionitisimpossibletoknowwhichfactorsareresponsiblefortheoutcomesRandomizedControlTrials(RCTs)inwhichindividualsarerandomlyassignedtotreatmentandcontrolgroupsarethegoldstandardinprovingcausalityIfyoucandemonstratethatoutcomesimprovedforthosewhoreceivedtheinterventionanddidnotimproveatallorimprovedlessforacomparisongroupyoucanbesomewhatcertainthattheinterventionwasthecause
Examplesofothernon-causalimpactevaluationsthatarerigorousandreliableincludecertainquasi-experimentalstudiesusingcomparisongroupdesignssmartinterruptedtimeseriesanalysesthatmodeltrendsovertimeinwaysthatallowestimationofinterventioneffectswelldesignedpre-posttestanalysesneedsassessmentsimplementationstudiesetcFormoreinformationondifferenttypesofevaluationsseetheSageResearchMethodswebsite
Lookfordifferencesingroups
BuildintimetoalsoexaminethepilotdatatoseeifgroupsareexperiencingdifferentresultsReviewingperformancebasedonoutcomemeasuresforapopulationinaggregatemayhidethesubtlewaysthatsomepeoplemaybeaffecteddifferentlysoitisnecessarytocarefullyanalyzethedatafordisparateoutcomesForinstancetheMovingtoOpportunity(MTO)studythatexaminedtheimpactofhousingvouchersonfamilieswholivedinhigh-povertyneighborhoodsfoundthatchildrenwhomovedbeforetheageof13hadbettereducationandhigherincomesasadultsHoweveradultsdidnotseeimprovededucationandemploymentoutcomes
Disaggregatingdataintomorespecificcategoriesofraceandethnicitycanhelpunderstandtrendsanddisparitiesinamoregranularwaywhichcanhelpdecisionmakerstargetpoliciesandinterventionsinamoreequitableway
EarlierintheRoadmapweworkedtoidentifystakeholdersandwhomightbepositivelyandnegativelyimpactedbyachangeinpolicyUsetheresultsofthatexercisetoinvolvethecommunityininterpretingdataanalysisandtogatherqualitativefeedbackfromthecommunitytocallattentiontoissuesForinstanceanincreaseinyouthdetentionmaybeexplainedbyinefficienciesorlackofaccesstotransportation
Designimplementationandevaluation
49
Designimplementationandevaluation
50
Shareyourstoryandsuccess
Sharetheresults
Storytelling
AnintegralstepinyourstrategyissharingtheresultsofyourpilotorpolicychangewithyourcommunityThisincludesdevelopingmessagingaroundtheworkofyourorganizationandcreatingaplantodisseminateyourstoriesWhytellstoriesaboutyourwork
Informandorunderstandasubjectorsituation
Giveneededcontextofdataorbitsofinformation
Relatablestoriesarememorableforreadersviewers
ReadersorviewerswillseetheinformationasmorecompellingwhenthereisademonstratedandrelatablechangeorimpactonthemselvesortheirfriendsneighborsandcolleaguesStoriesareaboutpeopleandgovernmentisfundamentallyabouthelpingpeopleSotellthestoriesofthepeopleyouandyourcitycolleagueshelped
Keepthesetipsinmindasyoudevelopyourstorywithyourteam
WhatwereyourfindingsandwhyshouldanyonecareTellyouraudiencehowtheproblemwassolvedWhathastheimpactbeenonresidentsandcitystafftodateWhatwasthesuccessandweretheremultiplesuccessesalongthewayIfyouhavepreliminaryresultsordatashowingimprovementtosharethisisthetimetodoit
WhowasinvolvedandimpactedReviewthelistofstakeholdersHowwereeachofthemimpactedbytheprogramHumanizeyourstoryandimpactbyaddingadditionalvoicesThesevoicescanbeoneofthefollowing
Apersonorfamilywhowaspositivelyaffectedbythechangesuccess
Aninvolvedthird-partydescribingboththeproblemandthesolution(ieanorganizationnonprofitprovidersbusinesstradeassociationsportsteametc)whoispartofthecommunitythatthegovernmentworkedwithonthechangesuccess
Someonefromthegovernmentitselfwhocanspeakaboutthechangesuccess
TellmoreaboutyourcityDescribeyourcityincludingpopulationdemographicsindustriesetcWhatareitschallengesWhatisitknownforWhatmakesituniqueHowisitsimilarordifferentfromothercitiesintheUS
Shareyourstoryandsuccess
51
DescribeyourcityrsquoschallengeWriteoutthechallengeyourcityisfacing(asdefinedinyourproblemstatement)HowdoesitaffectresidentsHowdoesthisproblemaffecttheworkofcitystaffmembers
AddthedetailsProvidedetailsonyourappliedresearchstrategyTalkaboutwhatyourcityhasdonenthepasttoresolvetheissueDiscusshowyourstrategyhaschangedwhenyoubegantoadoptthisapproachWasmoneysavedorinvested
Usethisstorymapexercisetoorganizeyourthoughtsandguaranteethatyouincludealloftheelementsofatraditionalstory
AfteryoudevelopyourmessagingcreateacommunicationsmarketingandoutreachstrategydesignedtospreadyourmessageYourstrategyshouldwalkyouthroughthefollowingkeyactivities
1 IdentifyandprofileyouraudienceWhencreatinganddisseminatingyourmessageitisalwaysgoodtothinkaboutyouraudienceWhoexactlywouldbeinterestedinwhatyouhavetosayThegeneralpopulationofthecityPeoplewhoworkinnonprofitsBusinessownersItmattersbecausenoteveryonewholivesinacitymaybethetargetoftheinformationBusinessownersmaybemoreinterestedindatashowingeffectsonthebusinessdistrictthanothersandsoonWewillcoverthislaterbutthisstepwillalsohelpyoutothinkofthemosteffectivewaytoreachyouraudience
2 Defineyourcalltoaction(ifyouhaveone)SpendsometimeconsideringwhyyouareinterestedintellingyourstoryDoyousimplywanttoupdatethepubliconyourprogressorcelebrateyoursuccessesAreyouinterestedinyouraudiencechangingitsbehaviororsupportingaspecificinitiativebeinglaunchedbyyourorganizationIfyouwanttohaveyouraudienceactorbehaveinacertainwaybespecificandincorporateitinyourmessaging
Shareyourstoryandsuccess
52
3 SelectperformancemeasuresThinkabouthowyouwoulddetermineifyourcampaignissuccessfulandselectappropriateperformancemeasuresYouwillrevisitthesemeasuresattheendofyourcampaigncommunicationsplanimplementationtodeterminewhetheryouneedtomodifyyourcommunicationsandoutreachstrategyforthefuture
4 MethodstocommunicateyourstoryTherearemanywaystocommunicatesuccessofyourprojectHerearesomesuggestionsonreachingaudiencesItisimportanttodeterminewhichmethodsarethemosteffectiveforyourdesiredaudiencetoreceiveinformationDoyourresearch(PewResearchCenterhasexcellentstudiesonthereachofmediadigitalmediaandmarketing)ForexampleifseniorsareyourtargetpopulationandtheyarelessinclinedtouseInstagram(thatistrue)youmaywanttoconsiderFacebookorthenightlylocalnewsbothofwhichareusedmoreoftenbyseniorsBelowisalistofsomemethodsandtoolsyoucanuse
i Pressreleasestolocalnewspapersandtvstations
ii ShortsmartphonevideosAgreatwaytoengageanaudienceisthroughvideoAskonequestionofsomeonewhoispartofthestoryMaybethereisasocialworkerwhocandiscusshungerinyourcityThesevideoscanbereusedonmultipleplatforms
iii Long-formwriting
iv TellingindividualstoriesthroughaseriesofphotosandshortnarrativesforpublicationonFacebookorInstagram
v Op-edsopinionpiecesplacedinthelocalpaper
vi Emailupdatestoyourlist
vii Connectingwithappropriateorganizationsthatareinvestedinthetopicandworkwiththecommunityandaskthemtodisseminateinformationtotheirmembers
viii EmployingldquoWordofMouthrdquo(WOM)tacticsbycreatingtownhallsorfreeeventsforresidentstoattendintheirownneighborhoods
5 DeterminethetypeofcollateralUsuallycampaignsthattargetresidentsrequireseveralavenuestoreachthemPeoplearenotalwaysinthesameplaceorspaceHavingakeenunderstandingofyouraudiencewillassistyouwithyourchoicesonwhatneedstobedevelopedSometimesthatmeansimagesforsocialmediaaonepageflierthatgivespeopleinformationonattendinganeventoranin-depthbrochure
Shareyourstoryandsuccess
53
6 ImplementyourstrategyKeepinmindstrategiescanbefluidSometimessomethingmaynotturnoutasexpectedevenifduringplanningsessionsitwasldquosupposedtordquoItisokYoucananalyzewhathappenedduringadebriefsessionandonceyoursurveysandothermeasurementshavebeenfinalized
Visualizations
Withtherightvisualizationyoucanreviewandshareinformationinsightsanalysisperformanceandtrendsallofwhichisvaluabletoacross-sectionofthepublicButwhichvisualizationisrightforyouTochoosetherighttypeofchartgraphdeterminethetypeofquestionyouwanttoanswerSomecommoncharttypesarebelowandmoreinformationcanbefoundhere
Exploreadditionalanalyticaltechniques
Duringthegoal-settingphaseyouestablishedoutputoutcomeandimplementationmeasuresBasedontheresultsofyourpilotyoumayneedtoadjustyourmeasuresWiththepilotdatainhandyoumayhavesomeideasofhowfutureanalysescouldbedifferentandpotentiallyimprovethequalityofyourprogramTakesometimetobrainstormotherrelevantdatasetsthatyoudidnothavethetimetopursueearlierincludingmessydepartmentaldatathatneedsmoreattentionorsourcesofpublicdatafromthestateandfederalgovernmentnonprofitsandevenprivatecompaniesthatcouldsupplementyourunderstandingofanissueUsingmachinelearningtechniquesyoumightbeabletobetter
Shareyourstoryandsuccess
54
predictdemandforhomelessservicesusingNOAAweatherforecastspreemptwatermainbreaksusingsoilstratigraphyorinvestineconomicdevelopmentbymonitoringrealestatelistingdata
Advancedanalyticaltechniquescanalsohelpyourimplementationrunmoreeffectivelybyintegratingdata-informedtargetinganddecisiontoolsintotheprocessByautomatingroutineworkflowsandsurfacingsubtletrendsthesetoolscanhelpyouiterateinamoreprecisemannerandprovidekeystrategicguidancetofront-lineemployeesthatwouldotherwisebeunavailableThesesortsofdata-science-enhancedprocessescanbeclassifiedasoneofsixprojecttypesasexploredfurtherbytheNewOrleansLANOLAlyticsteamFormoreguidanceonscopingasimilarprojecttheGovExadvancedanalyticsscopingguidegivesyouastep-by-stepoverviewofhowtobesuccessful
Usethespacebelowtolistyourideasforpossibleadvancedanalyticsprojectsbytype
Typeofproject Ideas
Findingtheneedleinthehaystack
Prioritizingforimpact
Earlywarningtools
Betterquickerdecisions
Optimizingresourceallocation
Experimentingforwhatworks
Keepthefeedbackloopgoing
GatherinformationonpublicsentimentaboutpolicychangethroughfocusgroupssurveysandpublicmeetingsledbyadvisorygroupmembersNotewhethernewinformationisavailablethatcanhelpanalyzeorcontextualizeyourprogress(egfederalorstatedatareports)Sharetheresultsofyourpilotandthepolicychangeprocesswithothercitiesandresearchers
Shareyourstoryandsuccess
55
WorkingwitharangeofevidencePresentlythereisalotofattentionandinvestmentinevidence-basedpolicymakingincludingthefederalbipartisanCommissiononEvidence-BasedPolicymakingthePewCharitableTrustsreportsandtoolkitsonHowStatesEngageinEvidence-BasedPolicymakingandtheLauraandJohnArnoldFoundationsupportedEvidence-BasedPolicymakingCollaborativeManyoftheseinitiativesfocusonrigorousevaluation--meaningrandomizedcontrolledtrials(RCT)orotherexperimentalmodels--asthebestkindofevidencetouseHoweverthisbodyofevidenceisofteninsufficientforthespecifickindsofquestionsthatgovernmentscanhavewhenmakingadecisionaboutpolicyInthisguideweproposeanapproachtosupportgovernmentsinincorporatingadditionaltypesofevidenceintotheirdecisionmaking
OnewaytounderstandarangeofevidenceistouseNestarsquosStandardsofEvidencetoframeanddefineevidenceTheBritishnonprofitNestaworkstoinjectpositivechangeintothewayUKgovernmentpoliciesandprogramsfunctionOneoftheirkeyareasofworkhasbeendevelopingWhatWorksCentresorganizationsthatcollectcommissionsynthesizeanddisseminateevidenceaboutinterventioneffectivenessacrossarangeofsocialimpactareasNestadevelopedthefollowingStandardsofEvidencetoprovideausefulwaytodocumenthowdifferentkindsofinformationdemonstratedifferentlevelsofevidence
Workingwitharangeofevidence
56
Theimportanceofpotentiallybiasedinformation
Onechallengepresentedbyanevidenceclassificationthatacknowledgesthevalueofnon-scientificformsofevidenceistofigureouthowtobalanceallofthepiecestogetherwhenyouusethemTounderstandhowtodothiswehavetothinkabouthowtopositionpotentiallybiasedinformationindecisionmaking
WhenwelookatinformationwhichhasnotbeenrigorouslyevaluatedweacceptthatthisinformationmightbebiasedThewholereasonweconductrigorousevaluationistoremovebiasfromourobservationsWeconductstructuredtrialstoproduceresultsthatareconsistentandgeneralizableIfafindingisunbiasedthisaccuracymdashthislackofbiasmdashcanbeprovedbecauseanyindependenttestercanreproduceitwhentheyfollowanidenticalprocess
YoucanlookatpotentiallybiasedinformationasjustinformationthatisearlieroninitspathtorefinementreplicationandverifiedaccuracyApieceofinformationbasedonanyoneindividualrsquosobservationsislikelytobebiasedbecauseitiscomingfromjustonesourceBiasisreducedbygatheringtogetherlotsofindependentsourcesofinformationandfiguringoutitsmostcommonattributes
MaybethebiasedinformationmakesaclaimthatistoolargemdashitsayssomethingistrueunderallconditionswhenitisreallyonlytrueundercertainconditionsorthataneffectofaninterventionisbiggerthanitstrueaverageorthataneffectlastsovertimewhenitdoesnotPerhapschangingtheclaimtobemorespecificmdashcriticallyreviewingtheinformationtoseeifspecificconditionsmustbemettoachievetheoutcomesmdashwouldmaketheclaimmoreaccurate
IfyoutestedyourclaimfounditdidnotholdupinsomecaseschangedittesteditagainuntilyourclaimrepeatedlyheldupthenyoucoulddemonstratethattheclaimwasnolongerbiasedbutaccurateThisprocessoftestingimprovingtestingandvalidatingiswhatwegetfromstructuredscientificstudy
Withindividualpiecesofinformation--crediblehypothesescasestudiesexpertopinion--youcannotassumethatkindofaccuracyAtthesametimetheseearlyobservationsarefoundationaltoanyreplicablescientificobservationsWewillnevergetanywhereifwedonotstartwiththesepiecesofinformationeveniftheymightbebiased
Combininglevelsofevidencetogetthemostdetailedinformationforyourcity
ItisespeciallyimportanttoacceptdifferentformsofevidencewhenyoufacequestionsthathavenotyetbeentestedbyindependentstudiesSomeofthemostimportantquestionsrelatetothingsthatareveryspecifictoyoursituationYoufaceaparticularcombinationof
Workingwitharangeofevidence
57
issuesactorstrendsandresourcesthatisultimatelydifferentfromeveryotherplaceintheworld
ThisisimportantbecauseaspolicypractitionersandnotscientistsyoucaremoreaboutyourspecificcasethanahypotheticalaveragegeneralcaseYoucarelessaboutthetotallikelyrangeofimpactsfromapolicythanhowthissingleimplementationwillworkYouwantinformationaboutpoliciesthatwillprovideyouwiththebestpossiblepredictionabouthowtheywillworkinyourcontext
ByputtingdifferentkindsofevidenceintothesameframeworkNestarsquosStandardsofEvidenceimplythatwhenyoucannotgetrigorouslytestedexperimentallyverifiedinformationitisappropriatetoworkwithwhatwedohaveThisisrelevantforthemunicipalimplementationcontextbecausenostudypulledfromaliteraturereviewwillaccountforallofthevariablesyoufaceinyourowncontext
Intheabsenceofhavingastudythatpredictshowaninterventionworksinyourspecificcontextyouwillneednon-scientificinformationtotrytofillinthosegapsYouwillneedtogatherstoriesfromresidentsstaffmembersandotherstakeholdersthatsitonLevels1or2ofthestandardsofevidence
Someofthekindsofinformationyouwillprobablygetmoreeasilyfrompeoplethanstudiesinclude
howpeopleincitieslikeyoursreactedtoaninterventioninpublicforumshowpeoplewhoimplementedaninterventioninacitylikeyoursfiguredouthowtogetitpassedhowthecitycommunicatedpubliclyabouttheapparentbenefitsanddrawbacksoftheinterventionthepoliticalacceptabilityoftheintervention
Youcanremainsimultaneouslyawarethattheinformationyouaregatheringthroughindividualstoriesisbiased--notgeneralizableinformedbyspecificperspectives--whilealsorecognizingthathavingsomeinformationaboutsomethingimportantisbetterthannoinformationEvidencefromthefirsttwoevidencelevelsareessentialbuildingblocksofknowledgeandperfectlyusefulwhenyoucannotgetdetailedunbiasedknowledge
Youhavetostartsomewhereandyoushouldalwaysstartwithevidenceyouhaveaboutyourveryspecificconditionsthatyouknowareimportantforsuccess
Workingwitharangeofevidence
58
Thankyou
Interviewswithcities
GovExheldsemi-structuredinterviewswithsixstaffmembersinfourcitiestoestablishtheinitialstepsoftheRoadmapWeaskedrespondentstotalkaboutaninitiativeorpolicythattheyhavebeenresponsibleforimplementingintheircityRespondentsfromthreecitiesspokeabouttheireffortstoaddresshomelessnessandrespondentsfromonecityspokeaboutblightAfterrespondentsidentifiedtheproblemtheyspokeabouthowthecityprioritizedthisissuethevariousstepsinvolvedandbarrierspresent
GovExconductedtextanalysisoftheseinterviewswhichrevealedthatpeoplewasthemostcommonlyusedtermduringtheseconversationswhendiscussingthestepstochangeaprogramorpolicyThisisathemethroughoutthisRoadmapThefirstsectionbeginswithidentifyingtherelevantpeopleTheconversationsaroundevidencearefocusedonhumanbehaviorratherthanmethodologyorstrengthofevidence
Thankyou
59
OncewehadadraftoftheRoadmapwefollowedupwithcitiesonanindividualbasistolearnmoreaboutwhatwouldbehelpfulWearegratefultothefollowingcitystafffortheinterviewsandindividualconversations
MichaelBaskinChiefPolicyOfficerChattanoogaTN
MelissaBridgesPerformanceandInnovationCoordinatorLittleRockAR
VictoriaCarreoacutenAdministrativeOfficerLasVegasNV
TyroneGrandstrandCommunityDevelopmentPlannerFargoND
TibLaughlinDirectorofGeneralServicesKansasCityKS
TimMorelandDirectorofPerformanceManagementandOpenDataChattanoogaTN
CharlesRosendahlManagementAnalystDentonTX
Thankyou
60
MelissaSiebenAssistantCountyAdministratorKansasCityKS
GinaThompsonDirectorofStrategicServicesLewisvilleTX
JHUFacultycollaboration
GovExalsoconvenedthecitycontactsandfacultypartnersinaseriesofAdvisoryGroupmeetingsThesemeetingsservedtoconnectcitiesandfacultytounderstandoneanothersperspective
StefanBaralBloombergSchoolofPublicHealth
AshleyBernerJHUSchoolofEducation
MelissaDavey-RothwellBloombergSchoolofPublicHealth
JonLinksBloombergSchoolofPublicHealth
TamarMendelsonBloombergSchoolofPublicHealth
KristinMmariBloombergSchoolofPublicHealth
KeshiaPollackPorterBloombergSchoolofPublicHealth
JoshSharfsteinBloombergSchoolofPublicHealth
LizStuartBloombergSchoolofPublicHealth
LindsayThompsonCareyBusinessSchool
DanielWebsterBloombergSchoolofPublicHealth
Partnersindataandevidence-basedpolicy
Wealsospokewithpartnerswhoareexpertsinthefieldofevidence-basedpolicywhosharedtheirknowledgeandresourcesofcurrentandfuturethinkinginevidence-informedthinking
WendyBarrenoAssociateDirectorWilsonSheehanLabforEconomicOpportunitiesUniversityofNotreDame
TiffanyDavenportDirectorofDataPartnershipsBaltimoresPromise
TomiHiersDirectorBaltimoreCivicSiteAnnieECaseyFoundation
ChrisKingsleySeniorAssociateAnnieECaseyFoundation
KathrynOliverAssociateProfessorLondonSchoolofHygieneandTropicalMedicine
Thankyou
61
KristenWatkinsSeniorPolicyAssociateJ-PAL
GovExParticipants
GovExAnalystMiriamMcKinneyanalyzedtheinterviewtranscriptsandconductedtextanalysisSeniorImplementationAdvisorEmilyShawwillintegrateRoadmapactivitieswithadditionalservicedeliveryactivitiesatGovEx
Thankyou
62
ReferencesAbrahamKGampHaskinsR(2017)Thepromiseofevidence-basedpolicymakingReportofthecommissiononevidence-basedpolicymakingWashingtonDCCommissiononEvidence-basedPolicymakingBigmove2Extendingthegreen(nd)Retrievedfromhttpswwwcityoflewisvillecomdoing-businesslewisville-2025big-move-2-extending-the-greenBatesMAampGlennersterR(2017)ThegeneralizabilitypuzzleRetrievedfromhttpsssirorgarticlesentrythe_generalizability_puzzleBuildwithnotfor(2018January)RetrievedfromhttpsbuildwithnotforwordpresscomCairneyPampOliverK(2017)Evidence-basedpolicymakingisnotlikeevidence-basedmedicinesohowfarshouldyougotobridgethedividebetweenevidenceandpolicyHealthResearchPolicyandSystemsdoi101186s12961-017-0192-xCairneyPOliverKampWellsteadA(2016)TobridgethedividebetweenevidenceandpolicyReduceambiguityasmuchasuncertaintyPublicAdministrationReview76(3)399-402Retrievedfromhttpdxdoiorg101111puar12555Centreforpublicimpact(nd)RetrievedfromhttpswwwcentreforpublicimpactorgCernaL(2013)ThenatureofpolicychangeandimplementationAreviewofdifferenttheoreticalapproachesOrganisationforEconomicCo-OperationandDevelopmentRetrievedfromhttpswwwoecdorgeduceriTheNatureofPolicyChangeandImplementationpdfColoradoSprings(nd)Retrievedfromhttpscoloradospringsgovcommunicationspagewhat-expect-when-you-speakupContinuumofcare(coc)program(nd)RetrievedfromhttpswwwhudexchangeinfoprogramscocCouncilpresidentkicksoff$12millionyouthfund(nd)Retrievedfromhttpwwwbaltimorecitycouncilcomcontentcouncil-president-kicks-12-million-youth-fundDurlakJAampDuPreEP(2008)ImplementationmattersAreviewofresearchontheinfluenceofimplementationonprogramoutcomesandthefactorsaffectingimplementationAmericanJournalofCommunityPsychology41327-350Retrievedfromhttpswwwncbinlmnihgovpubmed18322790Fishbonediagram(ishikawa)(nd)Retrievedfromhttpasqorglearn-about-qualitycause-analysis-toolsoverviewfishbonehtmlFivewhysGettingtotherootofaproblemquickly(nd)RetrievedfromhttpswwwmindtoolscompagesarticlenewTMC_5Whtm
References
63
FligsteinNampMcAdamD(2011)TowardageneraltheoryofstrategicactionfieldsSociologicalTheory29(1)1-26RetrievedfromhttpsociologyberkeleyedusitesdefaultfilesfacultyfligsteinFligstein20McAdam20Soc20Theory20PaperpdHeinkelP(2018January12)CitymullspermitsforbikesharingcompaniesRetrievedfromhttpwwwdentonrccomnewsdenton20180112city-mulls-permits-bike-sharing-companiesHowisareamedianincomecalculated(2012October)Retrievedfromhttpswwwhudexchangeinfofaqs843how-is-area-median-income-calculatedLampkinLMampHatryHP(2003)KeystepsinoutcomemanagementUrbanInstituteRetrievedfromhttpswwwurbanorgresearchpublicationkey-steps-outcome-managementMarshDampSmithM(2000)UnderstandingpolicynetworksTowardsadialecticalapproachPoliticalStudies484-21Retrievedfromhttpjournalssagepubcomdoiabs1011111467-924800247McCannL(nd)RetrievedfromhttplaurenellencomMoultonSampSandfortJR(2016)ThestrategicactionfieldframeworkforpolicyimplementationresearchPolicyStudiesJournal45(1)Retrievedfromhttponlinelibrarywileycomstore101111psj12147assetpsj12147pdfv=1ampt=jdkhhcp4amps=e184c625e1855a6b1615ccb99ee27b1c0b3212d8PaynterB(2018January1)What100citiesarelearningfromeachotherbysharingtheirdataRetrievedfromhttpswwwfastcompanycom40518004what-100-cities-are-learning-from-each-other-by-sharing-their-dataPeakacademy(nd)Retrievedfromhttpswwwdenvergovorgcontentdenvergovenmayors-officeprograms-initiativespeak-performancepeak-academyhtmlPinnockHBarwickMCarpenterCREldridgeSGrandesGGriffithsCJRycroft-MaloneJMeissnerPMurrayEPatelASheikhAampTaylorSJC(2017)Standardsforreportingimplementationstudies(stari)ExplanationandelaborationdocumentBMJOpenRetrievedfromhttpbmjopenbmjcomcontentbmjopen74e013318fullpdfProctorESilmereHRaghavanRHovmandPAaronsGBungerAGriffeyRampHensleyM(2011)OutcomesforimplementationresearchConceptualdistinctionsmeasurementchallengesandresearchagendaAdministrationandPolicyinMentalHealth38(2)65-76doi101007s10488-010-0319-7TandonSD(2018)ProgressincommunityhealthpartnershipsResearcheducationandactionRetrievedfromhttpswwwpressjhuedujournalsprogress-community-health-partnerships-research-education-and-actionProvanKGampKenisP(2007)ModesofnetworkgovernanceStructuremanagementandeffectivenessJournalofPublicAdministrationResearchandTheory
References
64
18(2)229-252Retrievedfromhttpsacademicoupcomjpartarticle182229935895SabatierPA(1988)Anadvocacycoalitionframeworkofpolicychangeandtheroleofpolicy-orientedlearningthereinPolicySciences21(3)129-168Retrievedfromhttpwwwjstororgstable4532139SandstromAampCarlssonL(2008)TheperformanceofpolicynetworksTherelationbetweennetworkstructureandnetworkperformancePolicyStudiesJournal36497-524Retrievedfromhttponlinelibrarywileycomdoi101111j1541-0072200800281xabstractSchwartzMampWilsonE(nd)WhocanaffordtoliveinahomeAlookatdatafromthe2006americancommunitysurveyRetrievedfromhttpswwwcensusgovhousingcensuspublicationswho-can-affordpdfShearerJCAbelsonAKouyateBLavisJNampWaltG(2016)WhydopolicieschangeInstitutionsinterestsideasandnetworksinthreecasesofpolicyreformHealthPolicyandPlanning31(9)1200-1211Retrievedfromhttpsdoiorg101093heapolczw052ThelessonsofadministrativedataHigh-profilepolicy-relevantresearchpoweredbyadministrativedata(nd)Retrievedfromhttpswwwpovertyactionlaborgsitesdefaultfilesresources20161114-The-Lessons-of-Administrative-DatapdfToolsandotherneedsforqualitativeandquantitativemethods(2015February)AgencyforHealthcareResearchandQualityRetrievedfromhttpwwwahrqgovprofessionalsquality-patient-safetytalkingqualityassesstoolshtml
WeaverRCampBagchi-SenS(2013)SpatialanalysisofurbandeclineThegeographyofblightAppliedGeography4061-70
References
65
UniversityPartners
Name Title Affiliation Area(s)ofexpertise
Contactinformation
Examplesofworkingwithnon-academicdecisionmakers
WendyPBarreno
AssociateDirector
WilsonSheehanLabforEconomicOpportunitiesUniversityofNotreDame
LEOsfocusareasincludehealtheducationhousingchildwelfareeconomicopportunitycriminaljustice
wbarrenonduduorleondedu
MelissaDavey-RothwellPhD
AssociateScientist
JohnsHopkinsBloombergSchoolofPublicHealth
Health mdavey1jhuedu410-614-5854
BaltimoreCityHealthDepartmentLeadersMarylandDepartmentofHealthAddictionRecoveryServicesinDCExecutiveDirectorsofcommunityorganizationsacrosstheUSthatfocusonHIVSTIsubstanceabuseandimplementationscience
ElizabethStuart
AssociateDeanforEducationandProfessorDepartmentsofMentalHealthBiostatisticsandHealthPolicyand
JohnsHopkinsBloombergSchoolofPublicHealth
Educationsocialpolicystatisticalmethods
estuartjhuedu
Resources
66
Management
Design-ThinkingResources
EngagementGuideforCivicTech
Journeymap
PerformanceManagementResources
DenverPeakAcademy
Proxymeasuresguide
Talkingaboutperformancewithoutcomplicatingit(video)
SettingPerformanceTargetsguide
Datavisualizationguide
Performancemanagementgettingstartedguide
Exercises
Fishboneexercise
5Whys
AdministrativeData
TheLessonsofAdministrativeData(JPAL)
EvaluationResources
Quantitativeandqualitativemethods(USDepartmentofHealthandHumanServicesAgencyforHealthcareResearchandQuality)
KeyStepsinOutcomeManagement(UrbanInstitute)
RCT-YES
Sageresearchmethods
BestPracticesforMixedMethodsResearchintheHealthSciences
ACCORDChecklist
Pilotandimplementationplanning
Benchmarkingexamplesandproxymeasures
Seattleopendataplaybook
Resources
67
Datasciencescopingguide
Resources
68
AboutGovExGovExhelpsgovernmentsusedataandevidencetoimproveresidentsrsquolivesWesupportgovernmentsinbuildingtheirinternalcapacitytooperatesustainableanalyticspracticeswhichimprovecommunityoutcomesinfieldsfrompublicsafetytoeconomicdevelopmentGovExhasaproventrackrecordofworkinginconcertwithpartnerstoprovidetechnicalassistanceandtrainingatscale
SinceourlaunchinApril2015GovExhas
Trainedover2000governmentemployeesinpersonandonlineintopicsincludingdatamanagementperformancemanagementopendatacommunityengagementandanalytics
Providedtechnicalassistancetomorethan100localgovernmentsacrosstheglobeimprovingtheiruseofdatatoachieveefficienciesinservicedeliveryandbetteroutcomesforresidents
Assessedthecapacityofmorethan130governmentalorganizationstodeterminehowleadersstaffandexternalstakeholdersmanageandusedata
Localgovernmentsareincreasinglyonthefrontlinesofaddressinglargescaleproblemswithglobalimplicationssuchasclimatechangepublicsafetyandeconomicinequality
Whileevidence-basedinterventionstohelpsolvethoseproblemsareavailablelocalgovernmentstaffhavelimitedtimeandcapacitytofindandimplementresearch-basedsolutionsAsaresultcitiescontinuetoworkthewaytheyalwayshave--makingincrementalimprovementstoprogramsbutnotalwayscreatingchangeatasystem-level
ThroughresourcedevelopmentandcollaborationwithJHUfacultyGovExissupportingcityleadersinusingdataandevidencetomakedecisionsandcreatinglastingrelationshipswithresearchers
TheRoadmapwasdevelopedbyGovExaspartofitspartnershipwithBloombergPhilanthropiesWhatWorksCitiesinitiative
AboutGovEx
69
- Title page
- Introduction and purpose of the Roadmap
- Using the Roadmap
-
- Gather information about the problem
- Strategically engage key partners
- Review the evidence and collect details
- Decide on a policy to implement based on local factors
- Establish a framework to monitor progress
- Assess data accessibility quality and security
- Design implementation and evaluation
- Share your story and success
-
- Working with a range of evidence
- Thank you
- References
- Resources
- About GovEx
-
expertsThepublicwasinvitedtopresentcommentsonideasfordecreasingoverdoseearlyinterventiontreatmentandrecoveryPublichealthandmedicalfacultyfromWestVirginiaUniversityMarshallUniversityandJohnsHopkinsUniversityprovidedresponsestothecommentsTheresultwasanOpioidResponsePlanthathealthcommissionerRahulGuptapresentedtoGovernorJimJustice
AsamplespectrumtoplanyourowncommunityparticipationprocesscreatedbyLaurenellenMcCannisbelow
SourceLaurenellenMcCann
Community-basedparticipatoryresearchisanotherwaytoworkwiththecommunitytoimproveoutcomesCBPRmeansthecommunityisinvolvedinallaspectsoftheresearchprocesswhichisofteniterativeInCBPRtraditionalresearchersandcommunitymembersshareknowledgeaswellasresourcesandcredit
CBPRInActionInSanFranciscoanenvironmentaljusticenonprofitteamedupwiththeCitysDepartmentofPublicHealthandanoutsideevaluatortounderstandtherootcausesofobesityandfoodinsecurityinlow-incomeneighborhoodsandidentifysolutionsThroughresearchmethodssuchasinterviewingresidentsandstoreownersdiagrammingstoreshelvesGISmappinganeconomicanalysisandreviewofsimilarpoliciesinnearbycitiestheteamobservedthatthestoresthatresidentshasveaccesstowerestockedwithlesshealthyfoodsandthatresidentswantedmorehealthyfoodandlessalcoholandtobaccoThisprojectresultedinlocalandstatelegislationtoencouragestorestostockfreshproduce
Strategicallyengagekeypartners
18
Developrelationshipswithsubjectmatterexperts
OnewaytokeepsubjectmatterexpertsengagedistoinvolvetheminthecommunityparticipationprocessasdescribedaboveIntheWestVirginiaexamplesubjectmatterexpertsrespondedtoandsynthesizedthepubliccommentsOtheropportunitiesaredescribedbelow
AdvisoryAcitydepartmentheadMayororstaffmembermaydevelopaone-on-oneadvisoryrelationshipwithuniversityfacultymemberoracollaborativebetweentheCityandUniversitytoaddressaparticularproblemAgreatplacetostartistolookforrelevantpracticecenterswithinyourlocaluniversityandreachingouttosubjectmatterexpertsInadditiontotheirexpertiseontherelevantpolicytopicheorshemayalsobeabletoidentifyotherexpertsandevenprovidesupportonhiringtherighttalenttoleadcertainprogramsAnadvisoryrelationshipwithauniversitysubjectmatterexpertcanalsohelpwhencitiesareconsideringbringinginnewprogramsTheseexpertsareoftenfamiliarwithprogramsthatarebasedonnationalmodelsandcanprovideinsightintoaprogramsoriginandlineageeffectivenessinotherplacesandthepotentialforimplementationinyourcityAcademicscanalsohelpidentifyindicatorsbasedongovernmentindicatorsorliteratureFinallyrelationshipsbetweenpolicymakersandresearchersprovidethemostbroadlyeffectiveroutetopullingresearchintopolicy
ResearchandevaluationAcademicexpertsmaysupportcitiesinevaluation-indeedevaluatingcityinitiativesisatraditionalmodelofcityuniversityrelationshipsSharingtheinformationyoucollectedintheGatherInformationphasecanhelpresearchersunderstandyourworkandgoalswhichinturnwillstrengthenthesupportthattheycanprovideDiscussingdetailsaboutyourinterventionandevaluationplanatthebeginningoftheprocessmeansthataresearchpartnercanhelpwiththeevaluationdesignanddatacollectionForinstancetheWilsonSheehanLabforEconomicOpportunitieswalkscitiesthroughconsideringthefollowingquestionsbeforepartneringonanevaluation
1 Isthereaclearpolicyprogrammodelwithobjectivescreeningcriteria
2 Isitpossibletoevaluatewhathappensbothwiththepolicyprogramandintheabsenceofthepolicyprogram
3 Arethereoneormoremeasurableoutcomesofinterest
4 Arethereenoughpeopleforrigorousquantitativeanalysis
5 Whatdatawillbenecessaryforthestudyandwhataretheprocessesforaccessingthesedatasource(s)
6 Isthereacompellingreasontoevaluate
Strategicallyengagekeypartners
19
InadditiontoassessingreadinesstoconductanevaluationresearcherscanhelpyouhoneinonthepurposeoftheevaluationTheWorldBankdescribesseveralpotentialusesofevaluationwhicharedefinedhereInbriefevaluationcanhelpdecisionmakers
makeresourceallocationdecisionsrethinkrootcausesofproblemsidentifynewproblemsdecideonabestalternativedeterminewhetherprogramsorpoliciesarebeingimplementedwithfidelitydeterminewhetherprogramsorpoliciesarehavingthedesiredimpact
Researcherscanalsoprovidesupportinsharingtheresultswiththescientificcommunitythroughpapersandconferenceswhichisvaluableincreatingapeernetworkofevidence-basedcitypractitioners
TrainingandtechnicalassistanceUniversitypartnersespeciallythoseatpracticecenterscanbehelpfulindevelopingorupdatingtrainingcurriculumtrainingthetraineranddeliveringtrainingtocitystaff
Strategicallyengagekeypartners
20
ReviewtheevidenceandcollectdetailsOnceyouhaveathoroughunderstandingoftheproblemdataandactorsyouwillneedtoconductyourownresearchonavailableevidenceandlearnaboutthelandscapeofpolicyoptionsYourgoalwillbetodevelopanaccessiblelistofinformationaboutpoliciesthathavehelpedsolvesimilarproblemsinotherplacesThischapterwillguideyouthroughthestepsofpreparingyourreviewersunderstandinglevelsofevidenceandcollectingthedetailsonevidencethatyouwillneedfordiscussionwithdecisionmakersinyourcity
Selectandprepareyourreviewers
AssignapersonorteamtoreviewrelevantinterventionsthathavedemonstratedsuccessinotherjurisdictionsTheirgoalwillbetoproduceareviewofalloftherelevantprogramspoliciesorinitiativesthatyourteamcouldconsiderimplementingTheywillbeconductingaliteraturereviewsupplementedbyinterviewswithexpertsandpeerswhohaveinformationaboutpotentialprograms
SincethepeopleyouassigntothistaskareprovidingthebaselineofpotentialoptionstoimplementitisimportantthatyouchoosepeopleyouareconfidentwillbethoughtfulandthoroughintheirworkItisalsocriticalthattheyareawareoftheimportanceofsourcecredibilityEnsureyourreviewershavefamiliarizedthemselveswithrulesforcriticallyanalyzinginformationsourcesbeforetheygetstartedIftheyareincludingnewsarticlesaspartoftheirsearchtheyshouldalsobeawareofmethodsforevaluatingnewscredibility
AskyourreviewerstodevelopaplanandtimelinetocompletethereviewThisstartswithsolidifyingagoverningresearchquestionOnewaytoknowyouarecreatingadetailedresearchquestionistousetheacronymPICOremindingyoutoincludereferencetothepopulationintervention(ifyouhaveoneinmind)comparisonandoutcomeelementsinyourresearchquestionForexampleifyouknowthatyouhaveapopulationofteenagerswhoarehomelessyouhavelearnedfromyourstakeholdersthatmanyoftheseteenagersidentifyasLGBTandyouhaveheardsomethingabouttheapproachofcreatingdrop-incentersfordeliveringsupportyoucouldask
ForLGBTyouthfacinghomelessness(population)howeffectivearedrop-incenters(intervention)relativetostandardsocialservices(comparison)atsettlingtheminpermanenthousing(outcome)
Ifyoudonothaveaninterventioninmindyoucanleaveouttheinterventionandcomparisonandsimplyask
Reviewtheevidenceandcollectdetails
21
HowcanwehelpLGBTyouthfacinghomelessness(population)obtainpermanenthousing(outcome)
OnceyourreviewershavetheirresearchquestionsusethequestionstocreatealistofkeywordstheycanuseintheirliteraturesearchThiswillhelpkeeptheircollectionofarticlesreasonablyfocusedHavethemdescribewhattheirsearchprocesswillbeincludinglistingthedatabasestheywillreviewandtheirplanforstoringtheinformationtheywillcollect(Herearesomeadditionalsuggestionsforstructuringareviewplan)
WhatyourreviewerswillbedoingisalightweightversionofarapidreviewTotheextentthattheywouldliketodevelopamethodologyforspecificsearchprotocolsincludingcriticalappraisalprocessestheycancertainlygoaheadanddoso
Usedifferentlevelsofevidenceforthemostcompletepicture
AsdiscussedintheWorkingwitharangeofevidencechaptertheNestaStandardsofEvidenceframework--reproducedbelow--ishelpfultounderstandthelevelofconfidencethataprogramorinterventionishavingapositiveimpactThisapproachletsyouputinterventionsthatyouhaveheardaboutlocallybutwhichhavenotbeenstudiedtogetherwiththosewithademonstratedcausaleffectThisishelpfulsinceanumberofpoliciesprogramsandpracticesthatyouwillwanttoconsidermaynotyethavemuchofaresearchrecord
Reviewtheevidenceandcollectdetails
22
NearlyanypieceofinformationaboutaninterventioncanbeclassifiedwithinthestandardsofevidenceForexampleifsomeonehasacrediblehypothesisabouthowaninterventioncouldproducebenefitthistypeofevidencewouldbeclassifiedasLevel1ThistypeofevidencecouldincludeinterventionsinothercitiesthatyoureadaboutinthenewsorhearaboutataconferenceForinstancewhenUSDAallowedschoolsandschooldistrictstoprovidefreebreakfastandlunchtoallstudentsthroughtheCommunityEligibilityProvision(CEP)manydistrictsimplementedthisoptionevenbeforetherewasevidenceofeffectivenessThiswouldfitintoLevel1sinceitbuildsonthelogicthatprovidingaccesstomealswilldecreasefoodinsecurity
YoumayalsofindpiecesofinformationthatqualifyasLevel2evidenceForinstanceasmoreschooldistrictsimplementedCEPprincipalstestifiedaboutpositivechangestheyobservedinstudentssuchasamorepositivedemeanorthatcanleadtobetteracademicperformanceDataalsoshowedthatthestudentsdescribedwerereceivingmoremealsthanbeforetheprogramsimplementationHoweveritwastoosoontobuildevidencetoshowacausalrelationshipbetweentheincreaseinfreemealsandincreasedacademicperformance
ldquoLower-levelrdquoevidencecanconsistofnuancedbutvaluableelementsthatrelatetoaninterventionrsquospotentialfitwithacommunityQualitativeinformationsuchastestimonyfrompeoplewhohaveexperiencedaninterventionandthosewhohavefacilitatedorimplementeditwillalwaysbeusefultoconsiderandprovidesdetailedlocalcontext
Experimentalstudies--describedstartingatLevel3withcontrolledexperimentsandgoingthroughLevel5withfullyfleshedoutprotocolsandprocessesforreplication--stilloccupythehigherlevelsofthisdescriptionofevidenceThisprioritizationshowsthevalueofrigorousstudyforevaluatingtheimpactofaparticularpolicyorprogramratherthanotherfactorsthatcouldbeinfluencingoutcomesofinterestRememberthatmultipletypesofevidencecancomplementrigorousimpactevaluationsandgivecityleadersadditionalinsightsandbroaderperspective
Accessinformation
Encourageyourreviewerstobegintheirsearchforpromisingpracticesingovernmentandnon-profitclearinghousesandinacademicpolicyjournalsWorkpublishedinthesesourceshasalreadybeensubjectedtoahighlevelofprofessionalscrutinyandgenerallycontainsagreatdealofdetailwhichmeansthatyourreviewersaremorelikelytobeabletopullouttheinformationyouneedtomakeagooddecision
Clearinghouses
Reviewtheevidenceandcollectdetails
23
Herearesomeoftheclearinghouses(collectionsofprogramevaluations)forevidence-basedpoliciesandprograms
ResultsFirstClearinghouseDatabaseThePewCharitableTrustsCollectstogethereightotherclearinghousestoprovideinformationoninterventionsinthefollowingcategoriesadultcriminaljusticeeducationchildwelfarehealthjuvenilejusticementalhealthsubstanceabuseandyouthdevelopmentIndividualclearinghousesincludedinResultsFirstare
BlueprintsforHealthyYouthDevelopment-foryouthdevelopmentandviolenceprevention
CaliforniaEvidence-BasedClearinghouse-forchildwelfareinterventions
TheLauraandJohnArnoldFoundationrsquosSocialProgramsThatWork-forsocialservicesprograms
TheUSDepartmentofEducationrsquosWhatWorksClearinghouse-foreducationinterventions
TheUSDepartmentofHealthandHumanServicesrsquoResearch-TestedInterventionPrograms(RTIPs)-forcancercontrolinterventions
TheUSDepartmentofHealthandHumanServicesrsquoTeenPregnancyPreventionEvidenceReview(TPPEvidenceReview)-forsexualriskbehaviorandsexualhealthoutcomes
TheUSDepartmentofJusticersquosCrimeSolutionsgov-forcriminaljusticejuvenilejusticeandcrimevictimservices
TheUSSubstanceAbuseandMentalHealthServicesAdministrationrsquosNationalRegistryofEvidence-basedProgramsandPractices(NREPP)-substanceuseandmentalhealthinterventions
TheUniversityofWisconsinPopulationHealthInstituteandRobertWoodJohnsonFoundationrsquosCountyHealthRankingsandRoadmapsWhatWorksforHealth(WhatWorksforHealth)-healthbehaviorsclinicalcaresocialandeconomicfactorsandthephysicalenvironment
UKWhatWorksCentresTheBritishgovernmenthascharteredanetworkofsubject-specificorganizationswhichcollectsynthesizeandreportoutevidenceontheeffectivenessofpolicyprogramsandpractices
NationalInstituteforHealthandCareExcellence(NICE)-healthandsocialcare
Reviewtheevidenceandcollectdetails
24
SuttonTrustEducationalEndowmentFoundation-educationalachievement
CollegeofPolicingWhatWorksCentreforCrimeReduction-crimereduction
EarlyInterventionFoundation-earlyintervention
WhatWorksCentreforLocalEconomicGrowth-localeconomicgrowth
CentreforAgeingBetter-qualityoflifeforolderpeople
WhatWorksCentreforWellbeing-wellbeing
WilsonSheehanLabforEconomicOpportunities(LEO)UniversityofNotreDameProvideslinkstotheirevaluationsofinterventionsincriminaljusticeself-sufficiencyeducationhealthandhousingandhomelessness
ClearinghouseforLaborEvaluationandResearchUSDepartmentofLaborProvidesresearchonworkforceandlabor-relatedpoliciesandprograms
J-PALEvaluationsAbdulLatifJameelPovertyActionLabStudiesonpoverty-reducinginterventionsfromaroundtheworld
CampbellSystematicReviewsCampbellCollaborationSystematicreviewofpolicyinterventionsbyscholarlycoordinatinggroupsinsevenissueareascrimeandjusticeeducationinternationaldevelopmentmethodssocialwelfareknowledgetranslationandimplementationanddisability
Benefit-CostResultsWashingtonStateInstituteforPublicPolicyAWashingtonState-charteredprogramtoestablishthelikelihoodofrelativecostsandbenefitsforprogramsacrossthefollowingareasjuvenilejusticeadultcriminaljusticechildwelfarepre-Kto12educationchildrensmentalhealthhealthcaresubstanceusedisordersadultmentalhealthpublichealthandPreventionworkforcedevelopmentandhighereducation
EffectiveInterventionsCentersforDiseaseControlandPreventionRepositoryforevidence-informedHIVpreventioncareinterventions(withstepsonhowtoimplement)
Journalsandindexes
OnceyouhavereviewedrelevantinformationaccessiblethroughclearinghousesitisworthtakingalookatindividualstudiespublishedasscholarlyarticlesYourreviewerswilldothisbysearchingarticleindexesandacademicjournalsUnfortunatelyanumberofusefulacademicjournalsandindexingplatformsmaynotbegenerallyaccessibletoyourreviewersifyoudonothaveasubscriptiontothemToaddressthisconsiderpartneringwithalocal
Reviewtheevidenceandcollectdetails
25
collegeoruniversitytoprovideyouwithalistofarticlesfromthefollowingsources(Thisisalsoanexcellentopportunitytoseekresearchsupportfromanyacademicstakeholdersyouidentifiedpreviously)
Indexes
GoogleScholarincludesmostpeer-reviewedonlineacademicjournalsandbooksconferencepapersthesesanddissertationspreprintsabstractstechnicalreportsandotherscholarlyliteratureincludingcourtopinionsandpatentsItallowsyoutospecifyapublicationrangeinyoursearchandprovideslinkstolikelysourcesforalllistedarticles
PolicyFileIndexisauniqueresourceforUSpublicpolicyresearchinthatitgrantsusersaccesstotimelyupdatedinformationfromover350publicpolicythinktanksnongovernmentalorganizationsresearchinstitutesuniversitycentersadvocacygroupsandotherentities
PolicyArchiveworkstocollectanddisseminatesummariesandfulltextsvideosreportsbriefsandmultimediamaterialofthinktankuniversitygovernmentandfoundation-fundedpolicyresearch
Anothersourceforhigh-qualityresearchisthereportsoftheCongressionalResearchServiceYoucanfindtheirreportsaggregatedhere
Specificjournals
Oftenthemostusefulacademicjournalsforunearthingpolicyorprogramreviewswillbeineachspecificsubjectarea--thatisineducationjournalsforeducationpoliciesYoucanuseageneralindexlikeGoogleScholartobegintolearnwhichjournalsaremostoftenusefulforyoursearchHoweveryoucanalsosearchthefollowingtwogeneralistpolicypublicationsforissuesandarticlesrelatedtoyourtopic
PublicAdministrationReviewJournalofPolicyAnalysisandManagement
Newssourcessubjectmatterexpertsandinterviews
AlthoughitisvaluabletoknowwhathasbeenexploredthroughascholarlyresearchframeworkitisalsousefultolearnwhatkindofinterventionsarebeingtriedinthefieldrightnowandhowpeoplefeelaboutthesetrialsSearchsubscriptionnewsaggregatorslikeLexisNexisorfreenewsaggregatorslikeGoogleNewsorYahooNewsforyourissueareaplusthewordpolicyorprogramtoseewhatkindsofinterventionsarebeingcoveredinthenews
Reviewtheevidenceandcollectdetails
26
DonotforgettocontacttheadvisorygroupmemberswhowerechosenfortheirsubjectmatterexpertiseInadditiontoprovidinggoodmaterialthroughinterviewsthemselvestheymayhaveideasaboutothersubjectmatterexpertsyoushouldcontacttoaskaboutpotentialinterventions
Peercitiesandnetworks
InadditiontoreadingabouttheworkofothercitiesitisidealifyouareabletospeakwithpeoplewhoareinvolvedwithcurrentimplementationsofinterestinginterventionsInyourcityyoulikelyhaverelationshipswithyourneighborsbutitcanbehelpfultolookoutsideofyourregionforsolutionsaswellTofindcounterpartsinothercitieswhoareworkingonaprogramyouwouldliketolearnmoreaboutyoucancontactstaffinintermediaryorganizations(likeGovExoryourprofessionalassociations)whomightbeinapositiontoputyouintouchimmediatelyorwhomightseeanothergoodrouteforsettingupaconversationHereisalistofnetworksthatmaybehelpful
InternationalCityCountyManagementAssociation(ICMA)NationalAssociationofCounties(NACO)NationalLeagueofCitiesUSConferenceofMayorsWhatWorksCities
Collectthedetails
Onceyourreviewershavefoundevidenceonpossiblepoliciesorprogramstoimplementtheyshouldcapturekeydetailsthatwillhelpdecisionmakersunderstandthepotentialeffectivenessandapplicabilityofeachpotentialintervention
Foreachsourcethathasprovidedinformationyouwanttoconveyusethetemplatebelowtocaptureasmanyofthefollowingdetailsaspossible(Youdonotneedtocompletethisforsourcesthatprovideduplicativeinformationorareotherwisenotthatuseful)
Programorpolicyname (egldquoHousingFirstrdquoorldquoblightedpropertiesregistryrdquo)
Sourcetype (egjournalarticleinterviewsubjectmatterexpertstudyassociationcommunityparticipationprocess)
Sourcedetails (egarticletitleandpublicationnameandtitleofinterviewee)
Evaluationmethodology
(egrandomizedevaluationcomparativeevaluationoftwodifferentgroupsbeforeafteranalysis)
Evidencelevel (egLevel12345)
Programorpolicy
Reviewtheevidenceandcollectdetails
27
description
Programorpolicygoal (1sentencecapturingtheprincipalaimoftheintervention)
Existingimplementations
(WherehastheinterventionbeentriedListasmanysitesasyouareawareof)
Characteristicsofimplementationsites
(ForeachimplementationsiteofferbasicgeographicIDmdashcityandstateThenyouwillwanttocollectthedetailswhicharerelevanttoyourdecisionmakersrsquoconsiderationsaboutwhatmakesalocationldquosimilartordquoorldquodifferentfromrdquoyoursituationYouwillatleastwanttoincludeforeachimplementationsitePopulationnumberPopulationdensity(ruralurbansuburban)Socio-economiccharacteristicsBeforestartingaskyourdecisionmakersaboutanyparticularcharacteristicsofprogramimplementationsitestheywouldwanttoknowbeforeseekingtoreplicate)
Characteristicsofimplementationpartners
(Anyavailabledetailsaboutorganizationsinstitutionsorpeoplewhowereinvolvedwithimplementingtheintervention-egindependentchildcarecenterpublicelementaryschoolcitypublichealthnurses)
Characteristicsoftargetpopulationeligibleprogramparticipants
(Anyavailabledetailsaboutpeoplewhoarethebeneficiariesoftheprogramorotherwiseincludedoraffectedbytheintervention-egnumberofprogramparticipantscategoriesofpropertyownersaffected)
Policyorprogramduration
(HowlonghastheinterventionbeeninplaceineachimplementationsiteHasitendedandifsowhy)
Whatbenefitscamefromeachimplementation
(Anyobservedpositiveeffectsoftheinterventionincludingeffectsize)
Statisticalestimatesofinterventioneffectiveness
(Anyavailablestatisticalestimatesoftheinterventionrsquoseffects)
Whatdrawbackswereobserved
(Anyavailabledetailsaboutunintendedorunavoidablenegativeeffectsoftheintervention)
Programorpolicycosts
(Anyavailabledetailsaboutprogramcostsperimplementation)
Reviewtheevidenceandcollectdetails
28
Reviewtheevidenceandcollectdetails
29
DecideonapolicytoimplementbasedonlocalfactorsTogetthecriticalbuy-infromleadershiptoimplementanewpolicyorprogramtheprojectleadshouldscheduleameetingwithdecisionmakerstopresenttheevidencesurroundingthepolicyoptionscapturedinthetemplateDependingonyourinsightintoyourcityleadershipswillingnesstomakechangeyoumaywishtosharetheresearchyouhaveconductedontherootcausesoftheproblemandsomeofyouranalysisontheimpactofcurrenteffortstodate
Considertheapplicabilityofanapproach
AsdiscussedinthepreviouschapterpolicymakersoftenlookforevidenceofeffectiveprogramsinothercitiesbutoftenthequestionarisesofhowgeneralizableaprogramisfromonecitytoanotherTheAbdulLatifJameelPovertyActionLab(J-PAL)suggeststhatinreviewingevidencefromanotherlocationitishelpfultoidentifythespecificmechanismthatwasused(deliveryofprogramorserviceinformationsharingetc)identifythelocalconditionsthatwereinplaceobservethebehaviorchangethatresultedfromthemechanismandfinallytounderstandlocalcontextandconditions(legalorcostconstraintsculturalnormsetc)Thissetofpresentationelementscanbedrawnfromthelistofdetailscollectedintheresearchtemplate
EducationpolicymakersinChicagoILusedfindingsfromprogramsinKenyaandIndiainwhichchildrenwhohadfallenbehindtheirpeersweregiveneducationalsupporttargetedtotheirleveloflearningThemechanismusedwastargetedteachingandthereweresimilarconditionspresentinallthreecountriesAlthoughsomestudentshadfallenbehindinthecurriculumteacherswereincentivizedtoteachgrade-levelmaterialsConsiderationsofhowthelocalcontextinChicagomightaffectimplementationofatargetedteachingprogramincludedunderstandinghowtheneighborhoodsthatthestudentswerefrommayhaveaffectedtheireducationaloutcomesandthehighcostofteachinginsmallgroupsInChicagoMatchEducationprovidedtwo-on-onetutoringwhichmatchedstudentswithwell-educatedindividualswhohavenothadformalteachertrainingforalower-costmodelTheresultswerepositiveStudentswhoweretutoredperformedbetterinmathandweremorelikelytopasshighschoolclasses
ThegeneralizabilityframeworkproposedbyJ-PALisbelowUsethisasatemplateasyouconsiderpotentialpolicysolutions
Decideonapolicytoimplementbasedonlocalfactors
30
SourceTheGeneralizabilityPuzzle
OnceyouhavenarrowedthepolicyoptionsdiscussthequestionsbelowforeachoptionsyouareconsideringThisstepiscriticalasunderstandinglocalcontextissoimportantinsuccessfulpolicyimplementation
CommunityContextQuestionnaire
Localcontext
DoesthisoptionmakesenseinourcommunityIsthisaonesizefitsallapproachIsanyparticulargroupabsentfromthisapproachIfsoisthatok
Youranswers
Effectiveness
WhatwouldweexpecttoobserveifthepilotisaseffectivehereasinCityX
Youranswers
Timing
DoesthetimingmakesenseIsthistherighttimetoimplementthispolicychange
Decideonapolicytoimplementbasedonlocalfactors
31
WherearethepotentialroadblocksthatcouldaffectthescheduleIsthereanywaytomitigatethisHowlongwouldittaketoseeanimpact
Youranswers
Implementation
DowehavethestaffcapacitytoimplementthisArethereadditionalpartnersthatyoucanbringtothetabletosupportimplementationDoesthismakesensegivenourformofgovernment
Youranswers
Authority
DoesourcityhavetheauthoritytoenactthisIfsowhatistheappropriatemechanismAretherelegalbarriersArethereanyexistingpreemptionissuesfromthestateorfederalgovernmentCanthosebeaddressedthroughlegislationadvocacyIstherepoliticalsupportoroppositiontothis
Youranswers
Cost
HowmuchwillthiscostannuallyHowwillwepayforitCanthepolicybeimplementedwithexistingfunding
IfsoisthisfundingreliableHowwouldthispolicyoptionaffectourcitysbudgetintheshortandlongtermWhatotherfundingoptionscanweconsiderIsthereanalternativethatwouldcostlessandgiveussimilarresultsWillanyconstituencybeadverselyaffectediffundingforprogramsthatimpactthemisshiftedtoanotherprogrampolicyorinitiative
Youranswers
Decideonapolicytoimplementbasedonlocalfactors
32
EstablishaframeworktomonitorprogressNowthatyouhavereviewedtheevidenceconsideredlocalcontextandchosenyourpolicyitistimetodeterminehowyourcitywillmonitorprogressofthisnewapproachCitystaffandleadersneedtoestablishtheinfrastructureforongoingcontinuousdecisionmakingbasedonwhetherthepolicychangeisleadingtothedesiredoutcomesforresidentsandwhetherthechangeismeaningfulandrationaltofrontlineworkersSettingarbitrarygoalsatthelocallevelthatconflictwithgoalsatotherlevelsisconfusingtotheindividualsresponsibleforachievingthegoalsGoalsthataresettoohightoachievecanbedemoralizingtostaff
ThissectionincludesstepstosetupavenueinwhichCitystaffandleadershiproutinelyvisitkeyindicatorstodeterminewhatisorisnotworkingandtomakedecisionsinreal-timeFormorereadingonperformancemanagementpleasevisittheGovExPerformanceManagementGettingStartedGuide
InperformancemanagementtermssuchasindicatormeasureandmetricmaybeusedinterchangeablyInthisguidewewilluseindicatorInyourworkthemostimportantthingistobeconsistentinwhichevertermyouuseThisvideomaybehelpfulinunderstandingthisconceptandcommunicatingittoothers
Establishthefundamentalsandgatheryourteam
FirstdeterminewhetherthereisanestablishedvenueorregularmeetingscheduleduringwhichyoucandiscussprogressIfyourcityhasanexistingperformancemanagementstructureonarelatedtopicaddingadiscussionoftheprogressofpolicychangetoyouragendacanbeagreatideaIfthiswillbeastandalonemeetingdecidehowfrequentlytheprojectteamanddecisionmakerswillmeetandputthosemeetingsonthecalendarfortheentirelengthoftheproject
DeterminewhowillstaffandattendroutinemeetingsonCityprogressThesemeetingsmayincludetheMayorandorCityManagerdepartmentheadstaffwhoareimplementingthepolicychangeandanalysts
Identifyindicators
ItiscriticalthatindicatorsbeanimportantcityandcommunitypriorityaswellassomethingtheCityplaysamajorroleinaddressingTheremaybeinstancesinwhichtheCitywillneedtoreachouttoorcollaboratewithexternalpartnerstogetamorecompletepictureofprogressontheoverallpolicychange
Establishaframeworktomonitorprogress
33
Anoutcomeindicatordescribesthedesiredeffectofthetargetedpopulation
ExampleBellevuesfamilyfriendlyneighborhoodsindicatormeasuresthepercentofresidentswhobelievetheyliveinaneighborhoodthatsupportsfamiliesparticularlythosewithchildren
WithyourkeypartnersrevisityourproblemstatementcurrentstateandfuturestateexerciseandbegintoidentifytheoutcomeindicatorsthatwillillustratesuccessFornowchooseasingletop-levelkeyperformanceindicator(KPI)ItiscriticalthattheKPI-thepieceyouwillmeasure-beapriorityforthecommunityandsomethingtheCityplaysamajorroleinaddressing
Outputindicatorsmeasuretheprocessofprogramorservicedelivery
ExampleSanDiegoMayorKevinFaulconersgoaltorepair1000milesofstreetby2020
ReviewingtheadministrativedatasetinventorybegintoparedowntherelevantprogramsanddatasetsthatsupportyourKPIWhatarethemeasuresofeffectivenessforeachprogramTheseareyouroutputindicators
Thediagrambelowillustratestherelationshipbetweenoutputandoutcomeindicators
ThinkaboutimplementationindicatorsasmeasuringstepsintheprogramsrolloutthatarenecessarytoobtaindesiredchangesImplementationoutcomescanfocusontheinterventionstaffdeliveringtheinterventionanorganizationorsystemandcanaddressfeasibilityadaptabilitysustainabilityandcostTwokeyquestionstoconsiderwhendevelopingimplementationindicatorsare
1 Istheinterventionacceptableandappropriatetothecommunityand
2 IsitfeasibletoimplementinterventionXatthistimewiththeseresources
Establishaframeworktomonitorprogress
34
Amoreextensivelistwithdefinitionsofpotentialimplementationoutcomesislinkedhere
UsethetablebelowtochoosethedesiredimplementationoutcomesforyourpilotandtoplanyourmeasurementtechniqueThisshouldincludelevelofanalysis(individualproviderorconsumerorganizationorsetting)yourmeasurementtechnique(surveysobservationinterviewfocusgroupsadministrativedataanalysis)andthephasewhenyouwillcollectthisinformation(earlymidlate)Theremaybemorethanonelevelofanalysismeasurementtechniqueandphaseofdatacollection
Implementationoutcome
Definitionofoutcome
Levelofanalysisandmeasurement
Yourmeasurementtechnique
Phaseofpilotwhenyouwillcollectdata
Usethespacebelowtocaptureyourindicatorwishlist
Indicatorwishlist
Outcomes_____________________________________________________________________________
Outputs________________________________________________________________________________
Implementationoutcomes______________________________________________________________
Setapublichigh-levelgoal
Thehigh-levelgoaliscriticalinestablishingdirectiontogalvanizethepeoplewhowillbedoingtheworkWhendiscussingthegoalusetheMayorsbullypulpitLookforkeymilestonesinCitygovernmenttosharethegoalstatementsuchasthebeginningofaMayoraladministrationaStateoftheCityspeechorotherpublicremarks
GoalsshouldbeldquoSMARTrdquo
Specificaspossible(innamingthepopulationwhowillbebetterofftheactionthecitywilltake)
Measurable
Achievable
Establishaframeworktomonitorprogress
35
Results-focused
Time-bound
ThegoalstatementshouldincludeyouroutcomeindicatorlikeChattanoogasgoalldquoWewillendchronicveteranhomelessnessbyDecember2016rdquoAdiagramtohepyoubuildaSMARTgoalaroundyouroutcomeindicatorisbelow
Component Description Yourturn
Specific WhowillbebetteroffWhatactionthecitywilltake
Measurable Howwillyoumeasuresuccess
Achievable Isthegoalattainable
Results-focused Whatistheexpectedresult
Time-bound When
Setrealisticambitioustargetsandtimelinesforeachindicator
ToestablisharealisticandambitioustargetforyourgoalstatementusetheadministrativedatacollectedforyourchosenKPItoreviewactualCityperformanceinthemostrecentyearcomparedwiththethreeorfourmostrecentyearsworthofdataBegintocalculatesummarystatistics(egtotalminimumaveragemedianmaximum)toidentifytrendsCreatecumulativeandmonthlychartstoidentifytrendsandanomaliesBasedonyouranalysisofhistoricaldataignoretheanomaliesandassumetheproductivitytrendswillfollowthesameoverallpatternfrommonth-to-monthFinallyapplyanincreasedproductivityfactortotheperformanceeachmonthtoforecastfutureprogressBasedonthisanalysischooseatargetthatinspiresimprovementwhileaccountingforreality
Manycitiesthataremorematureintheirperformancemanagementpracticesrecognizetheimportanceoftakingacommunity-levelapproachtoimprovingoutcomesforresidentsThesecitiesarecreatinggoaltaxonomiesthatincludemeasuresrelatedtopublicsafetyhousingandevenrecreationCitiesdriveprogressonthesegoalsbyimplementingstrategiesthatengagemultipledepartmentsandcollaboratewithserviceprovidersinthecommunity
EventuallyyouwilldevelopgoalstatementsfortheoutputsinordertocapturetheefforttoachievetheoutcomecontainedinyourgoalstatementYoursetofindicatorsshouldbecomposedofshort-andinterim-termsupportivemeasureswithaccuratetargetsandrelevanttimelinesAnexamplegoaltaxonomyfromtheCityofLasVegassResultsVegasprogramisbelowUsethistaxonomyasamodelforyourownsuiteofindicatorsandgoalstatements
Establishaframeworktomonitorprogress
36
Yourturn
Thetemplatebelowismeanttohelpyoucraftagoalstatementforeachsupportingmeasure
Goalstatement
Departments
Outcome
Activities
KPI
Target
Timeframe
Conductanddiscussrelevantanalysis
ThrougharoutineseriesofmeetingscontinuallymonitorprogressofKPIssupportingindicatorsandanyotherdatacollectedsuchasqualitativedatatoidentifytrendsthatwillindicatewhetherthechosenpolicyishavingthedesiredimpactThesedatapointsshoulddrivethediscussionduringthesemeetingspointingtoopportunitiesforfurtherinquiryinto
Establishaframeworktomonitorprogress
37
whatisdrivingtheobservedtrendsAnalystsshouldbeencouragedandtrainedtoaskquestionsofthedepartmentsproducingthedataandconductsitevisitswhenpossibletodeepentheirunderstandingofthedataPresentationoftheanalysisshouldfocusononeideaatatimetokeeptheconversationfocusedonkeypointsandspecificopportunitiesforimprovementYoumayconsiderdevelopingapublicfacingdashboardtosharethisprogressmorebroadly
AgeneralcomparisonofthestatusquodatawithwhatwasobservedduringthepilotcanprovidedecisionmakerswithaquickunderstandingofthedifferencesbetweentheoldandnewapproachesalthoughthiswillnotshowcausalityOftenthesecomparisonswillincludeoperationalmetricssuchasdaysordollarssavedbutrememberthattheimpactonpeoplewillbemostimportantIfyouareworkingwithanevaluationpartnertheywillsupportthisstep
Usethespacebelowtosummarizetheresultsofyouranalysis
MeasureTargetvalue(seesectiononsettingtargetsabove)
Pilotvalue(whatdidyouobserveduringthepilot)
Historicalaverage(ifavailable)
Results(WhoisbetteroffIsanyonenegativelyimpacted1-2sentencesreassessingyourgoalstatement)
DocumentyourprocessandprogressincludingfeedbackfromtheparticipantsimplementationpartnersandtheprojectteamDidanythingunexpectedoccurduringthepilotArethereinternalandexternalfactorsthatmayaffectyouroutcomesThisiswheretheimplementationoutcomescanbecanhelpfulIftheinterventiondidnotgoasplannedconsiderwhetheritisbecausetheinterventionwasnotacceptableorfeasible
UsethetemplatebelowcreatedbyDenversPeakAcademytocapturethisinformationIfthe306090daytimelinedoesnotmakesenseforyourcityfeelfreetocustomizeit
CurrentState
FutureState
30Day
60Day
90day
Whatwentwell(+)
Whatdidnotgowellshouldbechanged(-)
IfaquestionordiscussionpointarisesthatcannotbeansweredinthemeetinganassignedstaffmembershouldcapturethequestionduringthemeetingAfterwardthefollowupitemsshouldbesenttorelevantpartnersandthenextmeetingshouldbeginwithadiscussionof
Establishaframeworktomonitorprogress
38
thoseitems
Takeaction
DiscussionsaboutwhetheryourindicatorsaremovingintherightdirectionandwhatmightbedrivingtrendsmayleadtoadiscoverythattherearenotenoughresourcesallocatedtothispolicyorissueTheremaybelimitedfundingforstaffprogrammingortechnologythatisdrivingthedatainthewrongdirectionIfCityleadershipisintheroomthisisagreattimetomakethecaseforadditionalfundingduringthebudgetseasonBycollectinganalyzinganddiscussingdataonthispolicyroutinelyyouarebuildingevidencetomakethecasefortherightleveloffundingtoreachtheestablishedtargets
Establishaframeworktomonitorprogress
39
Assessdataaccessibilityqualityandsecurity
Determineaccessibilityandqualityofdata
OnceyouhavecompiledyourwishlistofindicatorsitistimetodeterminetheaccessibilityandqualityofthedataTopreventcomplicationyoumaywishtoprioritizerelevantindicatorsforwhichyouarealreadycollectingdataAvoidthetemptationofadoptingatechnologysolutiontocollectnewdataatthispointsuchasaphotosurveytoolorotherappFornowyouaretryingtodeterminewhichdatasetsarerelevantaccessibleandhighqualityBesuretoworkwithyourkeyinternalpartnerswhileyouareidentifyingyourmeasuresDepartmentstaffwhoworkonprogramsarefamiliarwiththeindicatorsthatwillaccuratelysupporttheoutcomesanddatabaseadministratorswillbevaluablewhendiscussingaccessibilityandqualityofthedata
Abrieflistofconsiderationsfortheaccessibilityofyourdataisbelow
ArewecurrentlycollectingthisdatasetorwillwebecollectingitforthefirsttimeHowfarbackdoestherecordgo(timehorizon)WouldweneednewtechnologytocollectoranalyzethedataCanthedatabecollectedmanuallyorcanitbeautomatedHowfrequentlyisthedataproduced(continuouslyhourlydailyweeklymonthlyquarterlyyearlyetc)IsthedataavailablefortheentireCitypopulationorasubsetofthepopulationHowgranularisthedata(individuallevelCensusblockneighborhoodcitywideetc)DoesthedatacontainsensitiveinformationIsthereaproxymeasurethatwecanusefornowIsthedatamachinereadable
AsnotedbytheWilsonSheehanLabforEconomicOpportunities(LEO)attheUniversityofNotreDameleveragingadministrativedatacansignificantlyreducethecostsandtimelinesofrigorousevaluationCitiescanestablishaninternaldatasharingagreementacrossdepartmentsincludinglanguagethatenablesadministrativedatatobeusedforresearchpurposesCitiescanalsoidentifyexternaladministrativedata-holderssuchasacountyschooldistrictpolicedepartmenthospitaloranotherstatethatmayhaveoutcomedataimportantformeasuringprogramimpactCitiescanleverageexistingrelationshipsandorworkwithresearchinstitutionstodevelopstreamlineddatasharingprocessesandagreementswiththeseexternalentities
Assessdataaccessibilityqualityandsecurity
40
Evaluatethequalityofyourdata
QualityisasimportantasaccessibilityToevaluatedataqualityyoumaywishtocreatearubricliketheonebelow
Value ValueDescription
3 ExcellentDataQualityThedataiscompleteaccurateandupdatedregularly
2 AdequateDataQualityThedataissomewhatcompleteaccurateandupdatedfairlyregularly
1 PoorDataQualityThedataisnotcompleteaccurateorupdatedregularly
ThisisagreattimetobuildonthedatainventorythatyoustartedinthegatherinformationpageSuggestedadditionalfieldstoaddtoyourdatainventorytemplatearebelow
Program DatasetTitle
DatasetDescription DataSource
Departmentorexternalpartner
DoestheCityalreadycollectthisdata
Whatisthenameoftheprogram
Human-readablenameofthedataset
Shouldbeunderstandablebynon-technicalusers
Whatistheoriginalsystemapplicationfilethathousesit
Departmentorexternalpartnerthatmaintainsthedata
Isthisdataalreadycollected
Updatefrequency-ThefileformatforthedatasetTypicallycsvxlsxshapefiletxtetc
Format-Howoftenthedatasetisupdated
Timerange-Theperiodoftimecoveredinthisparticulardataset
Datause-Howisthisdatasetcommonlyused
Datausers-Whotypicallyusesthisdataset
Dataownerstitle-Titleofthedataowner
Dataownersposition-DataOwnersjobtitle
Dataownersemailaddress-DataOwnersemailaddressWhomanagesthedataandorisresponsibleforgrantingpermissiontoaccessthedataWhounderstandswhatthedatasetincludesandcananswerquestionsaboutit
Dataqualityconcerns-Describetheconcernsyoumayhaveaboutthequalityofthedataset
Assessdataaccessibilityqualityandsecurity
41
Division-DivisionwherethedataismaintainedIfmorethanonedivisionisresponsiblelisttheprimaryownerofthedataset
Sensitivedatacomments-DescribetheissuesthatwouldpreventthecityfromsharingthedatapubliclyCommonsensitivityconcernsincludeprivacyviolationssecurityissuesorhighcostorstaffdemandstopostthedata
Publishingstatus-Thistheplacetoputanycommentsregardingthecurrentstatusofthedataset
Publishedlink-Thisisthelinktothepublisheddatasetontheopendataportalorcitywebsite
Reviewdatasecurityandprivacyprotectionprotocols
OftenapolicychangealsoinvolvesachangeinthedatathatisbeingcollectedhowthatdataisbeinghandledwhohasaccesstoitandahostofotheradjustmentsAsyoutakestockofthedataassetsthatareavailableforyourinitiativeitisvitaltoalsothinkthroughwhatwillbedonetoprotectthatdataandtheprivacyofthepeopleitrepresentsAbigpartofgooddatasecurityinvolvesstrongdatagovernancepracticesthatdefineandidentifysensitivedataandconsciouslyworktobalancethevalueandriskofstoringsuchsensitivedataOnewayofevaluatingyourdatacomesfromcaselawrelatedtotheUSFreedomofInformationActwhichwehavetranslatedtothechecklistbelow
1 Woulddisclosureresultinasubstantialinvasionofprivacy2 Haveweconsideredtheextentorvalueofthepublicinterestandthepurposeorobject
oftheindividualsseekingdisclosure3 Istheinformationavailablefromothersources4 Wastheinformationgivenwithanexpectationofconfidentiality5 Isitpossibletomouldreliefsoastolimittheinvasionofindividualprivacy
TheseriskscouldbemitigatedusingavarietyofstrategiesfromaggregatingdatatoobfuscatingindividualstoredactingfieldsthatmayincludePersonallyIdentifiableInformation(PII)orre-evaluatingyourdatacollectionmethodstofindalternativewaysofgettingtheinformationyouneedThesestepsshouldbecodifiedandmanagedbyadatagovernancecommitteeifpossibleForthesensitivedatathatisdeemedessentialcreateaplanaroundpotentialdatabreachesinordertoquicklyandconfidentlyrespondtoemergencysituations
ProactivelypreparingforthesesituationsisoftenthebeststrategyfordatasecuritybutdonotforgetthetechnicalsideInvolveyourITadministratorsinconversationsaboutdatastorageandaccessensuringthattheywillbeprotectedbyenterprise-gradecybersecuritysystems
Assessdataaccessibilityqualityandsecurity
42
Assessdataaccessibilityqualityandsecurity
43
Designimplementationandevaluation
Designtheimplementation
NowthatyouhaveyouraccountabilityplanandkeyindicatorsitistimetogetspecificaboutyourplanforimplementationThetemplatebelowwillhelpyouthinkthroughthespecificmechanismsofimplementingpolicychange
UnderstandingorganizationsandsystemsiscriticaltounderstandingandmakingdecisionsaboutwhereandhowinterventionsshouldbeappliedApolicyimplementedinasinglesettingmayshowpromisingresultsbutimplementingmultiplepoliciesandprogramsatthesametimewithdifferentactorsresponsibleforeachaspectcanhelpensuresustainablebehaviorchange
InSomervilleMAShapeUpSomerville(SUS)isacitywidepublichealthprogramthatisimplementedacrossmultiplelevelsofinfluenceandsystemsthroughinterventionsbeforeduringandafterschooltoincludechangestobuiltenvironmentsuchasbikelanesregionalmasstransportationandphysicaleducationequipmentinschoolsandgymstoincreasephysicalactivitywellnesspoliciesandschoolgardensnutritioneducationduringandafterschoolandhealthyrestaurantprogramstosupporthealthyfoodandhealthyeatinghabits
Usethetemplatebelowtoplanyourimplementation
Policyimplementationplanningtemplate
ListthespecificdesiredoutcomesInthepreviousstepyouworkedwithdepartmentstafftoestablishspecificmeasurableoutcomesandinterimindicatorsPrioritizetheindicatorsthatwilldemonstrateprogressUsethespacebelowtocapturethedesiredoutcomesofthepolicychange
ExampleShapeupSomervillebeganwiththegoalofpreventingobesityin1stthrough3rdgraders
Yourturn
____________________________________________________________________________
____________________________________________________________________________
Designimplementationandevaluation
44
SpecifythetargetpopulationAreyoudesigninganinterventionthattargetsonlycertainsitesAreyouplanningonimplementingaprogramorinitiativeforaspecifictargetaudienceClearlyspecifythepopulationyouplantotarget
IfyouplantoconductarigorousimpactevaluationasnotedbytheWilsonSheehanLabforEconomicOpportunitiesitisimportanttoestablishconcreteobjectiveeligibilitycriteriaandensurethatthesecriteriaareappliedconsistentlybyallstaffduringscreeningandintakeItisalsovitaltoidentifytheindividualswhoareeligibleforaparticularprogramorserviceevenifeveryoneeligiblecannotbeservedbyexistingresourcesThishelpscitiesidentifyanyunmetneedsandguidefutureresourceallocationwhilealsocreatingthepossibilityofarigorousimpactevaluationbyidentifyingapotentialcomparisongroupFinallywhenconductingscreeningandintakeitisimperativetocollectandmaintainbasicinformation(egnamedateofbirthgender)aboutallprogram-eligibleindividualsnotjustthosewhoparticipateintheprogram
ExampleIntheShapeUpSomervilleexampletheprogrambeganbytargeting1stthrough3rdgraders
Yourturn
____________________________________________________________________________
____________________________________________________________________________
DefinethelocationWherewillthepolicychangetakeplaceWillimplementationtakeplaceatonesiteormultiplesites
ExampleShapeUpSomervilleinterventionsweredeliveredinschools
Yourturn
____________________________________________________________________________
____________________________________________________________________________
ClarifyimplementationpartnersandarticulatetheirrolesWhatarethekeyorganizationsinstitutionsorpeopleresponsibleforimplementationTheprojectteamshouldbeasclearandspecificaspossibleinprovidingadescriptionoftheconcreteactivitiesthatimplementationpartnersaremeanttocarryoutPracticeprofilesmaybeahelpfultooltohelpdefinewhatisrequiredoftheactorswhoareresponsibleforimplementingtheintervention
Designimplementationandevaluation
45
ExampleInSomervillemayoralleadershipandengagementofmultiplecityagencies(publichealthprimarycareschoolsCambridgeHealthAllianceetc)streamlinedandconnectedthevariousShapeUpSomervilleinitiativesandgrantswhichhelpedtokeepthetalentandenergyfocusedonhealthyeatingandactivelivingforallsectorsofSomervilleSUSpartnersultimatelyincludedschoolsnonprofitsCityandstateagenciesandnationalfoundations
Yourturn
____________________________________________________________________________
____________________________________________________________________________
DescribetheinterventionWhatarethestepsactivitiesorprocessesRememberthatthemechanismthespecificstepsorprocessthatleadtobehaviorchangeiskeytounderstandingtheeffectivenessofthepolicychangeandsharingtheresultswithothers
ExampleGoalsofShapeUpSomervilleweretoinfluenceeverypartofanearlyelementaryschoolchildrsquosdayndashbeforeduringandafter-schooltopromotehealthyeatingphysicalactivityandtheirweightstatusInterventionincludedfoodserviceenhancementstoimprovebreakfastandlunchandstaffprofessionaldevelopmentwalktoschoolactivitiesSUSclassroomcurriculumandprofessionaldevelopmentschoolwellnesspolicydevelopmentSUSafter-schoolcurriculumandprofessionaldevelopmentoutreachandeducationtohomethroughmaterialsforumseventsldquoSUSapprovedrdquorestaurantscommunityoutreachandcapacitybuildingthroughpolicydevelopmenttrainingsmediaplacements
Yourturn
____________________________________________________________________________
____________________________________________________________________________
ClarifytimelineandmilestonesfordeliveryandevaluationHowfrequentlywilltheservicesinterventionbedeliveredHowlongwilltheprogramrunWhatisthetimetoexpectedimpacts(eg12monthsafterprogramcompletion)
Designimplementationandevaluation
46
ExampleShapeUpSomervilleledbyTuftsUniversityFriedmanSchoolofNutritionScienceandPolicyconductedaquasi-experimental(nonrandomized)interventionstudyamong1stthrough3rdgradestudentsinSomerville(intervention)andtwocomparisoncommunitiesfor9-monthsduringoneschoolyear(fall2002ndashspring2003)ThestudyshowedpositiveassociationoftheinterventiononchangeinBMISUSreleasedreportssummarizingevaluationsprogresstodateplansforadditionalprojectsandimpactstenandfifteenyearsintotheinitiative
Yourturn
____________________________________________________________________________
____________________________________________________________________________
PlanyouranalysisTheprevioussectionaddressedestablishinganaccountabilityframeworktomonitorprogressandimplementationDeterminewhichindicatorswillbediscussthroughthatprocesswhowillsharethedataandanalysisandhowfeedbackfromthoseconversationswillbeincorporatedbytheprojectteam
ExampleTheMayorrsquosofficeimplementedadata-drivendecisionmakingapproachcalledSomerStatwhichwassynergisticwiththeapproachestakenbytheresearchandevaluationteamsdeployedforthespecificgrants
Yourturn
____________________________________________________________________________
____________________________________________________________________________
PlanforsustainabilityIfthepolicyandprogramchangeissuccessfulhowwilltheprojectteamsustainandexpandtoadditionalsitesorpopulations
ExampleAlthoughmostofthepartnershipsforgedintheearlyyearsweremaintainednewgrantsandcollaborationsexpandedattentiontoolderpublicschoolstudentsactivetransportationandtheGreenLinesubwayextensionthebuiltenvironmentfarmerrsquosmarketeffortssuchasldquoGrowninSomervillerdquowithUnionMainStreeteconomicdevelopmentandhealthdisparitiesaswellasdatacollectionmonitoringandsurveillance
Yourturn
Designimplementationandevaluation
47
____________________________________________________________________________
____________________________________________________________________________
Pilotplanningtemplate
IfyouplantoconductapilotfirstdeterminethepurposeofthepilotDoyouwanttoobservewhetherthepolicychangeresultsinthedesiredbehaviorchangeAreyouconcernedaboutfeasibilityscalabilityorcostDoyouwanttotestwhetheryourdesiredresultscanbeachievedthroughthispolicyapproachClarifythegoalofthepilotandthequestionsyouhopetoanswerThiswillhelpyouidentifythekeyindicatorsthatyouwillusetomeasurewhetherthepilotissuccessfulBeforebeginningestablishabaselineofcurrentperformanceandsetyourtargetsaccordinglybasedonwhatyouhopetoaccomplishthroughthepilot
NextestablishthepilotplanforyourprojectteamincludingmilestonesforteamcheckinstodeterminewhethertomoveaheadorchangecourseThisisseparatefromthemeetingsthatwilloccuraspartoftheaccountabilityframeworkUsethespacebelowtocreateyourtimelineanddevelopyourmeetingschedule
Task Personresponsible Startdate Checkindate Deliverable
Planyourevaluation
WhileyouaredesigningtheimplementationitisalsotimetobegintodevelopyourplanforanalysisAsnotedbytheWilsonSheehanLabforEconomicOpportunitiesdecisionsaboutprogramsetup-includingoutreachscreeningintakedatatrackingandenrollment-candeterminewhetherornotarigorousimpactevaluationwillbepossibleAsdiscussedcityleadersmayfindithelpfultoincludeevaluationpartnersinearlyconversationsaboutprogramsetupDuringpreviousstagesyoumayhaveworkedwitharesearchertoplanyourproposedpolicychangeYoushouldbeclearaboutthepurposeoftheevaluationanddevelopanevaluationplanBelowareadditionalconsiderations
Causationvstrendanalysis
IntheEstablishaframeworktomonitorprogressstagetherewasguidanceontrackingtrendsovertimeItisimportanttonoteherethatmonitoringdataisdifferentfromanalyzingacausaleffectTrendanalysiswillillustratethatchangesareoccurringandmayleadtoquestionsthatcouldbeansweredbyyourevaluationbuttrendanalysisalonedoesnotprovideevidenceaboutwhatisdrivingthosechangesYoumayseechangesinyour
Designimplementationandevaluation
48
outcomeindicatorsbasedonvariablesoutsideoftheCityscontrolForinstanceachangeinfederaltradepolicycanhaveanimpactemploymentratesinyourCityoutsideofyourCitysnewjobtrainingandplacementprogram
ToaddresscausalityyoumustcomparetheoutcomesofindividualswhoreceivedtheinterventionwiththosewhodidnotWithoutacomparisongroupofsimilarindividualswhowerenotexposedtotheinterventionitisimpossibletoknowwhichfactorsareresponsiblefortheoutcomesRandomizedControlTrials(RCTs)inwhichindividualsarerandomlyassignedtotreatmentandcontrolgroupsarethegoldstandardinprovingcausalityIfyoucandemonstratethatoutcomesimprovedforthosewhoreceivedtheinterventionanddidnotimproveatallorimprovedlessforacomparisongroupyoucanbesomewhatcertainthattheinterventionwasthecause
Examplesofothernon-causalimpactevaluationsthatarerigorousandreliableincludecertainquasi-experimentalstudiesusingcomparisongroupdesignssmartinterruptedtimeseriesanalysesthatmodeltrendsovertimeinwaysthatallowestimationofinterventioneffectswelldesignedpre-posttestanalysesneedsassessmentsimplementationstudiesetcFormoreinformationondifferenttypesofevaluationsseetheSageResearchMethodswebsite
Lookfordifferencesingroups
BuildintimetoalsoexaminethepilotdatatoseeifgroupsareexperiencingdifferentresultsReviewingperformancebasedonoutcomemeasuresforapopulationinaggregatemayhidethesubtlewaysthatsomepeoplemaybeaffecteddifferentlysoitisnecessarytocarefullyanalyzethedatafordisparateoutcomesForinstancetheMovingtoOpportunity(MTO)studythatexaminedtheimpactofhousingvouchersonfamilieswholivedinhigh-povertyneighborhoodsfoundthatchildrenwhomovedbeforetheageof13hadbettereducationandhigherincomesasadultsHoweveradultsdidnotseeimprovededucationandemploymentoutcomes
Disaggregatingdataintomorespecificcategoriesofraceandethnicitycanhelpunderstandtrendsanddisparitiesinamoregranularwaywhichcanhelpdecisionmakerstargetpoliciesandinterventionsinamoreequitableway
EarlierintheRoadmapweworkedtoidentifystakeholdersandwhomightbepositivelyandnegativelyimpactedbyachangeinpolicyUsetheresultsofthatexercisetoinvolvethecommunityininterpretingdataanalysisandtogatherqualitativefeedbackfromthecommunitytocallattentiontoissuesForinstanceanincreaseinyouthdetentionmaybeexplainedbyinefficienciesorlackofaccesstotransportation
Designimplementationandevaluation
49
Designimplementationandevaluation
50
Shareyourstoryandsuccess
Sharetheresults
Storytelling
AnintegralstepinyourstrategyissharingtheresultsofyourpilotorpolicychangewithyourcommunityThisincludesdevelopingmessagingaroundtheworkofyourorganizationandcreatingaplantodisseminateyourstoriesWhytellstoriesaboutyourwork
Informandorunderstandasubjectorsituation
Giveneededcontextofdataorbitsofinformation
Relatablestoriesarememorableforreadersviewers
ReadersorviewerswillseetheinformationasmorecompellingwhenthereisademonstratedandrelatablechangeorimpactonthemselvesortheirfriendsneighborsandcolleaguesStoriesareaboutpeopleandgovernmentisfundamentallyabouthelpingpeopleSotellthestoriesofthepeopleyouandyourcitycolleagueshelped
Keepthesetipsinmindasyoudevelopyourstorywithyourteam
WhatwereyourfindingsandwhyshouldanyonecareTellyouraudiencehowtheproblemwassolvedWhathastheimpactbeenonresidentsandcitystafftodateWhatwasthesuccessandweretheremultiplesuccessesalongthewayIfyouhavepreliminaryresultsordatashowingimprovementtosharethisisthetimetodoit
WhowasinvolvedandimpactedReviewthelistofstakeholdersHowwereeachofthemimpactedbytheprogramHumanizeyourstoryandimpactbyaddingadditionalvoicesThesevoicescanbeoneofthefollowing
Apersonorfamilywhowaspositivelyaffectedbythechangesuccess
Aninvolvedthird-partydescribingboththeproblemandthesolution(ieanorganizationnonprofitprovidersbusinesstradeassociationsportsteametc)whoispartofthecommunitythatthegovernmentworkedwithonthechangesuccess
Someonefromthegovernmentitselfwhocanspeakaboutthechangesuccess
TellmoreaboutyourcityDescribeyourcityincludingpopulationdemographicsindustriesetcWhatareitschallengesWhatisitknownforWhatmakesituniqueHowisitsimilarordifferentfromothercitiesintheUS
Shareyourstoryandsuccess
51
DescribeyourcityrsquoschallengeWriteoutthechallengeyourcityisfacing(asdefinedinyourproblemstatement)HowdoesitaffectresidentsHowdoesthisproblemaffecttheworkofcitystaffmembers
AddthedetailsProvidedetailsonyourappliedresearchstrategyTalkaboutwhatyourcityhasdonenthepasttoresolvetheissueDiscusshowyourstrategyhaschangedwhenyoubegantoadoptthisapproachWasmoneysavedorinvested
Usethisstorymapexercisetoorganizeyourthoughtsandguaranteethatyouincludealloftheelementsofatraditionalstory
AfteryoudevelopyourmessagingcreateacommunicationsmarketingandoutreachstrategydesignedtospreadyourmessageYourstrategyshouldwalkyouthroughthefollowingkeyactivities
1 IdentifyandprofileyouraudienceWhencreatinganddisseminatingyourmessageitisalwaysgoodtothinkaboutyouraudienceWhoexactlywouldbeinterestedinwhatyouhavetosayThegeneralpopulationofthecityPeoplewhoworkinnonprofitsBusinessownersItmattersbecausenoteveryonewholivesinacitymaybethetargetoftheinformationBusinessownersmaybemoreinterestedindatashowingeffectsonthebusinessdistrictthanothersandsoonWewillcoverthislaterbutthisstepwillalsohelpyoutothinkofthemosteffectivewaytoreachyouraudience
2 Defineyourcalltoaction(ifyouhaveone)SpendsometimeconsideringwhyyouareinterestedintellingyourstoryDoyousimplywanttoupdatethepubliconyourprogressorcelebrateyoursuccessesAreyouinterestedinyouraudiencechangingitsbehaviororsupportingaspecificinitiativebeinglaunchedbyyourorganizationIfyouwanttohaveyouraudienceactorbehaveinacertainwaybespecificandincorporateitinyourmessaging
Shareyourstoryandsuccess
52
3 SelectperformancemeasuresThinkabouthowyouwoulddetermineifyourcampaignissuccessfulandselectappropriateperformancemeasuresYouwillrevisitthesemeasuresattheendofyourcampaigncommunicationsplanimplementationtodeterminewhetheryouneedtomodifyyourcommunicationsandoutreachstrategyforthefuture
4 MethodstocommunicateyourstoryTherearemanywaystocommunicatesuccessofyourprojectHerearesomesuggestionsonreachingaudiencesItisimportanttodeterminewhichmethodsarethemosteffectiveforyourdesiredaudiencetoreceiveinformationDoyourresearch(PewResearchCenterhasexcellentstudiesonthereachofmediadigitalmediaandmarketing)ForexampleifseniorsareyourtargetpopulationandtheyarelessinclinedtouseInstagram(thatistrue)youmaywanttoconsiderFacebookorthenightlylocalnewsbothofwhichareusedmoreoftenbyseniorsBelowisalistofsomemethodsandtoolsyoucanuse
i Pressreleasestolocalnewspapersandtvstations
ii ShortsmartphonevideosAgreatwaytoengageanaudienceisthroughvideoAskonequestionofsomeonewhoispartofthestoryMaybethereisasocialworkerwhocandiscusshungerinyourcityThesevideoscanbereusedonmultipleplatforms
iii Long-formwriting
iv TellingindividualstoriesthroughaseriesofphotosandshortnarrativesforpublicationonFacebookorInstagram
v Op-edsopinionpiecesplacedinthelocalpaper
vi Emailupdatestoyourlist
vii Connectingwithappropriateorganizationsthatareinvestedinthetopicandworkwiththecommunityandaskthemtodisseminateinformationtotheirmembers
viii EmployingldquoWordofMouthrdquo(WOM)tacticsbycreatingtownhallsorfreeeventsforresidentstoattendintheirownneighborhoods
5 DeterminethetypeofcollateralUsuallycampaignsthattargetresidentsrequireseveralavenuestoreachthemPeoplearenotalwaysinthesameplaceorspaceHavingakeenunderstandingofyouraudiencewillassistyouwithyourchoicesonwhatneedstobedevelopedSometimesthatmeansimagesforsocialmediaaonepageflierthatgivespeopleinformationonattendinganeventoranin-depthbrochure
Shareyourstoryandsuccess
53
6 ImplementyourstrategyKeepinmindstrategiescanbefluidSometimessomethingmaynotturnoutasexpectedevenifduringplanningsessionsitwasldquosupposedtordquoItisokYoucananalyzewhathappenedduringadebriefsessionandonceyoursurveysandothermeasurementshavebeenfinalized
Visualizations
Withtherightvisualizationyoucanreviewandshareinformationinsightsanalysisperformanceandtrendsallofwhichisvaluabletoacross-sectionofthepublicButwhichvisualizationisrightforyouTochoosetherighttypeofchartgraphdeterminethetypeofquestionyouwanttoanswerSomecommoncharttypesarebelowandmoreinformationcanbefoundhere
Exploreadditionalanalyticaltechniques
Duringthegoal-settingphaseyouestablishedoutputoutcomeandimplementationmeasuresBasedontheresultsofyourpilotyoumayneedtoadjustyourmeasuresWiththepilotdatainhandyoumayhavesomeideasofhowfutureanalysescouldbedifferentandpotentiallyimprovethequalityofyourprogramTakesometimetobrainstormotherrelevantdatasetsthatyoudidnothavethetimetopursueearlierincludingmessydepartmentaldatathatneedsmoreattentionorsourcesofpublicdatafromthestateandfederalgovernmentnonprofitsandevenprivatecompaniesthatcouldsupplementyourunderstandingofanissueUsingmachinelearningtechniquesyoumightbeabletobetter
Shareyourstoryandsuccess
54
predictdemandforhomelessservicesusingNOAAweatherforecastspreemptwatermainbreaksusingsoilstratigraphyorinvestineconomicdevelopmentbymonitoringrealestatelistingdata
Advancedanalyticaltechniquescanalsohelpyourimplementationrunmoreeffectivelybyintegratingdata-informedtargetinganddecisiontoolsintotheprocessByautomatingroutineworkflowsandsurfacingsubtletrendsthesetoolscanhelpyouiterateinamoreprecisemannerandprovidekeystrategicguidancetofront-lineemployeesthatwouldotherwisebeunavailableThesesortsofdata-science-enhancedprocessescanbeclassifiedasoneofsixprojecttypesasexploredfurtherbytheNewOrleansLANOLAlyticsteamFormoreguidanceonscopingasimilarprojecttheGovExadvancedanalyticsscopingguidegivesyouastep-by-stepoverviewofhowtobesuccessful
Usethespacebelowtolistyourideasforpossibleadvancedanalyticsprojectsbytype
Typeofproject Ideas
Findingtheneedleinthehaystack
Prioritizingforimpact
Earlywarningtools
Betterquickerdecisions
Optimizingresourceallocation
Experimentingforwhatworks
Keepthefeedbackloopgoing
GatherinformationonpublicsentimentaboutpolicychangethroughfocusgroupssurveysandpublicmeetingsledbyadvisorygroupmembersNotewhethernewinformationisavailablethatcanhelpanalyzeorcontextualizeyourprogress(egfederalorstatedatareports)Sharetheresultsofyourpilotandthepolicychangeprocesswithothercitiesandresearchers
Shareyourstoryandsuccess
55
WorkingwitharangeofevidencePresentlythereisalotofattentionandinvestmentinevidence-basedpolicymakingincludingthefederalbipartisanCommissiononEvidence-BasedPolicymakingthePewCharitableTrustsreportsandtoolkitsonHowStatesEngageinEvidence-BasedPolicymakingandtheLauraandJohnArnoldFoundationsupportedEvidence-BasedPolicymakingCollaborativeManyoftheseinitiativesfocusonrigorousevaluation--meaningrandomizedcontrolledtrials(RCT)orotherexperimentalmodels--asthebestkindofevidencetouseHoweverthisbodyofevidenceisofteninsufficientforthespecifickindsofquestionsthatgovernmentscanhavewhenmakingadecisionaboutpolicyInthisguideweproposeanapproachtosupportgovernmentsinincorporatingadditionaltypesofevidenceintotheirdecisionmaking
OnewaytounderstandarangeofevidenceistouseNestarsquosStandardsofEvidencetoframeanddefineevidenceTheBritishnonprofitNestaworkstoinjectpositivechangeintothewayUKgovernmentpoliciesandprogramsfunctionOneoftheirkeyareasofworkhasbeendevelopingWhatWorksCentresorganizationsthatcollectcommissionsynthesizeanddisseminateevidenceaboutinterventioneffectivenessacrossarangeofsocialimpactareasNestadevelopedthefollowingStandardsofEvidencetoprovideausefulwaytodocumenthowdifferentkindsofinformationdemonstratedifferentlevelsofevidence
Workingwitharangeofevidence
56
Theimportanceofpotentiallybiasedinformation
Onechallengepresentedbyanevidenceclassificationthatacknowledgesthevalueofnon-scientificformsofevidenceistofigureouthowtobalanceallofthepiecestogetherwhenyouusethemTounderstandhowtodothiswehavetothinkabouthowtopositionpotentiallybiasedinformationindecisionmaking
WhenwelookatinformationwhichhasnotbeenrigorouslyevaluatedweacceptthatthisinformationmightbebiasedThewholereasonweconductrigorousevaluationistoremovebiasfromourobservationsWeconductstructuredtrialstoproduceresultsthatareconsistentandgeneralizableIfafindingisunbiasedthisaccuracymdashthislackofbiasmdashcanbeprovedbecauseanyindependenttestercanreproduceitwhentheyfollowanidenticalprocess
YoucanlookatpotentiallybiasedinformationasjustinformationthatisearlieroninitspathtorefinementreplicationandverifiedaccuracyApieceofinformationbasedonanyoneindividualrsquosobservationsislikelytobebiasedbecauseitiscomingfromjustonesourceBiasisreducedbygatheringtogetherlotsofindependentsourcesofinformationandfiguringoutitsmostcommonattributes
MaybethebiasedinformationmakesaclaimthatistoolargemdashitsayssomethingistrueunderallconditionswhenitisreallyonlytrueundercertainconditionsorthataneffectofaninterventionisbiggerthanitstrueaverageorthataneffectlastsovertimewhenitdoesnotPerhapschangingtheclaimtobemorespecificmdashcriticallyreviewingtheinformationtoseeifspecificconditionsmustbemettoachievetheoutcomesmdashwouldmaketheclaimmoreaccurate
IfyoutestedyourclaimfounditdidnotholdupinsomecaseschangedittesteditagainuntilyourclaimrepeatedlyheldupthenyoucoulddemonstratethattheclaimwasnolongerbiasedbutaccurateThisprocessoftestingimprovingtestingandvalidatingiswhatwegetfromstructuredscientificstudy
Withindividualpiecesofinformation--crediblehypothesescasestudiesexpertopinion--youcannotassumethatkindofaccuracyAtthesametimetheseearlyobservationsarefoundationaltoanyreplicablescientificobservationsWewillnevergetanywhereifwedonotstartwiththesepiecesofinformationeveniftheymightbebiased
Combininglevelsofevidencetogetthemostdetailedinformationforyourcity
ItisespeciallyimportanttoacceptdifferentformsofevidencewhenyoufacequestionsthathavenotyetbeentestedbyindependentstudiesSomeofthemostimportantquestionsrelatetothingsthatareveryspecifictoyoursituationYoufaceaparticularcombinationof
Workingwitharangeofevidence
57
issuesactorstrendsandresourcesthatisultimatelydifferentfromeveryotherplaceintheworld
ThisisimportantbecauseaspolicypractitionersandnotscientistsyoucaremoreaboutyourspecificcasethanahypotheticalaveragegeneralcaseYoucarelessaboutthetotallikelyrangeofimpactsfromapolicythanhowthissingleimplementationwillworkYouwantinformationaboutpoliciesthatwillprovideyouwiththebestpossiblepredictionabouthowtheywillworkinyourcontext
ByputtingdifferentkindsofevidenceintothesameframeworkNestarsquosStandardsofEvidenceimplythatwhenyoucannotgetrigorouslytestedexperimentallyverifiedinformationitisappropriatetoworkwithwhatwedohaveThisisrelevantforthemunicipalimplementationcontextbecausenostudypulledfromaliteraturereviewwillaccountforallofthevariablesyoufaceinyourowncontext
Intheabsenceofhavingastudythatpredictshowaninterventionworksinyourspecificcontextyouwillneednon-scientificinformationtotrytofillinthosegapsYouwillneedtogatherstoriesfromresidentsstaffmembersandotherstakeholdersthatsitonLevels1or2ofthestandardsofevidence
Someofthekindsofinformationyouwillprobablygetmoreeasilyfrompeoplethanstudiesinclude
howpeopleincitieslikeyoursreactedtoaninterventioninpublicforumshowpeoplewhoimplementedaninterventioninacitylikeyoursfiguredouthowtogetitpassedhowthecitycommunicatedpubliclyabouttheapparentbenefitsanddrawbacksoftheinterventionthepoliticalacceptabilityoftheintervention
Youcanremainsimultaneouslyawarethattheinformationyouaregatheringthroughindividualstoriesisbiased--notgeneralizableinformedbyspecificperspectives--whilealsorecognizingthathavingsomeinformationaboutsomethingimportantisbetterthannoinformationEvidencefromthefirsttwoevidencelevelsareessentialbuildingblocksofknowledgeandperfectlyusefulwhenyoucannotgetdetailedunbiasedknowledge
Youhavetostartsomewhereandyoushouldalwaysstartwithevidenceyouhaveaboutyourveryspecificconditionsthatyouknowareimportantforsuccess
Workingwitharangeofevidence
58
Thankyou
Interviewswithcities
GovExheldsemi-structuredinterviewswithsixstaffmembersinfourcitiestoestablishtheinitialstepsoftheRoadmapWeaskedrespondentstotalkaboutaninitiativeorpolicythattheyhavebeenresponsibleforimplementingintheircityRespondentsfromthreecitiesspokeabouttheireffortstoaddresshomelessnessandrespondentsfromonecityspokeaboutblightAfterrespondentsidentifiedtheproblemtheyspokeabouthowthecityprioritizedthisissuethevariousstepsinvolvedandbarrierspresent
GovExconductedtextanalysisoftheseinterviewswhichrevealedthatpeoplewasthemostcommonlyusedtermduringtheseconversationswhendiscussingthestepstochangeaprogramorpolicyThisisathemethroughoutthisRoadmapThefirstsectionbeginswithidentifyingtherelevantpeopleTheconversationsaroundevidencearefocusedonhumanbehaviorratherthanmethodologyorstrengthofevidence
Thankyou
59
OncewehadadraftoftheRoadmapwefollowedupwithcitiesonanindividualbasistolearnmoreaboutwhatwouldbehelpfulWearegratefultothefollowingcitystafffortheinterviewsandindividualconversations
MichaelBaskinChiefPolicyOfficerChattanoogaTN
MelissaBridgesPerformanceandInnovationCoordinatorLittleRockAR
VictoriaCarreoacutenAdministrativeOfficerLasVegasNV
TyroneGrandstrandCommunityDevelopmentPlannerFargoND
TibLaughlinDirectorofGeneralServicesKansasCityKS
TimMorelandDirectorofPerformanceManagementandOpenDataChattanoogaTN
CharlesRosendahlManagementAnalystDentonTX
Thankyou
60
MelissaSiebenAssistantCountyAdministratorKansasCityKS
GinaThompsonDirectorofStrategicServicesLewisvilleTX
JHUFacultycollaboration
GovExalsoconvenedthecitycontactsandfacultypartnersinaseriesofAdvisoryGroupmeetingsThesemeetingsservedtoconnectcitiesandfacultytounderstandoneanothersperspective
StefanBaralBloombergSchoolofPublicHealth
AshleyBernerJHUSchoolofEducation
MelissaDavey-RothwellBloombergSchoolofPublicHealth
JonLinksBloombergSchoolofPublicHealth
TamarMendelsonBloombergSchoolofPublicHealth
KristinMmariBloombergSchoolofPublicHealth
KeshiaPollackPorterBloombergSchoolofPublicHealth
JoshSharfsteinBloombergSchoolofPublicHealth
LizStuartBloombergSchoolofPublicHealth
LindsayThompsonCareyBusinessSchool
DanielWebsterBloombergSchoolofPublicHealth
Partnersindataandevidence-basedpolicy
Wealsospokewithpartnerswhoareexpertsinthefieldofevidence-basedpolicywhosharedtheirknowledgeandresourcesofcurrentandfuturethinkinginevidence-informedthinking
WendyBarrenoAssociateDirectorWilsonSheehanLabforEconomicOpportunitiesUniversityofNotreDame
TiffanyDavenportDirectorofDataPartnershipsBaltimoresPromise
TomiHiersDirectorBaltimoreCivicSiteAnnieECaseyFoundation
ChrisKingsleySeniorAssociateAnnieECaseyFoundation
KathrynOliverAssociateProfessorLondonSchoolofHygieneandTropicalMedicine
Thankyou
61
KristenWatkinsSeniorPolicyAssociateJ-PAL
GovExParticipants
GovExAnalystMiriamMcKinneyanalyzedtheinterviewtranscriptsandconductedtextanalysisSeniorImplementationAdvisorEmilyShawwillintegrateRoadmapactivitieswithadditionalservicedeliveryactivitiesatGovEx
Thankyou
62
ReferencesAbrahamKGampHaskinsR(2017)Thepromiseofevidence-basedpolicymakingReportofthecommissiononevidence-basedpolicymakingWashingtonDCCommissiononEvidence-basedPolicymakingBigmove2Extendingthegreen(nd)Retrievedfromhttpswwwcityoflewisvillecomdoing-businesslewisville-2025big-move-2-extending-the-greenBatesMAampGlennersterR(2017)ThegeneralizabilitypuzzleRetrievedfromhttpsssirorgarticlesentrythe_generalizability_puzzleBuildwithnotfor(2018January)RetrievedfromhttpsbuildwithnotforwordpresscomCairneyPampOliverK(2017)Evidence-basedpolicymakingisnotlikeevidence-basedmedicinesohowfarshouldyougotobridgethedividebetweenevidenceandpolicyHealthResearchPolicyandSystemsdoi101186s12961-017-0192-xCairneyPOliverKampWellsteadA(2016)TobridgethedividebetweenevidenceandpolicyReduceambiguityasmuchasuncertaintyPublicAdministrationReview76(3)399-402Retrievedfromhttpdxdoiorg101111puar12555Centreforpublicimpact(nd)RetrievedfromhttpswwwcentreforpublicimpactorgCernaL(2013)ThenatureofpolicychangeandimplementationAreviewofdifferenttheoreticalapproachesOrganisationforEconomicCo-OperationandDevelopmentRetrievedfromhttpswwwoecdorgeduceriTheNatureofPolicyChangeandImplementationpdfColoradoSprings(nd)Retrievedfromhttpscoloradospringsgovcommunicationspagewhat-expect-when-you-speakupContinuumofcare(coc)program(nd)RetrievedfromhttpswwwhudexchangeinfoprogramscocCouncilpresidentkicksoff$12millionyouthfund(nd)Retrievedfromhttpwwwbaltimorecitycouncilcomcontentcouncil-president-kicks-12-million-youth-fundDurlakJAampDuPreEP(2008)ImplementationmattersAreviewofresearchontheinfluenceofimplementationonprogramoutcomesandthefactorsaffectingimplementationAmericanJournalofCommunityPsychology41327-350Retrievedfromhttpswwwncbinlmnihgovpubmed18322790Fishbonediagram(ishikawa)(nd)Retrievedfromhttpasqorglearn-about-qualitycause-analysis-toolsoverviewfishbonehtmlFivewhysGettingtotherootofaproblemquickly(nd)RetrievedfromhttpswwwmindtoolscompagesarticlenewTMC_5Whtm
References
63
FligsteinNampMcAdamD(2011)TowardageneraltheoryofstrategicactionfieldsSociologicalTheory29(1)1-26RetrievedfromhttpsociologyberkeleyedusitesdefaultfilesfacultyfligsteinFligstein20McAdam20Soc20Theory20PaperpdHeinkelP(2018January12)CitymullspermitsforbikesharingcompaniesRetrievedfromhttpwwwdentonrccomnewsdenton20180112city-mulls-permits-bike-sharing-companiesHowisareamedianincomecalculated(2012October)Retrievedfromhttpswwwhudexchangeinfofaqs843how-is-area-median-income-calculatedLampkinLMampHatryHP(2003)KeystepsinoutcomemanagementUrbanInstituteRetrievedfromhttpswwwurbanorgresearchpublicationkey-steps-outcome-managementMarshDampSmithM(2000)UnderstandingpolicynetworksTowardsadialecticalapproachPoliticalStudies484-21Retrievedfromhttpjournalssagepubcomdoiabs1011111467-924800247McCannL(nd)RetrievedfromhttplaurenellencomMoultonSampSandfortJR(2016)ThestrategicactionfieldframeworkforpolicyimplementationresearchPolicyStudiesJournal45(1)Retrievedfromhttponlinelibrarywileycomstore101111psj12147assetpsj12147pdfv=1ampt=jdkhhcp4amps=e184c625e1855a6b1615ccb99ee27b1c0b3212d8PaynterB(2018January1)What100citiesarelearningfromeachotherbysharingtheirdataRetrievedfromhttpswwwfastcompanycom40518004what-100-cities-are-learning-from-each-other-by-sharing-their-dataPeakacademy(nd)Retrievedfromhttpswwwdenvergovorgcontentdenvergovenmayors-officeprograms-initiativespeak-performancepeak-academyhtmlPinnockHBarwickMCarpenterCREldridgeSGrandesGGriffithsCJRycroft-MaloneJMeissnerPMurrayEPatelASheikhAampTaylorSJC(2017)Standardsforreportingimplementationstudies(stari)ExplanationandelaborationdocumentBMJOpenRetrievedfromhttpbmjopenbmjcomcontentbmjopen74e013318fullpdfProctorESilmereHRaghavanRHovmandPAaronsGBungerAGriffeyRampHensleyM(2011)OutcomesforimplementationresearchConceptualdistinctionsmeasurementchallengesandresearchagendaAdministrationandPolicyinMentalHealth38(2)65-76doi101007s10488-010-0319-7TandonSD(2018)ProgressincommunityhealthpartnershipsResearcheducationandactionRetrievedfromhttpswwwpressjhuedujournalsprogress-community-health-partnerships-research-education-and-actionProvanKGampKenisP(2007)ModesofnetworkgovernanceStructuremanagementandeffectivenessJournalofPublicAdministrationResearchandTheory
References
64
18(2)229-252Retrievedfromhttpsacademicoupcomjpartarticle182229935895SabatierPA(1988)Anadvocacycoalitionframeworkofpolicychangeandtheroleofpolicy-orientedlearningthereinPolicySciences21(3)129-168Retrievedfromhttpwwwjstororgstable4532139SandstromAampCarlssonL(2008)TheperformanceofpolicynetworksTherelationbetweennetworkstructureandnetworkperformancePolicyStudiesJournal36497-524Retrievedfromhttponlinelibrarywileycomdoi101111j1541-0072200800281xabstractSchwartzMampWilsonE(nd)WhocanaffordtoliveinahomeAlookatdatafromthe2006americancommunitysurveyRetrievedfromhttpswwwcensusgovhousingcensuspublicationswho-can-affordpdfShearerJCAbelsonAKouyateBLavisJNampWaltG(2016)WhydopolicieschangeInstitutionsinterestsideasandnetworksinthreecasesofpolicyreformHealthPolicyandPlanning31(9)1200-1211Retrievedfromhttpsdoiorg101093heapolczw052ThelessonsofadministrativedataHigh-profilepolicy-relevantresearchpoweredbyadministrativedata(nd)Retrievedfromhttpswwwpovertyactionlaborgsitesdefaultfilesresources20161114-The-Lessons-of-Administrative-DatapdfToolsandotherneedsforqualitativeandquantitativemethods(2015February)AgencyforHealthcareResearchandQualityRetrievedfromhttpwwwahrqgovprofessionalsquality-patient-safetytalkingqualityassesstoolshtml
WeaverRCampBagchi-SenS(2013)SpatialanalysisofurbandeclineThegeographyofblightAppliedGeography4061-70
References
65
UniversityPartners
Name Title Affiliation Area(s)ofexpertise
Contactinformation
Examplesofworkingwithnon-academicdecisionmakers
WendyPBarreno
AssociateDirector
WilsonSheehanLabforEconomicOpportunitiesUniversityofNotreDame
LEOsfocusareasincludehealtheducationhousingchildwelfareeconomicopportunitycriminaljustice
wbarrenonduduorleondedu
MelissaDavey-RothwellPhD
AssociateScientist
JohnsHopkinsBloombergSchoolofPublicHealth
Health mdavey1jhuedu410-614-5854
BaltimoreCityHealthDepartmentLeadersMarylandDepartmentofHealthAddictionRecoveryServicesinDCExecutiveDirectorsofcommunityorganizationsacrosstheUSthatfocusonHIVSTIsubstanceabuseandimplementationscience
ElizabethStuart
AssociateDeanforEducationandProfessorDepartmentsofMentalHealthBiostatisticsandHealthPolicyand
JohnsHopkinsBloombergSchoolofPublicHealth
Educationsocialpolicystatisticalmethods
estuartjhuedu
Resources
66
Management
Design-ThinkingResources
EngagementGuideforCivicTech
Journeymap
PerformanceManagementResources
DenverPeakAcademy
Proxymeasuresguide
Talkingaboutperformancewithoutcomplicatingit(video)
SettingPerformanceTargetsguide
Datavisualizationguide
Performancemanagementgettingstartedguide
Exercises
Fishboneexercise
5Whys
AdministrativeData
TheLessonsofAdministrativeData(JPAL)
EvaluationResources
Quantitativeandqualitativemethods(USDepartmentofHealthandHumanServicesAgencyforHealthcareResearchandQuality)
KeyStepsinOutcomeManagement(UrbanInstitute)
RCT-YES
Sageresearchmethods
BestPracticesforMixedMethodsResearchintheHealthSciences
ACCORDChecklist
Pilotandimplementationplanning
Benchmarkingexamplesandproxymeasures
Seattleopendataplaybook
Resources
67
Datasciencescopingguide
Resources
68
AboutGovExGovExhelpsgovernmentsusedataandevidencetoimproveresidentsrsquolivesWesupportgovernmentsinbuildingtheirinternalcapacitytooperatesustainableanalyticspracticeswhichimprovecommunityoutcomesinfieldsfrompublicsafetytoeconomicdevelopmentGovExhasaproventrackrecordofworkinginconcertwithpartnerstoprovidetechnicalassistanceandtrainingatscale
SinceourlaunchinApril2015GovExhas
Trainedover2000governmentemployeesinpersonandonlineintopicsincludingdatamanagementperformancemanagementopendatacommunityengagementandanalytics
Providedtechnicalassistancetomorethan100localgovernmentsacrosstheglobeimprovingtheiruseofdatatoachieveefficienciesinservicedeliveryandbetteroutcomesforresidents
Assessedthecapacityofmorethan130governmentalorganizationstodeterminehowleadersstaffandexternalstakeholdersmanageandusedata
Localgovernmentsareincreasinglyonthefrontlinesofaddressinglargescaleproblemswithglobalimplicationssuchasclimatechangepublicsafetyandeconomicinequality
Whileevidence-basedinterventionstohelpsolvethoseproblemsareavailablelocalgovernmentstaffhavelimitedtimeandcapacitytofindandimplementresearch-basedsolutionsAsaresultcitiescontinuetoworkthewaytheyalwayshave--makingincrementalimprovementstoprogramsbutnotalwayscreatingchangeatasystem-level
ThroughresourcedevelopmentandcollaborationwithJHUfacultyGovExissupportingcityleadersinusingdataandevidencetomakedecisionsandcreatinglastingrelationshipswithresearchers
TheRoadmapwasdevelopedbyGovExaspartofitspartnershipwithBloombergPhilanthropiesWhatWorksCitiesinitiative
AboutGovEx
69
- Title page
- Introduction and purpose of the Roadmap
- Using the Roadmap
-
- Gather information about the problem
- Strategically engage key partners
- Review the evidence and collect details
- Decide on a policy to implement based on local factors
- Establish a framework to monitor progress
- Assess data accessibility quality and security
- Design implementation and evaluation
- Share your story and success
-
- Working with a range of evidence
- Thank you
- References
- Resources
- About GovEx
-
Developrelationshipswithsubjectmatterexperts
OnewaytokeepsubjectmatterexpertsengagedistoinvolvetheminthecommunityparticipationprocessasdescribedaboveIntheWestVirginiaexamplesubjectmatterexpertsrespondedtoandsynthesizedthepubliccommentsOtheropportunitiesaredescribedbelow
AdvisoryAcitydepartmentheadMayororstaffmembermaydevelopaone-on-oneadvisoryrelationshipwithuniversityfacultymemberoracollaborativebetweentheCityandUniversitytoaddressaparticularproblemAgreatplacetostartistolookforrelevantpracticecenterswithinyourlocaluniversityandreachingouttosubjectmatterexpertsInadditiontotheirexpertiseontherelevantpolicytopicheorshemayalsobeabletoidentifyotherexpertsandevenprovidesupportonhiringtherighttalenttoleadcertainprogramsAnadvisoryrelationshipwithauniversitysubjectmatterexpertcanalsohelpwhencitiesareconsideringbringinginnewprogramsTheseexpertsareoftenfamiliarwithprogramsthatarebasedonnationalmodelsandcanprovideinsightintoaprogramsoriginandlineageeffectivenessinotherplacesandthepotentialforimplementationinyourcityAcademicscanalsohelpidentifyindicatorsbasedongovernmentindicatorsorliteratureFinallyrelationshipsbetweenpolicymakersandresearchersprovidethemostbroadlyeffectiveroutetopullingresearchintopolicy
ResearchandevaluationAcademicexpertsmaysupportcitiesinevaluation-indeedevaluatingcityinitiativesisatraditionalmodelofcityuniversityrelationshipsSharingtheinformationyoucollectedintheGatherInformationphasecanhelpresearchersunderstandyourworkandgoalswhichinturnwillstrengthenthesupportthattheycanprovideDiscussingdetailsaboutyourinterventionandevaluationplanatthebeginningoftheprocessmeansthataresearchpartnercanhelpwiththeevaluationdesignanddatacollectionForinstancetheWilsonSheehanLabforEconomicOpportunitieswalkscitiesthroughconsideringthefollowingquestionsbeforepartneringonanevaluation
1 Isthereaclearpolicyprogrammodelwithobjectivescreeningcriteria
2 Isitpossibletoevaluatewhathappensbothwiththepolicyprogramandintheabsenceofthepolicyprogram
3 Arethereoneormoremeasurableoutcomesofinterest
4 Arethereenoughpeopleforrigorousquantitativeanalysis
5 Whatdatawillbenecessaryforthestudyandwhataretheprocessesforaccessingthesedatasource(s)
6 Isthereacompellingreasontoevaluate
Strategicallyengagekeypartners
19
InadditiontoassessingreadinesstoconductanevaluationresearcherscanhelpyouhoneinonthepurposeoftheevaluationTheWorldBankdescribesseveralpotentialusesofevaluationwhicharedefinedhereInbriefevaluationcanhelpdecisionmakers
makeresourceallocationdecisionsrethinkrootcausesofproblemsidentifynewproblemsdecideonabestalternativedeterminewhetherprogramsorpoliciesarebeingimplementedwithfidelitydeterminewhetherprogramsorpoliciesarehavingthedesiredimpact
Researcherscanalsoprovidesupportinsharingtheresultswiththescientificcommunitythroughpapersandconferenceswhichisvaluableincreatingapeernetworkofevidence-basedcitypractitioners
TrainingandtechnicalassistanceUniversitypartnersespeciallythoseatpracticecenterscanbehelpfulindevelopingorupdatingtrainingcurriculumtrainingthetraineranddeliveringtrainingtocitystaff
Strategicallyengagekeypartners
20
ReviewtheevidenceandcollectdetailsOnceyouhaveathoroughunderstandingoftheproblemdataandactorsyouwillneedtoconductyourownresearchonavailableevidenceandlearnaboutthelandscapeofpolicyoptionsYourgoalwillbetodevelopanaccessiblelistofinformationaboutpoliciesthathavehelpedsolvesimilarproblemsinotherplacesThischapterwillguideyouthroughthestepsofpreparingyourreviewersunderstandinglevelsofevidenceandcollectingthedetailsonevidencethatyouwillneedfordiscussionwithdecisionmakersinyourcity
Selectandprepareyourreviewers
AssignapersonorteamtoreviewrelevantinterventionsthathavedemonstratedsuccessinotherjurisdictionsTheirgoalwillbetoproduceareviewofalloftherelevantprogramspoliciesorinitiativesthatyourteamcouldconsiderimplementingTheywillbeconductingaliteraturereviewsupplementedbyinterviewswithexpertsandpeerswhohaveinformationaboutpotentialprograms
SincethepeopleyouassigntothistaskareprovidingthebaselineofpotentialoptionstoimplementitisimportantthatyouchoosepeopleyouareconfidentwillbethoughtfulandthoroughintheirworkItisalsocriticalthattheyareawareoftheimportanceofsourcecredibilityEnsureyourreviewershavefamiliarizedthemselveswithrulesforcriticallyanalyzinginformationsourcesbeforetheygetstartedIftheyareincludingnewsarticlesaspartoftheirsearchtheyshouldalsobeawareofmethodsforevaluatingnewscredibility
AskyourreviewerstodevelopaplanandtimelinetocompletethereviewThisstartswithsolidifyingagoverningresearchquestionOnewaytoknowyouarecreatingadetailedresearchquestionistousetheacronymPICOremindingyoutoincludereferencetothepopulationintervention(ifyouhaveoneinmind)comparisonandoutcomeelementsinyourresearchquestionForexampleifyouknowthatyouhaveapopulationofteenagerswhoarehomelessyouhavelearnedfromyourstakeholdersthatmanyoftheseteenagersidentifyasLGBTandyouhaveheardsomethingabouttheapproachofcreatingdrop-incentersfordeliveringsupportyoucouldask
ForLGBTyouthfacinghomelessness(population)howeffectivearedrop-incenters(intervention)relativetostandardsocialservices(comparison)atsettlingtheminpermanenthousing(outcome)
Ifyoudonothaveaninterventioninmindyoucanleaveouttheinterventionandcomparisonandsimplyask
Reviewtheevidenceandcollectdetails
21
HowcanwehelpLGBTyouthfacinghomelessness(population)obtainpermanenthousing(outcome)
OnceyourreviewershavetheirresearchquestionsusethequestionstocreatealistofkeywordstheycanuseintheirliteraturesearchThiswillhelpkeeptheircollectionofarticlesreasonablyfocusedHavethemdescribewhattheirsearchprocesswillbeincludinglistingthedatabasestheywillreviewandtheirplanforstoringtheinformationtheywillcollect(Herearesomeadditionalsuggestionsforstructuringareviewplan)
WhatyourreviewerswillbedoingisalightweightversionofarapidreviewTotheextentthattheywouldliketodevelopamethodologyforspecificsearchprotocolsincludingcriticalappraisalprocessestheycancertainlygoaheadanddoso
Usedifferentlevelsofevidenceforthemostcompletepicture
AsdiscussedintheWorkingwitharangeofevidencechaptertheNestaStandardsofEvidenceframework--reproducedbelow--ishelpfultounderstandthelevelofconfidencethataprogramorinterventionishavingapositiveimpactThisapproachletsyouputinterventionsthatyouhaveheardaboutlocallybutwhichhavenotbeenstudiedtogetherwiththosewithademonstratedcausaleffectThisishelpfulsinceanumberofpoliciesprogramsandpracticesthatyouwillwanttoconsidermaynotyethavemuchofaresearchrecord
Reviewtheevidenceandcollectdetails
22
NearlyanypieceofinformationaboutaninterventioncanbeclassifiedwithinthestandardsofevidenceForexampleifsomeonehasacrediblehypothesisabouthowaninterventioncouldproducebenefitthistypeofevidencewouldbeclassifiedasLevel1ThistypeofevidencecouldincludeinterventionsinothercitiesthatyoureadaboutinthenewsorhearaboutataconferenceForinstancewhenUSDAallowedschoolsandschooldistrictstoprovidefreebreakfastandlunchtoallstudentsthroughtheCommunityEligibilityProvision(CEP)manydistrictsimplementedthisoptionevenbeforetherewasevidenceofeffectivenessThiswouldfitintoLevel1sinceitbuildsonthelogicthatprovidingaccesstomealswilldecreasefoodinsecurity
YoumayalsofindpiecesofinformationthatqualifyasLevel2evidenceForinstanceasmoreschooldistrictsimplementedCEPprincipalstestifiedaboutpositivechangestheyobservedinstudentssuchasamorepositivedemeanorthatcanleadtobetteracademicperformanceDataalsoshowedthatthestudentsdescribedwerereceivingmoremealsthanbeforetheprogramsimplementationHoweveritwastoosoontobuildevidencetoshowacausalrelationshipbetweentheincreaseinfreemealsandincreasedacademicperformance
ldquoLower-levelrdquoevidencecanconsistofnuancedbutvaluableelementsthatrelatetoaninterventionrsquospotentialfitwithacommunityQualitativeinformationsuchastestimonyfrompeoplewhohaveexperiencedaninterventionandthosewhohavefacilitatedorimplementeditwillalwaysbeusefultoconsiderandprovidesdetailedlocalcontext
Experimentalstudies--describedstartingatLevel3withcontrolledexperimentsandgoingthroughLevel5withfullyfleshedoutprotocolsandprocessesforreplication--stilloccupythehigherlevelsofthisdescriptionofevidenceThisprioritizationshowsthevalueofrigorousstudyforevaluatingtheimpactofaparticularpolicyorprogramratherthanotherfactorsthatcouldbeinfluencingoutcomesofinterestRememberthatmultipletypesofevidencecancomplementrigorousimpactevaluationsandgivecityleadersadditionalinsightsandbroaderperspective
Accessinformation
Encourageyourreviewerstobegintheirsearchforpromisingpracticesingovernmentandnon-profitclearinghousesandinacademicpolicyjournalsWorkpublishedinthesesourceshasalreadybeensubjectedtoahighlevelofprofessionalscrutinyandgenerallycontainsagreatdealofdetailwhichmeansthatyourreviewersaremorelikelytobeabletopullouttheinformationyouneedtomakeagooddecision
Clearinghouses
Reviewtheevidenceandcollectdetails
23
Herearesomeoftheclearinghouses(collectionsofprogramevaluations)forevidence-basedpoliciesandprograms
ResultsFirstClearinghouseDatabaseThePewCharitableTrustsCollectstogethereightotherclearinghousestoprovideinformationoninterventionsinthefollowingcategoriesadultcriminaljusticeeducationchildwelfarehealthjuvenilejusticementalhealthsubstanceabuseandyouthdevelopmentIndividualclearinghousesincludedinResultsFirstare
BlueprintsforHealthyYouthDevelopment-foryouthdevelopmentandviolenceprevention
CaliforniaEvidence-BasedClearinghouse-forchildwelfareinterventions
TheLauraandJohnArnoldFoundationrsquosSocialProgramsThatWork-forsocialservicesprograms
TheUSDepartmentofEducationrsquosWhatWorksClearinghouse-foreducationinterventions
TheUSDepartmentofHealthandHumanServicesrsquoResearch-TestedInterventionPrograms(RTIPs)-forcancercontrolinterventions
TheUSDepartmentofHealthandHumanServicesrsquoTeenPregnancyPreventionEvidenceReview(TPPEvidenceReview)-forsexualriskbehaviorandsexualhealthoutcomes
TheUSDepartmentofJusticersquosCrimeSolutionsgov-forcriminaljusticejuvenilejusticeandcrimevictimservices
TheUSSubstanceAbuseandMentalHealthServicesAdministrationrsquosNationalRegistryofEvidence-basedProgramsandPractices(NREPP)-substanceuseandmentalhealthinterventions
TheUniversityofWisconsinPopulationHealthInstituteandRobertWoodJohnsonFoundationrsquosCountyHealthRankingsandRoadmapsWhatWorksforHealth(WhatWorksforHealth)-healthbehaviorsclinicalcaresocialandeconomicfactorsandthephysicalenvironment
UKWhatWorksCentresTheBritishgovernmenthascharteredanetworkofsubject-specificorganizationswhichcollectsynthesizeandreportoutevidenceontheeffectivenessofpolicyprogramsandpractices
NationalInstituteforHealthandCareExcellence(NICE)-healthandsocialcare
Reviewtheevidenceandcollectdetails
24
SuttonTrustEducationalEndowmentFoundation-educationalachievement
CollegeofPolicingWhatWorksCentreforCrimeReduction-crimereduction
EarlyInterventionFoundation-earlyintervention
WhatWorksCentreforLocalEconomicGrowth-localeconomicgrowth
CentreforAgeingBetter-qualityoflifeforolderpeople
WhatWorksCentreforWellbeing-wellbeing
WilsonSheehanLabforEconomicOpportunities(LEO)UniversityofNotreDameProvideslinkstotheirevaluationsofinterventionsincriminaljusticeself-sufficiencyeducationhealthandhousingandhomelessness
ClearinghouseforLaborEvaluationandResearchUSDepartmentofLaborProvidesresearchonworkforceandlabor-relatedpoliciesandprograms
J-PALEvaluationsAbdulLatifJameelPovertyActionLabStudiesonpoverty-reducinginterventionsfromaroundtheworld
CampbellSystematicReviewsCampbellCollaborationSystematicreviewofpolicyinterventionsbyscholarlycoordinatinggroupsinsevenissueareascrimeandjusticeeducationinternationaldevelopmentmethodssocialwelfareknowledgetranslationandimplementationanddisability
Benefit-CostResultsWashingtonStateInstituteforPublicPolicyAWashingtonState-charteredprogramtoestablishthelikelihoodofrelativecostsandbenefitsforprogramsacrossthefollowingareasjuvenilejusticeadultcriminaljusticechildwelfarepre-Kto12educationchildrensmentalhealthhealthcaresubstanceusedisordersadultmentalhealthpublichealthandPreventionworkforcedevelopmentandhighereducation
EffectiveInterventionsCentersforDiseaseControlandPreventionRepositoryforevidence-informedHIVpreventioncareinterventions(withstepsonhowtoimplement)
Journalsandindexes
OnceyouhavereviewedrelevantinformationaccessiblethroughclearinghousesitisworthtakingalookatindividualstudiespublishedasscholarlyarticlesYourreviewerswilldothisbysearchingarticleindexesandacademicjournalsUnfortunatelyanumberofusefulacademicjournalsandindexingplatformsmaynotbegenerallyaccessibletoyourreviewersifyoudonothaveasubscriptiontothemToaddressthisconsiderpartneringwithalocal
Reviewtheevidenceandcollectdetails
25
collegeoruniversitytoprovideyouwithalistofarticlesfromthefollowingsources(Thisisalsoanexcellentopportunitytoseekresearchsupportfromanyacademicstakeholdersyouidentifiedpreviously)
Indexes
GoogleScholarincludesmostpeer-reviewedonlineacademicjournalsandbooksconferencepapersthesesanddissertationspreprintsabstractstechnicalreportsandotherscholarlyliteratureincludingcourtopinionsandpatentsItallowsyoutospecifyapublicationrangeinyoursearchandprovideslinkstolikelysourcesforalllistedarticles
PolicyFileIndexisauniqueresourceforUSpublicpolicyresearchinthatitgrantsusersaccesstotimelyupdatedinformationfromover350publicpolicythinktanksnongovernmentalorganizationsresearchinstitutesuniversitycentersadvocacygroupsandotherentities
PolicyArchiveworkstocollectanddisseminatesummariesandfulltextsvideosreportsbriefsandmultimediamaterialofthinktankuniversitygovernmentandfoundation-fundedpolicyresearch
Anothersourceforhigh-qualityresearchisthereportsoftheCongressionalResearchServiceYoucanfindtheirreportsaggregatedhere
Specificjournals
Oftenthemostusefulacademicjournalsforunearthingpolicyorprogramreviewswillbeineachspecificsubjectarea--thatisineducationjournalsforeducationpoliciesYoucanuseageneralindexlikeGoogleScholartobegintolearnwhichjournalsaremostoftenusefulforyoursearchHoweveryoucanalsosearchthefollowingtwogeneralistpolicypublicationsforissuesandarticlesrelatedtoyourtopic
PublicAdministrationReviewJournalofPolicyAnalysisandManagement
Newssourcessubjectmatterexpertsandinterviews
AlthoughitisvaluabletoknowwhathasbeenexploredthroughascholarlyresearchframeworkitisalsousefultolearnwhatkindofinterventionsarebeingtriedinthefieldrightnowandhowpeoplefeelaboutthesetrialsSearchsubscriptionnewsaggregatorslikeLexisNexisorfreenewsaggregatorslikeGoogleNewsorYahooNewsforyourissueareaplusthewordpolicyorprogramtoseewhatkindsofinterventionsarebeingcoveredinthenews
Reviewtheevidenceandcollectdetails
26
DonotforgettocontacttheadvisorygroupmemberswhowerechosenfortheirsubjectmatterexpertiseInadditiontoprovidinggoodmaterialthroughinterviewsthemselvestheymayhaveideasaboutothersubjectmatterexpertsyoushouldcontacttoaskaboutpotentialinterventions
Peercitiesandnetworks
InadditiontoreadingabouttheworkofothercitiesitisidealifyouareabletospeakwithpeoplewhoareinvolvedwithcurrentimplementationsofinterestinginterventionsInyourcityyoulikelyhaverelationshipswithyourneighborsbutitcanbehelpfultolookoutsideofyourregionforsolutionsaswellTofindcounterpartsinothercitieswhoareworkingonaprogramyouwouldliketolearnmoreaboutyoucancontactstaffinintermediaryorganizations(likeGovExoryourprofessionalassociations)whomightbeinapositiontoputyouintouchimmediatelyorwhomightseeanothergoodrouteforsettingupaconversationHereisalistofnetworksthatmaybehelpful
InternationalCityCountyManagementAssociation(ICMA)NationalAssociationofCounties(NACO)NationalLeagueofCitiesUSConferenceofMayorsWhatWorksCities
Collectthedetails
Onceyourreviewershavefoundevidenceonpossiblepoliciesorprogramstoimplementtheyshouldcapturekeydetailsthatwillhelpdecisionmakersunderstandthepotentialeffectivenessandapplicabilityofeachpotentialintervention
Foreachsourcethathasprovidedinformationyouwanttoconveyusethetemplatebelowtocaptureasmanyofthefollowingdetailsaspossible(Youdonotneedtocompletethisforsourcesthatprovideduplicativeinformationorareotherwisenotthatuseful)
Programorpolicyname (egldquoHousingFirstrdquoorldquoblightedpropertiesregistryrdquo)
Sourcetype (egjournalarticleinterviewsubjectmatterexpertstudyassociationcommunityparticipationprocess)
Sourcedetails (egarticletitleandpublicationnameandtitleofinterviewee)
Evaluationmethodology
(egrandomizedevaluationcomparativeevaluationoftwodifferentgroupsbeforeafteranalysis)
Evidencelevel (egLevel12345)
Programorpolicy
Reviewtheevidenceandcollectdetails
27
description
Programorpolicygoal (1sentencecapturingtheprincipalaimoftheintervention)
Existingimplementations
(WherehastheinterventionbeentriedListasmanysitesasyouareawareof)
Characteristicsofimplementationsites
(ForeachimplementationsiteofferbasicgeographicIDmdashcityandstateThenyouwillwanttocollectthedetailswhicharerelevanttoyourdecisionmakersrsquoconsiderationsaboutwhatmakesalocationldquosimilartordquoorldquodifferentfromrdquoyoursituationYouwillatleastwanttoincludeforeachimplementationsitePopulationnumberPopulationdensity(ruralurbansuburban)Socio-economiccharacteristicsBeforestartingaskyourdecisionmakersaboutanyparticularcharacteristicsofprogramimplementationsitestheywouldwanttoknowbeforeseekingtoreplicate)
Characteristicsofimplementationpartners
(Anyavailabledetailsaboutorganizationsinstitutionsorpeoplewhowereinvolvedwithimplementingtheintervention-egindependentchildcarecenterpublicelementaryschoolcitypublichealthnurses)
Characteristicsoftargetpopulationeligibleprogramparticipants
(Anyavailabledetailsaboutpeoplewhoarethebeneficiariesoftheprogramorotherwiseincludedoraffectedbytheintervention-egnumberofprogramparticipantscategoriesofpropertyownersaffected)
Policyorprogramduration
(HowlonghastheinterventionbeeninplaceineachimplementationsiteHasitendedandifsowhy)
Whatbenefitscamefromeachimplementation
(Anyobservedpositiveeffectsoftheinterventionincludingeffectsize)
Statisticalestimatesofinterventioneffectiveness
(Anyavailablestatisticalestimatesoftheinterventionrsquoseffects)
Whatdrawbackswereobserved
(Anyavailabledetailsaboutunintendedorunavoidablenegativeeffectsoftheintervention)
Programorpolicycosts
(Anyavailabledetailsaboutprogramcostsperimplementation)
Reviewtheevidenceandcollectdetails
28
Reviewtheevidenceandcollectdetails
29
DecideonapolicytoimplementbasedonlocalfactorsTogetthecriticalbuy-infromleadershiptoimplementanewpolicyorprogramtheprojectleadshouldscheduleameetingwithdecisionmakerstopresenttheevidencesurroundingthepolicyoptionscapturedinthetemplateDependingonyourinsightintoyourcityleadershipswillingnesstomakechangeyoumaywishtosharetheresearchyouhaveconductedontherootcausesoftheproblemandsomeofyouranalysisontheimpactofcurrenteffortstodate
Considertheapplicabilityofanapproach
AsdiscussedinthepreviouschapterpolicymakersoftenlookforevidenceofeffectiveprogramsinothercitiesbutoftenthequestionarisesofhowgeneralizableaprogramisfromonecitytoanotherTheAbdulLatifJameelPovertyActionLab(J-PAL)suggeststhatinreviewingevidencefromanotherlocationitishelpfultoidentifythespecificmechanismthatwasused(deliveryofprogramorserviceinformationsharingetc)identifythelocalconditionsthatwereinplaceobservethebehaviorchangethatresultedfromthemechanismandfinallytounderstandlocalcontextandconditions(legalorcostconstraintsculturalnormsetc)Thissetofpresentationelementscanbedrawnfromthelistofdetailscollectedintheresearchtemplate
EducationpolicymakersinChicagoILusedfindingsfromprogramsinKenyaandIndiainwhichchildrenwhohadfallenbehindtheirpeersweregiveneducationalsupporttargetedtotheirleveloflearningThemechanismusedwastargetedteachingandthereweresimilarconditionspresentinallthreecountriesAlthoughsomestudentshadfallenbehindinthecurriculumteacherswereincentivizedtoteachgrade-levelmaterialsConsiderationsofhowthelocalcontextinChicagomightaffectimplementationofatargetedteachingprogramincludedunderstandinghowtheneighborhoodsthatthestudentswerefrommayhaveaffectedtheireducationaloutcomesandthehighcostofteachinginsmallgroupsInChicagoMatchEducationprovidedtwo-on-onetutoringwhichmatchedstudentswithwell-educatedindividualswhohavenothadformalteachertrainingforalower-costmodelTheresultswerepositiveStudentswhoweretutoredperformedbetterinmathandweremorelikelytopasshighschoolclasses
ThegeneralizabilityframeworkproposedbyJ-PALisbelowUsethisasatemplateasyouconsiderpotentialpolicysolutions
Decideonapolicytoimplementbasedonlocalfactors
30
SourceTheGeneralizabilityPuzzle
OnceyouhavenarrowedthepolicyoptionsdiscussthequestionsbelowforeachoptionsyouareconsideringThisstepiscriticalasunderstandinglocalcontextissoimportantinsuccessfulpolicyimplementation
CommunityContextQuestionnaire
Localcontext
DoesthisoptionmakesenseinourcommunityIsthisaonesizefitsallapproachIsanyparticulargroupabsentfromthisapproachIfsoisthatok
Youranswers
Effectiveness
WhatwouldweexpecttoobserveifthepilotisaseffectivehereasinCityX
Youranswers
Timing
DoesthetimingmakesenseIsthistherighttimetoimplementthispolicychange
Decideonapolicytoimplementbasedonlocalfactors
31
WherearethepotentialroadblocksthatcouldaffectthescheduleIsthereanywaytomitigatethisHowlongwouldittaketoseeanimpact
Youranswers
Implementation
DowehavethestaffcapacitytoimplementthisArethereadditionalpartnersthatyoucanbringtothetabletosupportimplementationDoesthismakesensegivenourformofgovernment
Youranswers
Authority
DoesourcityhavetheauthoritytoenactthisIfsowhatistheappropriatemechanismAretherelegalbarriersArethereanyexistingpreemptionissuesfromthestateorfederalgovernmentCanthosebeaddressedthroughlegislationadvocacyIstherepoliticalsupportoroppositiontothis
Youranswers
Cost
HowmuchwillthiscostannuallyHowwillwepayforitCanthepolicybeimplementedwithexistingfunding
IfsoisthisfundingreliableHowwouldthispolicyoptionaffectourcitysbudgetintheshortandlongtermWhatotherfundingoptionscanweconsiderIsthereanalternativethatwouldcostlessandgiveussimilarresultsWillanyconstituencybeadverselyaffectediffundingforprogramsthatimpactthemisshiftedtoanotherprogrampolicyorinitiative
Youranswers
Decideonapolicytoimplementbasedonlocalfactors
32
EstablishaframeworktomonitorprogressNowthatyouhavereviewedtheevidenceconsideredlocalcontextandchosenyourpolicyitistimetodeterminehowyourcitywillmonitorprogressofthisnewapproachCitystaffandleadersneedtoestablishtheinfrastructureforongoingcontinuousdecisionmakingbasedonwhetherthepolicychangeisleadingtothedesiredoutcomesforresidentsandwhetherthechangeismeaningfulandrationaltofrontlineworkersSettingarbitrarygoalsatthelocallevelthatconflictwithgoalsatotherlevelsisconfusingtotheindividualsresponsibleforachievingthegoalsGoalsthataresettoohightoachievecanbedemoralizingtostaff
ThissectionincludesstepstosetupavenueinwhichCitystaffandleadershiproutinelyvisitkeyindicatorstodeterminewhatisorisnotworkingandtomakedecisionsinreal-timeFormorereadingonperformancemanagementpleasevisittheGovExPerformanceManagementGettingStartedGuide
InperformancemanagementtermssuchasindicatormeasureandmetricmaybeusedinterchangeablyInthisguidewewilluseindicatorInyourworkthemostimportantthingistobeconsistentinwhichevertermyouuseThisvideomaybehelpfulinunderstandingthisconceptandcommunicatingittoothers
Establishthefundamentalsandgatheryourteam
FirstdeterminewhetherthereisanestablishedvenueorregularmeetingscheduleduringwhichyoucandiscussprogressIfyourcityhasanexistingperformancemanagementstructureonarelatedtopicaddingadiscussionoftheprogressofpolicychangetoyouragendacanbeagreatideaIfthiswillbeastandalonemeetingdecidehowfrequentlytheprojectteamanddecisionmakerswillmeetandputthosemeetingsonthecalendarfortheentirelengthoftheproject
DeterminewhowillstaffandattendroutinemeetingsonCityprogressThesemeetingsmayincludetheMayorandorCityManagerdepartmentheadstaffwhoareimplementingthepolicychangeandanalysts
Identifyindicators
ItiscriticalthatindicatorsbeanimportantcityandcommunitypriorityaswellassomethingtheCityplaysamajorroleinaddressingTheremaybeinstancesinwhichtheCitywillneedtoreachouttoorcollaboratewithexternalpartnerstogetamorecompletepictureofprogressontheoverallpolicychange
Establishaframeworktomonitorprogress
33
Anoutcomeindicatordescribesthedesiredeffectofthetargetedpopulation
ExampleBellevuesfamilyfriendlyneighborhoodsindicatormeasuresthepercentofresidentswhobelievetheyliveinaneighborhoodthatsupportsfamiliesparticularlythosewithchildren
WithyourkeypartnersrevisityourproblemstatementcurrentstateandfuturestateexerciseandbegintoidentifytheoutcomeindicatorsthatwillillustratesuccessFornowchooseasingletop-levelkeyperformanceindicator(KPI)ItiscriticalthattheKPI-thepieceyouwillmeasure-beapriorityforthecommunityandsomethingtheCityplaysamajorroleinaddressing
Outputindicatorsmeasuretheprocessofprogramorservicedelivery
ExampleSanDiegoMayorKevinFaulconersgoaltorepair1000milesofstreetby2020
ReviewingtheadministrativedatasetinventorybegintoparedowntherelevantprogramsanddatasetsthatsupportyourKPIWhatarethemeasuresofeffectivenessforeachprogramTheseareyouroutputindicators
Thediagrambelowillustratestherelationshipbetweenoutputandoutcomeindicators
ThinkaboutimplementationindicatorsasmeasuringstepsintheprogramsrolloutthatarenecessarytoobtaindesiredchangesImplementationoutcomescanfocusontheinterventionstaffdeliveringtheinterventionanorganizationorsystemandcanaddressfeasibilityadaptabilitysustainabilityandcostTwokeyquestionstoconsiderwhendevelopingimplementationindicatorsare
1 Istheinterventionacceptableandappropriatetothecommunityand
2 IsitfeasibletoimplementinterventionXatthistimewiththeseresources
Establishaframeworktomonitorprogress
34
Amoreextensivelistwithdefinitionsofpotentialimplementationoutcomesislinkedhere
UsethetablebelowtochoosethedesiredimplementationoutcomesforyourpilotandtoplanyourmeasurementtechniqueThisshouldincludelevelofanalysis(individualproviderorconsumerorganizationorsetting)yourmeasurementtechnique(surveysobservationinterviewfocusgroupsadministrativedataanalysis)andthephasewhenyouwillcollectthisinformation(earlymidlate)Theremaybemorethanonelevelofanalysismeasurementtechniqueandphaseofdatacollection
Implementationoutcome
Definitionofoutcome
Levelofanalysisandmeasurement
Yourmeasurementtechnique
Phaseofpilotwhenyouwillcollectdata
Usethespacebelowtocaptureyourindicatorwishlist
Indicatorwishlist
Outcomes_____________________________________________________________________________
Outputs________________________________________________________________________________
Implementationoutcomes______________________________________________________________
Setapublichigh-levelgoal
Thehigh-levelgoaliscriticalinestablishingdirectiontogalvanizethepeoplewhowillbedoingtheworkWhendiscussingthegoalusetheMayorsbullypulpitLookforkeymilestonesinCitygovernmenttosharethegoalstatementsuchasthebeginningofaMayoraladministrationaStateoftheCityspeechorotherpublicremarks
GoalsshouldbeldquoSMARTrdquo
Specificaspossible(innamingthepopulationwhowillbebetterofftheactionthecitywilltake)
Measurable
Achievable
Establishaframeworktomonitorprogress
35
Results-focused
Time-bound
ThegoalstatementshouldincludeyouroutcomeindicatorlikeChattanoogasgoalldquoWewillendchronicveteranhomelessnessbyDecember2016rdquoAdiagramtohepyoubuildaSMARTgoalaroundyouroutcomeindicatorisbelow
Component Description Yourturn
Specific WhowillbebetteroffWhatactionthecitywilltake
Measurable Howwillyoumeasuresuccess
Achievable Isthegoalattainable
Results-focused Whatistheexpectedresult
Time-bound When
Setrealisticambitioustargetsandtimelinesforeachindicator
ToestablisharealisticandambitioustargetforyourgoalstatementusetheadministrativedatacollectedforyourchosenKPItoreviewactualCityperformanceinthemostrecentyearcomparedwiththethreeorfourmostrecentyearsworthofdataBegintocalculatesummarystatistics(egtotalminimumaveragemedianmaximum)toidentifytrendsCreatecumulativeandmonthlychartstoidentifytrendsandanomaliesBasedonyouranalysisofhistoricaldataignoretheanomaliesandassumetheproductivitytrendswillfollowthesameoverallpatternfrommonth-to-monthFinallyapplyanincreasedproductivityfactortotheperformanceeachmonthtoforecastfutureprogressBasedonthisanalysischooseatargetthatinspiresimprovementwhileaccountingforreality
Manycitiesthataremorematureintheirperformancemanagementpracticesrecognizetheimportanceoftakingacommunity-levelapproachtoimprovingoutcomesforresidentsThesecitiesarecreatinggoaltaxonomiesthatincludemeasuresrelatedtopublicsafetyhousingandevenrecreationCitiesdriveprogressonthesegoalsbyimplementingstrategiesthatengagemultipledepartmentsandcollaboratewithserviceprovidersinthecommunity
EventuallyyouwilldevelopgoalstatementsfortheoutputsinordertocapturetheefforttoachievetheoutcomecontainedinyourgoalstatementYoursetofindicatorsshouldbecomposedofshort-andinterim-termsupportivemeasureswithaccuratetargetsandrelevanttimelinesAnexamplegoaltaxonomyfromtheCityofLasVegassResultsVegasprogramisbelowUsethistaxonomyasamodelforyourownsuiteofindicatorsandgoalstatements
Establishaframeworktomonitorprogress
36
Yourturn
Thetemplatebelowismeanttohelpyoucraftagoalstatementforeachsupportingmeasure
Goalstatement
Departments
Outcome
Activities
KPI
Target
Timeframe
Conductanddiscussrelevantanalysis
ThrougharoutineseriesofmeetingscontinuallymonitorprogressofKPIssupportingindicatorsandanyotherdatacollectedsuchasqualitativedatatoidentifytrendsthatwillindicatewhetherthechosenpolicyishavingthedesiredimpactThesedatapointsshoulddrivethediscussionduringthesemeetingspointingtoopportunitiesforfurtherinquiryinto
Establishaframeworktomonitorprogress
37
whatisdrivingtheobservedtrendsAnalystsshouldbeencouragedandtrainedtoaskquestionsofthedepartmentsproducingthedataandconductsitevisitswhenpossibletodeepentheirunderstandingofthedataPresentationoftheanalysisshouldfocusononeideaatatimetokeeptheconversationfocusedonkeypointsandspecificopportunitiesforimprovementYoumayconsiderdevelopingapublicfacingdashboardtosharethisprogressmorebroadly
AgeneralcomparisonofthestatusquodatawithwhatwasobservedduringthepilotcanprovidedecisionmakerswithaquickunderstandingofthedifferencesbetweentheoldandnewapproachesalthoughthiswillnotshowcausalityOftenthesecomparisonswillincludeoperationalmetricssuchasdaysordollarssavedbutrememberthattheimpactonpeoplewillbemostimportantIfyouareworkingwithanevaluationpartnertheywillsupportthisstep
Usethespacebelowtosummarizetheresultsofyouranalysis
MeasureTargetvalue(seesectiononsettingtargetsabove)
Pilotvalue(whatdidyouobserveduringthepilot)
Historicalaverage(ifavailable)
Results(WhoisbetteroffIsanyonenegativelyimpacted1-2sentencesreassessingyourgoalstatement)
DocumentyourprocessandprogressincludingfeedbackfromtheparticipantsimplementationpartnersandtheprojectteamDidanythingunexpectedoccurduringthepilotArethereinternalandexternalfactorsthatmayaffectyouroutcomesThisiswheretheimplementationoutcomescanbecanhelpfulIftheinterventiondidnotgoasplannedconsiderwhetheritisbecausetheinterventionwasnotacceptableorfeasible
UsethetemplatebelowcreatedbyDenversPeakAcademytocapturethisinformationIfthe306090daytimelinedoesnotmakesenseforyourcityfeelfreetocustomizeit
CurrentState
FutureState
30Day
60Day
90day
Whatwentwell(+)
Whatdidnotgowellshouldbechanged(-)
IfaquestionordiscussionpointarisesthatcannotbeansweredinthemeetinganassignedstaffmembershouldcapturethequestionduringthemeetingAfterwardthefollowupitemsshouldbesenttorelevantpartnersandthenextmeetingshouldbeginwithadiscussionof
Establishaframeworktomonitorprogress
38
thoseitems
Takeaction
DiscussionsaboutwhetheryourindicatorsaremovingintherightdirectionandwhatmightbedrivingtrendsmayleadtoadiscoverythattherearenotenoughresourcesallocatedtothispolicyorissueTheremaybelimitedfundingforstaffprogrammingortechnologythatisdrivingthedatainthewrongdirectionIfCityleadershipisintheroomthisisagreattimetomakethecaseforadditionalfundingduringthebudgetseasonBycollectinganalyzinganddiscussingdataonthispolicyroutinelyyouarebuildingevidencetomakethecasefortherightleveloffundingtoreachtheestablishedtargets
Establishaframeworktomonitorprogress
39
Assessdataaccessibilityqualityandsecurity
Determineaccessibilityandqualityofdata
OnceyouhavecompiledyourwishlistofindicatorsitistimetodeterminetheaccessibilityandqualityofthedataTopreventcomplicationyoumaywishtoprioritizerelevantindicatorsforwhichyouarealreadycollectingdataAvoidthetemptationofadoptingatechnologysolutiontocollectnewdataatthispointsuchasaphotosurveytoolorotherappFornowyouaretryingtodeterminewhichdatasetsarerelevantaccessibleandhighqualityBesuretoworkwithyourkeyinternalpartnerswhileyouareidentifyingyourmeasuresDepartmentstaffwhoworkonprogramsarefamiliarwiththeindicatorsthatwillaccuratelysupporttheoutcomesanddatabaseadministratorswillbevaluablewhendiscussingaccessibilityandqualityofthedata
Abrieflistofconsiderationsfortheaccessibilityofyourdataisbelow
ArewecurrentlycollectingthisdatasetorwillwebecollectingitforthefirsttimeHowfarbackdoestherecordgo(timehorizon)WouldweneednewtechnologytocollectoranalyzethedataCanthedatabecollectedmanuallyorcanitbeautomatedHowfrequentlyisthedataproduced(continuouslyhourlydailyweeklymonthlyquarterlyyearlyetc)IsthedataavailablefortheentireCitypopulationorasubsetofthepopulationHowgranularisthedata(individuallevelCensusblockneighborhoodcitywideetc)DoesthedatacontainsensitiveinformationIsthereaproxymeasurethatwecanusefornowIsthedatamachinereadable
AsnotedbytheWilsonSheehanLabforEconomicOpportunities(LEO)attheUniversityofNotreDameleveragingadministrativedatacansignificantlyreducethecostsandtimelinesofrigorousevaluationCitiescanestablishaninternaldatasharingagreementacrossdepartmentsincludinglanguagethatenablesadministrativedatatobeusedforresearchpurposesCitiescanalsoidentifyexternaladministrativedata-holderssuchasacountyschooldistrictpolicedepartmenthospitaloranotherstatethatmayhaveoutcomedataimportantformeasuringprogramimpactCitiescanleverageexistingrelationshipsandorworkwithresearchinstitutionstodevelopstreamlineddatasharingprocessesandagreementswiththeseexternalentities
Assessdataaccessibilityqualityandsecurity
40
Evaluatethequalityofyourdata
QualityisasimportantasaccessibilityToevaluatedataqualityyoumaywishtocreatearubricliketheonebelow
Value ValueDescription
3 ExcellentDataQualityThedataiscompleteaccurateandupdatedregularly
2 AdequateDataQualityThedataissomewhatcompleteaccurateandupdatedfairlyregularly
1 PoorDataQualityThedataisnotcompleteaccurateorupdatedregularly
ThisisagreattimetobuildonthedatainventorythatyoustartedinthegatherinformationpageSuggestedadditionalfieldstoaddtoyourdatainventorytemplatearebelow
Program DatasetTitle
DatasetDescription DataSource
Departmentorexternalpartner
DoestheCityalreadycollectthisdata
Whatisthenameoftheprogram
Human-readablenameofthedataset
Shouldbeunderstandablebynon-technicalusers
Whatistheoriginalsystemapplicationfilethathousesit
Departmentorexternalpartnerthatmaintainsthedata
Isthisdataalreadycollected
Updatefrequency-ThefileformatforthedatasetTypicallycsvxlsxshapefiletxtetc
Format-Howoftenthedatasetisupdated
Timerange-Theperiodoftimecoveredinthisparticulardataset
Datause-Howisthisdatasetcommonlyused
Datausers-Whotypicallyusesthisdataset
Dataownerstitle-Titleofthedataowner
Dataownersposition-DataOwnersjobtitle
Dataownersemailaddress-DataOwnersemailaddressWhomanagesthedataandorisresponsibleforgrantingpermissiontoaccessthedataWhounderstandswhatthedatasetincludesandcananswerquestionsaboutit
Dataqualityconcerns-Describetheconcernsyoumayhaveaboutthequalityofthedataset
Assessdataaccessibilityqualityandsecurity
41
Division-DivisionwherethedataismaintainedIfmorethanonedivisionisresponsiblelisttheprimaryownerofthedataset
Sensitivedatacomments-DescribetheissuesthatwouldpreventthecityfromsharingthedatapubliclyCommonsensitivityconcernsincludeprivacyviolationssecurityissuesorhighcostorstaffdemandstopostthedata
Publishingstatus-Thistheplacetoputanycommentsregardingthecurrentstatusofthedataset
Publishedlink-Thisisthelinktothepublisheddatasetontheopendataportalorcitywebsite
Reviewdatasecurityandprivacyprotectionprotocols
OftenapolicychangealsoinvolvesachangeinthedatathatisbeingcollectedhowthatdataisbeinghandledwhohasaccesstoitandahostofotheradjustmentsAsyoutakestockofthedataassetsthatareavailableforyourinitiativeitisvitaltoalsothinkthroughwhatwillbedonetoprotectthatdataandtheprivacyofthepeopleitrepresentsAbigpartofgooddatasecurityinvolvesstrongdatagovernancepracticesthatdefineandidentifysensitivedataandconsciouslyworktobalancethevalueandriskofstoringsuchsensitivedataOnewayofevaluatingyourdatacomesfromcaselawrelatedtotheUSFreedomofInformationActwhichwehavetranslatedtothechecklistbelow
1 Woulddisclosureresultinasubstantialinvasionofprivacy2 Haveweconsideredtheextentorvalueofthepublicinterestandthepurposeorobject
oftheindividualsseekingdisclosure3 Istheinformationavailablefromothersources4 Wastheinformationgivenwithanexpectationofconfidentiality5 Isitpossibletomouldreliefsoastolimittheinvasionofindividualprivacy
TheseriskscouldbemitigatedusingavarietyofstrategiesfromaggregatingdatatoobfuscatingindividualstoredactingfieldsthatmayincludePersonallyIdentifiableInformation(PII)orre-evaluatingyourdatacollectionmethodstofindalternativewaysofgettingtheinformationyouneedThesestepsshouldbecodifiedandmanagedbyadatagovernancecommitteeifpossibleForthesensitivedatathatisdeemedessentialcreateaplanaroundpotentialdatabreachesinordertoquicklyandconfidentlyrespondtoemergencysituations
ProactivelypreparingforthesesituationsisoftenthebeststrategyfordatasecuritybutdonotforgetthetechnicalsideInvolveyourITadministratorsinconversationsaboutdatastorageandaccessensuringthattheywillbeprotectedbyenterprise-gradecybersecuritysystems
Assessdataaccessibilityqualityandsecurity
42
Assessdataaccessibilityqualityandsecurity
43
Designimplementationandevaluation
Designtheimplementation
NowthatyouhaveyouraccountabilityplanandkeyindicatorsitistimetogetspecificaboutyourplanforimplementationThetemplatebelowwillhelpyouthinkthroughthespecificmechanismsofimplementingpolicychange
UnderstandingorganizationsandsystemsiscriticaltounderstandingandmakingdecisionsaboutwhereandhowinterventionsshouldbeappliedApolicyimplementedinasinglesettingmayshowpromisingresultsbutimplementingmultiplepoliciesandprogramsatthesametimewithdifferentactorsresponsibleforeachaspectcanhelpensuresustainablebehaviorchange
InSomervilleMAShapeUpSomerville(SUS)isacitywidepublichealthprogramthatisimplementedacrossmultiplelevelsofinfluenceandsystemsthroughinterventionsbeforeduringandafterschooltoincludechangestobuiltenvironmentsuchasbikelanesregionalmasstransportationandphysicaleducationequipmentinschoolsandgymstoincreasephysicalactivitywellnesspoliciesandschoolgardensnutritioneducationduringandafterschoolandhealthyrestaurantprogramstosupporthealthyfoodandhealthyeatinghabits
Usethetemplatebelowtoplanyourimplementation
Policyimplementationplanningtemplate
ListthespecificdesiredoutcomesInthepreviousstepyouworkedwithdepartmentstafftoestablishspecificmeasurableoutcomesandinterimindicatorsPrioritizetheindicatorsthatwilldemonstrateprogressUsethespacebelowtocapturethedesiredoutcomesofthepolicychange
ExampleShapeupSomervillebeganwiththegoalofpreventingobesityin1stthrough3rdgraders
Yourturn
____________________________________________________________________________
____________________________________________________________________________
Designimplementationandevaluation
44
SpecifythetargetpopulationAreyoudesigninganinterventionthattargetsonlycertainsitesAreyouplanningonimplementingaprogramorinitiativeforaspecifictargetaudienceClearlyspecifythepopulationyouplantotarget
IfyouplantoconductarigorousimpactevaluationasnotedbytheWilsonSheehanLabforEconomicOpportunitiesitisimportanttoestablishconcreteobjectiveeligibilitycriteriaandensurethatthesecriteriaareappliedconsistentlybyallstaffduringscreeningandintakeItisalsovitaltoidentifytheindividualswhoareeligibleforaparticularprogramorserviceevenifeveryoneeligiblecannotbeservedbyexistingresourcesThishelpscitiesidentifyanyunmetneedsandguidefutureresourceallocationwhilealsocreatingthepossibilityofarigorousimpactevaluationbyidentifyingapotentialcomparisongroupFinallywhenconductingscreeningandintakeitisimperativetocollectandmaintainbasicinformation(egnamedateofbirthgender)aboutallprogram-eligibleindividualsnotjustthosewhoparticipateintheprogram
ExampleIntheShapeUpSomervilleexampletheprogrambeganbytargeting1stthrough3rdgraders
Yourturn
____________________________________________________________________________
____________________________________________________________________________
DefinethelocationWherewillthepolicychangetakeplaceWillimplementationtakeplaceatonesiteormultiplesites
ExampleShapeUpSomervilleinterventionsweredeliveredinschools
Yourturn
____________________________________________________________________________
____________________________________________________________________________
ClarifyimplementationpartnersandarticulatetheirrolesWhatarethekeyorganizationsinstitutionsorpeopleresponsibleforimplementationTheprojectteamshouldbeasclearandspecificaspossibleinprovidingadescriptionoftheconcreteactivitiesthatimplementationpartnersaremeanttocarryoutPracticeprofilesmaybeahelpfultooltohelpdefinewhatisrequiredoftheactorswhoareresponsibleforimplementingtheintervention
Designimplementationandevaluation
45
ExampleInSomervillemayoralleadershipandengagementofmultiplecityagencies(publichealthprimarycareschoolsCambridgeHealthAllianceetc)streamlinedandconnectedthevariousShapeUpSomervilleinitiativesandgrantswhichhelpedtokeepthetalentandenergyfocusedonhealthyeatingandactivelivingforallsectorsofSomervilleSUSpartnersultimatelyincludedschoolsnonprofitsCityandstateagenciesandnationalfoundations
Yourturn
____________________________________________________________________________
____________________________________________________________________________
DescribetheinterventionWhatarethestepsactivitiesorprocessesRememberthatthemechanismthespecificstepsorprocessthatleadtobehaviorchangeiskeytounderstandingtheeffectivenessofthepolicychangeandsharingtheresultswithothers
ExampleGoalsofShapeUpSomervilleweretoinfluenceeverypartofanearlyelementaryschoolchildrsquosdayndashbeforeduringandafter-schooltopromotehealthyeatingphysicalactivityandtheirweightstatusInterventionincludedfoodserviceenhancementstoimprovebreakfastandlunchandstaffprofessionaldevelopmentwalktoschoolactivitiesSUSclassroomcurriculumandprofessionaldevelopmentschoolwellnesspolicydevelopmentSUSafter-schoolcurriculumandprofessionaldevelopmentoutreachandeducationtohomethroughmaterialsforumseventsldquoSUSapprovedrdquorestaurantscommunityoutreachandcapacitybuildingthroughpolicydevelopmenttrainingsmediaplacements
Yourturn
____________________________________________________________________________
____________________________________________________________________________
ClarifytimelineandmilestonesfordeliveryandevaluationHowfrequentlywilltheservicesinterventionbedeliveredHowlongwilltheprogramrunWhatisthetimetoexpectedimpacts(eg12monthsafterprogramcompletion)
Designimplementationandevaluation
46
ExampleShapeUpSomervilleledbyTuftsUniversityFriedmanSchoolofNutritionScienceandPolicyconductedaquasi-experimental(nonrandomized)interventionstudyamong1stthrough3rdgradestudentsinSomerville(intervention)andtwocomparisoncommunitiesfor9-monthsduringoneschoolyear(fall2002ndashspring2003)ThestudyshowedpositiveassociationoftheinterventiononchangeinBMISUSreleasedreportssummarizingevaluationsprogresstodateplansforadditionalprojectsandimpactstenandfifteenyearsintotheinitiative
Yourturn
____________________________________________________________________________
____________________________________________________________________________
PlanyouranalysisTheprevioussectionaddressedestablishinganaccountabilityframeworktomonitorprogressandimplementationDeterminewhichindicatorswillbediscussthroughthatprocesswhowillsharethedataandanalysisandhowfeedbackfromthoseconversationswillbeincorporatedbytheprojectteam
ExampleTheMayorrsquosofficeimplementedadata-drivendecisionmakingapproachcalledSomerStatwhichwassynergisticwiththeapproachestakenbytheresearchandevaluationteamsdeployedforthespecificgrants
Yourturn
____________________________________________________________________________
____________________________________________________________________________
PlanforsustainabilityIfthepolicyandprogramchangeissuccessfulhowwilltheprojectteamsustainandexpandtoadditionalsitesorpopulations
ExampleAlthoughmostofthepartnershipsforgedintheearlyyearsweremaintainednewgrantsandcollaborationsexpandedattentiontoolderpublicschoolstudentsactivetransportationandtheGreenLinesubwayextensionthebuiltenvironmentfarmerrsquosmarketeffortssuchasldquoGrowninSomervillerdquowithUnionMainStreeteconomicdevelopmentandhealthdisparitiesaswellasdatacollectionmonitoringandsurveillance
Yourturn
Designimplementationandevaluation
47
____________________________________________________________________________
____________________________________________________________________________
Pilotplanningtemplate
IfyouplantoconductapilotfirstdeterminethepurposeofthepilotDoyouwanttoobservewhetherthepolicychangeresultsinthedesiredbehaviorchangeAreyouconcernedaboutfeasibilityscalabilityorcostDoyouwanttotestwhetheryourdesiredresultscanbeachievedthroughthispolicyapproachClarifythegoalofthepilotandthequestionsyouhopetoanswerThiswillhelpyouidentifythekeyindicatorsthatyouwillusetomeasurewhetherthepilotissuccessfulBeforebeginningestablishabaselineofcurrentperformanceandsetyourtargetsaccordinglybasedonwhatyouhopetoaccomplishthroughthepilot
NextestablishthepilotplanforyourprojectteamincludingmilestonesforteamcheckinstodeterminewhethertomoveaheadorchangecourseThisisseparatefromthemeetingsthatwilloccuraspartoftheaccountabilityframeworkUsethespacebelowtocreateyourtimelineanddevelopyourmeetingschedule
Task Personresponsible Startdate Checkindate Deliverable
Planyourevaluation
WhileyouaredesigningtheimplementationitisalsotimetobegintodevelopyourplanforanalysisAsnotedbytheWilsonSheehanLabforEconomicOpportunitiesdecisionsaboutprogramsetup-includingoutreachscreeningintakedatatrackingandenrollment-candeterminewhetherornotarigorousimpactevaluationwillbepossibleAsdiscussedcityleadersmayfindithelpfultoincludeevaluationpartnersinearlyconversationsaboutprogramsetupDuringpreviousstagesyoumayhaveworkedwitharesearchertoplanyourproposedpolicychangeYoushouldbeclearaboutthepurposeoftheevaluationanddevelopanevaluationplanBelowareadditionalconsiderations
Causationvstrendanalysis
IntheEstablishaframeworktomonitorprogressstagetherewasguidanceontrackingtrendsovertimeItisimportanttonoteherethatmonitoringdataisdifferentfromanalyzingacausaleffectTrendanalysiswillillustratethatchangesareoccurringandmayleadtoquestionsthatcouldbeansweredbyyourevaluationbuttrendanalysisalonedoesnotprovideevidenceaboutwhatisdrivingthosechangesYoumayseechangesinyour
Designimplementationandevaluation
48
outcomeindicatorsbasedonvariablesoutsideoftheCityscontrolForinstanceachangeinfederaltradepolicycanhaveanimpactemploymentratesinyourCityoutsideofyourCitysnewjobtrainingandplacementprogram
ToaddresscausalityyoumustcomparetheoutcomesofindividualswhoreceivedtheinterventionwiththosewhodidnotWithoutacomparisongroupofsimilarindividualswhowerenotexposedtotheinterventionitisimpossibletoknowwhichfactorsareresponsiblefortheoutcomesRandomizedControlTrials(RCTs)inwhichindividualsarerandomlyassignedtotreatmentandcontrolgroupsarethegoldstandardinprovingcausalityIfyoucandemonstratethatoutcomesimprovedforthosewhoreceivedtheinterventionanddidnotimproveatallorimprovedlessforacomparisongroupyoucanbesomewhatcertainthattheinterventionwasthecause
Examplesofothernon-causalimpactevaluationsthatarerigorousandreliableincludecertainquasi-experimentalstudiesusingcomparisongroupdesignssmartinterruptedtimeseriesanalysesthatmodeltrendsovertimeinwaysthatallowestimationofinterventioneffectswelldesignedpre-posttestanalysesneedsassessmentsimplementationstudiesetcFormoreinformationondifferenttypesofevaluationsseetheSageResearchMethodswebsite
Lookfordifferencesingroups
BuildintimetoalsoexaminethepilotdatatoseeifgroupsareexperiencingdifferentresultsReviewingperformancebasedonoutcomemeasuresforapopulationinaggregatemayhidethesubtlewaysthatsomepeoplemaybeaffecteddifferentlysoitisnecessarytocarefullyanalyzethedatafordisparateoutcomesForinstancetheMovingtoOpportunity(MTO)studythatexaminedtheimpactofhousingvouchersonfamilieswholivedinhigh-povertyneighborhoodsfoundthatchildrenwhomovedbeforetheageof13hadbettereducationandhigherincomesasadultsHoweveradultsdidnotseeimprovededucationandemploymentoutcomes
Disaggregatingdataintomorespecificcategoriesofraceandethnicitycanhelpunderstandtrendsanddisparitiesinamoregranularwaywhichcanhelpdecisionmakerstargetpoliciesandinterventionsinamoreequitableway
EarlierintheRoadmapweworkedtoidentifystakeholdersandwhomightbepositivelyandnegativelyimpactedbyachangeinpolicyUsetheresultsofthatexercisetoinvolvethecommunityininterpretingdataanalysisandtogatherqualitativefeedbackfromthecommunitytocallattentiontoissuesForinstanceanincreaseinyouthdetentionmaybeexplainedbyinefficienciesorlackofaccesstotransportation
Designimplementationandevaluation
49
Designimplementationandevaluation
50
Shareyourstoryandsuccess
Sharetheresults
Storytelling
AnintegralstepinyourstrategyissharingtheresultsofyourpilotorpolicychangewithyourcommunityThisincludesdevelopingmessagingaroundtheworkofyourorganizationandcreatingaplantodisseminateyourstoriesWhytellstoriesaboutyourwork
Informandorunderstandasubjectorsituation
Giveneededcontextofdataorbitsofinformation
Relatablestoriesarememorableforreadersviewers
ReadersorviewerswillseetheinformationasmorecompellingwhenthereisademonstratedandrelatablechangeorimpactonthemselvesortheirfriendsneighborsandcolleaguesStoriesareaboutpeopleandgovernmentisfundamentallyabouthelpingpeopleSotellthestoriesofthepeopleyouandyourcitycolleagueshelped
Keepthesetipsinmindasyoudevelopyourstorywithyourteam
WhatwereyourfindingsandwhyshouldanyonecareTellyouraudiencehowtheproblemwassolvedWhathastheimpactbeenonresidentsandcitystafftodateWhatwasthesuccessandweretheremultiplesuccessesalongthewayIfyouhavepreliminaryresultsordatashowingimprovementtosharethisisthetimetodoit
WhowasinvolvedandimpactedReviewthelistofstakeholdersHowwereeachofthemimpactedbytheprogramHumanizeyourstoryandimpactbyaddingadditionalvoicesThesevoicescanbeoneofthefollowing
Apersonorfamilywhowaspositivelyaffectedbythechangesuccess
Aninvolvedthird-partydescribingboththeproblemandthesolution(ieanorganizationnonprofitprovidersbusinesstradeassociationsportsteametc)whoispartofthecommunitythatthegovernmentworkedwithonthechangesuccess
Someonefromthegovernmentitselfwhocanspeakaboutthechangesuccess
TellmoreaboutyourcityDescribeyourcityincludingpopulationdemographicsindustriesetcWhatareitschallengesWhatisitknownforWhatmakesituniqueHowisitsimilarordifferentfromothercitiesintheUS
Shareyourstoryandsuccess
51
DescribeyourcityrsquoschallengeWriteoutthechallengeyourcityisfacing(asdefinedinyourproblemstatement)HowdoesitaffectresidentsHowdoesthisproblemaffecttheworkofcitystaffmembers
AddthedetailsProvidedetailsonyourappliedresearchstrategyTalkaboutwhatyourcityhasdonenthepasttoresolvetheissueDiscusshowyourstrategyhaschangedwhenyoubegantoadoptthisapproachWasmoneysavedorinvested
Usethisstorymapexercisetoorganizeyourthoughtsandguaranteethatyouincludealloftheelementsofatraditionalstory
AfteryoudevelopyourmessagingcreateacommunicationsmarketingandoutreachstrategydesignedtospreadyourmessageYourstrategyshouldwalkyouthroughthefollowingkeyactivities
1 IdentifyandprofileyouraudienceWhencreatinganddisseminatingyourmessageitisalwaysgoodtothinkaboutyouraudienceWhoexactlywouldbeinterestedinwhatyouhavetosayThegeneralpopulationofthecityPeoplewhoworkinnonprofitsBusinessownersItmattersbecausenoteveryonewholivesinacitymaybethetargetoftheinformationBusinessownersmaybemoreinterestedindatashowingeffectsonthebusinessdistrictthanothersandsoonWewillcoverthislaterbutthisstepwillalsohelpyoutothinkofthemosteffectivewaytoreachyouraudience
2 Defineyourcalltoaction(ifyouhaveone)SpendsometimeconsideringwhyyouareinterestedintellingyourstoryDoyousimplywanttoupdatethepubliconyourprogressorcelebrateyoursuccessesAreyouinterestedinyouraudiencechangingitsbehaviororsupportingaspecificinitiativebeinglaunchedbyyourorganizationIfyouwanttohaveyouraudienceactorbehaveinacertainwaybespecificandincorporateitinyourmessaging
Shareyourstoryandsuccess
52
3 SelectperformancemeasuresThinkabouthowyouwoulddetermineifyourcampaignissuccessfulandselectappropriateperformancemeasuresYouwillrevisitthesemeasuresattheendofyourcampaigncommunicationsplanimplementationtodeterminewhetheryouneedtomodifyyourcommunicationsandoutreachstrategyforthefuture
4 MethodstocommunicateyourstoryTherearemanywaystocommunicatesuccessofyourprojectHerearesomesuggestionsonreachingaudiencesItisimportanttodeterminewhichmethodsarethemosteffectiveforyourdesiredaudiencetoreceiveinformationDoyourresearch(PewResearchCenterhasexcellentstudiesonthereachofmediadigitalmediaandmarketing)ForexampleifseniorsareyourtargetpopulationandtheyarelessinclinedtouseInstagram(thatistrue)youmaywanttoconsiderFacebookorthenightlylocalnewsbothofwhichareusedmoreoftenbyseniorsBelowisalistofsomemethodsandtoolsyoucanuse
i Pressreleasestolocalnewspapersandtvstations
ii ShortsmartphonevideosAgreatwaytoengageanaudienceisthroughvideoAskonequestionofsomeonewhoispartofthestoryMaybethereisasocialworkerwhocandiscusshungerinyourcityThesevideoscanbereusedonmultipleplatforms
iii Long-formwriting
iv TellingindividualstoriesthroughaseriesofphotosandshortnarrativesforpublicationonFacebookorInstagram
v Op-edsopinionpiecesplacedinthelocalpaper
vi Emailupdatestoyourlist
vii Connectingwithappropriateorganizationsthatareinvestedinthetopicandworkwiththecommunityandaskthemtodisseminateinformationtotheirmembers
viii EmployingldquoWordofMouthrdquo(WOM)tacticsbycreatingtownhallsorfreeeventsforresidentstoattendintheirownneighborhoods
5 DeterminethetypeofcollateralUsuallycampaignsthattargetresidentsrequireseveralavenuestoreachthemPeoplearenotalwaysinthesameplaceorspaceHavingakeenunderstandingofyouraudiencewillassistyouwithyourchoicesonwhatneedstobedevelopedSometimesthatmeansimagesforsocialmediaaonepageflierthatgivespeopleinformationonattendinganeventoranin-depthbrochure
Shareyourstoryandsuccess
53
6 ImplementyourstrategyKeepinmindstrategiescanbefluidSometimessomethingmaynotturnoutasexpectedevenifduringplanningsessionsitwasldquosupposedtordquoItisokYoucananalyzewhathappenedduringadebriefsessionandonceyoursurveysandothermeasurementshavebeenfinalized
Visualizations
Withtherightvisualizationyoucanreviewandshareinformationinsightsanalysisperformanceandtrendsallofwhichisvaluabletoacross-sectionofthepublicButwhichvisualizationisrightforyouTochoosetherighttypeofchartgraphdeterminethetypeofquestionyouwanttoanswerSomecommoncharttypesarebelowandmoreinformationcanbefoundhere
Exploreadditionalanalyticaltechniques
Duringthegoal-settingphaseyouestablishedoutputoutcomeandimplementationmeasuresBasedontheresultsofyourpilotyoumayneedtoadjustyourmeasuresWiththepilotdatainhandyoumayhavesomeideasofhowfutureanalysescouldbedifferentandpotentiallyimprovethequalityofyourprogramTakesometimetobrainstormotherrelevantdatasetsthatyoudidnothavethetimetopursueearlierincludingmessydepartmentaldatathatneedsmoreattentionorsourcesofpublicdatafromthestateandfederalgovernmentnonprofitsandevenprivatecompaniesthatcouldsupplementyourunderstandingofanissueUsingmachinelearningtechniquesyoumightbeabletobetter
Shareyourstoryandsuccess
54
predictdemandforhomelessservicesusingNOAAweatherforecastspreemptwatermainbreaksusingsoilstratigraphyorinvestineconomicdevelopmentbymonitoringrealestatelistingdata
Advancedanalyticaltechniquescanalsohelpyourimplementationrunmoreeffectivelybyintegratingdata-informedtargetinganddecisiontoolsintotheprocessByautomatingroutineworkflowsandsurfacingsubtletrendsthesetoolscanhelpyouiterateinamoreprecisemannerandprovidekeystrategicguidancetofront-lineemployeesthatwouldotherwisebeunavailableThesesortsofdata-science-enhancedprocessescanbeclassifiedasoneofsixprojecttypesasexploredfurtherbytheNewOrleansLANOLAlyticsteamFormoreguidanceonscopingasimilarprojecttheGovExadvancedanalyticsscopingguidegivesyouastep-by-stepoverviewofhowtobesuccessful
Usethespacebelowtolistyourideasforpossibleadvancedanalyticsprojectsbytype
Typeofproject Ideas
Findingtheneedleinthehaystack
Prioritizingforimpact
Earlywarningtools
Betterquickerdecisions
Optimizingresourceallocation
Experimentingforwhatworks
Keepthefeedbackloopgoing
GatherinformationonpublicsentimentaboutpolicychangethroughfocusgroupssurveysandpublicmeetingsledbyadvisorygroupmembersNotewhethernewinformationisavailablethatcanhelpanalyzeorcontextualizeyourprogress(egfederalorstatedatareports)Sharetheresultsofyourpilotandthepolicychangeprocesswithothercitiesandresearchers
Shareyourstoryandsuccess
55
WorkingwitharangeofevidencePresentlythereisalotofattentionandinvestmentinevidence-basedpolicymakingincludingthefederalbipartisanCommissiononEvidence-BasedPolicymakingthePewCharitableTrustsreportsandtoolkitsonHowStatesEngageinEvidence-BasedPolicymakingandtheLauraandJohnArnoldFoundationsupportedEvidence-BasedPolicymakingCollaborativeManyoftheseinitiativesfocusonrigorousevaluation--meaningrandomizedcontrolledtrials(RCT)orotherexperimentalmodels--asthebestkindofevidencetouseHoweverthisbodyofevidenceisofteninsufficientforthespecifickindsofquestionsthatgovernmentscanhavewhenmakingadecisionaboutpolicyInthisguideweproposeanapproachtosupportgovernmentsinincorporatingadditionaltypesofevidenceintotheirdecisionmaking
OnewaytounderstandarangeofevidenceistouseNestarsquosStandardsofEvidencetoframeanddefineevidenceTheBritishnonprofitNestaworkstoinjectpositivechangeintothewayUKgovernmentpoliciesandprogramsfunctionOneoftheirkeyareasofworkhasbeendevelopingWhatWorksCentresorganizationsthatcollectcommissionsynthesizeanddisseminateevidenceaboutinterventioneffectivenessacrossarangeofsocialimpactareasNestadevelopedthefollowingStandardsofEvidencetoprovideausefulwaytodocumenthowdifferentkindsofinformationdemonstratedifferentlevelsofevidence
Workingwitharangeofevidence
56
Theimportanceofpotentiallybiasedinformation
Onechallengepresentedbyanevidenceclassificationthatacknowledgesthevalueofnon-scientificformsofevidenceistofigureouthowtobalanceallofthepiecestogetherwhenyouusethemTounderstandhowtodothiswehavetothinkabouthowtopositionpotentiallybiasedinformationindecisionmaking
WhenwelookatinformationwhichhasnotbeenrigorouslyevaluatedweacceptthatthisinformationmightbebiasedThewholereasonweconductrigorousevaluationistoremovebiasfromourobservationsWeconductstructuredtrialstoproduceresultsthatareconsistentandgeneralizableIfafindingisunbiasedthisaccuracymdashthislackofbiasmdashcanbeprovedbecauseanyindependenttestercanreproduceitwhentheyfollowanidenticalprocess
YoucanlookatpotentiallybiasedinformationasjustinformationthatisearlieroninitspathtorefinementreplicationandverifiedaccuracyApieceofinformationbasedonanyoneindividualrsquosobservationsislikelytobebiasedbecauseitiscomingfromjustonesourceBiasisreducedbygatheringtogetherlotsofindependentsourcesofinformationandfiguringoutitsmostcommonattributes
MaybethebiasedinformationmakesaclaimthatistoolargemdashitsayssomethingistrueunderallconditionswhenitisreallyonlytrueundercertainconditionsorthataneffectofaninterventionisbiggerthanitstrueaverageorthataneffectlastsovertimewhenitdoesnotPerhapschangingtheclaimtobemorespecificmdashcriticallyreviewingtheinformationtoseeifspecificconditionsmustbemettoachievetheoutcomesmdashwouldmaketheclaimmoreaccurate
IfyoutestedyourclaimfounditdidnotholdupinsomecaseschangedittesteditagainuntilyourclaimrepeatedlyheldupthenyoucoulddemonstratethattheclaimwasnolongerbiasedbutaccurateThisprocessoftestingimprovingtestingandvalidatingiswhatwegetfromstructuredscientificstudy
Withindividualpiecesofinformation--crediblehypothesescasestudiesexpertopinion--youcannotassumethatkindofaccuracyAtthesametimetheseearlyobservationsarefoundationaltoanyreplicablescientificobservationsWewillnevergetanywhereifwedonotstartwiththesepiecesofinformationeveniftheymightbebiased
Combininglevelsofevidencetogetthemostdetailedinformationforyourcity
ItisespeciallyimportanttoacceptdifferentformsofevidencewhenyoufacequestionsthathavenotyetbeentestedbyindependentstudiesSomeofthemostimportantquestionsrelatetothingsthatareveryspecifictoyoursituationYoufaceaparticularcombinationof
Workingwitharangeofevidence
57
issuesactorstrendsandresourcesthatisultimatelydifferentfromeveryotherplaceintheworld
ThisisimportantbecauseaspolicypractitionersandnotscientistsyoucaremoreaboutyourspecificcasethanahypotheticalaveragegeneralcaseYoucarelessaboutthetotallikelyrangeofimpactsfromapolicythanhowthissingleimplementationwillworkYouwantinformationaboutpoliciesthatwillprovideyouwiththebestpossiblepredictionabouthowtheywillworkinyourcontext
ByputtingdifferentkindsofevidenceintothesameframeworkNestarsquosStandardsofEvidenceimplythatwhenyoucannotgetrigorouslytestedexperimentallyverifiedinformationitisappropriatetoworkwithwhatwedohaveThisisrelevantforthemunicipalimplementationcontextbecausenostudypulledfromaliteraturereviewwillaccountforallofthevariablesyoufaceinyourowncontext
Intheabsenceofhavingastudythatpredictshowaninterventionworksinyourspecificcontextyouwillneednon-scientificinformationtotrytofillinthosegapsYouwillneedtogatherstoriesfromresidentsstaffmembersandotherstakeholdersthatsitonLevels1or2ofthestandardsofevidence
Someofthekindsofinformationyouwillprobablygetmoreeasilyfrompeoplethanstudiesinclude
howpeopleincitieslikeyoursreactedtoaninterventioninpublicforumshowpeoplewhoimplementedaninterventioninacitylikeyoursfiguredouthowtogetitpassedhowthecitycommunicatedpubliclyabouttheapparentbenefitsanddrawbacksoftheinterventionthepoliticalacceptabilityoftheintervention
Youcanremainsimultaneouslyawarethattheinformationyouaregatheringthroughindividualstoriesisbiased--notgeneralizableinformedbyspecificperspectives--whilealsorecognizingthathavingsomeinformationaboutsomethingimportantisbetterthannoinformationEvidencefromthefirsttwoevidencelevelsareessentialbuildingblocksofknowledgeandperfectlyusefulwhenyoucannotgetdetailedunbiasedknowledge
Youhavetostartsomewhereandyoushouldalwaysstartwithevidenceyouhaveaboutyourveryspecificconditionsthatyouknowareimportantforsuccess
Workingwitharangeofevidence
58
Thankyou
Interviewswithcities
GovExheldsemi-structuredinterviewswithsixstaffmembersinfourcitiestoestablishtheinitialstepsoftheRoadmapWeaskedrespondentstotalkaboutaninitiativeorpolicythattheyhavebeenresponsibleforimplementingintheircityRespondentsfromthreecitiesspokeabouttheireffortstoaddresshomelessnessandrespondentsfromonecityspokeaboutblightAfterrespondentsidentifiedtheproblemtheyspokeabouthowthecityprioritizedthisissuethevariousstepsinvolvedandbarrierspresent
GovExconductedtextanalysisoftheseinterviewswhichrevealedthatpeoplewasthemostcommonlyusedtermduringtheseconversationswhendiscussingthestepstochangeaprogramorpolicyThisisathemethroughoutthisRoadmapThefirstsectionbeginswithidentifyingtherelevantpeopleTheconversationsaroundevidencearefocusedonhumanbehaviorratherthanmethodologyorstrengthofevidence
Thankyou
59
OncewehadadraftoftheRoadmapwefollowedupwithcitiesonanindividualbasistolearnmoreaboutwhatwouldbehelpfulWearegratefultothefollowingcitystafffortheinterviewsandindividualconversations
MichaelBaskinChiefPolicyOfficerChattanoogaTN
MelissaBridgesPerformanceandInnovationCoordinatorLittleRockAR
VictoriaCarreoacutenAdministrativeOfficerLasVegasNV
TyroneGrandstrandCommunityDevelopmentPlannerFargoND
TibLaughlinDirectorofGeneralServicesKansasCityKS
TimMorelandDirectorofPerformanceManagementandOpenDataChattanoogaTN
CharlesRosendahlManagementAnalystDentonTX
Thankyou
60
MelissaSiebenAssistantCountyAdministratorKansasCityKS
GinaThompsonDirectorofStrategicServicesLewisvilleTX
JHUFacultycollaboration
GovExalsoconvenedthecitycontactsandfacultypartnersinaseriesofAdvisoryGroupmeetingsThesemeetingsservedtoconnectcitiesandfacultytounderstandoneanothersperspective
StefanBaralBloombergSchoolofPublicHealth
AshleyBernerJHUSchoolofEducation
MelissaDavey-RothwellBloombergSchoolofPublicHealth
JonLinksBloombergSchoolofPublicHealth
TamarMendelsonBloombergSchoolofPublicHealth
KristinMmariBloombergSchoolofPublicHealth
KeshiaPollackPorterBloombergSchoolofPublicHealth
JoshSharfsteinBloombergSchoolofPublicHealth
LizStuartBloombergSchoolofPublicHealth
LindsayThompsonCareyBusinessSchool
DanielWebsterBloombergSchoolofPublicHealth
Partnersindataandevidence-basedpolicy
Wealsospokewithpartnerswhoareexpertsinthefieldofevidence-basedpolicywhosharedtheirknowledgeandresourcesofcurrentandfuturethinkinginevidence-informedthinking
WendyBarrenoAssociateDirectorWilsonSheehanLabforEconomicOpportunitiesUniversityofNotreDame
TiffanyDavenportDirectorofDataPartnershipsBaltimoresPromise
TomiHiersDirectorBaltimoreCivicSiteAnnieECaseyFoundation
ChrisKingsleySeniorAssociateAnnieECaseyFoundation
KathrynOliverAssociateProfessorLondonSchoolofHygieneandTropicalMedicine
Thankyou
61
KristenWatkinsSeniorPolicyAssociateJ-PAL
GovExParticipants
GovExAnalystMiriamMcKinneyanalyzedtheinterviewtranscriptsandconductedtextanalysisSeniorImplementationAdvisorEmilyShawwillintegrateRoadmapactivitieswithadditionalservicedeliveryactivitiesatGovEx
Thankyou
62
ReferencesAbrahamKGampHaskinsR(2017)Thepromiseofevidence-basedpolicymakingReportofthecommissiononevidence-basedpolicymakingWashingtonDCCommissiononEvidence-basedPolicymakingBigmove2Extendingthegreen(nd)Retrievedfromhttpswwwcityoflewisvillecomdoing-businesslewisville-2025big-move-2-extending-the-greenBatesMAampGlennersterR(2017)ThegeneralizabilitypuzzleRetrievedfromhttpsssirorgarticlesentrythe_generalizability_puzzleBuildwithnotfor(2018January)RetrievedfromhttpsbuildwithnotforwordpresscomCairneyPampOliverK(2017)Evidence-basedpolicymakingisnotlikeevidence-basedmedicinesohowfarshouldyougotobridgethedividebetweenevidenceandpolicyHealthResearchPolicyandSystemsdoi101186s12961-017-0192-xCairneyPOliverKampWellsteadA(2016)TobridgethedividebetweenevidenceandpolicyReduceambiguityasmuchasuncertaintyPublicAdministrationReview76(3)399-402Retrievedfromhttpdxdoiorg101111puar12555Centreforpublicimpact(nd)RetrievedfromhttpswwwcentreforpublicimpactorgCernaL(2013)ThenatureofpolicychangeandimplementationAreviewofdifferenttheoreticalapproachesOrganisationforEconomicCo-OperationandDevelopmentRetrievedfromhttpswwwoecdorgeduceriTheNatureofPolicyChangeandImplementationpdfColoradoSprings(nd)Retrievedfromhttpscoloradospringsgovcommunicationspagewhat-expect-when-you-speakupContinuumofcare(coc)program(nd)RetrievedfromhttpswwwhudexchangeinfoprogramscocCouncilpresidentkicksoff$12millionyouthfund(nd)Retrievedfromhttpwwwbaltimorecitycouncilcomcontentcouncil-president-kicks-12-million-youth-fundDurlakJAampDuPreEP(2008)ImplementationmattersAreviewofresearchontheinfluenceofimplementationonprogramoutcomesandthefactorsaffectingimplementationAmericanJournalofCommunityPsychology41327-350Retrievedfromhttpswwwncbinlmnihgovpubmed18322790Fishbonediagram(ishikawa)(nd)Retrievedfromhttpasqorglearn-about-qualitycause-analysis-toolsoverviewfishbonehtmlFivewhysGettingtotherootofaproblemquickly(nd)RetrievedfromhttpswwwmindtoolscompagesarticlenewTMC_5Whtm
References
63
FligsteinNampMcAdamD(2011)TowardageneraltheoryofstrategicactionfieldsSociologicalTheory29(1)1-26RetrievedfromhttpsociologyberkeleyedusitesdefaultfilesfacultyfligsteinFligstein20McAdam20Soc20Theory20PaperpdHeinkelP(2018January12)CitymullspermitsforbikesharingcompaniesRetrievedfromhttpwwwdentonrccomnewsdenton20180112city-mulls-permits-bike-sharing-companiesHowisareamedianincomecalculated(2012October)Retrievedfromhttpswwwhudexchangeinfofaqs843how-is-area-median-income-calculatedLampkinLMampHatryHP(2003)KeystepsinoutcomemanagementUrbanInstituteRetrievedfromhttpswwwurbanorgresearchpublicationkey-steps-outcome-managementMarshDampSmithM(2000)UnderstandingpolicynetworksTowardsadialecticalapproachPoliticalStudies484-21Retrievedfromhttpjournalssagepubcomdoiabs1011111467-924800247McCannL(nd)RetrievedfromhttplaurenellencomMoultonSampSandfortJR(2016)ThestrategicactionfieldframeworkforpolicyimplementationresearchPolicyStudiesJournal45(1)Retrievedfromhttponlinelibrarywileycomstore101111psj12147assetpsj12147pdfv=1ampt=jdkhhcp4amps=e184c625e1855a6b1615ccb99ee27b1c0b3212d8PaynterB(2018January1)What100citiesarelearningfromeachotherbysharingtheirdataRetrievedfromhttpswwwfastcompanycom40518004what-100-cities-are-learning-from-each-other-by-sharing-their-dataPeakacademy(nd)Retrievedfromhttpswwwdenvergovorgcontentdenvergovenmayors-officeprograms-initiativespeak-performancepeak-academyhtmlPinnockHBarwickMCarpenterCREldridgeSGrandesGGriffithsCJRycroft-MaloneJMeissnerPMurrayEPatelASheikhAampTaylorSJC(2017)Standardsforreportingimplementationstudies(stari)ExplanationandelaborationdocumentBMJOpenRetrievedfromhttpbmjopenbmjcomcontentbmjopen74e013318fullpdfProctorESilmereHRaghavanRHovmandPAaronsGBungerAGriffeyRampHensleyM(2011)OutcomesforimplementationresearchConceptualdistinctionsmeasurementchallengesandresearchagendaAdministrationandPolicyinMentalHealth38(2)65-76doi101007s10488-010-0319-7TandonSD(2018)ProgressincommunityhealthpartnershipsResearcheducationandactionRetrievedfromhttpswwwpressjhuedujournalsprogress-community-health-partnerships-research-education-and-actionProvanKGampKenisP(2007)ModesofnetworkgovernanceStructuremanagementandeffectivenessJournalofPublicAdministrationResearchandTheory
References
64
18(2)229-252Retrievedfromhttpsacademicoupcomjpartarticle182229935895SabatierPA(1988)Anadvocacycoalitionframeworkofpolicychangeandtheroleofpolicy-orientedlearningthereinPolicySciences21(3)129-168Retrievedfromhttpwwwjstororgstable4532139SandstromAampCarlssonL(2008)TheperformanceofpolicynetworksTherelationbetweennetworkstructureandnetworkperformancePolicyStudiesJournal36497-524Retrievedfromhttponlinelibrarywileycomdoi101111j1541-0072200800281xabstractSchwartzMampWilsonE(nd)WhocanaffordtoliveinahomeAlookatdatafromthe2006americancommunitysurveyRetrievedfromhttpswwwcensusgovhousingcensuspublicationswho-can-affordpdfShearerJCAbelsonAKouyateBLavisJNampWaltG(2016)WhydopolicieschangeInstitutionsinterestsideasandnetworksinthreecasesofpolicyreformHealthPolicyandPlanning31(9)1200-1211Retrievedfromhttpsdoiorg101093heapolczw052ThelessonsofadministrativedataHigh-profilepolicy-relevantresearchpoweredbyadministrativedata(nd)Retrievedfromhttpswwwpovertyactionlaborgsitesdefaultfilesresources20161114-The-Lessons-of-Administrative-DatapdfToolsandotherneedsforqualitativeandquantitativemethods(2015February)AgencyforHealthcareResearchandQualityRetrievedfromhttpwwwahrqgovprofessionalsquality-patient-safetytalkingqualityassesstoolshtml
WeaverRCampBagchi-SenS(2013)SpatialanalysisofurbandeclineThegeographyofblightAppliedGeography4061-70
References
65
UniversityPartners
Name Title Affiliation Area(s)ofexpertise
Contactinformation
Examplesofworkingwithnon-academicdecisionmakers
WendyPBarreno
AssociateDirector
WilsonSheehanLabforEconomicOpportunitiesUniversityofNotreDame
LEOsfocusareasincludehealtheducationhousingchildwelfareeconomicopportunitycriminaljustice
wbarrenonduduorleondedu
MelissaDavey-RothwellPhD
AssociateScientist
JohnsHopkinsBloombergSchoolofPublicHealth
Health mdavey1jhuedu410-614-5854
BaltimoreCityHealthDepartmentLeadersMarylandDepartmentofHealthAddictionRecoveryServicesinDCExecutiveDirectorsofcommunityorganizationsacrosstheUSthatfocusonHIVSTIsubstanceabuseandimplementationscience
ElizabethStuart
AssociateDeanforEducationandProfessorDepartmentsofMentalHealthBiostatisticsandHealthPolicyand
JohnsHopkinsBloombergSchoolofPublicHealth
Educationsocialpolicystatisticalmethods
estuartjhuedu
Resources
66
Management
Design-ThinkingResources
EngagementGuideforCivicTech
Journeymap
PerformanceManagementResources
DenverPeakAcademy
Proxymeasuresguide
Talkingaboutperformancewithoutcomplicatingit(video)
SettingPerformanceTargetsguide
Datavisualizationguide
Performancemanagementgettingstartedguide
Exercises
Fishboneexercise
5Whys
AdministrativeData
TheLessonsofAdministrativeData(JPAL)
EvaluationResources
Quantitativeandqualitativemethods(USDepartmentofHealthandHumanServicesAgencyforHealthcareResearchandQuality)
KeyStepsinOutcomeManagement(UrbanInstitute)
RCT-YES
Sageresearchmethods
BestPracticesforMixedMethodsResearchintheHealthSciences
ACCORDChecklist
Pilotandimplementationplanning
Benchmarkingexamplesandproxymeasures
Seattleopendataplaybook
Resources
67
Datasciencescopingguide
Resources
68
AboutGovExGovExhelpsgovernmentsusedataandevidencetoimproveresidentsrsquolivesWesupportgovernmentsinbuildingtheirinternalcapacitytooperatesustainableanalyticspracticeswhichimprovecommunityoutcomesinfieldsfrompublicsafetytoeconomicdevelopmentGovExhasaproventrackrecordofworkinginconcertwithpartnerstoprovidetechnicalassistanceandtrainingatscale
SinceourlaunchinApril2015GovExhas
Trainedover2000governmentemployeesinpersonandonlineintopicsincludingdatamanagementperformancemanagementopendatacommunityengagementandanalytics
Providedtechnicalassistancetomorethan100localgovernmentsacrosstheglobeimprovingtheiruseofdatatoachieveefficienciesinservicedeliveryandbetteroutcomesforresidents
Assessedthecapacityofmorethan130governmentalorganizationstodeterminehowleadersstaffandexternalstakeholdersmanageandusedata
Localgovernmentsareincreasinglyonthefrontlinesofaddressinglargescaleproblemswithglobalimplicationssuchasclimatechangepublicsafetyandeconomicinequality
Whileevidence-basedinterventionstohelpsolvethoseproblemsareavailablelocalgovernmentstaffhavelimitedtimeandcapacitytofindandimplementresearch-basedsolutionsAsaresultcitiescontinuetoworkthewaytheyalwayshave--makingincrementalimprovementstoprogramsbutnotalwayscreatingchangeatasystem-level
ThroughresourcedevelopmentandcollaborationwithJHUfacultyGovExissupportingcityleadersinusingdataandevidencetomakedecisionsandcreatinglastingrelationshipswithresearchers
TheRoadmapwasdevelopedbyGovExaspartofitspartnershipwithBloombergPhilanthropiesWhatWorksCitiesinitiative
AboutGovEx
69
- Title page
- Introduction and purpose of the Roadmap
- Using the Roadmap
-
- Gather information about the problem
- Strategically engage key partners
- Review the evidence and collect details
- Decide on a policy to implement based on local factors
- Establish a framework to monitor progress
- Assess data accessibility quality and security
- Design implementation and evaluation
- Share your story and success
-
- Working with a range of evidence
- Thank you
- References
- Resources
- About GovEx
-
InadditiontoassessingreadinesstoconductanevaluationresearcherscanhelpyouhoneinonthepurposeoftheevaluationTheWorldBankdescribesseveralpotentialusesofevaluationwhicharedefinedhereInbriefevaluationcanhelpdecisionmakers
makeresourceallocationdecisionsrethinkrootcausesofproblemsidentifynewproblemsdecideonabestalternativedeterminewhetherprogramsorpoliciesarebeingimplementedwithfidelitydeterminewhetherprogramsorpoliciesarehavingthedesiredimpact
Researcherscanalsoprovidesupportinsharingtheresultswiththescientificcommunitythroughpapersandconferenceswhichisvaluableincreatingapeernetworkofevidence-basedcitypractitioners
TrainingandtechnicalassistanceUniversitypartnersespeciallythoseatpracticecenterscanbehelpfulindevelopingorupdatingtrainingcurriculumtrainingthetraineranddeliveringtrainingtocitystaff
Strategicallyengagekeypartners
20
ReviewtheevidenceandcollectdetailsOnceyouhaveathoroughunderstandingoftheproblemdataandactorsyouwillneedtoconductyourownresearchonavailableevidenceandlearnaboutthelandscapeofpolicyoptionsYourgoalwillbetodevelopanaccessiblelistofinformationaboutpoliciesthathavehelpedsolvesimilarproblemsinotherplacesThischapterwillguideyouthroughthestepsofpreparingyourreviewersunderstandinglevelsofevidenceandcollectingthedetailsonevidencethatyouwillneedfordiscussionwithdecisionmakersinyourcity
Selectandprepareyourreviewers
AssignapersonorteamtoreviewrelevantinterventionsthathavedemonstratedsuccessinotherjurisdictionsTheirgoalwillbetoproduceareviewofalloftherelevantprogramspoliciesorinitiativesthatyourteamcouldconsiderimplementingTheywillbeconductingaliteraturereviewsupplementedbyinterviewswithexpertsandpeerswhohaveinformationaboutpotentialprograms
SincethepeopleyouassigntothistaskareprovidingthebaselineofpotentialoptionstoimplementitisimportantthatyouchoosepeopleyouareconfidentwillbethoughtfulandthoroughintheirworkItisalsocriticalthattheyareawareoftheimportanceofsourcecredibilityEnsureyourreviewershavefamiliarizedthemselveswithrulesforcriticallyanalyzinginformationsourcesbeforetheygetstartedIftheyareincludingnewsarticlesaspartoftheirsearchtheyshouldalsobeawareofmethodsforevaluatingnewscredibility
AskyourreviewerstodevelopaplanandtimelinetocompletethereviewThisstartswithsolidifyingagoverningresearchquestionOnewaytoknowyouarecreatingadetailedresearchquestionistousetheacronymPICOremindingyoutoincludereferencetothepopulationintervention(ifyouhaveoneinmind)comparisonandoutcomeelementsinyourresearchquestionForexampleifyouknowthatyouhaveapopulationofteenagerswhoarehomelessyouhavelearnedfromyourstakeholdersthatmanyoftheseteenagersidentifyasLGBTandyouhaveheardsomethingabouttheapproachofcreatingdrop-incentersfordeliveringsupportyoucouldask
ForLGBTyouthfacinghomelessness(population)howeffectivearedrop-incenters(intervention)relativetostandardsocialservices(comparison)atsettlingtheminpermanenthousing(outcome)
Ifyoudonothaveaninterventioninmindyoucanleaveouttheinterventionandcomparisonandsimplyask
Reviewtheevidenceandcollectdetails
21
HowcanwehelpLGBTyouthfacinghomelessness(population)obtainpermanenthousing(outcome)
OnceyourreviewershavetheirresearchquestionsusethequestionstocreatealistofkeywordstheycanuseintheirliteraturesearchThiswillhelpkeeptheircollectionofarticlesreasonablyfocusedHavethemdescribewhattheirsearchprocesswillbeincludinglistingthedatabasestheywillreviewandtheirplanforstoringtheinformationtheywillcollect(Herearesomeadditionalsuggestionsforstructuringareviewplan)
WhatyourreviewerswillbedoingisalightweightversionofarapidreviewTotheextentthattheywouldliketodevelopamethodologyforspecificsearchprotocolsincludingcriticalappraisalprocessestheycancertainlygoaheadanddoso
Usedifferentlevelsofevidenceforthemostcompletepicture
AsdiscussedintheWorkingwitharangeofevidencechaptertheNestaStandardsofEvidenceframework--reproducedbelow--ishelpfultounderstandthelevelofconfidencethataprogramorinterventionishavingapositiveimpactThisapproachletsyouputinterventionsthatyouhaveheardaboutlocallybutwhichhavenotbeenstudiedtogetherwiththosewithademonstratedcausaleffectThisishelpfulsinceanumberofpoliciesprogramsandpracticesthatyouwillwanttoconsidermaynotyethavemuchofaresearchrecord
Reviewtheevidenceandcollectdetails
22
NearlyanypieceofinformationaboutaninterventioncanbeclassifiedwithinthestandardsofevidenceForexampleifsomeonehasacrediblehypothesisabouthowaninterventioncouldproducebenefitthistypeofevidencewouldbeclassifiedasLevel1ThistypeofevidencecouldincludeinterventionsinothercitiesthatyoureadaboutinthenewsorhearaboutataconferenceForinstancewhenUSDAallowedschoolsandschooldistrictstoprovidefreebreakfastandlunchtoallstudentsthroughtheCommunityEligibilityProvision(CEP)manydistrictsimplementedthisoptionevenbeforetherewasevidenceofeffectivenessThiswouldfitintoLevel1sinceitbuildsonthelogicthatprovidingaccesstomealswilldecreasefoodinsecurity
YoumayalsofindpiecesofinformationthatqualifyasLevel2evidenceForinstanceasmoreschooldistrictsimplementedCEPprincipalstestifiedaboutpositivechangestheyobservedinstudentssuchasamorepositivedemeanorthatcanleadtobetteracademicperformanceDataalsoshowedthatthestudentsdescribedwerereceivingmoremealsthanbeforetheprogramsimplementationHoweveritwastoosoontobuildevidencetoshowacausalrelationshipbetweentheincreaseinfreemealsandincreasedacademicperformance
ldquoLower-levelrdquoevidencecanconsistofnuancedbutvaluableelementsthatrelatetoaninterventionrsquospotentialfitwithacommunityQualitativeinformationsuchastestimonyfrompeoplewhohaveexperiencedaninterventionandthosewhohavefacilitatedorimplementeditwillalwaysbeusefultoconsiderandprovidesdetailedlocalcontext
Experimentalstudies--describedstartingatLevel3withcontrolledexperimentsandgoingthroughLevel5withfullyfleshedoutprotocolsandprocessesforreplication--stilloccupythehigherlevelsofthisdescriptionofevidenceThisprioritizationshowsthevalueofrigorousstudyforevaluatingtheimpactofaparticularpolicyorprogramratherthanotherfactorsthatcouldbeinfluencingoutcomesofinterestRememberthatmultipletypesofevidencecancomplementrigorousimpactevaluationsandgivecityleadersadditionalinsightsandbroaderperspective
Accessinformation
Encourageyourreviewerstobegintheirsearchforpromisingpracticesingovernmentandnon-profitclearinghousesandinacademicpolicyjournalsWorkpublishedinthesesourceshasalreadybeensubjectedtoahighlevelofprofessionalscrutinyandgenerallycontainsagreatdealofdetailwhichmeansthatyourreviewersaremorelikelytobeabletopullouttheinformationyouneedtomakeagooddecision
Clearinghouses
Reviewtheevidenceandcollectdetails
23
Herearesomeoftheclearinghouses(collectionsofprogramevaluations)forevidence-basedpoliciesandprograms
ResultsFirstClearinghouseDatabaseThePewCharitableTrustsCollectstogethereightotherclearinghousestoprovideinformationoninterventionsinthefollowingcategoriesadultcriminaljusticeeducationchildwelfarehealthjuvenilejusticementalhealthsubstanceabuseandyouthdevelopmentIndividualclearinghousesincludedinResultsFirstare
BlueprintsforHealthyYouthDevelopment-foryouthdevelopmentandviolenceprevention
CaliforniaEvidence-BasedClearinghouse-forchildwelfareinterventions
TheLauraandJohnArnoldFoundationrsquosSocialProgramsThatWork-forsocialservicesprograms
TheUSDepartmentofEducationrsquosWhatWorksClearinghouse-foreducationinterventions
TheUSDepartmentofHealthandHumanServicesrsquoResearch-TestedInterventionPrograms(RTIPs)-forcancercontrolinterventions
TheUSDepartmentofHealthandHumanServicesrsquoTeenPregnancyPreventionEvidenceReview(TPPEvidenceReview)-forsexualriskbehaviorandsexualhealthoutcomes
TheUSDepartmentofJusticersquosCrimeSolutionsgov-forcriminaljusticejuvenilejusticeandcrimevictimservices
TheUSSubstanceAbuseandMentalHealthServicesAdministrationrsquosNationalRegistryofEvidence-basedProgramsandPractices(NREPP)-substanceuseandmentalhealthinterventions
TheUniversityofWisconsinPopulationHealthInstituteandRobertWoodJohnsonFoundationrsquosCountyHealthRankingsandRoadmapsWhatWorksforHealth(WhatWorksforHealth)-healthbehaviorsclinicalcaresocialandeconomicfactorsandthephysicalenvironment
UKWhatWorksCentresTheBritishgovernmenthascharteredanetworkofsubject-specificorganizationswhichcollectsynthesizeandreportoutevidenceontheeffectivenessofpolicyprogramsandpractices
NationalInstituteforHealthandCareExcellence(NICE)-healthandsocialcare
Reviewtheevidenceandcollectdetails
24
SuttonTrustEducationalEndowmentFoundation-educationalachievement
CollegeofPolicingWhatWorksCentreforCrimeReduction-crimereduction
EarlyInterventionFoundation-earlyintervention
WhatWorksCentreforLocalEconomicGrowth-localeconomicgrowth
CentreforAgeingBetter-qualityoflifeforolderpeople
WhatWorksCentreforWellbeing-wellbeing
WilsonSheehanLabforEconomicOpportunities(LEO)UniversityofNotreDameProvideslinkstotheirevaluationsofinterventionsincriminaljusticeself-sufficiencyeducationhealthandhousingandhomelessness
ClearinghouseforLaborEvaluationandResearchUSDepartmentofLaborProvidesresearchonworkforceandlabor-relatedpoliciesandprograms
J-PALEvaluationsAbdulLatifJameelPovertyActionLabStudiesonpoverty-reducinginterventionsfromaroundtheworld
CampbellSystematicReviewsCampbellCollaborationSystematicreviewofpolicyinterventionsbyscholarlycoordinatinggroupsinsevenissueareascrimeandjusticeeducationinternationaldevelopmentmethodssocialwelfareknowledgetranslationandimplementationanddisability
Benefit-CostResultsWashingtonStateInstituteforPublicPolicyAWashingtonState-charteredprogramtoestablishthelikelihoodofrelativecostsandbenefitsforprogramsacrossthefollowingareasjuvenilejusticeadultcriminaljusticechildwelfarepre-Kto12educationchildrensmentalhealthhealthcaresubstanceusedisordersadultmentalhealthpublichealthandPreventionworkforcedevelopmentandhighereducation
EffectiveInterventionsCentersforDiseaseControlandPreventionRepositoryforevidence-informedHIVpreventioncareinterventions(withstepsonhowtoimplement)
Journalsandindexes
OnceyouhavereviewedrelevantinformationaccessiblethroughclearinghousesitisworthtakingalookatindividualstudiespublishedasscholarlyarticlesYourreviewerswilldothisbysearchingarticleindexesandacademicjournalsUnfortunatelyanumberofusefulacademicjournalsandindexingplatformsmaynotbegenerallyaccessibletoyourreviewersifyoudonothaveasubscriptiontothemToaddressthisconsiderpartneringwithalocal
Reviewtheevidenceandcollectdetails
25
collegeoruniversitytoprovideyouwithalistofarticlesfromthefollowingsources(Thisisalsoanexcellentopportunitytoseekresearchsupportfromanyacademicstakeholdersyouidentifiedpreviously)
Indexes
GoogleScholarincludesmostpeer-reviewedonlineacademicjournalsandbooksconferencepapersthesesanddissertationspreprintsabstractstechnicalreportsandotherscholarlyliteratureincludingcourtopinionsandpatentsItallowsyoutospecifyapublicationrangeinyoursearchandprovideslinkstolikelysourcesforalllistedarticles
PolicyFileIndexisauniqueresourceforUSpublicpolicyresearchinthatitgrantsusersaccesstotimelyupdatedinformationfromover350publicpolicythinktanksnongovernmentalorganizationsresearchinstitutesuniversitycentersadvocacygroupsandotherentities
PolicyArchiveworkstocollectanddisseminatesummariesandfulltextsvideosreportsbriefsandmultimediamaterialofthinktankuniversitygovernmentandfoundation-fundedpolicyresearch
Anothersourceforhigh-qualityresearchisthereportsoftheCongressionalResearchServiceYoucanfindtheirreportsaggregatedhere
Specificjournals
Oftenthemostusefulacademicjournalsforunearthingpolicyorprogramreviewswillbeineachspecificsubjectarea--thatisineducationjournalsforeducationpoliciesYoucanuseageneralindexlikeGoogleScholartobegintolearnwhichjournalsaremostoftenusefulforyoursearchHoweveryoucanalsosearchthefollowingtwogeneralistpolicypublicationsforissuesandarticlesrelatedtoyourtopic
PublicAdministrationReviewJournalofPolicyAnalysisandManagement
Newssourcessubjectmatterexpertsandinterviews
AlthoughitisvaluabletoknowwhathasbeenexploredthroughascholarlyresearchframeworkitisalsousefultolearnwhatkindofinterventionsarebeingtriedinthefieldrightnowandhowpeoplefeelaboutthesetrialsSearchsubscriptionnewsaggregatorslikeLexisNexisorfreenewsaggregatorslikeGoogleNewsorYahooNewsforyourissueareaplusthewordpolicyorprogramtoseewhatkindsofinterventionsarebeingcoveredinthenews
Reviewtheevidenceandcollectdetails
26
DonotforgettocontacttheadvisorygroupmemberswhowerechosenfortheirsubjectmatterexpertiseInadditiontoprovidinggoodmaterialthroughinterviewsthemselvestheymayhaveideasaboutothersubjectmatterexpertsyoushouldcontacttoaskaboutpotentialinterventions
Peercitiesandnetworks
InadditiontoreadingabouttheworkofothercitiesitisidealifyouareabletospeakwithpeoplewhoareinvolvedwithcurrentimplementationsofinterestinginterventionsInyourcityyoulikelyhaverelationshipswithyourneighborsbutitcanbehelpfultolookoutsideofyourregionforsolutionsaswellTofindcounterpartsinothercitieswhoareworkingonaprogramyouwouldliketolearnmoreaboutyoucancontactstaffinintermediaryorganizations(likeGovExoryourprofessionalassociations)whomightbeinapositiontoputyouintouchimmediatelyorwhomightseeanothergoodrouteforsettingupaconversationHereisalistofnetworksthatmaybehelpful
InternationalCityCountyManagementAssociation(ICMA)NationalAssociationofCounties(NACO)NationalLeagueofCitiesUSConferenceofMayorsWhatWorksCities
Collectthedetails
Onceyourreviewershavefoundevidenceonpossiblepoliciesorprogramstoimplementtheyshouldcapturekeydetailsthatwillhelpdecisionmakersunderstandthepotentialeffectivenessandapplicabilityofeachpotentialintervention
Foreachsourcethathasprovidedinformationyouwanttoconveyusethetemplatebelowtocaptureasmanyofthefollowingdetailsaspossible(Youdonotneedtocompletethisforsourcesthatprovideduplicativeinformationorareotherwisenotthatuseful)
Programorpolicyname (egldquoHousingFirstrdquoorldquoblightedpropertiesregistryrdquo)
Sourcetype (egjournalarticleinterviewsubjectmatterexpertstudyassociationcommunityparticipationprocess)
Sourcedetails (egarticletitleandpublicationnameandtitleofinterviewee)
Evaluationmethodology
(egrandomizedevaluationcomparativeevaluationoftwodifferentgroupsbeforeafteranalysis)
Evidencelevel (egLevel12345)
Programorpolicy
Reviewtheevidenceandcollectdetails
27
description
Programorpolicygoal (1sentencecapturingtheprincipalaimoftheintervention)
Existingimplementations
(WherehastheinterventionbeentriedListasmanysitesasyouareawareof)
Characteristicsofimplementationsites
(ForeachimplementationsiteofferbasicgeographicIDmdashcityandstateThenyouwillwanttocollectthedetailswhicharerelevanttoyourdecisionmakersrsquoconsiderationsaboutwhatmakesalocationldquosimilartordquoorldquodifferentfromrdquoyoursituationYouwillatleastwanttoincludeforeachimplementationsitePopulationnumberPopulationdensity(ruralurbansuburban)Socio-economiccharacteristicsBeforestartingaskyourdecisionmakersaboutanyparticularcharacteristicsofprogramimplementationsitestheywouldwanttoknowbeforeseekingtoreplicate)
Characteristicsofimplementationpartners
(Anyavailabledetailsaboutorganizationsinstitutionsorpeoplewhowereinvolvedwithimplementingtheintervention-egindependentchildcarecenterpublicelementaryschoolcitypublichealthnurses)
Characteristicsoftargetpopulationeligibleprogramparticipants
(Anyavailabledetailsaboutpeoplewhoarethebeneficiariesoftheprogramorotherwiseincludedoraffectedbytheintervention-egnumberofprogramparticipantscategoriesofpropertyownersaffected)
Policyorprogramduration
(HowlonghastheinterventionbeeninplaceineachimplementationsiteHasitendedandifsowhy)
Whatbenefitscamefromeachimplementation
(Anyobservedpositiveeffectsoftheinterventionincludingeffectsize)
Statisticalestimatesofinterventioneffectiveness
(Anyavailablestatisticalestimatesoftheinterventionrsquoseffects)
Whatdrawbackswereobserved
(Anyavailabledetailsaboutunintendedorunavoidablenegativeeffectsoftheintervention)
Programorpolicycosts
(Anyavailabledetailsaboutprogramcostsperimplementation)
Reviewtheevidenceandcollectdetails
28
Reviewtheevidenceandcollectdetails
29
DecideonapolicytoimplementbasedonlocalfactorsTogetthecriticalbuy-infromleadershiptoimplementanewpolicyorprogramtheprojectleadshouldscheduleameetingwithdecisionmakerstopresenttheevidencesurroundingthepolicyoptionscapturedinthetemplateDependingonyourinsightintoyourcityleadershipswillingnesstomakechangeyoumaywishtosharetheresearchyouhaveconductedontherootcausesoftheproblemandsomeofyouranalysisontheimpactofcurrenteffortstodate
Considertheapplicabilityofanapproach
AsdiscussedinthepreviouschapterpolicymakersoftenlookforevidenceofeffectiveprogramsinothercitiesbutoftenthequestionarisesofhowgeneralizableaprogramisfromonecitytoanotherTheAbdulLatifJameelPovertyActionLab(J-PAL)suggeststhatinreviewingevidencefromanotherlocationitishelpfultoidentifythespecificmechanismthatwasused(deliveryofprogramorserviceinformationsharingetc)identifythelocalconditionsthatwereinplaceobservethebehaviorchangethatresultedfromthemechanismandfinallytounderstandlocalcontextandconditions(legalorcostconstraintsculturalnormsetc)Thissetofpresentationelementscanbedrawnfromthelistofdetailscollectedintheresearchtemplate
EducationpolicymakersinChicagoILusedfindingsfromprogramsinKenyaandIndiainwhichchildrenwhohadfallenbehindtheirpeersweregiveneducationalsupporttargetedtotheirleveloflearningThemechanismusedwastargetedteachingandthereweresimilarconditionspresentinallthreecountriesAlthoughsomestudentshadfallenbehindinthecurriculumteacherswereincentivizedtoteachgrade-levelmaterialsConsiderationsofhowthelocalcontextinChicagomightaffectimplementationofatargetedteachingprogramincludedunderstandinghowtheneighborhoodsthatthestudentswerefrommayhaveaffectedtheireducationaloutcomesandthehighcostofteachinginsmallgroupsInChicagoMatchEducationprovidedtwo-on-onetutoringwhichmatchedstudentswithwell-educatedindividualswhohavenothadformalteachertrainingforalower-costmodelTheresultswerepositiveStudentswhoweretutoredperformedbetterinmathandweremorelikelytopasshighschoolclasses
ThegeneralizabilityframeworkproposedbyJ-PALisbelowUsethisasatemplateasyouconsiderpotentialpolicysolutions
Decideonapolicytoimplementbasedonlocalfactors
30
SourceTheGeneralizabilityPuzzle
OnceyouhavenarrowedthepolicyoptionsdiscussthequestionsbelowforeachoptionsyouareconsideringThisstepiscriticalasunderstandinglocalcontextissoimportantinsuccessfulpolicyimplementation
CommunityContextQuestionnaire
Localcontext
DoesthisoptionmakesenseinourcommunityIsthisaonesizefitsallapproachIsanyparticulargroupabsentfromthisapproachIfsoisthatok
Youranswers
Effectiveness
WhatwouldweexpecttoobserveifthepilotisaseffectivehereasinCityX
Youranswers
Timing
DoesthetimingmakesenseIsthistherighttimetoimplementthispolicychange
Decideonapolicytoimplementbasedonlocalfactors
31
WherearethepotentialroadblocksthatcouldaffectthescheduleIsthereanywaytomitigatethisHowlongwouldittaketoseeanimpact
Youranswers
Implementation
DowehavethestaffcapacitytoimplementthisArethereadditionalpartnersthatyoucanbringtothetabletosupportimplementationDoesthismakesensegivenourformofgovernment
Youranswers
Authority
DoesourcityhavetheauthoritytoenactthisIfsowhatistheappropriatemechanismAretherelegalbarriersArethereanyexistingpreemptionissuesfromthestateorfederalgovernmentCanthosebeaddressedthroughlegislationadvocacyIstherepoliticalsupportoroppositiontothis
Youranswers
Cost
HowmuchwillthiscostannuallyHowwillwepayforitCanthepolicybeimplementedwithexistingfunding
IfsoisthisfundingreliableHowwouldthispolicyoptionaffectourcitysbudgetintheshortandlongtermWhatotherfundingoptionscanweconsiderIsthereanalternativethatwouldcostlessandgiveussimilarresultsWillanyconstituencybeadverselyaffectediffundingforprogramsthatimpactthemisshiftedtoanotherprogrampolicyorinitiative
Youranswers
Decideonapolicytoimplementbasedonlocalfactors
32
EstablishaframeworktomonitorprogressNowthatyouhavereviewedtheevidenceconsideredlocalcontextandchosenyourpolicyitistimetodeterminehowyourcitywillmonitorprogressofthisnewapproachCitystaffandleadersneedtoestablishtheinfrastructureforongoingcontinuousdecisionmakingbasedonwhetherthepolicychangeisleadingtothedesiredoutcomesforresidentsandwhetherthechangeismeaningfulandrationaltofrontlineworkersSettingarbitrarygoalsatthelocallevelthatconflictwithgoalsatotherlevelsisconfusingtotheindividualsresponsibleforachievingthegoalsGoalsthataresettoohightoachievecanbedemoralizingtostaff
ThissectionincludesstepstosetupavenueinwhichCitystaffandleadershiproutinelyvisitkeyindicatorstodeterminewhatisorisnotworkingandtomakedecisionsinreal-timeFormorereadingonperformancemanagementpleasevisittheGovExPerformanceManagementGettingStartedGuide
InperformancemanagementtermssuchasindicatormeasureandmetricmaybeusedinterchangeablyInthisguidewewilluseindicatorInyourworkthemostimportantthingistobeconsistentinwhichevertermyouuseThisvideomaybehelpfulinunderstandingthisconceptandcommunicatingittoothers
Establishthefundamentalsandgatheryourteam
FirstdeterminewhetherthereisanestablishedvenueorregularmeetingscheduleduringwhichyoucandiscussprogressIfyourcityhasanexistingperformancemanagementstructureonarelatedtopicaddingadiscussionoftheprogressofpolicychangetoyouragendacanbeagreatideaIfthiswillbeastandalonemeetingdecidehowfrequentlytheprojectteamanddecisionmakerswillmeetandputthosemeetingsonthecalendarfortheentirelengthoftheproject
DeterminewhowillstaffandattendroutinemeetingsonCityprogressThesemeetingsmayincludetheMayorandorCityManagerdepartmentheadstaffwhoareimplementingthepolicychangeandanalysts
Identifyindicators
ItiscriticalthatindicatorsbeanimportantcityandcommunitypriorityaswellassomethingtheCityplaysamajorroleinaddressingTheremaybeinstancesinwhichtheCitywillneedtoreachouttoorcollaboratewithexternalpartnerstogetamorecompletepictureofprogressontheoverallpolicychange
Establishaframeworktomonitorprogress
33
Anoutcomeindicatordescribesthedesiredeffectofthetargetedpopulation
ExampleBellevuesfamilyfriendlyneighborhoodsindicatormeasuresthepercentofresidentswhobelievetheyliveinaneighborhoodthatsupportsfamiliesparticularlythosewithchildren
WithyourkeypartnersrevisityourproblemstatementcurrentstateandfuturestateexerciseandbegintoidentifytheoutcomeindicatorsthatwillillustratesuccessFornowchooseasingletop-levelkeyperformanceindicator(KPI)ItiscriticalthattheKPI-thepieceyouwillmeasure-beapriorityforthecommunityandsomethingtheCityplaysamajorroleinaddressing
Outputindicatorsmeasuretheprocessofprogramorservicedelivery
ExampleSanDiegoMayorKevinFaulconersgoaltorepair1000milesofstreetby2020
ReviewingtheadministrativedatasetinventorybegintoparedowntherelevantprogramsanddatasetsthatsupportyourKPIWhatarethemeasuresofeffectivenessforeachprogramTheseareyouroutputindicators
Thediagrambelowillustratestherelationshipbetweenoutputandoutcomeindicators
ThinkaboutimplementationindicatorsasmeasuringstepsintheprogramsrolloutthatarenecessarytoobtaindesiredchangesImplementationoutcomescanfocusontheinterventionstaffdeliveringtheinterventionanorganizationorsystemandcanaddressfeasibilityadaptabilitysustainabilityandcostTwokeyquestionstoconsiderwhendevelopingimplementationindicatorsare
1 Istheinterventionacceptableandappropriatetothecommunityand
2 IsitfeasibletoimplementinterventionXatthistimewiththeseresources
Establishaframeworktomonitorprogress
34
Amoreextensivelistwithdefinitionsofpotentialimplementationoutcomesislinkedhere
UsethetablebelowtochoosethedesiredimplementationoutcomesforyourpilotandtoplanyourmeasurementtechniqueThisshouldincludelevelofanalysis(individualproviderorconsumerorganizationorsetting)yourmeasurementtechnique(surveysobservationinterviewfocusgroupsadministrativedataanalysis)andthephasewhenyouwillcollectthisinformation(earlymidlate)Theremaybemorethanonelevelofanalysismeasurementtechniqueandphaseofdatacollection
Implementationoutcome
Definitionofoutcome
Levelofanalysisandmeasurement
Yourmeasurementtechnique
Phaseofpilotwhenyouwillcollectdata
Usethespacebelowtocaptureyourindicatorwishlist
Indicatorwishlist
Outcomes_____________________________________________________________________________
Outputs________________________________________________________________________________
Implementationoutcomes______________________________________________________________
Setapublichigh-levelgoal
Thehigh-levelgoaliscriticalinestablishingdirectiontogalvanizethepeoplewhowillbedoingtheworkWhendiscussingthegoalusetheMayorsbullypulpitLookforkeymilestonesinCitygovernmenttosharethegoalstatementsuchasthebeginningofaMayoraladministrationaStateoftheCityspeechorotherpublicremarks
GoalsshouldbeldquoSMARTrdquo
Specificaspossible(innamingthepopulationwhowillbebetterofftheactionthecitywilltake)
Measurable
Achievable
Establishaframeworktomonitorprogress
35
Results-focused
Time-bound
ThegoalstatementshouldincludeyouroutcomeindicatorlikeChattanoogasgoalldquoWewillendchronicveteranhomelessnessbyDecember2016rdquoAdiagramtohepyoubuildaSMARTgoalaroundyouroutcomeindicatorisbelow
Component Description Yourturn
Specific WhowillbebetteroffWhatactionthecitywilltake
Measurable Howwillyoumeasuresuccess
Achievable Isthegoalattainable
Results-focused Whatistheexpectedresult
Time-bound When
Setrealisticambitioustargetsandtimelinesforeachindicator
ToestablisharealisticandambitioustargetforyourgoalstatementusetheadministrativedatacollectedforyourchosenKPItoreviewactualCityperformanceinthemostrecentyearcomparedwiththethreeorfourmostrecentyearsworthofdataBegintocalculatesummarystatistics(egtotalminimumaveragemedianmaximum)toidentifytrendsCreatecumulativeandmonthlychartstoidentifytrendsandanomaliesBasedonyouranalysisofhistoricaldataignoretheanomaliesandassumetheproductivitytrendswillfollowthesameoverallpatternfrommonth-to-monthFinallyapplyanincreasedproductivityfactortotheperformanceeachmonthtoforecastfutureprogressBasedonthisanalysischooseatargetthatinspiresimprovementwhileaccountingforreality
Manycitiesthataremorematureintheirperformancemanagementpracticesrecognizetheimportanceoftakingacommunity-levelapproachtoimprovingoutcomesforresidentsThesecitiesarecreatinggoaltaxonomiesthatincludemeasuresrelatedtopublicsafetyhousingandevenrecreationCitiesdriveprogressonthesegoalsbyimplementingstrategiesthatengagemultipledepartmentsandcollaboratewithserviceprovidersinthecommunity
EventuallyyouwilldevelopgoalstatementsfortheoutputsinordertocapturetheefforttoachievetheoutcomecontainedinyourgoalstatementYoursetofindicatorsshouldbecomposedofshort-andinterim-termsupportivemeasureswithaccuratetargetsandrelevanttimelinesAnexamplegoaltaxonomyfromtheCityofLasVegassResultsVegasprogramisbelowUsethistaxonomyasamodelforyourownsuiteofindicatorsandgoalstatements
Establishaframeworktomonitorprogress
36
Yourturn
Thetemplatebelowismeanttohelpyoucraftagoalstatementforeachsupportingmeasure
Goalstatement
Departments
Outcome
Activities
KPI
Target
Timeframe
Conductanddiscussrelevantanalysis
ThrougharoutineseriesofmeetingscontinuallymonitorprogressofKPIssupportingindicatorsandanyotherdatacollectedsuchasqualitativedatatoidentifytrendsthatwillindicatewhetherthechosenpolicyishavingthedesiredimpactThesedatapointsshoulddrivethediscussionduringthesemeetingspointingtoopportunitiesforfurtherinquiryinto
Establishaframeworktomonitorprogress
37
whatisdrivingtheobservedtrendsAnalystsshouldbeencouragedandtrainedtoaskquestionsofthedepartmentsproducingthedataandconductsitevisitswhenpossibletodeepentheirunderstandingofthedataPresentationoftheanalysisshouldfocusononeideaatatimetokeeptheconversationfocusedonkeypointsandspecificopportunitiesforimprovementYoumayconsiderdevelopingapublicfacingdashboardtosharethisprogressmorebroadly
AgeneralcomparisonofthestatusquodatawithwhatwasobservedduringthepilotcanprovidedecisionmakerswithaquickunderstandingofthedifferencesbetweentheoldandnewapproachesalthoughthiswillnotshowcausalityOftenthesecomparisonswillincludeoperationalmetricssuchasdaysordollarssavedbutrememberthattheimpactonpeoplewillbemostimportantIfyouareworkingwithanevaluationpartnertheywillsupportthisstep
Usethespacebelowtosummarizetheresultsofyouranalysis
MeasureTargetvalue(seesectiononsettingtargetsabove)
Pilotvalue(whatdidyouobserveduringthepilot)
Historicalaverage(ifavailable)
Results(WhoisbetteroffIsanyonenegativelyimpacted1-2sentencesreassessingyourgoalstatement)
DocumentyourprocessandprogressincludingfeedbackfromtheparticipantsimplementationpartnersandtheprojectteamDidanythingunexpectedoccurduringthepilotArethereinternalandexternalfactorsthatmayaffectyouroutcomesThisiswheretheimplementationoutcomescanbecanhelpfulIftheinterventiondidnotgoasplannedconsiderwhetheritisbecausetheinterventionwasnotacceptableorfeasible
UsethetemplatebelowcreatedbyDenversPeakAcademytocapturethisinformationIfthe306090daytimelinedoesnotmakesenseforyourcityfeelfreetocustomizeit
CurrentState
FutureState
30Day
60Day
90day
Whatwentwell(+)
Whatdidnotgowellshouldbechanged(-)
IfaquestionordiscussionpointarisesthatcannotbeansweredinthemeetinganassignedstaffmembershouldcapturethequestionduringthemeetingAfterwardthefollowupitemsshouldbesenttorelevantpartnersandthenextmeetingshouldbeginwithadiscussionof
Establishaframeworktomonitorprogress
38
thoseitems
Takeaction
DiscussionsaboutwhetheryourindicatorsaremovingintherightdirectionandwhatmightbedrivingtrendsmayleadtoadiscoverythattherearenotenoughresourcesallocatedtothispolicyorissueTheremaybelimitedfundingforstaffprogrammingortechnologythatisdrivingthedatainthewrongdirectionIfCityleadershipisintheroomthisisagreattimetomakethecaseforadditionalfundingduringthebudgetseasonBycollectinganalyzinganddiscussingdataonthispolicyroutinelyyouarebuildingevidencetomakethecasefortherightleveloffundingtoreachtheestablishedtargets
Establishaframeworktomonitorprogress
39
Assessdataaccessibilityqualityandsecurity
Determineaccessibilityandqualityofdata
OnceyouhavecompiledyourwishlistofindicatorsitistimetodeterminetheaccessibilityandqualityofthedataTopreventcomplicationyoumaywishtoprioritizerelevantindicatorsforwhichyouarealreadycollectingdataAvoidthetemptationofadoptingatechnologysolutiontocollectnewdataatthispointsuchasaphotosurveytoolorotherappFornowyouaretryingtodeterminewhichdatasetsarerelevantaccessibleandhighqualityBesuretoworkwithyourkeyinternalpartnerswhileyouareidentifyingyourmeasuresDepartmentstaffwhoworkonprogramsarefamiliarwiththeindicatorsthatwillaccuratelysupporttheoutcomesanddatabaseadministratorswillbevaluablewhendiscussingaccessibilityandqualityofthedata
Abrieflistofconsiderationsfortheaccessibilityofyourdataisbelow
ArewecurrentlycollectingthisdatasetorwillwebecollectingitforthefirsttimeHowfarbackdoestherecordgo(timehorizon)WouldweneednewtechnologytocollectoranalyzethedataCanthedatabecollectedmanuallyorcanitbeautomatedHowfrequentlyisthedataproduced(continuouslyhourlydailyweeklymonthlyquarterlyyearlyetc)IsthedataavailablefortheentireCitypopulationorasubsetofthepopulationHowgranularisthedata(individuallevelCensusblockneighborhoodcitywideetc)DoesthedatacontainsensitiveinformationIsthereaproxymeasurethatwecanusefornowIsthedatamachinereadable
AsnotedbytheWilsonSheehanLabforEconomicOpportunities(LEO)attheUniversityofNotreDameleveragingadministrativedatacansignificantlyreducethecostsandtimelinesofrigorousevaluationCitiescanestablishaninternaldatasharingagreementacrossdepartmentsincludinglanguagethatenablesadministrativedatatobeusedforresearchpurposesCitiescanalsoidentifyexternaladministrativedata-holderssuchasacountyschooldistrictpolicedepartmenthospitaloranotherstatethatmayhaveoutcomedataimportantformeasuringprogramimpactCitiescanleverageexistingrelationshipsandorworkwithresearchinstitutionstodevelopstreamlineddatasharingprocessesandagreementswiththeseexternalentities
Assessdataaccessibilityqualityandsecurity
40
Evaluatethequalityofyourdata
QualityisasimportantasaccessibilityToevaluatedataqualityyoumaywishtocreatearubricliketheonebelow
Value ValueDescription
3 ExcellentDataQualityThedataiscompleteaccurateandupdatedregularly
2 AdequateDataQualityThedataissomewhatcompleteaccurateandupdatedfairlyregularly
1 PoorDataQualityThedataisnotcompleteaccurateorupdatedregularly
ThisisagreattimetobuildonthedatainventorythatyoustartedinthegatherinformationpageSuggestedadditionalfieldstoaddtoyourdatainventorytemplatearebelow
Program DatasetTitle
DatasetDescription DataSource
Departmentorexternalpartner
DoestheCityalreadycollectthisdata
Whatisthenameoftheprogram
Human-readablenameofthedataset
Shouldbeunderstandablebynon-technicalusers
Whatistheoriginalsystemapplicationfilethathousesit
Departmentorexternalpartnerthatmaintainsthedata
Isthisdataalreadycollected
Updatefrequency-ThefileformatforthedatasetTypicallycsvxlsxshapefiletxtetc
Format-Howoftenthedatasetisupdated
Timerange-Theperiodoftimecoveredinthisparticulardataset
Datause-Howisthisdatasetcommonlyused
Datausers-Whotypicallyusesthisdataset
Dataownerstitle-Titleofthedataowner
Dataownersposition-DataOwnersjobtitle
Dataownersemailaddress-DataOwnersemailaddressWhomanagesthedataandorisresponsibleforgrantingpermissiontoaccessthedataWhounderstandswhatthedatasetincludesandcananswerquestionsaboutit
Dataqualityconcerns-Describetheconcernsyoumayhaveaboutthequalityofthedataset
Assessdataaccessibilityqualityandsecurity
41
Division-DivisionwherethedataismaintainedIfmorethanonedivisionisresponsiblelisttheprimaryownerofthedataset
Sensitivedatacomments-DescribetheissuesthatwouldpreventthecityfromsharingthedatapubliclyCommonsensitivityconcernsincludeprivacyviolationssecurityissuesorhighcostorstaffdemandstopostthedata
Publishingstatus-Thistheplacetoputanycommentsregardingthecurrentstatusofthedataset
Publishedlink-Thisisthelinktothepublisheddatasetontheopendataportalorcitywebsite
Reviewdatasecurityandprivacyprotectionprotocols
OftenapolicychangealsoinvolvesachangeinthedatathatisbeingcollectedhowthatdataisbeinghandledwhohasaccesstoitandahostofotheradjustmentsAsyoutakestockofthedataassetsthatareavailableforyourinitiativeitisvitaltoalsothinkthroughwhatwillbedonetoprotectthatdataandtheprivacyofthepeopleitrepresentsAbigpartofgooddatasecurityinvolvesstrongdatagovernancepracticesthatdefineandidentifysensitivedataandconsciouslyworktobalancethevalueandriskofstoringsuchsensitivedataOnewayofevaluatingyourdatacomesfromcaselawrelatedtotheUSFreedomofInformationActwhichwehavetranslatedtothechecklistbelow
1 Woulddisclosureresultinasubstantialinvasionofprivacy2 Haveweconsideredtheextentorvalueofthepublicinterestandthepurposeorobject
oftheindividualsseekingdisclosure3 Istheinformationavailablefromothersources4 Wastheinformationgivenwithanexpectationofconfidentiality5 Isitpossibletomouldreliefsoastolimittheinvasionofindividualprivacy
TheseriskscouldbemitigatedusingavarietyofstrategiesfromaggregatingdatatoobfuscatingindividualstoredactingfieldsthatmayincludePersonallyIdentifiableInformation(PII)orre-evaluatingyourdatacollectionmethodstofindalternativewaysofgettingtheinformationyouneedThesestepsshouldbecodifiedandmanagedbyadatagovernancecommitteeifpossibleForthesensitivedatathatisdeemedessentialcreateaplanaroundpotentialdatabreachesinordertoquicklyandconfidentlyrespondtoemergencysituations
ProactivelypreparingforthesesituationsisoftenthebeststrategyfordatasecuritybutdonotforgetthetechnicalsideInvolveyourITadministratorsinconversationsaboutdatastorageandaccessensuringthattheywillbeprotectedbyenterprise-gradecybersecuritysystems
Assessdataaccessibilityqualityandsecurity
42
Assessdataaccessibilityqualityandsecurity
43
Designimplementationandevaluation
Designtheimplementation
NowthatyouhaveyouraccountabilityplanandkeyindicatorsitistimetogetspecificaboutyourplanforimplementationThetemplatebelowwillhelpyouthinkthroughthespecificmechanismsofimplementingpolicychange
UnderstandingorganizationsandsystemsiscriticaltounderstandingandmakingdecisionsaboutwhereandhowinterventionsshouldbeappliedApolicyimplementedinasinglesettingmayshowpromisingresultsbutimplementingmultiplepoliciesandprogramsatthesametimewithdifferentactorsresponsibleforeachaspectcanhelpensuresustainablebehaviorchange
InSomervilleMAShapeUpSomerville(SUS)isacitywidepublichealthprogramthatisimplementedacrossmultiplelevelsofinfluenceandsystemsthroughinterventionsbeforeduringandafterschooltoincludechangestobuiltenvironmentsuchasbikelanesregionalmasstransportationandphysicaleducationequipmentinschoolsandgymstoincreasephysicalactivitywellnesspoliciesandschoolgardensnutritioneducationduringandafterschoolandhealthyrestaurantprogramstosupporthealthyfoodandhealthyeatinghabits
Usethetemplatebelowtoplanyourimplementation
Policyimplementationplanningtemplate
ListthespecificdesiredoutcomesInthepreviousstepyouworkedwithdepartmentstafftoestablishspecificmeasurableoutcomesandinterimindicatorsPrioritizetheindicatorsthatwilldemonstrateprogressUsethespacebelowtocapturethedesiredoutcomesofthepolicychange
ExampleShapeupSomervillebeganwiththegoalofpreventingobesityin1stthrough3rdgraders
Yourturn
____________________________________________________________________________
____________________________________________________________________________
Designimplementationandevaluation
44
SpecifythetargetpopulationAreyoudesigninganinterventionthattargetsonlycertainsitesAreyouplanningonimplementingaprogramorinitiativeforaspecifictargetaudienceClearlyspecifythepopulationyouplantotarget
IfyouplantoconductarigorousimpactevaluationasnotedbytheWilsonSheehanLabforEconomicOpportunitiesitisimportanttoestablishconcreteobjectiveeligibilitycriteriaandensurethatthesecriteriaareappliedconsistentlybyallstaffduringscreeningandintakeItisalsovitaltoidentifytheindividualswhoareeligibleforaparticularprogramorserviceevenifeveryoneeligiblecannotbeservedbyexistingresourcesThishelpscitiesidentifyanyunmetneedsandguidefutureresourceallocationwhilealsocreatingthepossibilityofarigorousimpactevaluationbyidentifyingapotentialcomparisongroupFinallywhenconductingscreeningandintakeitisimperativetocollectandmaintainbasicinformation(egnamedateofbirthgender)aboutallprogram-eligibleindividualsnotjustthosewhoparticipateintheprogram
ExampleIntheShapeUpSomervilleexampletheprogrambeganbytargeting1stthrough3rdgraders
Yourturn
____________________________________________________________________________
____________________________________________________________________________
DefinethelocationWherewillthepolicychangetakeplaceWillimplementationtakeplaceatonesiteormultiplesites
ExampleShapeUpSomervilleinterventionsweredeliveredinschools
Yourturn
____________________________________________________________________________
____________________________________________________________________________
ClarifyimplementationpartnersandarticulatetheirrolesWhatarethekeyorganizationsinstitutionsorpeopleresponsibleforimplementationTheprojectteamshouldbeasclearandspecificaspossibleinprovidingadescriptionoftheconcreteactivitiesthatimplementationpartnersaremeanttocarryoutPracticeprofilesmaybeahelpfultooltohelpdefinewhatisrequiredoftheactorswhoareresponsibleforimplementingtheintervention
Designimplementationandevaluation
45
ExampleInSomervillemayoralleadershipandengagementofmultiplecityagencies(publichealthprimarycareschoolsCambridgeHealthAllianceetc)streamlinedandconnectedthevariousShapeUpSomervilleinitiativesandgrantswhichhelpedtokeepthetalentandenergyfocusedonhealthyeatingandactivelivingforallsectorsofSomervilleSUSpartnersultimatelyincludedschoolsnonprofitsCityandstateagenciesandnationalfoundations
Yourturn
____________________________________________________________________________
____________________________________________________________________________
DescribetheinterventionWhatarethestepsactivitiesorprocessesRememberthatthemechanismthespecificstepsorprocessthatleadtobehaviorchangeiskeytounderstandingtheeffectivenessofthepolicychangeandsharingtheresultswithothers
ExampleGoalsofShapeUpSomervilleweretoinfluenceeverypartofanearlyelementaryschoolchildrsquosdayndashbeforeduringandafter-schooltopromotehealthyeatingphysicalactivityandtheirweightstatusInterventionincludedfoodserviceenhancementstoimprovebreakfastandlunchandstaffprofessionaldevelopmentwalktoschoolactivitiesSUSclassroomcurriculumandprofessionaldevelopmentschoolwellnesspolicydevelopmentSUSafter-schoolcurriculumandprofessionaldevelopmentoutreachandeducationtohomethroughmaterialsforumseventsldquoSUSapprovedrdquorestaurantscommunityoutreachandcapacitybuildingthroughpolicydevelopmenttrainingsmediaplacements
Yourturn
____________________________________________________________________________
____________________________________________________________________________
ClarifytimelineandmilestonesfordeliveryandevaluationHowfrequentlywilltheservicesinterventionbedeliveredHowlongwilltheprogramrunWhatisthetimetoexpectedimpacts(eg12monthsafterprogramcompletion)
Designimplementationandevaluation
46
ExampleShapeUpSomervilleledbyTuftsUniversityFriedmanSchoolofNutritionScienceandPolicyconductedaquasi-experimental(nonrandomized)interventionstudyamong1stthrough3rdgradestudentsinSomerville(intervention)andtwocomparisoncommunitiesfor9-monthsduringoneschoolyear(fall2002ndashspring2003)ThestudyshowedpositiveassociationoftheinterventiononchangeinBMISUSreleasedreportssummarizingevaluationsprogresstodateplansforadditionalprojectsandimpactstenandfifteenyearsintotheinitiative
Yourturn
____________________________________________________________________________
____________________________________________________________________________
PlanyouranalysisTheprevioussectionaddressedestablishinganaccountabilityframeworktomonitorprogressandimplementationDeterminewhichindicatorswillbediscussthroughthatprocesswhowillsharethedataandanalysisandhowfeedbackfromthoseconversationswillbeincorporatedbytheprojectteam
ExampleTheMayorrsquosofficeimplementedadata-drivendecisionmakingapproachcalledSomerStatwhichwassynergisticwiththeapproachestakenbytheresearchandevaluationteamsdeployedforthespecificgrants
Yourturn
____________________________________________________________________________
____________________________________________________________________________
PlanforsustainabilityIfthepolicyandprogramchangeissuccessfulhowwilltheprojectteamsustainandexpandtoadditionalsitesorpopulations
ExampleAlthoughmostofthepartnershipsforgedintheearlyyearsweremaintainednewgrantsandcollaborationsexpandedattentiontoolderpublicschoolstudentsactivetransportationandtheGreenLinesubwayextensionthebuiltenvironmentfarmerrsquosmarketeffortssuchasldquoGrowninSomervillerdquowithUnionMainStreeteconomicdevelopmentandhealthdisparitiesaswellasdatacollectionmonitoringandsurveillance
Yourturn
Designimplementationandevaluation
47
____________________________________________________________________________
____________________________________________________________________________
Pilotplanningtemplate
IfyouplantoconductapilotfirstdeterminethepurposeofthepilotDoyouwanttoobservewhetherthepolicychangeresultsinthedesiredbehaviorchangeAreyouconcernedaboutfeasibilityscalabilityorcostDoyouwanttotestwhetheryourdesiredresultscanbeachievedthroughthispolicyapproachClarifythegoalofthepilotandthequestionsyouhopetoanswerThiswillhelpyouidentifythekeyindicatorsthatyouwillusetomeasurewhetherthepilotissuccessfulBeforebeginningestablishabaselineofcurrentperformanceandsetyourtargetsaccordinglybasedonwhatyouhopetoaccomplishthroughthepilot
NextestablishthepilotplanforyourprojectteamincludingmilestonesforteamcheckinstodeterminewhethertomoveaheadorchangecourseThisisseparatefromthemeetingsthatwilloccuraspartoftheaccountabilityframeworkUsethespacebelowtocreateyourtimelineanddevelopyourmeetingschedule
Task Personresponsible Startdate Checkindate Deliverable
Planyourevaluation
WhileyouaredesigningtheimplementationitisalsotimetobegintodevelopyourplanforanalysisAsnotedbytheWilsonSheehanLabforEconomicOpportunitiesdecisionsaboutprogramsetup-includingoutreachscreeningintakedatatrackingandenrollment-candeterminewhetherornotarigorousimpactevaluationwillbepossibleAsdiscussedcityleadersmayfindithelpfultoincludeevaluationpartnersinearlyconversationsaboutprogramsetupDuringpreviousstagesyoumayhaveworkedwitharesearchertoplanyourproposedpolicychangeYoushouldbeclearaboutthepurposeoftheevaluationanddevelopanevaluationplanBelowareadditionalconsiderations
Causationvstrendanalysis
IntheEstablishaframeworktomonitorprogressstagetherewasguidanceontrackingtrendsovertimeItisimportanttonoteherethatmonitoringdataisdifferentfromanalyzingacausaleffectTrendanalysiswillillustratethatchangesareoccurringandmayleadtoquestionsthatcouldbeansweredbyyourevaluationbuttrendanalysisalonedoesnotprovideevidenceaboutwhatisdrivingthosechangesYoumayseechangesinyour
Designimplementationandevaluation
48
outcomeindicatorsbasedonvariablesoutsideoftheCityscontrolForinstanceachangeinfederaltradepolicycanhaveanimpactemploymentratesinyourCityoutsideofyourCitysnewjobtrainingandplacementprogram
ToaddresscausalityyoumustcomparetheoutcomesofindividualswhoreceivedtheinterventionwiththosewhodidnotWithoutacomparisongroupofsimilarindividualswhowerenotexposedtotheinterventionitisimpossibletoknowwhichfactorsareresponsiblefortheoutcomesRandomizedControlTrials(RCTs)inwhichindividualsarerandomlyassignedtotreatmentandcontrolgroupsarethegoldstandardinprovingcausalityIfyoucandemonstratethatoutcomesimprovedforthosewhoreceivedtheinterventionanddidnotimproveatallorimprovedlessforacomparisongroupyoucanbesomewhatcertainthattheinterventionwasthecause
Examplesofothernon-causalimpactevaluationsthatarerigorousandreliableincludecertainquasi-experimentalstudiesusingcomparisongroupdesignssmartinterruptedtimeseriesanalysesthatmodeltrendsovertimeinwaysthatallowestimationofinterventioneffectswelldesignedpre-posttestanalysesneedsassessmentsimplementationstudiesetcFormoreinformationondifferenttypesofevaluationsseetheSageResearchMethodswebsite
Lookfordifferencesingroups
BuildintimetoalsoexaminethepilotdatatoseeifgroupsareexperiencingdifferentresultsReviewingperformancebasedonoutcomemeasuresforapopulationinaggregatemayhidethesubtlewaysthatsomepeoplemaybeaffecteddifferentlysoitisnecessarytocarefullyanalyzethedatafordisparateoutcomesForinstancetheMovingtoOpportunity(MTO)studythatexaminedtheimpactofhousingvouchersonfamilieswholivedinhigh-povertyneighborhoodsfoundthatchildrenwhomovedbeforetheageof13hadbettereducationandhigherincomesasadultsHoweveradultsdidnotseeimprovededucationandemploymentoutcomes
Disaggregatingdataintomorespecificcategoriesofraceandethnicitycanhelpunderstandtrendsanddisparitiesinamoregranularwaywhichcanhelpdecisionmakerstargetpoliciesandinterventionsinamoreequitableway
EarlierintheRoadmapweworkedtoidentifystakeholdersandwhomightbepositivelyandnegativelyimpactedbyachangeinpolicyUsetheresultsofthatexercisetoinvolvethecommunityininterpretingdataanalysisandtogatherqualitativefeedbackfromthecommunitytocallattentiontoissuesForinstanceanincreaseinyouthdetentionmaybeexplainedbyinefficienciesorlackofaccesstotransportation
Designimplementationandevaluation
49
Designimplementationandevaluation
50
Shareyourstoryandsuccess
Sharetheresults
Storytelling
AnintegralstepinyourstrategyissharingtheresultsofyourpilotorpolicychangewithyourcommunityThisincludesdevelopingmessagingaroundtheworkofyourorganizationandcreatingaplantodisseminateyourstoriesWhytellstoriesaboutyourwork
Informandorunderstandasubjectorsituation
Giveneededcontextofdataorbitsofinformation
Relatablestoriesarememorableforreadersviewers
ReadersorviewerswillseetheinformationasmorecompellingwhenthereisademonstratedandrelatablechangeorimpactonthemselvesortheirfriendsneighborsandcolleaguesStoriesareaboutpeopleandgovernmentisfundamentallyabouthelpingpeopleSotellthestoriesofthepeopleyouandyourcitycolleagueshelped
Keepthesetipsinmindasyoudevelopyourstorywithyourteam
WhatwereyourfindingsandwhyshouldanyonecareTellyouraudiencehowtheproblemwassolvedWhathastheimpactbeenonresidentsandcitystafftodateWhatwasthesuccessandweretheremultiplesuccessesalongthewayIfyouhavepreliminaryresultsordatashowingimprovementtosharethisisthetimetodoit
WhowasinvolvedandimpactedReviewthelistofstakeholdersHowwereeachofthemimpactedbytheprogramHumanizeyourstoryandimpactbyaddingadditionalvoicesThesevoicescanbeoneofthefollowing
Apersonorfamilywhowaspositivelyaffectedbythechangesuccess
Aninvolvedthird-partydescribingboththeproblemandthesolution(ieanorganizationnonprofitprovidersbusinesstradeassociationsportsteametc)whoispartofthecommunitythatthegovernmentworkedwithonthechangesuccess
Someonefromthegovernmentitselfwhocanspeakaboutthechangesuccess
TellmoreaboutyourcityDescribeyourcityincludingpopulationdemographicsindustriesetcWhatareitschallengesWhatisitknownforWhatmakesituniqueHowisitsimilarordifferentfromothercitiesintheUS
Shareyourstoryandsuccess
51
DescribeyourcityrsquoschallengeWriteoutthechallengeyourcityisfacing(asdefinedinyourproblemstatement)HowdoesitaffectresidentsHowdoesthisproblemaffecttheworkofcitystaffmembers
AddthedetailsProvidedetailsonyourappliedresearchstrategyTalkaboutwhatyourcityhasdonenthepasttoresolvetheissueDiscusshowyourstrategyhaschangedwhenyoubegantoadoptthisapproachWasmoneysavedorinvested
Usethisstorymapexercisetoorganizeyourthoughtsandguaranteethatyouincludealloftheelementsofatraditionalstory
AfteryoudevelopyourmessagingcreateacommunicationsmarketingandoutreachstrategydesignedtospreadyourmessageYourstrategyshouldwalkyouthroughthefollowingkeyactivities
1 IdentifyandprofileyouraudienceWhencreatinganddisseminatingyourmessageitisalwaysgoodtothinkaboutyouraudienceWhoexactlywouldbeinterestedinwhatyouhavetosayThegeneralpopulationofthecityPeoplewhoworkinnonprofitsBusinessownersItmattersbecausenoteveryonewholivesinacitymaybethetargetoftheinformationBusinessownersmaybemoreinterestedindatashowingeffectsonthebusinessdistrictthanothersandsoonWewillcoverthislaterbutthisstepwillalsohelpyoutothinkofthemosteffectivewaytoreachyouraudience
2 Defineyourcalltoaction(ifyouhaveone)SpendsometimeconsideringwhyyouareinterestedintellingyourstoryDoyousimplywanttoupdatethepubliconyourprogressorcelebrateyoursuccessesAreyouinterestedinyouraudiencechangingitsbehaviororsupportingaspecificinitiativebeinglaunchedbyyourorganizationIfyouwanttohaveyouraudienceactorbehaveinacertainwaybespecificandincorporateitinyourmessaging
Shareyourstoryandsuccess
52
3 SelectperformancemeasuresThinkabouthowyouwoulddetermineifyourcampaignissuccessfulandselectappropriateperformancemeasuresYouwillrevisitthesemeasuresattheendofyourcampaigncommunicationsplanimplementationtodeterminewhetheryouneedtomodifyyourcommunicationsandoutreachstrategyforthefuture
4 MethodstocommunicateyourstoryTherearemanywaystocommunicatesuccessofyourprojectHerearesomesuggestionsonreachingaudiencesItisimportanttodeterminewhichmethodsarethemosteffectiveforyourdesiredaudiencetoreceiveinformationDoyourresearch(PewResearchCenterhasexcellentstudiesonthereachofmediadigitalmediaandmarketing)ForexampleifseniorsareyourtargetpopulationandtheyarelessinclinedtouseInstagram(thatistrue)youmaywanttoconsiderFacebookorthenightlylocalnewsbothofwhichareusedmoreoftenbyseniorsBelowisalistofsomemethodsandtoolsyoucanuse
i Pressreleasestolocalnewspapersandtvstations
ii ShortsmartphonevideosAgreatwaytoengageanaudienceisthroughvideoAskonequestionofsomeonewhoispartofthestoryMaybethereisasocialworkerwhocandiscusshungerinyourcityThesevideoscanbereusedonmultipleplatforms
iii Long-formwriting
iv TellingindividualstoriesthroughaseriesofphotosandshortnarrativesforpublicationonFacebookorInstagram
v Op-edsopinionpiecesplacedinthelocalpaper
vi Emailupdatestoyourlist
vii Connectingwithappropriateorganizationsthatareinvestedinthetopicandworkwiththecommunityandaskthemtodisseminateinformationtotheirmembers
viii EmployingldquoWordofMouthrdquo(WOM)tacticsbycreatingtownhallsorfreeeventsforresidentstoattendintheirownneighborhoods
5 DeterminethetypeofcollateralUsuallycampaignsthattargetresidentsrequireseveralavenuestoreachthemPeoplearenotalwaysinthesameplaceorspaceHavingakeenunderstandingofyouraudiencewillassistyouwithyourchoicesonwhatneedstobedevelopedSometimesthatmeansimagesforsocialmediaaonepageflierthatgivespeopleinformationonattendinganeventoranin-depthbrochure
Shareyourstoryandsuccess
53
6 ImplementyourstrategyKeepinmindstrategiescanbefluidSometimessomethingmaynotturnoutasexpectedevenifduringplanningsessionsitwasldquosupposedtordquoItisokYoucananalyzewhathappenedduringadebriefsessionandonceyoursurveysandothermeasurementshavebeenfinalized
Visualizations
Withtherightvisualizationyoucanreviewandshareinformationinsightsanalysisperformanceandtrendsallofwhichisvaluabletoacross-sectionofthepublicButwhichvisualizationisrightforyouTochoosetherighttypeofchartgraphdeterminethetypeofquestionyouwanttoanswerSomecommoncharttypesarebelowandmoreinformationcanbefoundhere
Exploreadditionalanalyticaltechniques
Duringthegoal-settingphaseyouestablishedoutputoutcomeandimplementationmeasuresBasedontheresultsofyourpilotyoumayneedtoadjustyourmeasuresWiththepilotdatainhandyoumayhavesomeideasofhowfutureanalysescouldbedifferentandpotentiallyimprovethequalityofyourprogramTakesometimetobrainstormotherrelevantdatasetsthatyoudidnothavethetimetopursueearlierincludingmessydepartmentaldatathatneedsmoreattentionorsourcesofpublicdatafromthestateandfederalgovernmentnonprofitsandevenprivatecompaniesthatcouldsupplementyourunderstandingofanissueUsingmachinelearningtechniquesyoumightbeabletobetter
Shareyourstoryandsuccess
54
predictdemandforhomelessservicesusingNOAAweatherforecastspreemptwatermainbreaksusingsoilstratigraphyorinvestineconomicdevelopmentbymonitoringrealestatelistingdata
Advancedanalyticaltechniquescanalsohelpyourimplementationrunmoreeffectivelybyintegratingdata-informedtargetinganddecisiontoolsintotheprocessByautomatingroutineworkflowsandsurfacingsubtletrendsthesetoolscanhelpyouiterateinamoreprecisemannerandprovidekeystrategicguidancetofront-lineemployeesthatwouldotherwisebeunavailableThesesortsofdata-science-enhancedprocessescanbeclassifiedasoneofsixprojecttypesasexploredfurtherbytheNewOrleansLANOLAlyticsteamFormoreguidanceonscopingasimilarprojecttheGovExadvancedanalyticsscopingguidegivesyouastep-by-stepoverviewofhowtobesuccessful
Usethespacebelowtolistyourideasforpossibleadvancedanalyticsprojectsbytype
Typeofproject Ideas
Findingtheneedleinthehaystack
Prioritizingforimpact
Earlywarningtools
Betterquickerdecisions
Optimizingresourceallocation
Experimentingforwhatworks
Keepthefeedbackloopgoing
GatherinformationonpublicsentimentaboutpolicychangethroughfocusgroupssurveysandpublicmeetingsledbyadvisorygroupmembersNotewhethernewinformationisavailablethatcanhelpanalyzeorcontextualizeyourprogress(egfederalorstatedatareports)Sharetheresultsofyourpilotandthepolicychangeprocesswithothercitiesandresearchers
Shareyourstoryandsuccess
55
WorkingwitharangeofevidencePresentlythereisalotofattentionandinvestmentinevidence-basedpolicymakingincludingthefederalbipartisanCommissiononEvidence-BasedPolicymakingthePewCharitableTrustsreportsandtoolkitsonHowStatesEngageinEvidence-BasedPolicymakingandtheLauraandJohnArnoldFoundationsupportedEvidence-BasedPolicymakingCollaborativeManyoftheseinitiativesfocusonrigorousevaluation--meaningrandomizedcontrolledtrials(RCT)orotherexperimentalmodels--asthebestkindofevidencetouseHoweverthisbodyofevidenceisofteninsufficientforthespecifickindsofquestionsthatgovernmentscanhavewhenmakingadecisionaboutpolicyInthisguideweproposeanapproachtosupportgovernmentsinincorporatingadditionaltypesofevidenceintotheirdecisionmaking
OnewaytounderstandarangeofevidenceistouseNestarsquosStandardsofEvidencetoframeanddefineevidenceTheBritishnonprofitNestaworkstoinjectpositivechangeintothewayUKgovernmentpoliciesandprogramsfunctionOneoftheirkeyareasofworkhasbeendevelopingWhatWorksCentresorganizationsthatcollectcommissionsynthesizeanddisseminateevidenceaboutinterventioneffectivenessacrossarangeofsocialimpactareasNestadevelopedthefollowingStandardsofEvidencetoprovideausefulwaytodocumenthowdifferentkindsofinformationdemonstratedifferentlevelsofevidence
Workingwitharangeofevidence
56
Theimportanceofpotentiallybiasedinformation
Onechallengepresentedbyanevidenceclassificationthatacknowledgesthevalueofnon-scientificformsofevidenceistofigureouthowtobalanceallofthepiecestogetherwhenyouusethemTounderstandhowtodothiswehavetothinkabouthowtopositionpotentiallybiasedinformationindecisionmaking
WhenwelookatinformationwhichhasnotbeenrigorouslyevaluatedweacceptthatthisinformationmightbebiasedThewholereasonweconductrigorousevaluationistoremovebiasfromourobservationsWeconductstructuredtrialstoproduceresultsthatareconsistentandgeneralizableIfafindingisunbiasedthisaccuracymdashthislackofbiasmdashcanbeprovedbecauseanyindependenttestercanreproduceitwhentheyfollowanidenticalprocess
YoucanlookatpotentiallybiasedinformationasjustinformationthatisearlieroninitspathtorefinementreplicationandverifiedaccuracyApieceofinformationbasedonanyoneindividualrsquosobservationsislikelytobebiasedbecauseitiscomingfromjustonesourceBiasisreducedbygatheringtogetherlotsofindependentsourcesofinformationandfiguringoutitsmostcommonattributes
MaybethebiasedinformationmakesaclaimthatistoolargemdashitsayssomethingistrueunderallconditionswhenitisreallyonlytrueundercertainconditionsorthataneffectofaninterventionisbiggerthanitstrueaverageorthataneffectlastsovertimewhenitdoesnotPerhapschangingtheclaimtobemorespecificmdashcriticallyreviewingtheinformationtoseeifspecificconditionsmustbemettoachievetheoutcomesmdashwouldmaketheclaimmoreaccurate
IfyoutestedyourclaimfounditdidnotholdupinsomecaseschangedittesteditagainuntilyourclaimrepeatedlyheldupthenyoucoulddemonstratethattheclaimwasnolongerbiasedbutaccurateThisprocessoftestingimprovingtestingandvalidatingiswhatwegetfromstructuredscientificstudy
Withindividualpiecesofinformation--crediblehypothesescasestudiesexpertopinion--youcannotassumethatkindofaccuracyAtthesametimetheseearlyobservationsarefoundationaltoanyreplicablescientificobservationsWewillnevergetanywhereifwedonotstartwiththesepiecesofinformationeveniftheymightbebiased
Combininglevelsofevidencetogetthemostdetailedinformationforyourcity
ItisespeciallyimportanttoacceptdifferentformsofevidencewhenyoufacequestionsthathavenotyetbeentestedbyindependentstudiesSomeofthemostimportantquestionsrelatetothingsthatareveryspecifictoyoursituationYoufaceaparticularcombinationof
Workingwitharangeofevidence
57
issuesactorstrendsandresourcesthatisultimatelydifferentfromeveryotherplaceintheworld
ThisisimportantbecauseaspolicypractitionersandnotscientistsyoucaremoreaboutyourspecificcasethanahypotheticalaveragegeneralcaseYoucarelessaboutthetotallikelyrangeofimpactsfromapolicythanhowthissingleimplementationwillworkYouwantinformationaboutpoliciesthatwillprovideyouwiththebestpossiblepredictionabouthowtheywillworkinyourcontext
ByputtingdifferentkindsofevidenceintothesameframeworkNestarsquosStandardsofEvidenceimplythatwhenyoucannotgetrigorouslytestedexperimentallyverifiedinformationitisappropriatetoworkwithwhatwedohaveThisisrelevantforthemunicipalimplementationcontextbecausenostudypulledfromaliteraturereviewwillaccountforallofthevariablesyoufaceinyourowncontext
Intheabsenceofhavingastudythatpredictshowaninterventionworksinyourspecificcontextyouwillneednon-scientificinformationtotrytofillinthosegapsYouwillneedtogatherstoriesfromresidentsstaffmembersandotherstakeholdersthatsitonLevels1or2ofthestandardsofevidence
Someofthekindsofinformationyouwillprobablygetmoreeasilyfrompeoplethanstudiesinclude
howpeopleincitieslikeyoursreactedtoaninterventioninpublicforumshowpeoplewhoimplementedaninterventioninacitylikeyoursfiguredouthowtogetitpassedhowthecitycommunicatedpubliclyabouttheapparentbenefitsanddrawbacksoftheinterventionthepoliticalacceptabilityoftheintervention
Youcanremainsimultaneouslyawarethattheinformationyouaregatheringthroughindividualstoriesisbiased--notgeneralizableinformedbyspecificperspectives--whilealsorecognizingthathavingsomeinformationaboutsomethingimportantisbetterthannoinformationEvidencefromthefirsttwoevidencelevelsareessentialbuildingblocksofknowledgeandperfectlyusefulwhenyoucannotgetdetailedunbiasedknowledge
Youhavetostartsomewhereandyoushouldalwaysstartwithevidenceyouhaveaboutyourveryspecificconditionsthatyouknowareimportantforsuccess
Workingwitharangeofevidence
58
Thankyou
Interviewswithcities
GovExheldsemi-structuredinterviewswithsixstaffmembersinfourcitiestoestablishtheinitialstepsoftheRoadmapWeaskedrespondentstotalkaboutaninitiativeorpolicythattheyhavebeenresponsibleforimplementingintheircityRespondentsfromthreecitiesspokeabouttheireffortstoaddresshomelessnessandrespondentsfromonecityspokeaboutblightAfterrespondentsidentifiedtheproblemtheyspokeabouthowthecityprioritizedthisissuethevariousstepsinvolvedandbarrierspresent
GovExconductedtextanalysisoftheseinterviewswhichrevealedthatpeoplewasthemostcommonlyusedtermduringtheseconversationswhendiscussingthestepstochangeaprogramorpolicyThisisathemethroughoutthisRoadmapThefirstsectionbeginswithidentifyingtherelevantpeopleTheconversationsaroundevidencearefocusedonhumanbehaviorratherthanmethodologyorstrengthofevidence
Thankyou
59
OncewehadadraftoftheRoadmapwefollowedupwithcitiesonanindividualbasistolearnmoreaboutwhatwouldbehelpfulWearegratefultothefollowingcitystafffortheinterviewsandindividualconversations
MichaelBaskinChiefPolicyOfficerChattanoogaTN
MelissaBridgesPerformanceandInnovationCoordinatorLittleRockAR
VictoriaCarreoacutenAdministrativeOfficerLasVegasNV
TyroneGrandstrandCommunityDevelopmentPlannerFargoND
TibLaughlinDirectorofGeneralServicesKansasCityKS
TimMorelandDirectorofPerformanceManagementandOpenDataChattanoogaTN
CharlesRosendahlManagementAnalystDentonTX
Thankyou
60
MelissaSiebenAssistantCountyAdministratorKansasCityKS
GinaThompsonDirectorofStrategicServicesLewisvilleTX
JHUFacultycollaboration
GovExalsoconvenedthecitycontactsandfacultypartnersinaseriesofAdvisoryGroupmeetingsThesemeetingsservedtoconnectcitiesandfacultytounderstandoneanothersperspective
StefanBaralBloombergSchoolofPublicHealth
AshleyBernerJHUSchoolofEducation
MelissaDavey-RothwellBloombergSchoolofPublicHealth
JonLinksBloombergSchoolofPublicHealth
TamarMendelsonBloombergSchoolofPublicHealth
KristinMmariBloombergSchoolofPublicHealth
KeshiaPollackPorterBloombergSchoolofPublicHealth
JoshSharfsteinBloombergSchoolofPublicHealth
LizStuartBloombergSchoolofPublicHealth
LindsayThompsonCareyBusinessSchool
DanielWebsterBloombergSchoolofPublicHealth
Partnersindataandevidence-basedpolicy
Wealsospokewithpartnerswhoareexpertsinthefieldofevidence-basedpolicywhosharedtheirknowledgeandresourcesofcurrentandfuturethinkinginevidence-informedthinking
WendyBarrenoAssociateDirectorWilsonSheehanLabforEconomicOpportunitiesUniversityofNotreDame
TiffanyDavenportDirectorofDataPartnershipsBaltimoresPromise
TomiHiersDirectorBaltimoreCivicSiteAnnieECaseyFoundation
ChrisKingsleySeniorAssociateAnnieECaseyFoundation
KathrynOliverAssociateProfessorLondonSchoolofHygieneandTropicalMedicine
Thankyou
61
KristenWatkinsSeniorPolicyAssociateJ-PAL
GovExParticipants
GovExAnalystMiriamMcKinneyanalyzedtheinterviewtranscriptsandconductedtextanalysisSeniorImplementationAdvisorEmilyShawwillintegrateRoadmapactivitieswithadditionalservicedeliveryactivitiesatGovEx
Thankyou
62
ReferencesAbrahamKGampHaskinsR(2017)Thepromiseofevidence-basedpolicymakingReportofthecommissiononevidence-basedpolicymakingWashingtonDCCommissiononEvidence-basedPolicymakingBigmove2Extendingthegreen(nd)Retrievedfromhttpswwwcityoflewisvillecomdoing-businesslewisville-2025big-move-2-extending-the-greenBatesMAampGlennersterR(2017)ThegeneralizabilitypuzzleRetrievedfromhttpsssirorgarticlesentrythe_generalizability_puzzleBuildwithnotfor(2018January)RetrievedfromhttpsbuildwithnotforwordpresscomCairneyPampOliverK(2017)Evidence-basedpolicymakingisnotlikeevidence-basedmedicinesohowfarshouldyougotobridgethedividebetweenevidenceandpolicyHealthResearchPolicyandSystemsdoi101186s12961-017-0192-xCairneyPOliverKampWellsteadA(2016)TobridgethedividebetweenevidenceandpolicyReduceambiguityasmuchasuncertaintyPublicAdministrationReview76(3)399-402Retrievedfromhttpdxdoiorg101111puar12555Centreforpublicimpact(nd)RetrievedfromhttpswwwcentreforpublicimpactorgCernaL(2013)ThenatureofpolicychangeandimplementationAreviewofdifferenttheoreticalapproachesOrganisationforEconomicCo-OperationandDevelopmentRetrievedfromhttpswwwoecdorgeduceriTheNatureofPolicyChangeandImplementationpdfColoradoSprings(nd)Retrievedfromhttpscoloradospringsgovcommunicationspagewhat-expect-when-you-speakupContinuumofcare(coc)program(nd)RetrievedfromhttpswwwhudexchangeinfoprogramscocCouncilpresidentkicksoff$12millionyouthfund(nd)Retrievedfromhttpwwwbaltimorecitycouncilcomcontentcouncil-president-kicks-12-million-youth-fundDurlakJAampDuPreEP(2008)ImplementationmattersAreviewofresearchontheinfluenceofimplementationonprogramoutcomesandthefactorsaffectingimplementationAmericanJournalofCommunityPsychology41327-350Retrievedfromhttpswwwncbinlmnihgovpubmed18322790Fishbonediagram(ishikawa)(nd)Retrievedfromhttpasqorglearn-about-qualitycause-analysis-toolsoverviewfishbonehtmlFivewhysGettingtotherootofaproblemquickly(nd)RetrievedfromhttpswwwmindtoolscompagesarticlenewTMC_5Whtm
References
63
FligsteinNampMcAdamD(2011)TowardageneraltheoryofstrategicactionfieldsSociologicalTheory29(1)1-26RetrievedfromhttpsociologyberkeleyedusitesdefaultfilesfacultyfligsteinFligstein20McAdam20Soc20Theory20PaperpdHeinkelP(2018January12)CitymullspermitsforbikesharingcompaniesRetrievedfromhttpwwwdentonrccomnewsdenton20180112city-mulls-permits-bike-sharing-companiesHowisareamedianincomecalculated(2012October)Retrievedfromhttpswwwhudexchangeinfofaqs843how-is-area-median-income-calculatedLampkinLMampHatryHP(2003)KeystepsinoutcomemanagementUrbanInstituteRetrievedfromhttpswwwurbanorgresearchpublicationkey-steps-outcome-managementMarshDampSmithM(2000)UnderstandingpolicynetworksTowardsadialecticalapproachPoliticalStudies484-21Retrievedfromhttpjournalssagepubcomdoiabs1011111467-924800247McCannL(nd)RetrievedfromhttplaurenellencomMoultonSampSandfortJR(2016)ThestrategicactionfieldframeworkforpolicyimplementationresearchPolicyStudiesJournal45(1)Retrievedfromhttponlinelibrarywileycomstore101111psj12147assetpsj12147pdfv=1ampt=jdkhhcp4amps=e184c625e1855a6b1615ccb99ee27b1c0b3212d8PaynterB(2018January1)What100citiesarelearningfromeachotherbysharingtheirdataRetrievedfromhttpswwwfastcompanycom40518004what-100-cities-are-learning-from-each-other-by-sharing-their-dataPeakacademy(nd)Retrievedfromhttpswwwdenvergovorgcontentdenvergovenmayors-officeprograms-initiativespeak-performancepeak-academyhtmlPinnockHBarwickMCarpenterCREldridgeSGrandesGGriffithsCJRycroft-MaloneJMeissnerPMurrayEPatelASheikhAampTaylorSJC(2017)Standardsforreportingimplementationstudies(stari)ExplanationandelaborationdocumentBMJOpenRetrievedfromhttpbmjopenbmjcomcontentbmjopen74e013318fullpdfProctorESilmereHRaghavanRHovmandPAaronsGBungerAGriffeyRampHensleyM(2011)OutcomesforimplementationresearchConceptualdistinctionsmeasurementchallengesandresearchagendaAdministrationandPolicyinMentalHealth38(2)65-76doi101007s10488-010-0319-7TandonSD(2018)ProgressincommunityhealthpartnershipsResearcheducationandactionRetrievedfromhttpswwwpressjhuedujournalsprogress-community-health-partnerships-research-education-and-actionProvanKGampKenisP(2007)ModesofnetworkgovernanceStructuremanagementandeffectivenessJournalofPublicAdministrationResearchandTheory
References
64
18(2)229-252Retrievedfromhttpsacademicoupcomjpartarticle182229935895SabatierPA(1988)Anadvocacycoalitionframeworkofpolicychangeandtheroleofpolicy-orientedlearningthereinPolicySciences21(3)129-168Retrievedfromhttpwwwjstororgstable4532139SandstromAampCarlssonL(2008)TheperformanceofpolicynetworksTherelationbetweennetworkstructureandnetworkperformancePolicyStudiesJournal36497-524Retrievedfromhttponlinelibrarywileycomdoi101111j1541-0072200800281xabstractSchwartzMampWilsonE(nd)WhocanaffordtoliveinahomeAlookatdatafromthe2006americancommunitysurveyRetrievedfromhttpswwwcensusgovhousingcensuspublicationswho-can-affordpdfShearerJCAbelsonAKouyateBLavisJNampWaltG(2016)WhydopolicieschangeInstitutionsinterestsideasandnetworksinthreecasesofpolicyreformHealthPolicyandPlanning31(9)1200-1211Retrievedfromhttpsdoiorg101093heapolczw052ThelessonsofadministrativedataHigh-profilepolicy-relevantresearchpoweredbyadministrativedata(nd)Retrievedfromhttpswwwpovertyactionlaborgsitesdefaultfilesresources20161114-The-Lessons-of-Administrative-DatapdfToolsandotherneedsforqualitativeandquantitativemethods(2015February)AgencyforHealthcareResearchandQualityRetrievedfromhttpwwwahrqgovprofessionalsquality-patient-safetytalkingqualityassesstoolshtml
WeaverRCampBagchi-SenS(2013)SpatialanalysisofurbandeclineThegeographyofblightAppliedGeography4061-70
References
65
UniversityPartners
Name Title Affiliation Area(s)ofexpertise
Contactinformation
Examplesofworkingwithnon-academicdecisionmakers
WendyPBarreno
AssociateDirector
WilsonSheehanLabforEconomicOpportunitiesUniversityofNotreDame
LEOsfocusareasincludehealtheducationhousingchildwelfareeconomicopportunitycriminaljustice
wbarrenonduduorleondedu
MelissaDavey-RothwellPhD
AssociateScientist
JohnsHopkinsBloombergSchoolofPublicHealth
Health mdavey1jhuedu410-614-5854
BaltimoreCityHealthDepartmentLeadersMarylandDepartmentofHealthAddictionRecoveryServicesinDCExecutiveDirectorsofcommunityorganizationsacrosstheUSthatfocusonHIVSTIsubstanceabuseandimplementationscience
ElizabethStuart
AssociateDeanforEducationandProfessorDepartmentsofMentalHealthBiostatisticsandHealthPolicyand
JohnsHopkinsBloombergSchoolofPublicHealth
Educationsocialpolicystatisticalmethods
estuartjhuedu
Resources
66
Management
Design-ThinkingResources
EngagementGuideforCivicTech
Journeymap
PerformanceManagementResources
DenverPeakAcademy
Proxymeasuresguide
Talkingaboutperformancewithoutcomplicatingit(video)
SettingPerformanceTargetsguide
Datavisualizationguide
Performancemanagementgettingstartedguide
Exercises
Fishboneexercise
5Whys
AdministrativeData
TheLessonsofAdministrativeData(JPAL)
EvaluationResources
Quantitativeandqualitativemethods(USDepartmentofHealthandHumanServicesAgencyforHealthcareResearchandQuality)
KeyStepsinOutcomeManagement(UrbanInstitute)
RCT-YES
Sageresearchmethods
BestPracticesforMixedMethodsResearchintheHealthSciences
ACCORDChecklist
Pilotandimplementationplanning
Benchmarkingexamplesandproxymeasures
Seattleopendataplaybook
Resources
67
Datasciencescopingguide
Resources
68
AboutGovExGovExhelpsgovernmentsusedataandevidencetoimproveresidentsrsquolivesWesupportgovernmentsinbuildingtheirinternalcapacitytooperatesustainableanalyticspracticeswhichimprovecommunityoutcomesinfieldsfrompublicsafetytoeconomicdevelopmentGovExhasaproventrackrecordofworkinginconcertwithpartnerstoprovidetechnicalassistanceandtrainingatscale
SinceourlaunchinApril2015GovExhas
Trainedover2000governmentemployeesinpersonandonlineintopicsincludingdatamanagementperformancemanagementopendatacommunityengagementandanalytics
Providedtechnicalassistancetomorethan100localgovernmentsacrosstheglobeimprovingtheiruseofdatatoachieveefficienciesinservicedeliveryandbetteroutcomesforresidents
Assessedthecapacityofmorethan130governmentalorganizationstodeterminehowleadersstaffandexternalstakeholdersmanageandusedata
Localgovernmentsareincreasinglyonthefrontlinesofaddressinglargescaleproblemswithglobalimplicationssuchasclimatechangepublicsafetyandeconomicinequality
Whileevidence-basedinterventionstohelpsolvethoseproblemsareavailablelocalgovernmentstaffhavelimitedtimeandcapacitytofindandimplementresearch-basedsolutionsAsaresultcitiescontinuetoworkthewaytheyalwayshave--makingincrementalimprovementstoprogramsbutnotalwayscreatingchangeatasystem-level
ThroughresourcedevelopmentandcollaborationwithJHUfacultyGovExissupportingcityleadersinusingdataandevidencetomakedecisionsandcreatinglastingrelationshipswithresearchers
TheRoadmapwasdevelopedbyGovExaspartofitspartnershipwithBloombergPhilanthropiesWhatWorksCitiesinitiative
AboutGovEx
69
- Title page
- Introduction and purpose of the Roadmap
- Using the Roadmap
-
- Gather information about the problem
- Strategically engage key partners
- Review the evidence and collect details
- Decide on a policy to implement based on local factors
- Establish a framework to monitor progress
- Assess data accessibility quality and security
- Design implementation and evaluation
- Share your story and success
-
- Working with a range of evidence
- Thank you
- References
- Resources
- About GovEx
-
ReviewtheevidenceandcollectdetailsOnceyouhaveathoroughunderstandingoftheproblemdataandactorsyouwillneedtoconductyourownresearchonavailableevidenceandlearnaboutthelandscapeofpolicyoptionsYourgoalwillbetodevelopanaccessiblelistofinformationaboutpoliciesthathavehelpedsolvesimilarproblemsinotherplacesThischapterwillguideyouthroughthestepsofpreparingyourreviewersunderstandinglevelsofevidenceandcollectingthedetailsonevidencethatyouwillneedfordiscussionwithdecisionmakersinyourcity
Selectandprepareyourreviewers
AssignapersonorteamtoreviewrelevantinterventionsthathavedemonstratedsuccessinotherjurisdictionsTheirgoalwillbetoproduceareviewofalloftherelevantprogramspoliciesorinitiativesthatyourteamcouldconsiderimplementingTheywillbeconductingaliteraturereviewsupplementedbyinterviewswithexpertsandpeerswhohaveinformationaboutpotentialprograms
SincethepeopleyouassigntothistaskareprovidingthebaselineofpotentialoptionstoimplementitisimportantthatyouchoosepeopleyouareconfidentwillbethoughtfulandthoroughintheirworkItisalsocriticalthattheyareawareoftheimportanceofsourcecredibilityEnsureyourreviewershavefamiliarizedthemselveswithrulesforcriticallyanalyzinginformationsourcesbeforetheygetstartedIftheyareincludingnewsarticlesaspartoftheirsearchtheyshouldalsobeawareofmethodsforevaluatingnewscredibility
AskyourreviewerstodevelopaplanandtimelinetocompletethereviewThisstartswithsolidifyingagoverningresearchquestionOnewaytoknowyouarecreatingadetailedresearchquestionistousetheacronymPICOremindingyoutoincludereferencetothepopulationintervention(ifyouhaveoneinmind)comparisonandoutcomeelementsinyourresearchquestionForexampleifyouknowthatyouhaveapopulationofteenagerswhoarehomelessyouhavelearnedfromyourstakeholdersthatmanyoftheseteenagersidentifyasLGBTandyouhaveheardsomethingabouttheapproachofcreatingdrop-incentersfordeliveringsupportyoucouldask
ForLGBTyouthfacinghomelessness(population)howeffectivearedrop-incenters(intervention)relativetostandardsocialservices(comparison)atsettlingtheminpermanenthousing(outcome)
Ifyoudonothaveaninterventioninmindyoucanleaveouttheinterventionandcomparisonandsimplyask
Reviewtheevidenceandcollectdetails
21
HowcanwehelpLGBTyouthfacinghomelessness(population)obtainpermanenthousing(outcome)
OnceyourreviewershavetheirresearchquestionsusethequestionstocreatealistofkeywordstheycanuseintheirliteraturesearchThiswillhelpkeeptheircollectionofarticlesreasonablyfocusedHavethemdescribewhattheirsearchprocesswillbeincludinglistingthedatabasestheywillreviewandtheirplanforstoringtheinformationtheywillcollect(Herearesomeadditionalsuggestionsforstructuringareviewplan)
WhatyourreviewerswillbedoingisalightweightversionofarapidreviewTotheextentthattheywouldliketodevelopamethodologyforspecificsearchprotocolsincludingcriticalappraisalprocessestheycancertainlygoaheadanddoso
Usedifferentlevelsofevidenceforthemostcompletepicture
AsdiscussedintheWorkingwitharangeofevidencechaptertheNestaStandardsofEvidenceframework--reproducedbelow--ishelpfultounderstandthelevelofconfidencethataprogramorinterventionishavingapositiveimpactThisapproachletsyouputinterventionsthatyouhaveheardaboutlocallybutwhichhavenotbeenstudiedtogetherwiththosewithademonstratedcausaleffectThisishelpfulsinceanumberofpoliciesprogramsandpracticesthatyouwillwanttoconsidermaynotyethavemuchofaresearchrecord
Reviewtheevidenceandcollectdetails
22
NearlyanypieceofinformationaboutaninterventioncanbeclassifiedwithinthestandardsofevidenceForexampleifsomeonehasacrediblehypothesisabouthowaninterventioncouldproducebenefitthistypeofevidencewouldbeclassifiedasLevel1ThistypeofevidencecouldincludeinterventionsinothercitiesthatyoureadaboutinthenewsorhearaboutataconferenceForinstancewhenUSDAallowedschoolsandschooldistrictstoprovidefreebreakfastandlunchtoallstudentsthroughtheCommunityEligibilityProvision(CEP)manydistrictsimplementedthisoptionevenbeforetherewasevidenceofeffectivenessThiswouldfitintoLevel1sinceitbuildsonthelogicthatprovidingaccesstomealswilldecreasefoodinsecurity
YoumayalsofindpiecesofinformationthatqualifyasLevel2evidenceForinstanceasmoreschooldistrictsimplementedCEPprincipalstestifiedaboutpositivechangestheyobservedinstudentssuchasamorepositivedemeanorthatcanleadtobetteracademicperformanceDataalsoshowedthatthestudentsdescribedwerereceivingmoremealsthanbeforetheprogramsimplementationHoweveritwastoosoontobuildevidencetoshowacausalrelationshipbetweentheincreaseinfreemealsandincreasedacademicperformance
ldquoLower-levelrdquoevidencecanconsistofnuancedbutvaluableelementsthatrelatetoaninterventionrsquospotentialfitwithacommunityQualitativeinformationsuchastestimonyfrompeoplewhohaveexperiencedaninterventionandthosewhohavefacilitatedorimplementeditwillalwaysbeusefultoconsiderandprovidesdetailedlocalcontext
Experimentalstudies--describedstartingatLevel3withcontrolledexperimentsandgoingthroughLevel5withfullyfleshedoutprotocolsandprocessesforreplication--stilloccupythehigherlevelsofthisdescriptionofevidenceThisprioritizationshowsthevalueofrigorousstudyforevaluatingtheimpactofaparticularpolicyorprogramratherthanotherfactorsthatcouldbeinfluencingoutcomesofinterestRememberthatmultipletypesofevidencecancomplementrigorousimpactevaluationsandgivecityleadersadditionalinsightsandbroaderperspective
Accessinformation
Encourageyourreviewerstobegintheirsearchforpromisingpracticesingovernmentandnon-profitclearinghousesandinacademicpolicyjournalsWorkpublishedinthesesourceshasalreadybeensubjectedtoahighlevelofprofessionalscrutinyandgenerallycontainsagreatdealofdetailwhichmeansthatyourreviewersaremorelikelytobeabletopullouttheinformationyouneedtomakeagooddecision
Clearinghouses
Reviewtheevidenceandcollectdetails
23
Herearesomeoftheclearinghouses(collectionsofprogramevaluations)forevidence-basedpoliciesandprograms
ResultsFirstClearinghouseDatabaseThePewCharitableTrustsCollectstogethereightotherclearinghousestoprovideinformationoninterventionsinthefollowingcategoriesadultcriminaljusticeeducationchildwelfarehealthjuvenilejusticementalhealthsubstanceabuseandyouthdevelopmentIndividualclearinghousesincludedinResultsFirstare
BlueprintsforHealthyYouthDevelopment-foryouthdevelopmentandviolenceprevention
CaliforniaEvidence-BasedClearinghouse-forchildwelfareinterventions
TheLauraandJohnArnoldFoundationrsquosSocialProgramsThatWork-forsocialservicesprograms
TheUSDepartmentofEducationrsquosWhatWorksClearinghouse-foreducationinterventions
TheUSDepartmentofHealthandHumanServicesrsquoResearch-TestedInterventionPrograms(RTIPs)-forcancercontrolinterventions
TheUSDepartmentofHealthandHumanServicesrsquoTeenPregnancyPreventionEvidenceReview(TPPEvidenceReview)-forsexualriskbehaviorandsexualhealthoutcomes
TheUSDepartmentofJusticersquosCrimeSolutionsgov-forcriminaljusticejuvenilejusticeandcrimevictimservices
TheUSSubstanceAbuseandMentalHealthServicesAdministrationrsquosNationalRegistryofEvidence-basedProgramsandPractices(NREPP)-substanceuseandmentalhealthinterventions
TheUniversityofWisconsinPopulationHealthInstituteandRobertWoodJohnsonFoundationrsquosCountyHealthRankingsandRoadmapsWhatWorksforHealth(WhatWorksforHealth)-healthbehaviorsclinicalcaresocialandeconomicfactorsandthephysicalenvironment
UKWhatWorksCentresTheBritishgovernmenthascharteredanetworkofsubject-specificorganizationswhichcollectsynthesizeandreportoutevidenceontheeffectivenessofpolicyprogramsandpractices
NationalInstituteforHealthandCareExcellence(NICE)-healthandsocialcare
Reviewtheevidenceandcollectdetails
24
SuttonTrustEducationalEndowmentFoundation-educationalachievement
CollegeofPolicingWhatWorksCentreforCrimeReduction-crimereduction
EarlyInterventionFoundation-earlyintervention
WhatWorksCentreforLocalEconomicGrowth-localeconomicgrowth
CentreforAgeingBetter-qualityoflifeforolderpeople
WhatWorksCentreforWellbeing-wellbeing
WilsonSheehanLabforEconomicOpportunities(LEO)UniversityofNotreDameProvideslinkstotheirevaluationsofinterventionsincriminaljusticeself-sufficiencyeducationhealthandhousingandhomelessness
ClearinghouseforLaborEvaluationandResearchUSDepartmentofLaborProvidesresearchonworkforceandlabor-relatedpoliciesandprograms
J-PALEvaluationsAbdulLatifJameelPovertyActionLabStudiesonpoverty-reducinginterventionsfromaroundtheworld
CampbellSystematicReviewsCampbellCollaborationSystematicreviewofpolicyinterventionsbyscholarlycoordinatinggroupsinsevenissueareascrimeandjusticeeducationinternationaldevelopmentmethodssocialwelfareknowledgetranslationandimplementationanddisability
Benefit-CostResultsWashingtonStateInstituteforPublicPolicyAWashingtonState-charteredprogramtoestablishthelikelihoodofrelativecostsandbenefitsforprogramsacrossthefollowingareasjuvenilejusticeadultcriminaljusticechildwelfarepre-Kto12educationchildrensmentalhealthhealthcaresubstanceusedisordersadultmentalhealthpublichealthandPreventionworkforcedevelopmentandhighereducation
EffectiveInterventionsCentersforDiseaseControlandPreventionRepositoryforevidence-informedHIVpreventioncareinterventions(withstepsonhowtoimplement)
Journalsandindexes
OnceyouhavereviewedrelevantinformationaccessiblethroughclearinghousesitisworthtakingalookatindividualstudiespublishedasscholarlyarticlesYourreviewerswilldothisbysearchingarticleindexesandacademicjournalsUnfortunatelyanumberofusefulacademicjournalsandindexingplatformsmaynotbegenerallyaccessibletoyourreviewersifyoudonothaveasubscriptiontothemToaddressthisconsiderpartneringwithalocal
Reviewtheevidenceandcollectdetails
25
collegeoruniversitytoprovideyouwithalistofarticlesfromthefollowingsources(Thisisalsoanexcellentopportunitytoseekresearchsupportfromanyacademicstakeholdersyouidentifiedpreviously)
Indexes
GoogleScholarincludesmostpeer-reviewedonlineacademicjournalsandbooksconferencepapersthesesanddissertationspreprintsabstractstechnicalreportsandotherscholarlyliteratureincludingcourtopinionsandpatentsItallowsyoutospecifyapublicationrangeinyoursearchandprovideslinkstolikelysourcesforalllistedarticles
PolicyFileIndexisauniqueresourceforUSpublicpolicyresearchinthatitgrantsusersaccesstotimelyupdatedinformationfromover350publicpolicythinktanksnongovernmentalorganizationsresearchinstitutesuniversitycentersadvocacygroupsandotherentities
PolicyArchiveworkstocollectanddisseminatesummariesandfulltextsvideosreportsbriefsandmultimediamaterialofthinktankuniversitygovernmentandfoundation-fundedpolicyresearch
Anothersourceforhigh-qualityresearchisthereportsoftheCongressionalResearchServiceYoucanfindtheirreportsaggregatedhere
Specificjournals
Oftenthemostusefulacademicjournalsforunearthingpolicyorprogramreviewswillbeineachspecificsubjectarea--thatisineducationjournalsforeducationpoliciesYoucanuseageneralindexlikeGoogleScholartobegintolearnwhichjournalsaremostoftenusefulforyoursearchHoweveryoucanalsosearchthefollowingtwogeneralistpolicypublicationsforissuesandarticlesrelatedtoyourtopic
PublicAdministrationReviewJournalofPolicyAnalysisandManagement
Newssourcessubjectmatterexpertsandinterviews
AlthoughitisvaluabletoknowwhathasbeenexploredthroughascholarlyresearchframeworkitisalsousefultolearnwhatkindofinterventionsarebeingtriedinthefieldrightnowandhowpeoplefeelaboutthesetrialsSearchsubscriptionnewsaggregatorslikeLexisNexisorfreenewsaggregatorslikeGoogleNewsorYahooNewsforyourissueareaplusthewordpolicyorprogramtoseewhatkindsofinterventionsarebeingcoveredinthenews
Reviewtheevidenceandcollectdetails
26
DonotforgettocontacttheadvisorygroupmemberswhowerechosenfortheirsubjectmatterexpertiseInadditiontoprovidinggoodmaterialthroughinterviewsthemselvestheymayhaveideasaboutothersubjectmatterexpertsyoushouldcontacttoaskaboutpotentialinterventions
Peercitiesandnetworks
InadditiontoreadingabouttheworkofothercitiesitisidealifyouareabletospeakwithpeoplewhoareinvolvedwithcurrentimplementationsofinterestinginterventionsInyourcityyoulikelyhaverelationshipswithyourneighborsbutitcanbehelpfultolookoutsideofyourregionforsolutionsaswellTofindcounterpartsinothercitieswhoareworkingonaprogramyouwouldliketolearnmoreaboutyoucancontactstaffinintermediaryorganizations(likeGovExoryourprofessionalassociations)whomightbeinapositiontoputyouintouchimmediatelyorwhomightseeanothergoodrouteforsettingupaconversationHereisalistofnetworksthatmaybehelpful
InternationalCityCountyManagementAssociation(ICMA)NationalAssociationofCounties(NACO)NationalLeagueofCitiesUSConferenceofMayorsWhatWorksCities
Collectthedetails
Onceyourreviewershavefoundevidenceonpossiblepoliciesorprogramstoimplementtheyshouldcapturekeydetailsthatwillhelpdecisionmakersunderstandthepotentialeffectivenessandapplicabilityofeachpotentialintervention
Foreachsourcethathasprovidedinformationyouwanttoconveyusethetemplatebelowtocaptureasmanyofthefollowingdetailsaspossible(Youdonotneedtocompletethisforsourcesthatprovideduplicativeinformationorareotherwisenotthatuseful)
Programorpolicyname (egldquoHousingFirstrdquoorldquoblightedpropertiesregistryrdquo)
Sourcetype (egjournalarticleinterviewsubjectmatterexpertstudyassociationcommunityparticipationprocess)
Sourcedetails (egarticletitleandpublicationnameandtitleofinterviewee)
Evaluationmethodology
(egrandomizedevaluationcomparativeevaluationoftwodifferentgroupsbeforeafteranalysis)
Evidencelevel (egLevel12345)
Programorpolicy
Reviewtheevidenceandcollectdetails
27
description
Programorpolicygoal (1sentencecapturingtheprincipalaimoftheintervention)
Existingimplementations
(WherehastheinterventionbeentriedListasmanysitesasyouareawareof)
Characteristicsofimplementationsites
(ForeachimplementationsiteofferbasicgeographicIDmdashcityandstateThenyouwillwanttocollectthedetailswhicharerelevanttoyourdecisionmakersrsquoconsiderationsaboutwhatmakesalocationldquosimilartordquoorldquodifferentfromrdquoyoursituationYouwillatleastwanttoincludeforeachimplementationsitePopulationnumberPopulationdensity(ruralurbansuburban)Socio-economiccharacteristicsBeforestartingaskyourdecisionmakersaboutanyparticularcharacteristicsofprogramimplementationsitestheywouldwanttoknowbeforeseekingtoreplicate)
Characteristicsofimplementationpartners
(Anyavailabledetailsaboutorganizationsinstitutionsorpeoplewhowereinvolvedwithimplementingtheintervention-egindependentchildcarecenterpublicelementaryschoolcitypublichealthnurses)
Characteristicsoftargetpopulationeligibleprogramparticipants
(Anyavailabledetailsaboutpeoplewhoarethebeneficiariesoftheprogramorotherwiseincludedoraffectedbytheintervention-egnumberofprogramparticipantscategoriesofpropertyownersaffected)
Policyorprogramduration
(HowlonghastheinterventionbeeninplaceineachimplementationsiteHasitendedandifsowhy)
Whatbenefitscamefromeachimplementation
(Anyobservedpositiveeffectsoftheinterventionincludingeffectsize)
Statisticalestimatesofinterventioneffectiveness
(Anyavailablestatisticalestimatesoftheinterventionrsquoseffects)
Whatdrawbackswereobserved
(Anyavailabledetailsaboutunintendedorunavoidablenegativeeffectsoftheintervention)
Programorpolicycosts
(Anyavailabledetailsaboutprogramcostsperimplementation)
Reviewtheevidenceandcollectdetails
28
Reviewtheevidenceandcollectdetails
29
DecideonapolicytoimplementbasedonlocalfactorsTogetthecriticalbuy-infromleadershiptoimplementanewpolicyorprogramtheprojectleadshouldscheduleameetingwithdecisionmakerstopresenttheevidencesurroundingthepolicyoptionscapturedinthetemplateDependingonyourinsightintoyourcityleadershipswillingnesstomakechangeyoumaywishtosharetheresearchyouhaveconductedontherootcausesoftheproblemandsomeofyouranalysisontheimpactofcurrenteffortstodate
Considertheapplicabilityofanapproach
AsdiscussedinthepreviouschapterpolicymakersoftenlookforevidenceofeffectiveprogramsinothercitiesbutoftenthequestionarisesofhowgeneralizableaprogramisfromonecitytoanotherTheAbdulLatifJameelPovertyActionLab(J-PAL)suggeststhatinreviewingevidencefromanotherlocationitishelpfultoidentifythespecificmechanismthatwasused(deliveryofprogramorserviceinformationsharingetc)identifythelocalconditionsthatwereinplaceobservethebehaviorchangethatresultedfromthemechanismandfinallytounderstandlocalcontextandconditions(legalorcostconstraintsculturalnormsetc)Thissetofpresentationelementscanbedrawnfromthelistofdetailscollectedintheresearchtemplate
EducationpolicymakersinChicagoILusedfindingsfromprogramsinKenyaandIndiainwhichchildrenwhohadfallenbehindtheirpeersweregiveneducationalsupporttargetedtotheirleveloflearningThemechanismusedwastargetedteachingandthereweresimilarconditionspresentinallthreecountriesAlthoughsomestudentshadfallenbehindinthecurriculumteacherswereincentivizedtoteachgrade-levelmaterialsConsiderationsofhowthelocalcontextinChicagomightaffectimplementationofatargetedteachingprogramincludedunderstandinghowtheneighborhoodsthatthestudentswerefrommayhaveaffectedtheireducationaloutcomesandthehighcostofteachinginsmallgroupsInChicagoMatchEducationprovidedtwo-on-onetutoringwhichmatchedstudentswithwell-educatedindividualswhohavenothadformalteachertrainingforalower-costmodelTheresultswerepositiveStudentswhoweretutoredperformedbetterinmathandweremorelikelytopasshighschoolclasses
ThegeneralizabilityframeworkproposedbyJ-PALisbelowUsethisasatemplateasyouconsiderpotentialpolicysolutions
Decideonapolicytoimplementbasedonlocalfactors
30
SourceTheGeneralizabilityPuzzle
OnceyouhavenarrowedthepolicyoptionsdiscussthequestionsbelowforeachoptionsyouareconsideringThisstepiscriticalasunderstandinglocalcontextissoimportantinsuccessfulpolicyimplementation
CommunityContextQuestionnaire
Localcontext
DoesthisoptionmakesenseinourcommunityIsthisaonesizefitsallapproachIsanyparticulargroupabsentfromthisapproachIfsoisthatok
Youranswers
Effectiveness
WhatwouldweexpecttoobserveifthepilotisaseffectivehereasinCityX
Youranswers
Timing
DoesthetimingmakesenseIsthistherighttimetoimplementthispolicychange
Decideonapolicytoimplementbasedonlocalfactors
31
WherearethepotentialroadblocksthatcouldaffectthescheduleIsthereanywaytomitigatethisHowlongwouldittaketoseeanimpact
Youranswers
Implementation
DowehavethestaffcapacitytoimplementthisArethereadditionalpartnersthatyoucanbringtothetabletosupportimplementationDoesthismakesensegivenourformofgovernment
Youranswers
Authority
DoesourcityhavetheauthoritytoenactthisIfsowhatistheappropriatemechanismAretherelegalbarriersArethereanyexistingpreemptionissuesfromthestateorfederalgovernmentCanthosebeaddressedthroughlegislationadvocacyIstherepoliticalsupportoroppositiontothis
Youranswers
Cost
HowmuchwillthiscostannuallyHowwillwepayforitCanthepolicybeimplementedwithexistingfunding
IfsoisthisfundingreliableHowwouldthispolicyoptionaffectourcitysbudgetintheshortandlongtermWhatotherfundingoptionscanweconsiderIsthereanalternativethatwouldcostlessandgiveussimilarresultsWillanyconstituencybeadverselyaffectediffundingforprogramsthatimpactthemisshiftedtoanotherprogrampolicyorinitiative
Youranswers
Decideonapolicytoimplementbasedonlocalfactors
32
EstablishaframeworktomonitorprogressNowthatyouhavereviewedtheevidenceconsideredlocalcontextandchosenyourpolicyitistimetodeterminehowyourcitywillmonitorprogressofthisnewapproachCitystaffandleadersneedtoestablishtheinfrastructureforongoingcontinuousdecisionmakingbasedonwhetherthepolicychangeisleadingtothedesiredoutcomesforresidentsandwhetherthechangeismeaningfulandrationaltofrontlineworkersSettingarbitrarygoalsatthelocallevelthatconflictwithgoalsatotherlevelsisconfusingtotheindividualsresponsibleforachievingthegoalsGoalsthataresettoohightoachievecanbedemoralizingtostaff
ThissectionincludesstepstosetupavenueinwhichCitystaffandleadershiproutinelyvisitkeyindicatorstodeterminewhatisorisnotworkingandtomakedecisionsinreal-timeFormorereadingonperformancemanagementpleasevisittheGovExPerformanceManagementGettingStartedGuide
InperformancemanagementtermssuchasindicatormeasureandmetricmaybeusedinterchangeablyInthisguidewewilluseindicatorInyourworkthemostimportantthingistobeconsistentinwhichevertermyouuseThisvideomaybehelpfulinunderstandingthisconceptandcommunicatingittoothers
Establishthefundamentalsandgatheryourteam
FirstdeterminewhetherthereisanestablishedvenueorregularmeetingscheduleduringwhichyoucandiscussprogressIfyourcityhasanexistingperformancemanagementstructureonarelatedtopicaddingadiscussionoftheprogressofpolicychangetoyouragendacanbeagreatideaIfthiswillbeastandalonemeetingdecidehowfrequentlytheprojectteamanddecisionmakerswillmeetandputthosemeetingsonthecalendarfortheentirelengthoftheproject
DeterminewhowillstaffandattendroutinemeetingsonCityprogressThesemeetingsmayincludetheMayorandorCityManagerdepartmentheadstaffwhoareimplementingthepolicychangeandanalysts
Identifyindicators
ItiscriticalthatindicatorsbeanimportantcityandcommunitypriorityaswellassomethingtheCityplaysamajorroleinaddressingTheremaybeinstancesinwhichtheCitywillneedtoreachouttoorcollaboratewithexternalpartnerstogetamorecompletepictureofprogressontheoverallpolicychange
Establishaframeworktomonitorprogress
33
Anoutcomeindicatordescribesthedesiredeffectofthetargetedpopulation
ExampleBellevuesfamilyfriendlyneighborhoodsindicatormeasuresthepercentofresidentswhobelievetheyliveinaneighborhoodthatsupportsfamiliesparticularlythosewithchildren
WithyourkeypartnersrevisityourproblemstatementcurrentstateandfuturestateexerciseandbegintoidentifytheoutcomeindicatorsthatwillillustratesuccessFornowchooseasingletop-levelkeyperformanceindicator(KPI)ItiscriticalthattheKPI-thepieceyouwillmeasure-beapriorityforthecommunityandsomethingtheCityplaysamajorroleinaddressing
Outputindicatorsmeasuretheprocessofprogramorservicedelivery
ExampleSanDiegoMayorKevinFaulconersgoaltorepair1000milesofstreetby2020
ReviewingtheadministrativedatasetinventorybegintoparedowntherelevantprogramsanddatasetsthatsupportyourKPIWhatarethemeasuresofeffectivenessforeachprogramTheseareyouroutputindicators
Thediagrambelowillustratestherelationshipbetweenoutputandoutcomeindicators
ThinkaboutimplementationindicatorsasmeasuringstepsintheprogramsrolloutthatarenecessarytoobtaindesiredchangesImplementationoutcomescanfocusontheinterventionstaffdeliveringtheinterventionanorganizationorsystemandcanaddressfeasibilityadaptabilitysustainabilityandcostTwokeyquestionstoconsiderwhendevelopingimplementationindicatorsare
1 Istheinterventionacceptableandappropriatetothecommunityand
2 IsitfeasibletoimplementinterventionXatthistimewiththeseresources
Establishaframeworktomonitorprogress
34
Amoreextensivelistwithdefinitionsofpotentialimplementationoutcomesislinkedhere
UsethetablebelowtochoosethedesiredimplementationoutcomesforyourpilotandtoplanyourmeasurementtechniqueThisshouldincludelevelofanalysis(individualproviderorconsumerorganizationorsetting)yourmeasurementtechnique(surveysobservationinterviewfocusgroupsadministrativedataanalysis)andthephasewhenyouwillcollectthisinformation(earlymidlate)Theremaybemorethanonelevelofanalysismeasurementtechniqueandphaseofdatacollection
Implementationoutcome
Definitionofoutcome
Levelofanalysisandmeasurement
Yourmeasurementtechnique
Phaseofpilotwhenyouwillcollectdata
Usethespacebelowtocaptureyourindicatorwishlist
Indicatorwishlist
Outcomes_____________________________________________________________________________
Outputs________________________________________________________________________________
Implementationoutcomes______________________________________________________________
Setapublichigh-levelgoal
Thehigh-levelgoaliscriticalinestablishingdirectiontogalvanizethepeoplewhowillbedoingtheworkWhendiscussingthegoalusetheMayorsbullypulpitLookforkeymilestonesinCitygovernmenttosharethegoalstatementsuchasthebeginningofaMayoraladministrationaStateoftheCityspeechorotherpublicremarks
GoalsshouldbeldquoSMARTrdquo
Specificaspossible(innamingthepopulationwhowillbebetterofftheactionthecitywilltake)
Measurable
Achievable
Establishaframeworktomonitorprogress
35
Results-focused
Time-bound
ThegoalstatementshouldincludeyouroutcomeindicatorlikeChattanoogasgoalldquoWewillendchronicveteranhomelessnessbyDecember2016rdquoAdiagramtohepyoubuildaSMARTgoalaroundyouroutcomeindicatorisbelow
Component Description Yourturn
Specific WhowillbebetteroffWhatactionthecitywilltake
Measurable Howwillyoumeasuresuccess
Achievable Isthegoalattainable
Results-focused Whatistheexpectedresult
Time-bound When
Setrealisticambitioustargetsandtimelinesforeachindicator
ToestablisharealisticandambitioustargetforyourgoalstatementusetheadministrativedatacollectedforyourchosenKPItoreviewactualCityperformanceinthemostrecentyearcomparedwiththethreeorfourmostrecentyearsworthofdataBegintocalculatesummarystatistics(egtotalminimumaveragemedianmaximum)toidentifytrendsCreatecumulativeandmonthlychartstoidentifytrendsandanomaliesBasedonyouranalysisofhistoricaldataignoretheanomaliesandassumetheproductivitytrendswillfollowthesameoverallpatternfrommonth-to-monthFinallyapplyanincreasedproductivityfactortotheperformanceeachmonthtoforecastfutureprogressBasedonthisanalysischooseatargetthatinspiresimprovementwhileaccountingforreality
Manycitiesthataremorematureintheirperformancemanagementpracticesrecognizetheimportanceoftakingacommunity-levelapproachtoimprovingoutcomesforresidentsThesecitiesarecreatinggoaltaxonomiesthatincludemeasuresrelatedtopublicsafetyhousingandevenrecreationCitiesdriveprogressonthesegoalsbyimplementingstrategiesthatengagemultipledepartmentsandcollaboratewithserviceprovidersinthecommunity
EventuallyyouwilldevelopgoalstatementsfortheoutputsinordertocapturetheefforttoachievetheoutcomecontainedinyourgoalstatementYoursetofindicatorsshouldbecomposedofshort-andinterim-termsupportivemeasureswithaccuratetargetsandrelevanttimelinesAnexamplegoaltaxonomyfromtheCityofLasVegassResultsVegasprogramisbelowUsethistaxonomyasamodelforyourownsuiteofindicatorsandgoalstatements
Establishaframeworktomonitorprogress
36
Yourturn
Thetemplatebelowismeanttohelpyoucraftagoalstatementforeachsupportingmeasure
Goalstatement
Departments
Outcome
Activities
KPI
Target
Timeframe
Conductanddiscussrelevantanalysis
ThrougharoutineseriesofmeetingscontinuallymonitorprogressofKPIssupportingindicatorsandanyotherdatacollectedsuchasqualitativedatatoidentifytrendsthatwillindicatewhetherthechosenpolicyishavingthedesiredimpactThesedatapointsshoulddrivethediscussionduringthesemeetingspointingtoopportunitiesforfurtherinquiryinto
Establishaframeworktomonitorprogress
37
whatisdrivingtheobservedtrendsAnalystsshouldbeencouragedandtrainedtoaskquestionsofthedepartmentsproducingthedataandconductsitevisitswhenpossibletodeepentheirunderstandingofthedataPresentationoftheanalysisshouldfocusononeideaatatimetokeeptheconversationfocusedonkeypointsandspecificopportunitiesforimprovementYoumayconsiderdevelopingapublicfacingdashboardtosharethisprogressmorebroadly
AgeneralcomparisonofthestatusquodatawithwhatwasobservedduringthepilotcanprovidedecisionmakerswithaquickunderstandingofthedifferencesbetweentheoldandnewapproachesalthoughthiswillnotshowcausalityOftenthesecomparisonswillincludeoperationalmetricssuchasdaysordollarssavedbutrememberthattheimpactonpeoplewillbemostimportantIfyouareworkingwithanevaluationpartnertheywillsupportthisstep
Usethespacebelowtosummarizetheresultsofyouranalysis
MeasureTargetvalue(seesectiononsettingtargetsabove)
Pilotvalue(whatdidyouobserveduringthepilot)
Historicalaverage(ifavailable)
Results(WhoisbetteroffIsanyonenegativelyimpacted1-2sentencesreassessingyourgoalstatement)
DocumentyourprocessandprogressincludingfeedbackfromtheparticipantsimplementationpartnersandtheprojectteamDidanythingunexpectedoccurduringthepilotArethereinternalandexternalfactorsthatmayaffectyouroutcomesThisiswheretheimplementationoutcomescanbecanhelpfulIftheinterventiondidnotgoasplannedconsiderwhetheritisbecausetheinterventionwasnotacceptableorfeasible
UsethetemplatebelowcreatedbyDenversPeakAcademytocapturethisinformationIfthe306090daytimelinedoesnotmakesenseforyourcityfeelfreetocustomizeit
CurrentState
FutureState
30Day
60Day
90day
Whatwentwell(+)
Whatdidnotgowellshouldbechanged(-)
IfaquestionordiscussionpointarisesthatcannotbeansweredinthemeetinganassignedstaffmembershouldcapturethequestionduringthemeetingAfterwardthefollowupitemsshouldbesenttorelevantpartnersandthenextmeetingshouldbeginwithadiscussionof
Establishaframeworktomonitorprogress
38
thoseitems
Takeaction
DiscussionsaboutwhetheryourindicatorsaremovingintherightdirectionandwhatmightbedrivingtrendsmayleadtoadiscoverythattherearenotenoughresourcesallocatedtothispolicyorissueTheremaybelimitedfundingforstaffprogrammingortechnologythatisdrivingthedatainthewrongdirectionIfCityleadershipisintheroomthisisagreattimetomakethecaseforadditionalfundingduringthebudgetseasonBycollectinganalyzinganddiscussingdataonthispolicyroutinelyyouarebuildingevidencetomakethecasefortherightleveloffundingtoreachtheestablishedtargets
Establishaframeworktomonitorprogress
39
Assessdataaccessibilityqualityandsecurity
Determineaccessibilityandqualityofdata
OnceyouhavecompiledyourwishlistofindicatorsitistimetodeterminetheaccessibilityandqualityofthedataTopreventcomplicationyoumaywishtoprioritizerelevantindicatorsforwhichyouarealreadycollectingdataAvoidthetemptationofadoptingatechnologysolutiontocollectnewdataatthispointsuchasaphotosurveytoolorotherappFornowyouaretryingtodeterminewhichdatasetsarerelevantaccessibleandhighqualityBesuretoworkwithyourkeyinternalpartnerswhileyouareidentifyingyourmeasuresDepartmentstaffwhoworkonprogramsarefamiliarwiththeindicatorsthatwillaccuratelysupporttheoutcomesanddatabaseadministratorswillbevaluablewhendiscussingaccessibilityandqualityofthedata
Abrieflistofconsiderationsfortheaccessibilityofyourdataisbelow
ArewecurrentlycollectingthisdatasetorwillwebecollectingitforthefirsttimeHowfarbackdoestherecordgo(timehorizon)WouldweneednewtechnologytocollectoranalyzethedataCanthedatabecollectedmanuallyorcanitbeautomatedHowfrequentlyisthedataproduced(continuouslyhourlydailyweeklymonthlyquarterlyyearlyetc)IsthedataavailablefortheentireCitypopulationorasubsetofthepopulationHowgranularisthedata(individuallevelCensusblockneighborhoodcitywideetc)DoesthedatacontainsensitiveinformationIsthereaproxymeasurethatwecanusefornowIsthedatamachinereadable
AsnotedbytheWilsonSheehanLabforEconomicOpportunities(LEO)attheUniversityofNotreDameleveragingadministrativedatacansignificantlyreducethecostsandtimelinesofrigorousevaluationCitiescanestablishaninternaldatasharingagreementacrossdepartmentsincludinglanguagethatenablesadministrativedatatobeusedforresearchpurposesCitiescanalsoidentifyexternaladministrativedata-holderssuchasacountyschooldistrictpolicedepartmenthospitaloranotherstatethatmayhaveoutcomedataimportantformeasuringprogramimpactCitiescanleverageexistingrelationshipsandorworkwithresearchinstitutionstodevelopstreamlineddatasharingprocessesandagreementswiththeseexternalentities
Assessdataaccessibilityqualityandsecurity
40
Evaluatethequalityofyourdata
QualityisasimportantasaccessibilityToevaluatedataqualityyoumaywishtocreatearubricliketheonebelow
Value ValueDescription
3 ExcellentDataQualityThedataiscompleteaccurateandupdatedregularly
2 AdequateDataQualityThedataissomewhatcompleteaccurateandupdatedfairlyregularly
1 PoorDataQualityThedataisnotcompleteaccurateorupdatedregularly
ThisisagreattimetobuildonthedatainventorythatyoustartedinthegatherinformationpageSuggestedadditionalfieldstoaddtoyourdatainventorytemplatearebelow
Program DatasetTitle
DatasetDescription DataSource
Departmentorexternalpartner
DoestheCityalreadycollectthisdata
Whatisthenameoftheprogram
Human-readablenameofthedataset
Shouldbeunderstandablebynon-technicalusers
Whatistheoriginalsystemapplicationfilethathousesit
Departmentorexternalpartnerthatmaintainsthedata
Isthisdataalreadycollected
Updatefrequency-ThefileformatforthedatasetTypicallycsvxlsxshapefiletxtetc
Format-Howoftenthedatasetisupdated
Timerange-Theperiodoftimecoveredinthisparticulardataset
Datause-Howisthisdatasetcommonlyused
Datausers-Whotypicallyusesthisdataset
Dataownerstitle-Titleofthedataowner
Dataownersposition-DataOwnersjobtitle
Dataownersemailaddress-DataOwnersemailaddressWhomanagesthedataandorisresponsibleforgrantingpermissiontoaccessthedataWhounderstandswhatthedatasetincludesandcananswerquestionsaboutit
Dataqualityconcerns-Describetheconcernsyoumayhaveaboutthequalityofthedataset
Assessdataaccessibilityqualityandsecurity
41
Division-DivisionwherethedataismaintainedIfmorethanonedivisionisresponsiblelisttheprimaryownerofthedataset
Sensitivedatacomments-DescribetheissuesthatwouldpreventthecityfromsharingthedatapubliclyCommonsensitivityconcernsincludeprivacyviolationssecurityissuesorhighcostorstaffdemandstopostthedata
Publishingstatus-Thistheplacetoputanycommentsregardingthecurrentstatusofthedataset
Publishedlink-Thisisthelinktothepublisheddatasetontheopendataportalorcitywebsite
Reviewdatasecurityandprivacyprotectionprotocols
OftenapolicychangealsoinvolvesachangeinthedatathatisbeingcollectedhowthatdataisbeinghandledwhohasaccesstoitandahostofotheradjustmentsAsyoutakestockofthedataassetsthatareavailableforyourinitiativeitisvitaltoalsothinkthroughwhatwillbedonetoprotectthatdataandtheprivacyofthepeopleitrepresentsAbigpartofgooddatasecurityinvolvesstrongdatagovernancepracticesthatdefineandidentifysensitivedataandconsciouslyworktobalancethevalueandriskofstoringsuchsensitivedataOnewayofevaluatingyourdatacomesfromcaselawrelatedtotheUSFreedomofInformationActwhichwehavetranslatedtothechecklistbelow
1 Woulddisclosureresultinasubstantialinvasionofprivacy2 Haveweconsideredtheextentorvalueofthepublicinterestandthepurposeorobject
oftheindividualsseekingdisclosure3 Istheinformationavailablefromothersources4 Wastheinformationgivenwithanexpectationofconfidentiality5 Isitpossibletomouldreliefsoastolimittheinvasionofindividualprivacy
TheseriskscouldbemitigatedusingavarietyofstrategiesfromaggregatingdatatoobfuscatingindividualstoredactingfieldsthatmayincludePersonallyIdentifiableInformation(PII)orre-evaluatingyourdatacollectionmethodstofindalternativewaysofgettingtheinformationyouneedThesestepsshouldbecodifiedandmanagedbyadatagovernancecommitteeifpossibleForthesensitivedatathatisdeemedessentialcreateaplanaroundpotentialdatabreachesinordertoquicklyandconfidentlyrespondtoemergencysituations
ProactivelypreparingforthesesituationsisoftenthebeststrategyfordatasecuritybutdonotforgetthetechnicalsideInvolveyourITadministratorsinconversationsaboutdatastorageandaccessensuringthattheywillbeprotectedbyenterprise-gradecybersecuritysystems
Assessdataaccessibilityqualityandsecurity
42
Assessdataaccessibilityqualityandsecurity
43
Designimplementationandevaluation
Designtheimplementation
NowthatyouhaveyouraccountabilityplanandkeyindicatorsitistimetogetspecificaboutyourplanforimplementationThetemplatebelowwillhelpyouthinkthroughthespecificmechanismsofimplementingpolicychange
UnderstandingorganizationsandsystemsiscriticaltounderstandingandmakingdecisionsaboutwhereandhowinterventionsshouldbeappliedApolicyimplementedinasinglesettingmayshowpromisingresultsbutimplementingmultiplepoliciesandprogramsatthesametimewithdifferentactorsresponsibleforeachaspectcanhelpensuresustainablebehaviorchange
InSomervilleMAShapeUpSomerville(SUS)isacitywidepublichealthprogramthatisimplementedacrossmultiplelevelsofinfluenceandsystemsthroughinterventionsbeforeduringandafterschooltoincludechangestobuiltenvironmentsuchasbikelanesregionalmasstransportationandphysicaleducationequipmentinschoolsandgymstoincreasephysicalactivitywellnesspoliciesandschoolgardensnutritioneducationduringandafterschoolandhealthyrestaurantprogramstosupporthealthyfoodandhealthyeatinghabits
Usethetemplatebelowtoplanyourimplementation
Policyimplementationplanningtemplate
ListthespecificdesiredoutcomesInthepreviousstepyouworkedwithdepartmentstafftoestablishspecificmeasurableoutcomesandinterimindicatorsPrioritizetheindicatorsthatwilldemonstrateprogressUsethespacebelowtocapturethedesiredoutcomesofthepolicychange
ExampleShapeupSomervillebeganwiththegoalofpreventingobesityin1stthrough3rdgraders
Yourturn
____________________________________________________________________________
____________________________________________________________________________
Designimplementationandevaluation
44
SpecifythetargetpopulationAreyoudesigninganinterventionthattargetsonlycertainsitesAreyouplanningonimplementingaprogramorinitiativeforaspecifictargetaudienceClearlyspecifythepopulationyouplantotarget
IfyouplantoconductarigorousimpactevaluationasnotedbytheWilsonSheehanLabforEconomicOpportunitiesitisimportanttoestablishconcreteobjectiveeligibilitycriteriaandensurethatthesecriteriaareappliedconsistentlybyallstaffduringscreeningandintakeItisalsovitaltoidentifytheindividualswhoareeligibleforaparticularprogramorserviceevenifeveryoneeligiblecannotbeservedbyexistingresourcesThishelpscitiesidentifyanyunmetneedsandguidefutureresourceallocationwhilealsocreatingthepossibilityofarigorousimpactevaluationbyidentifyingapotentialcomparisongroupFinallywhenconductingscreeningandintakeitisimperativetocollectandmaintainbasicinformation(egnamedateofbirthgender)aboutallprogram-eligibleindividualsnotjustthosewhoparticipateintheprogram
ExampleIntheShapeUpSomervilleexampletheprogrambeganbytargeting1stthrough3rdgraders
Yourturn
____________________________________________________________________________
____________________________________________________________________________
DefinethelocationWherewillthepolicychangetakeplaceWillimplementationtakeplaceatonesiteormultiplesites
ExampleShapeUpSomervilleinterventionsweredeliveredinschools
Yourturn
____________________________________________________________________________
____________________________________________________________________________
ClarifyimplementationpartnersandarticulatetheirrolesWhatarethekeyorganizationsinstitutionsorpeopleresponsibleforimplementationTheprojectteamshouldbeasclearandspecificaspossibleinprovidingadescriptionoftheconcreteactivitiesthatimplementationpartnersaremeanttocarryoutPracticeprofilesmaybeahelpfultooltohelpdefinewhatisrequiredoftheactorswhoareresponsibleforimplementingtheintervention
Designimplementationandevaluation
45
ExampleInSomervillemayoralleadershipandengagementofmultiplecityagencies(publichealthprimarycareschoolsCambridgeHealthAllianceetc)streamlinedandconnectedthevariousShapeUpSomervilleinitiativesandgrantswhichhelpedtokeepthetalentandenergyfocusedonhealthyeatingandactivelivingforallsectorsofSomervilleSUSpartnersultimatelyincludedschoolsnonprofitsCityandstateagenciesandnationalfoundations
Yourturn
____________________________________________________________________________
____________________________________________________________________________
DescribetheinterventionWhatarethestepsactivitiesorprocessesRememberthatthemechanismthespecificstepsorprocessthatleadtobehaviorchangeiskeytounderstandingtheeffectivenessofthepolicychangeandsharingtheresultswithothers
ExampleGoalsofShapeUpSomervilleweretoinfluenceeverypartofanearlyelementaryschoolchildrsquosdayndashbeforeduringandafter-schooltopromotehealthyeatingphysicalactivityandtheirweightstatusInterventionincludedfoodserviceenhancementstoimprovebreakfastandlunchandstaffprofessionaldevelopmentwalktoschoolactivitiesSUSclassroomcurriculumandprofessionaldevelopmentschoolwellnesspolicydevelopmentSUSafter-schoolcurriculumandprofessionaldevelopmentoutreachandeducationtohomethroughmaterialsforumseventsldquoSUSapprovedrdquorestaurantscommunityoutreachandcapacitybuildingthroughpolicydevelopmenttrainingsmediaplacements
Yourturn
____________________________________________________________________________
____________________________________________________________________________
ClarifytimelineandmilestonesfordeliveryandevaluationHowfrequentlywilltheservicesinterventionbedeliveredHowlongwilltheprogramrunWhatisthetimetoexpectedimpacts(eg12monthsafterprogramcompletion)
Designimplementationandevaluation
46
ExampleShapeUpSomervilleledbyTuftsUniversityFriedmanSchoolofNutritionScienceandPolicyconductedaquasi-experimental(nonrandomized)interventionstudyamong1stthrough3rdgradestudentsinSomerville(intervention)andtwocomparisoncommunitiesfor9-monthsduringoneschoolyear(fall2002ndashspring2003)ThestudyshowedpositiveassociationoftheinterventiononchangeinBMISUSreleasedreportssummarizingevaluationsprogresstodateplansforadditionalprojectsandimpactstenandfifteenyearsintotheinitiative
Yourturn
____________________________________________________________________________
____________________________________________________________________________
PlanyouranalysisTheprevioussectionaddressedestablishinganaccountabilityframeworktomonitorprogressandimplementationDeterminewhichindicatorswillbediscussthroughthatprocesswhowillsharethedataandanalysisandhowfeedbackfromthoseconversationswillbeincorporatedbytheprojectteam
ExampleTheMayorrsquosofficeimplementedadata-drivendecisionmakingapproachcalledSomerStatwhichwassynergisticwiththeapproachestakenbytheresearchandevaluationteamsdeployedforthespecificgrants
Yourturn
____________________________________________________________________________
____________________________________________________________________________
PlanforsustainabilityIfthepolicyandprogramchangeissuccessfulhowwilltheprojectteamsustainandexpandtoadditionalsitesorpopulations
ExampleAlthoughmostofthepartnershipsforgedintheearlyyearsweremaintainednewgrantsandcollaborationsexpandedattentiontoolderpublicschoolstudentsactivetransportationandtheGreenLinesubwayextensionthebuiltenvironmentfarmerrsquosmarketeffortssuchasldquoGrowninSomervillerdquowithUnionMainStreeteconomicdevelopmentandhealthdisparitiesaswellasdatacollectionmonitoringandsurveillance
Yourturn
Designimplementationandevaluation
47
____________________________________________________________________________
____________________________________________________________________________
Pilotplanningtemplate
IfyouplantoconductapilotfirstdeterminethepurposeofthepilotDoyouwanttoobservewhetherthepolicychangeresultsinthedesiredbehaviorchangeAreyouconcernedaboutfeasibilityscalabilityorcostDoyouwanttotestwhetheryourdesiredresultscanbeachievedthroughthispolicyapproachClarifythegoalofthepilotandthequestionsyouhopetoanswerThiswillhelpyouidentifythekeyindicatorsthatyouwillusetomeasurewhetherthepilotissuccessfulBeforebeginningestablishabaselineofcurrentperformanceandsetyourtargetsaccordinglybasedonwhatyouhopetoaccomplishthroughthepilot
NextestablishthepilotplanforyourprojectteamincludingmilestonesforteamcheckinstodeterminewhethertomoveaheadorchangecourseThisisseparatefromthemeetingsthatwilloccuraspartoftheaccountabilityframeworkUsethespacebelowtocreateyourtimelineanddevelopyourmeetingschedule
Task Personresponsible Startdate Checkindate Deliverable
Planyourevaluation
WhileyouaredesigningtheimplementationitisalsotimetobegintodevelopyourplanforanalysisAsnotedbytheWilsonSheehanLabforEconomicOpportunitiesdecisionsaboutprogramsetup-includingoutreachscreeningintakedatatrackingandenrollment-candeterminewhetherornotarigorousimpactevaluationwillbepossibleAsdiscussedcityleadersmayfindithelpfultoincludeevaluationpartnersinearlyconversationsaboutprogramsetupDuringpreviousstagesyoumayhaveworkedwitharesearchertoplanyourproposedpolicychangeYoushouldbeclearaboutthepurposeoftheevaluationanddevelopanevaluationplanBelowareadditionalconsiderations
Causationvstrendanalysis
IntheEstablishaframeworktomonitorprogressstagetherewasguidanceontrackingtrendsovertimeItisimportanttonoteherethatmonitoringdataisdifferentfromanalyzingacausaleffectTrendanalysiswillillustratethatchangesareoccurringandmayleadtoquestionsthatcouldbeansweredbyyourevaluationbuttrendanalysisalonedoesnotprovideevidenceaboutwhatisdrivingthosechangesYoumayseechangesinyour
Designimplementationandevaluation
48
outcomeindicatorsbasedonvariablesoutsideoftheCityscontrolForinstanceachangeinfederaltradepolicycanhaveanimpactemploymentratesinyourCityoutsideofyourCitysnewjobtrainingandplacementprogram
ToaddresscausalityyoumustcomparetheoutcomesofindividualswhoreceivedtheinterventionwiththosewhodidnotWithoutacomparisongroupofsimilarindividualswhowerenotexposedtotheinterventionitisimpossibletoknowwhichfactorsareresponsiblefortheoutcomesRandomizedControlTrials(RCTs)inwhichindividualsarerandomlyassignedtotreatmentandcontrolgroupsarethegoldstandardinprovingcausalityIfyoucandemonstratethatoutcomesimprovedforthosewhoreceivedtheinterventionanddidnotimproveatallorimprovedlessforacomparisongroupyoucanbesomewhatcertainthattheinterventionwasthecause
Examplesofothernon-causalimpactevaluationsthatarerigorousandreliableincludecertainquasi-experimentalstudiesusingcomparisongroupdesignssmartinterruptedtimeseriesanalysesthatmodeltrendsovertimeinwaysthatallowestimationofinterventioneffectswelldesignedpre-posttestanalysesneedsassessmentsimplementationstudiesetcFormoreinformationondifferenttypesofevaluationsseetheSageResearchMethodswebsite
Lookfordifferencesingroups
BuildintimetoalsoexaminethepilotdatatoseeifgroupsareexperiencingdifferentresultsReviewingperformancebasedonoutcomemeasuresforapopulationinaggregatemayhidethesubtlewaysthatsomepeoplemaybeaffecteddifferentlysoitisnecessarytocarefullyanalyzethedatafordisparateoutcomesForinstancetheMovingtoOpportunity(MTO)studythatexaminedtheimpactofhousingvouchersonfamilieswholivedinhigh-povertyneighborhoodsfoundthatchildrenwhomovedbeforetheageof13hadbettereducationandhigherincomesasadultsHoweveradultsdidnotseeimprovededucationandemploymentoutcomes
Disaggregatingdataintomorespecificcategoriesofraceandethnicitycanhelpunderstandtrendsanddisparitiesinamoregranularwaywhichcanhelpdecisionmakerstargetpoliciesandinterventionsinamoreequitableway
EarlierintheRoadmapweworkedtoidentifystakeholdersandwhomightbepositivelyandnegativelyimpactedbyachangeinpolicyUsetheresultsofthatexercisetoinvolvethecommunityininterpretingdataanalysisandtogatherqualitativefeedbackfromthecommunitytocallattentiontoissuesForinstanceanincreaseinyouthdetentionmaybeexplainedbyinefficienciesorlackofaccesstotransportation
Designimplementationandevaluation
49
Designimplementationandevaluation
50
Shareyourstoryandsuccess
Sharetheresults
Storytelling
AnintegralstepinyourstrategyissharingtheresultsofyourpilotorpolicychangewithyourcommunityThisincludesdevelopingmessagingaroundtheworkofyourorganizationandcreatingaplantodisseminateyourstoriesWhytellstoriesaboutyourwork
Informandorunderstandasubjectorsituation
Giveneededcontextofdataorbitsofinformation
Relatablestoriesarememorableforreadersviewers
ReadersorviewerswillseetheinformationasmorecompellingwhenthereisademonstratedandrelatablechangeorimpactonthemselvesortheirfriendsneighborsandcolleaguesStoriesareaboutpeopleandgovernmentisfundamentallyabouthelpingpeopleSotellthestoriesofthepeopleyouandyourcitycolleagueshelped
Keepthesetipsinmindasyoudevelopyourstorywithyourteam
WhatwereyourfindingsandwhyshouldanyonecareTellyouraudiencehowtheproblemwassolvedWhathastheimpactbeenonresidentsandcitystafftodateWhatwasthesuccessandweretheremultiplesuccessesalongthewayIfyouhavepreliminaryresultsordatashowingimprovementtosharethisisthetimetodoit
WhowasinvolvedandimpactedReviewthelistofstakeholdersHowwereeachofthemimpactedbytheprogramHumanizeyourstoryandimpactbyaddingadditionalvoicesThesevoicescanbeoneofthefollowing
Apersonorfamilywhowaspositivelyaffectedbythechangesuccess
Aninvolvedthird-partydescribingboththeproblemandthesolution(ieanorganizationnonprofitprovidersbusinesstradeassociationsportsteametc)whoispartofthecommunitythatthegovernmentworkedwithonthechangesuccess
Someonefromthegovernmentitselfwhocanspeakaboutthechangesuccess
TellmoreaboutyourcityDescribeyourcityincludingpopulationdemographicsindustriesetcWhatareitschallengesWhatisitknownforWhatmakesituniqueHowisitsimilarordifferentfromothercitiesintheUS
Shareyourstoryandsuccess
51
DescribeyourcityrsquoschallengeWriteoutthechallengeyourcityisfacing(asdefinedinyourproblemstatement)HowdoesitaffectresidentsHowdoesthisproblemaffecttheworkofcitystaffmembers
AddthedetailsProvidedetailsonyourappliedresearchstrategyTalkaboutwhatyourcityhasdonenthepasttoresolvetheissueDiscusshowyourstrategyhaschangedwhenyoubegantoadoptthisapproachWasmoneysavedorinvested
Usethisstorymapexercisetoorganizeyourthoughtsandguaranteethatyouincludealloftheelementsofatraditionalstory
AfteryoudevelopyourmessagingcreateacommunicationsmarketingandoutreachstrategydesignedtospreadyourmessageYourstrategyshouldwalkyouthroughthefollowingkeyactivities
1 IdentifyandprofileyouraudienceWhencreatinganddisseminatingyourmessageitisalwaysgoodtothinkaboutyouraudienceWhoexactlywouldbeinterestedinwhatyouhavetosayThegeneralpopulationofthecityPeoplewhoworkinnonprofitsBusinessownersItmattersbecausenoteveryonewholivesinacitymaybethetargetoftheinformationBusinessownersmaybemoreinterestedindatashowingeffectsonthebusinessdistrictthanothersandsoonWewillcoverthislaterbutthisstepwillalsohelpyoutothinkofthemosteffectivewaytoreachyouraudience
2 Defineyourcalltoaction(ifyouhaveone)SpendsometimeconsideringwhyyouareinterestedintellingyourstoryDoyousimplywanttoupdatethepubliconyourprogressorcelebrateyoursuccessesAreyouinterestedinyouraudiencechangingitsbehaviororsupportingaspecificinitiativebeinglaunchedbyyourorganizationIfyouwanttohaveyouraudienceactorbehaveinacertainwaybespecificandincorporateitinyourmessaging
Shareyourstoryandsuccess
52
3 SelectperformancemeasuresThinkabouthowyouwoulddetermineifyourcampaignissuccessfulandselectappropriateperformancemeasuresYouwillrevisitthesemeasuresattheendofyourcampaigncommunicationsplanimplementationtodeterminewhetheryouneedtomodifyyourcommunicationsandoutreachstrategyforthefuture
4 MethodstocommunicateyourstoryTherearemanywaystocommunicatesuccessofyourprojectHerearesomesuggestionsonreachingaudiencesItisimportanttodeterminewhichmethodsarethemosteffectiveforyourdesiredaudiencetoreceiveinformationDoyourresearch(PewResearchCenterhasexcellentstudiesonthereachofmediadigitalmediaandmarketing)ForexampleifseniorsareyourtargetpopulationandtheyarelessinclinedtouseInstagram(thatistrue)youmaywanttoconsiderFacebookorthenightlylocalnewsbothofwhichareusedmoreoftenbyseniorsBelowisalistofsomemethodsandtoolsyoucanuse
i Pressreleasestolocalnewspapersandtvstations
ii ShortsmartphonevideosAgreatwaytoengageanaudienceisthroughvideoAskonequestionofsomeonewhoispartofthestoryMaybethereisasocialworkerwhocandiscusshungerinyourcityThesevideoscanbereusedonmultipleplatforms
iii Long-formwriting
iv TellingindividualstoriesthroughaseriesofphotosandshortnarrativesforpublicationonFacebookorInstagram
v Op-edsopinionpiecesplacedinthelocalpaper
vi Emailupdatestoyourlist
vii Connectingwithappropriateorganizationsthatareinvestedinthetopicandworkwiththecommunityandaskthemtodisseminateinformationtotheirmembers
viii EmployingldquoWordofMouthrdquo(WOM)tacticsbycreatingtownhallsorfreeeventsforresidentstoattendintheirownneighborhoods
5 DeterminethetypeofcollateralUsuallycampaignsthattargetresidentsrequireseveralavenuestoreachthemPeoplearenotalwaysinthesameplaceorspaceHavingakeenunderstandingofyouraudiencewillassistyouwithyourchoicesonwhatneedstobedevelopedSometimesthatmeansimagesforsocialmediaaonepageflierthatgivespeopleinformationonattendinganeventoranin-depthbrochure
Shareyourstoryandsuccess
53
6 ImplementyourstrategyKeepinmindstrategiescanbefluidSometimessomethingmaynotturnoutasexpectedevenifduringplanningsessionsitwasldquosupposedtordquoItisokYoucananalyzewhathappenedduringadebriefsessionandonceyoursurveysandothermeasurementshavebeenfinalized
Visualizations
Withtherightvisualizationyoucanreviewandshareinformationinsightsanalysisperformanceandtrendsallofwhichisvaluabletoacross-sectionofthepublicButwhichvisualizationisrightforyouTochoosetherighttypeofchartgraphdeterminethetypeofquestionyouwanttoanswerSomecommoncharttypesarebelowandmoreinformationcanbefoundhere
Exploreadditionalanalyticaltechniques
Duringthegoal-settingphaseyouestablishedoutputoutcomeandimplementationmeasuresBasedontheresultsofyourpilotyoumayneedtoadjustyourmeasuresWiththepilotdatainhandyoumayhavesomeideasofhowfutureanalysescouldbedifferentandpotentiallyimprovethequalityofyourprogramTakesometimetobrainstormotherrelevantdatasetsthatyoudidnothavethetimetopursueearlierincludingmessydepartmentaldatathatneedsmoreattentionorsourcesofpublicdatafromthestateandfederalgovernmentnonprofitsandevenprivatecompaniesthatcouldsupplementyourunderstandingofanissueUsingmachinelearningtechniquesyoumightbeabletobetter
Shareyourstoryandsuccess
54
predictdemandforhomelessservicesusingNOAAweatherforecastspreemptwatermainbreaksusingsoilstratigraphyorinvestineconomicdevelopmentbymonitoringrealestatelistingdata
Advancedanalyticaltechniquescanalsohelpyourimplementationrunmoreeffectivelybyintegratingdata-informedtargetinganddecisiontoolsintotheprocessByautomatingroutineworkflowsandsurfacingsubtletrendsthesetoolscanhelpyouiterateinamoreprecisemannerandprovidekeystrategicguidancetofront-lineemployeesthatwouldotherwisebeunavailableThesesortsofdata-science-enhancedprocessescanbeclassifiedasoneofsixprojecttypesasexploredfurtherbytheNewOrleansLANOLAlyticsteamFormoreguidanceonscopingasimilarprojecttheGovExadvancedanalyticsscopingguidegivesyouastep-by-stepoverviewofhowtobesuccessful
Usethespacebelowtolistyourideasforpossibleadvancedanalyticsprojectsbytype
Typeofproject Ideas
Findingtheneedleinthehaystack
Prioritizingforimpact
Earlywarningtools
Betterquickerdecisions
Optimizingresourceallocation
Experimentingforwhatworks
Keepthefeedbackloopgoing
GatherinformationonpublicsentimentaboutpolicychangethroughfocusgroupssurveysandpublicmeetingsledbyadvisorygroupmembersNotewhethernewinformationisavailablethatcanhelpanalyzeorcontextualizeyourprogress(egfederalorstatedatareports)Sharetheresultsofyourpilotandthepolicychangeprocesswithothercitiesandresearchers
Shareyourstoryandsuccess
55
WorkingwitharangeofevidencePresentlythereisalotofattentionandinvestmentinevidence-basedpolicymakingincludingthefederalbipartisanCommissiononEvidence-BasedPolicymakingthePewCharitableTrustsreportsandtoolkitsonHowStatesEngageinEvidence-BasedPolicymakingandtheLauraandJohnArnoldFoundationsupportedEvidence-BasedPolicymakingCollaborativeManyoftheseinitiativesfocusonrigorousevaluation--meaningrandomizedcontrolledtrials(RCT)orotherexperimentalmodels--asthebestkindofevidencetouseHoweverthisbodyofevidenceisofteninsufficientforthespecifickindsofquestionsthatgovernmentscanhavewhenmakingadecisionaboutpolicyInthisguideweproposeanapproachtosupportgovernmentsinincorporatingadditionaltypesofevidenceintotheirdecisionmaking
OnewaytounderstandarangeofevidenceistouseNestarsquosStandardsofEvidencetoframeanddefineevidenceTheBritishnonprofitNestaworkstoinjectpositivechangeintothewayUKgovernmentpoliciesandprogramsfunctionOneoftheirkeyareasofworkhasbeendevelopingWhatWorksCentresorganizationsthatcollectcommissionsynthesizeanddisseminateevidenceaboutinterventioneffectivenessacrossarangeofsocialimpactareasNestadevelopedthefollowingStandardsofEvidencetoprovideausefulwaytodocumenthowdifferentkindsofinformationdemonstratedifferentlevelsofevidence
Workingwitharangeofevidence
56
Theimportanceofpotentiallybiasedinformation
Onechallengepresentedbyanevidenceclassificationthatacknowledgesthevalueofnon-scientificformsofevidenceistofigureouthowtobalanceallofthepiecestogetherwhenyouusethemTounderstandhowtodothiswehavetothinkabouthowtopositionpotentiallybiasedinformationindecisionmaking
WhenwelookatinformationwhichhasnotbeenrigorouslyevaluatedweacceptthatthisinformationmightbebiasedThewholereasonweconductrigorousevaluationistoremovebiasfromourobservationsWeconductstructuredtrialstoproduceresultsthatareconsistentandgeneralizableIfafindingisunbiasedthisaccuracymdashthislackofbiasmdashcanbeprovedbecauseanyindependenttestercanreproduceitwhentheyfollowanidenticalprocess
YoucanlookatpotentiallybiasedinformationasjustinformationthatisearlieroninitspathtorefinementreplicationandverifiedaccuracyApieceofinformationbasedonanyoneindividualrsquosobservationsislikelytobebiasedbecauseitiscomingfromjustonesourceBiasisreducedbygatheringtogetherlotsofindependentsourcesofinformationandfiguringoutitsmostcommonattributes
MaybethebiasedinformationmakesaclaimthatistoolargemdashitsayssomethingistrueunderallconditionswhenitisreallyonlytrueundercertainconditionsorthataneffectofaninterventionisbiggerthanitstrueaverageorthataneffectlastsovertimewhenitdoesnotPerhapschangingtheclaimtobemorespecificmdashcriticallyreviewingtheinformationtoseeifspecificconditionsmustbemettoachievetheoutcomesmdashwouldmaketheclaimmoreaccurate
IfyoutestedyourclaimfounditdidnotholdupinsomecaseschangedittesteditagainuntilyourclaimrepeatedlyheldupthenyoucoulddemonstratethattheclaimwasnolongerbiasedbutaccurateThisprocessoftestingimprovingtestingandvalidatingiswhatwegetfromstructuredscientificstudy
Withindividualpiecesofinformation--crediblehypothesescasestudiesexpertopinion--youcannotassumethatkindofaccuracyAtthesametimetheseearlyobservationsarefoundationaltoanyreplicablescientificobservationsWewillnevergetanywhereifwedonotstartwiththesepiecesofinformationeveniftheymightbebiased
Combininglevelsofevidencetogetthemostdetailedinformationforyourcity
ItisespeciallyimportanttoacceptdifferentformsofevidencewhenyoufacequestionsthathavenotyetbeentestedbyindependentstudiesSomeofthemostimportantquestionsrelatetothingsthatareveryspecifictoyoursituationYoufaceaparticularcombinationof
Workingwitharangeofevidence
57
issuesactorstrendsandresourcesthatisultimatelydifferentfromeveryotherplaceintheworld
ThisisimportantbecauseaspolicypractitionersandnotscientistsyoucaremoreaboutyourspecificcasethanahypotheticalaveragegeneralcaseYoucarelessaboutthetotallikelyrangeofimpactsfromapolicythanhowthissingleimplementationwillworkYouwantinformationaboutpoliciesthatwillprovideyouwiththebestpossiblepredictionabouthowtheywillworkinyourcontext
ByputtingdifferentkindsofevidenceintothesameframeworkNestarsquosStandardsofEvidenceimplythatwhenyoucannotgetrigorouslytestedexperimentallyverifiedinformationitisappropriatetoworkwithwhatwedohaveThisisrelevantforthemunicipalimplementationcontextbecausenostudypulledfromaliteraturereviewwillaccountforallofthevariablesyoufaceinyourowncontext
Intheabsenceofhavingastudythatpredictshowaninterventionworksinyourspecificcontextyouwillneednon-scientificinformationtotrytofillinthosegapsYouwillneedtogatherstoriesfromresidentsstaffmembersandotherstakeholdersthatsitonLevels1or2ofthestandardsofevidence
Someofthekindsofinformationyouwillprobablygetmoreeasilyfrompeoplethanstudiesinclude
howpeopleincitieslikeyoursreactedtoaninterventioninpublicforumshowpeoplewhoimplementedaninterventioninacitylikeyoursfiguredouthowtogetitpassedhowthecitycommunicatedpubliclyabouttheapparentbenefitsanddrawbacksoftheinterventionthepoliticalacceptabilityoftheintervention
Youcanremainsimultaneouslyawarethattheinformationyouaregatheringthroughindividualstoriesisbiased--notgeneralizableinformedbyspecificperspectives--whilealsorecognizingthathavingsomeinformationaboutsomethingimportantisbetterthannoinformationEvidencefromthefirsttwoevidencelevelsareessentialbuildingblocksofknowledgeandperfectlyusefulwhenyoucannotgetdetailedunbiasedknowledge
Youhavetostartsomewhereandyoushouldalwaysstartwithevidenceyouhaveaboutyourveryspecificconditionsthatyouknowareimportantforsuccess
Workingwitharangeofevidence
58
Thankyou
Interviewswithcities
GovExheldsemi-structuredinterviewswithsixstaffmembersinfourcitiestoestablishtheinitialstepsoftheRoadmapWeaskedrespondentstotalkaboutaninitiativeorpolicythattheyhavebeenresponsibleforimplementingintheircityRespondentsfromthreecitiesspokeabouttheireffortstoaddresshomelessnessandrespondentsfromonecityspokeaboutblightAfterrespondentsidentifiedtheproblemtheyspokeabouthowthecityprioritizedthisissuethevariousstepsinvolvedandbarrierspresent
GovExconductedtextanalysisoftheseinterviewswhichrevealedthatpeoplewasthemostcommonlyusedtermduringtheseconversationswhendiscussingthestepstochangeaprogramorpolicyThisisathemethroughoutthisRoadmapThefirstsectionbeginswithidentifyingtherelevantpeopleTheconversationsaroundevidencearefocusedonhumanbehaviorratherthanmethodologyorstrengthofevidence
Thankyou
59
OncewehadadraftoftheRoadmapwefollowedupwithcitiesonanindividualbasistolearnmoreaboutwhatwouldbehelpfulWearegratefultothefollowingcitystafffortheinterviewsandindividualconversations
MichaelBaskinChiefPolicyOfficerChattanoogaTN
MelissaBridgesPerformanceandInnovationCoordinatorLittleRockAR
VictoriaCarreoacutenAdministrativeOfficerLasVegasNV
TyroneGrandstrandCommunityDevelopmentPlannerFargoND
TibLaughlinDirectorofGeneralServicesKansasCityKS
TimMorelandDirectorofPerformanceManagementandOpenDataChattanoogaTN
CharlesRosendahlManagementAnalystDentonTX
Thankyou
60
MelissaSiebenAssistantCountyAdministratorKansasCityKS
GinaThompsonDirectorofStrategicServicesLewisvilleTX
JHUFacultycollaboration
GovExalsoconvenedthecitycontactsandfacultypartnersinaseriesofAdvisoryGroupmeetingsThesemeetingsservedtoconnectcitiesandfacultytounderstandoneanothersperspective
StefanBaralBloombergSchoolofPublicHealth
AshleyBernerJHUSchoolofEducation
MelissaDavey-RothwellBloombergSchoolofPublicHealth
JonLinksBloombergSchoolofPublicHealth
TamarMendelsonBloombergSchoolofPublicHealth
KristinMmariBloombergSchoolofPublicHealth
KeshiaPollackPorterBloombergSchoolofPublicHealth
JoshSharfsteinBloombergSchoolofPublicHealth
LizStuartBloombergSchoolofPublicHealth
LindsayThompsonCareyBusinessSchool
DanielWebsterBloombergSchoolofPublicHealth
Partnersindataandevidence-basedpolicy
Wealsospokewithpartnerswhoareexpertsinthefieldofevidence-basedpolicywhosharedtheirknowledgeandresourcesofcurrentandfuturethinkinginevidence-informedthinking
WendyBarrenoAssociateDirectorWilsonSheehanLabforEconomicOpportunitiesUniversityofNotreDame
TiffanyDavenportDirectorofDataPartnershipsBaltimoresPromise
TomiHiersDirectorBaltimoreCivicSiteAnnieECaseyFoundation
ChrisKingsleySeniorAssociateAnnieECaseyFoundation
KathrynOliverAssociateProfessorLondonSchoolofHygieneandTropicalMedicine
Thankyou
61
KristenWatkinsSeniorPolicyAssociateJ-PAL
GovExParticipants
GovExAnalystMiriamMcKinneyanalyzedtheinterviewtranscriptsandconductedtextanalysisSeniorImplementationAdvisorEmilyShawwillintegrateRoadmapactivitieswithadditionalservicedeliveryactivitiesatGovEx
Thankyou
62
ReferencesAbrahamKGampHaskinsR(2017)Thepromiseofevidence-basedpolicymakingReportofthecommissiononevidence-basedpolicymakingWashingtonDCCommissiononEvidence-basedPolicymakingBigmove2Extendingthegreen(nd)Retrievedfromhttpswwwcityoflewisvillecomdoing-businesslewisville-2025big-move-2-extending-the-greenBatesMAampGlennersterR(2017)ThegeneralizabilitypuzzleRetrievedfromhttpsssirorgarticlesentrythe_generalizability_puzzleBuildwithnotfor(2018January)RetrievedfromhttpsbuildwithnotforwordpresscomCairneyPampOliverK(2017)Evidence-basedpolicymakingisnotlikeevidence-basedmedicinesohowfarshouldyougotobridgethedividebetweenevidenceandpolicyHealthResearchPolicyandSystemsdoi101186s12961-017-0192-xCairneyPOliverKampWellsteadA(2016)TobridgethedividebetweenevidenceandpolicyReduceambiguityasmuchasuncertaintyPublicAdministrationReview76(3)399-402Retrievedfromhttpdxdoiorg101111puar12555Centreforpublicimpact(nd)RetrievedfromhttpswwwcentreforpublicimpactorgCernaL(2013)ThenatureofpolicychangeandimplementationAreviewofdifferenttheoreticalapproachesOrganisationforEconomicCo-OperationandDevelopmentRetrievedfromhttpswwwoecdorgeduceriTheNatureofPolicyChangeandImplementationpdfColoradoSprings(nd)Retrievedfromhttpscoloradospringsgovcommunicationspagewhat-expect-when-you-speakupContinuumofcare(coc)program(nd)RetrievedfromhttpswwwhudexchangeinfoprogramscocCouncilpresidentkicksoff$12millionyouthfund(nd)Retrievedfromhttpwwwbaltimorecitycouncilcomcontentcouncil-president-kicks-12-million-youth-fundDurlakJAampDuPreEP(2008)ImplementationmattersAreviewofresearchontheinfluenceofimplementationonprogramoutcomesandthefactorsaffectingimplementationAmericanJournalofCommunityPsychology41327-350Retrievedfromhttpswwwncbinlmnihgovpubmed18322790Fishbonediagram(ishikawa)(nd)Retrievedfromhttpasqorglearn-about-qualitycause-analysis-toolsoverviewfishbonehtmlFivewhysGettingtotherootofaproblemquickly(nd)RetrievedfromhttpswwwmindtoolscompagesarticlenewTMC_5Whtm
References
63
FligsteinNampMcAdamD(2011)TowardageneraltheoryofstrategicactionfieldsSociologicalTheory29(1)1-26RetrievedfromhttpsociologyberkeleyedusitesdefaultfilesfacultyfligsteinFligstein20McAdam20Soc20Theory20PaperpdHeinkelP(2018January12)CitymullspermitsforbikesharingcompaniesRetrievedfromhttpwwwdentonrccomnewsdenton20180112city-mulls-permits-bike-sharing-companiesHowisareamedianincomecalculated(2012October)Retrievedfromhttpswwwhudexchangeinfofaqs843how-is-area-median-income-calculatedLampkinLMampHatryHP(2003)KeystepsinoutcomemanagementUrbanInstituteRetrievedfromhttpswwwurbanorgresearchpublicationkey-steps-outcome-managementMarshDampSmithM(2000)UnderstandingpolicynetworksTowardsadialecticalapproachPoliticalStudies484-21Retrievedfromhttpjournalssagepubcomdoiabs1011111467-924800247McCannL(nd)RetrievedfromhttplaurenellencomMoultonSampSandfortJR(2016)ThestrategicactionfieldframeworkforpolicyimplementationresearchPolicyStudiesJournal45(1)Retrievedfromhttponlinelibrarywileycomstore101111psj12147assetpsj12147pdfv=1ampt=jdkhhcp4amps=e184c625e1855a6b1615ccb99ee27b1c0b3212d8PaynterB(2018January1)What100citiesarelearningfromeachotherbysharingtheirdataRetrievedfromhttpswwwfastcompanycom40518004what-100-cities-are-learning-from-each-other-by-sharing-their-dataPeakacademy(nd)Retrievedfromhttpswwwdenvergovorgcontentdenvergovenmayors-officeprograms-initiativespeak-performancepeak-academyhtmlPinnockHBarwickMCarpenterCREldridgeSGrandesGGriffithsCJRycroft-MaloneJMeissnerPMurrayEPatelASheikhAampTaylorSJC(2017)Standardsforreportingimplementationstudies(stari)ExplanationandelaborationdocumentBMJOpenRetrievedfromhttpbmjopenbmjcomcontentbmjopen74e013318fullpdfProctorESilmereHRaghavanRHovmandPAaronsGBungerAGriffeyRampHensleyM(2011)OutcomesforimplementationresearchConceptualdistinctionsmeasurementchallengesandresearchagendaAdministrationandPolicyinMentalHealth38(2)65-76doi101007s10488-010-0319-7TandonSD(2018)ProgressincommunityhealthpartnershipsResearcheducationandactionRetrievedfromhttpswwwpressjhuedujournalsprogress-community-health-partnerships-research-education-and-actionProvanKGampKenisP(2007)ModesofnetworkgovernanceStructuremanagementandeffectivenessJournalofPublicAdministrationResearchandTheory
References
64
18(2)229-252Retrievedfromhttpsacademicoupcomjpartarticle182229935895SabatierPA(1988)Anadvocacycoalitionframeworkofpolicychangeandtheroleofpolicy-orientedlearningthereinPolicySciences21(3)129-168Retrievedfromhttpwwwjstororgstable4532139SandstromAampCarlssonL(2008)TheperformanceofpolicynetworksTherelationbetweennetworkstructureandnetworkperformancePolicyStudiesJournal36497-524Retrievedfromhttponlinelibrarywileycomdoi101111j1541-0072200800281xabstractSchwartzMampWilsonE(nd)WhocanaffordtoliveinahomeAlookatdatafromthe2006americancommunitysurveyRetrievedfromhttpswwwcensusgovhousingcensuspublicationswho-can-affordpdfShearerJCAbelsonAKouyateBLavisJNampWaltG(2016)WhydopolicieschangeInstitutionsinterestsideasandnetworksinthreecasesofpolicyreformHealthPolicyandPlanning31(9)1200-1211Retrievedfromhttpsdoiorg101093heapolczw052ThelessonsofadministrativedataHigh-profilepolicy-relevantresearchpoweredbyadministrativedata(nd)Retrievedfromhttpswwwpovertyactionlaborgsitesdefaultfilesresources20161114-The-Lessons-of-Administrative-DatapdfToolsandotherneedsforqualitativeandquantitativemethods(2015February)AgencyforHealthcareResearchandQualityRetrievedfromhttpwwwahrqgovprofessionalsquality-patient-safetytalkingqualityassesstoolshtml
WeaverRCampBagchi-SenS(2013)SpatialanalysisofurbandeclineThegeographyofblightAppliedGeography4061-70
References
65
UniversityPartners
Name Title Affiliation Area(s)ofexpertise
Contactinformation
Examplesofworkingwithnon-academicdecisionmakers
WendyPBarreno
AssociateDirector
WilsonSheehanLabforEconomicOpportunitiesUniversityofNotreDame
LEOsfocusareasincludehealtheducationhousingchildwelfareeconomicopportunitycriminaljustice
wbarrenonduduorleondedu
MelissaDavey-RothwellPhD
AssociateScientist
JohnsHopkinsBloombergSchoolofPublicHealth
Health mdavey1jhuedu410-614-5854
BaltimoreCityHealthDepartmentLeadersMarylandDepartmentofHealthAddictionRecoveryServicesinDCExecutiveDirectorsofcommunityorganizationsacrosstheUSthatfocusonHIVSTIsubstanceabuseandimplementationscience
ElizabethStuart
AssociateDeanforEducationandProfessorDepartmentsofMentalHealthBiostatisticsandHealthPolicyand
JohnsHopkinsBloombergSchoolofPublicHealth
Educationsocialpolicystatisticalmethods
estuartjhuedu
Resources
66
Management
Design-ThinkingResources
EngagementGuideforCivicTech
Journeymap
PerformanceManagementResources
DenverPeakAcademy
Proxymeasuresguide
Talkingaboutperformancewithoutcomplicatingit(video)
SettingPerformanceTargetsguide
Datavisualizationguide
Performancemanagementgettingstartedguide
Exercises
Fishboneexercise
5Whys
AdministrativeData
TheLessonsofAdministrativeData(JPAL)
EvaluationResources
Quantitativeandqualitativemethods(USDepartmentofHealthandHumanServicesAgencyforHealthcareResearchandQuality)
KeyStepsinOutcomeManagement(UrbanInstitute)
RCT-YES
Sageresearchmethods
BestPracticesforMixedMethodsResearchintheHealthSciences
ACCORDChecklist
Pilotandimplementationplanning
Benchmarkingexamplesandproxymeasures
Seattleopendataplaybook
Resources
67
Datasciencescopingguide
Resources
68
AboutGovExGovExhelpsgovernmentsusedataandevidencetoimproveresidentsrsquolivesWesupportgovernmentsinbuildingtheirinternalcapacitytooperatesustainableanalyticspracticeswhichimprovecommunityoutcomesinfieldsfrompublicsafetytoeconomicdevelopmentGovExhasaproventrackrecordofworkinginconcertwithpartnerstoprovidetechnicalassistanceandtrainingatscale
SinceourlaunchinApril2015GovExhas
Trainedover2000governmentemployeesinpersonandonlineintopicsincludingdatamanagementperformancemanagementopendatacommunityengagementandanalytics
Providedtechnicalassistancetomorethan100localgovernmentsacrosstheglobeimprovingtheiruseofdatatoachieveefficienciesinservicedeliveryandbetteroutcomesforresidents
Assessedthecapacityofmorethan130governmentalorganizationstodeterminehowleadersstaffandexternalstakeholdersmanageandusedata
Localgovernmentsareincreasinglyonthefrontlinesofaddressinglargescaleproblemswithglobalimplicationssuchasclimatechangepublicsafetyandeconomicinequality
Whileevidence-basedinterventionstohelpsolvethoseproblemsareavailablelocalgovernmentstaffhavelimitedtimeandcapacitytofindandimplementresearch-basedsolutionsAsaresultcitiescontinuetoworkthewaytheyalwayshave--makingincrementalimprovementstoprogramsbutnotalwayscreatingchangeatasystem-level
ThroughresourcedevelopmentandcollaborationwithJHUfacultyGovExissupportingcityleadersinusingdataandevidencetomakedecisionsandcreatinglastingrelationshipswithresearchers
TheRoadmapwasdevelopedbyGovExaspartofitspartnershipwithBloombergPhilanthropiesWhatWorksCitiesinitiative
AboutGovEx
69
- Title page
- Introduction and purpose of the Roadmap
- Using the Roadmap
-
- Gather information about the problem
- Strategically engage key partners
- Review the evidence and collect details
- Decide on a policy to implement based on local factors
- Establish a framework to monitor progress
- Assess data accessibility quality and security
- Design implementation and evaluation
- Share your story and success
-
- Working with a range of evidence
- Thank you
- References
- Resources
- About GovEx
-
HowcanwehelpLGBTyouthfacinghomelessness(population)obtainpermanenthousing(outcome)
OnceyourreviewershavetheirresearchquestionsusethequestionstocreatealistofkeywordstheycanuseintheirliteraturesearchThiswillhelpkeeptheircollectionofarticlesreasonablyfocusedHavethemdescribewhattheirsearchprocesswillbeincludinglistingthedatabasestheywillreviewandtheirplanforstoringtheinformationtheywillcollect(Herearesomeadditionalsuggestionsforstructuringareviewplan)
WhatyourreviewerswillbedoingisalightweightversionofarapidreviewTotheextentthattheywouldliketodevelopamethodologyforspecificsearchprotocolsincludingcriticalappraisalprocessestheycancertainlygoaheadanddoso
Usedifferentlevelsofevidenceforthemostcompletepicture
AsdiscussedintheWorkingwitharangeofevidencechaptertheNestaStandardsofEvidenceframework--reproducedbelow--ishelpfultounderstandthelevelofconfidencethataprogramorinterventionishavingapositiveimpactThisapproachletsyouputinterventionsthatyouhaveheardaboutlocallybutwhichhavenotbeenstudiedtogetherwiththosewithademonstratedcausaleffectThisishelpfulsinceanumberofpoliciesprogramsandpracticesthatyouwillwanttoconsidermaynotyethavemuchofaresearchrecord
Reviewtheevidenceandcollectdetails
22
NearlyanypieceofinformationaboutaninterventioncanbeclassifiedwithinthestandardsofevidenceForexampleifsomeonehasacrediblehypothesisabouthowaninterventioncouldproducebenefitthistypeofevidencewouldbeclassifiedasLevel1ThistypeofevidencecouldincludeinterventionsinothercitiesthatyoureadaboutinthenewsorhearaboutataconferenceForinstancewhenUSDAallowedschoolsandschooldistrictstoprovidefreebreakfastandlunchtoallstudentsthroughtheCommunityEligibilityProvision(CEP)manydistrictsimplementedthisoptionevenbeforetherewasevidenceofeffectivenessThiswouldfitintoLevel1sinceitbuildsonthelogicthatprovidingaccesstomealswilldecreasefoodinsecurity
YoumayalsofindpiecesofinformationthatqualifyasLevel2evidenceForinstanceasmoreschooldistrictsimplementedCEPprincipalstestifiedaboutpositivechangestheyobservedinstudentssuchasamorepositivedemeanorthatcanleadtobetteracademicperformanceDataalsoshowedthatthestudentsdescribedwerereceivingmoremealsthanbeforetheprogramsimplementationHoweveritwastoosoontobuildevidencetoshowacausalrelationshipbetweentheincreaseinfreemealsandincreasedacademicperformance
ldquoLower-levelrdquoevidencecanconsistofnuancedbutvaluableelementsthatrelatetoaninterventionrsquospotentialfitwithacommunityQualitativeinformationsuchastestimonyfrompeoplewhohaveexperiencedaninterventionandthosewhohavefacilitatedorimplementeditwillalwaysbeusefultoconsiderandprovidesdetailedlocalcontext
Experimentalstudies--describedstartingatLevel3withcontrolledexperimentsandgoingthroughLevel5withfullyfleshedoutprotocolsandprocessesforreplication--stilloccupythehigherlevelsofthisdescriptionofevidenceThisprioritizationshowsthevalueofrigorousstudyforevaluatingtheimpactofaparticularpolicyorprogramratherthanotherfactorsthatcouldbeinfluencingoutcomesofinterestRememberthatmultipletypesofevidencecancomplementrigorousimpactevaluationsandgivecityleadersadditionalinsightsandbroaderperspective
Accessinformation
Encourageyourreviewerstobegintheirsearchforpromisingpracticesingovernmentandnon-profitclearinghousesandinacademicpolicyjournalsWorkpublishedinthesesourceshasalreadybeensubjectedtoahighlevelofprofessionalscrutinyandgenerallycontainsagreatdealofdetailwhichmeansthatyourreviewersaremorelikelytobeabletopullouttheinformationyouneedtomakeagooddecision
Clearinghouses
Reviewtheevidenceandcollectdetails
23
Herearesomeoftheclearinghouses(collectionsofprogramevaluations)forevidence-basedpoliciesandprograms
ResultsFirstClearinghouseDatabaseThePewCharitableTrustsCollectstogethereightotherclearinghousestoprovideinformationoninterventionsinthefollowingcategoriesadultcriminaljusticeeducationchildwelfarehealthjuvenilejusticementalhealthsubstanceabuseandyouthdevelopmentIndividualclearinghousesincludedinResultsFirstare
BlueprintsforHealthyYouthDevelopment-foryouthdevelopmentandviolenceprevention
CaliforniaEvidence-BasedClearinghouse-forchildwelfareinterventions
TheLauraandJohnArnoldFoundationrsquosSocialProgramsThatWork-forsocialservicesprograms
TheUSDepartmentofEducationrsquosWhatWorksClearinghouse-foreducationinterventions
TheUSDepartmentofHealthandHumanServicesrsquoResearch-TestedInterventionPrograms(RTIPs)-forcancercontrolinterventions
TheUSDepartmentofHealthandHumanServicesrsquoTeenPregnancyPreventionEvidenceReview(TPPEvidenceReview)-forsexualriskbehaviorandsexualhealthoutcomes
TheUSDepartmentofJusticersquosCrimeSolutionsgov-forcriminaljusticejuvenilejusticeandcrimevictimservices
TheUSSubstanceAbuseandMentalHealthServicesAdministrationrsquosNationalRegistryofEvidence-basedProgramsandPractices(NREPP)-substanceuseandmentalhealthinterventions
TheUniversityofWisconsinPopulationHealthInstituteandRobertWoodJohnsonFoundationrsquosCountyHealthRankingsandRoadmapsWhatWorksforHealth(WhatWorksforHealth)-healthbehaviorsclinicalcaresocialandeconomicfactorsandthephysicalenvironment
UKWhatWorksCentresTheBritishgovernmenthascharteredanetworkofsubject-specificorganizationswhichcollectsynthesizeandreportoutevidenceontheeffectivenessofpolicyprogramsandpractices
NationalInstituteforHealthandCareExcellence(NICE)-healthandsocialcare
Reviewtheevidenceandcollectdetails
24
SuttonTrustEducationalEndowmentFoundation-educationalachievement
CollegeofPolicingWhatWorksCentreforCrimeReduction-crimereduction
EarlyInterventionFoundation-earlyintervention
WhatWorksCentreforLocalEconomicGrowth-localeconomicgrowth
CentreforAgeingBetter-qualityoflifeforolderpeople
WhatWorksCentreforWellbeing-wellbeing
WilsonSheehanLabforEconomicOpportunities(LEO)UniversityofNotreDameProvideslinkstotheirevaluationsofinterventionsincriminaljusticeself-sufficiencyeducationhealthandhousingandhomelessness
ClearinghouseforLaborEvaluationandResearchUSDepartmentofLaborProvidesresearchonworkforceandlabor-relatedpoliciesandprograms
J-PALEvaluationsAbdulLatifJameelPovertyActionLabStudiesonpoverty-reducinginterventionsfromaroundtheworld
CampbellSystematicReviewsCampbellCollaborationSystematicreviewofpolicyinterventionsbyscholarlycoordinatinggroupsinsevenissueareascrimeandjusticeeducationinternationaldevelopmentmethodssocialwelfareknowledgetranslationandimplementationanddisability
Benefit-CostResultsWashingtonStateInstituteforPublicPolicyAWashingtonState-charteredprogramtoestablishthelikelihoodofrelativecostsandbenefitsforprogramsacrossthefollowingareasjuvenilejusticeadultcriminaljusticechildwelfarepre-Kto12educationchildrensmentalhealthhealthcaresubstanceusedisordersadultmentalhealthpublichealthandPreventionworkforcedevelopmentandhighereducation
EffectiveInterventionsCentersforDiseaseControlandPreventionRepositoryforevidence-informedHIVpreventioncareinterventions(withstepsonhowtoimplement)
Journalsandindexes
OnceyouhavereviewedrelevantinformationaccessiblethroughclearinghousesitisworthtakingalookatindividualstudiespublishedasscholarlyarticlesYourreviewerswilldothisbysearchingarticleindexesandacademicjournalsUnfortunatelyanumberofusefulacademicjournalsandindexingplatformsmaynotbegenerallyaccessibletoyourreviewersifyoudonothaveasubscriptiontothemToaddressthisconsiderpartneringwithalocal
Reviewtheevidenceandcollectdetails
25
collegeoruniversitytoprovideyouwithalistofarticlesfromthefollowingsources(Thisisalsoanexcellentopportunitytoseekresearchsupportfromanyacademicstakeholdersyouidentifiedpreviously)
Indexes
GoogleScholarincludesmostpeer-reviewedonlineacademicjournalsandbooksconferencepapersthesesanddissertationspreprintsabstractstechnicalreportsandotherscholarlyliteratureincludingcourtopinionsandpatentsItallowsyoutospecifyapublicationrangeinyoursearchandprovideslinkstolikelysourcesforalllistedarticles
PolicyFileIndexisauniqueresourceforUSpublicpolicyresearchinthatitgrantsusersaccesstotimelyupdatedinformationfromover350publicpolicythinktanksnongovernmentalorganizationsresearchinstitutesuniversitycentersadvocacygroupsandotherentities
PolicyArchiveworkstocollectanddisseminatesummariesandfulltextsvideosreportsbriefsandmultimediamaterialofthinktankuniversitygovernmentandfoundation-fundedpolicyresearch
Anothersourceforhigh-qualityresearchisthereportsoftheCongressionalResearchServiceYoucanfindtheirreportsaggregatedhere
Specificjournals
Oftenthemostusefulacademicjournalsforunearthingpolicyorprogramreviewswillbeineachspecificsubjectarea--thatisineducationjournalsforeducationpoliciesYoucanuseageneralindexlikeGoogleScholartobegintolearnwhichjournalsaremostoftenusefulforyoursearchHoweveryoucanalsosearchthefollowingtwogeneralistpolicypublicationsforissuesandarticlesrelatedtoyourtopic
PublicAdministrationReviewJournalofPolicyAnalysisandManagement
Newssourcessubjectmatterexpertsandinterviews
AlthoughitisvaluabletoknowwhathasbeenexploredthroughascholarlyresearchframeworkitisalsousefultolearnwhatkindofinterventionsarebeingtriedinthefieldrightnowandhowpeoplefeelaboutthesetrialsSearchsubscriptionnewsaggregatorslikeLexisNexisorfreenewsaggregatorslikeGoogleNewsorYahooNewsforyourissueareaplusthewordpolicyorprogramtoseewhatkindsofinterventionsarebeingcoveredinthenews
Reviewtheevidenceandcollectdetails
26
DonotforgettocontacttheadvisorygroupmemberswhowerechosenfortheirsubjectmatterexpertiseInadditiontoprovidinggoodmaterialthroughinterviewsthemselvestheymayhaveideasaboutothersubjectmatterexpertsyoushouldcontacttoaskaboutpotentialinterventions
Peercitiesandnetworks
InadditiontoreadingabouttheworkofothercitiesitisidealifyouareabletospeakwithpeoplewhoareinvolvedwithcurrentimplementationsofinterestinginterventionsInyourcityyoulikelyhaverelationshipswithyourneighborsbutitcanbehelpfultolookoutsideofyourregionforsolutionsaswellTofindcounterpartsinothercitieswhoareworkingonaprogramyouwouldliketolearnmoreaboutyoucancontactstaffinintermediaryorganizations(likeGovExoryourprofessionalassociations)whomightbeinapositiontoputyouintouchimmediatelyorwhomightseeanothergoodrouteforsettingupaconversationHereisalistofnetworksthatmaybehelpful
InternationalCityCountyManagementAssociation(ICMA)NationalAssociationofCounties(NACO)NationalLeagueofCitiesUSConferenceofMayorsWhatWorksCities
Collectthedetails
Onceyourreviewershavefoundevidenceonpossiblepoliciesorprogramstoimplementtheyshouldcapturekeydetailsthatwillhelpdecisionmakersunderstandthepotentialeffectivenessandapplicabilityofeachpotentialintervention
Foreachsourcethathasprovidedinformationyouwanttoconveyusethetemplatebelowtocaptureasmanyofthefollowingdetailsaspossible(Youdonotneedtocompletethisforsourcesthatprovideduplicativeinformationorareotherwisenotthatuseful)
Programorpolicyname (egldquoHousingFirstrdquoorldquoblightedpropertiesregistryrdquo)
Sourcetype (egjournalarticleinterviewsubjectmatterexpertstudyassociationcommunityparticipationprocess)
Sourcedetails (egarticletitleandpublicationnameandtitleofinterviewee)
Evaluationmethodology
(egrandomizedevaluationcomparativeevaluationoftwodifferentgroupsbeforeafteranalysis)
Evidencelevel (egLevel12345)
Programorpolicy
Reviewtheevidenceandcollectdetails
27
description
Programorpolicygoal (1sentencecapturingtheprincipalaimoftheintervention)
Existingimplementations
(WherehastheinterventionbeentriedListasmanysitesasyouareawareof)
Characteristicsofimplementationsites
(ForeachimplementationsiteofferbasicgeographicIDmdashcityandstateThenyouwillwanttocollectthedetailswhicharerelevanttoyourdecisionmakersrsquoconsiderationsaboutwhatmakesalocationldquosimilartordquoorldquodifferentfromrdquoyoursituationYouwillatleastwanttoincludeforeachimplementationsitePopulationnumberPopulationdensity(ruralurbansuburban)Socio-economiccharacteristicsBeforestartingaskyourdecisionmakersaboutanyparticularcharacteristicsofprogramimplementationsitestheywouldwanttoknowbeforeseekingtoreplicate)
Characteristicsofimplementationpartners
(Anyavailabledetailsaboutorganizationsinstitutionsorpeoplewhowereinvolvedwithimplementingtheintervention-egindependentchildcarecenterpublicelementaryschoolcitypublichealthnurses)
Characteristicsoftargetpopulationeligibleprogramparticipants
(Anyavailabledetailsaboutpeoplewhoarethebeneficiariesoftheprogramorotherwiseincludedoraffectedbytheintervention-egnumberofprogramparticipantscategoriesofpropertyownersaffected)
Policyorprogramduration
(HowlonghastheinterventionbeeninplaceineachimplementationsiteHasitendedandifsowhy)
Whatbenefitscamefromeachimplementation
(Anyobservedpositiveeffectsoftheinterventionincludingeffectsize)
Statisticalestimatesofinterventioneffectiveness
(Anyavailablestatisticalestimatesoftheinterventionrsquoseffects)
Whatdrawbackswereobserved
(Anyavailabledetailsaboutunintendedorunavoidablenegativeeffectsoftheintervention)
Programorpolicycosts
(Anyavailabledetailsaboutprogramcostsperimplementation)
Reviewtheevidenceandcollectdetails
28
Reviewtheevidenceandcollectdetails
29
DecideonapolicytoimplementbasedonlocalfactorsTogetthecriticalbuy-infromleadershiptoimplementanewpolicyorprogramtheprojectleadshouldscheduleameetingwithdecisionmakerstopresenttheevidencesurroundingthepolicyoptionscapturedinthetemplateDependingonyourinsightintoyourcityleadershipswillingnesstomakechangeyoumaywishtosharetheresearchyouhaveconductedontherootcausesoftheproblemandsomeofyouranalysisontheimpactofcurrenteffortstodate
Considertheapplicabilityofanapproach
AsdiscussedinthepreviouschapterpolicymakersoftenlookforevidenceofeffectiveprogramsinothercitiesbutoftenthequestionarisesofhowgeneralizableaprogramisfromonecitytoanotherTheAbdulLatifJameelPovertyActionLab(J-PAL)suggeststhatinreviewingevidencefromanotherlocationitishelpfultoidentifythespecificmechanismthatwasused(deliveryofprogramorserviceinformationsharingetc)identifythelocalconditionsthatwereinplaceobservethebehaviorchangethatresultedfromthemechanismandfinallytounderstandlocalcontextandconditions(legalorcostconstraintsculturalnormsetc)Thissetofpresentationelementscanbedrawnfromthelistofdetailscollectedintheresearchtemplate
EducationpolicymakersinChicagoILusedfindingsfromprogramsinKenyaandIndiainwhichchildrenwhohadfallenbehindtheirpeersweregiveneducationalsupporttargetedtotheirleveloflearningThemechanismusedwastargetedteachingandthereweresimilarconditionspresentinallthreecountriesAlthoughsomestudentshadfallenbehindinthecurriculumteacherswereincentivizedtoteachgrade-levelmaterialsConsiderationsofhowthelocalcontextinChicagomightaffectimplementationofatargetedteachingprogramincludedunderstandinghowtheneighborhoodsthatthestudentswerefrommayhaveaffectedtheireducationaloutcomesandthehighcostofteachinginsmallgroupsInChicagoMatchEducationprovidedtwo-on-onetutoringwhichmatchedstudentswithwell-educatedindividualswhohavenothadformalteachertrainingforalower-costmodelTheresultswerepositiveStudentswhoweretutoredperformedbetterinmathandweremorelikelytopasshighschoolclasses
ThegeneralizabilityframeworkproposedbyJ-PALisbelowUsethisasatemplateasyouconsiderpotentialpolicysolutions
Decideonapolicytoimplementbasedonlocalfactors
30
SourceTheGeneralizabilityPuzzle
OnceyouhavenarrowedthepolicyoptionsdiscussthequestionsbelowforeachoptionsyouareconsideringThisstepiscriticalasunderstandinglocalcontextissoimportantinsuccessfulpolicyimplementation
CommunityContextQuestionnaire
Localcontext
DoesthisoptionmakesenseinourcommunityIsthisaonesizefitsallapproachIsanyparticulargroupabsentfromthisapproachIfsoisthatok
Youranswers
Effectiveness
WhatwouldweexpecttoobserveifthepilotisaseffectivehereasinCityX
Youranswers
Timing
DoesthetimingmakesenseIsthistherighttimetoimplementthispolicychange
Decideonapolicytoimplementbasedonlocalfactors
31
WherearethepotentialroadblocksthatcouldaffectthescheduleIsthereanywaytomitigatethisHowlongwouldittaketoseeanimpact
Youranswers
Implementation
DowehavethestaffcapacitytoimplementthisArethereadditionalpartnersthatyoucanbringtothetabletosupportimplementationDoesthismakesensegivenourformofgovernment
Youranswers
Authority
DoesourcityhavetheauthoritytoenactthisIfsowhatistheappropriatemechanismAretherelegalbarriersArethereanyexistingpreemptionissuesfromthestateorfederalgovernmentCanthosebeaddressedthroughlegislationadvocacyIstherepoliticalsupportoroppositiontothis
Youranswers
Cost
HowmuchwillthiscostannuallyHowwillwepayforitCanthepolicybeimplementedwithexistingfunding
IfsoisthisfundingreliableHowwouldthispolicyoptionaffectourcitysbudgetintheshortandlongtermWhatotherfundingoptionscanweconsiderIsthereanalternativethatwouldcostlessandgiveussimilarresultsWillanyconstituencybeadverselyaffectediffundingforprogramsthatimpactthemisshiftedtoanotherprogrampolicyorinitiative
Youranswers
Decideonapolicytoimplementbasedonlocalfactors
32
EstablishaframeworktomonitorprogressNowthatyouhavereviewedtheevidenceconsideredlocalcontextandchosenyourpolicyitistimetodeterminehowyourcitywillmonitorprogressofthisnewapproachCitystaffandleadersneedtoestablishtheinfrastructureforongoingcontinuousdecisionmakingbasedonwhetherthepolicychangeisleadingtothedesiredoutcomesforresidentsandwhetherthechangeismeaningfulandrationaltofrontlineworkersSettingarbitrarygoalsatthelocallevelthatconflictwithgoalsatotherlevelsisconfusingtotheindividualsresponsibleforachievingthegoalsGoalsthataresettoohightoachievecanbedemoralizingtostaff
ThissectionincludesstepstosetupavenueinwhichCitystaffandleadershiproutinelyvisitkeyindicatorstodeterminewhatisorisnotworkingandtomakedecisionsinreal-timeFormorereadingonperformancemanagementpleasevisittheGovExPerformanceManagementGettingStartedGuide
InperformancemanagementtermssuchasindicatormeasureandmetricmaybeusedinterchangeablyInthisguidewewilluseindicatorInyourworkthemostimportantthingistobeconsistentinwhichevertermyouuseThisvideomaybehelpfulinunderstandingthisconceptandcommunicatingittoothers
Establishthefundamentalsandgatheryourteam
FirstdeterminewhetherthereisanestablishedvenueorregularmeetingscheduleduringwhichyoucandiscussprogressIfyourcityhasanexistingperformancemanagementstructureonarelatedtopicaddingadiscussionoftheprogressofpolicychangetoyouragendacanbeagreatideaIfthiswillbeastandalonemeetingdecidehowfrequentlytheprojectteamanddecisionmakerswillmeetandputthosemeetingsonthecalendarfortheentirelengthoftheproject
DeterminewhowillstaffandattendroutinemeetingsonCityprogressThesemeetingsmayincludetheMayorandorCityManagerdepartmentheadstaffwhoareimplementingthepolicychangeandanalysts
Identifyindicators
ItiscriticalthatindicatorsbeanimportantcityandcommunitypriorityaswellassomethingtheCityplaysamajorroleinaddressingTheremaybeinstancesinwhichtheCitywillneedtoreachouttoorcollaboratewithexternalpartnerstogetamorecompletepictureofprogressontheoverallpolicychange
Establishaframeworktomonitorprogress
33
Anoutcomeindicatordescribesthedesiredeffectofthetargetedpopulation
ExampleBellevuesfamilyfriendlyneighborhoodsindicatormeasuresthepercentofresidentswhobelievetheyliveinaneighborhoodthatsupportsfamiliesparticularlythosewithchildren
WithyourkeypartnersrevisityourproblemstatementcurrentstateandfuturestateexerciseandbegintoidentifytheoutcomeindicatorsthatwillillustratesuccessFornowchooseasingletop-levelkeyperformanceindicator(KPI)ItiscriticalthattheKPI-thepieceyouwillmeasure-beapriorityforthecommunityandsomethingtheCityplaysamajorroleinaddressing
Outputindicatorsmeasuretheprocessofprogramorservicedelivery
ExampleSanDiegoMayorKevinFaulconersgoaltorepair1000milesofstreetby2020
ReviewingtheadministrativedatasetinventorybegintoparedowntherelevantprogramsanddatasetsthatsupportyourKPIWhatarethemeasuresofeffectivenessforeachprogramTheseareyouroutputindicators
Thediagrambelowillustratestherelationshipbetweenoutputandoutcomeindicators
ThinkaboutimplementationindicatorsasmeasuringstepsintheprogramsrolloutthatarenecessarytoobtaindesiredchangesImplementationoutcomescanfocusontheinterventionstaffdeliveringtheinterventionanorganizationorsystemandcanaddressfeasibilityadaptabilitysustainabilityandcostTwokeyquestionstoconsiderwhendevelopingimplementationindicatorsare
1 Istheinterventionacceptableandappropriatetothecommunityand
2 IsitfeasibletoimplementinterventionXatthistimewiththeseresources
Establishaframeworktomonitorprogress
34
Amoreextensivelistwithdefinitionsofpotentialimplementationoutcomesislinkedhere
UsethetablebelowtochoosethedesiredimplementationoutcomesforyourpilotandtoplanyourmeasurementtechniqueThisshouldincludelevelofanalysis(individualproviderorconsumerorganizationorsetting)yourmeasurementtechnique(surveysobservationinterviewfocusgroupsadministrativedataanalysis)andthephasewhenyouwillcollectthisinformation(earlymidlate)Theremaybemorethanonelevelofanalysismeasurementtechniqueandphaseofdatacollection
Implementationoutcome
Definitionofoutcome
Levelofanalysisandmeasurement
Yourmeasurementtechnique
Phaseofpilotwhenyouwillcollectdata
Usethespacebelowtocaptureyourindicatorwishlist
Indicatorwishlist
Outcomes_____________________________________________________________________________
Outputs________________________________________________________________________________
Implementationoutcomes______________________________________________________________
Setapublichigh-levelgoal
Thehigh-levelgoaliscriticalinestablishingdirectiontogalvanizethepeoplewhowillbedoingtheworkWhendiscussingthegoalusetheMayorsbullypulpitLookforkeymilestonesinCitygovernmenttosharethegoalstatementsuchasthebeginningofaMayoraladministrationaStateoftheCityspeechorotherpublicremarks
GoalsshouldbeldquoSMARTrdquo
Specificaspossible(innamingthepopulationwhowillbebetterofftheactionthecitywilltake)
Measurable
Achievable
Establishaframeworktomonitorprogress
35
Results-focused
Time-bound
ThegoalstatementshouldincludeyouroutcomeindicatorlikeChattanoogasgoalldquoWewillendchronicveteranhomelessnessbyDecember2016rdquoAdiagramtohepyoubuildaSMARTgoalaroundyouroutcomeindicatorisbelow
Component Description Yourturn
Specific WhowillbebetteroffWhatactionthecitywilltake
Measurable Howwillyoumeasuresuccess
Achievable Isthegoalattainable
Results-focused Whatistheexpectedresult
Time-bound When
Setrealisticambitioustargetsandtimelinesforeachindicator
ToestablisharealisticandambitioustargetforyourgoalstatementusetheadministrativedatacollectedforyourchosenKPItoreviewactualCityperformanceinthemostrecentyearcomparedwiththethreeorfourmostrecentyearsworthofdataBegintocalculatesummarystatistics(egtotalminimumaveragemedianmaximum)toidentifytrendsCreatecumulativeandmonthlychartstoidentifytrendsandanomaliesBasedonyouranalysisofhistoricaldataignoretheanomaliesandassumetheproductivitytrendswillfollowthesameoverallpatternfrommonth-to-monthFinallyapplyanincreasedproductivityfactortotheperformanceeachmonthtoforecastfutureprogressBasedonthisanalysischooseatargetthatinspiresimprovementwhileaccountingforreality
Manycitiesthataremorematureintheirperformancemanagementpracticesrecognizetheimportanceoftakingacommunity-levelapproachtoimprovingoutcomesforresidentsThesecitiesarecreatinggoaltaxonomiesthatincludemeasuresrelatedtopublicsafetyhousingandevenrecreationCitiesdriveprogressonthesegoalsbyimplementingstrategiesthatengagemultipledepartmentsandcollaboratewithserviceprovidersinthecommunity
EventuallyyouwilldevelopgoalstatementsfortheoutputsinordertocapturetheefforttoachievetheoutcomecontainedinyourgoalstatementYoursetofindicatorsshouldbecomposedofshort-andinterim-termsupportivemeasureswithaccuratetargetsandrelevanttimelinesAnexamplegoaltaxonomyfromtheCityofLasVegassResultsVegasprogramisbelowUsethistaxonomyasamodelforyourownsuiteofindicatorsandgoalstatements
Establishaframeworktomonitorprogress
36
Yourturn
Thetemplatebelowismeanttohelpyoucraftagoalstatementforeachsupportingmeasure
Goalstatement
Departments
Outcome
Activities
KPI
Target
Timeframe
Conductanddiscussrelevantanalysis
ThrougharoutineseriesofmeetingscontinuallymonitorprogressofKPIssupportingindicatorsandanyotherdatacollectedsuchasqualitativedatatoidentifytrendsthatwillindicatewhetherthechosenpolicyishavingthedesiredimpactThesedatapointsshoulddrivethediscussionduringthesemeetingspointingtoopportunitiesforfurtherinquiryinto
Establishaframeworktomonitorprogress
37
whatisdrivingtheobservedtrendsAnalystsshouldbeencouragedandtrainedtoaskquestionsofthedepartmentsproducingthedataandconductsitevisitswhenpossibletodeepentheirunderstandingofthedataPresentationoftheanalysisshouldfocusononeideaatatimetokeeptheconversationfocusedonkeypointsandspecificopportunitiesforimprovementYoumayconsiderdevelopingapublicfacingdashboardtosharethisprogressmorebroadly
AgeneralcomparisonofthestatusquodatawithwhatwasobservedduringthepilotcanprovidedecisionmakerswithaquickunderstandingofthedifferencesbetweentheoldandnewapproachesalthoughthiswillnotshowcausalityOftenthesecomparisonswillincludeoperationalmetricssuchasdaysordollarssavedbutrememberthattheimpactonpeoplewillbemostimportantIfyouareworkingwithanevaluationpartnertheywillsupportthisstep
Usethespacebelowtosummarizetheresultsofyouranalysis
MeasureTargetvalue(seesectiononsettingtargetsabove)
Pilotvalue(whatdidyouobserveduringthepilot)
Historicalaverage(ifavailable)
Results(WhoisbetteroffIsanyonenegativelyimpacted1-2sentencesreassessingyourgoalstatement)
DocumentyourprocessandprogressincludingfeedbackfromtheparticipantsimplementationpartnersandtheprojectteamDidanythingunexpectedoccurduringthepilotArethereinternalandexternalfactorsthatmayaffectyouroutcomesThisiswheretheimplementationoutcomescanbecanhelpfulIftheinterventiondidnotgoasplannedconsiderwhetheritisbecausetheinterventionwasnotacceptableorfeasible
UsethetemplatebelowcreatedbyDenversPeakAcademytocapturethisinformationIfthe306090daytimelinedoesnotmakesenseforyourcityfeelfreetocustomizeit
CurrentState
FutureState
30Day
60Day
90day
Whatwentwell(+)
Whatdidnotgowellshouldbechanged(-)
IfaquestionordiscussionpointarisesthatcannotbeansweredinthemeetinganassignedstaffmembershouldcapturethequestionduringthemeetingAfterwardthefollowupitemsshouldbesenttorelevantpartnersandthenextmeetingshouldbeginwithadiscussionof
Establishaframeworktomonitorprogress
38
thoseitems
Takeaction
DiscussionsaboutwhetheryourindicatorsaremovingintherightdirectionandwhatmightbedrivingtrendsmayleadtoadiscoverythattherearenotenoughresourcesallocatedtothispolicyorissueTheremaybelimitedfundingforstaffprogrammingortechnologythatisdrivingthedatainthewrongdirectionIfCityleadershipisintheroomthisisagreattimetomakethecaseforadditionalfundingduringthebudgetseasonBycollectinganalyzinganddiscussingdataonthispolicyroutinelyyouarebuildingevidencetomakethecasefortherightleveloffundingtoreachtheestablishedtargets
Establishaframeworktomonitorprogress
39
Assessdataaccessibilityqualityandsecurity
Determineaccessibilityandqualityofdata
OnceyouhavecompiledyourwishlistofindicatorsitistimetodeterminetheaccessibilityandqualityofthedataTopreventcomplicationyoumaywishtoprioritizerelevantindicatorsforwhichyouarealreadycollectingdataAvoidthetemptationofadoptingatechnologysolutiontocollectnewdataatthispointsuchasaphotosurveytoolorotherappFornowyouaretryingtodeterminewhichdatasetsarerelevantaccessibleandhighqualityBesuretoworkwithyourkeyinternalpartnerswhileyouareidentifyingyourmeasuresDepartmentstaffwhoworkonprogramsarefamiliarwiththeindicatorsthatwillaccuratelysupporttheoutcomesanddatabaseadministratorswillbevaluablewhendiscussingaccessibilityandqualityofthedata
Abrieflistofconsiderationsfortheaccessibilityofyourdataisbelow
ArewecurrentlycollectingthisdatasetorwillwebecollectingitforthefirsttimeHowfarbackdoestherecordgo(timehorizon)WouldweneednewtechnologytocollectoranalyzethedataCanthedatabecollectedmanuallyorcanitbeautomatedHowfrequentlyisthedataproduced(continuouslyhourlydailyweeklymonthlyquarterlyyearlyetc)IsthedataavailablefortheentireCitypopulationorasubsetofthepopulationHowgranularisthedata(individuallevelCensusblockneighborhoodcitywideetc)DoesthedatacontainsensitiveinformationIsthereaproxymeasurethatwecanusefornowIsthedatamachinereadable
AsnotedbytheWilsonSheehanLabforEconomicOpportunities(LEO)attheUniversityofNotreDameleveragingadministrativedatacansignificantlyreducethecostsandtimelinesofrigorousevaluationCitiescanestablishaninternaldatasharingagreementacrossdepartmentsincludinglanguagethatenablesadministrativedatatobeusedforresearchpurposesCitiescanalsoidentifyexternaladministrativedata-holderssuchasacountyschooldistrictpolicedepartmenthospitaloranotherstatethatmayhaveoutcomedataimportantformeasuringprogramimpactCitiescanleverageexistingrelationshipsandorworkwithresearchinstitutionstodevelopstreamlineddatasharingprocessesandagreementswiththeseexternalentities
Assessdataaccessibilityqualityandsecurity
40
Evaluatethequalityofyourdata
QualityisasimportantasaccessibilityToevaluatedataqualityyoumaywishtocreatearubricliketheonebelow
Value ValueDescription
3 ExcellentDataQualityThedataiscompleteaccurateandupdatedregularly
2 AdequateDataQualityThedataissomewhatcompleteaccurateandupdatedfairlyregularly
1 PoorDataQualityThedataisnotcompleteaccurateorupdatedregularly
ThisisagreattimetobuildonthedatainventorythatyoustartedinthegatherinformationpageSuggestedadditionalfieldstoaddtoyourdatainventorytemplatearebelow
Program DatasetTitle
DatasetDescription DataSource
Departmentorexternalpartner
DoestheCityalreadycollectthisdata
Whatisthenameoftheprogram
Human-readablenameofthedataset
Shouldbeunderstandablebynon-technicalusers
Whatistheoriginalsystemapplicationfilethathousesit
Departmentorexternalpartnerthatmaintainsthedata
Isthisdataalreadycollected
Updatefrequency-ThefileformatforthedatasetTypicallycsvxlsxshapefiletxtetc
Format-Howoftenthedatasetisupdated
Timerange-Theperiodoftimecoveredinthisparticulardataset
Datause-Howisthisdatasetcommonlyused
Datausers-Whotypicallyusesthisdataset
Dataownerstitle-Titleofthedataowner
Dataownersposition-DataOwnersjobtitle
Dataownersemailaddress-DataOwnersemailaddressWhomanagesthedataandorisresponsibleforgrantingpermissiontoaccessthedataWhounderstandswhatthedatasetincludesandcananswerquestionsaboutit
Dataqualityconcerns-Describetheconcernsyoumayhaveaboutthequalityofthedataset
Assessdataaccessibilityqualityandsecurity
41
Division-DivisionwherethedataismaintainedIfmorethanonedivisionisresponsiblelisttheprimaryownerofthedataset
Sensitivedatacomments-DescribetheissuesthatwouldpreventthecityfromsharingthedatapubliclyCommonsensitivityconcernsincludeprivacyviolationssecurityissuesorhighcostorstaffdemandstopostthedata
Publishingstatus-Thistheplacetoputanycommentsregardingthecurrentstatusofthedataset
Publishedlink-Thisisthelinktothepublisheddatasetontheopendataportalorcitywebsite
Reviewdatasecurityandprivacyprotectionprotocols
OftenapolicychangealsoinvolvesachangeinthedatathatisbeingcollectedhowthatdataisbeinghandledwhohasaccesstoitandahostofotheradjustmentsAsyoutakestockofthedataassetsthatareavailableforyourinitiativeitisvitaltoalsothinkthroughwhatwillbedonetoprotectthatdataandtheprivacyofthepeopleitrepresentsAbigpartofgooddatasecurityinvolvesstrongdatagovernancepracticesthatdefineandidentifysensitivedataandconsciouslyworktobalancethevalueandriskofstoringsuchsensitivedataOnewayofevaluatingyourdatacomesfromcaselawrelatedtotheUSFreedomofInformationActwhichwehavetranslatedtothechecklistbelow
1 Woulddisclosureresultinasubstantialinvasionofprivacy2 Haveweconsideredtheextentorvalueofthepublicinterestandthepurposeorobject
oftheindividualsseekingdisclosure3 Istheinformationavailablefromothersources4 Wastheinformationgivenwithanexpectationofconfidentiality5 Isitpossibletomouldreliefsoastolimittheinvasionofindividualprivacy
TheseriskscouldbemitigatedusingavarietyofstrategiesfromaggregatingdatatoobfuscatingindividualstoredactingfieldsthatmayincludePersonallyIdentifiableInformation(PII)orre-evaluatingyourdatacollectionmethodstofindalternativewaysofgettingtheinformationyouneedThesestepsshouldbecodifiedandmanagedbyadatagovernancecommitteeifpossibleForthesensitivedatathatisdeemedessentialcreateaplanaroundpotentialdatabreachesinordertoquicklyandconfidentlyrespondtoemergencysituations
ProactivelypreparingforthesesituationsisoftenthebeststrategyfordatasecuritybutdonotforgetthetechnicalsideInvolveyourITadministratorsinconversationsaboutdatastorageandaccessensuringthattheywillbeprotectedbyenterprise-gradecybersecuritysystems
Assessdataaccessibilityqualityandsecurity
42
Assessdataaccessibilityqualityandsecurity
43
Designimplementationandevaluation
Designtheimplementation
NowthatyouhaveyouraccountabilityplanandkeyindicatorsitistimetogetspecificaboutyourplanforimplementationThetemplatebelowwillhelpyouthinkthroughthespecificmechanismsofimplementingpolicychange
UnderstandingorganizationsandsystemsiscriticaltounderstandingandmakingdecisionsaboutwhereandhowinterventionsshouldbeappliedApolicyimplementedinasinglesettingmayshowpromisingresultsbutimplementingmultiplepoliciesandprogramsatthesametimewithdifferentactorsresponsibleforeachaspectcanhelpensuresustainablebehaviorchange
InSomervilleMAShapeUpSomerville(SUS)isacitywidepublichealthprogramthatisimplementedacrossmultiplelevelsofinfluenceandsystemsthroughinterventionsbeforeduringandafterschooltoincludechangestobuiltenvironmentsuchasbikelanesregionalmasstransportationandphysicaleducationequipmentinschoolsandgymstoincreasephysicalactivitywellnesspoliciesandschoolgardensnutritioneducationduringandafterschoolandhealthyrestaurantprogramstosupporthealthyfoodandhealthyeatinghabits
Usethetemplatebelowtoplanyourimplementation
Policyimplementationplanningtemplate
ListthespecificdesiredoutcomesInthepreviousstepyouworkedwithdepartmentstafftoestablishspecificmeasurableoutcomesandinterimindicatorsPrioritizetheindicatorsthatwilldemonstrateprogressUsethespacebelowtocapturethedesiredoutcomesofthepolicychange
ExampleShapeupSomervillebeganwiththegoalofpreventingobesityin1stthrough3rdgraders
Yourturn
____________________________________________________________________________
____________________________________________________________________________
Designimplementationandevaluation
44
SpecifythetargetpopulationAreyoudesigninganinterventionthattargetsonlycertainsitesAreyouplanningonimplementingaprogramorinitiativeforaspecifictargetaudienceClearlyspecifythepopulationyouplantotarget
IfyouplantoconductarigorousimpactevaluationasnotedbytheWilsonSheehanLabforEconomicOpportunitiesitisimportanttoestablishconcreteobjectiveeligibilitycriteriaandensurethatthesecriteriaareappliedconsistentlybyallstaffduringscreeningandintakeItisalsovitaltoidentifytheindividualswhoareeligibleforaparticularprogramorserviceevenifeveryoneeligiblecannotbeservedbyexistingresourcesThishelpscitiesidentifyanyunmetneedsandguidefutureresourceallocationwhilealsocreatingthepossibilityofarigorousimpactevaluationbyidentifyingapotentialcomparisongroupFinallywhenconductingscreeningandintakeitisimperativetocollectandmaintainbasicinformation(egnamedateofbirthgender)aboutallprogram-eligibleindividualsnotjustthosewhoparticipateintheprogram
ExampleIntheShapeUpSomervilleexampletheprogrambeganbytargeting1stthrough3rdgraders
Yourturn
____________________________________________________________________________
____________________________________________________________________________
DefinethelocationWherewillthepolicychangetakeplaceWillimplementationtakeplaceatonesiteormultiplesites
ExampleShapeUpSomervilleinterventionsweredeliveredinschools
Yourturn
____________________________________________________________________________
____________________________________________________________________________
ClarifyimplementationpartnersandarticulatetheirrolesWhatarethekeyorganizationsinstitutionsorpeopleresponsibleforimplementationTheprojectteamshouldbeasclearandspecificaspossibleinprovidingadescriptionoftheconcreteactivitiesthatimplementationpartnersaremeanttocarryoutPracticeprofilesmaybeahelpfultooltohelpdefinewhatisrequiredoftheactorswhoareresponsibleforimplementingtheintervention
Designimplementationandevaluation
45
ExampleInSomervillemayoralleadershipandengagementofmultiplecityagencies(publichealthprimarycareschoolsCambridgeHealthAllianceetc)streamlinedandconnectedthevariousShapeUpSomervilleinitiativesandgrantswhichhelpedtokeepthetalentandenergyfocusedonhealthyeatingandactivelivingforallsectorsofSomervilleSUSpartnersultimatelyincludedschoolsnonprofitsCityandstateagenciesandnationalfoundations
Yourturn
____________________________________________________________________________
____________________________________________________________________________
DescribetheinterventionWhatarethestepsactivitiesorprocessesRememberthatthemechanismthespecificstepsorprocessthatleadtobehaviorchangeiskeytounderstandingtheeffectivenessofthepolicychangeandsharingtheresultswithothers
ExampleGoalsofShapeUpSomervilleweretoinfluenceeverypartofanearlyelementaryschoolchildrsquosdayndashbeforeduringandafter-schooltopromotehealthyeatingphysicalactivityandtheirweightstatusInterventionincludedfoodserviceenhancementstoimprovebreakfastandlunchandstaffprofessionaldevelopmentwalktoschoolactivitiesSUSclassroomcurriculumandprofessionaldevelopmentschoolwellnesspolicydevelopmentSUSafter-schoolcurriculumandprofessionaldevelopmentoutreachandeducationtohomethroughmaterialsforumseventsldquoSUSapprovedrdquorestaurantscommunityoutreachandcapacitybuildingthroughpolicydevelopmenttrainingsmediaplacements
Yourturn
____________________________________________________________________________
____________________________________________________________________________
ClarifytimelineandmilestonesfordeliveryandevaluationHowfrequentlywilltheservicesinterventionbedeliveredHowlongwilltheprogramrunWhatisthetimetoexpectedimpacts(eg12monthsafterprogramcompletion)
Designimplementationandevaluation
46
ExampleShapeUpSomervilleledbyTuftsUniversityFriedmanSchoolofNutritionScienceandPolicyconductedaquasi-experimental(nonrandomized)interventionstudyamong1stthrough3rdgradestudentsinSomerville(intervention)andtwocomparisoncommunitiesfor9-monthsduringoneschoolyear(fall2002ndashspring2003)ThestudyshowedpositiveassociationoftheinterventiononchangeinBMISUSreleasedreportssummarizingevaluationsprogresstodateplansforadditionalprojectsandimpactstenandfifteenyearsintotheinitiative
Yourturn
____________________________________________________________________________
____________________________________________________________________________
PlanyouranalysisTheprevioussectionaddressedestablishinganaccountabilityframeworktomonitorprogressandimplementationDeterminewhichindicatorswillbediscussthroughthatprocesswhowillsharethedataandanalysisandhowfeedbackfromthoseconversationswillbeincorporatedbytheprojectteam
ExampleTheMayorrsquosofficeimplementedadata-drivendecisionmakingapproachcalledSomerStatwhichwassynergisticwiththeapproachestakenbytheresearchandevaluationteamsdeployedforthespecificgrants
Yourturn
____________________________________________________________________________
____________________________________________________________________________
PlanforsustainabilityIfthepolicyandprogramchangeissuccessfulhowwilltheprojectteamsustainandexpandtoadditionalsitesorpopulations
ExampleAlthoughmostofthepartnershipsforgedintheearlyyearsweremaintainednewgrantsandcollaborationsexpandedattentiontoolderpublicschoolstudentsactivetransportationandtheGreenLinesubwayextensionthebuiltenvironmentfarmerrsquosmarketeffortssuchasldquoGrowninSomervillerdquowithUnionMainStreeteconomicdevelopmentandhealthdisparitiesaswellasdatacollectionmonitoringandsurveillance
Yourturn
Designimplementationandevaluation
47
____________________________________________________________________________
____________________________________________________________________________
Pilotplanningtemplate
IfyouplantoconductapilotfirstdeterminethepurposeofthepilotDoyouwanttoobservewhetherthepolicychangeresultsinthedesiredbehaviorchangeAreyouconcernedaboutfeasibilityscalabilityorcostDoyouwanttotestwhetheryourdesiredresultscanbeachievedthroughthispolicyapproachClarifythegoalofthepilotandthequestionsyouhopetoanswerThiswillhelpyouidentifythekeyindicatorsthatyouwillusetomeasurewhetherthepilotissuccessfulBeforebeginningestablishabaselineofcurrentperformanceandsetyourtargetsaccordinglybasedonwhatyouhopetoaccomplishthroughthepilot
NextestablishthepilotplanforyourprojectteamincludingmilestonesforteamcheckinstodeterminewhethertomoveaheadorchangecourseThisisseparatefromthemeetingsthatwilloccuraspartoftheaccountabilityframeworkUsethespacebelowtocreateyourtimelineanddevelopyourmeetingschedule
Task Personresponsible Startdate Checkindate Deliverable
Planyourevaluation
WhileyouaredesigningtheimplementationitisalsotimetobegintodevelopyourplanforanalysisAsnotedbytheWilsonSheehanLabforEconomicOpportunitiesdecisionsaboutprogramsetup-includingoutreachscreeningintakedatatrackingandenrollment-candeterminewhetherornotarigorousimpactevaluationwillbepossibleAsdiscussedcityleadersmayfindithelpfultoincludeevaluationpartnersinearlyconversationsaboutprogramsetupDuringpreviousstagesyoumayhaveworkedwitharesearchertoplanyourproposedpolicychangeYoushouldbeclearaboutthepurposeoftheevaluationanddevelopanevaluationplanBelowareadditionalconsiderations
Causationvstrendanalysis
IntheEstablishaframeworktomonitorprogressstagetherewasguidanceontrackingtrendsovertimeItisimportanttonoteherethatmonitoringdataisdifferentfromanalyzingacausaleffectTrendanalysiswillillustratethatchangesareoccurringandmayleadtoquestionsthatcouldbeansweredbyyourevaluationbuttrendanalysisalonedoesnotprovideevidenceaboutwhatisdrivingthosechangesYoumayseechangesinyour
Designimplementationandevaluation
48
outcomeindicatorsbasedonvariablesoutsideoftheCityscontrolForinstanceachangeinfederaltradepolicycanhaveanimpactemploymentratesinyourCityoutsideofyourCitysnewjobtrainingandplacementprogram
ToaddresscausalityyoumustcomparetheoutcomesofindividualswhoreceivedtheinterventionwiththosewhodidnotWithoutacomparisongroupofsimilarindividualswhowerenotexposedtotheinterventionitisimpossibletoknowwhichfactorsareresponsiblefortheoutcomesRandomizedControlTrials(RCTs)inwhichindividualsarerandomlyassignedtotreatmentandcontrolgroupsarethegoldstandardinprovingcausalityIfyoucandemonstratethatoutcomesimprovedforthosewhoreceivedtheinterventionanddidnotimproveatallorimprovedlessforacomparisongroupyoucanbesomewhatcertainthattheinterventionwasthecause
Examplesofothernon-causalimpactevaluationsthatarerigorousandreliableincludecertainquasi-experimentalstudiesusingcomparisongroupdesignssmartinterruptedtimeseriesanalysesthatmodeltrendsovertimeinwaysthatallowestimationofinterventioneffectswelldesignedpre-posttestanalysesneedsassessmentsimplementationstudiesetcFormoreinformationondifferenttypesofevaluationsseetheSageResearchMethodswebsite
Lookfordifferencesingroups
BuildintimetoalsoexaminethepilotdatatoseeifgroupsareexperiencingdifferentresultsReviewingperformancebasedonoutcomemeasuresforapopulationinaggregatemayhidethesubtlewaysthatsomepeoplemaybeaffecteddifferentlysoitisnecessarytocarefullyanalyzethedatafordisparateoutcomesForinstancetheMovingtoOpportunity(MTO)studythatexaminedtheimpactofhousingvouchersonfamilieswholivedinhigh-povertyneighborhoodsfoundthatchildrenwhomovedbeforetheageof13hadbettereducationandhigherincomesasadultsHoweveradultsdidnotseeimprovededucationandemploymentoutcomes
Disaggregatingdataintomorespecificcategoriesofraceandethnicitycanhelpunderstandtrendsanddisparitiesinamoregranularwaywhichcanhelpdecisionmakerstargetpoliciesandinterventionsinamoreequitableway
EarlierintheRoadmapweworkedtoidentifystakeholdersandwhomightbepositivelyandnegativelyimpactedbyachangeinpolicyUsetheresultsofthatexercisetoinvolvethecommunityininterpretingdataanalysisandtogatherqualitativefeedbackfromthecommunitytocallattentiontoissuesForinstanceanincreaseinyouthdetentionmaybeexplainedbyinefficienciesorlackofaccesstotransportation
Designimplementationandevaluation
49
Designimplementationandevaluation
50
Shareyourstoryandsuccess
Sharetheresults
Storytelling
AnintegralstepinyourstrategyissharingtheresultsofyourpilotorpolicychangewithyourcommunityThisincludesdevelopingmessagingaroundtheworkofyourorganizationandcreatingaplantodisseminateyourstoriesWhytellstoriesaboutyourwork
Informandorunderstandasubjectorsituation
Giveneededcontextofdataorbitsofinformation
Relatablestoriesarememorableforreadersviewers
ReadersorviewerswillseetheinformationasmorecompellingwhenthereisademonstratedandrelatablechangeorimpactonthemselvesortheirfriendsneighborsandcolleaguesStoriesareaboutpeopleandgovernmentisfundamentallyabouthelpingpeopleSotellthestoriesofthepeopleyouandyourcitycolleagueshelped
Keepthesetipsinmindasyoudevelopyourstorywithyourteam
WhatwereyourfindingsandwhyshouldanyonecareTellyouraudiencehowtheproblemwassolvedWhathastheimpactbeenonresidentsandcitystafftodateWhatwasthesuccessandweretheremultiplesuccessesalongthewayIfyouhavepreliminaryresultsordatashowingimprovementtosharethisisthetimetodoit
WhowasinvolvedandimpactedReviewthelistofstakeholdersHowwereeachofthemimpactedbytheprogramHumanizeyourstoryandimpactbyaddingadditionalvoicesThesevoicescanbeoneofthefollowing
Apersonorfamilywhowaspositivelyaffectedbythechangesuccess
Aninvolvedthird-partydescribingboththeproblemandthesolution(ieanorganizationnonprofitprovidersbusinesstradeassociationsportsteametc)whoispartofthecommunitythatthegovernmentworkedwithonthechangesuccess
Someonefromthegovernmentitselfwhocanspeakaboutthechangesuccess
TellmoreaboutyourcityDescribeyourcityincludingpopulationdemographicsindustriesetcWhatareitschallengesWhatisitknownforWhatmakesituniqueHowisitsimilarordifferentfromothercitiesintheUS
Shareyourstoryandsuccess
51
DescribeyourcityrsquoschallengeWriteoutthechallengeyourcityisfacing(asdefinedinyourproblemstatement)HowdoesitaffectresidentsHowdoesthisproblemaffecttheworkofcitystaffmembers
AddthedetailsProvidedetailsonyourappliedresearchstrategyTalkaboutwhatyourcityhasdonenthepasttoresolvetheissueDiscusshowyourstrategyhaschangedwhenyoubegantoadoptthisapproachWasmoneysavedorinvested
Usethisstorymapexercisetoorganizeyourthoughtsandguaranteethatyouincludealloftheelementsofatraditionalstory
AfteryoudevelopyourmessagingcreateacommunicationsmarketingandoutreachstrategydesignedtospreadyourmessageYourstrategyshouldwalkyouthroughthefollowingkeyactivities
1 IdentifyandprofileyouraudienceWhencreatinganddisseminatingyourmessageitisalwaysgoodtothinkaboutyouraudienceWhoexactlywouldbeinterestedinwhatyouhavetosayThegeneralpopulationofthecityPeoplewhoworkinnonprofitsBusinessownersItmattersbecausenoteveryonewholivesinacitymaybethetargetoftheinformationBusinessownersmaybemoreinterestedindatashowingeffectsonthebusinessdistrictthanothersandsoonWewillcoverthislaterbutthisstepwillalsohelpyoutothinkofthemosteffectivewaytoreachyouraudience
2 Defineyourcalltoaction(ifyouhaveone)SpendsometimeconsideringwhyyouareinterestedintellingyourstoryDoyousimplywanttoupdatethepubliconyourprogressorcelebrateyoursuccessesAreyouinterestedinyouraudiencechangingitsbehaviororsupportingaspecificinitiativebeinglaunchedbyyourorganizationIfyouwanttohaveyouraudienceactorbehaveinacertainwaybespecificandincorporateitinyourmessaging
Shareyourstoryandsuccess
52
3 SelectperformancemeasuresThinkabouthowyouwoulddetermineifyourcampaignissuccessfulandselectappropriateperformancemeasuresYouwillrevisitthesemeasuresattheendofyourcampaigncommunicationsplanimplementationtodeterminewhetheryouneedtomodifyyourcommunicationsandoutreachstrategyforthefuture
4 MethodstocommunicateyourstoryTherearemanywaystocommunicatesuccessofyourprojectHerearesomesuggestionsonreachingaudiencesItisimportanttodeterminewhichmethodsarethemosteffectiveforyourdesiredaudiencetoreceiveinformationDoyourresearch(PewResearchCenterhasexcellentstudiesonthereachofmediadigitalmediaandmarketing)ForexampleifseniorsareyourtargetpopulationandtheyarelessinclinedtouseInstagram(thatistrue)youmaywanttoconsiderFacebookorthenightlylocalnewsbothofwhichareusedmoreoftenbyseniorsBelowisalistofsomemethodsandtoolsyoucanuse
i Pressreleasestolocalnewspapersandtvstations
ii ShortsmartphonevideosAgreatwaytoengageanaudienceisthroughvideoAskonequestionofsomeonewhoispartofthestoryMaybethereisasocialworkerwhocandiscusshungerinyourcityThesevideoscanbereusedonmultipleplatforms
iii Long-formwriting
iv TellingindividualstoriesthroughaseriesofphotosandshortnarrativesforpublicationonFacebookorInstagram
v Op-edsopinionpiecesplacedinthelocalpaper
vi Emailupdatestoyourlist
vii Connectingwithappropriateorganizationsthatareinvestedinthetopicandworkwiththecommunityandaskthemtodisseminateinformationtotheirmembers
viii EmployingldquoWordofMouthrdquo(WOM)tacticsbycreatingtownhallsorfreeeventsforresidentstoattendintheirownneighborhoods
5 DeterminethetypeofcollateralUsuallycampaignsthattargetresidentsrequireseveralavenuestoreachthemPeoplearenotalwaysinthesameplaceorspaceHavingakeenunderstandingofyouraudiencewillassistyouwithyourchoicesonwhatneedstobedevelopedSometimesthatmeansimagesforsocialmediaaonepageflierthatgivespeopleinformationonattendinganeventoranin-depthbrochure
Shareyourstoryandsuccess
53
6 ImplementyourstrategyKeepinmindstrategiescanbefluidSometimessomethingmaynotturnoutasexpectedevenifduringplanningsessionsitwasldquosupposedtordquoItisokYoucananalyzewhathappenedduringadebriefsessionandonceyoursurveysandothermeasurementshavebeenfinalized
Visualizations
Withtherightvisualizationyoucanreviewandshareinformationinsightsanalysisperformanceandtrendsallofwhichisvaluabletoacross-sectionofthepublicButwhichvisualizationisrightforyouTochoosetherighttypeofchartgraphdeterminethetypeofquestionyouwanttoanswerSomecommoncharttypesarebelowandmoreinformationcanbefoundhere
Exploreadditionalanalyticaltechniques
Duringthegoal-settingphaseyouestablishedoutputoutcomeandimplementationmeasuresBasedontheresultsofyourpilotyoumayneedtoadjustyourmeasuresWiththepilotdatainhandyoumayhavesomeideasofhowfutureanalysescouldbedifferentandpotentiallyimprovethequalityofyourprogramTakesometimetobrainstormotherrelevantdatasetsthatyoudidnothavethetimetopursueearlierincludingmessydepartmentaldatathatneedsmoreattentionorsourcesofpublicdatafromthestateandfederalgovernmentnonprofitsandevenprivatecompaniesthatcouldsupplementyourunderstandingofanissueUsingmachinelearningtechniquesyoumightbeabletobetter
Shareyourstoryandsuccess
54
predictdemandforhomelessservicesusingNOAAweatherforecastspreemptwatermainbreaksusingsoilstratigraphyorinvestineconomicdevelopmentbymonitoringrealestatelistingdata
Advancedanalyticaltechniquescanalsohelpyourimplementationrunmoreeffectivelybyintegratingdata-informedtargetinganddecisiontoolsintotheprocessByautomatingroutineworkflowsandsurfacingsubtletrendsthesetoolscanhelpyouiterateinamoreprecisemannerandprovidekeystrategicguidancetofront-lineemployeesthatwouldotherwisebeunavailableThesesortsofdata-science-enhancedprocessescanbeclassifiedasoneofsixprojecttypesasexploredfurtherbytheNewOrleansLANOLAlyticsteamFormoreguidanceonscopingasimilarprojecttheGovExadvancedanalyticsscopingguidegivesyouastep-by-stepoverviewofhowtobesuccessful
Usethespacebelowtolistyourideasforpossibleadvancedanalyticsprojectsbytype
Typeofproject Ideas
Findingtheneedleinthehaystack
Prioritizingforimpact
Earlywarningtools
Betterquickerdecisions
Optimizingresourceallocation
Experimentingforwhatworks
Keepthefeedbackloopgoing
GatherinformationonpublicsentimentaboutpolicychangethroughfocusgroupssurveysandpublicmeetingsledbyadvisorygroupmembersNotewhethernewinformationisavailablethatcanhelpanalyzeorcontextualizeyourprogress(egfederalorstatedatareports)Sharetheresultsofyourpilotandthepolicychangeprocesswithothercitiesandresearchers
Shareyourstoryandsuccess
55
WorkingwitharangeofevidencePresentlythereisalotofattentionandinvestmentinevidence-basedpolicymakingincludingthefederalbipartisanCommissiononEvidence-BasedPolicymakingthePewCharitableTrustsreportsandtoolkitsonHowStatesEngageinEvidence-BasedPolicymakingandtheLauraandJohnArnoldFoundationsupportedEvidence-BasedPolicymakingCollaborativeManyoftheseinitiativesfocusonrigorousevaluation--meaningrandomizedcontrolledtrials(RCT)orotherexperimentalmodels--asthebestkindofevidencetouseHoweverthisbodyofevidenceisofteninsufficientforthespecifickindsofquestionsthatgovernmentscanhavewhenmakingadecisionaboutpolicyInthisguideweproposeanapproachtosupportgovernmentsinincorporatingadditionaltypesofevidenceintotheirdecisionmaking
OnewaytounderstandarangeofevidenceistouseNestarsquosStandardsofEvidencetoframeanddefineevidenceTheBritishnonprofitNestaworkstoinjectpositivechangeintothewayUKgovernmentpoliciesandprogramsfunctionOneoftheirkeyareasofworkhasbeendevelopingWhatWorksCentresorganizationsthatcollectcommissionsynthesizeanddisseminateevidenceaboutinterventioneffectivenessacrossarangeofsocialimpactareasNestadevelopedthefollowingStandardsofEvidencetoprovideausefulwaytodocumenthowdifferentkindsofinformationdemonstratedifferentlevelsofevidence
Workingwitharangeofevidence
56
Theimportanceofpotentiallybiasedinformation
Onechallengepresentedbyanevidenceclassificationthatacknowledgesthevalueofnon-scientificformsofevidenceistofigureouthowtobalanceallofthepiecestogetherwhenyouusethemTounderstandhowtodothiswehavetothinkabouthowtopositionpotentiallybiasedinformationindecisionmaking
WhenwelookatinformationwhichhasnotbeenrigorouslyevaluatedweacceptthatthisinformationmightbebiasedThewholereasonweconductrigorousevaluationistoremovebiasfromourobservationsWeconductstructuredtrialstoproduceresultsthatareconsistentandgeneralizableIfafindingisunbiasedthisaccuracymdashthislackofbiasmdashcanbeprovedbecauseanyindependenttestercanreproduceitwhentheyfollowanidenticalprocess
YoucanlookatpotentiallybiasedinformationasjustinformationthatisearlieroninitspathtorefinementreplicationandverifiedaccuracyApieceofinformationbasedonanyoneindividualrsquosobservationsislikelytobebiasedbecauseitiscomingfromjustonesourceBiasisreducedbygatheringtogetherlotsofindependentsourcesofinformationandfiguringoutitsmostcommonattributes
MaybethebiasedinformationmakesaclaimthatistoolargemdashitsayssomethingistrueunderallconditionswhenitisreallyonlytrueundercertainconditionsorthataneffectofaninterventionisbiggerthanitstrueaverageorthataneffectlastsovertimewhenitdoesnotPerhapschangingtheclaimtobemorespecificmdashcriticallyreviewingtheinformationtoseeifspecificconditionsmustbemettoachievetheoutcomesmdashwouldmaketheclaimmoreaccurate
IfyoutestedyourclaimfounditdidnotholdupinsomecaseschangedittesteditagainuntilyourclaimrepeatedlyheldupthenyoucoulddemonstratethattheclaimwasnolongerbiasedbutaccurateThisprocessoftestingimprovingtestingandvalidatingiswhatwegetfromstructuredscientificstudy
Withindividualpiecesofinformation--crediblehypothesescasestudiesexpertopinion--youcannotassumethatkindofaccuracyAtthesametimetheseearlyobservationsarefoundationaltoanyreplicablescientificobservationsWewillnevergetanywhereifwedonotstartwiththesepiecesofinformationeveniftheymightbebiased
Combininglevelsofevidencetogetthemostdetailedinformationforyourcity
ItisespeciallyimportanttoacceptdifferentformsofevidencewhenyoufacequestionsthathavenotyetbeentestedbyindependentstudiesSomeofthemostimportantquestionsrelatetothingsthatareveryspecifictoyoursituationYoufaceaparticularcombinationof
Workingwitharangeofevidence
57
issuesactorstrendsandresourcesthatisultimatelydifferentfromeveryotherplaceintheworld
ThisisimportantbecauseaspolicypractitionersandnotscientistsyoucaremoreaboutyourspecificcasethanahypotheticalaveragegeneralcaseYoucarelessaboutthetotallikelyrangeofimpactsfromapolicythanhowthissingleimplementationwillworkYouwantinformationaboutpoliciesthatwillprovideyouwiththebestpossiblepredictionabouthowtheywillworkinyourcontext
ByputtingdifferentkindsofevidenceintothesameframeworkNestarsquosStandardsofEvidenceimplythatwhenyoucannotgetrigorouslytestedexperimentallyverifiedinformationitisappropriatetoworkwithwhatwedohaveThisisrelevantforthemunicipalimplementationcontextbecausenostudypulledfromaliteraturereviewwillaccountforallofthevariablesyoufaceinyourowncontext
Intheabsenceofhavingastudythatpredictshowaninterventionworksinyourspecificcontextyouwillneednon-scientificinformationtotrytofillinthosegapsYouwillneedtogatherstoriesfromresidentsstaffmembersandotherstakeholdersthatsitonLevels1or2ofthestandardsofevidence
Someofthekindsofinformationyouwillprobablygetmoreeasilyfrompeoplethanstudiesinclude
howpeopleincitieslikeyoursreactedtoaninterventioninpublicforumshowpeoplewhoimplementedaninterventioninacitylikeyoursfiguredouthowtogetitpassedhowthecitycommunicatedpubliclyabouttheapparentbenefitsanddrawbacksoftheinterventionthepoliticalacceptabilityoftheintervention
Youcanremainsimultaneouslyawarethattheinformationyouaregatheringthroughindividualstoriesisbiased--notgeneralizableinformedbyspecificperspectives--whilealsorecognizingthathavingsomeinformationaboutsomethingimportantisbetterthannoinformationEvidencefromthefirsttwoevidencelevelsareessentialbuildingblocksofknowledgeandperfectlyusefulwhenyoucannotgetdetailedunbiasedknowledge
Youhavetostartsomewhereandyoushouldalwaysstartwithevidenceyouhaveaboutyourveryspecificconditionsthatyouknowareimportantforsuccess
Workingwitharangeofevidence
58
Thankyou
Interviewswithcities
GovExheldsemi-structuredinterviewswithsixstaffmembersinfourcitiestoestablishtheinitialstepsoftheRoadmapWeaskedrespondentstotalkaboutaninitiativeorpolicythattheyhavebeenresponsibleforimplementingintheircityRespondentsfromthreecitiesspokeabouttheireffortstoaddresshomelessnessandrespondentsfromonecityspokeaboutblightAfterrespondentsidentifiedtheproblemtheyspokeabouthowthecityprioritizedthisissuethevariousstepsinvolvedandbarrierspresent
GovExconductedtextanalysisoftheseinterviewswhichrevealedthatpeoplewasthemostcommonlyusedtermduringtheseconversationswhendiscussingthestepstochangeaprogramorpolicyThisisathemethroughoutthisRoadmapThefirstsectionbeginswithidentifyingtherelevantpeopleTheconversationsaroundevidencearefocusedonhumanbehaviorratherthanmethodologyorstrengthofevidence
Thankyou
59
OncewehadadraftoftheRoadmapwefollowedupwithcitiesonanindividualbasistolearnmoreaboutwhatwouldbehelpfulWearegratefultothefollowingcitystafffortheinterviewsandindividualconversations
MichaelBaskinChiefPolicyOfficerChattanoogaTN
MelissaBridgesPerformanceandInnovationCoordinatorLittleRockAR
VictoriaCarreoacutenAdministrativeOfficerLasVegasNV
TyroneGrandstrandCommunityDevelopmentPlannerFargoND
TibLaughlinDirectorofGeneralServicesKansasCityKS
TimMorelandDirectorofPerformanceManagementandOpenDataChattanoogaTN
CharlesRosendahlManagementAnalystDentonTX
Thankyou
60
MelissaSiebenAssistantCountyAdministratorKansasCityKS
GinaThompsonDirectorofStrategicServicesLewisvilleTX
JHUFacultycollaboration
GovExalsoconvenedthecitycontactsandfacultypartnersinaseriesofAdvisoryGroupmeetingsThesemeetingsservedtoconnectcitiesandfacultytounderstandoneanothersperspective
StefanBaralBloombergSchoolofPublicHealth
AshleyBernerJHUSchoolofEducation
MelissaDavey-RothwellBloombergSchoolofPublicHealth
JonLinksBloombergSchoolofPublicHealth
TamarMendelsonBloombergSchoolofPublicHealth
KristinMmariBloombergSchoolofPublicHealth
KeshiaPollackPorterBloombergSchoolofPublicHealth
JoshSharfsteinBloombergSchoolofPublicHealth
LizStuartBloombergSchoolofPublicHealth
LindsayThompsonCareyBusinessSchool
DanielWebsterBloombergSchoolofPublicHealth
Partnersindataandevidence-basedpolicy
Wealsospokewithpartnerswhoareexpertsinthefieldofevidence-basedpolicywhosharedtheirknowledgeandresourcesofcurrentandfuturethinkinginevidence-informedthinking
WendyBarrenoAssociateDirectorWilsonSheehanLabforEconomicOpportunitiesUniversityofNotreDame
TiffanyDavenportDirectorofDataPartnershipsBaltimoresPromise
TomiHiersDirectorBaltimoreCivicSiteAnnieECaseyFoundation
ChrisKingsleySeniorAssociateAnnieECaseyFoundation
KathrynOliverAssociateProfessorLondonSchoolofHygieneandTropicalMedicine
Thankyou
61
KristenWatkinsSeniorPolicyAssociateJ-PAL
GovExParticipants
GovExAnalystMiriamMcKinneyanalyzedtheinterviewtranscriptsandconductedtextanalysisSeniorImplementationAdvisorEmilyShawwillintegrateRoadmapactivitieswithadditionalservicedeliveryactivitiesatGovEx
Thankyou
62
ReferencesAbrahamKGampHaskinsR(2017)Thepromiseofevidence-basedpolicymakingReportofthecommissiononevidence-basedpolicymakingWashingtonDCCommissiononEvidence-basedPolicymakingBigmove2Extendingthegreen(nd)Retrievedfromhttpswwwcityoflewisvillecomdoing-businesslewisville-2025big-move-2-extending-the-greenBatesMAampGlennersterR(2017)ThegeneralizabilitypuzzleRetrievedfromhttpsssirorgarticlesentrythe_generalizability_puzzleBuildwithnotfor(2018January)RetrievedfromhttpsbuildwithnotforwordpresscomCairneyPampOliverK(2017)Evidence-basedpolicymakingisnotlikeevidence-basedmedicinesohowfarshouldyougotobridgethedividebetweenevidenceandpolicyHealthResearchPolicyandSystemsdoi101186s12961-017-0192-xCairneyPOliverKampWellsteadA(2016)TobridgethedividebetweenevidenceandpolicyReduceambiguityasmuchasuncertaintyPublicAdministrationReview76(3)399-402Retrievedfromhttpdxdoiorg101111puar12555Centreforpublicimpact(nd)RetrievedfromhttpswwwcentreforpublicimpactorgCernaL(2013)ThenatureofpolicychangeandimplementationAreviewofdifferenttheoreticalapproachesOrganisationforEconomicCo-OperationandDevelopmentRetrievedfromhttpswwwoecdorgeduceriTheNatureofPolicyChangeandImplementationpdfColoradoSprings(nd)Retrievedfromhttpscoloradospringsgovcommunicationspagewhat-expect-when-you-speakupContinuumofcare(coc)program(nd)RetrievedfromhttpswwwhudexchangeinfoprogramscocCouncilpresidentkicksoff$12millionyouthfund(nd)Retrievedfromhttpwwwbaltimorecitycouncilcomcontentcouncil-president-kicks-12-million-youth-fundDurlakJAampDuPreEP(2008)ImplementationmattersAreviewofresearchontheinfluenceofimplementationonprogramoutcomesandthefactorsaffectingimplementationAmericanJournalofCommunityPsychology41327-350Retrievedfromhttpswwwncbinlmnihgovpubmed18322790Fishbonediagram(ishikawa)(nd)Retrievedfromhttpasqorglearn-about-qualitycause-analysis-toolsoverviewfishbonehtmlFivewhysGettingtotherootofaproblemquickly(nd)RetrievedfromhttpswwwmindtoolscompagesarticlenewTMC_5Whtm
References
63
FligsteinNampMcAdamD(2011)TowardageneraltheoryofstrategicactionfieldsSociologicalTheory29(1)1-26RetrievedfromhttpsociologyberkeleyedusitesdefaultfilesfacultyfligsteinFligstein20McAdam20Soc20Theory20PaperpdHeinkelP(2018January12)CitymullspermitsforbikesharingcompaniesRetrievedfromhttpwwwdentonrccomnewsdenton20180112city-mulls-permits-bike-sharing-companiesHowisareamedianincomecalculated(2012October)Retrievedfromhttpswwwhudexchangeinfofaqs843how-is-area-median-income-calculatedLampkinLMampHatryHP(2003)KeystepsinoutcomemanagementUrbanInstituteRetrievedfromhttpswwwurbanorgresearchpublicationkey-steps-outcome-managementMarshDampSmithM(2000)UnderstandingpolicynetworksTowardsadialecticalapproachPoliticalStudies484-21Retrievedfromhttpjournalssagepubcomdoiabs1011111467-924800247McCannL(nd)RetrievedfromhttplaurenellencomMoultonSampSandfortJR(2016)ThestrategicactionfieldframeworkforpolicyimplementationresearchPolicyStudiesJournal45(1)Retrievedfromhttponlinelibrarywileycomstore101111psj12147assetpsj12147pdfv=1ampt=jdkhhcp4amps=e184c625e1855a6b1615ccb99ee27b1c0b3212d8PaynterB(2018January1)What100citiesarelearningfromeachotherbysharingtheirdataRetrievedfromhttpswwwfastcompanycom40518004what-100-cities-are-learning-from-each-other-by-sharing-their-dataPeakacademy(nd)Retrievedfromhttpswwwdenvergovorgcontentdenvergovenmayors-officeprograms-initiativespeak-performancepeak-academyhtmlPinnockHBarwickMCarpenterCREldridgeSGrandesGGriffithsCJRycroft-MaloneJMeissnerPMurrayEPatelASheikhAampTaylorSJC(2017)Standardsforreportingimplementationstudies(stari)ExplanationandelaborationdocumentBMJOpenRetrievedfromhttpbmjopenbmjcomcontentbmjopen74e013318fullpdfProctorESilmereHRaghavanRHovmandPAaronsGBungerAGriffeyRampHensleyM(2011)OutcomesforimplementationresearchConceptualdistinctionsmeasurementchallengesandresearchagendaAdministrationandPolicyinMentalHealth38(2)65-76doi101007s10488-010-0319-7TandonSD(2018)ProgressincommunityhealthpartnershipsResearcheducationandactionRetrievedfromhttpswwwpressjhuedujournalsprogress-community-health-partnerships-research-education-and-actionProvanKGampKenisP(2007)ModesofnetworkgovernanceStructuremanagementandeffectivenessJournalofPublicAdministrationResearchandTheory
References
64
18(2)229-252Retrievedfromhttpsacademicoupcomjpartarticle182229935895SabatierPA(1988)Anadvocacycoalitionframeworkofpolicychangeandtheroleofpolicy-orientedlearningthereinPolicySciences21(3)129-168Retrievedfromhttpwwwjstororgstable4532139SandstromAampCarlssonL(2008)TheperformanceofpolicynetworksTherelationbetweennetworkstructureandnetworkperformancePolicyStudiesJournal36497-524Retrievedfromhttponlinelibrarywileycomdoi101111j1541-0072200800281xabstractSchwartzMampWilsonE(nd)WhocanaffordtoliveinahomeAlookatdatafromthe2006americancommunitysurveyRetrievedfromhttpswwwcensusgovhousingcensuspublicationswho-can-affordpdfShearerJCAbelsonAKouyateBLavisJNampWaltG(2016)WhydopolicieschangeInstitutionsinterestsideasandnetworksinthreecasesofpolicyreformHealthPolicyandPlanning31(9)1200-1211Retrievedfromhttpsdoiorg101093heapolczw052ThelessonsofadministrativedataHigh-profilepolicy-relevantresearchpoweredbyadministrativedata(nd)Retrievedfromhttpswwwpovertyactionlaborgsitesdefaultfilesresources20161114-The-Lessons-of-Administrative-DatapdfToolsandotherneedsforqualitativeandquantitativemethods(2015February)AgencyforHealthcareResearchandQualityRetrievedfromhttpwwwahrqgovprofessionalsquality-patient-safetytalkingqualityassesstoolshtml
WeaverRCampBagchi-SenS(2013)SpatialanalysisofurbandeclineThegeographyofblightAppliedGeography4061-70
References
65
UniversityPartners
Name Title Affiliation Area(s)ofexpertise
Contactinformation
Examplesofworkingwithnon-academicdecisionmakers
WendyPBarreno
AssociateDirector
WilsonSheehanLabforEconomicOpportunitiesUniversityofNotreDame
LEOsfocusareasincludehealtheducationhousingchildwelfareeconomicopportunitycriminaljustice
wbarrenonduduorleondedu
MelissaDavey-RothwellPhD
AssociateScientist
JohnsHopkinsBloombergSchoolofPublicHealth
Health mdavey1jhuedu410-614-5854
BaltimoreCityHealthDepartmentLeadersMarylandDepartmentofHealthAddictionRecoveryServicesinDCExecutiveDirectorsofcommunityorganizationsacrosstheUSthatfocusonHIVSTIsubstanceabuseandimplementationscience
ElizabethStuart
AssociateDeanforEducationandProfessorDepartmentsofMentalHealthBiostatisticsandHealthPolicyand
JohnsHopkinsBloombergSchoolofPublicHealth
Educationsocialpolicystatisticalmethods
estuartjhuedu
Resources
66
Management
Design-ThinkingResources
EngagementGuideforCivicTech
Journeymap
PerformanceManagementResources
DenverPeakAcademy
Proxymeasuresguide
Talkingaboutperformancewithoutcomplicatingit(video)
SettingPerformanceTargetsguide
Datavisualizationguide
Performancemanagementgettingstartedguide
Exercises
Fishboneexercise
5Whys
AdministrativeData
TheLessonsofAdministrativeData(JPAL)
EvaluationResources
Quantitativeandqualitativemethods(USDepartmentofHealthandHumanServicesAgencyforHealthcareResearchandQuality)
KeyStepsinOutcomeManagement(UrbanInstitute)
RCT-YES
Sageresearchmethods
BestPracticesforMixedMethodsResearchintheHealthSciences
ACCORDChecklist
Pilotandimplementationplanning
Benchmarkingexamplesandproxymeasures
Seattleopendataplaybook
Resources
67
Datasciencescopingguide
Resources
68
AboutGovExGovExhelpsgovernmentsusedataandevidencetoimproveresidentsrsquolivesWesupportgovernmentsinbuildingtheirinternalcapacitytooperatesustainableanalyticspracticeswhichimprovecommunityoutcomesinfieldsfrompublicsafetytoeconomicdevelopmentGovExhasaproventrackrecordofworkinginconcertwithpartnerstoprovidetechnicalassistanceandtrainingatscale
SinceourlaunchinApril2015GovExhas
Trainedover2000governmentemployeesinpersonandonlineintopicsincludingdatamanagementperformancemanagementopendatacommunityengagementandanalytics
Providedtechnicalassistancetomorethan100localgovernmentsacrosstheglobeimprovingtheiruseofdatatoachieveefficienciesinservicedeliveryandbetteroutcomesforresidents
Assessedthecapacityofmorethan130governmentalorganizationstodeterminehowleadersstaffandexternalstakeholdersmanageandusedata
Localgovernmentsareincreasinglyonthefrontlinesofaddressinglargescaleproblemswithglobalimplicationssuchasclimatechangepublicsafetyandeconomicinequality
Whileevidence-basedinterventionstohelpsolvethoseproblemsareavailablelocalgovernmentstaffhavelimitedtimeandcapacitytofindandimplementresearch-basedsolutionsAsaresultcitiescontinuetoworkthewaytheyalwayshave--makingincrementalimprovementstoprogramsbutnotalwayscreatingchangeatasystem-level
ThroughresourcedevelopmentandcollaborationwithJHUfacultyGovExissupportingcityleadersinusingdataandevidencetomakedecisionsandcreatinglastingrelationshipswithresearchers
TheRoadmapwasdevelopedbyGovExaspartofitspartnershipwithBloombergPhilanthropiesWhatWorksCitiesinitiative
AboutGovEx
69
- Title page
- Introduction and purpose of the Roadmap
- Using the Roadmap
-
- Gather information about the problem
- Strategically engage key partners
- Review the evidence and collect details
- Decide on a policy to implement based on local factors
- Establish a framework to monitor progress
- Assess data accessibility quality and security
- Design implementation and evaluation
- Share your story and success
-
- Working with a range of evidence
- Thank you
- References
- Resources
- About GovEx
-
NearlyanypieceofinformationaboutaninterventioncanbeclassifiedwithinthestandardsofevidenceForexampleifsomeonehasacrediblehypothesisabouthowaninterventioncouldproducebenefitthistypeofevidencewouldbeclassifiedasLevel1ThistypeofevidencecouldincludeinterventionsinothercitiesthatyoureadaboutinthenewsorhearaboutataconferenceForinstancewhenUSDAallowedschoolsandschooldistrictstoprovidefreebreakfastandlunchtoallstudentsthroughtheCommunityEligibilityProvision(CEP)manydistrictsimplementedthisoptionevenbeforetherewasevidenceofeffectivenessThiswouldfitintoLevel1sinceitbuildsonthelogicthatprovidingaccesstomealswilldecreasefoodinsecurity
YoumayalsofindpiecesofinformationthatqualifyasLevel2evidenceForinstanceasmoreschooldistrictsimplementedCEPprincipalstestifiedaboutpositivechangestheyobservedinstudentssuchasamorepositivedemeanorthatcanleadtobetteracademicperformanceDataalsoshowedthatthestudentsdescribedwerereceivingmoremealsthanbeforetheprogramsimplementationHoweveritwastoosoontobuildevidencetoshowacausalrelationshipbetweentheincreaseinfreemealsandincreasedacademicperformance
ldquoLower-levelrdquoevidencecanconsistofnuancedbutvaluableelementsthatrelatetoaninterventionrsquospotentialfitwithacommunityQualitativeinformationsuchastestimonyfrompeoplewhohaveexperiencedaninterventionandthosewhohavefacilitatedorimplementeditwillalwaysbeusefultoconsiderandprovidesdetailedlocalcontext
Experimentalstudies--describedstartingatLevel3withcontrolledexperimentsandgoingthroughLevel5withfullyfleshedoutprotocolsandprocessesforreplication--stilloccupythehigherlevelsofthisdescriptionofevidenceThisprioritizationshowsthevalueofrigorousstudyforevaluatingtheimpactofaparticularpolicyorprogramratherthanotherfactorsthatcouldbeinfluencingoutcomesofinterestRememberthatmultipletypesofevidencecancomplementrigorousimpactevaluationsandgivecityleadersadditionalinsightsandbroaderperspective
Accessinformation
Encourageyourreviewerstobegintheirsearchforpromisingpracticesingovernmentandnon-profitclearinghousesandinacademicpolicyjournalsWorkpublishedinthesesourceshasalreadybeensubjectedtoahighlevelofprofessionalscrutinyandgenerallycontainsagreatdealofdetailwhichmeansthatyourreviewersaremorelikelytobeabletopullouttheinformationyouneedtomakeagooddecision
Clearinghouses
Reviewtheevidenceandcollectdetails
23
Herearesomeoftheclearinghouses(collectionsofprogramevaluations)forevidence-basedpoliciesandprograms
ResultsFirstClearinghouseDatabaseThePewCharitableTrustsCollectstogethereightotherclearinghousestoprovideinformationoninterventionsinthefollowingcategoriesadultcriminaljusticeeducationchildwelfarehealthjuvenilejusticementalhealthsubstanceabuseandyouthdevelopmentIndividualclearinghousesincludedinResultsFirstare
BlueprintsforHealthyYouthDevelopment-foryouthdevelopmentandviolenceprevention
CaliforniaEvidence-BasedClearinghouse-forchildwelfareinterventions
TheLauraandJohnArnoldFoundationrsquosSocialProgramsThatWork-forsocialservicesprograms
TheUSDepartmentofEducationrsquosWhatWorksClearinghouse-foreducationinterventions
TheUSDepartmentofHealthandHumanServicesrsquoResearch-TestedInterventionPrograms(RTIPs)-forcancercontrolinterventions
TheUSDepartmentofHealthandHumanServicesrsquoTeenPregnancyPreventionEvidenceReview(TPPEvidenceReview)-forsexualriskbehaviorandsexualhealthoutcomes
TheUSDepartmentofJusticersquosCrimeSolutionsgov-forcriminaljusticejuvenilejusticeandcrimevictimservices
TheUSSubstanceAbuseandMentalHealthServicesAdministrationrsquosNationalRegistryofEvidence-basedProgramsandPractices(NREPP)-substanceuseandmentalhealthinterventions
TheUniversityofWisconsinPopulationHealthInstituteandRobertWoodJohnsonFoundationrsquosCountyHealthRankingsandRoadmapsWhatWorksforHealth(WhatWorksforHealth)-healthbehaviorsclinicalcaresocialandeconomicfactorsandthephysicalenvironment
UKWhatWorksCentresTheBritishgovernmenthascharteredanetworkofsubject-specificorganizationswhichcollectsynthesizeandreportoutevidenceontheeffectivenessofpolicyprogramsandpractices
NationalInstituteforHealthandCareExcellence(NICE)-healthandsocialcare
Reviewtheevidenceandcollectdetails
24
SuttonTrustEducationalEndowmentFoundation-educationalachievement
CollegeofPolicingWhatWorksCentreforCrimeReduction-crimereduction
EarlyInterventionFoundation-earlyintervention
WhatWorksCentreforLocalEconomicGrowth-localeconomicgrowth
CentreforAgeingBetter-qualityoflifeforolderpeople
WhatWorksCentreforWellbeing-wellbeing
WilsonSheehanLabforEconomicOpportunities(LEO)UniversityofNotreDameProvideslinkstotheirevaluationsofinterventionsincriminaljusticeself-sufficiencyeducationhealthandhousingandhomelessness
ClearinghouseforLaborEvaluationandResearchUSDepartmentofLaborProvidesresearchonworkforceandlabor-relatedpoliciesandprograms
J-PALEvaluationsAbdulLatifJameelPovertyActionLabStudiesonpoverty-reducinginterventionsfromaroundtheworld
CampbellSystematicReviewsCampbellCollaborationSystematicreviewofpolicyinterventionsbyscholarlycoordinatinggroupsinsevenissueareascrimeandjusticeeducationinternationaldevelopmentmethodssocialwelfareknowledgetranslationandimplementationanddisability
Benefit-CostResultsWashingtonStateInstituteforPublicPolicyAWashingtonState-charteredprogramtoestablishthelikelihoodofrelativecostsandbenefitsforprogramsacrossthefollowingareasjuvenilejusticeadultcriminaljusticechildwelfarepre-Kto12educationchildrensmentalhealthhealthcaresubstanceusedisordersadultmentalhealthpublichealthandPreventionworkforcedevelopmentandhighereducation
EffectiveInterventionsCentersforDiseaseControlandPreventionRepositoryforevidence-informedHIVpreventioncareinterventions(withstepsonhowtoimplement)
Journalsandindexes
OnceyouhavereviewedrelevantinformationaccessiblethroughclearinghousesitisworthtakingalookatindividualstudiespublishedasscholarlyarticlesYourreviewerswilldothisbysearchingarticleindexesandacademicjournalsUnfortunatelyanumberofusefulacademicjournalsandindexingplatformsmaynotbegenerallyaccessibletoyourreviewersifyoudonothaveasubscriptiontothemToaddressthisconsiderpartneringwithalocal
Reviewtheevidenceandcollectdetails
25
collegeoruniversitytoprovideyouwithalistofarticlesfromthefollowingsources(Thisisalsoanexcellentopportunitytoseekresearchsupportfromanyacademicstakeholdersyouidentifiedpreviously)
Indexes
GoogleScholarincludesmostpeer-reviewedonlineacademicjournalsandbooksconferencepapersthesesanddissertationspreprintsabstractstechnicalreportsandotherscholarlyliteratureincludingcourtopinionsandpatentsItallowsyoutospecifyapublicationrangeinyoursearchandprovideslinkstolikelysourcesforalllistedarticles
PolicyFileIndexisauniqueresourceforUSpublicpolicyresearchinthatitgrantsusersaccesstotimelyupdatedinformationfromover350publicpolicythinktanksnongovernmentalorganizationsresearchinstitutesuniversitycentersadvocacygroupsandotherentities
PolicyArchiveworkstocollectanddisseminatesummariesandfulltextsvideosreportsbriefsandmultimediamaterialofthinktankuniversitygovernmentandfoundation-fundedpolicyresearch
Anothersourceforhigh-qualityresearchisthereportsoftheCongressionalResearchServiceYoucanfindtheirreportsaggregatedhere
Specificjournals
Oftenthemostusefulacademicjournalsforunearthingpolicyorprogramreviewswillbeineachspecificsubjectarea--thatisineducationjournalsforeducationpoliciesYoucanuseageneralindexlikeGoogleScholartobegintolearnwhichjournalsaremostoftenusefulforyoursearchHoweveryoucanalsosearchthefollowingtwogeneralistpolicypublicationsforissuesandarticlesrelatedtoyourtopic
PublicAdministrationReviewJournalofPolicyAnalysisandManagement
Newssourcessubjectmatterexpertsandinterviews
AlthoughitisvaluabletoknowwhathasbeenexploredthroughascholarlyresearchframeworkitisalsousefultolearnwhatkindofinterventionsarebeingtriedinthefieldrightnowandhowpeoplefeelaboutthesetrialsSearchsubscriptionnewsaggregatorslikeLexisNexisorfreenewsaggregatorslikeGoogleNewsorYahooNewsforyourissueareaplusthewordpolicyorprogramtoseewhatkindsofinterventionsarebeingcoveredinthenews
Reviewtheevidenceandcollectdetails
26
DonotforgettocontacttheadvisorygroupmemberswhowerechosenfortheirsubjectmatterexpertiseInadditiontoprovidinggoodmaterialthroughinterviewsthemselvestheymayhaveideasaboutothersubjectmatterexpertsyoushouldcontacttoaskaboutpotentialinterventions
Peercitiesandnetworks
InadditiontoreadingabouttheworkofothercitiesitisidealifyouareabletospeakwithpeoplewhoareinvolvedwithcurrentimplementationsofinterestinginterventionsInyourcityyoulikelyhaverelationshipswithyourneighborsbutitcanbehelpfultolookoutsideofyourregionforsolutionsaswellTofindcounterpartsinothercitieswhoareworkingonaprogramyouwouldliketolearnmoreaboutyoucancontactstaffinintermediaryorganizations(likeGovExoryourprofessionalassociations)whomightbeinapositiontoputyouintouchimmediatelyorwhomightseeanothergoodrouteforsettingupaconversationHereisalistofnetworksthatmaybehelpful
InternationalCityCountyManagementAssociation(ICMA)NationalAssociationofCounties(NACO)NationalLeagueofCitiesUSConferenceofMayorsWhatWorksCities
Collectthedetails
Onceyourreviewershavefoundevidenceonpossiblepoliciesorprogramstoimplementtheyshouldcapturekeydetailsthatwillhelpdecisionmakersunderstandthepotentialeffectivenessandapplicabilityofeachpotentialintervention
Foreachsourcethathasprovidedinformationyouwanttoconveyusethetemplatebelowtocaptureasmanyofthefollowingdetailsaspossible(Youdonotneedtocompletethisforsourcesthatprovideduplicativeinformationorareotherwisenotthatuseful)
Programorpolicyname (egldquoHousingFirstrdquoorldquoblightedpropertiesregistryrdquo)
Sourcetype (egjournalarticleinterviewsubjectmatterexpertstudyassociationcommunityparticipationprocess)
Sourcedetails (egarticletitleandpublicationnameandtitleofinterviewee)
Evaluationmethodology
(egrandomizedevaluationcomparativeevaluationoftwodifferentgroupsbeforeafteranalysis)
Evidencelevel (egLevel12345)
Programorpolicy
Reviewtheevidenceandcollectdetails
27
description
Programorpolicygoal (1sentencecapturingtheprincipalaimoftheintervention)
Existingimplementations
(WherehastheinterventionbeentriedListasmanysitesasyouareawareof)
Characteristicsofimplementationsites
(ForeachimplementationsiteofferbasicgeographicIDmdashcityandstateThenyouwillwanttocollectthedetailswhicharerelevanttoyourdecisionmakersrsquoconsiderationsaboutwhatmakesalocationldquosimilartordquoorldquodifferentfromrdquoyoursituationYouwillatleastwanttoincludeforeachimplementationsitePopulationnumberPopulationdensity(ruralurbansuburban)Socio-economiccharacteristicsBeforestartingaskyourdecisionmakersaboutanyparticularcharacteristicsofprogramimplementationsitestheywouldwanttoknowbeforeseekingtoreplicate)
Characteristicsofimplementationpartners
(Anyavailabledetailsaboutorganizationsinstitutionsorpeoplewhowereinvolvedwithimplementingtheintervention-egindependentchildcarecenterpublicelementaryschoolcitypublichealthnurses)
Characteristicsoftargetpopulationeligibleprogramparticipants
(Anyavailabledetailsaboutpeoplewhoarethebeneficiariesoftheprogramorotherwiseincludedoraffectedbytheintervention-egnumberofprogramparticipantscategoriesofpropertyownersaffected)
Policyorprogramduration
(HowlonghastheinterventionbeeninplaceineachimplementationsiteHasitendedandifsowhy)
Whatbenefitscamefromeachimplementation
(Anyobservedpositiveeffectsoftheinterventionincludingeffectsize)
Statisticalestimatesofinterventioneffectiveness
(Anyavailablestatisticalestimatesoftheinterventionrsquoseffects)
Whatdrawbackswereobserved
(Anyavailabledetailsaboutunintendedorunavoidablenegativeeffectsoftheintervention)
Programorpolicycosts
(Anyavailabledetailsaboutprogramcostsperimplementation)
Reviewtheevidenceandcollectdetails
28
Reviewtheevidenceandcollectdetails
29
DecideonapolicytoimplementbasedonlocalfactorsTogetthecriticalbuy-infromleadershiptoimplementanewpolicyorprogramtheprojectleadshouldscheduleameetingwithdecisionmakerstopresenttheevidencesurroundingthepolicyoptionscapturedinthetemplateDependingonyourinsightintoyourcityleadershipswillingnesstomakechangeyoumaywishtosharetheresearchyouhaveconductedontherootcausesoftheproblemandsomeofyouranalysisontheimpactofcurrenteffortstodate
Considertheapplicabilityofanapproach
AsdiscussedinthepreviouschapterpolicymakersoftenlookforevidenceofeffectiveprogramsinothercitiesbutoftenthequestionarisesofhowgeneralizableaprogramisfromonecitytoanotherTheAbdulLatifJameelPovertyActionLab(J-PAL)suggeststhatinreviewingevidencefromanotherlocationitishelpfultoidentifythespecificmechanismthatwasused(deliveryofprogramorserviceinformationsharingetc)identifythelocalconditionsthatwereinplaceobservethebehaviorchangethatresultedfromthemechanismandfinallytounderstandlocalcontextandconditions(legalorcostconstraintsculturalnormsetc)Thissetofpresentationelementscanbedrawnfromthelistofdetailscollectedintheresearchtemplate
EducationpolicymakersinChicagoILusedfindingsfromprogramsinKenyaandIndiainwhichchildrenwhohadfallenbehindtheirpeersweregiveneducationalsupporttargetedtotheirleveloflearningThemechanismusedwastargetedteachingandthereweresimilarconditionspresentinallthreecountriesAlthoughsomestudentshadfallenbehindinthecurriculumteacherswereincentivizedtoteachgrade-levelmaterialsConsiderationsofhowthelocalcontextinChicagomightaffectimplementationofatargetedteachingprogramincludedunderstandinghowtheneighborhoodsthatthestudentswerefrommayhaveaffectedtheireducationaloutcomesandthehighcostofteachinginsmallgroupsInChicagoMatchEducationprovidedtwo-on-onetutoringwhichmatchedstudentswithwell-educatedindividualswhohavenothadformalteachertrainingforalower-costmodelTheresultswerepositiveStudentswhoweretutoredperformedbetterinmathandweremorelikelytopasshighschoolclasses
ThegeneralizabilityframeworkproposedbyJ-PALisbelowUsethisasatemplateasyouconsiderpotentialpolicysolutions
Decideonapolicytoimplementbasedonlocalfactors
30
SourceTheGeneralizabilityPuzzle
OnceyouhavenarrowedthepolicyoptionsdiscussthequestionsbelowforeachoptionsyouareconsideringThisstepiscriticalasunderstandinglocalcontextissoimportantinsuccessfulpolicyimplementation
CommunityContextQuestionnaire
Localcontext
DoesthisoptionmakesenseinourcommunityIsthisaonesizefitsallapproachIsanyparticulargroupabsentfromthisapproachIfsoisthatok
Youranswers
Effectiveness
WhatwouldweexpecttoobserveifthepilotisaseffectivehereasinCityX
Youranswers
Timing
DoesthetimingmakesenseIsthistherighttimetoimplementthispolicychange
Decideonapolicytoimplementbasedonlocalfactors
31
WherearethepotentialroadblocksthatcouldaffectthescheduleIsthereanywaytomitigatethisHowlongwouldittaketoseeanimpact
Youranswers
Implementation
DowehavethestaffcapacitytoimplementthisArethereadditionalpartnersthatyoucanbringtothetabletosupportimplementationDoesthismakesensegivenourformofgovernment
Youranswers
Authority
DoesourcityhavetheauthoritytoenactthisIfsowhatistheappropriatemechanismAretherelegalbarriersArethereanyexistingpreemptionissuesfromthestateorfederalgovernmentCanthosebeaddressedthroughlegislationadvocacyIstherepoliticalsupportoroppositiontothis
Youranswers
Cost
HowmuchwillthiscostannuallyHowwillwepayforitCanthepolicybeimplementedwithexistingfunding
IfsoisthisfundingreliableHowwouldthispolicyoptionaffectourcitysbudgetintheshortandlongtermWhatotherfundingoptionscanweconsiderIsthereanalternativethatwouldcostlessandgiveussimilarresultsWillanyconstituencybeadverselyaffectediffundingforprogramsthatimpactthemisshiftedtoanotherprogrampolicyorinitiative
Youranswers
Decideonapolicytoimplementbasedonlocalfactors
32
EstablishaframeworktomonitorprogressNowthatyouhavereviewedtheevidenceconsideredlocalcontextandchosenyourpolicyitistimetodeterminehowyourcitywillmonitorprogressofthisnewapproachCitystaffandleadersneedtoestablishtheinfrastructureforongoingcontinuousdecisionmakingbasedonwhetherthepolicychangeisleadingtothedesiredoutcomesforresidentsandwhetherthechangeismeaningfulandrationaltofrontlineworkersSettingarbitrarygoalsatthelocallevelthatconflictwithgoalsatotherlevelsisconfusingtotheindividualsresponsibleforachievingthegoalsGoalsthataresettoohightoachievecanbedemoralizingtostaff
ThissectionincludesstepstosetupavenueinwhichCitystaffandleadershiproutinelyvisitkeyindicatorstodeterminewhatisorisnotworkingandtomakedecisionsinreal-timeFormorereadingonperformancemanagementpleasevisittheGovExPerformanceManagementGettingStartedGuide
InperformancemanagementtermssuchasindicatormeasureandmetricmaybeusedinterchangeablyInthisguidewewilluseindicatorInyourworkthemostimportantthingistobeconsistentinwhichevertermyouuseThisvideomaybehelpfulinunderstandingthisconceptandcommunicatingittoothers
Establishthefundamentalsandgatheryourteam
FirstdeterminewhetherthereisanestablishedvenueorregularmeetingscheduleduringwhichyoucandiscussprogressIfyourcityhasanexistingperformancemanagementstructureonarelatedtopicaddingadiscussionoftheprogressofpolicychangetoyouragendacanbeagreatideaIfthiswillbeastandalonemeetingdecidehowfrequentlytheprojectteamanddecisionmakerswillmeetandputthosemeetingsonthecalendarfortheentirelengthoftheproject
DeterminewhowillstaffandattendroutinemeetingsonCityprogressThesemeetingsmayincludetheMayorandorCityManagerdepartmentheadstaffwhoareimplementingthepolicychangeandanalysts
Identifyindicators
ItiscriticalthatindicatorsbeanimportantcityandcommunitypriorityaswellassomethingtheCityplaysamajorroleinaddressingTheremaybeinstancesinwhichtheCitywillneedtoreachouttoorcollaboratewithexternalpartnerstogetamorecompletepictureofprogressontheoverallpolicychange
Establishaframeworktomonitorprogress
33
Anoutcomeindicatordescribesthedesiredeffectofthetargetedpopulation
ExampleBellevuesfamilyfriendlyneighborhoodsindicatormeasuresthepercentofresidentswhobelievetheyliveinaneighborhoodthatsupportsfamiliesparticularlythosewithchildren
WithyourkeypartnersrevisityourproblemstatementcurrentstateandfuturestateexerciseandbegintoidentifytheoutcomeindicatorsthatwillillustratesuccessFornowchooseasingletop-levelkeyperformanceindicator(KPI)ItiscriticalthattheKPI-thepieceyouwillmeasure-beapriorityforthecommunityandsomethingtheCityplaysamajorroleinaddressing
Outputindicatorsmeasuretheprocessofprogramorservicedelivery
ExampleSanDiegoMayorKevinFaulconersgoaltorepair1000milesofstreetby2020
ReviewingtheadministrativedatasetinventorybegintoparedowntherelevantprogramsanddatasetsthatsupportyourKPIWhatarethemeasuresofeffectivenessforeachprogramTheseareyouroutputindicators
Thediagrambelowillustratestherelationshipbetweenoutputandoutcomeindicators
ThinkaboutimplementationindicatorsasmeasuringstepsintheprogramsrolloutthatarenecessarytoobtaindesiredchangesImplementationoutcomescanfocusontheinterventionstaffdeliveringtheinterventionanorganizationorsystemandcanaddressfeasibilityadaptabilitysustainabilityandcostTwokeyquestionstoconsiderwhendevelopingimplementationindicatorsare
1 Istheinterventionacceptableandappropriatetothecommunityand
2 IsitfeasibletoimplementinterventionXatthistimewiththeseresources
Establishaframeworktomonitorprogress
34
Amoreextensivelistwithdefinitionsofpotentialimplementationoutcomesislinkedhere
UsethetablebelowtochoosethedesiredimplementationoutcomesforyourpilotandtoplanyourmeasurementtechniqueThisshouldincludelevelofanalysis(individualproviderorconsumerorganizationorsetting)yourmeasurementtechnique(surveysobservationinterviewfocusgroupsadministrativedataanalysis)andthephasewhenyouwillcollectthisinformation(earlymidlate)Theremaybemorethanonelevelofanalysismeasurementtechniqueandphaseofdatacollection
Implementationoutcome
Definitionofoutcome
Levelofanalysisandmeasurement
Yourmeasurementtechnique
Phaseofpilotwhenyouwillcollectdata
Usethespacebelowtocaptureyourindicatorwishlist
Indicatorwishlist
Outcomes_____________________________________________________________________________
Outputs________________________________________________________________________________
Implementationoutcomes______________________________________________________________
Setapublichigh-levelgoal
Thehigh-levelgoaliscriticalinestablishingdirectiontogalvanizethepeoplewhowillbedoingtheworkWhendiscussingthegoalusetheMayorsbullypulpitLookforkeymilestonesinCitygovernmenttosharethegoalstatementsuchasthebeginningofaMayoraladministrationaStateoftheCityspeechorotherpublicremarks
GoalsshouldbeldquoSMARTrdquo
Specificaspossible(innamingthepopulationwhowillbebetterofftheactionthecitywilltake)
Measurable
Achievable
Establishaframeworktomonitorprogress
35
Results-focused
Time-bound
ThegoalstatementshouldincludeyouroutcomeindicatorlikeChattanoogasgoalldquoWewillendchronicveteranhomelessnessbyDecember2016rdquoAdiagramtohepyoubuildaSMARTgoalaroundyouroutcomeindicatorisbelow
Component Description Yourturn
Specific WhowillbebetteroffWhatactionthecitywilltake
Measurable Howwillyoumeasuresuccess
Achievable Isthegoalattainable
Results-focused Whatistheexpectedresult
Time-bound When
Setrealisticambitioustargetsandtimelinesforeachindicator
ToestablisharealisticandambitioustargetforyourgoalstatementusetheadministrativedatacollectedforyourchosenKPItoreviewactualCityperformanceinthemostrecentyearcomparedwiththethreeorfourmostrecentyearsworthofdataBegintocalculatesummarystatistics(egtotalminimumaveragemedianmaximum)toidentifytrendsCreatecumulativeandmonthlychartstoidentifytrendsandanomaliesBasedonyouranalysisofhistoricaldataignoretheanomaliesandassumetheproductivitytrendswillfollowthesameoverallpatternfrommonth-to-monthFinallyapplyanincreasedproductivityfactortotheperformanceeachmonthtoforecastfutureprogressBasedonthisanalysischooseatargetthatinspiresimprovementwhileaccountingforreality
Manycitiesthataremorematureintheirperformancemanagementpracticesrecognizetheimportanceoftakingacommunity-levelapproachtoimprovingoutcomesforresidentsThesecitiesarecreatinggoaltaxonomiesthatincludemeasuresrelatedtopublicsafetyhousingandevenrecreationCitiesdriveprogressonthesegoalsbyimplementingstrategiesthatengagemultipledepartmentsandcollaboratewithserviceprovidersinthecommunity
EventuallyyouwilldevelopgoalstatementsfortheoutputsinordertocapturetheefforttoachievetheoutcomecontainedinyourgoalstatementYoursetofindicatorsshouldbecomposedofshort-andinterim-termsupportivemeasureswithaccuratetargetsandrelevanttimelinesAnexamplegoaltaxonomyfromtheCityofLasVegassResultsVegasprogramisbelowUsethistaxonomyasamodelforyourownsuiteofindicatorsandgoalstatements
Establishaframeworktomonitorprogress
36
Yourturn
Thetemplatebelowismeanttohelpyoucraftagoalstatementforeachsupportingmeasure
Goalstatement
Departments
Outcome
Activities
KPI
Target
Timeframe
Conductanddiscussrelevantanalysis
ThrougharoutineseriesofmeetingscontinuallymonitorprogressofKPIssupportingindicatorsandanyotherdatacollectedsuchasqualitativedatatoidentifytrendsthatwillindicatewhetherthechosenpolicyishavingthedesiredimpactThesedatapointsshoulddrivethediscussionduringthesemeetingspointingtoopportunitiesforfurtherinquiryinto
Establishaframeworktomonitorprogress
37
whatisdrivingtheobservedtrendsAnalystsshouldbeencouragedandtrainedtoaskquestionsofthedepartmentsproducingthedataandconductsitevisitswhenpossibletodeepentheirunderstandingofthedataPresentationoftheanalysisshouldfocusononeideaatatimetokeeptheconversationfocusedonkeypointsandspecificopportunitiesforimprovementYoumayconsiderdevelopingapublicfacingdashboardtosharethisprogressmorebroadly
AgeneralcomparisonofthestatusquodatawithwhatwasobservedduringthepilotcanprovidedecisionmakerswithaquickunderstandingofthedifferencesbetweentheoldandnewapproachesalthoughthiswillnotshowcausalityOftenthesecomparisonswillincludeoperationalmetricssuchasdaysordollarssavedbutrememberthattheimpactonpeoplewillbemostimportantIfyouareworkingwithanevaluationpartnertheywillsupportthisstep
Usethespacebelowtosummarizetheresultsofyouranalysis
MeasureTargetvalue(seesectiononsettingtargetsabove)
Pilotvalue(whatdidyouobserveduringthepilot)
Historicalaverage(ifavailable)
Results(WhoisbetteroffIsanyonenegativelyimpacted1-2sentencesreassessingyourgoalstatement)
DocumentyourprocessandprogressincludingfeedbackfromtheparticipantsimplementationpartnersandtheprojectteamDidanythingunexpectedoccurduringthepilotArethereinternalandexternalfactorsthatmayaffectyouroutcomesThisiswheretheimplementationoutcomescanbecanhelpfulIftheinterventiondidnotgoasplannedconsiderwhetheritisbecausetheinterventionwasnotacceptableorfeasible
UsethetemplatebelowcreatedbyDenversPeakAcademytocapturethisinformationIfthe306090daytimelinedoesnotmakesenseforyourcityfeelfreetocustomizeit
CurrentState
FutureState
30Day
60Day
90day
Whatwentwell(+)
Whatdidnotgowellshouldbechanged(-)
IfaquestionordiscussionpointarisesthatcannotbeansweredinthemeetinganassignedstaffmembershouldcapturethequestionduringthemeetingAfterwardthefollowupitemsshouldbesenttorelevantpartnersandthenextmeetingshouldbeginwithadiscussionof
Establishaframeworktomonitorprogress
38
thoseitems
Takeaction
DiscussionsaboutwhetheryourindicatorsaremovingintherightdirectionandwhatmightbedrivingtrendsmayleadtoadiscoverythattherearenotenoughresourcesallocatedtothispolicyorissueTheremaybelimitedfundingforstaffprogrammingortechnologythatisdrivingthedatainthewrongdirectionIfCityleadershipisintheroomthisisagreattimetomakethecaseforadditionalfundingduringthebudgetseasonBycollectinganalyzinganddiscussingdataonthispolicyroutinelyyouarebuildingevidencetomakethecasefortherightleveloffundingtoreachtheestablishedtargets
Establishaframeworktomonitorprogress
39
Assessdataaccessibilityqualityandsecurity
Determineaccessibilityandqualityofdata
OnceyouhavecompiledyourwishlistofindicatorsitistimetodeterminetheaccessibilityandqualityofthedataTopreventcomplicationyoumaywishtoprioritizerelevantindicatorsforwhichyouarealreadycollectingdataAvoidthetemptationofadoptingatechnologysolutiontocollectnewdataatthispointsuchasaphotosurveytoolorotherappFornowyouaretryingtodeterminewhichdatasetsarerelevantaccessibleandhighqualityBesuretoworkwithyourkeyinternalpartnerswhileyouareidentifyingyourmeasuresDepartmentstaffwhoworkonprogramsarefamiliarwiththeindicatorsthatwillaccuratelysupporttheoutcomesanddatabaseadministratorswillbevaluablewhendiscussingaccessibilityandqualityofthedata
Abrieflistofconsiderationsfortheaccessibilityofyourdataisbelow
ArewecurrentlycollectingthisdatasetorwillwebecollectingitforthefirsttimeHowfarbackdoestherecordgo(timehorizon)WouldweneednewtechnologytocollectoranalyzethedataCanthedatabecollectedmanuallyorcanitbeautomatedHowfrequentlyisthedataproduced(continuouslyhourlydailyweeklymonthlyquarterlyyearlyetc)IsthedataavailablefortheentireCitypopulationorasubsetofthepopulationHowgranularisthedata(individuallevelCensusblockneighborhoodcitywideetc)DoesthedatacontainsensitiveinformationIsthereaproxymeasurethatwecanusefornowIsthedatamachinereadable
AsnotedbytheWilsonSheehanLabforEconomicOpportunities(LEO)attheUniversityofNotreDameleveragingadministrativedatacansignificantlyreducethecostsandtimelinesofrigorousevaluationCitiescanestablishaninternaldatasharingagreementacrossdepartmentsincludinglanguagethatenablesadministrativedatatobeusedforresearchpurposesCitiescanalsoidentifyexternaladministrativedata-holderssuchasacountyschooldistrictpolicedepartmenthospitaloranotherstatethatmayhaveoutcomedataimportantformeasuringprogramimpactCitiescanleverageexistingrelationshipsandorworkwithresearchinstitutionstodevelopstreamlineddatasharingprocessesandagreementswiththeseexternalentities
Assessdataaccessibilityqualityandsecurity
40
Evaluatethequalityofyourdata
QualityisasimportantasaccessibilityToevaluatedataqualityyoumaywishtocreatearubricliketheonebelow
Value ValueDescription
3 ExcellentDataQualityThedataiscompleteaccurateandupdatedregularly
2 AdequateDataQualityThedataissomewhatcompleteaccurateandupdatedfairlyregularly
1 PoorDataQualityThedataisnotcompleteaccurateorupdatedregularly
ThisisagreattimetobuildonthedatainventorythatyoustartedinthegatherinformationpageSuggestedadditionalfieldstoaddtoyourdatainventorytemplatearebelow
Program DatasetTitle
DatasetDescription DataSource
Departmentorexternalpartner
DoestheCityalreadycollectthisdata
Whatisthenameoftheprogram
Human-readablenameofthedataset
Shouldbeunderstandablebynon-technicalusers
Whatistheoriginalsystemapplicationfilethathousesit
Departmentorexternalpartnerthatmaintainsthedata
Isthisdataalreadycollected
Updatefrequency-ThefileformatforthedatasetTypicallycsvxlsxshapefiletxtetc
Format-Howoftenthedatasetisupdated
Timerange-Theperiodoftimecoveredinthisparticulardataset
Datause-Howisthisdatasetcommonlyused
Datausers-Whotypicallyusesthisdataset
Dataownerstitle-Titleofthedataowner
Dataownersposition-DataOwnersjobtitle
Dataownersemailaddress-DataOwnersemailaddressWhomanagesthedataandorisresponsibleforgrantingpermissiontoaccessthedataWhounderstandswhatthedatasetincludesandcananswerquestionsaboutit
Dataqualityconcerns-Describetheconcernsyoumayhaveaboutthequalityofthedataset
Assessdataaccessibilityqualityandsecurity
41
Division-DivisionwherethedataismaintainedIfmorethanonedivisionisresponsiblelisttheprimaryownerofthedataset
Sensitivedatacomments-DescribetheissuesthatwouldpreventthecityfromsharingthedatapubliclyCommonsensitivityconcernsincludeprivacyviolationssecurityissuesorhighcostorstaffdemandstopostthedata
Publishingstatus-Thistheplacetoputanycommentsregardingthecurrentstatusofthedataset
Publishedlink-Thisisthelinktothepublisheddatasetontheopendataportalorcitywebsite
Reviewdatasecurityandprivacyprotectionprotocols
OftenapolicychangealsoinvolvesachangeinthedatathatisbeingcollectedhowthatdataisbeinghandledwhohasaccesstoitandahostofotheradjustmentsAsyoutakestockofthedataassetsthatareavailableforyourinitiativeitisvitaltoalsothinkthroughwhatwillbedonetoprotectthatdataandtheprivacyofthepeopleitrepresentsAbigpartofgooddatasecurityinvolvesstrongdatagovernancepracticesthatdefineandidentifysensitivedataandconsciouslyworktobalancethevalueandriskofstoringsuchsensitivedataOnewayofevaluatingyourdatacomesfromcaselawrelatedtotheUSFreedomofInformationActwhichwehavetranslatedtothechecklistbelow
1 Woulddisclosureresultinasubstantialinvasionofprivacy2 Haveweconsideredtheextentorvalueofthepublicinterestandthepurposeorobject
oftheindividualsseekingdisclosure3 Istheinformationavailablefromothersources4 Wastheinformationgivenwithanexpectationofconfidentiality5 Isitpossibletomouldreliefsoastolimittheinvasionofindividualprivacy
TheseriskscouldbemitigatedusingavarietyofstrategiesfromaggregatingdatatoobfuscatingindividualstoredactingfieldsthatmayincludePersonallyIdentifiableInformation(PII)orre-evaluatingyourdatacollectionmethodstofindalternativewaysofgettingtheinformationyouneedThesestepsshouldbecodifiedandmanagedbyadatagovernancecommitteeifpossibleForthesensitivedatathatisdeemedessentialcreateaplanaroundpotentialdatabreachesinordertoquicklyandconfidentlyrespondtoemergencysituations
ProactivelypreparingforthesesituationsisoftenthebeststrategyfordatasecuritybutdonotforgetthetechnicalsideInvolveyourITadministratorsinconversationsaboutdatastorageandaccessensuringthattheywillbeprotectedbyenterprise-gradecybersecuritysystems
Assessdataaccessibilityqualityandsecurity
42
Assessdataaccessibilityqualityandsecurity
43
Designimplementationandevaluation
Designtheimplementation
NowthatyouhaveyouraccountabilityplanandkeyindicatorsitistimetogetspecificaboutyourplanforimplementationThetemplatebelowwillhelpyouthinkthroughthespecificmechanismsofimplementingpolicychange
UnderstandingorganizationsandsystemsiscriticaltounderstandingandmakingdecisionsaboutwhereandhowinterventionsshouldbeappliedApolicyimplementedinasinglesettingmayshowpromisingresultsbutimplementingmultiplepoliciesandprogramsatthesametimewithdifferentactorsresponsibleforeachaspectcanhelpensuresustainablebehaviorchange
InSomervilleMAShapeUpSomerville(SUS)isacitywidepublichealthprogramthatisimplementedacrossmultiplelevelsofinfluenceandsystemsthroughinterventionsbeforeduringandafterschooltoincludechangestobuiltenvironmentsuchasbikelanesregionalmasstransportationandphysicaleducationequipmentinschoolsandgymstoincreasephysicalactivitywellnesspoliciesandschoolgardensnutritioneducationduringandafterschoolandhealthyrestaurantprogramstosupporthealthyfoodandhealthyeatinghabits
Usethetemplatebelowtoplanyourimplementation
Policyimplementationplanningtemplate
ListthespecificdesiredoutcomesInthepreviousstepyouworkedwithdepartmentstafftoestablishspecificmeasurableoutcomesandinterimindicatorsPrioritizetheindicatorsthatwilldemonstrateprogressUsethespacebelowtocapturethedesiredoutcomesofthepolicychange
ExampleShapeupSomervillebeganwiththegoalofpreventingobesityin1stthrough3rdgraders
Yourturn
____________________________________________________________________________
____________________________________________________________________________
Designimplementationandevaluation
44
SpecifythetargetpopulationAreyoudesigninganinterventionthattargetsonlycertainsitesAreyouplanningonimplementingaprogramorinitiativeforaspecifictargetaudienceClearlyspecifythepopulationyouplantotarget
IfyouplantoconductarigorousimpactevaluationasnotedbytheWilsonSheehanLabforEconomicOpportunitiesitisimportanttoestablishconcreteobjectiveeligibilitycriteriaandensurethatthesecriteriaareappliedconsistentlybyallstaffduringscreeningandintakeItisalsovitaltoidentifytheindividualswhoareeligibleforaparticularprogramorserviceevenifeveryoneeligiblecannotbeservedbyexistingresourcesThishelpscitiesidentifyanyunmetneedsandguidefutureresourceallocationwhilealsocreatingthepossibilityofarigorousimpactevaluationbyidentifyingapotentialcomparisongroupFinallywhenconductingscreeningandintakeitisimperativetocollectandmaintainbasicinformation(egnamedateofbirthgender)aboutallprogram-eligibleindividualsnotjustthosewhoparticipateintheprogram
ExampleIntheShapeUpSomervilleexampletheprogrambeganbytargeting1stthrough3rdgraders
Yourturn
____________________________________________________________________________
____________________________________________________________________________
DefinethelocationWherewillthepolicychangetakeplaceWillimplementationtakeplaceatonesiteormultiplesites
ExampleShapeUpSomervilleinterventionsweredeliveredinschools
Yourturn
____________________________________________________________________________
____________________________________________________________________________
ClarifyimplementationpartnersandarticulatetheirrolesWhatarethekeyorganizationsinstitutionsorpeopleresponsibleforimplementationTheprojectteamshouldbeasclearandspecificaspossibleinprovidingadescriptionoftheconcreteactivitiesthatimplementationpartnersaremeanttocarryoutPracticeprofilesmaybeahelpfultooltohelpdefinewhatisrequiredoftheactorswhoareresponsibleforimplementingtheintervention
Designimplementationandevaluation
45
ExampleInSomervillemayoralleadershipandengagementofmultiplecityagencies(publichealthprimarycareschoolsCambridgeHealthAllianceetc)streamlinedandconnectedthevariousShapeUpSomervilleinitiativesandgrantswhichhelpedtokeepthetalentandenergyfocusedonhealthyeatingandactivelivingforallsectorsofSomervilleSUSpartnersultimatelyincludedschoolsnonprofitsCityandstateagenciesandnationalfoundations
Yourturn
____________________________________________________________________________
____________________________________________________________________________
DescribetheinterventionWhatarethestepsactivitiesorprocessesRememberthatthemechanismthespecificstepsorprocessthatleadtobehaviorchangeiskeytounderstandingtheeffectivenessofthepolicychangeandsharingtheresultswithothers
ExampleGoalsofShapeUpSomervilleweretoinfluenceeverypartofanearlyelementaryschoolchildrsquosdayndashbeforeduringandafter-schooltopromotehealthyeatingphysicalactivityandtheirweightstatusInterventionincludedfoodserviceenhancementstoimprovebreakfastandlunchandstaffprofessionaldevelopmentwalktoschoolactivitiesSUSclassroomcurriculumandprofessionaldevelopmentschoolwellnesspolicydevelopmentSUSafter-schoolcurriculumandprofessionaldevelopmentoutreachandeducationtohomethroughmaterialsforumseventsldquoSUSapprovedrdquorestaurantscommunityoutreachandcapacitybuildingthroughpolicydevelopmenttrainingsmediaplacements
Yourturn
____________________________________________________________________________
____________________________________________________________________________
ClarifytimelineandmilestonesfordeliveryandevaluationHowfrequentlywilltheservicesinterventionbedeliveredHowlongwilltheprogramrunWhatisthetimetoexpectedimpacts(eg12monthsafterprogramcompletion)
Designimplementationandevaluation
46
ExampleShapeUpSomervilleledbyTuftsUniversityFriedmanSchoolofNutritionScienceandPolicyconductedaquasi-experimental(nonrandomized)interventionstudyamong1stthrough3rdgradestudentsinSomerville(intervention)andtwocomparisoncommunitiesfor9-monthsduringoneschoolyear(fall2002ndashspring2003)ThestudyshowedpositiveassociationoftheinterventiononchangeinBMISUSreleasedreportssummarizingevaluationsprogresstodateplansforadditionalprojectsandimpactstenandfifteenyearsintotheinitiative
Yourturn
____________________________________________________________________________
____________________________________________________________________________
PlanyouranalysisTheprevioussectionaddressedestablishinganaccountabilityframeworktomonitorprogressandimplementationDeterminewhichindicatorswillbediscussthroughthatprocesswhowillsharethedataandanalysisandhowfeedbackfromthoseconversationswillbeincorporatedbytheprojectteam
ExampleTheMayorrsquosofficeimplementedadata-drivendecisionmakingapproachcalledSomerStatwhichwassynergisticwiththeapproachestakenbytheresearchandevaluationteamsdeployedforthespecificgrants
Yourturn
____________________________________________________________________________
____________________________________________________________________________
PlanforsustainabilityIfthepolicyandprogramchangeissuccessfulhowwilltheprojectteamsustainandexpandtoadditionalsitesorpopulations
ExampleAlthoughmostofthepartnershipsforgedintheearlyyearsweremaintainednewgrantsandcollaborationsexpandedattentiontoolderpublicschoolstudentsactivetransportationandtheGreenLinesubwayextensionthebuiltenvironmentfarmerrsquosmarketeffortssuchasldquoGrowninSomervillerdquowithUnionMainStreeteconomicdevelopmentandhealthdisparitiesaswellasdatacollectionmonitoringandsurveillance
Yourturn
Designimplementationandevaluation
47
____________________________________________________________________________
____________________________________________________________________________
Pilotplanningtemplate
IfyouplantoconductapilotfirstdeterminethepurposeofthepilotDoyouwanttoobservewhetherthepolicychangeresultsinthedesiredbehaviorchangeAreyouconcernedaboutfeasibilityscalabilityorcostDoyouwanttotestwhetheryourdesiredresultscanbeachievedthroughthispolicyapproachClarifythegoalofthepilotandthequestionsyouhopetoanswerThiswillhelpyouidentifythekeyindicatorsthatyouwillusetomeasurewhetherthepilotissuccessfulBeforebeginningestablishabaselineofcurrentperformanceandsetyourtargetsaccordinglybasedonwhatyouhopetoaccomplishthroughthepilot
NextestablishthepilotplanforyourprojectteamincludingmilestonesforteamcheckinstodeterminewhethertomoveaheadorchangecourseThisisseparatefromthemeetingsthatwilloccuraspartoftheaccountabilityframeworkUsethespacebelowtocreateyourtimelineanddevelopyourmeetingschedule
Task Personresponsible Startdate Checkindate Deliverable
Planyourevaluation
WhileyouaredesigningtheimplementationitisalsotimetobegintodevelopyourplanforanalysisAsnotedbytheWilsonSheehanLabforEconomicOpportunitiesdecisionsaboutprogramsetup-includingoutreachscreeningintakedatatrackingandenrollment-candeterminewhetherornotarigorousimpactevaluationwillbepossibleAsdiscussedcityleadersmayfindithelpfultoincludeevaluationpartnersinearlyconversationsaboutprogramsetupDuringpreviousstagesyoumayhaveworkedwitharesearchertoplanyourproposedpolicychangeYoushouldbeclearaboutthepurposeoftheevaluationanddevelopanevaluationplanBelowareadditionalconsiderations
Causationvstrendanalysis
IntheEstablishaframeworktomonitorprogressstagetherewasguidanceontrackingtrendsovertimeItisimportanttonoteherethatmonitoringdataisdifferentfromanalyzingacausaleffectTrendanalysiswillillustratethatchangesareoccurringandmayleadtoquestionsthatcouldbeansweredbyyourevaluationbuttrendanalysisalonedoesnotprovideevidenceaboutwhatisdrivingthosechangesYoumayseechangesinyour
Designimplementationandevaluation
48
outcomeindicatorsbasedonvariablesoutsideoftheCityscontrolForinstanceachangeinfederaltradepolicycanhaveanimpactemploymentratesinyourCityoutsideofyourCitysnewjobtrainingandplacementprogram
ToaddresscausalityyoumustcomparetheoutcomesofindividualswhoreceivedtheinterventionwiththosewhodidnotWithoutacomparisongroupofsimilarindividualswhowerenotexposedtotheinterventionitisimpossibletoknowwhichfactorsareresponsiblefortheoutcomesRandomizedControlTrials(RCTs)inwhichindividualsarerandomlyassignedtotreatmentandcontrolgroupsarethegoldstandardinprovingcausalityIfyoucandemonstratethatoutcomesimprovedforthosewhoreceivedtheinterventionanddidnotimproveatallorimprovedlessforacomparisongroupyoucanbesomewhatcertainthattheinterventionwasthecause
Examplesofothernon-causalimpactevaluationsthatarerigorousandreliableincludecertainquasi-experimentalstudiesusingcomparisongroupdesignssmartinterruptedtimeseriesanalysesthatmodeltrendsovertimeinwaysthatallowestimationofinterventioneffectswelldesignedpre-posttestanalysesneedsassessmentsimplementationstudiesetcFormoreinformationondifferenttypesofevaluationsseetheSageResearchMethodswebsite
Lookfordifferencesingroups
BuildintimetoalsoexaminethepilotdatatoseeifgroupsareexperiencingdifferentresultsReviewingperformancebasedonoutcomemeasuresforapopulationinaggregatemayhidethesubtlewaysthatsomepeoplemaybeaffecteddifferentlysoitisnecessarytocarefullyanalyzethedatafordisparateoutcomesForinstancetheMovingtoOpportunity(MTO)studythatexaminedtheimpactofhousingvouchersonfamilieswholivedinhigh-povertyneighborhoodsfoundthatchildrenwhomovedbeforetheageof13hadbettereducationandhigherincomesasadultsHoweveradultsdidnotseeimprovededucationandemploymentoutcomes
Disaggregatingdataintomorespecificcategoriesofraceandethnicitycanhelpunderstandtrendsanddisparitiesinamoregranularwaywhichcanhelpdecisionmakerstargetpoliciesandinterventionsinamoreequitableway
EarlierintheRoadmapweworkedtoidentifystakeholdersandwhomightbepositivelyandnegativelyimpactedbyachangeinpolicyUsetheresultsofthatexercisetoinvolvethecommunityininterpretingdataanalysisandtogatherqualitativefeedbackfromthecommunitytocallattentiontoissuesForinstanceanincreaseinyouthdetentionmaybeexplainedbyinefficienciesorlackofaccesstotransportation
Designimplementationandevaluation
49
Designimplementationandevaluation
50
Shareyourstoryandsuccess
Sharetheresults
Storytelling
AnintegralstepinyourstrategyissharingtheresultsofyourpilotorpolicychangewithyourcommunityThisincludesdevelopingmessagingaroundtheworkofyourorganizationandcreatingaplantodisseminateyourstoriesWhytellstoriesaboutyourwork
Informandorunderstandasubjectorsituation
Giveneededcontextofdataorbitsofinformation
Relatablestoriesarememorableforreadersviewers
ReadersorviewerswillseetheinformationasmorecompellingwhenthereisademonstratedandrelatablechangeorimpactonthemselvesortheirfriendsneighborsandcolleaguesStoriesareaboutpeopleandgovernmentisfundamentallyabouthelpingpeopleSotellthestoriesofthepeopleyouandyourcitycolleagueshelped
Keepthesetipsinmindasyoudevelopyourstorywithyourteam
WhatwereyourfindingsandwhyshouldanyonecareTellyouraudiencehowtheproblemwassolvedWhathastheimpactbeenonresidentsandcitystafftodateWhatwasthesuccessandweretheremultiplesuccessesalongthewayIfyouhavepreliminaryresultsordatashowingimprovementtosharethisisthetimetodoit
WhowasinvolvedandimpactedReviewthelistofstakeholdersHowwereeachofthemimpactedbytheprogramHumanizeyourstoryandimpactbyaddingadditionalvoicesThesevoicescanbeoneofthefollowing
Apersonorfamilywhowaspositivelyaffectedbythechangesuccess
Aninvolvedthird-partydescribingboththeproblemandthesolution(ieanorganizationnonprofitprovidersbusinesstradeassociationsportsteametc)whoispartofthecommunitythatthegovernmentworkedwithonthechangesuccess
Someonefromthegovernmentitselfwhocanspeakaboutthechangesuccess
TellmoreaboutyourcityDescribeyourcityincludingpopulationdemographicsindustriesetcWhatareitschallengesWhatisitknownforWhatmakesituniqueHowisitsimilarordifferentfromothercitiesintheUS
Shareyourstoryandsuccess
51
DescribeyourcityrsquoschallengeWriteoutthechallengeyourcityisfacing(asdefinedinyourproblemstatement)HowdoesitaffectresidentsHowdoesthisproblemaffecttheworkofcitystaffmembers
AddthedetailsProvidedetailsonyourappliedresearchstrategyTalkaboutwhatyourcityhasdonenthepasttoresolvetheissueDiscusshowyourstrategyhaschangedwhenyoubegantoadoptthisapproachWasmoneysavedorinvested
Usethisstorymapexercisetoorganizeyourthoughtsandguaranteethatyouincludealloftheelementsofatraditionalstory
AfteryoudevelopyourmessagingcreateacommunicationsmarketingandoutreachstrategydesignedtospreadyourmessageYourstrategyshouldwalkyouthroughthefollowingkeyactivities
1 IdentifyandprofileyouraudienceWhencreatinganddisseminatingyourmessageitisalwaysgoodtothinkaboutyouraudienceWhoexactlywouldbeinterestedinwhatyouhavetosayThegeneralpopulationofthecityPeoplewhoworkinnonprofitsBusinessownersItmattersbecausenoteveryonewholivesinacitymaybethetargetoftheinformationBusinessownersmaybemoreinterestedindatashowingeffectsonthebusinessdistrictthanothersandsoonWewillcoverthislaterbutthisstepwillalsohelpyoutothinkofthemosteffectivewaytoreachyouraudience
2 Defineyourcalltoaction(ifyouhaveone)SpendsometimeconsideringwhyyouareinterestedintellingyourstoryDoyousimplywanttoupdatethepubliconyourprogressorcelebrateyoursuccessesAreyouinterestedinyouraudiencechangingitsbehaviororsupportingaspecificinitiativebeinglaunchedbyyourorganizationIfyouwanttohaveyouraudienceactorbehaveinacertainwaybespecificandincorporateitinyourmessaging
Shareyourstoryandsuccess
52
3 SelectperformancemeasuresThinkabouthowyouwoulddetermineifyourcampaignissuccessfulandselectappropriateperformancemeasuresYouwillrevisitthesemeasuresattheendofyourcampaigncommunicationsplanimplementationtodeterminewhetheryouneedtomodifyyourcommunicationsandoutreachstrategyforthefuture
4 MethodstocommunicateyourstoryTherearemanywaystocommunicatesuccessofyourprojectHerearesomesuggestionsonreachingaudiencesItisimportanttodeterminewhichmethodsarethemosteffectiveforyourdesiredaudiencetoreceiveinformationDoyourresearch(PewResearchCenterhasexcellentstudiesonthereachofmediadigitalmediaandmarketing)ForexampleifseniorsareyourtargetpopulationandtheyarelessinclinedtouseInstagram(thatistrue)youmaywanttoconsiderFacebookorthenightlylocalnewsbothofwhichareusedmoreoftenbyseniorsBelowisalistofsomemethodsandtoolsyoucanuse
i Pressreleasestolocalnewspapersandtvstations
ii ShortsmartphonevideosAgreatwaytoengageanaudienceisthroughvideoAskonequestionofsomeonewhoispartofthestoryMaybethereisasocialworkerwhocandiscusshungerinyourcityThesevideoscanbereusedonmultipleplatforms
iii Long-formwriting
iv TellingindividualstoriesthroughaseriesofphotosandshortnarrativesforpublicationonFacebookorInstagram
v Op-edsopinionpiecesplacedinthelocalpaper
vi Emailupdatestoyourlist
vii Connectingwithappropriateorganizationsthatareinvestedinthetopicandworkwiththecommunityandaskthemtodisseminateinformationtotheirmembers
viii EmployingldquoWordofMouthrdquo(WOM)tacticsbycreatingtownhallsorfreeeventsforresidentstoattendintheirownneighborhoods
5 DeterminethetypeofcollateralUsuallycampaignsthattargetresidentsrequireseveralavenuestoreachthemPeoplearenotalwaysinthesameplaceorspaceHavingakeenunderstandingofyouraudiencewillassistyouwithyourchoicesonwhatneedstobedevelopedSometimesthatmeansimagesforsocialmediaaonepageflierthatgivespeopleinformationonattendinganeventoranin-depthbrochure
Shareyourstoryandsuccess
53
6 ImplementyourstrategyKeepinmindstrategiescanbefluidSometimessomethingmaynotturnoutasexpectedevenifduringplanningsessionsitwasldquosupposedtordquoItisokYoucananalyzewhathappenedduringadebriefsessionandonceyoursurveysandothermeasurementshavebeenfinalized
Visualizations
Withtherightvisualizationyoucanreviewandshareinformationinsightsanalysisperformanceandtrendsallofwhichisvaluabletoacross-sectionofthepublicButwhichvisualizationisrightforyouTochoosetherighttypeofchartgraphdeterminethetypeofquestionyouwanttoanswerSomecommoncharttypesarebelowandmoreinformationcanbefoundhere
Exploreadditionalanalyticaltechniques
Duringthegoal-settingphaseyouestablishedoutputoutcomeandimplementationmeasuresBasedontheresultsofyourpilotyoumayneedtoadjustyourmeasuresWiththepilotdatainhandyoumayhavesomeideasofhowfutureanalysescouldbedifferentandpotentiallyimprovethequalityofyourprogramTakesometimetobrainstormotherrelevantdatasetsthatyoudidnothavethetimetopursueearlierincludingmessydepartmentaldatathatneedsmoreattentionorsourcesofpublicdatafromthestateandfederalgovernmentnonprofitsandevenprivatecompaniesthatcouldsupplementyourunderstandingofanissueUsingmachinelearningtechniquesyoumightbeabletobetter
Shareyourstoryandsuccess
54
predictdemandforhomelessservicesusingNOAAweatherforecastspreemptwatermainbreaksusingsoilstratigraphyorinvestineconomicdevelopmentbymonitoringrealestatelistingdata
Advancedanalyticaltechniquescanalsohelpyourimplementationrunmoreeffectivelybyintegratingdata-informedtargetinganddecisiontoolsintotheprocessByautomatingroutineworkflowsandsurfacingsubtletrendsthesetoolscanhelpyouiterateinamoreprecisemannerandprovidekeystrategicguidancetofront-lineemployeesthatwouldotherwisebeunavailableThesesortsofdata-science-enhancedprocessescanbeclassifiedasoneofsixprojecttypesasexploredfurtherbytheNewOrleansLANOLAlyticsteamFormoreguidanceonscopingasimilarprojecttheGovExadvancedanalyticsscopingguidegivesyouastep-by-stepoverviewofhowtobesuccessful
Usethespacebelowtolistyourideasforpossibleadvancedanalyticsprojectsbytype
Typeofproject Ideas
Findingtheneedleinthehaystack
Prioritizingforimpact
Earlywarningtools
Betterquickerdecisions
Optimizingresourceallocation
Experimentingforwhatworks
Keepthefeedbackloopgoing
GatherinformationonpublicsentimentaboutpolicychangethroughfocusgroupssurveysandpublicmeetingsledbyadvisorygroupmembersNotewhethernewinformationisavailablethatcanhelpanalyzeorcontextualizeyourprogress(egfederalorstatedatareports)Sharetheresultsofyourpilotandthepolicychangeprocesswithothercitiesandresearchers
Shareyourstoryandsuccess
55
WorkingwitharangeofevidencePresentlythereisalotofattentionandinvestmentinevidence-basedpolicymakingincludingthefederalbipartisanCommissiononEvidence-BasedPolicymakingthePewCharitableTrustsreportsandtoolkitsonHowStatesEngageinEvidence-BasedPolicymakingandtheLauraandJohnArnoldFoundationsupportedEvidence-BasedPolicymakingCollaborativeManyoftheseinitiativesfocusonrigorousevaluation--meaningrandomizedcontrolledtrials(RCT)orotherexperimentalmodels--asthebestkindofevidencetouseHoweverthisbodyofevidenceisofteninsufficientforthespecifickindsofquestionsthatgovernmentscanhavewhenmakingadecisionaboutpolicyInthisguideweproposeanapproachtosupportgovernmentsinincorporatingadditionaltypesofevidenceintotheirdecisionmaking
OnewaytounderstandarangeofevidenceistouseNestarsquosStandardsofEvidencetoframeanddefineevidenceTheBritishnonprofitNestaworkstoinjectpositivechangeintothewayUKgovernmentpoliciesandprogramsfunctionOneoftheirkeyareasofworkhasbeendevelopingWhatWorksCentresorganizationsthatcollectcommissionsynthesizeanddisseminateevidenceaboutinterventioneffectivenessacrossarangeofsocialimpactareasNestadevelopedthefollowingStandardsofEvidencetoprovideausefulwaytodocumenthowdifferentkindsofinformationdemonstratedifferentlevelsofevidence
Workingwitharangeofevidence
56
Theimportanceofpotentiallybiasedinformation
Onechallengepresentedbyanevidenceclassificationthatacknowledgesthevalueofnon-scientificformsofevidenceistofigureouthowtobalanceallofthepiecestogetherwhenyouusethemTounderstandhowtodothiswehavetothinkabouthowtopositionpotentiallybiasedinformationindecisionmaking
WhenwelookatinformationwhichhasnotbeenrigorouslyevaluatedweacceptthatthisinformationmightbebiasedThewholereasonweconductrigorousevaluationistoremovebiasfromourobservationsWeconductstructuredtrialstoproduceresultsthatareconsistentandgeneralizableIfafindingisunbiasedthisaccuracymdashthislackofbiasmdashcanbeprovedbecauseanyindependenttestercanreproduceitwhentheyfollowanidenticalprocess
YoucanlookatpotentiallybiasedinformationasjustinformationthatisearlieroninitspathtorefinementreplicationandverifiedaccuracyApieceofinformationbasedonanyoneindividualrsquosobservationsislikelytobebiasedbecauseitiscomingfromjustonesourceBiasisreducedbygatheringtogetherlotsofindependentsourcesofinformationandfiguringoutitsmostcommonattributes
MaybethebiasedinformationmakesaclaimthatistoolargemdashitsayssomethingistrueunderallconditionswhenitisreallyonlytrueundercertainconditionsorthataneffectofaninterventionisbiggerthanitstrueaverageorthataneffectlastsovertimewhenitdoesnotPerhapschangingtheclaimtobemorespecificmdashcriticallyreviewingtheinformationtoseeifspecificconditionsmustbemettoachievetheoutcomesmdashwouldmaketheclaimmoreaccurate
IfyoutestedyourclaimfounditdidnotholdupinsomecaseschangedittesteditagainuntilyourclaimrepeatedlyheldupthenyoucoulddemonstratethattheclaimwasnolongerbiasedbutaccurateThisprocessoftestingimprovingtestingandvalidatingiswhatwegetfromstructuredscientificstudy
Withindividualpiecesofinformation--crediblehypothesescasestudiesexpertopinion--youcannotassumethatkindofaccuracyAtthesametimetheseearlyobservationsarefoundationaltoanyreplicablescientificobservationsWewillnevergetanywhereifwedonotstartwiththesepiecesofinformationeveniftheymightbebiased
Combininglevelsofevidencetogetthemostdetailedinformationforyourcity
ItisespeciallyimportanttoacceptdifferentformsofevidencewhenyoufacequestionsthathavenotyetbeentestedbyindependentstudiesSomeofthemostimportantquestionsrelatetothingsthatareveryspecifictoyoursituationYoufaceaparticularcombinationof
Workingwitharangeofevidence
57
issuesactorstrendsandresourcesthatisultimatelydifferentfromeveryotherplaceintheworld
ThisisimportantbecauseaspolicypractitionersandnotscientistsyoucaremoreaboutyourspecificcasethanahypotheticalaveragegeneralcaseYoucarelessaboutthetotallikelyrangeofimpactsfromapolicythanhowthissingleimplementationwillworkYouwantinformationaboutpoliciesthatwillprovideyouwiththebestpossiblepredictionabouthowtheywillworkinyourcontext
ByputtingdifferentkindsofevidenceintothesameframeworkNestarsquosStandardsofEvidenceimplythatwhenyoucannotgetrigorouslytestedexperimentallyverifiedinformationitisappropriatetoworkwithwhatwedohaveThisisrelevantforthemunicipalimplementationcontextbecausenostudypulledfromaliteraturereviewwillaccountforallofthevariablesyoufaceinyourowncontext
Intheabsenceofhavingastudythatpredictshowaninterventionworksinyourspecificcontextyouwillneednon-scientificinformationtotrytofillinthosegapsYouwillneedtogatherstoriesfromresidentsstaffmembersandotherstakeholdersthatsitonLevels1or2ofthestandardsofevidence
Someofthekindsofinformationyouwillprobablygetmoreeasilyfrompeoplethanstudiesinclude
howpeopleincitieslikeyoursreactedtoaninterventioninpublicforumshowpeoplewhoimplementedaninterventioninacitylikeyoursfiguredouthowtogetitpassedhowthecitycommunicatedpubliclyabouttheapparentbenefitsanddrawbacksoftheinterventionthepoliticalacceptabilityoftheintervention
Youcanremainsimultaneouslyawarethattheinformationyouaregatheringthroughindividualstoriesisbiased--notgeneralizableinformedbyspecificperspectives--whilealsorecognizingthathavingsomeinformationaboutsomethingimportantisbetterthannoinformationEvidencefromthefirsttwoevidencelevelsareessentialbuildingblocksofknowledgeandperfectlyusefulwhenyoucannotgetdetailedunbiasedknowledge
Youhavetostartsomewhereandyoushouldalwaysstartwithevidenceyouhaveaboutyourveryspecificconditionsthatyouknowareimportantforsuccess
Workingwitharangeofevidence
58
Thankyou
Interviewswithcities
GovExheldsemi-structuredinterviewswithsixstaffmembersinfourcitiestoestablishtheinitialstepsoftheRoadmapWeaskedrespondentstotalkaboutaninitiativeorpolicythattheyhavebeenresponsibleforimplementingintheircityRespondentsfromthreecitiesspokeabouttheireffortstoaddresshomelessnessandrespondentsfromonecityspokeaboutblightAfterrespondentsidentifiedtheproblemtheyspokeabouthowthecityprioritizedthisissuethevariousstepsinvolvedandbarrierspresent
GovExconductedtextanalysisoftheseinterviewswhichrevealedthatpeoplewasthemostcommonlyusedtermduringtheseconversationswhendiscussingthestepstochangeaprogramorpolicyThisisathemethroughoutthisRoadmapThefirstsectionbeginswithidentifyingtherelevantpeopleTheconversationsaroundevidencearefocusedonhumanbehaviorratherthanmethodologyorstrengthofevidence
Thankyou
59
OncewehadadraftoftheRoadmapwefollowedupwithcitiesonanindividualbasistolearnmoreaboutwhatwouldbehelpfulWearegratefultothefollowingcitystafffortheinterviewsandindividualconversations
MichaelBaskinChiefPolicyOfficerChattanoogaTN
MelissaBridgesPerformanceandInnovationCoordinatorLittleRockAR
VictoriaCarreoacutenAdministrativeOfficerLasVegasNV
TyroneGrandstrandCommunityDevelopmentPlannerFargoND
TibLaughlinDirectorofGeneralServicesKansasCityKS
TimMorelandDirectorofPerformanceManagementandOpenDataChattanoogaTN
CharlesRosendahlManagementAnalystDentonTX
Thankyou
60
MelissaSiebenAssistantCountyAdministratorKansasCityKS
GinaThompsonDirectorofStrategicServicesLewisvilleTX
JHUFacultycollaboration
GovExalsoconvenedthecitycontactsandfacultypartnersinaseriesofAdvisoryGroupmeetingsThesemeetingsservedtoconnectcitiesandfacultytounderstandoneanothersperspective
StefanBaralBloombergSchoolofPublicHealth
AshleyBernerJHUSchoolofEducation
MelissaDavey-RothwellBloombergSchoolofPublicHealth
JonLinksBloombergSchoolofPublicHealth
TamarMendelsonBloombergSchoolofPublicHealth
KristinMmariBloombergSchoolofPublicHealth
KeshiaPollackPorterBloombergSchoolofPublicHealth
JoshSharfsteinBloombergSchoolofPublicHealth
LizStuartBloombergSchoolofPublicHealth
LindsayThompsonCareyBusinessSchool
DanielWebsterBloombergSchoolofPublicHealth
Partnersindataandevidence-basedpolicy
Wealsospokewithpartnerswhoareexpertsinthefieldofevidence-basedpolicywhosharedtheirknowledgeandresourcesofcurrentandfuturethinkinginevidence-informedthinking
WendyBarrenoAssociateDirectorWilsonSheehanLabforEconomicOpportunitiesUniversityofNotreDame
TiffanyDavenportDirectorofDataPartnershipsBaltimoresPromise
TomiHiersDirectorBaltimoreCivicSiteAnnieECaseyFoundation
ChrisKingsleySeniorAssociateAnnieECaseyFoundation
KathrynOliverAssociateProfessorLondonSchoolofHygieneandTropicalMedicine
Thankyou
61
KristenWatkinsSeniorPolicyAssociateJ-PAL
GovExParticipants
GovExAnalystMiriamMcKinneyanalyzedtheinterviewtranscriptsandconductedtextanalysisSeniorImplementationAdvisorEmilyShawwillintegrateRoadmapactivitieswithadditionalservicedeliveryactivitiesatGovEx
Thankyou
62
ReferencesAbrahamKGampHaskinsR(2017)Thepromiseofevidence-basedpolicymakingReportofthecommissiononevidence-basedpolicymakingWashingtonDCCommissiononEvidence-basedPolicymakingBigmove2Extendingthegreen(nd)Retrievedfromhttpswwwcityoflewisvillecomdoing-businesslewisville-2025big-move-2-extending-the-greenBatesMAampGlennersterR(2017)ThegeneralizabilitypuzzleRetrievedfromhttpsssirorgarticlesentrythe_generalizability_puzzleBuildwithnotfor(2018January)RetrievedfromhttpsbuildwithnotforwordpresscomCairneyPampOliverK(2017)Evidence-basedpolicymakingisnotlikeevidence-basedmedicinesohowfarshouldyougotobridgethedividebetweenevidenceandpolicyHealthResearchPolicyandSystemsdoi101186s12961-017-0192-xCairneyPOliverKampWellsteadA(2016)TobridgethedividebetweenevidenceandpolicyReduceambiguityasmuchasuncertaintyPublicAdministrationReview76(3)399-402Retrievedfromhttpdxdoiorg101111puar12555Centreforpublicimpact(nd)RetrievedfromhttpswwwcentreforpublicimpactorgCernaL(2013)ThenatureofpolicychangeandimplementationAreviewofdifferenttheoreticalapproachesOrganisationforEconomicCo-OperationandDevelopmentRetrievedfromhttpswwwoecdorgeduceriTheNatureofPolicyChangeandImplementationpdfColoradoSprings(nd)Retrievedfromhttpscoloradospringsgovcommunicationspagewhat-expect-when-you-speakupContinuumofcare(coc)program(nd)RetrievedfromhttpswwwhudexchangeinfoprogramscocCouncilpresidentkicksoff$12millionyouthfund(nd)Retrievedfromhttpwwwbaltimorecitycouncilcomcontentcouncil-president-kicks-12-million-youth-fundDurlakJAampDuPreEP(2008)ImplementationmattersAreviewofresearchontheinfluenceofimplementationonprogramoutcomesandthefactorsaffectingimplementationAmericanJournalofCommunityPsychology41327-350Retrievedfromhttpswwwncbinlmnihgovpubmed18322790Fishbonediagram(ishikawa)(nd)Retrievedfromhttpasqorglearn-about-qualitycause-analysis-toolsoverviewfishbonehtmlFivewhysGettingtotherootofaproblemquickly(nd)RetrievedfromhttpswwwmindtoolscompagesarticlenewTMC_5Whtm
References
63
FligsteinNampMcAdamD(2011)TowardageneraltheoryofstrategicactionfieldsSociologicalTheory29(1)1-26RetrievedfromhttpsociologyberkeleyedusitesdefaultfilesfacultyfligsteinFligstein20McAdam20Soc20Theory20PaperpdHeinkelP(2018January12)CitymullspermitsforbikesharingcompaniesRetrievedfromhttpwwwdentonrccomnewsdenton20180112city-mulls-permits-bike-sharing-companiesHowisareamedianincomecalculated(2012October)Retrievedfromhttpswwwhudexchangeinfofaqs843how-is-area-median-income-calculatedLampkinLMampHatryHP(2003)KeystepsinoutcomemanagementUrbanInstituteRetrievedfromhttpswwwurbanorgresearchpublicationkey-steps-outcome-managementMarshDampSmithM(2000)UnderstandingpolicynetworksTowardsadialecticalapproachPoliticalStudies484-21Retrievedfromhttpjournalssagepubcomdoiabs1011111467-924800247McCannL(nd)RetrievedfromhttplaurenellencomMoultonSampSandfortJR(2016)ThestrategicactionfieldframeworkforpolicyimplementationresearchPolicyStudiesJournal45(1)Retrievedfromhttponlinelibrarywileycomstore101111psj12147assetpsj12147pdfv=1ampt=jdkhhcp4amps=e184c625e1855a6b1615ccb99ee27b1c0b3212d8PaynterB(2018January1)What100citiesarelearningfromeachotherbysharingtheirdataRetrievedfromhttpswwwfastcompanycom40518004what-100-cities-are-learning-from-each-other-by-sharing-their-dataPeakacademy(nd)Retrievedfromhttpswwwdenvergovorgcontentdenvergovenmayors-officeprograms-initiativespeak-performancepeak-academyhtmlPinnockHBarwickMCarpenterCREldridgeSGrandesGGriffithsCJRycroft-MaloneJMeissnerPMurrayEPatelASheikhAampTaylorSJC(2017)Standardsforreportingimplementationstudies(stari)ExplanationandelaborationdocumentBMJOpenRetrievedfromhttpbmjopenbmjcomcontentbmjopen74e013318fullpdfProctorESilmereHRaghavanRHovmandPAaronsGBungerAGriffeyRampHensleyM(2011)OutcomesforimplementationresearchConceptualdistinctionsmeasurementchallengesandresearchagendaAdministrationandPolicyinMentalHealth38(2)65-76doi101007s10488-010-0319-7TandonSD(2018)ProgressincommunityhealthpartnershipsResearcheducationandactionRetrievedfromhttpswwwpressjhuedujournalsprogress-community-health-partnerships-research-education-and-actionProvanKGampKenisP(2007)ModesofnetworkgovernanceStructuremanagementandeffectivenessJournalofPublicAdministrationResearchandTheory
References
64
18(2)229-252Retrievedfromhttpsacademicoupcomjpartarticle182229935895SabatierPA(1988)Anadvocacycoalitionframeworkofpolicychangeandtheroleofpolicy-orientedlearningthereinPolicySciences21(3)129-168Retrievedfromhttpwwwjstororgstable4532139SandstromAampCarlssonL(2008)TheperformanceofpolicynetworksTherelationbetweennetworkstructureandnetworkperformancePolicyStudiesJournal36497-524Retrievedfromhttponlinelibrarywileycomdoi101111j1541-0072200800281xabstractSchwartzMampWilsonE(nd)WhocanaffordtoliveinahomeAlookatdatafromthe2006americancommunitysurveyRetrievedfromhttpswwwcensusgovhousingcensuspublicationswho-can-affordpdfShearerJCAbelsonAKouyateBLavisJNampWaltG(2016)WhydopolicieschangeInstitutionsinterestsideasandnetworksinthreecasesofpolicyreformHealthPolicyandPlanning31(9)1200-1211Retrievedfromhttpsdoiorg101093heapolczw052ThelessonsofadministrativedataHigh-profilepolicy-relevantresearchpoweredbyadministrativedata(nd)Retrievedfromhttpswwwpovertyactionlaborgsitesdefaultfilesresources20161114-The-Lessons-of-Administrative-DatapdfToolsandotherneedsforqualitativeandquantitativemethods(2015February)AgencyforHealthcareResearchandQualityRetrievedfromhttpwwwahrqgovprofessionalsquality-patient-safetytalkingqualityassesstoolshtml
WeaverRCampBagchi-SenS(2013)SpatialanalysisofurbandeclineThegeographyofblightAppliedGeography4061-70
References
65
UniversityPartners
Name Title Affiliation Area(s)ofexpertise
Contactinformation
Examplesofworkingwithnon-academicdecisionmakers
WendyPBarreno
AssociateDirector
WilsonSheehanLabforEconomicOpportunitiesUniversityofNotreDame
LEOsfocusareasincludehealtheducationhousingchildwelfareeconomicopportunitycriminaljustice
wbarrenonduduorleondedu
MelissaDavey-RothwellPhD
AssociateScientist
JohnsHopkinsBloombergSchoolofPublicHealth
Health mdavey1jhuedu410-614-5854
BaltimoreCityHealthDepartmentLeadersMarylandDepartmentofHealthAddictionRecoveryServicesinDCExecutiveDirectorsofcommunityorganizationsacrosstheUSthatfocusonHIVSTIsubstanceabuseandimplementationscience
ElizabethStuart
AssociateDeanforEducationandProfessorDepartmentsofMentalHealthBiostatisticsandHealthPolicyand
JohnsHopkinsBloombergSchoolofPublicHealth
Educationsocialpolicystatisticalmethods
estuartjhuedu
Resources
66
Management
Design-ThinkingResources
EngagementGuideforCivicTech
Journeymap
PerformanceManagementResources
DenverPeakAcademy
Proxymeasuresguide
Talkingaboutperformancewithoutcomplicatingit(video)
SettingPerformanceTargetsguide
Datavisualizationguide
Performancemanagementgettingstartedguide
Exercises
Fishboneexercise
5Whys
AdministrativeData
TheLessonsofAdministrativeData(JPAL)
EvaluationResources
Quantitativeandqualitativemethods(USDepartmentofHealthandHumanServicesAgencyforHealthcareResearchandQuality)
KeyStepsinOutcomeManagement(UrbanInstitute)
RCT-YES
Sageresearchmethods
BestPracticesforMixedMethodsResearchintheHealthSciences
ACCORDChecklist
Pilotandimplementationplanning
Benchmarkingexamplesandproxymeasures
Seattleopendataplaybook
Resources
67
Datasciencescopingguide
Resources
68
AboutGovExGovExhelpsgovernmentsusedataandevidencetoimproveresidentsrsquolivesWesupportgovernmentsinbuildingtheirinternalcapacitytooperatesustainableanalyticspracticeswhichimprovecommunityoutcomesinfieldsfrompublicsafetytoeconomicdevelopmentGovExhasaproventrackrecordofworkinginconcertwithpartnerstoprovidetechnicalassistanceandtrainingatscale
SinceourlaunchinApril2015GovExhas
Trainedover2000governmentemployeesinpersonandonlineintopicsincludingdatamanagementperformancemanagementopendatacommunityengagementandanalytics
Providedtechnicalassistancetomorethan100localgovernmentsacrosstheglobeimprovingtheiruseofdatatoachieveefficienciesinservicedeliveryandbetteroutcomesforresidents
Assessedthecapacityofmorethan130governmentalorganizationstodeterminehowleadersstaffandexternalstakeholdersmanageandusedata
Localgovernmentsareincreasinglyonthefrontlinesofaddressinglargescaleproblemswithglobalimplicationssuchasclimatechangepublicsafetyandeconomicinequality
Whileevidence-basedinterventionstohelpsolvethoseproblemsareavailablelocalgovernmentstaffhavelimitedtimeandcapacitytofindandimplementresearch-basedsolutionsAsaresultcitiescontinuetoworkthewaytheyalwayshave--makingincrementalimprovementstoprogramsbutnotalwayscreatingchangeatasystem-level
ThroughresourcedevelopmentandcollaborationwithJHUfacultyGovExissupportingcityleadersinusingdataandevidencetomakedecisionsandcreatinglastingrelationshipswithresearchers
TheRoadmapwasdevelopedbyGovExaspartofitspartnershipwithBloombergPhilanthropiesWhatWorksCitiesinitiative
AboutGovEx
69
- Title page
- Introduction and purpose of the Roadmap
- Using the Roadmap
-
- Gather information about the problem
- Strategically engage key partners
- Review the evidence and collect details
- Decide on a policy to implement based on local factors
- Establish a framework to monitor progress
- Assess data accessibility quality and security
- Design implementation and evaluation
- Share your story and success
-
- Working with a range of evidence
- Thank you
- References
- Resources
- About GovEx
-
Herearesomeoftheclearinghouses(collectionsofprogramevaluations)forevidence-basedpoliciesandprograms
ResultsFirstClearinghouseDatabaseThePewCharitableTrustsCollectstogethereightotherclearinghousestoprovideinformationoninterventionsinthefollowingcategoriesadultcriminaljusticeeducationchildwelfarehealthjuvenilejusticementalhealthsubstanceabuseandyouthdevelopmentIndividualclearinghousesincludedinResultsFirstare
BlueprintsforHealthyYouthDevelopment-foryouthdevelopmentandviolenceprevention
CaliforniaEvidence-BasedClearinghouse-forchildwelfareinterventions
TheLauraandJohnArnoldFoundationrsquosSocialProgramsThatWork-forsocialservicesprograms
TheUSDepartmentofEducationrsquosWhatWorksClearinghouse-foreducationinterventions
TheUSDepartmentofHealthandHumanServicesrsquoResearch-TestedInterventionPrograms(RTIPs)-forcancercontrolinterventions
TheUSDepartmentofHealthandHumanServicesrsquoTeenPregnancyPreventionEvidenceReview(TPPEvidenceReview)-forsexualriskbehaviorandsexualhealthoutcomes
TheUSDepartmentofJusticersquosCrimeSolutionsgov-forcriminaljusticejuvenilejusticeandcrimevictimservices
TheUSSubstanceAbuseandMentalHealthServicesAdministrationrsquosNationalRegistryofEvidence-basedProgramsandPractices(NREPP)-substanceuseandmentalhealthinterventions
TheUniversityofWisconsinPopulationHealthInstituteandRobertWoodJohnsonFoundationrsquosCountyHealthRankingsandRoadmapsWhatWorksforHealth(WhatWorksforHealth)-healthbehaviorsclinicalcaresocialandeconomicfactorsandthephysicalenvironment
UKWhatWorksCentresTheBritishgovernmenthascharteredanetworkofsubject-specificorganizationswhichcollectsynthesizeandreportoutevidenceontheeffectivenessofpolicyprogramsandpractices
NationalInstituteforHealthandCareExcellence(NICE)-healthandsocialcare
Reviewtheevidenceandcollectdetails
24
SuttonTrustEducationalEndowmentFoundation-educationalachievement
CollegeofPolicingWhatWorksCentreforCrimeReduction-crimereduction
EarlyInterventionFoundation-earlyintervention
WhatWorksCentreforLocalEconomicGrowth-localeconomicgrowth
CentreforAgeingBetter-qualityoflifeforolderpeople
WhatWorksCentreforWellbeing-wellbeing
WilsonSheehanLabforEconomicOpportunities(LEO)UniversityofNotreDameProvideslinkstotheirevaluationsofinterventionsincriminaljusticeself-sufficiencyeducationhealthandhousingandhomelessness
ClearinghouseforLaborEvaluationandResearchUSDepartmentofLaborProvidesresearchonworkforceandlabor-relatedpoliciesandprograms
J-PALEvaluationsAbdulLatifJameelPovertyActionLabStudiesonpoverty-reducinginterventionsfromaroundtheworld
CampbellSystematicReviewsCampbellCollaborationSystematicreviewofpolicyinterventionsbyscholarlycoordinatinggroupsinsevenissueareascrimeandjusticeeducationinternationaldevelopmentmethodssocialwelfareknowledgetranslationandimplementationanddisability
Benefit-CostResultsWashingtonStateInstituteforPublicPolicyAWashingtonState-charteredprogramtoestablishthelikelihoodofrelativecostsandbenefitsforprogramsacrossthefollowingareasjuvenilejusticeadultcriminaljusticechildwelfarepre-Kto12educationchildrensmentalhealthhealthcaresubstanceusedisordersadultmentalhealthpublichealthandPreventionworkforcedevelopmentandhighereducation
EffectiveInterventionsCentersforDiseaseControlandPreventionRepositoryforevidence-informedHIVpreventioncareinterventions(withstepsonhowtoimplement)
Journalsandindexes
OnceyouhavereviewedrelevantinformationaccessiblethroughclearinghousesitisworthtakingalookatindividualstudiespublishedasscholarlyarticlesYourreviewerswilldothisbysearchingarticleindexesandacademicjournalsUnfortunatelyanumberofusefulacademicjournalsandindexingplatformsmaynotbegenerallyaccessibletoyourreviewersifyoudonothaveasubscriptiontothemToaddressthisconsiderpartneringwithalocal
Reviewtheevidenceandcollectdetails
25
collegeoruniversitytoprovideyouwithalistofarticlesfromthefollowingsources(Thisisalsoanexcellentopportunitytoseekresearchsupportfromanyacademicstakeholdersyouidentifiedpreviously)
Indexes
GoogleScholarincludesmostpeer-reviewedonlineacademicjournalsandbooksconferencepapersthesesanddissertationspreprintsabstractstechnicalreportsandotherscholarlyliteratureincludingcourtopinionsandpatentsItallowsyoutospecifyapublicationrangeinyoursearchandprovideslinkstolikelysourcesforalllistedarticles
PolicyFileIndexisauniqueresourceforUSpublicpolicyresearchinthatitgrantsusersaccesstotimelyupdatedinformationfromover350publicpolicythinktanksnongovernmentalorganizationsresearchinstitutesuniversitycentersadvocacygroupsandotherentities
PolicyArchiveworkstocollectanddisseminatesummariesandfulltextsvideosreportsbriefsandmultimediamaterialofthinktankuniversitygovernmentandfoundation-fundedpolicyresearch
Anothersourceforhigh-qualityresearchisthereportsoftheCongressionalResearchServiceYoucanfindtheirreportsaggregatedhere
Specificjournals
Oftenthemostusefulacademicjournalsforunearthingpolicyorprogramreviewswillbeineachspecificsubjectarea--thatisineducationjournalsforeducationpoliciesYoucanuseageneralindexlikeGoogleScholartobegintolearnwhichjournalsaremostoftenusefulforyoursearchHoweveryoucanalsosearchthefollowingtwogeneralistpolicypublicationsforissuesandarticlesrelatedtoyourtopic
PublicAdministrationReviewJournalofPolicyAnalysisandManagement
Newssourcessubjectmatterexpertsandinterviews
AlthoughitisvaluabletoknowwhathasbeenexploredthroughascholarlyresearchframeworkitisalsousefultolearnwhatkindofinterventionsarebeingtriedinthefieldrightnowandhowpeoplefeelaboutthesetrialsSearchsubscriptionnewsaggregatorslikeLexisNexisorfreenewsaggregatorslikeGoogleNewsorYahooNewsforyourissueareaplusthewordpolicyorprogramtoseewhatkindsofinterventionsarebeingcoveredinthenews
Reviewtheevidenceandcollectdetails
26
DonotforgettocontacttheadvisorygroupmemberswhowerechosenfortheirsubjectmatterexpertiseInadditiontoprovidinggoodmaterialthroughinterviewsthemselvestheymayhaveideasaboutothersubjectmatterexpertsyoushouldcontacttoaskaboutpotentialinterventions
Peercitiesandnetworks
InadditiontoreadingabouttheworkofothercitiesitisidealifyouareabletospeakwithpeoplewhoareinvolvedwithcurrentimplementationsofinterestinginterventionsInyourcityyoulikelyhaverelationshipswithyourneighborsbutitcanbehelpfultolookoutsideofyourregionforsolutionsaswellTofindcounterpartsinothercitieswhoareworkingonaprogramyouwouldliketolearnmoreaboutyoucancontactstaffinintermediaryorganizations(likeGovExoryourprofessionalassociations)whomightbeinapositiontoputyouintouchimmediatelyorwhomightseeanothergoodrouteforsettingupaconversationHereisalistofnetworksthatmaybehelpful
InternationalCityCountyManagementAssociation(ICMA)NationalAssociationofCounties(NACO)NationalLeagueofCitiesUSConferenceofMayorsWhatWorksCities
Collectthedetails
Onceyourreviewershavefoundevidenceonpossiblepoliciesorprogramstoimplementtheyshouldcapturekeydetailsthatwillhelpdecisionmakersunderstandthepotentialeffectivenessandapplicabilityofeachpotentialintervention
Foreachsourcethathasprovidedinformationyouwanttoconveyusethetemplatebelowtocaptureasmanyofthefollowingdetailsaspossible(Youdonotneedtocompletethisforsourcesthatprovideduplicativeinformationorareotherwisenotthatuseful)
Programorpolicyname (egldquoHousingFirstrdquoorldquoblightedpropertiesregistryrdquo)
Sourcetype (egjournalarticleinterviewsubjectmatterexpertstudyassociationcommunityparticipationprocess)
Sourcedetails (egarticletitleandpublicationnameandtitleofinterviewee)
Evaluationmethodology
(egrandomizedevaluationcomparativeevaluationoftwodifferentgroupsbeforeafteranalysis)
Evidencelevel (egLevel12345)
Programorpolicy
Reviewtheevidenceandcollectdetails
27
description
Programorpolicygoal (1sentencecapturingtheprincipalaimoftheintervention)
Existingimplementations
(WherehastheinterventionbeentriedListasmanysitesasyouareawareof)
Characteristicsofimplementationsites
(ForeachimplementationsiteofferbasicgeographicIDmdashcityandstateThenyouwillwanttocollectthedetailswhicharerelevanttoyourdecisionmakersrsquoconsiderationsaboutwhatmakesalocationldquosimilartordquoorldquodifferentfromrdquoyoursituationYouwillatleastwanttoincludeforeachimplementationsitePopulationnumberPopulationdensity(ruralurbansuburban)Socio-economiccharacteristicsBeforestartingaskyourdecisionmakersaboutanyparticularcharacteristicsofprogramimplementationsitestheywouldwanttoknowbeforeseekingtoreplicate)
Characteristicsofimplementationpartners
(Anyavailabledetailsaboutorganizationsinstitutionsorpeoplewhowereinvolvedwithimplementingtheintervention-egindependentchildcarecenterpublicelementaryschoolcitypublichealthnurses)
Characteristicsoftargetpopulationeligibleprogramparticipants
(Anyavailabledetailsaboutpeoplewhoarethebeneficiariesoftheprogramorotherwiseincludedoraffectedbytheintervention-egnumberofprogramparticipantscategoriesofpropertyownersaffected)
Policyorprogramduration
(HowlonghastheinterventionbeeninplaceineachimplementationsiteHasitendedandifsowhy)
Whatbenefitscamefromeachimplementation
(Anyobservedpositiveeffectsoftheinterventionincludingeffectsize)
Statisticalestimatesofinterventioneffectiveness
(Anyavailablestatisticalestimatesoftheinterventionrsquoseffects)
Whatdrawbackswereobserved
(Anyavailabledetailsaboutunintendedorunavoidablenegativeeffectsoftheintervention)
Programorpolicycosts
(Anyavailabledetailsaboutprogramcostsperimplementation)
Reviewtheevidenceandcollectdetails
28
Reviewtheevidenceandcollectdetails
29
DecideonapolicytoimplementbasedonlocalfactorsTogetthecriticalbuy-infromleadershiptoimplementanewpolicyorprogramtheprojectleadshouldscheduleameetingwithdecisionmakerstopresenttheevidencesurroundingthepolicyoptionscapturedinthetemplateDependingonyourinsightintoyourcityleadershipswillingnesstomakechangeyoumaywishtosharetheresearchyouhaveconductedontherootcausesoftheproblemandsomeofyouranalysisontheimpactofcurrenteffortstodate
Considertheapplicabilityofanapproach
AsdiscussedinthepreviouschapterpolicymakersoftenlookforevidenceofeffectiveprogramsinothercitiesbutoftenthequestionarisesofhowgeneralizableaprogramisfromonecitytoanotherTheAbdulLatifJameelPovertyActionLab(J-PAL)suggeststhatinreviewingevidencefromanotherlocationitishelpfultoidentifythespecificmechanismthatwasused(deliveryofprogramorserviceinformationsharingetc)identifythelocalconditionsthatwereinplaceobservethebehaviorchangethatresultedfromthemechanismandfinallytounderstandlocalcontextandconditions(legalorcostconstraintsculturalnormsetc)Thissetofpresentationelementscanbedrawnfromthelistofdetailscollectedintheresearchtemplate
EducationpolicymakersinChicagoILusedfindingsfromprogramsinKenyaandIndiainwhichchildrenwhohadfallenbehindtheirpeersweregiveneducationalsupporttargetedtotheirleveloflearningThemechanismusedwastargetedteachingandthereweresimilarconditionspresentinallthreecountriesAlthoughsomestudentshadfallenbehindinthecurriculumteacherswereincentivizedtoteachgrade-levelmaterialsConsiderationsofhowthelocalcontextinChicagomightaffectimplementationofatargetedteachingprogramincludedunderstandinghowtheneighborhoodsthatthestudentswerefrommayhaveaffectedtheireducationaloutcomesandthehighcostofteachinginsmallgroupsInChicagoMatchEducationprovidedtwo-on-onetutoringwhichmatchedstudentswithwell-educatedindividualswhohavenothadformalteachertrainingforalower-costmodelTheresultswerepositiveStudentswhoweretutoredperformedbetterinmathandweremorelikelytopasshighschoolclasses
ThegeneralizabilityframeworkproposedbyJ-PALisbelowUsethisasatemplateasyouconsiderpotentialpolicysolutions
Decideonapolicytoimplementbasedonlocalfactors
30
SourceTheGeneralizabilityPuzzle
OnceyouhavenarrowedthepolicyoptionsdiscussthequestionsbelowforeachoptionsyouareconsideringThisstepiscriticalasunderstandinglocalcontextissoimportantinsuccessfulpolicyimplementation
CommunityContextQuestionnaire
Localcontext
DoesthisoptionmakesenseinourcommunityIsthisaonesizefitsallapproachIsanyparticulargroupabsentfromthisapproachIfsoisthatok
Youranswers
Effectiveness
WhatwouldweexpecttoobserveifthepilotisaseffectivehereasinCityX
Youranswers
Timing
DoesthetimingmakesenseIsthistherighttimetoimplementthispolicychange
Decideonapolicytoimplementbasedonlocalfactors
31
WherearethepotentialroadblocksthatcouldaffectthescheduleIsthereanywaytomitigatethisHowlongwouldittaketoseeanimpact
Youranswers
Implementation
DowehavethestaffcapacitytoimplementthisArethereadditionalpartnersthatyoucanbringtothetabletosupportimplementationDoesthismakesensegivenourformofgovernment
Youranswers
Authority
DoesourcityhavetheauthoritytoenactthisIfsowhatistheappropriatemechanismAretherelegalbarriersArethereanyexistingpreemptionissuesfromthestateorfederalgovernmentCanthosebeaddressedthroughlegislationadvocacyIstherepoliticalsupportoroppositiontothis
Youranswers
Cost
HowmuchwillthiscostannuallyHowwillwepayforitCanthepolicybeimplementedwithexistingfunding
IfsoisthisfundingreliableHowwouldthispolicyoptionaffectourcitysbudgetintheshortandlongtermWhatotherfundingoptionscanweconsiderIsthereanalternativethatwouldcostlessandgiveussimilarresultsWillanyconstituencybeadverselyaffectediffundingforprogramsthatimpactthemisshiftedtoanotherprogrampolicyorinitiative
Youranswers
Decideonapolicytoimplementbasedonlocalfactors
32
EstablishaframeworktomonitorprogressNowthatyouhavereviewedtheevidenceconsideredlocalcontextandchosenyourpolicyitistimetodeterminehowyourcitywillmonitorprogressofthisnewapproachCitystaffandleadersneedtoestablishtheinfrastructureforongoingcontinuousdecisionmakingbasedonwhetherthepolicychangeisleadingtothedesiredoutcomesforresidentsandwhetherthechangeismeaningfulandrationaltofrontlineworkersSettingarbitrarygoalsatthelocallevelthatconflictwithgoalsatotherlevelsisconfusingtotheindividualsresponsibleforachievingthegoalsGoalsthataresettoohightoachievecanbedemoralizingtostaff
ThissectionincludesstepstosetupavenueinwhichCitystaffandleadershiproutinelyvisitkeyindicatorstodeterminewhatisorisnotworkingandtomakedecisionsinreal-timeFormorereadingonperformancemanagementpleasevisittheGovExPerformanceManagementGettingStartedGuide
InperformancemanagementtermssuchasindicatormeasureandmetricmaybeusedinterchangeablyInthisguidewewilluseindicatorInyourworkthemostimportantthingistobeconsistentinwhichevertermyouuseThisvideomaybehelpfulinunderstandingthisconceptandcommunicatingittoothers
Establishthefundamentalsandgatheryourteam
FirstdeterminewhetherthereisanestablishedvenueorregularmeetingscheduleduringwhichyoucandiscussprogressIfyourcityhasanexistingperformancemanagementstructureonarelatedtopicaddingadiscussionoftheprogressofpolicychangetoyouragendacanbeagreatideaIfthiswillbeastandalonemeetingdecidehowfrequentlytheprojectteamanddecisionmakerswillmeetandputthosemeetingsonthecalendarfortheentirelengthoftheproject
DeterminewhowillstaffandattendroutinemeetingsonCityprogressThesemeetingsmayincludetheMayorandorCityManagerdepartmentheadstaffwhoareimplementingthepolicychangeandanalysts
Identifyindicators
ItiscriticalthatindicatorsbeanimportantcityandcommunitypriorityaswellassomethingtheCityplaysamajorroleinaddressingTheremaybeinstancesinwhichtheCitywillneedtoreachouttoorcollaboratewithexternalpartnerstogetamorecompletepictureofprogressontheoverallpolicychange
Establishaframeworktomonitorprogress
33
Anoutcomeindicatordescribesthedesiredeffectofthetargetedpopulation
ExampleBellevuesfamilyfriendlyneighborhoodsindicatormeasuresthepercentofresidentswhobelievetheyliveinaneighborhoodthatsupportsfamiliesparticularlythosewithchildren
WithyourkeypartnersrevisityourproblemstatementcurrentstateandfuturestateexerciseandbegintoidentifytheoutcomeindicatorsthatwillillustratesuccessFornowchooseasingletop-levelkeyperformanceindicator(KPI)ItiscriticalthattheKPI-thepieceyouwillmeasure-beapriorityforthecommunityandsomethingtheCityplaysamajorroleinaddressing
Outputindicatorsmeasuretheprocessofprogramorservicedelivery
ExampleSanDiegoMayorKevinFaulconersgoaltorepair1000milesofstreetby2020
ReviewingtheadministrativedatasetinventorybegintoparedowntherelevantprogramsanddatasetsthatsupportyourKPIWhatarethemeasuresofeffectivenessforeachprogramTheseareyouroutputindicators
Thediagrambelowillustratestherelationshipbetweenoutputandoutcomeindicators
ThinkaboutimplementationindicatorsasmeasuringstepsintheprogramsrolloutthatarenecessarytoobtaindesiredchangesImplementationoutcomescanfocusontheinterventionstaffdeliveringtheinterventionanorganizationorsystemandcanaddressfeasibilityadaptabilitysustainabilityandcostTwokeyquestionstoconsiderwhendevelopingimplementationindicatorsare
1 Istheinterventionacceptableandappropriatetothecommunityand
2 IsitfeasibletoimplementinterventionXatthistimewiththeseresources
Establishaframeworktomonitorprogress
34
Amoreextensivelistwithdefinitionsofpotentialimplementationoutcomesislinkedhere
UsethetablebelowtochoosethedesiredimplementationoutcomesforyourpilotandtoplanyourmeasurementtechniqueThisshouldincludelevelofanalysis(individualproviderorconsumerorganizationorsetting)yourmeasurementtechnique(surveysobservationinterviewfocusgroupsadministrativedataanalysis)andthephasewhenyouwillcollectthisinformation(earlymidlate)Theremaybemorethanonelevelofanalysismeasurementtechniqueandphaseofdatacollection
Implementationoutcome
Definitionofoutcome
Levelofanalysisandmeasurement
Yourmeasurementtechnique
Phaseofpilotwhenyouwillcollectdata
Usethespacebelowtocaptureyourindicatorwishlist
Indicatorwishlist
Outcomes_____________________________________________________________________________
Outputs________________________________________________________________________________
Implementationoutcomes______________________________________________________________
Setapublichigh-levelgoal
Thehigh-levelgoaliscriticalinestablishingdirectiontogalvanizethepeoplewhowillbedoingtheworkWhendiscussingthegoalusetheMayorsbullypulpitLookforkeymilestonesinCitygovernmenttosharethegoalstatementsuchasthebeginningofaMayoraladministrationaStateoftheCityspeechorotherpublicremarks
GoalsshouldbeldquoSMARTrdquo
Specificaspossible(innamingthepopulationwhowillbebetterofftheactionthecitywilltake)
Measurable
Achievable
Establishaframeworktomonitorprogress
35
Results-focused
Time-bound
ThegoalstatementshouldincludeyouroutcomeindicatorlikeChattanoogasgoalldquoWewillendchronicveteranhomelessnessbyDecember2016rdquoAdiagramtohepyoubuildaSMARTgoalaroundyouroutcomeindicatorisbelow
Component Description Yourturn
Specific WhowillbebetteroffWhatactionthecitywilltake
Measurable Howwillyoumeasuresuccess
Achievable Isthegoalattainable
Results-focused Whatistheexpectedresult
Time-bound When
Setrealisticambitioustargetsandtimelinesforeachindicator
ToestablisharealisticandambitioustargetforyourgoalstatementusetheadministrativedatacollectedforyourchosenKPItoreviewactualCityperformanceinthemostrecentyearcomparedwiththethreeorfourmostrecentyearsworthofdataBegintocalculatesummarystatistics(egtotalminimumaveragemedianmaximum)toidentifytrendsCreatecumulativeandmonthlychartstoidentifytrendsandanomaliesBasedonyouranalysisofhistoricaldataignoretheanomaliesandassumetheproductivitytrendswillfollowthesameoverallpatternfrommonth-to-monthFinallyapplyanincreasedproductivityfactortotheperformanceeachmonthtoforecastfutureprogressBasedonthisanalysischooseatargetthatinspiresimprovementwhileaccountingforreality
Manycitiesthataremorematureintheirperformancemanagementpracticesrecognizetheimportanceoftakingacommunity-levelapproachtoimprovingoutcomesforresidentsThesecitiesarecreatinggoaltaxonomiesthatincludemeasuresrelatedtopublicsafetyhousingandevenrecreationCitiesdriveprogressonthesegoalsbyimplementingstrategiesthatengagemultipledepartmentsandcollaboratewithserviceprovidersinthecommunity
EventuallyyouwilldevelopgoalstatementsfortheoutputsinordertocapturetheefforttoachievetheoutcomecontainedinyourgoalstatementYoursetofindicatorsshouldbecomposedofshort-andinterim-termsupportivemeasureswithaccuratetargetsandrelevanttimelinesAnexamplegoaltaxonomyfromtheCityofLasVegassResultsVegasprogramisbelowUsethistaxonomyasamodelforyourownsuiteofindicatorsandgoalstatements
Establishaframeworktomonitorprogress
36
Yourturn
Thetemplatebelowismeanttohelpyoucraftagoalstatementforeachsupportingmeasure
Goalstatement
Departments
Outcome
Activities
KPI
Target
Timeframe
Conductanddiscussrelevantanalysis
ThrougharoutineseriesofmeetingscontinuallymonitorprogressofKPIssupportingindicatorsandanyotherdatacollectedsuchasqualitativedatatoidentifytrendsthatwillindicatewhetherthechosenpolicyishavingthedesiredimpactThesedatapointsshoulddrivethediscussionduringthesemeetingspointingtoopportunitiesforfurtherinquiryinto
Establishaframeworktomonitorprogress
37
whatisdrivingtheobservedtrendsAnalystsshouldbeencouragedandtrainedtoaskquestionsofthedepartmentsproducingthedataandconductsitevisitswhenpossibletodeepentheirunderstandingofthedataPresentationoftheanalysisshouldfocusononeideaatatimetokeeptheconversationfocusedonkeypointsandspecificopportunitiesforimprovementYoumayconsiderdevelopingapublicfacingdashboardtosharethisprogressmorebroadly
AgeneralcomparisonofthestatusquodatawithwhatwasobservedduringthepilotcanprovidedecisionmakerswithaquickunderstandingofthedifferencesbetweentheoldandnewapproachesalthoughthiswillnotshowcausalityOftenthesecomparisonswillincludeoperationalmetricssuchasdaysordollarssavedbutrememberthattheimpactonpeoplewillbemostimportantIfyouareworkingwithanevaluationpartnertheywillsupportthisstep
Usethespacebelowtosummarizetheresultsofyouranalysis
MeasureTargetvalue(seesectiononsettingtargetsabove)
Pilotvalue(whatdidyouobserveduringthepilot)
Historicalaverage(ifavailable)
Results(WhoisbetteroffIsanyonenegativelyimpacted1-2sentencesreassessingyourgoalstatement)
DocumentyourprocessandprogressincludingfeedbackfromtheparticipantsimplementationpartnersandtheprojectteamDidanythingunexpectedoccurduringthepilotArethereinternalandexternalfactorsthatmayaffectyouroutcomesThisiswheretheimplementationoutcomescanbecanhelpfulIftheinterventiondidnotgoasplannedconsiderwhetheritisbecausetheinterventionwasnotacceptableorfeasible
UsethetemplatebelowcreatedbyDenversPeakAcademytocapturethisinformationIfthe306090daytimelinedoesnotmakesenseforyourcityfeelfreetocustomizeit
CurrentState
FutureState
30Day
60Day
90day
Whatwentwell(+)
Whatdidnotgowellshouldbechanged(-)
IfaquestionordiscussionpointarisesthatcannotbeansweredinthemeetinganassignedstaffmembershouldcapturethequestionduringthemeetingAfterwardthefollowupitemsshouldbesenttorelevantpartnersandthenextmeetingshouldbeginwithadiscussionof
Establishaframeworktomonitorprogress
38
thoseitems
Takeaction
DiscussionsaboutwhetheryourindicatorsaremovingintherightdirectionandwhatmightbedrivingtrendsmayleadtoadiscoverythattherearenotenoughresourcesallocatedtothispolicyorissueTheremaybelimitedfundingforstaffprogrammingortechnologythatisdrivingthedatainthewrongdirectionIfCityleadershipisintheroomthisisagreattimetomakethecaseforadditionalfundingduringthebudgetseasonBycollectinganalyzinganddiscussingdataonthispolicyroutinelyyouarebuildingevidencetomakethecasefortherightleveloffundingtoreachtheestablishedtargets
Establishaframeworktomonitorprogress
39
Assessdataaccessibilityqualityandsecurity
Determineaccessibilityandqualityofdata
OnceyouhavecompiledyourwishlistofindicatorsitistimetodeterminetheaccessibilityandqualityofthedataTopreventcomplicationyoumaywishtoprioritizerelevantindicatorsforwhichyouarealreadycollectingdataAvoidthetemptationofadoptingatechnologysolutiontocollectnewdataatthispointsuchasaphotosurveytoolorotherappFornowyouaretryingtodeterminewhichdatasetsarerelevantaccessibleandhighqualityBesuretoworkwithyourkeyinternalpartnerswhileyouareidentifyingyourmeasuresDepartmentstaffwhoworkonprogramsarefamiliarwiththeindicatorsthatwillaccuratelysupporttheoutcomesanddatabaseadministratorswillbevaluablewhendiscussingaccessibilityandqualityofthedata
Abrieflistofconsiderationsfortheaccessibilityofyourdataisbelow
ArewecurrentlycollectingthisdatasetorwillwebecollectingitforthefirsttimeHowfarbackdoestherecordgo(timehorizon)WouldweneednewtechnologytocollectoranalyzethedataCanthedatabecollectedmanuallyorcanitbeautomatedHowfrequentlyisthedataproduced(continuouslyhourlydailyweeklymonthlyquarterlyyearlyetc)IsthedataavailablefortheentireCitypopulationorasubsetofthepopulationHowgranularisthedata(individuallevelCensusblockneighborhoodcitywideetc)DoesthedatacontainsensitiveinformationIsthereaproxymeasurethatwecanusefornowIsthedatamachinereadable
AsnotedbytheWilsonSheehanLabforEconomicOpportunities(LEO)attheUniversityofNotreDameleveragingadministrativedatacansignificantlyreducethecostsandtimelinesofrigorousevaluationCitiescanestablishaninternaldatasharingagreementacrossdepartmentsincludinglanguagethatenablesadministrativedatatobeusedforresearchpurposesCitiescanalsoidentifyexternaladministrativedata-holderssuchasacountyschooldistrictpolicedepartmenthospitaloranotherstatethatmayhaveoutcomedataimportantformeasuringprogramimpactCitiescanleverageexistingrelationshipsandorworkwithresearchinstitutionstodevelopstreamlineddatasharingprocessesandagreementswiththeseexternalentities
Assessdataaccessibilityqualityandsecurity
40
Evaluatethequalityofyourdata
QualityisasimportantasaccessibilityToevaluatedataqualityyoumaywishtocreatearubricliketheonebelow
Value ValueDescription
3 ExcellentDataQualityThedataiscompleteaccurateandupdatedregularly
2 AdequateDataQualityThedataissomewhatcompleteaccurateandupdatedfairlyregularly
1 PoorDataQualityThedataisnotcompleteaccurateorupdatedregularly
ThisisagreattimetobuildonthedatainventorythatyoustartedinthegatherinformationpageSuggestedadditionalfieldstoaddtoyourdatainventorytemplatearebelow
Program DatasetTitle
DatasetDescription DataSource
Departmentorexternalpartner
DoestheCityalreadycollectthisdata
Whatisthenameoftheprogram
Human-readablenameofthedataset
Shouldbeunderstandablebynon-technicalusers
Whatistheoriginalsystemapplicationfilethathousesit
Departmentorexternalpartnerthatmaintainsthedata
Isthisdataalreadycollected
Updatefrequency-ThefileformatforthedatasetTypicallycsvxlsxshapefiletxtetc
Format-Howoftenthedatasetisupdated
Timerange-Theperiodoftimecoveredinthisparticulardataset
Datause-Howisthisdatasetcommonlyused
Datausers-Whotypicallyusesthisdataset
Dataownerstitle-Titleofthedataowner
Dataownersposition-DataOwnersjobtitle
Dataownersemailaddress-DataOwnersemailaddressWhomanagesthedataandorisresponsibleforgrantingpermissiontoaccessthedataWhounderstandswhatthedatasetincludesandcananswerquestionsaboutit
Dataqualityconcerns-Describetheconcernsyoumayhaveaboutthequalityofthedataset
Assessdataaccessibilityqualityandsecurity
41
Division-DivisionwherethedataismaintainedIfmorethanonedivisionisresponsiblelisttheprimaryownerofthedataset
Sensitivedatacomments-DescribetheissuesthatwouldpreventthecityfromsharingthedatapubliclyCommonsensitivityconcernsincludeprivacyviolationssecurityissuesorhighcostorstaffdemandstopostthedata
Publishingstatus-Thistheplacetoputanycommentsregardingthecurrentstatusofthedataset
Publishedlink-Thisisthelinktothepublisheddatasetontheopendataportalorcitywebsite
Reviewdatasecurityandprivacyprotectionprotocols
OftenapolicychangealsoinvolvesachangeinthedatathatisbeingcollectedhowthatdataisbeinghandledwhohasaccesstoitandahostofotheradjustmentsAsyoutakestockofthedataassetsthatareavailableforyourinitiativeitisvitaltoalsothinkthroughwhatwillbedonetoprotectthatdataandtheprivacyofthepeopleitrepresentsAbigpartofgooddatasecurityinvolvesstrongdatagovernancepracticesthatdefineandidentifysensitivedataandconsciouslyworktobalancethevalueandriskofstoringsuchsensitivedataOnewayofevaluatingyourdatacomesfromcaselawrelatedtotheUSFreedomofInformationActwhichwehavetranslatedtothechecklistbelow
1 Woulddisclosureresultinasubstantialinvasionofprivacy2 Haveweconsideredtheextentorvalueofthepublicinterestandthepurposeorobject
oftheindividualsseekingdisclosure3 Istheinformationavailablefromothersources4 Wastheinformationgivenwithanexpectationofconfidentiality5 Isitpossibletomouldreliefsoastolimittheinvasionofindividualprivacy
TheseriskscouldbemitigatedusingavarietyofstrategiesfromaggregatingdatatoobfuscatingindividualstoredactingfieldsthatmayincludePersonallyIdentifiableInformation(PII)orre-evaluatingyourdatacollectionmethodstofindalternativewaysofgettingtheinformationyouneedThesestepsshouldbecodifiedandmanagedbyadatagovernancecommitteeifpossibleForthesensitivedatathatisdeemedessentialcreateaplanaroundpotentialdatabreachesinordertoquicklyandconfidentlyrespondtoemergencysituations
ProactivelypreparingforthesesituationsisoftenthebeststrategyfordatasecuritybutdonotforgetthetechnicalsideInvolveyourITadministratorsinconversationsaboutdatastorageandaccessensuringthattheywillbeprotectedbyenterprise-gradecybersecuritysystems
Assessdataaccessibilityqualityandsecurity
42
Assessdataaccessibilityqualityandsecurity
43
Designimplementationandevaluation
Designtheimplementation
NowthatyouhaveyouraccountabilityplanandkeyindicatorsitistimetogetspecificaboutyourplanforimplementationThetemplatebelowwillhelpyouthinkthroughthespecificmechanismsofimplementingpolicychange
UnderstandingorganizationsandsystemsiscriticaltounderstandingandmakingdecisionsaboutwhereandhowinterventionsshouldbeappliedApolicyimplementedinasinglesettingmayshowpromisingresultsbutimplementingmultiplepoliciesandprogramsatthesametimewithdifferentactorsresponsibleforeachaspectcanhelpensuresustainablebehaviorchange
InSomervilleMAShapeUpSomerville(SUS)isacitywidepublichealthprogramthatisimplementedacrossmultiplelevelsofinfluenceandsystemsthroughinterventionsbeforeduringandafterschooltoincludechangestobuiltenvironmentsuchasbikelanesregionalmasstransportationandphysicaleducationequipmentinschoolsandgymstoincreasephysicalactivitywellnesspoliciesandschoolgardensnutritioneducationduringandafterschoolandhealthyrestaurantprogramstosupporthealthyfoodandhealthyeatinghabits
Usethetemplatebelowtoplanyourimplementation
Policyimplementationplanningtemplate
ListthespecificdesiredoutcomesInthepreviousstepyouworkedwithdepartmentstafftoestablishspecificmeasurableoutcomesandinterimindicatorsPrioritizetheindicatorsthatwilldemonstrateprogressUsethespacebelowtocapturethedesiredoutcomesofthepolicychange
ExampleShapeupSomervillebeganwiththegoalofpreventingobesityin1stthrough3rdgraders
Yourturn
____________________________________________________________________________
____________________________________________________________________________
Designimplementationandevaluation
44
SpecifythetargetpopulationAreyoudesigninganinterventionthattargetsonlycertainsitesAreyouplanningonimplementingaprogramorinitiativeforaspecifictargetaudienceClearlyspecifythepopulationyouplantotarget
IfyouplantoconductarigorousimpactevaluationasnotedbytheWilsonSheehanLabforEconomicOpportunitiesitisimportanttoestablishconcreteobjectiveeligibilitycriteriaandensurethatthesecriteriaareappliedconsistentlybyallstaffduringscreeningandintakeItisalsovitaltoidentifytheindividualswhoareeligibleforaparticularprogramorserviceevenifeveryoneeligiblecannotbeservedbyexistingresourcesThishelpscitiesidentifyanyunmetneedsandguidefutureresourceallocationwhilealsocreatingthepossibilityofarigorousimpactevaluationbyidentifyingapotentialcomparisongroupFinallywhenconductingscreeningandintakeitisimperativetocollectandmaintainbasicinformation(egnamedateofbirthgender)aboutallprogram-eligibleindividualsnotjustthosewhoparticipateintheprogram
ExampleIntheShapeUpSomervilleexampletheprogrambeganbytargeting1stthrough3rdgraders
Yourturn
____________________________________________________________________________
____________________________________________________________________________
DefinethelocationWherewillthepolicychangetakeplaceWillimplementationtakeplaceatonesiteormultiplesites
ExampleShapeUpSomervilleinterventionsweredeliveredinschools
Yourturn
____________________________________________________________________________
____________________________________________________________________________
ClarifyimplementationpartnersandarticulatetheirrolesWhatarethekeyorganizationsinstitutionsorpeopleresponsibleforimplementationTheprojectteamshouldbeasclearandspecificaspossibleinprovidingadescriptionoftheconcreteactivitiesthatimplementationpartnersaremeanttocarryoutPracticeprofilesmaybeahelpfultooltohelpdefinewhatisrequiredoftheactorswhoareresponsibleforimplementingtheintervention
Designimplementationandevaluation
45
ExampleInSomervillemayoralleadershipandengagementofmultiplecityagencies(publichealthprimarycareschoolsCambridgeHealthAllianceetc)streamlinedandconnectedthevariousShapeUpSomervilleinitiativesandgrantswhichhelpedtokeepthetalentandenergyfocusedonhealthyeatingandactivelivingforallsectorsofSomervilleSUSpartnersultimatelyincludedschoolsnonprofitsCityandstateagenciesandnationalfoundations
Yourturn
____________________________________________________________________________
____________________________________________________________________________
DescribetheinterventionWhatarethestepsactivitiesorprocessesRememberthatthemechanismthespecificstepsorprocessthatleadtobehaviorchangeiskeytounderstandingtheeffectivenessofthepolicychangeandsharingtheresultswithothers
ExampleGoalsofShapeUpSomervilleweretoinfluenceeverypartofanearlyelementaryschoolchildrsquosdayndashbeforeduringandafter-schooltopromotehealthyeatingphysicalactivityandtheirweightstatusInterventionincludedfoodserviceenhancementstoimprovebreakfastandlunchandstaffprofessionaldevelopmentwalktoschoolactivitiesSUSclassroomcurriculumandprofessionaldevelopmentschoolwellnesspolicydevelopmentSUSafter-schoolcurriculumandprofessionaldevelopmentoutreachandeducationtohomethroughmaterialsforumseventsldquoSUSapprovedrdquorestaurantscommunityoutreachandcapacitybuildingthroughpolicydevelopmenttrainingsmediaplacements
Yourturn
____________________________________________________________________________
____________________________________________________________________________
ClarifytimelineandmilestonesfordeliveryandevaluationHowfrequentlywilltheservicesinterventionbedeliveredHowlongwilltheprogramrunWhatisthetimetoexpectedimpacts(eg12monthsafterprogramcompletion)
Designimplementationandevaluation
46
ExampleShapeUpSomervilleledbyTuftsUniversityFriedmanSchoolofNutritionScienceandPolicyconductedaquasi-experimental(nonrandomized)interventionstudyamong1stthrough3rdgradestudentsinSomerville(intervention)andtwocomparisoncommunitiesfor9-monthsduringoneschoolyear(fall2002ndashspring2003)ThestudyshowedpositiveassociationoftheinterventiononchangeinBMISUSreleasedreportssummarizingevaluationsprogresstodateplansforadditionalprojectsandimpactstenandfifteenyearsintotheinitiative
Yourturn
____________________________________________________________________________
____________________________________________________________________________
PlanyouranalysisTheprevioussectionaddressedestablishinganaccountabilityframeworktomonitorprogressandimplementationDeterminewhichindicatorswillbediscussthroughthatprocesswhowillsharethedataandanalysisandhowfeedbackfromthoseconversationswillbeincorporatedbytheprojectteam
ExampleTheMayorrsquosofficeimplementedadata-drivendecisionmakingapproachcalledSomerStatwhichwassynergisticwiththeapproachestakenbytheresearchandevaluationteamsdeployedforthespecificgrants
Yourturn
____________________________________________________________________________
____________________________________________________________________________
PlanforsustainabilityIfthepolicyandprogramchangeissuccessfulhowwilltheprojectteamsustainandexpandtoadditionalsitesorpopulations
ExampleAlthoughmostofthepartnershipsforgedintheearlyyearsweremaintainednewgrantsandcollaborationsexpandedattentiontoolderpublicschoolstudentsactivetransportationandtheGreenLinesubwayextensionthebuiltenvironmentfarmerrsquosmarketeffortssuchasldquoGrowninSomervillerdquowithUnionMainStreeteconomicdevelopmentandhealthdisparitiesaswellasdatacollectionmonitoringandsurveillance
Yourturn
Designimplementationandevaluation
47
____________________________________________________________________________
____________________________________________________________________________
Pilotplanningtemplate
IfyouplantoconductapilotfirstdeterminethepurposeofthepilotDoyouwanttoobservewhetherthepolicychangeresultsinthedesiredbehaviorchangeAreyouconcernedaboutfeasibilityscalabilityorcostDoyouwanttotestwhetheryourdesiredresultscanbeachievedthroughthispolicyapproachClarifythegoalofthepilotandthequestionsyouhopetoanswerThiswillhelpyouidentifythekeyindicatorsthatyouwillusetomeasurewhetherthepilotissuccessfulBeforebeginningestablishabaselineofcurrentperformanceandsetyourtargetsaccordinglybasedonwhatyouhopetoaccomplishthroughthepilot
NextestablishthepilotplanforyourprojectteamincludingmilestonesforteamcheckinstodeterminewhethertomoveaheadorchangecourseThisisseparatefromthemeetingsthatwilloccuraspartoftheaccountabilityframeworkUsethespacebelowtocreateyourtimelineanddevelopyourmeetingschedule
Task Personresponsible Startdate Checkindate Deliverable
Planyourevaluation
WhileyouaredesigningtheimplementationitisalsotimetobegintodevelopyourplanforanalysisAsnotedbytheWilsonSheehanLabforEconomicOpportunitiesdecisionsaboutprogramsetup-includingoutreachscreeningintakedatatrackingandenrollment-candeterminewhetherornotarigorousimpactevaluationwillbepossibleAsdiscussedcityleadersmayfindithelpfultoincludeevaluationpartnersinearlyconversationsaboutprogramsetupDuringpreviousstagesyoumayhaveworkedwitharesearchertoplanyourproposedpolicychangeYoushouldbeclearaboutthepurposeoftheevaluationanddevelopanevaluationplanBelowareadditionalconsiderations
Causationvstrendanalysis
IntheEstablishaframeworktomonitorprogressstagetherewasguidanceontrackingtrendsovertimeItisimportanttonoteherethatmonitoringdataisdifferentfromanalyzingacausaleffectTrendanalysiswillillustratethatchangesareoccurringandmayleadtoquestionsthatcouldbeansweredbyyourevaluationbuttrendanalysisalonedoesnotprovideevidenceaboutwhatisdrivingthosechangesYoumayseechangesinyour
Designimplementationandevaluation
48
outcomeindicatorsbasedonvariablesoutsideoftheCityscontrolForinstanceachangeinfederaltradepolicycanhaveanimpactemploymentratesinyourCityoutsideofyourCitysnewjobtrainingandplacementprogram
ToaddresscausalityyoumustcomparetheoutcomesofindividualswhoreceivedtheinterventionwiththosewhodidnotWithoutacomparisongroupofsimilarindividualswhowerenotexposedtotheinterventionitisimpossibletoknowwhichfactorsareresponsiblefortheoutcomesRandomizedControlTrials(RCTs)inwhichindividualsarerandomlyassignedtotreatmentandcontrolgroupsarethegoldstandardinprovingcausalityIfyoucandemonstratethatoutcomesimprovedforthosewhoreceivedtheinterventionanddidnotimproveatallorimprovedlessforacomparisongroupyoucanbesomewhatcertainthattheinterventionwasthecause
Examplesofothernon-causalimpactevaluationsthatarerigorousandreliableincludecertainquasi-experimentalstudiesusingcomparisongroupdesignssmartinterruptedtimeseriesanalysesthatmodeltrendsovertimeinwaysthatallowestimationofinterventioneffectswelldesignedpre-posttestanalysesneedsassessmentsimplementationstudiesetcFormoreinformationondifferenttypesofevaluationsseetheSageResearchMethodswebsite
Lookfordifferencesingroups
BuildintimetoalsoexaminethepilotdatatoseeifgroupsareexperiencingdifferentresultsReviewingperformancebasedonoutcomemeasuresforapopulationinaggregatemayhidethesubtlewaysthatsomepeoplemaybeaffecteddifferentlysoitisnecessarytocarefullyanalyzethedatafordisparateoutcomesForinstancetheMovingtoOpportunity(MTO)studythatexaminedtheimpactofhousingvouchersonfamilieswholivedinhigh-povertyneighborhoodsfoundthatchildrenwhomovedbeforetheageof13hadbettereducationandhigherincomesasadultsHoweveradultsdidnotseeimprovededucationandemploymentoutcomes
Disaggregatingdataintomorespecificcategoriesofraceandethnicitycanhelpunderstandtrendsanddisparitiesinamoregranularwaywhichcanhelpdecisionmakerstargetpoliciesandinterventionsinamoreequitableway
EarlierintheRoadmapweworkedtoidentifystakeholdersandwhomightbepositivelyandnegativelyimpactedbyachangeinpolicyUsetheresultsofthatexercisetoinvolvethecommunityininterpretingdataanalysisandtogatherqualitativefeedbackfromthecommunitytocallattentiontoissuesForinstanceanincreaseinyouthdetentionmaybeexplainedbyinefficienciesorlackofaccesstotransportation
Designimplementationandevaluation
49
Designimplementationandevaluation
50
Shareyourstoryandsuccess
Sharetheresults
Storytelling
AnintegralstepinyourstrategyissharingtheresultsofyourpilotorpolicychangewithyourcommunityThisincludesdevelopingmessagingaroundtheworkofyourorganizationandcreatingaplantodisseminateyourstoriesWhytellstoriesaboutyourwork
Informandorunderstandasubjectorsituation
Giveneededcontextofdataorbitsofinformation
Relatablestoriesarememorableforreadersviewers
ReadersorviewerswillseetheinformationasmorecompellingwhenthereisademonstratedandrelatablechangeorimpactonthemselvesortheirfriendsneighborsandcolleaguesStoriesareaboutpeopleandgovernmentisfundamentallyabouthelpingpeopleSotellthestoriesofthepeopleyouandyourcitycolleagueshelped
Keepthesetipsinmindasyoudevelopyourstorywithyourteam
WhatwereyourfindingsandwhyshouldanyonecareTellyouraudiencehowtheproblemwassolvedWhathastheimpactbeenonresidentsandcitystafftodateWhatwasthesuccessandweretheremultiplesuccessesalongthewayIfyouhavepreliminaryresultsordatashowingimprovementtosharethisisthetimetodoit
WhowasinvolvedandimpactedReviewthelistofstakeholdersHowwereeachofthemimpactedbytheprogramHumanizeyourstoryandimpactbyaddingadditionalvoicesThesevoicescanbeoneofthefollowing
Apersonorfamilywhowaspositivelyaffectedbythechangesuccess
Aninvolvedthird-partydescribingboththeproblemandthesolution(ieanorganizationnonprofitprovidersbusinesstradeassociationsportsteametc)whoispartofthecommunitythatthegovernmentworkedwithonthechangesuccess
Someonefromthegovernmentitselfwhocanspeakaboutthechangesuccess
TellmoreaboutyourcityDescribeyourcityincludingpopulationdemographicsindustriesetcWhatareitschallengesWhatisitknownforWhatmakesituniqueHowisitsimilarordifferentfromothercitiesintheUS
Shareyourstoryandsuccess
51
DescribeyourcityrsquoschallengeWriteoutthechallengeyourcityisfacing(asdefinedinyourproblemstatement)HowdoesitaffectresidentsHowdoesthisproblemaffecttheworkofcitystaffmembers
AddthedetailsProvidedetailsonyourappliedresearchstrategyTalkaboutwhatyourcityhasdonenthepasttoresolvetheissueDiscusshowyourstrategyhaschangedwhenyoubegantoadoptthisapproachWasmoneysavedorinvested
Usethisstorymapexercisetoorganizeyourthoughtsandguaranteethatyouincludealloftheelementsofatraditionalstory
AfteryoudevelopyourmessagingcreateacommunicationsmarketingandoutreachstrategydesignedtospreadyourmessageYourstrategyshouldwalkyouthroughthefollowingkeyactivities
1 IdentifyandprofileyouraudienceWhencreatinganddisseminatingyourmessageitisalwaysgoodtothinkaboutyouraudienceWhoexactlywouldbeinterestedinwhatyouhavetosayThegeneralpopulationofthecityPeoplewhoworkinnonprofitsBusinessownersItmattersbecausenoteveryonewholivesinacitymaybethetargetoftheinformationBusinessownersmaybemoreinterestedindatashowingeffectsonthebusinessdistrictthanothersandsoonWewillcoverthislaterbutthisstepwillalsohelpyoutothinkofthemosteffectivewaytoreachyouraudience
2 Defineyourcalltoaction(ifyouhaveone)SpendsometimeconsideringwhyyouareinterestedintellingyourstoryDoyousimplywanttoupdatethepubliconyourprogressorcelebrateyoursuccessesAreyouinterestedinyouraudiencechangingitsbehaviororsupportingaspecificinitiativebeinglaunchedbyyourorganizationIfyouwanttohaveyouraudienceactorbehaveinacertainwaybespecificandincorporateitinyourmessaging
Shareyourstoryandsuccess
52
3 SelectperformancemeasuresThinkabouthowyouwoulddetermineifyourcampaignissuccessfulandselectappropriateperformancemeasuresYouwillrevisitthesemeasuresattheendofyourcampaigncommunicationsplanimplementationtodeterminewhetheryouneedtomodifyyourcommunicationsandoutreachstrategyforthefuture
4 MethodstocommunicateyourstoryTherearemanywaystocommunicatesuccessofyourprojectHerearesomesuggestionsonreachingaudiencesItisimportanttodeterminewhichmethodsarethemosteffectiveforyourdesiredaudiencetoreceiveinformationDoyourresearch(PewResearchCenterhasexcellentstudiesonthereachofmediadigitalmediaandmarketing)ForexampleifseniorsareyourtargetpopulationandtheyarelessinclinedtouseInstagram(thatistrue)youmaywanttoconsiderFacebookorthenightlylocalnewsbothofwhichareusedmoreoftenbyseniorsBelowisalistofsomemethodsandtoolsyoucanuse
i Pressreleasestolocalnewspapersandtvstations
ii ShortsmartphonevideosAgreatwaytoengageanaudienceisthroughvideoAskonequestionofsomeonewhoispartofthestoryMaybethereisasocialworkerwhocandiscusshungerinyourcityThesevideoscanbereusedonmultipleplatforms
iii Long-formwriting
iv TellingindividualstoriesthroughaseriesofphotosandshortnarrativesforpublicationonFacebookorInstagram
v Op-edsopinionpiecesplacedinthelocalpaper
vi Emailupdatestoyourlist
vii Connectingwithappropriateorganizationsthatareinvestedinthetopicandworkwiththecommunityandaskthemtodisseminateinformationtotheirmembers
viii EmployingldquoWordofMouthrdquo(WOM)tacticsbycreatingtownhallsorfreeeventsforresidentstoattendintheirownneighborhoods
5 DeterminethetypeofcollateralUsuallycampaignsthattargetresidentsrequireseveralavenuestoreachthemPeoplearenotalwaysinthesameplaceorspaceHavingakeenunderstandingofyouraudiencewillassistyouwithyourchoicesonwhatneedstobedevelopedSometimesthatmeansimagesforsocialmediaaonepageflierthatgivespeopleinformationonattendinganeventoranin-depthbrochure
Shareyourstoryandsuccess
53
6 ImplementyourstrategyKeepinmindstrategiescanbefluidSometimessomethingmaynotturnoutasexpectedevenifduringplanningsessionsitwasldquosupposedtordquoItisokYoucananalyzewhathappenedduringadebriefsessionandonceyoursurveysandothermeasurementshavebeenfinalized
Visualizations
Withtherightvisualizationyoucanreviewandshareinformationinsightsanalysisperformanceandtrendsallofwhichisvaluabletoacross-sectionofthepublicButwhichvisualizationisrightforyouTochoosetherighttypeofchartgraphdeterminethetypeofquestionyouwanttoanswerSomecommoncharttypesarebelowandmoreinformationcanbefoundhere
Exploreadditionalanalyticaltechniques
Duringthegoal-settingphaseyouestablishedoutputoutcomeandimplementationmeasuresBasedontheresultsofyourpilotyoumayneedtoadjustyourmeasuresWiththepilotdatainhandyoumayhavesomeideasofhowfutureanalysescouldbedifferentandpotentiallyimprovethequalityofyourprogramTakesometimetobrainstormotherrelevantdatasetsthatyoudidnothavethetimetopursueearlierincludingmessydepartmentaldatathatneedsmoreattentionorsourcesofpublicdatafromthestateandfederalgovernmentnonprofitsandevenprivatecompaniesthatcouldsupplementyourunderstandingofanissueUsingmachinelearningtechniquesyoumightbeabletobetter
Shareyourstoryandsuccess
54
predictdemandforhomelessservicesusingNOAAweatherforecastspreemptwatermainbreaksusingsoilstratigraphyorinvestineconomicdevelopmentbymonitoringrealestatelistingdata
Advancedanalyticaltechniquescanalsohelpyourimplementationrunmoreeffectivelybyintegratingdata-informedtargetinganddecisiontoolsintotheprocessByautomatingroutineworkflowsandsurfacingsubtletrendsthesetoolscanhelpyouiterateinamoreprecisemannerandprovidekeystrategicguidancetofront-lineemployeesthatwouldotherwisebeunavailableThesesortsofdata-science-enhancedprocessescanbeclassifiedasoneofsixprojecttypesasexploredfurtherbytheNewOrleansLANOLAlyticsteamFormoreguidanceonscopingasimilarprojecttheGovExadvancedanalyticsscopingguidegivesyouastep-by-stepoverviewofhowtobesuccessful
Usethespacebelowtolistyourideasforpossibleadvancedanalyticsprojectsbytype
Typeofproject Ideas
Findingtheneedleinthehaystack
Prioritizingforimpact
Earlywarningtools
Betterquickerdecisions
Optimizingresourceallocation
Experimentingforwhatworks
Keepthefeedbackloopgoing
GatherinformationonpublicsentimentaboutpolicychangethroughfocusgroupssurveysandpublicmeetingsledbyadvisorygroupmembersNotewhethernewinformationisavailablethatcanhelpanalyzeorcontextualizeyourprogress(egfederalorstatedatareports)Sharetheresultsofyourpilotandthepolicychangeprocesswithothercitiesandresearchers
Shareyourstoryandsuccess
55
WorkingwitharangeofevidencePresentlythereisalotofattentionandinvestmentinevidence-basedpolicymakingincludingthefederalbipartisanCommissiononEvidence-BasedPolicymakingthePewCharitableTrustsreportsandtoolkitsonHowStatesEngageinEvidence-BasedPolicymakingandtheLauraandJohnArnoldFoundationsupportedEvidence-BasedPolicymakingCollaborativeManyoftheseinitiativesfocusonrigorousevaluation--meaningrandomizedcontrolledtrials(RCT)orotherexperimentalmodels--asthebestkindofevidencetouseHoweverthisbodyofevidenceisofteninsufficientforthespecifickindsofquestionsthatgovernmentscanhavewhenmakingadecisionaboutpolicyInthisguideweproposeanapproachtosupportgovernmentsinincorporatingadditionaltypesofevidenceintotheirdecisionmaking
OnewaytounderstandarangeofevidenceistouseNestarsquosStandardsofEvidencetoframeanddefineevidenceTheBritishnonprofitNestaworkstoinjectpositivechangeintothewayUKgovernmentpoliciesandprogramsfunctionOneoftheirkeyareasofworkhasbeendevelopingWhatWorksCentresorganizationsthatcollectcommissionsynthesizeanddisseminateevidenceaboutinterventioneffectivenessacrossarangeofsocialimpactareasNestadevelopedthefollowingStandardsofEvidencetoprovideausefulwaytodocumenthowdifferentkindsofinformationdemonstratedifferentlevelsofevidence
Workingwitharangeofevidence
56
Theimportanceofpotentiallybiasedinformation
Onechallengepresentedbyanevidenceclassificationthatacknowledgesthevalueofnon-scientificformsofevidenceistofigureouthowtobalanceallofthepiecestogetherwhenyouusethemTounderstandhowtodothiswehavetothinkabouthowtopositionpotentiallybiasedinformationindecisionmaking
WhenwelookatinformationwhichhasnotbeenrigorouslyevaluatedweacceptthatthisinformationmightbebiasedThewholereasonweconductrigorousevaluationistoremovebiasfromourobservationsWeconductstructuredtrialstoproduceresultsthatareconsistentandgeneralizableIfafindingisunbiasedthisaccuracymdashthislackofbiasmdashcanbeprovedbecauseanyindependenttestercanreproduceitwhentheyfollowanidenticalprocess
YoucanlookatpotentiallybiasedinformationasjustinformationthatisearlieroninitspathtorefinementreplicationandverifiedaccuracyApieceofinformationbasedonanyoneindividualrsquosobservationsislikelytobebiasedbecauseitiscomingfromjustonesourceBiasisreducedbygatheringtogetherlotsofindependentsourcesofinformationandfiguringoutitsmostcommonattributes
MaybethebiasedinformationmakesaclaimthatistoolargemdashitsayssomethingistrueunderallconditionswhenitisreallyonlytrueundercertainconditionsorthataneffectofaninterventionisbiggerthanitstrueaverageorthataneffectlastsovertimewhenitdoesnotPerhapschangingtheclaimtobemorespecificmdashcriticallyreviewingtheinformationtoseeifspecificconditionsmustbemettoachievetheoutcomesmdashwouldmaketheclaimmoreaccurate
IfyoutestedyourclaimfounditdidnotholdupinsomecaseschangedittesteditagainuntilyourclaimrepeatedlyheldupthenyoucoulddemonstratethattheclaimwasnolongerbiasedbutaccurateThisprocessoftestingimprovingtestingandvalidatingiswhatwegetfromstructuredscientificstudy
Withindividualpiecesofinformation--crediblehypothesescasestudiesexpertopinion--youcannotassumethatkindofaccuracyAtthesametimetheseearlyobservationsarefoundationaltoanyreplicablescientificobservationsWewillnevergetanywhereifwedonotstartwiththesepiecesofinformationeveniftheymightbebiased
Combininglevelsofevidencetogetthemostdetailedinformationforyourcity
ItisespeciallyimportanttoacceptdifferentformsofevidencewhenyoufacequestionsthathavenotyetbeentestedbyindependentstudiesSomeofthemostimportantquestionsrelatetothingsthatareveryspecifictoyoursituationYoufaceaparticularcombinationof
Workingwitharangeofevidence
57
issuesactorstrendsandresourcesthatisultimatelydifferentfromeveryotherplaceintheworld
ThisisimportantbecauseaspolicypractitionersandnotscientistsyoucaremoreaboutyourspecificcasethanahypotheticalaveragegeneralcaseYoucarelessaboutthetotallikelyrangeofimpactsfromapolicythanhowthissingleimplementationwillworkYouwantinformationaboutpoliciesthatwillprovideyouwiththebestpossiblepredictionabouthowtheywillworkinyourcontext
ByputtingdifferentkindsofevidenceintothesameframeworkNestarsquosStandardsofEvidenceimplythatwhenyoucannotgetrigorouslytestedexperimentallyverifiedinformationitisappropriatetoworkwithwhatwedohaveThisisrelevantforthemunicipalimplementationcontextbecausenostudypulledfromaliteraturereviewwillaccountforallofthevariablesyoufaceinyourowncontext
Intheabsenceofhavingastudythatpredictshowaninterventionworksinyourspecificcontextyouwillneednon-scientificinformationtotrytofillinthosegapsYouwillneedtogatherstoriesfromresidentsstaffmembersandotherstakeholdersthatsitonLevels1or2ofthestandardsofevidence
Someofthekindsofinformationyouwillprobablygetmoreeasilyfrompeoplethanstudiesinclude
howpeopleincitieslikeyoursreactedtoaninterventioninpublicforumshowpeoplewhoimplementedaninterventioninacitylikeyoursfiguredouthowtogetitpassedhowthecitycommunicatedpubliclyabouttheapparentbenefitsanddrawbacksoftheinterventionthepoliticalacceptabilityoftheintervention
Youcanremainsimultaneouslyawarethattheinformationyouaregatheringthroughindividualstoriesisbiased--notgeneralizableinformedbyspecificperspectives--whilealsorecognizingthathavingsomeinformationaboutsomethingimportantisbetterthannoinformationEvidencefromthefirsttwoevidencelevelsareessentialbuildingblocksofknowledgeandperfectlyusefulwhenyoucannotgetdetailedunbiasedknowledge
Youhavetostartsomewhereandyoushouldalwaysstartwithevidenceyouhaveaboutyourveryspecificconditionsthatyouknowareimportantforsuccess
Workingwitharangeofevidence
58
Thankyou
Interviewswithcities
GovExheldsemi-structuredinterviewswithsixstaffmembersinfourcitiestoestablishtheinitialstepsoftheRoadmapWeaskedrespondentstotalkaboutaninitiativeorpolicythattheyhavebeenresponsibleforimplementingintheircityRespondentsfromthreecitiesspokeabouttheireffortstoaddresshomelessnessandrespondentsfromonecityspokeaboutblightAfterrespondentsidentifiedtheproblemtheyspokeabouthowthecityprioritizedthisissuethevariousstepsinvolvedandbarrierspresent
GovExconductedtextanalysisoftheseinterviewswhichrevealedthatpeoplewasthemostcommonlyusedtermduringtheseconversationswhendiscussingthestepstochangeaprogramorpolicyThisisathemethroughoutthisRoadmapThefirstsectionbeginswithidentifyingtherelevantpeopleTheconversationsaroundevidencearefocusedonhumanbehaviorratherthanmethodologyorstrengthofevidence
Thankyou
59
OncewehadadraftoftheRoadmapwefollowedupwithcitiesonanindividualbasistolearnmoreaboutwhatwouldbehelpfulWearegratefultothefollowingcitystafffortheinterviewsandindividualconversations
MichaelBaskinChiefPolicyOfficerChattanoogaTN
MelissaBridgesPerformanceandInnovationCoordinatorLittleRockAR
VictoriaCarreoacutenAdministrativeOfficerLasVegasNV
TyroneGrandstrandCommunityDevelopmentPlannerFargoND
TibLaughlinDirectorofGeneralServicesKansasCityKS
TimMorelandDirectorofPerformanceManagementandOpenDataChattanoogaTN
CharlesRosendahlManagementAnalystDentonTX
Thankyou
60
MelissaSiebenAssistantCountyAdministratorKansasCityKS
GinaThompsonDirectorofStrategicServicesLewisvilleTX
JHUFacultycollaboration
GovExalsoconvenedthecitycontactsandfacultypartnersinaseriesofAdvisoryGroupmeetingsThesemeetingsservedtoconnectcitiesandfacultytounderstandoneanothersperspective
StefanBaralBloombergSchoolofPublicHealth
AshleyBernerJHUSchoolofEducation
MelissaDavey-RothwellBloombergSchoolofPublicHealth
JonLinksBloombergSchoolofPublicHealth
TamarMendelsonBloombergSchoolofPublicHealth
KristinMmariBloombergSchoolofPublicHealth
KeshiaPollackPorterBloombergSchoolofPublicHealth
JoshSharfsteinBloombergSchoolofPublicHealth
LizStuartBloombergSchoolofPublicHealth
LindsayThompsonCareyBusinessSchool
DanielWebsterBloombergSchoolofPublicHealth
Partnersindataandevidence-basedpolicy
Wealsospokewithpartnerswhoareexpertsinthefieldofevidence-basedpolicywhosharedtheirknowledgeandresourcesofcurrentandfuturethinkinginevidence-informedthinking
WendyBarrenoAssociateDirectorWilsonSheehanLabforEconomicOpportunitiesUniversityofNotreDame
TiffanyDavenportDirectorofDataPartnershipsBaltimoresPromise
TomiHiersDirectorBaltimoreCivicSiteAnnieECaseyFoundation
ChrisKingsleySeniorAssociateAnnieECaseyFoundation
KathrynOliverAssociateProfessorLondonSchoolofHygieneandTropicalMedicine
Thankyou
61
KristenWatkinsSeniorPolicyAssociateJ-PAL
GovExParticipants
GovExAnalystMiriamMcKinneyanalyzedtheinterviewtranscriptsandconductedtextanalysisSeniorImplementationAdvisorEmilyShawwillintegrateRoadmapactivitieswithadditionalservicedeliveryactivitiesatGovEx
Thankyou
62
ReferencesAbrahamKGampHaskinsR(2017)Thepromiseofevidence-basedpolicymakingReportofthecommissiononevidence-basedpolicymakingWashingtonDCCommissiononEvidence-basedPolicymakingBigmove2Extendingthegreen(nd)Retrievedfromhttpswwwcityoflewisvillecomdoing-businesslewisville-2025big-move-2-extending-the-greenBatesMAampGlennersterR(2017)ThegeneralizabilitypuzzleRetrievedfromhttpsssirorgarticlesentrythe_generalizability_puzzleBuildwithnotfor(2018January)RetrievedfromhttpsbuildwithnotforwordpresscomCairneyPampOliverK(2017)Evidence-basedpolicymakingisnotlikeevidence-basedmedicinesohowfarshouldyougotobridgethedividebetweenevidenceandpolicyHealthResearchPolicyandSystemsdoi101186s12961-017-0192-xCairneyPOliverKampWellsteadA(2016)TobridgethedividebetweenevidenceandpolicyReduceambiguityasmuchasuncertaintyPublicAdministrationReview76(3)399-402Retrievedfromhttpdxdoiorg101111puar12555Centreforpublicimpact(nd)RetrievedfromhttpswwwcentreforpublicimpactorgCernaL(2013)ThenatureofpolicychangeandimplementationAreviewofdifferenttheoreticalapproachesOrganisationforEconomicCo-OperationandDevelopmentRetrievedfromhttpswwwoecdorgeduceriTheNatureofPolicyChangeandImplementationpdfColoradoSprings(nd)Retrievedfromhttpscoloradospringsgovcommunicationspagewhat-expect-when-you-speakupContinuumofcare(coc)program(nd)RetrievedfromhttpswwwhudexchangeinfoprogramscocCouncilpresidentkicksoff$12millionyouthfund(nd)Retrievedfromhttpwwwbaltimorecitycouncilcomcontentcouncil-president-kicks-12-million-youth-fundDurlakJAampDuPreEP(2008)ImplementationmattersAreviewofresearchontheinfluenceofimplementationonprogramoutcomesandthefactorsaffectingimplementationAmericanJournalofCommunityPsychology41327-350Retrievedfromhttpswwwncbinlmnihgovpubmed18322790Fishbonediagram(ishikawa)(nd)Retrievedfromhttpasqorglearn-about-qualitycause-analysis-toolsoverviewfishbonehtmlFivewhysGettingtotherootofaproblemquickly(nd)RetrievedfromhttpswwwmindtoolscompagesarticlenewTMC_5Whtm
References
63
FligsteinNampMcAdamD(2011)TowardageneraltheoryofstrategicactionfieldsSociologicalTheory29(1)1-26RetrievedfromhttpsociologyberkeleyedusitesdefaultfilesfacultyfligsteinFligstein20McAdam20Soc20Theory20PaperpdHeinkelP(2018January12)CitymullspermitsforbikesharingcompaniesRetrievedfromhttpwwwdentonrccomnewsdenton20180112city-mulls-permits-bike-sharing-companiesHowisareamedianincomecalculated(2012October)Retrievedfromhttpswwwhudexchangeinfofaqs843how-is-area-median-income-calculatedLampkinLMampHatryHP(2003)KeystepsinoutcomemanagementUrbanInstituteRetrievedfromhttpswwwurbanorgresearchpublicationkey-steps-outcome-managementMarshDampSmithM(2000)UnderstandingpolicynetworksTowardsadialecticalapproachPoliticalStudies484-21Retrievedfromhttpjournalssagepubcomdoiabs1011111467-924800247McCannL(nd)RetrievedfromhttplaurenellencomMoultonSampSandfortJR(2016)ThestrategicactionfieldframeworkforpolicyimplementationresearchPolicyStudiesJournal45(1)Retrievedfromhttponlinelibrarywileycomstore101111psj12147assetpsj12147pdfv=1ampt=jdkhhcp4amps=e184c625e1855a6b1615ccb99ee27b1c0b3212d8PaynterB(2018January1)What100citiesarelearningfromeachotherbysharingtheirdataRetrievedfromhttpswwwfastcompanycom40518004what-100-cities-are-learning-from-each-other-by-sharing-their-dataPeakacademy(nd)Retrievedfromhttpswwwdenvergovorgcontentdenvergovenmayors-officeprograms-initiativespeak-performancepeak-academyhtmlPinnockHBarwickMCarpenterCREldridgeSGrandesGGriffithsCJRycroft-MaloneJMeissnerPMurrayEPatelASheikhAampTaylorSJC(2017)Standardsforreportingimplementationstudies(stari)ExplanationandelaborationdocumentBMJOpenRetrievedfromhttpbmjopenbmjcomcontentbmjopen74e013318fullpdfProctorESilmereHRaghavanRHovmandPAaronsGBungerAGriffeyRampHensleyM(2011)OutcomesforimplementationresearchConceptualdistinctionsmeasurementchallengesandresearchagendaAdministrationandPolicyinMentalHealth38(2)65-76doi101007s10488-010-0319-7TandonSD(2018)ProgressincommunityhealthpartnershipsResearcheducationandactionRetrievedfromhttpswwwpressjhuedujournalsprogress-community-health-partnerships-research-education-and-actionProvanKGampKenisP(2007)ModesofnetworkgovernanceStructuremanagementandeffectivenessJournalofPublicAdministrationResearchandTheory
References
64
18(2)229-252Retrievedfromhttpsacademicoupcomjpartarticle182229935895SabatierPA(1988)Anadvocacycoalitionframeworkofpolicychangeandtheroleofpolicy-orientedlearningthereinPolicySciences21(3)129-168Retrievedfromhttpwwwjstororgstable4532139SandstromAampCarlssonL(2008)TheperformanceofpolicynetworksTherelationbetweennetworkstructureandnetworkperformancePolicyStudiesJournal36497-524Retrievedfromhttponlinelibrarywileycomdoi101111j1541-0072200800281xabstractSchwartzMampWilsonE(nd)WhocanaffordtoliveinahomeAlookatdatafromthe2006americancommunitysurveyRetrievedfromhttpswwwcensusgovhousingcensuspublicationswho-can-affordpdfShearerJCAbelsonAKouyateBLavisJNampWaltG(2016)WhydopolicieschangeInstitutionsinterestsideasandnetworksinthreecasesofpolicyreformHealthPolicyandPlanning31(9)1200-1211Retrievedfromhttpsdoiorg101093heapolczw052ThelessonsofadministrativedataHigh-profilepolicy-relevantresearchpoweredbyadministrativedata(nd)Retrievedfromhttpswwwpovertyactionlaborgsitesdefaultfilesresources20161114-The-Lessons-of-Administrative-DatapdfToolsandotherneedsforqualitativeandquantitativemethods(2015February)AgencyforHealthcareResearchandQualityRetrievedfromhttpwwwahrqgovprofessionalsquality-patient-safetytalkingqualityassesstoolshtml
WeaverRCampBagchi-SenS(2013)SpatialanalysisofurbandeclineThegeographyofblightAppliedGeography4061-70
References
65
UniversityPartners
Name Title Affiliation Area(s)ofexpertise
Contactinformation
Examplesofworkingwithnon-academicdecisionmakers
WendyPBarreno
AssociateDirector
WilsonSheehanLabforEconomicOpportunitiesUniversityofNotreDame
LEOsfocusareasincludehealtheducationhousingchildwelfareeconomicopportunitycriminaljustice
wbarrenonduduorleondedu
MelissaDavey-RothwellPhD
AssociateScientist
JohnsHopkinsBloombergSchoolofPublicHealth
Health mdavey1jhuedu410-614-5854
BaltimoreCityHealthDepartmentLeadersMarylandDepartmentofHealthAddictionRecoveryServicesinDCExecutiveDirectorsofcommunityorganizationsacrosstheUSthatfocusonHIVSTIsubstanceabuseandimplementationscience
ElizabethStuart
AssociateDeanforEducationandProfessorDepartmentsofMentalHealthBiostatisticsandHealthPolicyand
JohnsHopkinsBloombergSchoolofPublicHealth
Educationsocialpolicystatisticalmethods
estuartjhuedu
Resources
66
Management
Design-ThinkingResources
EngagementGuideforCivicTech
Journeymap
PerformanceManagementResources
DenverPeakAcademy
Proxymeasuresguide
Talkingaboutperformancewithoutcomplicatingit(video)
SettingPerformanceTargetsguide
Datavisualizationguide
Performancemanagementgettingstartedguide
Exercises
Fishboneexercise
5Whys
AdministrativeData
TheLessonsofAdministrativeData(JPAL)
EvaluationResources
Quantitativeandqualitativemethods(USDepartmentofHealthandHumanServicesAgencyforHealthcareResearchandQuality)
KeyStepsinOutcomeManagement(UrbanInstitute)
RCT-YES
Sageresearchmethods
BestPracticesforMixedMethodsResearchintheHealthSciences
ACCORDChecklist
Pilotandimplementationplanning
Benchmarkingexamplesandproxymeasures
Seattleopendataplaybook
Resources
67
Datasciencescopingguide
Resources
68
AboutGovExGovExhelpsgovernmentsusedataandevidencetoimproveresidentsrsquolivesWesupportgovernmentsinbuildingtheirinternalcapacitytooperatesustainableanalyticspracticeswhichimprovecommunityoutcomesinfieldsfrompublicsafetytoeconomicdevelopmentGovExhasaproventrackrecordofworkinginconcertwithpartnerstoprovidetechnicalassistanceandtrainingatscale
SinceourlaunchinApril2015GovExhas
Trainedover2000governmentemployeesinpersonandonlineintopicsincludingdatamanagementperformancemanagementopendatacommunityengagementandanalytics
Providedtechnicalassistancetomorethan100localgovernmentsacrosstheglobeimprovingtheiruseofdatatoachieveefficienciesinservicedeliveryandbetteroutcomesforresidents
Assessedthecapacityofmorethan130governmentalorganizationstodeterminehowleadersstaffandexternalstakeholdersmanageandusedata
Localgovernmentsareincreasinglyonthefrontlinesofaddressinglargescaleproblemswithglobalimplicationssuchasclimatechangepublicsafetyandeconomicinequality
Whileevidence-basedinterventionstohelpsolvethoseproblemsareavailablelocalgovernmentstaffhavelimitedtimeandcapacitytofindandimplementresearch-basedsolutionsAsaresultcitiescontinuetoworkthewaytheyalwayshave--makingincrementalimprovementstoprogramsbutnotalwayscreatingchangeatasystem-level
ThroughresourcedevelopmentandcollaborationwithJHUfacultyGovExissupportingcityleadersinusingdataandevidencetomakedecisionsandcreatinglastingrelationshipswithresearchers
TheRoadmapwasdevelopedbyGovExaspartofitspartnershipwithBloombergPhilanthropiesWhatWorksCitiesinitiative
AboutGovEx
69
- Title page
- Introduction and purpose of the Roadmap
- Using the Roadmap
-
- Gather information about the problem
- Strategically engage key partners
- Review the evidence and collect details
- Decide on a policy to implement based on local factors
- Establish a framework to monitor progress
- Assess data accessibility quality and security
- Design implementation and evaluation
- Share your story and success
-
- Working with a range of evidence
- Thank you
- References
- Resources
- About GovEx
-
SuttonTrustEducationalEndowmentFoundation-educationalachievement
CollegeofPolicingWhatWorksCentreforCrimeReduction-crimereduction
EarlyInterventionFoundation-earlyintervention
WhatWorksCentreforLocalEconomicGrowth-localeconomicgrowth
CentreforAgeingBetter-qualityoflifeforolderpeople
WhatWorksCentreforWellbeing-wellbeing
WilsonSheehanLabforEconomicOpportunities(LEO)UniversityofNotreDameProvideslinkstotheirevaluationsofinterventionsincriminaljusticeself-sufficiencyeducationhealthandhousingandhomelessness
ClearinghouseforLaborEvaluationandResearchUSDepartmentofLaborProvidesresearchonworkforceandlabor-relatedpoliciesandprograms
J-PALEvaluationsAbdulLatifJameelPovertyActionLabStudiesonpoverty-reducinginterventionsfromaroundtheworld
CampbellSystematicReviewsCampbellCollaborationSystematicreviewofpolicyinterventionsbyscholarlycoordinatinggroupsinsevenissueareascrimeandjusticeeducationinternationaldevelopmentmethodssocialwelfareknowledgetranslationandimplementationanddisability
Benefit-CostResultsWashingtonStateInstituteforPublicPolicyAWashingtonState-charteredprogramtoestablishthelikelihoodofrelativecostsandbenefitsforprogramsacrossthefollowingareasjuvenilejusticeadultcriminaljusticechildwelfarepre-Kto12educationchildrensmentalhealthhealthcaresubstanceusedisordersadultmentalhealthpublichealthandPreventionworkforcedevelopmentandhighereducation
EffectiveInterventionsCentersforDiseaseControlandPreventionRepositoryforevidence-informedHIVpreventioncareinterventions(withstepsonhowtoimplement)
Journalsandindexes
OnceyouhavereviewedrelevantinformationaccessiblethroughclearinghousesitisworthtakingalookatindividualstudiespublishedasscholarlyarticlesYourreviewerswilldothisbysearchingarticleindexesandacademicjournalsUnfortunatelyanumberofusefulacademicjournalsandindexingplatformsmaynotbegenerallyaccessibletoyourreviewersifyoudonothaveasubscriptiontothemToaddressthisconsiderpartneringwithalocal
Reviewtheevidenceandcollectdetails
25
collegeoruniversitytoprovideyouwithalistofarticlesfromthefollowingsources(Thisisalsoanexcellentopportunitytoseekresearchsupportfromanyacademicstakeholdersyouidentifiedpreviously)
Indexes
GoogleScholarincludesmostpeer-reviewedonlineacademicjournalsandbooksconferencepapersthesesanddissertationspreprintsabstractstechnicalreportsandotherscholarlyliteratureincludingcourtopinionsandpatentsItallowsyoutospecifyapublicationrangeinyoursearchandprovideslinkstolikelysourcesforalllistedarticles
PolicyFileIndexisauniqueresourceforUSpublicpolicyresearchinthatitgrantsusersaccesstotimelyupdatedinformationfromover350publicpolicythinktanksnongovernmentalorganizationsresearchinstitutesuniversitycentersadvocacygroupsandotherentities
PolicyArchiveworkstocollectanddisseminatesummariesandfulltextsvideosreportsbriefsandmultimediamaterialofthinktankuniversitygovernmentandfoundation-fundedpolicyresearch
Anothersourceforhigh-qualityresearchisthereportsoftheCongressionalResearchServiceYoucanfindtheirreportsaggregatedhere
Specificjournals
Oftenthemostusefulacademicjournalsforunearthingpolicyorprogramreviewswillbeineachspecificsubjectarea--thatisineducationjournalsforeducationpoliciesYoucanuseageneralindexlikeGoogleScholartobegintolearnwhichjournalsaremostoftenusefulforyoursearchHoweveryoucanalsosearchthefollowingtwogeneralistpolicypublicationsforissuesandarticlesrelatedtoyourtopic
PublicAdministrationReviewJournalofPolicyAnalysisandManagement
Newssourcessubjectmatterexpertsandinterviews
AlthoughitisvaluabletoknowwhathasbeenexploredthroughascholarlyresearchframeworkitisalsousefultolearnwhatkindofinterventionsarebeingtriedinthefieldrightnowandhowpeoplefeelaboutthesetrialsSearchsubscriptionnewsaggregatorslikeLexisNexisorfreenewsaggregatorslikeGoogleNewsorYahooNewsforyourissueareaplusthewordpolicyorprogramtoseewhatkindsofinterventionsarebeingcoveredinthenews
Reviewtheevidenceandcollectdetails
26
DonotforgettocontacttheadvisorygroupmemberswhowerechosenfortheirsubjectmatterexpertiseInadditiontoprovidinggoodmaterialthroughinterviewsthemselvestheymayhaveideasaboutothersubjectmatterexpertsyoushouldcontacttoaskaboutpotentialinterventions
Peercitiesandnetworks
InadditiontoreadingabouttheworkofothercitiesitisidealifyouareabletospeakwithpeoplewhoareinvolvedwithcurrentimplementationsofinterestinginterventionsInyourcityyoulikelyhaverelationshipswithyourneighborsbutitcanbehelpfultolookoutsideofyourregionforsolutionsaswellTofindcounterpartsinothercitieswhoareworkingonaprogramyouwouldliketolearnmoreaboutyoucancontactstaffinintermediaryorganizations(likeGovExoryourprofessionalassociations)whomightbeinapositiontoputyouintouchimmediatelyorwhomightseeanothergoodrouteforsettingupaconversationHereisalistofnetworksthatmaybehelpful
InternationalCityCountyManagementAssociation(ICMA)NationalAssociationofCounties(NACO)NationalLeagueofCitiesUSConferenceofMayorsWhatWorksCities
Collectthedetails
Onceyourreviewershavefoundevidenceonpossiblepoliciesorprogramstoimplementtheyshouldcapturekeydetailsthatwillhelpdecisionmakersunderstandthepotentialeffectivenessandapplicabilityofeachpotentialintervention
Foreachsourcethathasprovidedinformationyouwanttoconveyusethetemplatebelowtocaptureasmanyofthefollowingdetailsaspossible(Youdonotneedtocompletethisforsourcesthatprovideduplicativeinformationorareotherwisenotthatuseful)
Programorpolicyname (egldquoHousingFirstrdquoorldquoblightedpropertiesregistryrdquo)
Sourcetype (egjournalarticleinterviewsubjectmatterexpertstudyassociationcommunityparticipationprocess)
Sourcedetails (egarticletitleandpublicationnameandtitleofinterviewee)
Evaluationmethodology
(egrandomizedevaluationcomparativeevaluationoftwodifferentgroupsbeforeafteranalysis)
Evidencelevel (egLevel12345)
Programorpolicy
Reviewtheevidenceandcollectdetails
27
description
Programorpolicygoal (1sentencecapturingtheprincipalaimoftheintervention)
Existingimplementations
(WherehastheinterventionbeentriedListasmanysitesasyouareawareof)
Characteristicsofimplementationsites
(ForeachimplementationsiteofferbasicgeographicIDmdashcityandstateThenyouwillwanttocollectthedetailswhicharerelevanttoyourdecisionmakersrsquoconsiderationsaboutwhatmakesalocationldquosimilartordquoorldquodifferentfromrdquoyoursituationYouwillatleastwanttoincludeforeachimplementationsitePopulationnumberPopulationdensity(ruralurbansuburban)Socio-economiccharacteristicsBeforestartingaskyourdecisionmakersaboutanyparticularcharacteristicsofprogramimplementationsitestheywouldwanttoknowbeforeseekingtoreplicate)
Characteristicsofimplementationpartners
(Anyavailabledetailsaboutorganizationsinstitutionsorpeoplewhowereinvolvedwithimplementingtheintervention-egindependentchildcarecenterpublicelementaryschoolcitypublichealthnurses)
Characteristicsoftargetpopulationeligibleprogramparticipants
(Anyavailabledetailsaboutpeoplewhoarethebeneficiariesoftheprogramorotherwiseincludedoraffectedbytheintervention-egnumberofprogramparticipantscategoriesofpropertyownersaffected)
Policyorprogramduration
(HowlonghastheinterventionbeeninplaceineachimplementationsiteHasitendedandifsowhy)
Whatbenefitscamefromeachimplementation
(Anyobservedpositiveeffectsoftheinterventionincludingeffectsize)
Statisticalestimatesofinterventioneffectiveness
(Anyavailablestatisticalestimatesoftheinterventionrsquoseffects)
Whatdrawbackswereobserved
(Anyavailabledetailsaboutunintendedorunavoidablenegativeeffectsoftheintervention)
Programorpolicycosts
(Anyavailabledetailsaboutprogramcostsperimplementation)
Reviewtheevidenceandcollectdetails
28
Reviewtheevidenceandcollectdetails
29
DecideonapolicytoimplementbasedonlocalfactorsTogetthecriticalbuy-infromleadershiptoimplementanewpolicyorprogramtheprojectleadshouldscheduleameetingwithdecisionmakerstopresenttheevidencesurroundingthepolicyoptionscapturedinthetemplateDependingonyourinsightintoyourcityleadershipswillingnesstomakechangeyoumaywishtosharetheresearchyouhaveconductedontherootcausesoftheproblemandsomeofyouranalysisontheimpactofcurrenteffortstodate
Considertheapplicabilityofanapproach
AsdiscussedinthepreviouschapterpolicymakersoftenlookforevidenceofeffectiveprogramsinothercitiesbutoftenthequestionarisesofhowgeneralizableaprogramisfromonecitytoanotherTheAbdulLatifJameelPovertyActionLab(J-PAL)suggeststhatinreviewingevidencefromanotherlocationitishelpfultoidentifythespecificmechanismthatwasused(deliveryofprogramorserviceinformationsharingetc)identifythelocalconditionsthatwereinplaceobservethebehaviorchangethatresultedfromthemechanismandfinallytounderstandlocalcontextandconditions(legalorcostconstraintsculturalnormsetc)Thissetofpresentationelementscanbedrawnfromthelistofdetailscollectedintheresearchtemplate
EducationpolicymakersinChicagoILusedfindingsfromprogramsinKenyaandIndiainwhichchildrenwhohadfallenbehindtheirpeersweregiveneducationalsupporttargetedtotheirleveloflearningThemechanismusedwastargetedteachingandthereweresimilarconditionspresentinallthreecountriesAlthoughsomestudentshadfallenbehindinthecurriculumteacherswereincentivizedtoteachgrade-levelmaterialsConsiderationsofhowthelocalcontextinChicagomightaffectimplementationofatargetedteachingprogramincludedunderstandinghowtheneighborhoodsthatthestudentswerefrommayhaveaffectedtheireducationaloutcomesandthehighcostofteachinginsmallgroupsInChicagoMatchEducationprovidedtwo-on-onetutoringwhichmatchedstudentswithwell-educatedindividualswhohavenothadformalteachertrainingforalower-costmodelTheresultswerepositiveStudentswhoweretutoredperformedbetterinmathandweremorelikelytopasshighschoolclasses
ThegeneralizabilityframeworkproposedbyJ-PALisbelowUsethisasatemplateasyouconsiderpotentialpolicysolutions
Decideonapolicytoimplementbasedonlocalfactors
30
SourceTheGeneralizabilityPuzzle
OnceyouhavenarrowedthepolicyoptionsdiscussthequestionsbelowforeachoptionsyouareconsideringThisstepiscriticalasunderstandinglocalcontextissoimportantinsuccessfulpolicyimplementation
CommunityContextQuestionnaire
Localcontext
DoesthisoptionmakesenseinourcommunityIsthisaonesizefitsallapproachIsanyparticulargroupabsentfromthisapproachIfsoisthatok
Youranswers
Effectiveness
WhatwouldweexpecttoobserveifthepilotisaseffectivehereasinCityX
Youranswers
Timing
DoesthetimingmakesenseIsthistherighttimetoimplementthispolicychange
Decideonapolicytoimplementbasedonlocalfactors
31
WherearethepotentialroadblocksthatcouldaffectthescheduleIsthereanywaytomitigatethisHowlongwouldittaketoseeanimpact
Youranswers
Implementation
DowehavethestaffcapacitytoimplementthisArethereadditionalpartnersthatyoucanbringtothetabletosupportimplementationDoesthismakesensegivenourformofgovernment
Youranswers
Authority
DoesourcityhavetheauthoritytoenactthisIfsowhatistheappropriatemechanismAretherelegalbarriersArethereanyexistingpreemptionissuesfromthestateorfederalgovernmentCanthosebeaddressedthroughlegislationadvocacyIstherepoliticalsupportoroppositiontothis
Youranswers
Cost
HowmuchwillthiscostannuallyHowwillwepayforitCanthepolicybeimplementedwithexistingfunding
IfsoisthisfundingreliableHowwouldthispolicyoptionaffectourcitysbudgetintheshortandlongtermWhatotherfundingoptionscanweconsiderIsthereanalternativethatwouldcostlessandgiveussimilarresultsWillanyconstituencybeadverselyaffectediffundingforprogramsthatimpactthemisshiftedtoanotherprogrampolicyorinitiative
Youranswers
Decideonapolicytoimplementbasedonlocalfactors
32
EstablishaframeworktomonitorprogressNowthatyouhavereviewedtheevidenceconsideredlocalcontextandchosenyourpolicyitistimetodeterminehowyourcitywillmonitorprogressofthisnewapproachCitystaffandleadersneedtoestablishtheinfrastructureforongoingcontinuousdecisionmakingbasedonwhetherthepolicychangeisleadingtothedesiredoutcomesforresidentsandwhetherthechangeismeaningfulandrationaltofrontlineworkersSettingarbitrarygoalsatthelocallevelthatconflictwithgoalsatotherlevelsisconfusingtotheindividualsresponsibleforachievingthegoalsGoalsthataresettoohightoachievecanbedemoralizingtostaff
ThissectionincludesstepstosetupavenueinwhichCitystaffandleadershiproutinelyvisitkeyindicatorstodeterminewhatisorisnotworkingandtomakedecisionsinreal-timeFormorereadingonperformancemanagementpleasevisittheGovExPerformanceManagementGettingStartedGuide
InperformancemanagementtermssuchasindicatormeasureandmetricmaybeusedinterchangeablyInthisguidewewilluseindicatorInyourworkthemostimportantthingistobeconsistentinwhichevertermyouuseThisvideomaybehelpfulinunderstandingthisconceptandcommunicatingittoothers
Establishthefundamentalsandgatheryourteam
FirstdeterminewhetherthereisanestablishedvenueorregularmeetingscheduleduringwhichyoucandiscussprogressIfyourcityhasanexistingperformancemanagementstructureonarelatedtopicaddingadiscussionoftheprogressofpolicychangetoyouragendacanbeagreatideaIfthiswillbeastandalonemeetingdecidehowfrequentlytheprojectteamanddecisionmakerswillmeetandputthosemeetingsonthecalendarfortheentirelengthoftheproject
DeterminewhowillstaffandattendroutinemeetingsonCityprogressThesemeetingsmayincludetheMayorandorCityManagerdepartmentheadstaffwhoareimplementingthepolicychangeandanalysts
Identifyindicators
ItiscriticalthatindicatorsbeanimportantcityandcommunitypriorityaswellassomethingtheCityplaysamajorroleinaddressingTheremaybeinstancesinwhichtheCitywillneedtoreachouttoorcollaboratewithexternalpartnerstogetamorecompletepictureofprogressontheoverallpolicychange
Establishaframeworktomonitorprogress
33
Anoutcomeindicatordescribesthedesiredeffectofthetargetedpopulation
ExampleBellevuesfamilyfriendlyneighborhoodsindicatormeasuresthepercentofresidentswhobelievetheyliveinaneighborhoodthatsupportsfamiliesparticularlythosewithchildren
WithyourkeypartnersrevisityourproblemstatementcurrentstateandfuturestateexerciseandbegintoidentifytheoutcomeindicatorsthatwillillustratesuccessFornowchooseasingletop-levelkeyperformanceindicator(KPI)ItiscriticalthattheKPI-thepieceyouwillmeasure-beapriorityforthecommunityandsomethingtheCityplaysamajorroleinaddressing
Outputindicatorsmeasuretheprocessofprogramorservicedelivery
ExampleSanDiegoMayorKevinFaulconersgoaltorepair1000milesofstreetby2020
ReviewingtheadministrativedatasetinventorybegintoparedowntherelevantprogramsanddatasetsthatsupportyourKPIWhatarethemeasuresofeffectivenessforeachprogramTheseareyouroutputindicators
Thediagrambelowillustratestherelationshipbetweenoutputandoutcomeindicators
ThinkaboutimplementationindicatorsasmeasuringstepsintheprogramsrolloutthatarenecessarytoobtaindesiredchangesImplementationoutcomescanfocusontheinterventionstaffdeliveringtheinterventionanorganizationorsystemandcanaddressfeasibilityadaptabilitysustainabilityandcostTwokeyquestionstoconsiderwhendevelopingimplementationindicatorsare
1 Istheinterventionacceptableandappropriatetothecommunityand
2 IsitfeasibletoimplementinterventionXatthistimewiththeseresources
Establishaframeworktomonitorprogress
34
Amoreextensivelistwithdefinitionsofpotentialimplementationoutcomesislinkedhere
UsethetablebelowtochoosethedesiredimplementationoutcomesforyourpilotandtoplanyourmeasurementtechniqueThisshouldincludelevelofanalysis(individualproviderorconsumerorganizationorsetting)yourmeasurementtechnique(surveysobservationinterviewfocusgroupsadministrativedataanalysis)andthephasewhenyouwillcollectthisinformation(earlymidlate)Theremaybemorethanonelevelofanalysismeasurementtechniqueandphaseofdatacollection
Implementationoutcome
Definitionofoutcome
Levelofanalysisandmeasurement
Yourmeasurementtechnique
Phaseofpilotwhenyouwillcollectdata
Usethespacebelowtocaptureyourindicatorwishlist
Indicatorwishlist
Outcomes_____________________________________________________________________________
Outputs________________________________________________________________________________
Implementationoutcomes______________________________________________________________
Setapublichigh-levelgoal
Thehigh-levelgoaliscriticalinestablishingdirectiontogalvanizethepeoplewhowillbedoingtheworkWhendiscussingthegoalusetheMayorsbullypulpitLookforkeymilestonesinCitygovernmenttosharethegoalstatementsuchasthebeginningofaMayoraladministrationaStateoftheCityspeechorotherpublicremarks
GoalsshouldbeldquoSMARTrdquo
Specificaspossible(innamingthepopulationwhowillbebetterofftheactionthecitywilltake)
Measurable
Achievable
Establishaframeworktomonitorprogress
35
Results-focused
Time-bound
ThegoalstatementshouldincludeyouroutcomeindicatorlikeChattanoogasgoalldquoWewillendchronicveteranhomelessnessbyDecember2016rdquoAdiagramtohepyoubuildaSMARTgoalaroundyouroutcomeindicatorisbelow
Component Description Yourturn
Specific WhowillbebetteroffWhatactionthecitywilltake
Measurable Howwillyoumeasuresuccess
Achievable Isthegoalattainable
Results-focused Whatistheexpectedresult
Time-bound When
Setrealisticambitioustargetsandtimelinesforeachindicator
ToestablisharealisticandambitioustargetforyourgoalstatementusetheadministrativedatacollectedforyourchosenKPItoreviewactualCityperformanceinthemostrecentyearcomparedwiththethreeorfourmostrecentyearsworthofdataBegintocalculatesummarystatistics(egtotalminimumaveragemedianmaximum)toidentifytrendsCreatecumulativeandmonthlychartstoidentifytrendsandanomaliesBasedonyouranalysisofhistoricaldataignoretheanomaliesandassumetheproductivitytrendswillfollowthesameoverallpatternfrommonth-to-monthFinallyapplyanincreasedproductivityfactortotheperformanceeachmonthtoforecastfutureprogressBasedonthisanalysischooseatargetthatinspiresimprovementwhileaccountingforreality
Manycitiesthataremorematureintheirperformancemanagementpracticesrecognizetheimportanceoftakingacommunity-levelapproachtoimprovingoutcomesforresidentsThesecitiesarecreatinggoaltaxonomiesthatincludemeasuresrelatedtopublicsafetyhousingandevenrecreationCitiesdriveprogressonthesegoalsbyimplementingstrategiesthatengagemultipledepartmentsandcollaboratewithserviceprovidersinthecommunity
EventuallyyouwilldevelopgoalstatementsfortheoutputsinordertocapturetheefforttoachievetheoutcomecontainedinyourgoalstatementYoursetofindicatorsshouldbecomposedofshort-andinterim-termsupportivemeasureswithaccuratetargetsandrelevanttimelinesAnexamplegoaltaxonomyfromtheCityofLasVegassResultsVegasprogramisbelowUsethistaxonomyasamodelforyourownsuiteofindicatorsandgoalstatements
Establishaframeworktomonitorprogress
36
Yourturn
Thetemplatebelowismeanttohelpyoucraftagoalstatementforeachsupportingmeasure
Goalstatement
Departments
Outcome
Activities
KPI
Target
Timeframe
Conductanddiscussrelevantanalysis
ThrougharoutineseriesofmeetingscontinuallymonitorprogressofKPIssupportingindicatorsandanyotherdatacollectedsuchasqualitativedatatoidentifytrendsthatwillindicatewhetherthechosenpolicyishavingthedesiredimpactThesedatapointsshoulddrivethediscussionduringthesemeetingspointingtoopportunitiesforfurtherinquiryinto
Establishaframeworktomonitorprogress
37
whatisdrivingtheobservedtrendsAnalystsshouldbeencouragedandtrainedtoaskquestionsofthedepartmentsproducingthedataandconductsitevisitswhenpossibletodeepentheirunderstandingofthedataPresentationoftheanalysisshouldfocusononeideaatatimetokeeptheconversationfocusedonkeypointsandspecificopportunitiesforimprovementYoumayconsiderdevelopingapublicfacingdashboardtosharethisprogressmorebroadly
AgeneralcomparisonofthestatusquodatawithwhatwasobservedduringthepilotcanprovidedecisionmakerswithaquickunderstandingofthedifferencesbetweentheoldandnewapproachesalthoughthiswillnotshowcausalityOftenthesecomparisonswillincludeoperationalmetricssuchasdaysordollarssavedbutrememberthattheimpactonpeoplewillbemostimportantIfyouareworkingwithanevaluationpartnertheywillsupportthisstep
Usethespacebelowtosummarizetheresultsofyouranalysis
MeasureTargetvalue(seesectiononsettingtargetsabove)
Pilotvalue(whatdidyouobserveduringthepilot)
Historicalaverage(ifavailable)
Results(WhoisbetteroffIsanyonenegativelyimpacted1-2sentencesreassessingyourgoalstatement)
DocumentyourprocessandprogressincludingfeedbackfromtheparticipantsimplementationpartnersandtheprojectteamDidanythingunexpectedoccurduringthepilotArethereinternalandexternalfactorsthatmayaffectyouroutcomesThisiswheretheimplementationoutcomescanbecanhelpfulIftheinterventiondidnotgoasplannedconsiderwhetheritisbecausetheinterventionwasnotacceptableorfeasible
UsethetemplatebelowcreatedbyDenversPeakAcademytocapturethisinformationIfthe306090daytimelinedoesnotmakesenseforyourcityfeelfreetocustomizeit
CurrentState
FutureState
30Day
60Day
90day
Whatwentwell(+)
Whatdidnotgowellshouldbechanged(-)
IfaquestionordiscussionpointarisesthatcannotbeansweredinthemeetinganassignedstaffmembershouldcapturethequestionduringthemeetingAfterwardthefollowupitemsshouldbesenttorelevantpartnersandthenextmeetingshouldbeginwithadiscussionof
Establishaframeworktomonitorprogress
38
thoseitems
Takeaction
DiscussionsaboutwhetheryourindicatorsaremovingintherightdirectionandwhatmightbedrivingtrendsmayleadtoadiscoverythattherearenotenoughresourcesallocatedtothispolicyorissueTheremaybelimitedfundingforstaffprogrammingortechnologythatisdrivingthedatainthewrongdirectionIfCityleadershipisintheroomthisisagreattimetomakethecaseforadditionalfundingduringthebudgetseasonBycollectinganalyzinganddiscussingdataonthispolicyroutinelyyouarebuildingevidencetomakethecasefortherightleveloffundingtoreachtheestablishedtargets
Establishaframeworktomonitorprogress
39
Assessdataaccessibilityqualityandsecurity
Determineaccessibilityandqualityofdata
OnceyouhavecompiledyourwishlistofindicatorsitistimetodeterminetheaccessibilityandqualityofthedataTopreventcomplicationyoumaywishtoprioritizerelevantindicatorsforwhichyouarealreadycollectingdataAvoidthetemptationofadoptingatechnologysolutiontocollectnewdataatthispointsuchasaphotosurveytoolorotherappFornowyouaretryingtodeterminewhichdatasetsarerelevantaccessibleandhighqualityBesuretoworkwithyourkeyinternalpartnerswhileyouareidentifyingyourmeasuresDepartmentstaffwhoworkonprogramsarefamiliarwiththeindicatorsthatwillaccuratelysupporttheoutcomesanddatabaseadministratorswillbevaluablewhendiscussingaccessibilityandqualityofthedata
Abrieflistofconsiderationsfortheaccessibilityofyourdataisbelow
ArewecurrentlycollectingthisdatasetorwillwebecollectingitforthefirsttimeHowfarbackdoestherecordgo(timehorizon)WouldweneednewtechnologytocollectoranalyzethedataCanthedatabecollectedmanuallyorcanitbeautomatedHowfrequentlyisthedataproduced(continuouslyhourlydailyweeklymonthlyquarterlyyearlyetc)IsthedataavailablefortheentireCitypopulationorasubsetofthepopulationHowgranularisthedata(individuallevelCensusblockneighborhoodcitywideetc)DoesthedatacontainsensitiveinformationIsthereaproxymeasurethatwecanusefornowIsthedatamachinereadable
AsnotedbytheWilsonSheehanLabforEconomicOpportunities(LEO)attheUniversityofNotreDameleveragingadministrativedatacansignificantlyreducethecostsandtimelinesofrigorousevaluationCitiescanestablishaninternaldatasharingagreementacrossdepartmentsincludinglanguagethatenablesadministrativedatatobeusedforresearchpurposesCitiescanalsoidentifyexternaladministrativedata-holderssuchasacountyschooldistrictpolicedepartmenthospitaloranotherstatethatmayhaveoutcomedataimportantformeasuringprogramimpactCitiescanleverageexistingrelationshipsandorworkwithresearchinstitutionstodevelopstreamlineddatasharingprocessesandagreementswiththeseexternalentities
Assessdataaccessibilityqualityandsecurity
40
Evaluatethequalityofyourdata
QualityisasimportantasaccessibilityToevaluatedataqualityyoumaywishtocreatearubricliketheonebelow
Value ValueDescription
3 ExcellentDataQualityThedataiscompleteaccurateandupdatedregularly
2 AdequateDataQualityThedataissomewhatcompleteaccurateandupdatedfairlyregularly
1 PoorDataQualityThedataisnotcompleteaccurateorupdatedregularly
ThisisagreattimetobuildonthedatainventorythatyoustartedinthegatherinformationpageSuggestedadditionalfieldstoaddtoyourdatainventorytemplatearebelow
Program DatasetTitle
DatasetDescription DataSource
Departmentorexternalpartner
DoestheCityalreadycollectthisdata
Whatisthenameoftheprogram
Human-readablenameofthedataset
Shouldbeunderstandablebynon-technicalusers
Whatistheoriginalsystemapplicationfilethathousesit
Departmentorexternalpartnerthatmaintainsthedata
Isthisdataalreadycollected
Updatefrequency-ThefileformatforthedatasetTypicallycsvxlsxshapefiletxtetc
Format-Howoftenthedatasetisupdated
Timerange-Theperiodoftimecoveredinthisparticulardataset
Datause-Howisthisdatasetcommonlyused
Datausers-Whotypicallyusesthisdataset
Dataownerstitle-Titleofthedataowner
Dataownersposition-DataOwnersjobtitle
Dataownersemailaddress-DataOwnersemailaddressWhomanagesthedataandorisresponsibleforgrantingpermissiontoaccessthedataWhounderstandswhatthedatasetincludesandcananswerquestionsaboutit
Dataqualityconcerns-Describetheconcernsyoumayhaveaboutthequalityofthedataset
Assessdataaccessibilityqualityandsecurity
41
Division-DivisionwherethedataismaintainedIfmorethanonedivisionisresponsiblelisttheprimaryownerofthedataset
Sensitivedatacomments-DescribetheissuesthatwouldpreventthecityfromsharingthedatapubliclyCommonsensitivityconcernsincludeprivacyviolationssecurityissuesorhighcostorstaffdemandstopostthedata
Publishingstatus-Thistheplacetoputanycommentsregardingthecurrentstatusofthedataset
Publishedlink-Thisisthelinktothepublisheddatasetontheopendataportalorcitywebsite
Reviewdatasecurityandprivacyprotectionprotocols
OftenapolicychangealsoinvolvesachangeinthedatathatisbeingcollectedhowthatdataisbeinghandledwhohasaccesstoitandahostofotheradjustmentsAsyoutakestockofthedataassetsthatareavailableforyourinitiativeitisvitaltoalsothinkthroughwhatwillbedonetoprotectthatdataandtheprivacyofthepeopleitrepresentsAbigpartofgooddatasecurityinvolvesstrongdatagovernancepracticesthatdefineandidentifysensitivedataandconsciouslyworktobalancethevalueandriskofstoringsuchsensitivedataOnewayofevaluatingyourdatacomesfromcaselawrelatedtotheUSFreedomofInformationActwhichwehavetranslatedtothechecklistbelow
1 Woulddisclosureresultinasubstantialinvasionofprivacy2 Haveweconsideredtheextentorvalueofthepublicinterestandthepurposeorobject
oftheindividualsseekingdisclosure3 Istheinformationavailablefromothersources4 Wastheinformationgivenwithanexpectationofconfidentiality5 Isitpossibletomouldreliefsoastolimittheinvasionofindividualprivacy
TheseriskscouldbemitigatedusingavarietyofstrategiesfromaggregatingdatatoobfuscatingindividualstoredactingfieldsthatmayincludePersonallyIdentifiableInformation(PII)orre-evaluatingyourdatacollectionmethodstofindalternativewaysofgettingtheinformationyouneedThesestepsshouldbecodifiedandmanagedbyadatagovernancecommitteeifpossibleForthesensitivedatathatisdeemedessentialcreateaplanaroundpotentialdatabreachesinordertoquicklyandconfidentlyrespondtoemergencysituations
ProactivelypreparingforthesesituationsisoftenthebeststrategyfordatasecuritybutdonotforgetthetechnicalsideInvolveyourITadministratorsinconversationsaboutdatastorageandaccessensuringthattheywillbeprotectedbyenterprise-gradecybersecuritysystems
Assessdataaccessibilityqualityandsecurity
42
Assessdataaccessibilityqualityandsecurity
43
Designimplementationandevaluation
Designtheimplementation
NowthatyouhaveyouraccountabilityplanandkeyindicatorsitistimetogetspecificaboutyourplanforimplementationThetemplatebelowwillhelpyouthinkthroughthespecificmechanismsofimplementingpolicychange
UnderstandingorganizationsandsystemsiscriticaltounderstandingandmakingdecisionsaboutwhereandhowinterventionsshouldbeappliedApolicyimplementedinasinglesettingmayshowpromisingresultsbutimplementingmultiplepoliciesandprogramsatthesametimewithdifferentactorsresponsibleforeachaspectcanhelpensuresustainablebehaviorchange
InSomervilleMAShapeUpSomerville(SUS)isacitywidepublichealthprogramthatisimplementedacrossmultiplelevelsofinfluenceandsystemsthroughinterventionsbeforeduringandafterschooltoincludechangestobuiltenvironmentsuchasbikelanesregionalmasstransportationandphysicaleducationequipmentinschoolsandgymstoincreasephysicalactivitywellnesspoliciesandschoolgardensnutritioneducationduringandafterschoolandhealthyrestaurantprogramstosupporthealthyfoodandhealthyeatinghabits
Usethetemplatebelowtoplanyourimplementation
Policyimplementationplanningtemplate
ListthespecificdesiredoutcomesInthepreviousstepyouworkedwithdepartmentstafftoestablishspecificmeasurableoutcomesandinterimindicatorsPrioritizetheindicatorsthatwilldemonstrateprogressUsethespacebelowtocapturethedesiredoutcomesofthepolicychange
ExampleShapeupSomervillebeganwiththegoalofpreventingobesityin1stthrough3rdgraders
Yourturn
____________________________________________________________________________
____________________________________________________________________________
Designimplementationandevaluation
44
SpecifythetargetpopulationAreyoudesigninganinterventionthattargetsonlycertainsitesAreyouplanningonimplementingaprogramorinitiativeforaspecifictargetaudienceClearlyspecifythepopulationyouplantotarget
IfyouplantoconductarigorousimpactevaluationasnotedbytheWilsonSheehanLabforEconomicOpportunitiesitisimportanttoestablishconcreteobjectiveeligibilitycriteriaandensurethatthesecriteriaareappliedconsistentlybyallstaffduringscreeningandintakeItisalsovitaltoidentifytheindividualswhoareeligibleforaparticularprogramorserviceevenifeveryoneeligiblecannotbeservedbyexistingresourcesThishelpscitiesidentifyanyunmetneedsandguidefutureresourceallocationwhilealsocreatingthepossibilityofarigorousimpactevaluationbyidentifyingapotentialcomparisongroupFinallywhenconductingscreeningandintakeitisimperativetocollectandmaintainbasicinformation(egnamedateofbirthgender)aboutallprogram-eligibleindividualsnotjustthosewhoparticipateintheprogram
ExampleIntheShapeUpSomervilleexampletheprogrambeganbytargeting1stthrough3rdgraders
Yourturn
____________________________________________________________________________
____________________________________________________________________________
DefinethelocationWherewillthepolicychangetakeplaceWillimplementationtakeplaceatonesiteormultiplesites
ExampleShapeUpSomervilleinterventionsweredeliveredinschools
Yourturn
____________________________________________________________________________
____________________________________________________________________________
ClarifyimplementationpartnersandarticulatetheirrolesWhatarethekeyorganizationsinstitutionsorpeopleresponsibleforimplementationTheprojectteamshouldbeasclearandspecificaspossibleinprovidingadescriptionoftheconcreteactivitiesthatimplementationpartnersaremeanttocarryoutPracticeprofilesmaybeahelpfultooltohelpdefinewhatisrequiredoftheactorswhoareresponsibleforimplementingtheintervention
Designimplementationandevaluation
45
ExampleInSomervillemayoralleadershipandengagementofmultiplecityagencies(publichealthprimarycareschoolsCambridgeHealthAllianceetc)streamlinedandconnectedthevariousShapeUpSomervilleinitiativesandgrantswhichhelpedtokeepthetalentandenergyfocusedonhealthyeatingandactivelivingforallsectorsofSomervilleSUSpartnersultimatelyincludedschoolsnonprofitsCityandstateagenciesandnationalfoundations
Yourturn
____________________________________________________________________________
____________________________________________________________________________
DescribetheinterventionWhatarethestepsactivitiesorprocessesRememberthatthemechanismthespecificstepsorprocessthatleadtobehaviorchangeiskeytounderstandingtheeffectivenessofthepolicychangeandsharingtheresultswithothers
ExampleGoalsofShapeUpSomervilleweretoinfluenceeverypartofanearlyelementaryschoolchildrsquosdayndashbeforeduringandafter-schooltopromotehealthyeatingphysicalactivityandtheirweightstatusInterventionincludedfoodserviceenhancementstoimprovebreakfastandlunchandstaffprofessionaldevelopmentwalktoschoolactivitiesSUSclassroomcurriculumandprofessionaldevelopmentschoolwellnesspolicydevelopmentSUSafter-schoolcurriculumandprofessionaldevelopmentoutreachandeducationtohomethroughmaterialsforumseventsldquoSUSapprovedrdquorestaurantscommunityoutreachandcapacitybuildingthroughpolicydevelopmenttrainingsmediaplacements
Yourturn
____________________________________________________________________________
____________________________________________________________________________
ClarifytimelineandmilestonesfordeliveryandevaluationHowfrequentlywilltheservicesinterventionbedeliveredHowlongwilltheprogramrunWhatisthetimetoexpectedimpacts(eg12monthsafterprogramcompletion)
Designimplementationandevaluation
46
ExampleShapeUpSomervilleledbyTuftsUniversityFriedmanSchoolofNutritionScienceandPolicyconductedaquasi-experimental(nonrandomized)interventionstudyamong1stthrough3rdgradestudentsinSomerville(intervention)andtwocomparisoncommunitiesfor9-monthsduringoneschoolyear(fall2002ndashspring2003)ThestudyshowedpositiveassociationoftheinterventiononchangeinBMISUSreleasedreportssummarizingevaluationsprogresstodateplansforadditionalprojectsandimpactstenandfifteenyearsintotheinitiative
Yourturn
____________________________________________________________________________
____________________________________________________________________________
PlanyouranalysisTheprevioussectionaddressedestablishinganaccountabilityframeworktomonitorprogressandimplementationDeterminewhichindicatorswillbediscussthroughthatprocesswhowillsharethedataandanalysisandhowfeedbackfromthoseconversationswillbeincorporatedbytheprojectteam
ExampleTheMayorrsquosofficeimplementedadata-drivendecisionmakingapproachcalledSomerStatwhichwassynergisticwiththeapproachestakenbytheresearchandevaluationteamsdeployedforthespecificgrants
Yourturn
____________________________________________________________________________
____________________________________________________________________________
PlanforsustainabilityIfthepolicyandprogramchangeissuccessfulhowwilltheprojectteamsustainandexpandtoadditionalsitesorpopulations
ExampleAlthoughmostofthepartnershipsforgedintheearlyyearsweremaintainednewgrantsandcollaborationsexpandedattentiontoolderpublicschoolstudentsactivetransportationandtheGreenLinesubwayextensionthebuiltenvironmentfarmerrsquosmarketeffortssuchasldquoGrowninSomervillerdquowithUnionMainStreeteconomicdevelopmentandhealthdisparitiesaswellasdatacollectionmonitoringandsurveillance
Yourturn
Designimplementationandevaluation
47
____________________________________________________________________________
____________________________________________________________________________
Pilotplanningtemplate
IfyouplantoconductapilotfirstdeterminethepurposeofthepilotDoyouwanttoobservewhetherthepolicychangeresultsinthedesiredbehaviorchangeAreyouconcernedaboutfeasibilityscalabilityorcostDoyouwanttotestwhetheryourdesiredresultscanbeachievedthroughthispolicyapproachClarifythegoalofthepilotandthequestionsyouhopetoanswerThiswillhelpyouidentifythekeyindicatorsthatyouwillusetomeasurewhetherthepilotissuccessfulBeforebeginningestablishabaselineofcurrentperformanceandsetyourtargetsaccordinglybasedonwhatyouhopetoaccomplishthroughthepilot
NextestablishthepilotplanforyourprojectteamincludingmilestonesforteamcheckinstodeterminewhethertomoveaheadorchangecourseThisisseparatefromthemeetingsthatwilloccuraspartoftheaccountabilityframeworkUsethespacebelowtocreateyourtimelineanddevelopyourmeetingschedule
Task Personresponsible Startdate Checkindate Deliverable
Planyourevaluation
WhileyouaredesigningtheimplementationitisalsotimetobegintodevelopyourplanforanalysisAsnotedbytheWilsonSheehanLabforEconomicOpportunitiesdecisionsaboutprogramsetup-includingoutreachscreeningintakedatatrackingandenrollment-candeterminewhetherornotarigorousimpactevaluationwillbepossibleAsdiscussedcityleadersmayfindithelpfultoincludeevaluationpartnersinearlyconversationsaboutprogramsetupDuringpreviousstagesyoumayhaveworkedwitharesearchertoplanyourproposedpolicychangeYoushouldbeclearaboutthepurposeoftheevaluationanddevelopanevaluationplanBelowareadditionalconsiderations
Causationvstrendanalysis
IntheEstablishaframeworktomonitorprogressstagetherewasguidanceontrackingtrendsovertimeItisimportanttonoteherethatmonitoringdataisdifferentfromanalyzingacausaleffectTrendanalysiswillillustratethatchangesareoccurringandmayleadtoquestionsthatcouldbeansweredbyyourevaluationbuttrendanalysisalonedoesnotprovideevidenceaboutwhatisdrivingthosechangesYoumayseechangesinyour
Designimplementationandevaluation
48
outcomeindicatorsbasedonvariablesoutsideoftheCityscontrolForinstanceachangeinfederaltradepolicycanhaveanimpactemploymentratesinyourCityoutsideofyourCitysnewjobtrainingandplacementprogram
ToaddresscausalityyoumustcomparetheoutcomesofindividualswhoreceivedtheinterventionwiththosewhodidnotWithoutacomparisongroupofsimilarindividualswhowerenotexposedtotheinterventionitisimpossibletoknowwhichfactorsareresponsiblefortheoutcomesRandomizedControlTrials(RCTs)inwhichindividualsarerandomlyassignedtotreatmentandcontrolgroupsarethegoldstandardinprovingcausalityIfyoucandemonstratethatoutcomesimprovedforthosewhoreceivedtheinterventionanddidnotimproveatallorimprovedlessforacomparisongroupyoucanbesomewhatcertainthattheinterventionwasthecause
Examplesofothernon-causalimpactevaluationsthatarerigorousandreliableincludecertainquasi-experimentalstudiesusingcomparisongroupdesignssmartinterruptedtimeseriesanalysesthatmodeltrendsovertimeinwaysthatallowestimationofinterventioneffectswelldesignedpre-posttestanalysesneedsassessmentsimplementationstudiesetcFormoreinformationondifferenttypesofevaluationsseetheSageResearchMethodswebsite
Lookfordifferencesingroups
BuildintimetoalsoexaminethepilotdatatoseeifgroupsareexperiencingdifferentresultsReviewingperformancebasedonoutcomemeasuresforapopulationinaggregatemayhidethesubtlewaysthatsomepeoplemaybeaffecteddifferentlysoitisnecessarytocarefullyanalyzethedatafordisparateoutcomesForinstancetheMovingtoOpportunity(MTO)studythatexaminedtheimpactofhousingvouchersonfamilieswholivedinhigh-povertyneighborhoodsfoundthatchildrenwhomovedbeforetheageof13hadbettereducationandhigherincomesasadultsHoweveradultsdidnotseeimprovededucationandemploymentoutcomes
Disaggregatingdataintomorespecificcategoriesofraceandethnicitycanhelpunderstandtrendsanddisparitiesinamoregranularwaywhichcanhelpdecisionmakerstargetpoliciesandinterventionsinamoreequitableway
EarlierintheRoadmapweworkedtoidentifystakeholdersandwhomightbepositivelyandnegativelyimpactedbyachangeinpolicyUsetheresultsofthatexercisetoinvolvethecommunityininterpretingdataanalysisandtogatherqualitativefeedbackfromthecommunitytocallattentiontoissuesForinstanceanincreaseinyouthdetentionmaybeexplainedbyinefficienciesorlackofaccesstotransportation
Designimplementationandevaluation
49
Designimplementationandevaluation
50
Shareyourstoryandsuccess
Sharetheresults
Storytelling
AnintegralstepinyourstrategyissharingtheresultsofyourpilotorpolicychangewithyourcommunityThisincludesdevelopingmessagingaroundtheworkofyourorganizationandcreatingaplantodisseminateyourstoriesWhytellstoriesaboutyourwork
Informandorunderstandasubjectorsituation
Giveneededcontextofdataorbitsofinformation
Relatablestoriesarememorableforreadersviewers
ReadersorviewerswillseetheinformationasmorecompellingwhenthereisademonstratedandrelatablechangeorimpactonthemselvesortheirfriendsneighborsandcolleaguesStoriesareaboutpeopleandgovernmentisfundamentallyabouthelpingpeopleSotellthestoriesofthepeopleyouandyourcitycolleagueshelped
Keepthesetipsinmindasyoudevelopyourstorywithyourteam
WhatwereyourfindingsandwhyshouldanyonecareTellyouraudiencehowtheproblemwassolvedWhathastheimpactbeenonresidentsandcitystafftodateWhatwasthesuccessandweretheremultiplesuccessesalongthewayIfyouhavepreliminaryresultsordatashowingimprovementtosharethisisthetimetodoit
WhowasinvolvedandimpactedReviewthelistofstakeholdersHowwereeachofthemimpactedbytheprogramHumanizeyourstoryandimpactbyaddingadditionalvoicesThesevoicescanbeoneofthefollowing
Apersonorfamilywhowaspositivelyaffectedbythechangesuccess
Aninvolvedthird-partydescribingboththeproblemandthesolution(ieanorganizationnonprofitprovidersbusinesstradeassociationsportsteametc)whoispartofthecommunitythatthegovernmentworkedwithonthechangesuccess
Someonefromthegovernmentitselfwhocanspeakaboutthechangesuccess
TellmoreaboutyourcityDescribeyourcityincludingpopulationdemographicsindustriesetcWhatareitschallengesWhatisitknownforWhatmakesituniqueHowisitsimilarordifferentfromothercitiesintheUS
Shareyourstoryandsuccess
51
DescribeyourcityrsquoschallengeWriteoutthechallengeyourcityisfacing(asdefinedinyourproblemstatement)HowdoesitaffectresidentsHowdoesthisproblemaffecttheworkofcitystaffmembers
AddthedetailsProvidedetailsonyourappliedresearchstrategyTalkaboutwhatyourcityhasdonenthepasttoresolvetheissueDiscusshowyourstrategyhaschangedwhenyoubegantoadoptthisapproachWasmoneysavedorinvested
Usethisstorymapexercisetoorganizeyourthoughtsandguaranteethatyouincludealloftheelementsofatraditionalstory
AfteryoudevelopyourmessagingcreateacommunicationsmarketingandoutreachstrategydesignedtospreadyourmessageYourstrategyshouldwalkyouthroughthefollowingkeyactivities
1 IdentifyandprofileyouraudienceWhencreatinganddisseminatingyourmessageitisalwaysgoodtothinkaboutyouraudienceWhoexactlywouldbeinterestedinwhatyouhavetosayThegeneralpopulationofthecityPeoplewhoworkinnonprofitsBusinessownersItmattersbecausenoteveryonewholivesinacitymaybethetargetoftheinformationBusinessownersmaybemoreinterestedindatashowingeffectsonthebusinessdistrictthanothersandsoonWewillcoverthislaterbutthisstepwillalsohelpyoutothinkofthemosteffectivewaytoreachyouraudience
2 Defineyourcalltoaction(ifyouhaveone)SpendsometimeconsideringwhyyouareinterestedintellingyourstoryDoyousimplywanttoupdatethepubliconyourprogressorcelebrateyoursuccessesAreyouinterestedinyouraudiencechangingitsbehaviororsupportingaspecificinitiativebeinglaunchedbyyourorganizationIfyouwanttohaveyouraudienceactorbehaveinacertainwaybespecificandincorporateitinyourmessaging
Shareyourstoryandsuccess
52
3 SelectperformancemeasuresThinkabouthowyouwoulddetermineifyourcampaignissuccessfulandselectappropriateperformancemeasuresYouwillrevisitthesemeasuresattheendofyourcampaigncommunicationsplanimplementationtodeterminewhetheryouneedtomodifyyourcommunicationsandoutreachstrategyforthefuture
4 MethodstocommunicateyourstoryTherearemanywaystocommunicatesuccessofyourprojectHerearesomesuggestionsonreachingaudiencesItisimportanttodeterminewhichmethodsarethemosteffectiveforyourdesiredaudiencetoreceiveinformationDoyourresearch(PewResearchCenterhasexcellentstudiesonthereachofmediadigitalmediaandmarketing)ForexampleifseniorsareyourtargetpopulationandtheyarelessinclinedtouseInstagram(thatistrue)youmaywanttoconsiderFacebookorthenightlylocalnewsbothofwhichareusedmoreoftenbyseniorsBelowisalistofsomemethodsandtoolsyoucanuse
i Pressreleasestolocalnewspapersandtvstations
ii ShortsmartphonevideosAgreatwaytoengageanaudienceisthroughvideoAskonequestionofsomeonewhoispartofthestoryMaybethereisasocialworkerwhocandiscusshungerinyourcityThesevideoscanbereusedonmultipleplatforms
iii Long-formwriting
iv TellingindividualstoriesthroughaseriesofphotosandshortnarrativesforpublicationonFacebookorInstagram
v Op-edsopinionpiecesplacedinthelocalpaper
vi Emailupdatestoyourlist
vii Connectingwithappropriateorganizationsthatareinvestedinthetopicandworkwiththecommunityandaskthemtodisseminateinformationtotheirmembers
viii EmployingldquoWordofMouthrdquo(WOM)tacticsbycreatingtownhallsorfreeeventsforresidentstoattendintheirownneighborhoods
5 DeterminethetypeofcollateralUsuallycampaignsthattargetresidentsrequireseveralavenuestoreachthemPeoplearenotalwaysinthesameplaceorspaceHavingakeenunderstandingofyouraudiencewillassistyouwithyourchoicesonwhatneedstobedevelopedSometimesthatmeansimagesforsocialmediaaonepageflierthatgivespeopleinformationonattendinganeventoranin-depthbrochure
Shareyourstoryandsuccess
53
6 ImplementyourstrategyKeepinmindstrategiescanbefluidSometimessomethingmaynotturnoutasexpectedevenifduringplanningsessionsitwasldquosupposedtordquoItisokYoucananalyzewhathappenedduringadebriefsessionandonceyoursurveysandothermeasurementshavebeenfinalized
Visualizations
Withtherightvisualizationyoucanreviewandshareinformationinsightsanalysisperformanceandtrendsallofwhichisvaluabletoacross-sectionofthepublicButwhichvisualizationisrightforyouTochoosetherighttypeofchartgraphdeterminethetypeofquestionyouwanttoanswerSomecommoncharttypesarebelowandmoreinformationcanbefoundhere
Exploreadditionalanalyticaltechniques
Duringthegoal-settingphaseyouestablishedoutputoutcomeandimplementationmeasuresBasedontheresultsofyourpilotyoumayneedtoadjustyourmeasuresWiththepilotdatainhandyoumayhavesomeideasofhowfutureanalysescouldbedifferentandpotentiallyimprovethequalityofyourprogramTakesometimetobrainstormotherrelevantdatasetsthatyoudidnothavethetimetopursueearlierincludingmessydepartmentaldatathatneedsmoreattentionorsourcesofpublicdatafromthestateandfederalgovernmentnonprofitsandevenprivatecompaniesthatcouldsupplementyourunderstandingofanissueUsingmachinelearningtechniquesyoumightbeabletobetter
Shareyourstoryandsuccess
54
predictdemandforhomelessservicesusingNOAAweatherforecastspreemptwatermainbreaksusingsoilstratigraphyorinvestineconomicdevelopmentbymonitoringrealestatelistingdata
Advancedanalyticaltechniquescanalsohelpyourimplementationrunmoreeffectivelybyintegratingdata-informedtargetinganddecisiontoolsintotheprocessByautomatingroutineworkflowsandsurfacingsubtletrendsthesetoolscanhelpyouiterateinamoreprecisemannerandprovidekeystrategicguidancetofront-lineemployeesthatwouldotherwisebeunavailableThesesortsofdata-science-enhancedprocessescanbeclassifiedasoneofsixprojecttypesasexploredfurtherbytheNewOrleansLANOLAlyticsteamFormoreguidanceonscopingasimilarprojecttheGovExadvancedanalyticsscopingguidegivesyouastep-by-stepoverviewofhowtobesuccessful
Usethespacebelowtolistyourideasforpossibleadvancedanalyticsprojectsbytype
Typeofproject Ideas
Findingtheneedleinthehaystack
Prioritizingforimpact
Earlywarningtools
Betterquickerdecisions
Optimizingresourceallocation
Experimentingforwhatworks
Keepthefeedbackloopgoing
GatherinformationonpublicsentimentaboutpolicychangethroughfocusgroupssurveysandpublicmeetingsledbyadvisorygroupmembersNotewhethernewinformationisavailablethatcanhelpanalyzeorcontextualizeyourprogress(egfederalorstatedatareports)Sharetheresultsofyourpilotandthepolicychangeprocesswithothercitiesandresearchers
Shareyourstoryandsuccess
55
WorkingwitharangeofevidencePresentlythereisalotofattentionandinvestmentinevidence-basedpolicymakingincludingthefederalbipartisanCommissiononEvidence-BasedPolicymakingthePewCharitableTrustsreportsandtoolkitsonHowStatesEngageinEvidence-BasedPolicymakingandtheLauraandJohnArnoldFoundationsupportedEvidence-BasedPolicymakingCollaborativeManyoftheseinitiativesfocusonrigorousevaluation--meaningrandomizedcontrolledtrials(RCT)orotherexperimentalmodels--asthebestkindofevidencetouseHoweverthisbodyofevidenceisofteninsufficientforthespecifickindsofquestionsthatgovernmentscanhavewhenmakingadecisionaboutpolicyInthisguideweproposeanapproachtosupportgovernmentsinincorporatingadditionaltypesofevidenceintotheirdecisionmaking
OnewaytounderstandarangeofevidenceistouseNestarsquosStandardsofEvidencetoframeanddefineevidenceTheBritishnonprofitNestaworkstoinjectpositivechangeintothewayUKgovernmentpoliciesandprogramsfunctionOneoftheirkeyareasofworkhasbeendevelopingWhatWorksCentresorganizationsthatcollectcommissionsynthesizeanddisseminateevidenceaboutinterventioneffectivenessacrossarangeofsocialimpactareasNestadevelopedthefollowingStandardsofEvidencetoprovideausefulwaytodocumenthowdifferentkindsofinformationdemonstratedifferentlevelsofevidence
Workingwitharangeofevidence
56
Theimportanceofpotentiallybiasedinformation
Onechallengepresentedbyanevidenceclassificationthatacknowledgesthevalueofnon-scientificformsofevidenceistofigureouthowtobalanceallofthepiecestogetherwhenyouusethemTounderstandhowtodothiswehavetothinkabouthowtopositionpotentiallybiasedinformationindecisionmaking
WhenwelookatinformationwhichhasnotbeenrigorouslyevaluatedweacceptthatthisinformationmightbebiasedThewholereasonweconductrigorousevaluationistoremovebiasfromourobservationsWeconductstructuredtrialstoproduceresultsthatareconsistentandgeneralizableIfafindingisunbiasedthisaccuracymdashthislackofbiasmdashcanbeprovedbecauseanyindependenttestercanreproduceitwhentheyfollowanidenticalprocess
YoucanlookatpotentiallybiasedinformationasjustinformationthatisearlieroninitspathtorefinementreplicationandverifiedaccuracyApieceofinformationbasedonanyoneindividualrsquosobservationsislikelytobebiasedbecauseitiscomingfromjustonesourceBiasisreducedbygatheringtogetherlotsofindependentsourcesofinformationandfiguringoutitsmostcommonattributes
MaybethebiasedinformationmakesaclaimthatistoolargemdashitsayssomethingistrueunderallconditionswhenitisreallyonlytrueundercertainconditionsorthataneffectofaninterventionisbiggerthanitstrueaverageorthataneffectlastsovertimewhenitdoesnotPerhapschangingtheclaimtobemorespecificmdashcriticallyreviewingtheinformationtoseeifspecificconditionsmustbemettoachievetheoutcomesmdashwouldmaketheclaimmoreaccurate
IfyoutestedyourclaimfounditdidnotholdupinsomecaseschangedittesteditagainuntilyourclaimrepeatedlyheldupthenyoucoulddemonstratethattheclaimwasnolongerbiasedbutaccurateThisprocessoftestingimprovingtestingandvalidatingiswhatwegetfromstructuredscientificstudy
Withindividualpiecesofinformation--crediblehypothesescasestudiesexpertopinion--youcannotassumethatkindofaccuracyAtthesametimetheseearlyobservationsarefoundationaltoanyreplicablescientificobservationsWewillnevergetanywhereifwedonotstartwiththesepiecesofinformationeveniftheymightbebiased
Combininglevelsofevidencetogetthemostdetailedinformationforyourcity
ItisespeciallyimportanttoacceptdifferentformsofevidencewhenyoufacequestionsthathavenotyetbeentestedbyindependentstudiesSomeofthemostimportantquestionsrelatetothingsthatareveryspecifictoyoursituationYoufaceaparticularcombinationof
Workingwitharangeofevidence
57
issuesactorstrendsandresourcesthatisultimatelydifferentfromeveryotherplaceintheworld
ThisisimportantbecauseaspolicypractitionersandnotscientistsyoucaremoreaboutyourspecificcasethanahypotheticalaveragegeneralcaseYoucarelessaboutthetotallikelyrangeofimpactsfromapolicythanhowthissingleimplementationwillworkYouwantinformationaboutpoliciesthatwillprovideyouwiththebestpossiblepredictionabouthowtheywillworkinyourcontext
ByputtingdifferentkindsofevidenceintothesameframeworkNestarsquosStandardsofEvidenceimplythatwhenyoucannotgetrigorouslytestedexperimentallyverifiedinformationitisappropriatetoworkwithwhatwedohaveThisisrelevantforthemunicipalimplementationcontextbecausenostudypulledfromaliteraturereviewwillaccountforallofthevariablesyoufaceinyourowncontext
Intheabsenceofhavingastudythatpredictshowaninterventionworksinyourspecificcontextyouwillneednon-scientificinformationtotrytofillinthosegapsYouwillneedtogatherstoriesfromresidentsstaffmembersandotherstakeholdersthatsitonLevels1or2ofthestandardsofevidence
Someofthekindsofinformationyouwillprobablygetmoreeasilyfrompeoplethanstudiesinclude
howpeopleincitieslikeyoursreactedtoaninterventioninpublicforumshowpeoplewhoimplementedaninterventioninacitylikeyoursfiguredouthowtogetitpassedhowthecitycommunicatedpubliclyabouttheapparentbenefitsanddrawbacksoftheinterventionthepoliticalacceptabilityoftheintervention
Youcanremainsimultaneouslyawarethattheinformationyouaregatheringthroughindividualstoriesisbiased--notgeneralizableinformedbyspecificperspectives--whilealsorecognizingthathavingsomeinformationaboutsomethingimportantisbetterthannoinformationEvidencefromthefirsttwoevidencelevelsareessentialbuildingblocksofknowledgeandperfectlyusefulwhenyoucannotgetdetailedunbiasedknowledge
Youhavetostartsomewhereandyoushouldalwaysstartwithevidenceyouhaveaboutyourveryspecificconditionsthatyouknowareimportantforsuccess
Workingwitharangeofevidence
58
Thankyou
Interviewswithcities
GovExheldsemi-structuredinterviewswithsixstaffmembersinfourcitiestoestablishtheinitialstepsoftheRoadmapWeaskedrespondentstotalkaboutaninitiativeorpolicythattheyhavebeenresponsibleforimplementingintheircityRespondentsfromthreecitiesspokeabouttheireffortstoaddresshomelessnessandrespondentsfromonecityspokeaboutblightAfterrespondentsidentifiedtheproblemtheyspokeabouthowthecityprioritizedthisissuethevariousstepsinvolvedandbarrierspresent
GovExconductedtextanalysisoftheseinterviewswhichrevealedthatpeoplewasthemostcommonlyusedtermduringtheseconversationswhendiscussingthestepstochangeaprogramorpolicyThisisathemethroughoutthisRoadmapThefirstsectionbeginswithidentifyingtherelevantpeopleTheconversationsaroundevidencearefocusedonhumanbehaviorratherthanmethodologyorstrengthofevidence
Thankyou
59
OncewehadadraftoftheRoadmapwefollowedupwithcitiesonanindividualbasistolearnmoreaboutwhatwouldbehelpfulWearegratefultothefollowingcitystafffortheinterviewsandindividualconversations
MichaelBaskinChiefPolicyOfficerChattanoogaTN
MelissaBridgesPerformanceandInnovationCoordinatorLittleRockAR
VictoriaCarreoacutenAdministrativeOfficerLasVegasNV
TyroneGrandstrandCommunityDevelopmentPlannerFargoND
TibLaughlinDirectorofGeneralServicesKansasCityKS
TimMorelandDirectorofPerformanceManagementandOpenDataChattanoogaTN
CharlesRosendahlManagementAnalystDentonTX
Thankyou
60
MelissaSiebenAssistantCountyAdministratorKansasCityKS
GinaThompsonDirectorofStrategicServicesLewisvilleTX
JHUFacultycollaboration
GovExalsoconvenedthecitycontactsandfacultypartnersinaseriesofAdvisoryGroupmeetingsThesemeetingsservedtoconnectcitiesandfacultytounderstandoneanothersperspective
StefanBaralBloombergSchoolofPublicHealth
AshleyBernerJHUSchoolofEducation
MelissaDavey-RothwellBloombergSchoolofPublicHealth
JonLinksBloombergSchoolofPublicHealth
TamarMendelsonBloombergSchoolofPublicHealth
KristinMmariBloombergSchoolofPublicHealth
KeshiaPollackPorterBloombergSchoolofPublicHealth
JoshSharfsteinBloombergSchoolofPublicHealth
LizStuartBloombergSchoolofPublicHealth
LindsayThompsonCareyBusinessSchool
DanielWebsterBloombergSchoolofPublicHealth
Partnersindataandevidence-basedpolicy
Wealsospokewithpartnerswhoareexpertsinthefieldofevidence-basedpolicywhosharedtheirknowledgeandresourcesofcurrentandfuturethinkinginevidence-informedthinking
WendyBarrenoAssociateDirectorWilsonSheehanLabforEconomicOpportunitiesUniversityofNotreDame
TiffanyDavenportDirectorofDataPartnershipsBaltimoresPromise
TomiHiersDirectorBaltimoreCivicSiteAnnieECaseyFoundation
ChrisKingsleySeniorAssociateAnnieECaseyFoundation
KathrynOliverAssociateProfessorLondonSchoolofHygieneandTropicalMedicine
Thankyou
61
KristenWatkinsSeniorPolicyAssociateJ-PAL
GovExParticipants
GovExAnalystMiriamMcKinneyanalyzedtheinterviewtranscriptsandconductedtextanalysisSeniorImplementationAdvisorEmilyShawwillintegrateRoadmapactivitieswithadditionalservicedeliveryactivitiesatGovEx
Thankyou
62
ReferencesAbrahamKGampHaskinsR(2017)Thepromiseofevidence-basedpolicymakingReportofthecommissiononevidence-basedpolicymakingWashingtonDCCommissiononEvidence-basedPolicymakingBigmove2Extendingthegreen(nd)Retrievedfromhttpswwwcityoflewisvillecomdoing-businesslewisville-2025big-move-2-extending-the-greenBatesMAampGlennersterR(2017)ThegeneralizabilitypuzzleRetrievedfromhttpsssirorgarticlesentrythe_generalizability_puzzleBuildwithnotfor(2018January)RetrievedfromhttpsbuildwithnotforwordpresscomCairneyPampOliverK(2017)Evidence-basedpolicymakingisnotlikeevidence-basedmedicinesohowfarshouldyougotobridgethedividebetweenevidenceandpolicyHealthResearchPolicyandSystemsdoi101186s12961-017-0192-xCairneyPOliverKampWellsteadA(2016)TobridgethedividebetweenevidenceandpolicyReduceambiguityasmuchasuncertaintyPublicAdministrationReview76(3)399-402Retrievedfromhttpdxdoiorg101111puar12555Centreforpublicimpact(nd)RetrievedfromhttpswwwcentreforpublicimpactorgCernaL(2013)ThenatureofpolicychangeandimplementationAreviewofdifferenttheoreticalapproachesOrganisationforEconomicCo-OperationandDevelopmentRetrievedfromhttpswwwoecdorgeduceriTheNatureofPolicyChangeandImplementationpdfColoradoSprings(nd)Retrievedfromhttpscoloradospringsgovcommunicationspagewhat-expect-when-you-speakupContinuumofcare(coc)program(nd)RetrievedfromhttpswwwhudexchangeinfoprogramscocCouncilpresidentkicksoff$12millionyouthfund(nd)Retrievedfromhttpwwwbaltimorecitycouncilcomcontentcouncil-president-kicks-12-million-youth-fundDurlakJAampDuPreEP(2008)ImplementationmattersAreviewofresearchontheinfluenceofimplementationonprogramoutcomesandthefactorsaffectingimplementationAmericanJournalofCommunityPsychology41327-350Retrievedfromhttpswwwncbinlmnihgovpubmed18322790Fishbonediagram(ishikawa)(nd)Retrievedfromhttpasqorglearn-about-qualitycause-analysis-toolsoverviewfishbonehtmlFivewhysGettingtotherootofaproblemquickly(nd)RetrievedfromhttpswwwmindtoolscompagesarticlenewTMC_5Whtm
References
63
FligsteinNampMcAdamD(2011)TowardageneraltheoryofstrategicactionfieldsSociologicalTheory29(1)1-26RetrievedfromhttpsociologyberkeleyedusitesdefaultfilesfacultyfligsteinFligstein20McAdam20Soc20Theory20PaperpdHeinkelP(2018January12)CitymullspermitsforbikesharingcompaniesRetrievedfromhttpwwwdentonrccomnewsdenton20180112city-mulls-permits-bike-sharing-companiesHowisareamedianincomecalculated(2012October)Retrievedfromhttpswwwhudexchangeinfofaqs843how-is-area-median-income-calculatedLampkinLMampHatryHP(2003)KeystepsinoutcomemanagementUrbanInstituteRetrievedfromhttpswwwurbanorgresearchpublicationkey-steps-outcome-managementMarshDampSmithM(2000)UnderstandingpolicynetworksTowardsadialecticalapproachPoliticalStudies484-21Retrievedfromhttpjournalssagepubcomdoiabs1011111467-924800247McCannL(nd)RetrievedfromhttplaurenellencomMoultonSampSandfortJR(2016)ThestrategicactionfieldframeworkforpolicyimplementationresearchPolicyStudiesJournal45(1)Retrievedfromhttponlinelibrarywileycomstore101111psj12147assetpsj12147pdfv=1ampt=jdkhhcp4amps=e184c625e1855a6b1615ccb99ee27b1c0b3212d8PaynterB(2018January1)What100citiesarelearningfromeachotherbysharingtheirdataRetrievedfromhttpswwwfastcompanycom40518004what-100-cities-are-learning-from-each-other-by-sharing-their-dataPeakacademy(nd)Retrievedfromhttpswwwdenvergovorgcontentdenvergovenmayors-officeprograms-initiativespeak-performancepeak-academyhtmlPinnockHBarwickMCarpenterCREldridgeSGrandesGGriffithsCJRycroft-MaloneJMeissnerPMurrayEPatelASheikhAampTaylorSJC(2017)Standardsforreportingimplementationstudies(stari)ExplanationandelaborationdocumentBMJOpenRetrievedfromhttpbmjopenbmjcomcontentbmjopen74e013318fullpdfProctorESilmereHRaghavanRHovmandPAaronsGBungerAGriffeyRampHensleyM(2011)OutcomesforimplementationresearchConceptualdistinctionsmeasurementchallengesandresearchagendaAdministrationandPolicyinMentalHealth38(2)65-76doi101007s10488-010-0319-7TandonSD(2018)ProgressincommunityhealthpartnershipsResearcheducationandactionRetrievedfromhttpswwwpressjhuedujournalsprogress-community-health-partnerships-research-education-and-actionProvanKGampKenisP(2007)ModesofnetworkgovernanceStructuremanagementandeffectivenessJournalofPublicAdministrationResearchandTheory
References
64
18(2)229-252Retrievedfromhttpsacademicoupcomjpartarticle182229935895SabatierPA(1988)Anadvocacycoalitionframeworkofpolicychangeandtheroleofpolicy-orientedlearningthereinPolicySciences21(3)129-168Retrievedfromhttpwwwjstororgstable4532139SandstromAampCarlssonL(2008)TheperformanceofpolicynetworksTherelationbetweennetworkstructureandnetworkperformancePolicyStudiesJournal36497-524Retrievedfromhttponlinelibrarywileycomdoi101111j1541-0072200800281xabstractSchwartzMampWilsonE(nd)WhocanaffordtoliveinahomeAlookatdatafromthe2006americancommunitysurveyRetrievedfromhttpswwwcensusgovhousingcensuspublicationswho-can-affordpdfShearerJCAbelsonAKouyateBLavisJNampWaltG(2016)WhydopolicieschangeInstitutionsinterestsideasandnetworksinthreecasesofpolicyreformHealthPolicyandPlanning31(9)1200-1211Retrievedfromhttpsdoiorg101093heapolczw052ThelessonsofadministrativedataHigh-profilepolicy-relevantresearchpoweredbyadministrativedata(nd)Retrievedfromhttpswwwpovertyactionlaborgsitesdefaultfilesresources20161114-The-Lessons-of-Administrative-DatapdfToolsandotherneedsforqualitativeandquantitativemethods(2015February)AgencyforHealthcareResearchandQualityRetrievedfromhttpwwwahrqgovprofessionalsquality-patient-safetytalkingqualityassesstoolshtml
WeaverRCampBagchi-SenS(2013)SpatialanalysisofurbandeclineThegeographyofblightAppliedGeography4061-70
References
65
UniversityPartners
Name Title Affiliation Area(s)ofexpertise
Contactinformation
Examplesofworkingwithnon-academicdecisionmakers
WendyPBarreno
AssociateDirector
WilsonSheehanLabforEconomicOpportunitiesUniversityofNotreDame
LEOsfocusareasincludehealtheducationhousingchildwelfareeconomicopportunitycriminaljustice
wbarrenonduduorleondedu
MelissaDavey-RothwellPhD
AssociateScientist
JohnsHopkinsBloombergSchoolofPublicHealth
Health mdavey1jhuedu410-614-5854
BaltimoreCityHealthDepartmentLeadersMarylandDepartmentofHealthAddictionRecoveryServicesinDCExecutiveDirectorsofcommunityorganizationsacrosstheUSthatfocusonHIVSTIsubstanceabuseandimplementationscience
ElizabethStuart
AssociateDeanforEducationandProfessorDepartmentsofMentalHealthBiostatisticsandHealthPolicyand
JohnsHopkinsBloombergSchoolofPublicHealth
Educationsocialpolicystatisticalmethods
estuartjhuedu
Resources
66
Management
Design-ThinkingResources
EngagementGuideforCivicTech
Journeymap
PerformanceManagementResources
DenverPeakAcademy
Proxymeasuresguide
Talkingaboutperformancewithoutcomplicatingit(video)
SettingPerformanceTargetsguide
Datavisualizationguide
Performancemanagementgettingstartedguide
Exercises
Fishboneexercise
5Whys
AdministrativeData
TheLessonsofAdministrativeData(JPAL)
EvaluationResources
Quantitativeandqualitativemethods(USDepartmentofHealthandHumanServicesAgencyforHealthcareResearchandQuality)
KeyStepsinOutcomeManagement(UrbanInstitute)
RCT-YES
Sageresearchmethods
BestPracticesforMixedMethodsResearchintheHealthSciences
ACCORDChecklist
Pilotandimplementationplanning
Benchmarkingexamplesandproxymeasures
Seattleopendataplaybook
Resources
67
Datasciencescopingguide
Resources
68
AboutGovExGovExhelpsgovernmentsusedataandevidencetoimproveresidentsrsquolivesWesupportgovernmentsinbuildingtheirinternalcapacitytooperatesustainableanalyticspracticeswhichimprovecommunityoutcomesinfieldsfrompublicsafetytoeconomicdevelopmentGovExhasaproventrackrecordofworkinginconcertwithpartnerstoprovidetechnicalassistanceandtrainingatscale
SinceourlaunchinApril2015GovExhas
Trainedover2000governmentemployeesinpersonandonlineintopicsincludingdatamanagementperformancemanagementopendatacommunityengagementandanalytics
Providedtechnicalassistancetomorethan100localgovernmentsacrosstheglobeimprovingtheiruseofdatatoachieveefficienciesinservicedeliveryandbetteroutcomesforresidents
Assessedthecapacityofmorethan130governmentalorganizationstodeterminehowleadersstaffandexternalstakeholdersmanageandusedata
Localgovernmentsareincreasinglyonthefrontlinesofaddressinglargescaleproblemswithglobalimplicationssuchasclimatechangepublicsafetyandeconomicinequality
Whileevidence-basedinterventionstohelpsolvethoseproblemsareavailablelocalgovernmentstaffhavelimitedtimeandcapacitytofindandimplementresearch-basedsolutionsAsaresultcitiescontinuetoworkthewaytheyalwayshave--makingincrementalimprovementstoprogramsbutnotalwayscreatingchangeatasystem-level
ThroughresourcedevelopmentandcollaborationwithJHUfacultyGovExissupportingcityleadersinusingdataandevidencetomakedecisionsandcreatinglastingrelationshipswithresearchers
TheRoadmapwasdevelopedbyGovExaspartofitspartnershipwithBloombergPhilanthropiesWhatWorksCitiesinitiative
AboutGovEx
69
- Title page
- Introduction and purpose of the Roadmap
- Using the Roadmap
-
- Gather information about the problem
- Strategically engage key partners
- Review the evidence and collect details
- Decide on a policy to implement based on local factors
- Establish a framework to monitor progress
- Assess data accessibility quality and security
- Design implementation and evaluation
- Share your story and success
-
- Working with a range of evidence
- Thank you
- References
- Resources
- About GovEx
-
collegeoruniversitytoprovideyouwithalistofarticlesfromthefollowingsources(Thisisalsoanexcellentopportunitytoseekresearchsupportfromanyacademicstakeholdersyouidentifiedpreviously)
Indexes
GoogleScholarincludesmostpeer-reviewedonlineacademicjournalsandbooksconferencepapersthesesanddissertationspreprintsabstractstechnicalreportsandotherscholarlyliteratureincludingcourtopinionsandpatentsItallowsyoutospecifyapublicationrangeinyoursearchandprovideslinkstolikelysourcesforalllistedarticles
PolicyFileIndexisauniqueresourceforUSpublicpolicyresearchinthatitgrantsusersaccesstotimelyupdatedinformationfromover350publicpolicythinktanksnongovernmentalorganizationsresearchinstitutesuniversitycentersadvocacygroupsandotherentities
PolicyArchiveworkstocollectanddisseminatesummariesandfulltextsvideosreportsbriefsandmultimediamaterialofthinktankuniversitygovernmentandfoundation-fundedpolicyresearch
Anothersourceforhigh-qualityresearchisthereportsoftheCongressionalResearchServiceYoucanfindtheirreportsaggregatedhere
Specificjournals
Oftenthemostusefulacademicjournalsforunearthingpolicyorprogramreviewswillbeineachspecificsubjectarea--thatisineducationjournalsforeducationpoliciesYoucanuseageneralindexlikeGoogleScholartobegintolearnwhichjournalsaremostoftenusefulforyoursearchHoweveryoucanalsosearchthefollowingtwogeneralistpolicypublicationsforissuesandarticlesrelatedtoyourtopic
PublicAdministrationReviewJournalofPolicyAnalysisandManagement
Newssourcessubjectmatterexpertsandinterviews
AlthoughitisvaluabletoknowwhathasbeenexploredthroughascholarlyresearchframeworkitisalsousefultolearnwhatkindofinterventionsarebeingtriedinthefieldrightnowandhowpeoplefeelaboutthesetrialsSearchsubscriptionnewsaggregatorslikeLexisNexisorfreenewsaggregatorslikeGoogleNewsorYahooNewsforyourissueareaplusthewordpolicyorprogramtoseewhatkindsofinterventionsarebeingcoveredinthenews
Reviewtheevidenceandcollectdetails
26
DonotforgettocontacttheadvisorygroupmemberswhowerechosenfortheirsubjectmatterexpertiseInadditiontoprovidinggoodmaterialthroughinterviewsthemselvestheymayhaveideasaboutothersubjectmatterexpertsyoushouldcontacttoaskaboutpotentialinterventions
Peercitiesandnetworks
InadditiontoreadingabouttheworkofothercitiesitisidealifyouareabletospeakwithpeoplewhoareinvolvedwithcurrentimplementationsofinterestinginterventionsInyourcityyoulikelyhaverelationshipswithyourneighborsbutitcanbehelpfultolookoutsideofyourregionforsolutionsaswellTofindcounterpartsinothercitieswhoareworkingonaprogramyouwouldliketolearnmoreaboutyoucancontactstaffinintermediaryorganizations(likeGovExoryourprofessionalassociations)whomightbeinapositiontoputyouintouchimmediatelyorwhomightseeanothergoodrouteforsettingupaconversationHereisalistofnetworksthatmaybehelpful
InternationalCityCountyManagementAssociation(ICMA)NationalAssociationofCounties(NACO)NationalLeagueofCitiesUSConferenceofMayorsWhatWorksCities
Collectthedetails
Onceyourreviewershavefoundevidenceonpossiblepoliciesorprogramstoimplementtheyshouldcapturekeydetailsthatwillhelpdecisionmakersunderstandthepotentialeffectivenessandapplicabilityofeachpotentialintervention
Foreachsourcethathasprovidedinformationyouwanttoconveyusethetemplatebelowtocaptureasmanyofthefollowingdetailsaspossible(Youdonotneedtocompletethisforsourcesthatprovideduplicativeinformationorareotherwisenotthatuseful)
Programorpolicyname (egldquoHousingFirstrdquoorldquoblightedpropertiesregistryrdquo)
Sourcetype (egjournalarticleinterviewsubjectmatterexpertstudyassociationcommunityparticipationprocess)
Sourcedetails (egarticletitleandpublicationnameandtitleofinterviewee)
Evaluationmethodology
(egrandomizedevaluationcomparativeevaluationoftwodifferentgroupsbeforeafteranalysis)
Evidencelevel (egLevel12345)
Programorpolicy
Reviewtheevidenceandcollectdetails
27
description
Programorpolicygoal (1sentencecapturingtheprincipalaimoftheintervention)
Existingimplementations
(WherehastheinterventionbeentriedListasmanysitesasyouareawareof)
Characteristicsofimplementationsites
(ForeachimplementationsiteofferbasicgeographicIDmdashcityandstateThenyouwillwanttocollectthedetailswhicharerelevanttoyourdecisionmakersrsquoconsiderationsaboutwhatmakesalocationldquosimilartordquoorldquodifferentfromrdquoyoursituationYouwillatleastwanttoincludeforeachimplementationsitePopulationnumberPopulationdensity(ruralurbansuburban)Socio-economiccharacteristicsBeforestartingaskyourdecisionmakersaboutanyparticularcharacteristicsofprogramimplementationsitestheywouldwanttoknowbeforeseekingtoreplicate)
Characteristicsofimplementationpartners
(Anyavailabledetailsaboutorganizationsinstitutionsorpeoplewhowereinvolvedwithimplementingtheintervention-egindependentchildcarecenterpublicelementaryschoolcitypublichealthnurses)
Characteristicsoftargetpopulationeligibleprogramparticipants
(Anyavailabledetailsaboutpeoplewhoarethebeneficiariesoftheprogramorotherwiseincludedoraffectedbytheintervention-egnumberofprogramparticipantscategoriesofpropertyownersaffected)
Policyorprogramduration
(HowlonghastheinterventionbeeninplaceineachimplementationsiteHasitendedandifsowhy)
Whatbenefitscamefromeachimplementation
(Anyobservedpositiveeffectsoftheinterventionincludingeffectsize)
Statisticalestimatesofinterventioneffectiveness
(Anyavailablestatisticalestimatesoftheinterventionrsquoseffects)
Whatdrawbackswereobserved
(Anyavailabledetailsaboutunintendedorunavoidablenegativeeffectsoftheintervention)
Programorpolicycosts
(Anyavailabledetailsaboutprogramcostsperimplementation)
Reviewtheevidenceandcollectdetails
28
Reviewtheevidenceandcollectdetails
29
DecideonapolicytoimplementbasedonlocalfactorsTogetthecriticalbuy-infromleadershiptoimplementanewpolicyorprogramtheprojectleadshouldscheduleameetingwithdecisionmakerstopresenttheevidencesurroundingthepolicyoptionscapturedinthetemplateDependingonyourinsightintoyourcityleadershipswillingnesstomakechangeyoumaywishtosharetheresearchyouhaveconductedontherootcausesoftheproblemandsomeofyouranalysisontheimpactofcurrenteffortstodate
Considertheapplicabilityofanapproach
AsdiscussedinthepreviouschapterpolicymakersoftenlookforevidenceofeffectiveprogramsinothercitiesbutoftenthequestionarisesofhowgeneralizableaprogramisfromonecitytoanotherTheAbdulLatifJameelPovertyActionLab(J-PAL)suggeststhatinreviewingevidencefromanotherlocationitishelpfultoidentifythespecificmechanismthatwasused(deliveryofprogramorserviceinformationsharingetc)identifythelocalconditionsthatwereinplaceobservethebehaviorchangethatresultedfromthemechanismandfinallytounderstandlocalcontextandconditions(legalorcostconstraintsculturalnormsetc)Thissetofpresentationelementscanbedrawnfromthelistofdetailscollectedintheresearchtemplate
EducationpolicymakersinChicagoILusedfindingsfromprogramsinKenyaandIndiainwhichchildrenwhohadfallenbehindtheirpeersweregiveneducationalsupporttargetedtotheirleveloflearningThemechanismusedwastargetedteachingandthereweresimilarconditionspresentinallthreecountriesAlthoughsomestudentshadfallenbehindinthecurriculumteacherswereincentivizedtoteachgrade-levelmaterialsConsiderationsofhowthelocalcontextinChicagomightaffectimplementationofatargetedteachingprogramincludedunderstandinghowtheneighborhoodsthatthestudentswerefrommayhaveaffectedtheireducationaloutcomesandthehighcostofteachinginsmallgroupsInChicagoMatchEducationprovidedtwo-on-onetutoringwhichmatchedstudentswithwell-educatedindividualswhohavenothadformalteachertrainingforalower-costmodelTheresultswerepositiveStudentswhoweretutoredperformedbetterinmathandweremorelikelytopasshighschoolclasses
ThegeneralizabilityframeworkproposedbyJ-PALisbelowUsethisasatemplateasyouconsiderpotentialpolicysolutions
Decideonapolicytoimplementbasedonlocalfactors
30
SourceTheGeneralizabilityPuzzle
OnceyouhavenarrowedthepolicyoptionsdiscussthequestionsbelowforeachoptionsyouareconsideringThisstepiscriticalasunderstandinglocalcontextissoimportantinsuccessfulpolicyimplementation
CommunityContextQuestionnaire
Localcontext
DoesthisoptionmakesenseinourcommunityIsthisaonesizefitsallapproachIsanyparticulargroupabsentfromthisapproachIfsoisthatok
Youranswers
Effectiveness
WhatwouldweexpecttoobserveifthepilotisaseffectivehereasinCityX
Youranswers
Timing
DoesthetimingmakesenseIsthistherighttimetoimplementthispolicychange
Decideonapolicytoimplementbasedonlocalfactors
31
WherearethepotentialroadblocksthatcouldaffectthescheduleIsthereanywaytomitigatethisHowlongwouldittaketoseeanimpact
Youranswers
Implementation
DowehavethestaffcapacitytoimplementthisArethereadditionalpartnersthatyoucanbringtothetabletosupportimplementationDoesthismakesensegivenourformofgovernment
Youranswers
Authority
DoesourcityhavetheauthoritytoenactthisIfsowhatistheappropriatemechanismAretherelegalbarriersArethereanyexistingpreemptionissuesfromthestateorfederalgovernmentCanthosebeaddressedthroughlegislationadvocacyIstherepoliticalsupportoroppositiontothis
Youranswers
Cost
HowmuchwillthiscostannuallyHowwillwepayforitCanthepolicybeimplementedwithexistingfunding
IfsoisthisfundingreliableHowwouldthispolicyoptionaffectourcitysbudgetintheshortandlongtermWhatotherfundingoptionscanweconsiderIsthereanalternativethatwouldcostlessandgiveussimilarresultsWillanyconstituencybeadverselyaffectediffundingforprogramsthatimpactthemisshiftedtoanotherprogrampolicyorinitiative
Youranswers
Decideonapolicytoimplementbasedonlocalfactors
32
EstablishaframeworktomonitorprogressNowthatyouhavereviewedtheevidenceconsideredlocalcontextandchosenyourpolicyitistimetodeterminehowyourcitywillmonitorprogressofthisnewapproachCitystaffandleadersneedtoestablishtheinfrastructureforongoingcontinuousdecisionmakingbasedonwhetherthepolicychangeisleadingtothedesiredoutcomesforresidentsandwhetherthechangeismeaningfulandrationaltofrontlineworkersSettingarbitrarygoalsatthelocallevelthatconflictwithgoalsatotherlevelsisconfusingtotheindividualsresponsibleforachievingthegoalsGoalsthataresettoohightoachievecanbedemoralizingtostaff
ThissectionincludesstepstosetupavenueinwhichCitystaffandleadershiproutinelyvisitkeyindicatorstodeterminewhatisorisnotworkingandtomakedecisionsinreal-timeFormorereadingonperformancemanagementpleasevisittheGovExPerformanceManagementGettingStartedGuide
InperformancemanagementtermssuchasindicatormeasureandmetricmaybeusedinterchangeablyInthisguidewewilluseindicatorInyourworkthemostimportantthingistobeconsistentinwhichevertermyouuseThisvideomaybehelpfulinunderstandingthisconceptandcommunicatingittoothers
Establishthefundamentalsandgatheryourteam
FirstdeterminewhetherthereisanestablishedvenueorregularmeetingscheduleduringwhichyoucandiscussprogressIfyourcityhasanexistingperformancemanagementstructureonarelatedtopicaddingadiscussionoftheprogressofpolicychangetoyouragendacanbeagreatideaIfthiswillbeastandalonemeetingdecidehowfrequentlytheprojectteamanddecisionmakerswillmeetandputthosemeetingsonthecalendarfortheentirelengthoftheproject
DeterminewhowillstaffandattendroutinemeetingsonCityprogressThesemeetingsmayincludetheMayorandorCityManagerdepartmentheadstaffwhoareimplementingthepolicychangeandanalysts
Identifyindicators
ItiscriticalthatindicatorsbeanimportantcityandcommunitypriorityaswellassomethingtheCityplaysamajorroleinaddressingTheremaybeinstancesinwhichtheCitywillneedtoreachouttoorcollaboratewithexternalpartnerstogetamorecompletepictureofprogressontheoverallpolicychange
Establishaframeworktomonitorprogress
33
Anoutcomeindicatordescribesthedesiredeffectofthetargetedpopulation
ExampleBellevuesfamilyfriendlyneighborhoodsindicatormeasuresthepercentofresidentswhobelievetheyliveinaneighborhoodthatsupportsfamiliesparticularlythosewithchildren
WithyourkeypartnersrevisityourproblemstatementcurrentstateandfuturestateexerciseandbegintoidentifytheoutcomeindicatorsthatwillillustratesuccessFornowchooseasingletop-levelkeyperformanceindicator(KPI)ItiscriticalthattheKPI-thepieceyouwillmeasure-beapriorityforthecommunityandsomethingtheCityplaysamajorroleinaddressing
Outputindicatorsmeasuretheprocessofprogramorservicedelivery
ExampleSanDiegoMayorKevinFaulconersgoaltorepair1000milesofstreetby2020
ReviewingtheadministrativedatasetinventorybegintoparedowntherelevantprogramsanddatasetsthatsupportyourKPIWhatarethemeasuresofeffectivenessforeachprogramTheseareyouroutputindicators
Thediagrambelowillustratestherelationshipbetweenoutputandoutcomeindicators
ThinkaboutimplementationindicatorsasmeasuringstepsintheprogramsrolloutthatarenecessarytoobtaindesiredchangesImplementationoutcomescanfocusontheinterventionstaffdeliveringtheinterventionanorganizationorsystemandcanaddressfeasibilityadaptabilitysustainabilityandcostTwokeyquestionstoconsiderwhendevelopingimplementationindicatorsare
1 Istheinterventionacceptableandappropriatetothecommunityand
2 IsitfeasibletoimplementinterventionXatthistimewiththeseresources
Establishaframeworktomonitorprogress
34
Amoreextensivelistwithdefinitionsofpotentialimplementationoutcomesislinkedhere
UsethetablebelowtochoosethedesiredimplementationoutcomesforyourpilotandtoplanyourmeasurementtechniqueThisshouldincludelevelofanalysis(individualproviderorconsumerorganizationorsetting)yourmeasurementtechnique(surveysobservationinterviewfocusgroupsadministrativedataanalysis)andthephasewhenyouwillcollectthisinformation(earlymidlate)Theremaybemorethanonelevelofanalysismeasurementtechniqueandphaseofdatacollection
Implementationoutcome
Definitionofoutcome
Levelofanalysisandmeasurement
Yourmeasurementtechnique
Phaseofpilotwhenyouwillcollectdata
Usethespacebelowtocaptureyourindicatorwishlist
Indicatorwishlist
Outcomes_____________________________________________________________________________
Outputs________________________________________________________________________________
Implementationoutcomes______________________________________________________________
Setapublichigh-levelgoal
Thehigh-levelgoaliscriticalinestablishingdirectiontogalvanizethepeoplewhowillbedoingtheworkWhendiscussingthegoalusetheMayorsbullypulpitLookforkeymilestonesinCitygovernmenttosharethegoalstatementsuchasthebeginningofaMayoraladministrationaStateoftheCityspeechorotherpublicremarks
GoalsshouldbeldquoSMARTrdquo
Specificaspossible(innamingthepopulationwhowillbebetterofftheactionthecitywilltake)
Measurable
Achievable
Establishaframeworktomonitorprogress
35
Results-focused
Time-bound
ThegoalstatementshouldincludeyouroutcomeindicatorlikeChattanoogasgoalldquoWewillendchronicveteranhomelessnessbyDecember2016rdquoAdiagramtohepyoubuildaSMARTgoalaroundyouroutcomeindicatorisbelow
Component Description Yourturn
Specific WhowillbebetteroffWhatactionthecitywilltake
Measurable Howwillyoumeasuresuccess
Achievable Isthegoalattainable
Results-focused Whatistheexpectedresult
Time-bound When
Setrealisticambitioustargetsandtimelinesforeachindicator
ToestablisharealisticandambitioustargetforyourgoalstatementusetheadministrativedatacollectedforyourchosenKPItoreviewactualCityperformanceinthemostrecentyearcomparedwiththethreeorfourmostrecentyearsworthofdataBegintocalculatesummarystatistics(egtotalminimumaveragemedianmaximum)toidentifytrendsCreatecumulativeandmonthlychartstoidentifytrendsandanomaliesBasedonyouranalysisofhistoricaldataignoretheanomaliesandassumetheproductivitytrendswillfollowthesameoverallpatternfrommonth-to-monthFinallyapplyanincreasedproductivityfactortotheperformanceeachmonthtoforecastfutureprogressBasedonthisanalysischooseatargetthatinspiresimprovementwhileaccountingforreality
Manycitiesthataremorematureintheirperformancemanagementpracticesrecognizetheimportanceoftakingacommunity-levelapproachtoimprovingoutcomesforresidentsThesecitiesarecreatinggoaltaxonomiesthatincludemeasuresrelatedtopublicsafetyhousingandevenrecreationCitiesdriveprogressonthesegoalsbyimplementingstrategiesthatengagemultipledepartmentsandcollaboratewithserviceprovidersinthecommunity
EventuallyyouwilldevelopgoalstatementsfortheoutputsinordertocapturetheefforttoachievetheoutcomecontainedinyourgoalstatementYoursetofindicatorsshouldbecomposedofshort-andinterim-termsupportivemeasureswithaccuratetargetsandrelevanttimelinesAnexamplegoaltaxonomyfromtheCityofLasVegassResultsVegasprogramisbelowUsethistaxonomyasamodelforyourownsuiteofindicatorsandgoalstatements
Establishaframeworktomonitorprogress
36
Yourturn
Thetemplatebelowismeanttohelpyoucraftagoalstatementforeachsupportingmeasure
Goalstatement
Departments
Outcome
Activities
KPI
Target
Timeframe
Conductanddiscussrelevantanalysis
ThrougharoutineseriesofmeetingscontinuallymonitorprogressofKPIssupportingindicatorsandanyotherdatacollectedsuchasqualitativedatatoidentifytrendsthatwillindicatewhetherthechosenpolicyishavingthedesiredimpactThesedatapointsshoulddrivethediscussionduringthesemeetingspointingtoopportunitiesforfurtherinquiryinto
Establishaframeworktomonitorprogress
37
whatisdrivingtheobservedtrendsAnalystsshouldbeencouragedandtrainedtoaskquestionsofthedepartmentsproducingthedataandconductsitevisitswhenpossibletodeepentheirunderstandingofthedataPresentationoftheanalysisshouldfocusononeideaatatimetokeeptheconversationfocusedonkeypointsandspecificopportunitiesforimprovementYoumayconsiderdevelopingapublicfacingdashboardtosharethisprogressmorebroadly
AgeneralcomparisonofthestatusquodatawithwhatwasobservedduringthepilotcanprovidedecisionmakerswithaquickunderstandingofthedifferencesbetweentheoldandnewapproachesalthoughthiswillnotshowcausalityOftenthesecomparisonswillincludeoperationalmetricssuchasdaysordollarssavedbutrememberthattheimpactonpeoplewillbemostimportantIfyouareworkingwithanevaluationpartnertheywillsupportthisstep
Usethespacebelowtosummarizetheresultsofyouranalysis
MeasureTargetvalue(seesectiononsettingtargetsabove)
Pilotvalue(whatdidyouobserveduringthepilot)
Historicalaverage(ifavailable)
Results(WhoisbetteroffIsanyonenegativelyimpacted1-2sentencesreassessingyourgoalstatement)
DocumentyourprocessandprogressincludingfeedbackfromtheparticipantsimplementationpartnersandtheprojectteamDidanythingunexpectedoccurduringthepilotArethereinternalandexternalfactorsthatmayaffectyouroutcomesThisiswheretheimplementationoutcomescanbecanhelpfulIftheinterventiondidnotgoasplannedconsiderwhetheritisbecausetheinterventionwasnotacceptableorfeasible
UsethetemplatebelowcreatedbyDenversPeakAcademytocapturethisinformationIfthe306090daytimelinedoesnotmakesenseforyourcityfeelfreetocustomizeit
CurrentState
FutureState
30Day
60Day
90day
Whatwentwell(+)
Whatdidnotgowellshouldbechanged(-)
IfaquestionordiscussionpointarisesthatcannotbeansweredinthemeetinganassignedstaffmembershouldcapturethequestionduringthemeetingAfterwardthefollowupitemsshouldbesenttorelevantpartnersandthenextmeetingshouldbeginwithadiscussionof
Establishaframeworktomonitorprogress
38
thoseitems
Takeaction
DiscussionsaboutwhetheryourindicatorsaremovingintherightdirectionandwhatmightbedrivingtrendsmayleadtoadiscoverythattherearenotenoughresourcesallocatedtothispolicyorissueTheremaybelimitedfundingforstaffprogrammingortechnologythatisdrivingthedatainthewrongdirectionIfCityleadershipisintheroomthisisagreattimetomakethecaseforadditionalfundingduringthebudgetseasonBycollectinganalyzinganddiscussingdataonthispolicyroutinelyyouarebuildingevidencetomakethecasefortherightleveloffundingtoreachtheestablishedtargets
Establishaframeworktomonitorprogress
39
Assessdataaccessibilityqualityandsecurity
Determineaccessibilityandqualityofdata
OnceyouhavecompiledyourwishlistofindicatorsitistimetodeterminetheaccessibilityandqualityofthedataTopreventcomplicationyoumaywishtoprioritizerelevantindicatorsforwhichyouarealreadycollectingdataAvoidthetemptationofadoptingatechnologysolutiontocollectnewdataatthispointsuchasaphotosurveytoolorotherappFornowyouaretryingtodeterminewhichdatasetsarerelevantaccessibleandhighqualityBesuretoworkwithyourkeyinternalpartnerswhileyouareidentifyingyourmeasuresDepartmentstaffwhoworkonprogramsarefamiliarwiththeindicatorsthatwillaccuratelysupporttheoutcomesanddatabaseadministratorswillbevaluablewhendiscussingaccessibilityandqualityofthedata
Abrieflistofconsiderationsfortheaccessibilityofyourdataisbelow
ArewecurrentlycollectingthisdatasetorwillwebecollectingitforthefirsttimeHowfarbackdoestherecordgo(timehorizon)WouldweneednewtechnologytocollectoranalyzethedataCanthedatabecollectedmanuallyorcanitbeautomatedHowfrequentlyisthedataproduced(continuouslyhourlydailyweeklymonthlyquarterlyyearlyetc)IsthedataavailablefortheentireCitypopulationorasubsetofthepopulationHowgranularisthedata(individuallevelCensusblockneighborhoodcitywideetc)DoesthedatacontainsensitiveinformationIsthereaproxymeasurethatwecanusefornowIsthedatamachinereadable
AsnotedbytheWilsonSheehanLabforEconomicOpportunities(LEO)attheUniversityofNotreDameleveragingadministrativedatacansignificantlyreducethecostsandtimelinesofrigorousevaluationCitiescanestablishaninternaldatasharingagreementacrossdepartmentsincludinglanguagethatenablesadministrativedatatobeusedforresearchpurposesCitiescanalsoidentifyexternaladministrativedata-holderssuchasacountyschooldistrictpolicedepartmenthospitaloranotherstatethatmayhaveoutcomedataimportantformeasuringprogramimpactCitiescanleverageexistingrelationshipsandorworkwithresearchinstitutionstodevelopstreamlineddatasharingprocessesandagreementswiththeseexternalentities
Assessdataaccessibilityqualityandsecurity
40
Evaluatethequalityofyourdata
QualityisasimportantasaccessibilityToevaluatedataqualityyoumaywishtocreatearubricliketheonebelow
Value ValueDescription
3 ExcellentDataQualityThedataiscompleteaccurateandupdatedregularly
2 AdequateDataQualityThedataissomewhatcompleteaccurateandupdatedfairlyregularly
1 PoorDataQualityThedataisnotcompleteaccurateorupdatedregularly
ThisisagreattimetobuildonthedatainventorythatyoustartedinthegatherinformationpageSuggestedadditionalfieldstoaddtoyourdatainventorytemplatearebelow
Program DatasetTitle
DatasetDescription DataSource
Departmentorexternalpartner
DoestheCityalreadycollectthisdata
Whatisthenameoftheprogram
Human-readablenameofthedataset
Shouldbeunderstandablebynon-technicalusers
Whatistheoriginalsystemapplicationfilethathousesit
Departmentorexternalpartnerthatmaintainsthedata
Isthisdataalreadycollected
Updatefrequency-ThefileformatforthedatasetTypicallycsvxlsxshapefiletxtetc
Format-Howoftenthedatasetisupdated
Timerange-Theperiodoftimecoveredinthisparticulardataset
Datause-Howisthisdatasetcommonlyused
Datausers-Whotypicallyusesthisdataset
Dataownerstitle-Titleofthedataowner
Dataownersposition-DataOwnersjobtitle
Dataownersemailaddress-DataOwnersemailaddressWhomanagesthedataandorisresponsibleforgrantingpermissiontoaccessthedataWhounderstandswhatthedatasetincludesandcananswerquestionsaboutit
Dataqualityconcerns-Describetheconcernsyoumayhaveaboutthequalityofthedataset
Assessdataaccessibilityqualityandsecurity
41
Division-DivisionwherethedataismaintainedIfmorethanonedivisionisresponsiblelisttheprimaryownerofthedataset
Sensitivedatacomments-DescribetheissuesthatwouldpreventthecityfromsharingthedatapubliclyCommonsensitivityconcernsincludeprivacyviolationssecurityissuesorhighcostorstaffdemandstopostthedata
Publishingstatus-Thistheplacetoputanycommentsregardingthecurrentstatusofthedataset
Publishedlink-Thisisthelinktothepublisheddatasetontheopendataportalorcitywebsite
Reviewdatasecurityandprivacyprotectionprotocols
OftenapolicychangealsoinvolvesachangeinthedatathatisbeingcollectedhowthatdataisbeinghandledwhohasaccesstoitandahostofotheradjustmentsAsyoutakestockofthedataassetsthatareavailableforyourinitiativeitisvitaltoalsothinkthroughwhatwillbedonetoprotectthatdataandtheprivacyofthepeopleitrepresentsAbigpartofgooddatasecurityinvolvesstrongdatagovernancepracticesthatdefineandidentifysensitivedataandconsciouslyworktobalancethevalueandriskofstoringsuchsensitivedataOnewayofevaluatingyourdatacomesfromcaselawrelatedtotheUSFreedomofInformationActwhichwehavetranslatedtothechecklistbelow
1 Woulddisclosureresultinasubstantialinvasionofprivacy2 Haveweconsideredtheextentorvalueofthepublicinterestandthepurposeorobject
oftheindividualsseekingdisclosure3 Istheinformationavailablefromothersources4 Wastheinformationgivenwithanexpectationofconfidentiality5 Isitpossibletomouldreliefsoastolimittheinvasionofindividualprivacy
TheseriskscouldbemitigatedusingavarietyofstrategiesfromaggregatingdatatoobfuscatingindividualstoredactingfieldsthatmayincludePersonallyIdentifiableInformation(PII)orre-evaluatingyourdatacollectionmethodstofindalternativewaysofgettingtheinformationyouneedThesestepsshouldbecodifiedandmanagedbyadatagovernancecommitteeifpossibleForthesensitivedatathatisdeemedessentialcreateaplanaroundpotentialdatabreachesinordertoquicklyandconfidentlyrespondtoemergencysituations
ProactivelypreparingforthesesituationsisoftenthebeststrategyfordatasecuritybutdonotforgetthetechnicalsideInvolveyourITadministratorsinconversationsaboutdatastorageandaccessensuringthattheywillbeprotectedbyenterprise-gradecybersecuritysystems
Assessdataaccessibilityqualityandsecurity
42
Assessdataaccessibilityqualityandsecurity
43
Designimplementationandevaluation
Designtheimplementation
NowthatyouhaveyouraccountabilityplanandkeyindicatorsitistimetogetspecificaboutyourplanforimplementationThetemplatebelowwillhelpyouthinkthroughthespecificmechanismsofimplementingpolicychange
UnderstandingorganizationsandsystemsiscriticaltounderstandingandmakingdecisionsaboutwhereandhowinterventionsshouldbeappliedApolicyimplementedinasinglesettingmayshowpromisingresultsbutimplementingmultiplepoliciesandprogramsatthesametimewithdifferentactorsresponsibleforeachaspectcanhelpensuresustainablebehaviorchange
InSomervilleMAShapeUpSomerville(SUS)isacitywidepublichealthprogramthatisimplementedacrossmultiplelevelsofinfluenceandsystemsthroughinterventionsbeforeduringandafterschooltoincludechangestobuiltenvironmentsuchasbikelanesregionalmasstransportationandphysicaleducationequipmentinschoolsandgymstoincreasephysicalactivitywellnesspoliciesandschoolgardensnutritioneducationduringandafterschoolandhealthyrestaurantprogramstosupporthealthyfoodandhealthyeatinghabits
Usethetemplatebelowtoplanyourimplementation
Policyimplementationplanningtemplate
ListthespecificdesiredoutcomesInthepreviousstepyouworkedwithdepartmentstafftoestablishspecificmeasurableoutcomesandinterimindicatorsPrioritizetheindicatorsthatwilldemonstrateprogressUsethespacebelowtocapturethedesiredoutcomesofthepolicychange
ExampleShapeupSomervillebeganwiththegoalofpreventingobesityin1stthrough3rdgraders
Yourturn
____________________________________________________________________________
____________________________________________________________________________
Designimplementationandevaluation
44
SpecifythetargetpopulationAreyoudesigninganinterventionthattargetsonlycertainsitesAreyouplanningonimplementingaprogramorinitiativeforaspecifictargetaudienceClearlyspecifythepopulationyouplantotarget
IfyouplantoconductarigorousimpactevaluationasnotedbytheWilsonSheehanLabforEconomicOpportunitiesitisimportanttoestablishconcreteobjectiveeligibilitycriteriaandensurethatthesecriteriaareappliedconsistentlybyallstaffduringscreeningandintakeItisalsovitaltoidentifytheindividualswhoareeligibleforaparticularprogramorserviceevenifeveryoneeligiblecannotbeservedbyexistingresourcesThishelpscitiesidentifyanyunmetneedsandguidefutureresourceallocationwhilealsocreatingthepossibilityofarigorousimpactevaluationbyidentifyingapotentialcomparisongroupFinallywhenconductingscreeningandintakeitisimperativetocollectandmaintainbasicinformation(egnamedateofbirthgender)aboutallprogram-eligibleindividualsnotjustthosewhoparticipateintheprogram
ExampleIntheShapeUpSomervilleexampletheprogrambeganbytargeting1stthrough3rdgraders
Yourturn
____________________________________________________________________________
____________________________________________________________________________
DefinethelocationWherewillthepolicychangetakeplaceWillimplementationtakeplaceatonesiteormultiplesites
ExampleShapeUpSomervilleinterventionsweredeliveredinschools
Yourturn
____________________________________________________________________________
____________________________________________________________________________
ClarifyimplementationpartnersandarticulatetheirrolesWhatarethekeyorganizationsinstitutionsorpeopleresponsibleforimplementationTheprojectteamshouldbeasclearandspecificaspossibleinprovidingadescriptionoftheconcreteactivitiesthatimplementationpartnersaremeanttocarryoutPracticeprofilesmaybeahelpfultooltohelpdefinewhatisrequiredoftheactorswhoareresponsibleforimplementingtheintervention
Designimplementationandevaluation
45
ExampleInSomervillemayoralleadershipandengagementofmultiplecityagencies(publichealthprimarycareschoolsCambridgeHealthAllianceetc)streamlinedandconnectedthevariousShapeUpSomervilleinitiativesandgrantswhichhelpedtokeepthetalentandenergyfocusedonhealthyeatingandactivelivingforallsectorsofSomervilleSUSpartnersultimatelyincludedschoolsnonprofitsCityandstateagenciesandnationalfoundations
Yourturn
____________________________________________________________________________
____________________________________________________________________________
DescribetheinterventionWhatarethestepsactivitiesorprocessesRememberthatthemechanismthespecificstepsorprocessthatleadtobehaviorchangeiskeytounderstandingtheeffectivenessofthepolicychangeandsharingtheresultswithothers
ExampleGoalsofShapeUpSomervilleweretoinfluenceeverypartofanearlyelementaryschoolchildrsquosdayndashbeforeduringandafter-schooltopromotehealthyeatingphysicalactivityandtheirweightstatusInterventionincludedfoodserviceenhancementstoimprovebreakfastandlunchandstaffprofessionaldevelopmentwalktoschoolactivitiesSUSclassroomcurriculumandprofessionaldevelopmentschoolwellnesspolicydevelopmentSUSafter-schoolcurriculumandprofessionaldevelopmentoutreachandeducationtohomethroughmaterialsforumseventsldquoSUSapprovedrdquorestaurantscommunityoutreachandcapacitybuildingthroughpolicydevelopmenttrainingsmediaplacements
Yourturn
____________________________________________________________________________
____________________________________________________________________________
ClarifytimelineandmilestonesfordeliveryandevaluationHowfrequentlywilltheservicesinterventionbedeliveredHowlongwilltheprogramrunWhatisthetimetoexpectedimpacts(eg12monthsafterprogramcompletion)
Designimplementationandevaluation
46
ExampleShapeUpSomervilleledbyTuftsUniversityFriedmanSchoolofNutritionScienceandPolicyconductedaquasi-experimental(nonrandomized)interventionstudyamong1stthrough3rdgradestudentsinSomerville(intervention)andtwocomparisoncommunitiesfor9-monthsduringoneschoolyear(fall2002ndashspring2003)ThestudyshowedpositiveassociationoftheinterventiononchangeinBMISUSreleasedreportssummarizingevaluationsprogresstodateplansforadditionalprojectsandimpactstenandfifteenyearsintotheinitiative
Yourturn
____________________________________________________________________________
____________________________________________________________________________
PlanyouranalysisTheprevioussectionaddressedestablishinganaccountabilityframeworktomonitorprogressandimplementationDeterminewhichindicatorswillbediscussthroughthatprocesswhowillsharethedataandanalysisandhowfeedbackfromthoseconversationswillbeincorporatedbytheprojectteam
ExampleTheMayorrsquosofficeimplementedadata-drivendecisionmakingapproachcalledSomerStatwhichwassynergisticwiththeapproachestakenbytheresearchandevaluationteamsdeployedforthespecificgrants
Yourturn
____________________________________________________________________________
____________________________________________________________________________
PlanforsustainabilityIfthepolicyandprogramchangeissuccessfulhowwilltheprojectteamsustainandexpandtoadditionalsitesorpopulations
ExampleAlthoughmostofthepartnershipsforgedintheearlyyearsweremaintainednewgrantsandcollaborationsexpandedattentiontoolderpublicschoolstudentsactivetransportationandtheGreenLinesubwayextensionthebuiltenvironmentfarmerrsquosmarketeffortssuchasldquoGrowninSomervillerdquowithUnionMainStreeteconomicdevelopmentandhealthdisparitiesaswellasdatacollectionmonitoringandsurveillance
Yourturn
Designimplementationandevaluation
47
____________________________________________________________________________
____________________________________________________________________________
Pilotplanningtemplate
IfyouplantoconductapilotfirstdeterminethepurposeofthepilotDoyouwanttoobservewhetherthepolicychangeresultsinthedesiredbehaviorchangeAreyouconcernedaboutfeasibilityscalabilityorcostDoyouwanttotestwhetheryourdesiredresultscanbeachievedthroughthispolicyapproachClarifythegoalofthepilotandthequestionsyouhopetoanswerThiswillhelpyouidentifythekeyindicatorsthatyouwillusetomeasurewhetherthepilotissuccessfulBeforebeginningestablishabaselineofcurrentperformanceandsetyourtargetsaccordinglybasedonwhatyouhopetoaccomplishthroughthepilot
NextestablishthepilotplanforyourprojectteamincludingmilestonesforteamcheckinstodeterminewhethertomoveaheadorchangecourseThisisseparatefromthemeetingsthatwilloccuraspartoftheaccountabilityframeworkUsethespacebelowtocreateyourtimelineanddevelopyourmeetingschedule
Task Personresponsible Startdate Checkindate Deliverable
Planyourevaluation
WhileyouaredesigningtheimplementationitisalsotimetobegintodevelopyourplanforanalysisAsnotedbytheWilsonSheehanLabforEconomicOpportunitiesdecisionsaboutprogramsetup-includingoutreachscreeningintakedatatrackingandenrollment-candeterminewhetherornotarigorousimpactevaluationwillbepossibleAsdiscussedcityleadersmayfindithelpfultoincludeevaluationpartnersinearlyconversationsaboutprogramsetupDuringpreviousstagesyoumayhaveworkedwitharesearchertoplanyourproposedpolicychangeYoushouldbeclearaboutthepurposeoftheevaluationanddevelopanevaluationplanBelowareadditionalconsiderations
Causationvstrendanalysis
IntheEstablishaframeworktomonitorprogressstagetherewasguidanceontrackingtrendsovertimeItisimportanttonoteherethatmonitoringdataisdifferentfromanalyzingacausaleffectTrendanalysiswillillustratethatchangesareoccurringandmayleadtoquestionsthatcouldbeansweredbyyourevaluationbuttrendanalysisalonedoesnotprovideevidenceaboutwhatisdrivingthosechangesYoumayseechangesinyour
Designimplementationandevaluation
48
outcomeindicatorsbasedonvariablesoutsideoftheCityscontrolForinstanceachangeinfederaltradepolicycanhaveanimpactemploymentratesinyourCityoutsideofyourCitysnewjobtrainingandplacementprogram
ToaddresscausalityyoumustcomparetheoutcomesofindividualswhoreceivedtheinterventionwiththosewhodidnotWithoutacomparisongroupofsimilarindividualswhowerenotexposedtotheinterventionitisimpossibletoknowwhichfactorsareresponsiblefortheoutcomesRandomizedControlTrials(RCTs)inwhichindividualsarerandomlyassignedtotreatmentandcontrolgroupsarethegoldstandardinprovingcausalityIfyoucandemonstratethatoutcomesimprovedforthosewhoreceivedtheinterventionanddidnotimproveatallorimprovedlessforacomparisongroupyoucanbesomewhatcertainthattheinterventionwasthecause
Examplesofothernon-causalimpactevaluationsthatarerigorousandreliableincludecertainquasi-experimentalstudiesusingcomparisongroupdesignssmartinterruptedtimeseriesanalysesthatmodeltrendsovertimeinwaysthatallowestimationofinterventioneffectswelldesignedpre-posttestanalysesneedsassessmentsimplementationstudiesetcFormoreinformationondifferenttypesofevaluationsseetheSageResearchMethodswebsite
Lookfordifferencesingroups
BuildintimetoalsoexaminethepilotdatatoseeifgroupsareexperiencingdifferentresultsReviewingperformancebasedonoutcomemeasuresforapopulationinaggregatemayhidethesubtlewaysthatsomepeoplemaybeaffecteddifferentlysoitisnecessarytocarefullyanalyzethedatafordisparateoutcomesForinstancetheMovingtoOpportunity(MTO)studythatexaminedtheimpactofhousingvouchersonfamilieswholivedinhigh-povertyneighborhoodsfoundthatchildrenwhomovedbeforetheageof13hadbettereducationandhigherincomesasadultsHoweveradultsdidnotseeimprovededucationandemploymentoutcomes
Disaggregatingdataintomorespecificcategoriesofraceandethnicitycanhelpunderstandtrendsanddisparitiesinamoregranularwaywhichcanhelpdecisionmakerstargetpoliciesandinterventionsinamoreequitableway
EarlierintheRoadmapweworkedtoidentifystakeholdersandwhomightbepositivelyandnegativelyimpactedbyachangeinpolicyUsetheresultsofthatexercisetoinvolvethecommunityininterpretingdataanalysisandtogatherqualitativefeedbackfromthecommunitytocallattentiontoissuesForinstanceanincreaseinyouthdetentionmaybeexplainedbyinefficienciesorlackofaccesstotransportation
Designimplementationandevaluation
49
Designimplementationandevaluation
50
Shareyourstoryandsuccess
Sharetheresults
Storytelling
AnintegralstepinyourstrategyissharingtheresultsofyourpilotorpolicychangewithyourcommunityThisincludesdevelopingmessagingaroundtheworkofyourorganizationandcreatingaplantodisseminateyourstoriesWhytellstoriesaboutyourwork
Informandorunderstandasubjectorsituation
Giveneededcontextofdataorbitsofinformation
Relatablestoriesarememorableforreadersviewers
ReadersorviewerswillseetheinformationasmorecompellingwhenthereisademonstratedandrelatablechangeorimpactonthemselvesortheirfriendsneighborsandcolleaguesStoriesareaboutpeopleandgovernmentisfundamentallyabouthelpingpeopleSotellthestoriesofthepeopleyouandyourcitycolleagueshelped
Keepthesetipsinmindasyoudevelopyourstorywithyourteam
WhatwereyourfindingsandwhyshouldanyonecareTellyouraudiencehowtheproblemwassolvedWhathastheimpactbeenonresidentsandcitystafftodateWhatwasthesuccessandweretheremultiplesuccessesalongthewayIfyouhavepreliminaryresultsordatashowingimprovementtosharethisisthetimetodoit
WhowasinvolvedandimpactedReviewthelistofstakeholdersHowwereeachofthemimpactedbytheprogramHumanizeyourstoryandimpactbyaddingadditionalvoicesThesevoicescanbeoneofthefollowing
Apersonorfamilywhowaspositivelyaffectedbythechangesuccess
Aninvolvedthird-partydescribingboththeproblemandthesolution(ieanorganizationnonprofitprovidersbusinesstradeassociationsportsteametc)whoispartofthecommunitythatthegovernmentworkedwithonthechangesuccess
Someonefromthegovernmentitselfwhocanspeakaboutthechangesuccess
TellmoreaboutyourcityDescribeyourcityincludingpopulationdemographicsindustriesetcWhatareitschallengesWhatisitknownforWhatmakesituniqueHowisitsimilarordifferentfromothercitiesintheUS
Shareyourstoryandsuccess
51
DescribeyourcityrsquoschallengeWriteoutthechallengeyourcityisfacing(asdefinedinyourproblemstatement)HowdoesitaffectresidentsHowdoesthisproblemaffecttheworkofcitystaffmembers
AddthedetailsProvidedetailsonyourappliedresearchstrategyTalkaboutwhatyourcityhasdonenthepasttoresolvetheissueDiscusshowyourstrategyhaschangedwhenyoubegantoadoptthisapproachWasmoneysavedorinvested
Usethisstorymapexercisetoorganizeyourthoughtsandguaranteethatyouincludealloftheelementsofatraditionalstory
AfteryoudevelopyourmessagingcreateacommunicationsmarketingandoutreachstrategydesignedtospreadyourmessageYourstrategyshouldwalkyouthroughthefollowingkeyactivities
1 IdentifyandprofileyouraudienceWhencreatinganddisseminatingyourmessageitisalwaysgoodtothinkaboutyouraudienceWhoexactlywouldbeinterestedinwhatyouhavetosayThegeneralpopulationofthecityPeoplewhoworkinnonprofitsBusinessownersItmattersbecausenoteveryonewholivesinacitymaybethetargetoftheinformationBusinessownersmaybemoreinterestedindatashowingeffectsonthebusinessdistrictthanothersandsoonWewillcoverthislaterbutthisstepwillalsohelpyoutothinkofthemosteffectivewaytoreachyouraudience
2 Defineyourcalltoaction(ifyouhaveone)SpendsometimeconsideringwhyyouareinterestedintellingyourstoryDoyousimplywanttoupdatethepubliconyourprogressorcelebrateyoursuccessesAreyouinterestedinyouraudiencechangingitsbehaviororsupportingaspecificinitiativebeinglaunchedbyyourorganizationIfyouwanttohaveyouraudienceactorbehaveinacertainwaybespecificandincorporateitinyourmessaging
Shareyourstoryandsuccess
52
3 SelectperformancemeasuresThinkabouthowyouwoulddetermineifyourcampaignissuccessfulandselectappropriateperformancemeasuresYouwillrevisitthesemeasuresattheendofyourcampaigncommunicationsplanimplementationtodeterminewhetheryouneedtomodifyyourcommunicationsandoutreachstrategyforthefuture
4 MethodstocommunicateyourstoryTherearemanywaystocommunicatesuccessofyourprojectHerearesomesuggestionsonreachingaudiencesItisimportanttodeterminewhichmethodsarethemosteffectiveforyourdesiredaudiencetoreceiveinformationDoyourresearch(PewResearchCenterhasexcellentstudiesonthereachofmediadigitalmediaandmarketing)ForexampleifseniorsareyourtargetpopulationandtheyarelessinclinedtouseInstagram(thatistrue)youmaywanttoconsiderFacebookorthenightlylocalnewsbothofwhichareusedmoreoftenbyseniorsBelowisalistofsomemethodsandtoolsyoucanuse
i Pressreleasestolocalnewspapersandtvstations
ii ShortsmartphonevideosAgreatwaytoengageanaudienceisthroughvideoAskonequestionofsomeonewhoispartofthestoryMaybethereisasocialworkerwhocandiscusshungerinyourcityThesevideoscanbereusedonmultipleplatforms
iii Long-formwriting
iv TellingindividualstoriesthroughaseriesofphotosandshortnarrativesforpublicationonFacebookorInstagram
v Op-edsopinionpiecesplacedinthelocalpaper
vi Emailupdatestoyourlist
vii Connectingwithappropriateorganizationsthatareinvestedinthetopicandworkwiththecommunityandaskthemtodisseminateinformationtotheirmembers
viii EmployingldquoWordofMouthrdquo(WOM)tacticsbycreatingtownhallsorfreeeventsforresidentstoattendintheirownneighborhoods
5 DeterminethetypeofcollateralUsuallycampaignsthattargetresidentsrequireseveralavenuestoreachthemPeoplearenotalwaysinthesameplaceorspaceHavingakeenunderstandingofyouraudiencewillassistyouwithyourchoicesonwhatneedstobedevelopedSometimesthatmeansimagesforsocialmediaaonepageflierthatgivespeopleinformationonattendinganeventoranin-depthbrochure
Shareyourstoryandsuccess
53
6 ImplementyourstrategyKeepinmindstrategiescanbefluidSometimessomethingmaynotturnoutasexpectedevenifduringplanningsessionsitwasldquosupposedtordquoItisokYoucananalyzewhathappenedduringadebriefsessionandonceyoursurveysandothermeasurementshavebeenfinalized
Visualizations
Withtherightvisualizationyoucanreviewandshareinformationinsightsanalysisperformanceandtrendsallofwhichisvaluabletoacross-sectionofthepublicButwhichvisualizationisrightforyouTochoosetherighttypeofchartgraphdeterminethetypeofquestionyouwanttoanswerSomecommoncharttypesarebelowandmoreinformationcanbefoundhere
Exploreadditionalanalyticaltechniques
Duringthegoal-settingphaseyouestablishedoutputoutcomeandimplementationmeasuresBasedontheresultsofyourpilotyoumayneedtoadjustyourmeasuresWiththepilotdatainhandyoumayhavesomeideasofhowfutureanalysescouldbedifferentandpotentiallyimprovethequalityofyourprogramTakesometimetobrainstormotherrelevantdatasetsthatyoudidnothavethetimetopursueearlierincludingmessydepartmentaldatathatneedsmoreattentionorsourcesofpublicdatafromthestateandfederalgovernmentnonprofitsandevenprivatecompaniesthatcouldsupplementyourunderstandingofanissueUsingmachinelearningtechniquesyoumightbeabletobetter
Shareyourstoryandsuccess
54
predictdemandforhomelessservicesusingNOAAweatherforecastspreemptwatermainbreaksusingsoilstratigraphyorinvestineconomicdevelopmentbymonitoringrealestatelistingdata
Advancedanalyticaltechniquescanalsohelpyourimplementationrunmoreeffectivelybyintegratingdata-informedtargetinganddecisiontoolsintotheprocessByautomatingroutineworkflowsandsurfacingsubtletrendsthesetoolscanhelpyouiterateinamoreprecisemannerandprovidekeystrategicguidancetofront-lineemployeesthatwouldotherwisebeunavailableThesesortsofdata-science-enhancedprocessescanbeclassifiedasoneofsixprojecttypesasexploredfurtherbytheNewOrleansLANOLAlyticsteamFormoreguidanceonscopingasimilarprojecttheGovExadvancedanalyticsscopingguidegivesyouastep-by-stepoverviewofhowtobesuccessful
Usethespacebelowtolistyourideasforpossibleadvancedanalyticsprojectsbytype
Typeofproject Ideas
Findingtheneedleinthehaystack
Prioritizingforimpact
Earlywarningtools
Betterquickerdecisions
Optimizingresourceallocation
Experimentingforwhatworks
Keepthefeedbackloopgoing
GatherinformationonpublicsentimentaboutpolicychangethroughfocusgroupssurveysandpublicmeetingsledbyadvisorygroupmembersNotewhethernewinformationisavailablethatcanhelpanalyzeorcontextualizeyourprogress(egfederalorstatedatareports)Sharetheresultsofyourpilotandthepolicychangeprocesswithothercitiesandresearchers
Shareyourstoryandsuccess
55
WorkingwitharangeofevidencePresentlythereisalotofattentionandinvestmentinevidence-basedpolicymakingincludingthefederalbipartisanCommissiononEvidence-BasedPolicymakingthePewCharitableTrustsreportsandtoolkitsonHowStatesEngageinEvidence-BasedPolicymakingandtheLauraandJohnArnoldFoundationsupportedEvidence-BasedPolicymakingCollaborativeManyoftheseinitiativesfocusonrigorousevaluation--meaningrandomizedcontrolledtrials(RCT)orotherexperimentalmodels--asthebestkindofevidencetouseHoweverthisbodyofevidenceisofteninsufficientforthespecifickindsofquestionsthatgovernmentscanhavewhenmakingadecisionaboutpolicyInthisguideweproposeanapproachtosupportgovernmentsinincorporatingadditionaltypesofevidenceintotheirdecisionmaking
OnewaytounderstandarangeofevidenceistouseNestarsquosStandardsofEvidencetoframeanddefineevidenceTheBritishnonprofitNestaworkstoinjectpositivechangeintothewayUKgovernmentpoliciesandprogramsfunctionOneoftheirkeyareasofworkhasbeendevelopingWhatWorksCentresorganizationsthatcollectcommissionsynthesizeanddisseminateevidenceaboutinterventioneffectivenessacrossarangeofsocialimpactareasNestadevelopedthefollowingStandardsofEvidencetoprovideausefulwaytodocumenthowdifferentkindsofinformationdemonstratedifferentlevelsofevidence
Workingwitharangeofevidence
56
Theimportanceofpotentiallybiasedinformation
Onechallengepresentedbyanevidenceclassificationthatacknowledgesthevalueofnon-scientificformsofevidenceistofigureouthowtobalanceallofthepiecestogetherwhenyouusethemTounderstandhowtodothiswehavetothinkabouthowtopositionpotentiallybiasedinformationindecisionmaking
WhenwelookatinformationwhichhasnotbeenrigorouslyevaluatedweacceptthatthisinformationmightbebiasedThewholereasonweconductrigorousevaluationistoremovebiasfromourobservationsWeconductstructuredtrialstoproduceresultsthatareconsistentandgeneralizableIfafindingisunbiasedthisaccuracymdashthislackofbiasmdashcanbeprovedbecauseanyindependenttestercanreproduceitwhentheyfollowanidenticalprocess
YoucanlookatpotentiallybiasedinformationasjustinformationthatisearlieroninitspathtorefinementreplicationandverifiedaccuracyApieceofinformationbasedonanyoneindividualrsquosobservationsislikelytobebiasedbecauseitiscomingfromjustonesourceBiasisreducedbygatheringtogetherlotsofindependentsourcesofinformationandfiguringoutitsmostcommonattributes
MaybethebiasedinformationmakesaclaimthatistoolargemdashitsayssomethingistrueunderallconditionswhenitisreallyonlytrueundercertainconditionsorthataneffectofaninterventionisbiggerthanitstrueaverageorthataneffectlastsovertimewhenitdoesnotPerhapschangingtheclaimtobemorespecificmdashcriticallyreviewingtheinformationtoseeifspecificconditionsmustbemettoachievetheoutcomesmdashwouldmaketheclaimmoreaccurate
IfyoutestedyourclaimfounditdidnotholdupinsomecaseschangedittesteditagainuntilyourclaimrepeatedlyheldupthenyoucoulddemonstratethattheclaimwasnolongerbiasedbutaccurateThisprocessoftestingimprovingtestingandvalidatingiswhatwegetfromstructuredscientificstudy
Withindividualpiecesofinformation--crediblehypothesescasestudiesexpertopinion--youcannotassumethatkindofaccuracyAtthesametimetheseearlyobservationsarefoundationaltoanyreplicablescientificobservationsWewillnevergetanywhereifwedonotstartwiththesepiecesofinformationeveniftheymightbebiased
Combininglevelsofevidencetogetthemostdetailedinformationforyourcity
ItisespeciallyimportanttoacceptdifferentformsofevidencewhenyoufacequestionsthathavenotyetbeentestedbyindependentstudiesSomeofthemostimportantquestionsrelatetothingsthatareveryspecifictoyoursituationYoufaceaparticularcombinationof
Workingwitharangeofevidence
57
issuesactorstrendsandresourcesthatisultimatelydifferentfromeveryotherplaceintheworld
ThisisimportantbecauseaspolicypractitionersandnotscientistsyoucaremoreaboutyourspecificcasethanahypotheticalaveragegeneralcaseYoucarelessaboutthetotallikelyrangeofimpactsfromapolicythanhowthissingleimplementationwillworkYouwantinformationaboutpoliciesthatwillprovideyouwiththebestpossiblepredictionabouthowtheywillworkinyourcontext
ByputtingdifferentkindsofevidenceintothesameframeworkNestarsquosStandardsofEvidenceimplythatwhenyoucannotgetrigorouslytestedexperimentallyverifiedinformationitisappropriatetoworkwithwhatwedohaveThisisrelevantforthemunicipalimplementationcontextbecausenostudypulledfromaliteraturereviewwillaccountforallofthevariablesyoufaceinyourowncontext
Intheabsenceofhavingastudythatpredictshowaninterventionworksinyourspecificcontextyouwillneednon-scientificinformationtotrytofillinthosegapsYouwillneedtogatherstoriesfromresidentsstaffmembersandotherstakeholdersthatsitonLevels1or2ofthestandardsofevidence
Someofthekindsofinformationyouwillprobablygetmoreeasilyfrompeoplethanstudiesinclude
howpeopleincitieslikeyoursreactedtoaninterventioninpublicforumshowpeoplewhoimplementedaninterventioninacitylikeyoursfiguredouthowtogetitpassedhowthecitycommunicatedpubliclyabouttheapparentbenefitsanddrawbacksoftheinterventionthepoliticalacceptabilityoftheintervention
Youcanremainsimultaneouslyawarethattheinformationyouaregatheringthroughindividualstoriesisbiased--notgeneralizableinformedbyspecificperspectives--whilealsorecognizingthathavingsomeinformationaboutsomethingimportantisbetterthannoinformationEvidencefromthefirsttwoevidencelevelsareessentialbuildingblocksofknowledgeandperfectlyusefulwhenyoucannotgetdetailedunbiasedknowledge
Youhavetostartsomewhereandyoushouldalwaysstartwithevidenceyouhaveaboutyourveryspecificconditionsthatyouknowareimportantforsuccess
Workingwitharangeofevidence
58
Thankyou
Interviewswithcities
GovExheldsemi-structuredinterviewswithsixstaffmembersinfourcitiestoestablishtheinitialstepsoftheRoadmapWeaskedrespondentstotalkaboutaninitiativeorpolicythattheyhavebeenresponsibleforimplementingintheircityRespondentsfromthreecitiesspokeabouttheireffortstoaddresshomelessnessandrespondentsfromonecityspokeaboutblightAfterrespondentsidentifiedtheproblemtheyspokeabouthowthecityprioritizedthisissuethevariousstepsinvolvedandbarrierspresent
GovExconductedtextanalysisoftheseinterviewswhichrevealedthatpeoplewasthemostcommonlyusedtermduringtheseconversationswhendiscussingthestepstochangeaprogramorpolicyThisisathemethroughoutthisRoadmapThefirstsectionbeginswithidentifyingtherelevantpeopleTheconversationsaroundevidencearefocusedonhumanbehaviorratherthanmethodologyorstrengthofevidence
Thankyou
59
OncewehadadraftoftheRoadmapwefollowedupwithcitiesonanindividualbasistolearnmoreaboutwhatwouldbehelpfulWearegratefultothefollowingcitystafffortheinterviewsandindividualconversations
MichaelBaskinChiefPolicyOfficerChattanoogaTN
MelissaBridgesPerformanceandInnovationCoordinatorLittleRockAR
VictoriaCarreoacutenAdministrativeOfficerLasVegasNV
TyroneGrandstrandCommunityDevelopmentPlannerFargoND
TibLaughlinDirectorofGeneralServicesKansasCityKS
TimMorelandDirectorofPerformanceManagementandOpenDataChattanoogaTN
CharlesRosendahlManagementAnalystDentonTX
Thankyou
60
MelissaSiebenAssistantCountyAdministratorKansasCityKS
GinaThompsonDirectorofStrategicServicesLewisvilleTX
JHUFacultycollaboration
GovExalsoconvenedthecitycontactsandfacultypartnersinaseriesofAdvisoryGroupmeetingsThesemeetingsservedtoconnectcitiesandfacultytounderstandoneanothersperspective
StefanBaralBloombergSchoolofPublicHealth
AshleyBernerJHUSchoolofEducation
MelissaDavey-RothwellBloombergSchoolofPublicHealth
JonLinksBloombergSchoolofPublicHealth
TamarMendelsonBloombergSchoolofPublicHealth
KristinMmariBloombergSchoolofPublicHealth
KeshiaPollackPorterBloombergSchoolofPublicHealth
JoshSharfsteinBloombergSchoolofPublicHealth
LizStuartBloombergSchoolofPublicHealth
LindsayThompsonCareyBusinessSchool
DanielWebsterBloombergSchoolofPublicHealth
Partnersindataandevidence-basedpolicy
Wealsospokewithpartnerswhoareexpertsinthefieldofevidence-basedpolicywhosharedtheirknowledgeandresourcesofcurrentandfuturethinkinginevidence-informedthinking
WendyBarrenoAssociateDirectorWilsonSheehanLabforEconomicOpportunitiesUniversityofNotreDame
TiffanyDavenportDirectorofDataPartnershipsBaltimoresPromise
TomiHiersDirectorBaltimoreCivicSiteAnnieECaseyFoundation
ChrisKingsleySeniorAssociateAnnieECaseyFoundation
KathrynOliverAssociateProfessorLondonSchoolofHygieneandTropicalMedicine
Thankyou
61
KristenWatkinsSeniorPolicyAssociateJ-PAL
GovExParticipants
GovExAnalystMiriamMcKinneyanalyzedtheinterviewtranscriptsandconductedtextanalysisSeniorImplementationAdvisorEmilyShawwillintegrateRoadmapactivitieswithadditionalservicedeliveryactivitiesatGovEx
Thankyou
62
ReferencesAbrahamKGampHaskinsR(2017)Thepromiseofevidence-basedpolicymakingReportofthecommissiononevidence-basedpolicymakingWashingtonDCCommissiononEvidence-basedPolicymakingBigmove2Extendingthegreen(nd)Retrievedfromhttpswwwcityoflewisvillecomdoing-businesslewisville-2025big-move-2-extending-the-greenBatesMAampGlennersterR(2017)ThegeneralizabilitypuzzleRetrievedfromhttpsssirorgarticlesentrythe_generalizability_puzzleBuildwithnotfor(2018January)RetrievedfromhttpsbuildwithnotforwordpresscomCairneyPampOliverK(2017)Evidence-basedpolicymakingisnotlikeevidence-basedmedicinesohowfarshouldyougotobridgethedividebetweenevidenceandpolicyHealthResearchPolicyandSystemsdoi101186s12961-017-0192-xCairneyPOliverKampWellsteadA(2016)TobridgethedividebetweenevidenceandpolicyReduceambiguityasmuchasuncertaintyPublicAdministrationReview76(3)399-402Retrievedfromhttpdxdoiorg101111puar12555Centreforpublicimpact(nd)RetrievedfromhttpswwwcentreforpublicimpactorgCernaL(2013)ThenatureofpolicychangeandimplementationAreviewofdifferenttheoreticalapproachesOrganisationforEconomicCo-OperationandDevelopmentRetrievedfromhttpswwwoecdorgeduceriTheNatureofPolicyChangeandImplementationpdfColoradoSprings(nd)Retrievedfromhttpscoloradospringsgovcommunicationspagewhat-expect-when-you-speakupContinuumofcare(coc)program(nd)RetrievedfromhttpswwwhudexchangeinfoprogramscocCouncilpresidentkicksoff$12millionyouthfund(nd)Retrievedfromhttpwwwbaltimorecitycouncilcomcontentcouncil-president-kicks-12-million-youth-fundDurlakJAampDuPreEP(2008)ImplementationmattersAreviewofresearchontheinfluenceofimplementationonprogramoutcomesandthefactorsaffectingimplementationAmericanJournalofCommunityPsychology41327-350Retrievedfromhttpswwwncbinlmnihgovpubmed18322790Fishbonediagram(ishikawa)(nd)Retrievedfromhttpasqorglearn-about-qualitycause-analysis-toolsoverviewfishbonehtmlFivewhysGettingtotherootofaproblemquickly(nd)RetrievedfromhttpswwwmindtoolscompagesarticlenewTMC_5Whtm
References
63
FligsteinNampMcAdamD(2011)TowardageneraltheoryofstrategicactionfieldsSociologicalTheory29(1)1-26RetrievedfromhttpsociologyberkeleyedusitesdefaultfilesfacultyfligsteinFligstein20McAdam20Soc20Theory20PaperpdHeinkelP(2018January12)CitymullspermitsforbikesharingcompaniesRetrievedfromhttpwwwdentonrccomnewsdenton20180112city-mulls-permits-bike-sharing-companiesHowisareamedianincomecalculated(2012October)Retrievedfromhttpswwwhudexchangeinfofaqs843how-is-area-median-income-calculatedLampkinLMampHatryHP(2003)KeystepsinoutcomemanagementUrbanInstituteRetrievedfromhttpswwwurbanorgresearchpublicationkey-steps-outcome-managementMarshDampSmithM(2000)UnderstandingpolicynetworksTowardsadialecticalapproachPoliticalStudies484-21Retrievedfromhttpjournalssagepubcomdoiabs1011111467-924800247McCannL(nd)RetrievedfromhttplaurenellencomMoultonSampSandfortJR(2016)ThestrategicactionfieldframeworkforpolicyimplementationresearchPolicyStudiesJournal45(1)Retrievedfromhttponlinelibrarywileycomstore101111psj12147assetpsj12147pdfv=1ampt=jdkhhcp4amps=e184c625e1855a6b1615ccb99ee27b1c0b3212d8PaynterB(2018January1)What100citiesarelearningfromeachotherbysharingtheirdataRetrievedfromhttpswwwfastcompanycom40518004what-100-cities-are-learning-from-each-other-by-sharing-their-dataPeakacademy(nd)Retrievedfromhttpswwwdenvergovorgcontentdenvergovenmayors-officeprograms-initiativespeak-performancepeak-academyhtmlPinnockHBarwickMCarpenterCREldridgeSGrandesGGriffithsCJRycroft-MaloneJMeissnerPMurrayEPatelASheikhAampTaylorSJC(2017)Standardsforreportingimplementationstudies(stari)ExplanationandelaborationdocumentBMJOpenRetrievedfromhttpbmjopenbmjcomcontentbmjopen74e013318fullpdfProctorESilmereHRaghavanRHovmandPAaronsGBungerAGriffeyRampHensleyM(2011)OutcomesforimplementationresearchConceptualdistinctionsmeasurementchallengesandresearchagendaAdministrationandPolicyinMentalHealth38(2)65-76doi101007s10488-010-0319-7TandonSD(2018)ProgressincommunityhealthpartnershipsResearcheducationandactionRetrievedfromhttpswwwpressjhuedujournalsprogress-community-health-partnerships-research-education-and-actionProvanKGampKenisP(2007)ModesofnetworkgovernanceStructuremanagementandeffectivenessJournalofPublicAdministrationResearchandTheory
References
64
18(2)229-252Retrievedfromhttpsacademicoupcomjpartarticle182229935895SabatierPA(1988)Anadvocacycoalitionframeworkofpolicychangeandtheroleofpolicy-orientedlearningthereinPolicySciences21(3)129-168Retrievedfromhttpwwwjstororgstable4532139SandstromAampCarlssonL(2008)TheperformanceofpolicynetworksTherelationbetweennetworkstructureandnetworkperformancePolicyStudiesJournal36497-524Retrievedfromhttponlinelibrarywileycomdoi101111j1541-0072200800281xabstractSchwartzMampWilsonE(nd)WhocanaffordtoliveinahomeAlookatdatafromthe2006americancommunitysurveyRetrievedfromhttpswwwcensusgovhousingcensuspublicationswho-can-affordpdfShearerJCAbelsonAKouyateBLavisJNampWaltG(2016)WhydopolicieschangeInstitutionsinterestsideasandnetworksinthreecasesofpolicyreformHealthPolicyandPlanning31(9)1200-1211Retrievedfromhttpsdoiorg101093heapolczw052ThelessonsofadministrativedataHigh-profilepolicy-relevantresearchpoweredbyadministrativedata(nd)Retrievedfromhttpswwwpovertyactionlaborgsitesdefaultfilesresources20161114-The-Lessons-of-Administrative-DatapdfToolsandotherneedsforqualitativeandquantitativemethods(2015February)AgencyforHealthcareResearchandQualityRetrievedfromhttpwwwahrqgovprofessionalsquality-patient-safetytalkingqualityassesstoolshtml
WeaverRCampBagchi-SenS(2013)SpatialanalysisofurbandeclineThegeographyofblightAppliedGeography4061-70
References
65
UniversityPartners
Name Title Affiliation Area(s)ofexpertise
Contactinformation
Examplesofworkingwithnon-academicdecisionmakers
WendyPBarreno
AssociateDirector
WilsonSheehanLabforEconomicOpportunitiesUniversityofNotreDame
LEOsfocusareasincludehealtheducationhousingchildwelfareeconomicopportunitycriminaljustice
wbarrenonduduorleondedu
MelissaDavey-RothwellPhD
AssociateScientist
JohnsHopkinsBloombergSchoolofPublicHealth
Health mdavey1jhuedu410-614-5854
BaltimoreCityHealthDepartmentLeadersMarylandDepartmentofHealthAddictionRecoveryServicesinDCExecutiveDirectorsofcommunityorganizationsacrosstheUSthatfocusonHIVSTIsubstanceabuseandimplementationscience
ElizabethStuart
AssociateDeanforEducationandProfessorDepartmentsofMentalHealthBiostatisticsandHealthPolicyand
JohnsHopkinsBloombergSchoolofPublicHealth
Educationsocialpolicystatisticalmethods
estuartjhuedu
Resources
66
Management
Design-ThinkingResources
EngagementGuideforCivicTech
Journeymap
PerformanceManagementResources
DenverPeakAcademy
Proxymeasuresguide
Talkingaboutperformancewithoutcomplicatingit(video)
SettingPerformanceTargetsguide
Datavisualizationguide
Performancemanagementgettingstartedguide
Exercises
Fishboneexercise
5Whys
AdministrativeData
TheLessonsofAdministrativeData(JPAL)
EvaluationResources
Quantitativeandqualitativemethods(USDepartmentofHealthandHumanServicesAgencyforHealthcareResearchandQuality)
KeyStepsinOutcomeManagement(UrbanInstitute)
RCT-YES
Sageresearchmethods
BestPracticesforMixedMethodsResearchintheHealthSciences
ACCORDChecklist
Pilotandimplementationplanning
Benchmarkingexamplesandproxymeasures
Seattleopendataplaybook
Resources
67
Datasciencescopingguide
Resources
68
AboutGovExGovExhelpsgovernmentsusedataandevidencetoimproveresidentsrsquolivesWesupportgovernmentsinbuildingtheirinternalcapacitytooperatesustainableanalyticspracticeswhichimprovecommunityoutcomesinfieldsfrompublicsafetytoeconomicdevelopmentGovExhasaproventrackrecordofworkinginconcertwithpartnerstoprovidetechnicalassistanceandtrainingatscale
SinceourlaunchinApril2015GovExhas
Trainedover2000governmentemployeesinpersonandonlineintopicsincludingdatamanagementperformancemanagementopendatacommunityengagementandanalytics
Providedtechnicalassistancetomorethan100localgovernmentsacrosstheglobeimprovingtheiruseofdatatoachieveefficienciesinservicedeliveryandbetteroutcomesforresidents
Assessedthecapacityofmorethan130governmentalorganizationstodeterminehowleadersstaffandexternalstakeholdersmanageandusedata
Localgovernmentsareincreasinglyonthefrontlinesofaddressinglargescaleproblemswithglobalimplicationssuchasclimatechangepublicsafetyandeconomicinequality
Whileevidence-basedinterventionstohelpsolvethoseproblemsareavailablelocalgovernmentstaffhavelimitedtimeandcapacitytofindandimplementresearch-basedsolutionsAsaresultcitiescontinuetoworkthewaytheyalwayshave--makingincrementalimprovementstoprogramsbutnotalwayscreatingchangeatasystem-level
ThroughresourcedevelopmentandcollaborationwithJHUfacultyGovExissupportingcityleadersinusingdataandevidencetomakedecisionsandcreatinglastingrelationshipswithresearchers
TheRoadmapwasdevelopedbyGovExaspartofitspartnershipwithBloombergPhilanthropiesWhatWorksCitiesinitiative
AboutGovEx
69
- Title page
- Introduction and purpose of the Roadmap
- Using the Roadmap
-
- Gather information about the problem
- Strategically engage key partners
- Review the evidence and collect details
- Decide on a policy to implement based on local factors
- Establish a framework to monitor progress
- Assess data accessibility quality and security
- Design implementation and evaluation
- Share your story and success
-
- Working with a range of evidence
- Thank you
- References
- Resources
- About GovEx
-
DonotforgettocontacttheadvisorygroupmemberswhowerechosenfortheirsubjectmatterexpertiseInadditiontoprovidinggoodmaterialthroughinterviewsthemselvestheymayhaveideasaboutothersubjectmatterexpertsyoushouldcontacttoaskaboutpotentialinterventions
Peercitiesandnetworks
InadditiontoreadingabouttheworkofothercitiesitisidealifyouareabletospeakwithpeoplewhoareinvolvedwithcurrentimplementationsofinterestinginterventionsInyourcityyoulikelyhaverelationshipswithyourneighborsbutitcanbehelpfultolookoutsideofyourregionforsolutionsaswellTofindcounterpartsinothercitieswhoareworkingonaprogramyouwouldliketolearnmoreaboutyoucancontactstaffinintermediaryorganizations(likeGovExoryourprofessionalassociations)whomightbeinapositiontoputyouintouchimmediatelyorwhomightseeanothergoodrouteforsettingupaconversationHereisalistofnetworksthatmaybehelpful
InternationalCityCountyManagementAssociation(ICMA)NationalAssociationofCounties(NACO)NationalLeagueofCitiesUSConferenceofMayorsWhatWorksCities
Collectthedetails
Onceyourreviewershavefoundevidenceonpossiblepoliciesorprogramstoimplementtheyshouldcapturekeydetailsthatwillhelpdecisionmakersunderstandthepotentialeffectivenessandapplicabilityofeachpotentialintervention
Foreachsourcethathasprovidedinformationyouwanttoconveyusethetemplatebelowtocaptureasmanyofthefollowingdetailsaspossible(Youdonotneedtocompletethisforsourcesthatprovideduplicativeinformationorareotherwisenotthatuseful)
Programorpolicyname (egldquoHousingFirstrdquoorldquoblightedpropertiesregistryrdquo)
Sourcetype (egjournalarticleinterviewsubjectmatterexpertstudyassociationcommunityparticipationprocess)
Sourcedetails (egarticletitleandpublicationnameandtitleofinterviewee)
Evaluationmethodology
(egrandomizedevaluationcomparativeevaluationoftwodifferentgroupsbeforeafteranalysis)
Evidencelevel (egLevel12345)
Programorpolicy
Reviewtheevidenceandcollectdetails
27
description
Programorpolicygoal (1sentencecapturingtheprincipalaimoftheintervention)
Existingimplementations
(WherehastheinterventionbeentriedListasmanysitesasyouareawareof)
Characteristicsofimplementationsites
(ForeachimplementationsiteofferbasicgeographicIDmdashcityandstateThenyouwillwanttocollectthedetailswhicharerelevanttoyourdecisionmakersrsquoconsiderationsaboutwhatmakesalocationldquosimilartordquoorldquodifferentfromrdquoyoursituationYouwillatleastwanttoincludeforeachimplementationsitePopulationnumberPopulationdensity(ruralurbansuburban)Socio-economiccharacteristicsBeforestartingaskyourdecisionmakersaboutanyparticularcharacteristicsofprogramimplementationsitestheywouldwanttoknowbeforeseekingtoreplicate)
Characteristicsofimplementationpartners
(Anyavailabledetailsaboutorganizationsinstitutionsorpeoplewhowereinvolvedwithimplementingtheintervention-egindependentchildcarecenterpublicelementaryschoolcitypublichealthnurses)
Characteristicsoftargetpopulationeligibleprogramparticipants
(Anyavailabledetailsaboutpeoplewhoarethebeneficiariesoftheprogramorotherwiseincludedoraffectedbytheintervention-egnumberofprogramparticipantscategoriesofpropertyownersaffected)
Policyorprogramduration
(HowlonghastheinterventionbeeninplaceineachimplementationsiteHasitendedandifsowhy)
Whatbenefitscamefromeachimplementation
(Anyobservedpositiveeffectsoftheinterventionincludingeffectsize)
Statisticalestimatesofinterventioneffectiveness
(Anyavailablestatisticalestimatesoftheinterventionrsquoseffects)
Whatdrawbackswereobserved
(Anyavailabledetailsaboutunintendedorunavoidablenegativeeffectsoftheintervention)
Programorpolicycosts
(Anyavailabledetailsaboutprogramcostsperimplementation)
Reviewtheevidenceandcollectdetails
28
Reviewtheevidenceandcollectdetails
29
DecideonapolicytoimplementbasedonlocalfactorsTogetthecriticalbuy-infromleadershiptoimplementanewpolicyorprogramtheprojectleadshouldscheduleameetingwithdecisionmakerstopresenttheevidencesurroundingthepolicyoptionscapturedinthetemplateDependingonyourinsightintoyourcityleadershipswillingnesstomakechangeyoumaywishtosharetheresearchyouhaveconductedontherootcausesoftheproblemandsomeofyouranalysisontheimpactofcurrenteffortstodate
Considertheapplicabilityofanapproach
AsdiscussedinthepreviouschapterpolicymakersoftenlookforevidenceofeffectiveprogramsinothercitiesbutoftenthequestionarisesofhowgeneralizableaprogramisfromonecitytoanotherTheAbdulLatifJameelPovertyActionLab(J-PAL)suggeststhatinreviewingevidencefromanotherlocationitishelpfultoidentifythespecificmechanismthatwasused(deliveryofprogramorserviceinformationsharingetc)identifythelocalconditionsthatwereinplaceobservethebehaviorchangethatresultedfromthemechanismandfinallytounderstandlocalcontextandconditions(legalorcostconstraintsculturalnormsetc)Thissetofpresentationelementscanbedrawnfromthelistofdetailscollectedintheresearchtemplate
EducationpolicymakersinChicagoILusedfindingsfromprogramsinKenyaandIndiainwhichchildrenwhohadfallenbehindtheirpeersweregiveneducationalsupporttargetedtotheirleveloflearningThemechanismusedwastargetedteachingandthereweresimilarconditionspresentinallthreecountriesAlthoughsomestudentshadfallenbehindinthecurriculumteacherswereincentivizedtoteachgrade-levelmaterialsConsiderationsofhowthelocalcontextinChicagomightaffectimplementationofatargetedteachingprogramincludedunderstandinghowtheneighborhoodsthatthestudentswerefrommayhaveaffectedtheireducationaloutcomesandthehighcostofteachinginsmallgroupsInChicagoMatchEducationprovidedtwo-on-onetutoringwhichmatchedstudentswithwell-educatedindividualswhohavenothadformalteachertrainingforalower-costmodelTheresultswerepositiveStudentswhoweretutoredperformedbetterinmathandweremorelikelytopasshighschoolclasses
ThegeneralizabilityframeworkproposedbyJ-PALisbelowUsethisasatemplateasyouconsiderpotentialpolicysolutions
Decideonapolicytoimplementbasedonlocalfactors
30
SourceTheGeneralizabilityPuzzle
OnceyouhavenarrowedthepolicyoptionsdiscussthequestionsbelowforeachoptionsyouareconsideringThisstepiscriticalasunderstandinglocalcontextissoimportantinsuccessfulpolicyimplementation
CommunityContextQuestionnaire
Localcontext
DoesthisoptionmakesenseinourcommunityIsthisaonesizefitsallapproachIsanyparticulargroupabsentfromthisapproachIfsoisthatok
Youranswers
Effectiveness
WhatwouldweexpecttoobserveifthepilotisaseffectivehereasinCityX
Youranswers
Timing
DoesthetimingmakesenseIsthistherighttimetoimplementthispolicychange
Decideonapolicytoimplementbasedonlocalfactors
31
WherearethepotentialroadblocksthatcouldaffectthescheduleIsthereanywaytomitigatethisHowlongwouldittaketoseeanimpact
Youranswers
Implementation
DowehavethestaffcapacitytoimplementthisArethereadditionalpartnersthatyoucanbringtothetabletosupportimplementationDoesthismakesensegivenourformofgovernment
Youranswers
Authority
DoesourcityhavetheauthoritytoenactthisIfsowhatistheappropriatemechanismAretherelegalbarriersArethereanyexistingpreemptionissuesfromthestateorfederalgovernmentCanthosebeaddressedthroughlegislationadvocacyIstherepoliticalsupportoroppositiontothis
Youranswers
Cost
HowmuchwillthiscostannuallyHowwillwepayforitCanthepolicybeimplementedwithexistingfunding
IfsoisthisfundingreliableHowwouldthispolicyoptionaffectourcitysbudgetintheshortandlongtermWhatotherfundingoptionscanweconsiderIsthereanalternativethatwouldcostlessandgiveussimilarresultsWillanyconstituencybeadverselyaffectediffundingforprogramsthatimpactthemisshiftedtoanotherprogrampolicyorinitiative
Youranswers
Decideonapolicytoimplementbasedonlocalfactors
32
EstablishaframeworktomonitorprogressNowthatyouhavereviewedtheevidenceconsideredlocalcontextandchosenyourpolicyitistimetodeterminehowyourcitywillmonitorprogressofthisnewapproachCitystaffandleadersneedtoestablishtheinfrastructureforongoingcontinuousdecisionmakingbasedonwhetherthepolicychangeisleadingtothedesiredoutcomesforresidentsandwhetherthechangeismeaningfulandrationaltofrontlineworkersSettingarbitrarygoalsatthelocallevelthatconflictwithgoalsatotherlevelsisconfusingtotheindividualsresponsibleforachievingthegoalsGoalsthataresettoohightoachievecanbedemoralizingtostaff
ThissectionincludesstepstosetupavenueinwhichCitystaffandleadershiproutinelyvisitkeyindicatorstodeterminewhatisorisnotworkingandtomakedecisionsinreal-timeFormorereadingonperformancemanagementpleasevisittheGovExPerformanceManagementGettingStartedGuide
InperformancemanagementtermssuchasindicatormeasureandmetricmaybeusedinterchangeablyInthisguidewewilluseindicatorInyourworkthemostimportantthingistobeconsistentinwhichevertermyouuseThisvideomaybehelpfulinunderstandingthisconceptandcommunicatingittoothers
Establishthefundamentalsandgatheryourteam
FirstdeterminewhetherthereisanestablishedvenueorregularmeetingscheduleduringwhichyoucandiscussprogressIfyourcityhasanexistingperformancemanagementstructureonarelatedtopicaddingadiscussionoftheprogressofpolicychangetoyouragendacanbeagreatideaIfthiswillbeastandalonemeetingdecidehowfrequentlytheprojectteamanddecisionmakerswillmeetandputthosemeetingsonthecalendarfortheentirelengthoftheproject
DeterminewhowillstaffandattendroutinemeetingsonCityprogressThesemeetingsmayincludetheMayorandorCityManagerdepartmentheadstaffwhoareimplementingthepolicychangeandanalysts
Identifyindicators
ItiscriticalthatindicatorsbeanimportantcityandcommunitypriorityaswellassomethingtheCityplaysamajorroleinaddressingTheremaybeinstancesinwhichtheCitywillneedtoreachouttoorcollaboratewithexternalpartnerstogetamorecompletepictureofprogressontheoverallpolicychange
Establishaframeworktomonitorprogress
33
Anoutcomeindicatordescribesthedesiredeffectofthetargetedpopulation
ExampleBellevuesfamilyfriendlyneighborhoodsindicatormeasuresthepercentofresidentswhobelievetheyliveinaneighborhoodthatsupportsfamiliesparticularlythosewithchildren
WithyourkeypartnersrevisityourproblemstatementcurrentstateandfuturestateexerciseandbegintoidentifytheoutcomeindicatorsthatwillillustratesuccessFornowchooseasingletop-levelkeyperformanceindicator(KPI)ItiscriticalthattheKPI-thepieceyouwillmeasure-beapriorityforthecommunityandsomethingtheCityplaysamajorroleinaddressing
Outputindicatorsmeasuretheprocessofprogramorservicedelivery
ExampleSanDiegoMayorKevinFaulconersgoaltorepair1000milesofstreetby2020
ReviewingtheadministrativedatasetinventorybegintoparedowntherelevantprogramsanddatasetsthatsupportyourKPIWhatarethemeasuresofeffectivenessforeachprogramTheseareyouroutputindicators
Thediagrambelowillustratestherelationshipbetweenoutputandoutcomeindicators
ThinkaboutimplementationindicatorsasmeasuringstepsintheprogramsrolloutthatarenecessarytoobtaindesiredchangesImplementationoutcomescanfocusontheinterventionstaffdeliveringtheinterventionanorganizationorsystemandcanaddressfeasibilityadaptabilitysustainabilityandcostTwokeyquestionstoconsiderwhendevelopingimplementationindicatorsare
1 Istheinterventionacceptableandappropriatetothecommunityand
2 IsitfeasibletoimplementinterventionXatthistimewiththeseresources
Establishaframeworktomonitorprogress
34
Amoreextensivelistwithdefinitionsofpotentialimplementationoutcomesislinkedhere
UsethetablebelowtochoosethedesiredimplementationoutcomesforyourpilotandtoplanyourmeasurementtechniqueThisshouldincludelevelofanalysis(individualproviderorconsumerorganizationorsetting)yourmeasurementtechnique(surveysobservationinterviewfocusgroupsadministrativedataanalysis)andthephasewhenyouwillcollectthisinformation(earlymidlate)Theremaybemorethanonelevelofanalysismeasurementtechniqueandphaseofdatacollection
Implementationoutcome
Definitionofoutcome
Levelofanalysisandmeasurement
Yourmeasurementtechnique
Phaseofpilotwhenyouwillcollectdata
Usethespacebelowtocaptureyourindicatorwishlist
Indicatorwishlist
Outcomes_____________________________________________________________________________
Outputs________________________________________________________________________________
Implementationoutcomes______________________________________________________________
Setapublichigh-levelgoal
Thehigh-levelgoaliscriticalinestablishingdirectiontogalvanizethepeoplewhowillbedoingtheworkWhendiscussingthegoalusetheMayorsbullypulpitLookforkeymilestonesinCitygovernmenttosharethegoalstatementsuchasthebeginningofaMayoraladministrationaStateoftheCityspeechorotherpublicremarks
GoalsshouldbeldquoSMARTrdquo
Specificaspossible(innamingthepopulationwhowillbebetterofftheactionthecitywilltake)
Measurable
Achievable
Establishaframeworktomonitorprogress
35
Results-focused
Time-bound
ThegoalstatementshouldincludeyouroutcomeindicatorlikeChattanoogasgoalldquoWewillendchronicveteranhomelessnessbyDecember2016rdquoAdiagramtohepyoubuildaSMARTgoalaroundyouroutcomeindicatorisbelow
Component Description Yourturn
Specific WhowillbebetteroffWhatactionthecitywilltake
Measurable Howwillyoumeasuresuccess
Achievable Isthegoalattainable
Results-focused Whatistheexpectedresult
Time-bound When
Setrealisticambitioustargetsandtimelinesforeachindicator
ToestablisharealisticandambitioustargetforyourgoalstatementusetheadministrativedatacollectedforyourchosenKPItoreviewactualCityperformanceinthemostrecentyearcomparedwiththethreeorfourmostrecentyearsworthofdataBegintocalculatesummarystatistics(egtotalminimumaveragemedianmaximum)toidentifytrendsCreatecumulativeandmonthlychartstoidentifytrendsandanomaliesBasedonyouranalysisofhistoricaldataignoretheanomaliesandassumetheproductivitytrendswillfollowthesameoverallpatternfrommonth-to-monthFinallyapplyanincreasedproductivityfactortotheperformanceeachmonthtoforecastfutureprogressBasedonthisanalysischooseatargetthatinspiresimprovementwhileaccountingforreality
Manycitiesthataremorematureintheirperformancemanagementpracticesrecognizetheimportanceoftakingacommunity-levelapproachtoimprovingoutcomesforresidentsThesecitiesarecreatinggoaltaxonomiesthatincludemeasuresrelatedtopublicsafetyhousingandevenrecreationCitiesdriveprogressonthesegoalsbyimplementingstrategiesthatengagemultipledepartmentsandcollaboratewithserviceprovidersinthecommunity
EventuallyyouwilldevelopgoalstatementsfortheoutputsinordertocapturetheefforttoachievetheoutcomecontainedinyourgoalstatementYoursetofindicatorsshouldbecomposedofshort-andinterim-termsupportivemeasureswithaccuratetargetsandrelevanttimelinesAnexamplegoaltaxonomyfromtheCityofLasVegassResultsVegasprogramisbelowUsethistaxonomyasamodelforyourownsuiteofindicatorsandgoalstatements
Establishaframeworktomonitorprogress
36
Yourturn
Thetemplatebelowismeanttohelpyoucraftagoalstatementforeachsupportingmeasure
Goalstatement
Departments
Outcome
Activities
KPI
Target
Timeframe
Conductanddiscussrelevantanalysis
ThrougharoutineseriesofmeetingscontinuallymonitorprogressofKPIssupportingindicatorsandanyotherdatacollectedsuchasqualitativedatatoidentifytrendsthatwillindicatewhetherthechosenpolicyishavingthedesiredimpactThesedatapointsshoulddrivethediscussionduringthesemeetingspointingtoopportunitiesforfurtherinquiryinto
Establishaframeworktomonitorprogress
37
whatisdrivingtheobservedtrendsAnalystsshouldbeencouragedandtrainedtoaskquestionsofthedepartmentsproducingthedataandconductsitevisitswhenpossibletodeepentheirunderstandingofthedataPresentationoftheanalysisshouldfocusononeideaatatimetokeeptheconversationfocusedonkeypointsandspecificopportunitiesforimprovementYoumayconsiderdevelopingapublicfacingdashboardtosharethisprogressmorebroadly
AgeneralcomparisonofthestatusquodatawithwhatwasobservedduringthepilotcanprovidedecisionmakerswithaquickunderstandingofthedifferencesbetweentheoldandnewapproachesalthoughthiswillnotshowcausalityOftenthesecomparisonswillincludeoperationalmetricssuchasdaysordollarssavedbutrememberthattheimpactonpeoplewillbemostimportantIfyouareworkingwithanevaluationpartnertheywillsupportthisstep
Usethespacebelowtosummarizetheresultsofyouranalysis
MeasureTargetvalue(seesectiononsettingtargetsabove)
Pilotvalue(whatdidyouobserveduringthepilot)
Historicalaverage(ifavailable)
Results(WhoisbetteroffIsanyonenegativelyimpacted1-2sentencesreassessingyourgoalstatement)
DocumentyourprocessandprogressincludingfeedbackfromtheparticipantsimplementationpartnersandtheprojectteamDidanythingunexpectedoccurduringthepilotArethereinternalandexternalfactorsthatmayaffectyouroutcomesThisiswheretheimplementationoutcomescanbecanhelpfulIftheinterventiondidnotgoasplannedconsiderwhetheritisbecausetheinterventionwasnotacceptableorfeasible
UsethetemplatebelowcreatedbyDenversPeakAcademytocapturethisinformationIfthe306090daytimelinedoesnotmakesenseforyourcityfeelfreetocustomizeit
CurrentState
FutureState
30Day
60Day
90day
Whatwentwell(+)
Whatdidnotgowellshouldbechanged(-)
IfaquestionordiscussionpointarisesthatcannotbeansweredinthemeetinganassignedstaffmembershouldcapturethequestionduringthemeetingAfterwardthefollowupitemsshouldbesenttorelevantpartnersandthenextmeetingshouldbeginwithadiscussionof
Establishaframeworktomonitorprogress
38
thoseitems
Takeaction
DiscussionsaboutwhetheryourindicatorsaremovingintherightdirectionandwhatmightbedrivingtrendsmayleadtoadiscoverythattherearenotenoughresourcesallocatedtothispolicyorissueTheremaybelimitedfundingforstaffprogrammingortechnologythatisdrivingthedatainthewrongdirectionIfCityleadershipisintheroomthisisagreattimetomakethecaseforadditionalfundingduringthebudgetseasonBycollectinganalyzinganddiscussingdataonthispolicyroutinelyyouarebuildingevidencetomakethecasefortherightleveloffundingtoreachtheestablishedtargets
Establishaframeworktomonitorprogress
39
Assessdataaccessibilityqualityandsecurity
Determineaccessibilityandqualityofdata
OnceyouhavecompiledyourwishlistofindicatorsitistimetodeterminetheaccessibilityandqualityofthedataTopreventcomplicationyoumaywishtoprioritizerelevantindicatorsforwhichyouarealreadycollectingdataAvoidthetemptationofadoptingatechnologysolutiontocollectnewdataatthispointsuchasaphotosurveytoolorotherappFornowyouaretryingtodeterminewhichdatasetsarerelevantaccessibleandhighqualityBesuretoworkwithyourkeyinternalpartnerswhileyouareidentifyingyourmeasuresDepartmentstaffwhoworkonprogramsarefamiliarwiththeindicatorsthatwillaccuratelysupporttheoutcomesanddatabaseadministratorswillbevaluablewhendiscussingaccessibilityandqualityofthedata
Abrieflistofconsiderationsfortheaccessibilityofyourdataisbelow
ArewecurrentlycollectingthisdatasetorwillwebecollectingitforthefirsttimeHowfarbackdoestherecordgo(timehorizon)WouldweneednewtechnologytocollectoranalyzethedataCanthedatabecollectedmanuallyorcanitbeautomatedHowfrequentlyisthedataproduced(continuouslyhourlydailyweeklymonthlyquarterlyyearlyetc)IsthedataavailablefortheentireCitypopulationorasubsetofthepopulationHowgranularisthedata(individuallevelCensusblockneighborhoodcitywideetc)DoesthedatacontainsensitiveinformationIsthereaproxymeasurethatwecanusefornowIsthedatamachinereadable
AsnotedbytheWilsonSheehanLabforEconomicOpportunities(LEO)attheUniversityofNotreDameleveragingadministrativedatacansignificantlyreducethecostsandtimelinesofrigorousevaluationCitiescanestablishaninternaldatasharingagreementacrossdepartmentsincludinglanguagethatenablesadministrativedatatobeusedforresearchpurposesCitiescanalsoidentifyexternaladministrativedata-holderssuchasacountyschooldistrictpolicedepartmenthospitaloranotherstatethatmayhaveoutcomedataimportantformeasuringprogramimpactCitiescanleverageexistingrelationshipsandorworkwithresearchinstitutionstodevelopstreamlineddatasharingprocessesandagreementswiththeseexternalentities
Assessdataaccessibilityqualityandsecurity
40
Evaluatethequalityofyourdata
QualityisasimportantasaccessibilityToevaluatedataqualityyoumaywishtocreatearubricliketheonebelow
Value ValueDescription
3 ExcellentDataQualityThedataiscompleteaccurateandupdatedregularly
2 AdequateDataQualityThedataissomewhatcompleteaccurateandupdatedfairlyregularly
1 PoorDataQualityThedataisnotcompleteaccurateorupdatedregularly
ThisisagreattimetobuildonthedatainventorythatyoustartedinthegatherinformationpageSuggestedadditionalfieldstoaddtoyourdatainventorytemplatearebelow
Program DatasetTitle
DatasetDescription DataSource
Departmentorexternalpartner
DoestheCityalreadycollectthisdata
Whatisthenameoftheprogram
Human-readablenameofthedataset
Shouldbeunderstandablebynon-technicalusers
Whatistheoriginalsystemapplicationfilethathousesit
Departmentorexternalpartnerthatmaintainsthedata
Isthisdataalreadycollected
Updatefrequency-ThefileformatforthedatasetTypicallycsvxlsxshapefiletxtetc
Format-Howoftenthedatasetisupdated
Timerange-Theperiodoftimecoveredinthisparticulardataset
Datause-Howisthisdatasetcommonlyused
Datausers-Whotypicallyusesthisdataset
Dataownerstitle-Titleofthedataowner
Dataownersposition-DataOwnersjobtitle
Dataownersemailaddress-DataOwnersemailaddressWhomanagesthedataandorisresponsibleforgrantingpermissiontoaccessthedataWhounderstandswhatthedatasetincludesandcananswerquestionsaboutit
Dataqualityconcerns-Describetheconcernsyoumayhaveaboutthequalityofthedataset
Assessdataaccessibilityqualityandsecurity
41
Division-DivisionwherethedataismaintainedIfmorethanonedivisionisresponsiblelisttheprimaryownerofthedataset
Sensitivedatacomments-DescribetheissuesthatwouldpreventthecityfromsharingthedatapubliclyCommonsensitivityconcernsincludeprivacyviolationssecurityissuesorhighcostorstaffdemandstopostthedata
Publishingstatus-Thistheplacetoputanycommentsregardingthecurrentstatusofthedataset
Publishedlink-Thisisthelinktothepublisheddatasetontheopendataportalorcitywebsite
Reviewdatasecurityandprivacyprotectionprotocols
OftenapolicychangealsoinvolvesachangeinthedatathatisbeingcollectedhowthatdataisbeinghandledwhohasaccesstoitandahostofotheradjustmentsAsyoutakestockofthedataassetsthatareavailableforyourinitiativeitisvitaltoalsothinkthroughwhatwillbedonetoprotectthatdataandtheprivacyofthepeopleitrepresentsAbigpartofgooddatasecurityinvolvesstrongdatagovernancepracticesthatdefineandidentifysensitivedataandconsciouslyworktobalancethevalueandriskofstoringsuchsensitivedataOnewayofevaluatingyourdatacomesfromcaselawrelatedtotheUSFreedomofInformationActwhichwehavetranslatedtothechecklistbelow
1 Woulddisclosureresultinasubstantialinvasionofprivacy2 Haveweconsideredtheextentorvalueofthepublicinterestandthepurposeorobject
oftheindividualsseekingdisclosure3 Istheinformationavailablefromothersources4 Wastheinformationgivenwithanexpectationofconfidentiality5 Isitpossibletomouldreliefsoastolimittheinvasionofindividualprivacy
TheseriskscouldbemitigatedusingavarietyofstrategiesfromaggregatingdatatoobfuscatingindividualstoredactingfieldsthatmayincludePersonallyIdentifiableInformation(PII)orre-evaluatingyourdatacollectionmethodstofindalternativewaysofgettingtheinformationyouneedThesestepsshouldbecodifiedandmanagedbyadatagovernancecommitteeifpossibleForthesensitivedatathatisdeemedessentialcreateaplanaroundpotentialdatabreachesinordertoquicklyandconfidentlyrespondtoemergencysituations
ProactivelypreparingforthesesituationsisoftenthebeststrategyfordatasecuritybutdonotforgetthetechnicalsideInvolveyourITadministratorsinconversationsaboutdatastorageandaccessensuringthattheywillbeprotectedbyenterprise-gradecybersecuritysystems
Assessdataaccessibilityqualityandsecurity
42
Assessdataaccessibilityqualityandsecurity
43
Designimplementationandevaluation
Designtheimplementation
NowthatyouhaveyouraccountabilityplanandkeyindicatorsitistimetogetspecificaboutyourplanforimplementationThetemplatebelowwillhelpyouthinkthroughthespecificmechanismsofimplementingpolicychange
UnderstandingorganizationsandsystemsiscriticaltounderstandingandmakingdecisionsaboutwhereandhowinterventionsshouldbeappliedApolicyimplementedinasinglesettingmayshowpromisingresultsbutimplementingmultiplepoliciesandprogramsatthesametimewithdifferentactorsresponsibleforeachaspectcanhelpensuresustainablebehaviorchange
InSomervilleMAShapeUpSomerville(SUS)isacitywidepublichealthprogramthatisimplementedacrossmultiplelevelsofinfluenceandsystemsthroughinterventionsbeforeduringandafterschooltoincludechangestobuiltenvironmentsuchasbikelanesregionalmasstransportationandphysicaleducationequipmentinschoolsandgymstoincreasephysicalactivitywellnesspoliciesandschoolgardensnutritioneducationduringandafterschoolandhealthyrestaurantprogramstosupporthealthyfoodandhealthyeatinghabits
Usethetemplatebelowtoplanyourimplementation
Policyimplementationplanningtemplate
ListthespecificdesiredoutcomesInthepreviousstepyouworkedwithdepartmentstafftoestablishspecificmeasurableoutcomesandinterimindicatorsPrioritizetheindicatorsthatwilldemonstrateprogressUsethespacebelowtocapturethedesiredoutcomesofthepolicychange
ExampleShapeupSomervillebeganwiththegoalofpreventingobesityin1stthrough3rdgraders
Yourturn
____________________________________________________________________________
____________________________________________________________________________
Designimplementationandevaluation
44
SpecifythetargetpopulationAreyoudesigninganinterventionthattargetsonlycertainsitesAreyouplanningonimplementingaprogramorinitiativeforaspecifictargetaudienceClearlyspecifythepopulationyouplantotarget
IfyouplantoconductarigorousimpactevaluationasnotedbytheWilsonSheehanLabforEconomicOpportunitiesitisimportanttoestablishconcreteobjectiveeligibilitycriteriaandensurethatthesecriteriaareappliedconsistentlybyallstaffduringscreeningandintakeItisalsovitaltoidentifytheindividualswhoareeligibleforaparticularprogramorserviceevenifeveryoneeligiblecannotbeservedbyexistingresourcesThishelpscitiesidentifyanyunmetneedsandguidefutureresourceallocationwhilealsocreatingthepossibilityofarigorousimpactevaluationbyidentifyingapotentialcomparisongroupFinallywhenconductingscreeningandintakeitisimperativetocollectandmaintainbasicinformation(egnamedateofbirthgender)aboutallprogram-eligibleindividualsnotjustthosewhoparticipateintheprogram
ExampleIntheShapeUpSomervilleexampletheprogrambeganbytargeting1stthrough3rdgraders
Yourturn
____________________________________________________________________________
____________________________________________________________________________
DefinethelocationWherewillthepolicychangetakeplaceWillimplementationtakeplaceatonesiteormultiplesites
ExampleShapeUpSomervilleinterventionsweredeliveredinschools
Yourturn
____________________________________________________________________________
____________________________________________________________________________
ClarifyimplementationpartnersandarticulatetheirrolesWhatarethekeyorganizationsinstitutionsorpeopleresponsibleforimplementationTheprojectteamshouldbeasclearandspecificaspossibleinprovidingadescriptionoftheconcreteactivitiesthatimplementationpartnersaremeanttocarryoutPracticeprofilesmaybeahelpfultooltohelpdefinewhatisrequiredoftheactorswhoareresponsibleforimplementingtheintervention
Designimplementationandevaluation
45
ExampleInSomervillemayoralleadershipandengagementofmultiplecityagencies(publichealthprimarycareschoolsCambridgeHealthAllianceetc)streamlinedandconnectedthevariousShapeUpSomervilleinitiativesandgrantswhichhelpedtokeepthetalentandenergyfocusedonhealthyeatingandactivelivingforallsectorsofSomervilleSUSpartnersultimatelyincludedschoolsnonprofitsCityandstateagenciesandnationalfoundations
Yourturn
____________________________________________________________________________
____________________________________________________________________________
DescribetheinterventionWhatarethestepsactivitiesorprocessesRememberthatthemechanismthespecificstepsorprocessthatleadtobehaviorchangeiskeytounderstandingtheeffectivenessofthepolicychangeandsharingtheresultswithothers
ExampleGoalsofShapeUpSomervilleweretoinfluenceeverypartofanearlyelementaryschoolchildrsquosdayndashbeforeduringandafter-schooltopromotehealthyeatingphysicalactivityandtheirweightstatusInterventionincludedfoodserviceenhancementstoimprovebreakfastandlunchandstaffprofessionaldevelopmentwalktoschoolactivitiesSUSclassroomcurriculumandprofessionaldevelopmentschoolwellnesspolicydevelopmentSUSafter-schoolcurriculumandprofessionaldevelopmentoutreachandeducationtohomethroughmaterialsforumseventsldquoSUSapprovedrdquorestaurantscommunityoutreachandcapacitybuildingthroughpolicydevelopmenttrainingsmediaplacements
Yourturn
____________________________________________________________________________
____________________________________________________________________________
ClarifytimelineandmilestonesfordeliveryandevaluationHowfrequentlywilltheservicesinterventionbedeliveredHowlongwilltheprogramrunWhatisthetimetoexpectedimpacts(eg12monthsafterprogramcompletion)
Designimplementationandevaluation
46
ExampleShapeUpSomervilleledbyTuftsUniversityFriedmanSchoolofNutritionScienceandPolicyconductedaquasi-experimental(nonrandomized)interventionstudyamong1stthrough3rdgradestudentsinSomerville(intervention)andtwocomparisoncommunitiesfor9-monthsduringoneschoolyear(fall2002ndashspring2003)ThestudyshowedpositiveassociationoftheinterventiononchangeinBMISUSreleasedreportssummarizingevaluationsprogresstodateplansforadditionalprojectsandimpactstenandfifteenyearsintotheinitiative
Yourturn
____________________________________________________________________________
____________________________________________________________________________
PlanyouranalysisTheprevioussectionaddressedestablishinganaccountabilityframeworktomonitorprogressandimplementationDeterminewhichindicatorswillbediscussthroughthatprocesswhowillsharethedataandanalysisandhowfeedbackfromthoseconversationswillbeincorporatedbytheprojectteam
ExampleTheMayorrsquosofficeimplementedadata-drivendecisionmakingapproachcalledSomerStatwhichwassynergisticwiththeapproachestakenbytheresearchandevaluationteamsdeployedforthespecificgrants
Yourturn
____________________________________________________________________________
____________________________________________________________________________
PlanforsustainabilityIfthepolicyandprogramchangeissuccessfulhowwilltheprojectteamsustainandexpandtoadditionalsitesorpopulations
ExampleAlthoughmostofthepartnershipsforgedintheearlyyearsweremaintainednewgrantsandcollaborationsexpandedattentiontoolderpublicschoolstudentsactivetransportationandtheGreenLinesubwayextensionthebuiltenvironmentfarmerrsquosmarketeffortssuchasldquoGrowninSomervillerdquowithUnionMainStreeteconomicdevelopmentandhealthdisparitiesaswellasdatacollectionmonitoringandsurveillance
Yourturn
Designimplementationandevaluation
47
____________________________________________________________________________
____________________________________________________________________________
Pilotplanningtemplate
IfyouplantoconductapilotfirstdeterminethepurposeofthepilotDoyouwanttoobservewhetherthepolicychangeresultsinthedesiredbehaviorchangeAreyouconcernedaboutfeasibilityscalabilityorcostDoyouwanttotestwhetheryourdesiredresultscanbeachievedthroughthispolicyapproachClarifythegoalofthepilotandthequestionsyouhopetoanswerThiswillhelpyouidentifythekeyindicatorsthatyouwillusetomeasurewhetherthepilotissuccessfulBeforebeginningestablishabaselineofcurrentperformanceandsetyourtargetsaccordinglybasedonwhatyouhopetoaccomplishthroughthepilot
NextestablishthepilotplanforyourprojectteamincludingmilestonesforteamcheckinstodeterminewhethertomoveaheadorchangecourseThisisseparatefromthemeetingsthatwilloccuraspartoftheaccountabilityframeworkUsethespacebelowtocreateyourtimelineanddevelopyourmeetingschedule
Task Personresponsible Startdate Checkindate Deliverable
Planyourevaluation
WhileyouaredesigningtheimplementationitisalsotimetobegintodevelopyourplanforanalysisAsnotedbytheWilsonSheehanLabforEconomicOpportunitiesdecisionsaboutprogramsetup-includingoutreachscreeningintakedatatrackingandenrollment-candeterminewhetherornotarigorousimpactevaluationwillbepossibleAsdiscussedcityleadersmayfindithelpfultoincludeevaluationpartnersinearlyconversationsaboutprogramsetupDuringpreviousstagesyoumayhaveworkedwitharesearchertoplanyourproposedpolicychangeYoushouldbeclearaboutthepurposeoftheevaluationanddevelopanevaluationplanBelowareadditionalconsiderations
Causationvstrendanalysis
IntheEstablishaframeworktomonitorprogressstagetherewasguidanceontrackingtrendsovertimeItisimportanttonoteherethatmonitoringdataisdifferentfromanalyzingacausaleffectTrendanalysiswillillustratethatchangesareoccurringandmayleadtoquestionsthatcouldbeansweredbyyourevaluationbuttrendanalysisalonedoesnotprovideevidenceaboutwhatisdrivingthosechangesYoumayseechangesinyour
Designimplementationandevaluation
48
outcomeindicatorsbasedonvariablesoutsideoftheCityscontrolForinstanceachangeinfederaltradepolicycanhaveanimpactemploymentratesinyourCityoutsideofyourCitysnewjobtrainingandplacementprogram
ToaddresscausalityyoumustcomparetheoutcomesofindividualswhoreceivedtheinterventionwiththosewhodidnotWithoutacomparisongroupofsimilarindividualswhowerenotexposedtotheinterventionitisimpossibletoknowwhichfactorsareresponsiblefortheoutcomesRandomizedControlTrials(RCTs)inwhichindividualsarerandomlyassignedtotreatmentandcontrolgroupsarethegoldstandardinprovingcausalityIfyoucandemonstratethatoutcomesimprovedforthosewhoreceivedtheinterventionanddidnotimproveatallorimprovedlessforacomparisongroupyoucanbesomewhatcertainthattheinterventionwasthecause
Examplesofothernon-causalimpactevaluationsthatarerigorousandreliableincludecertainquasi-experimentalstudiesusingcomparisongroupdesignssmartinterruptedtimeseriesanalysesthatmodeltrendsovertimeinwaysthatallowestimationofinterventioneffectswelldesignedpre-posttestanalysesneedsassessmentsimplementationstudiesetcFormoreinformationondifferenttypesofevaluationsseetheSageResearchMethodswebsite
Lookfordifferencesingroups
BuildintimetoalsoexaminethepilotdatatoseeifgroupsareexperiencingdifferentresultsReviewingperformancebasedonoutcomemeasuresforapopulationinaggregatemayhidethesubtlewaysthatsomepeoplemaybeaffecteddifferentlysoitisnecessarytocarefullyanalyzethedatafordisparateoutcomesForinstancetheMovingtoOpportunity(MTO)studythatexaminedtheimpactofhousingvouchersonfamilieswholivedinhigh-povertyneighborhoodsfoundthatchildrenwhomovedbeforetheageof13hadbettereducationandhigherincomesasadultsHoweveradultsdidnotseeimprovededucationandemploymentoutcomes
Disaggregatingdataintomorespecificcategoriesofraceandethnicitycanhelpunderstandtrendsanddisparitiesinamoregranularwaywhichcanhelpdecisionmakerstargetpoliciesandinterventionsinamoreequitableway
EarlierintheRoadmapweworkedtoidentifystakeholdersandwhomightbepositivelyandnegativelyimpactedbyachangeinpolicyUsetheresultsofthatexercisetoinvolvethecommunityininterpretingdataanalysisandtogatherqualitativefeedbackfromthecommunitytocallattentiontoissuesForinstanceanincreaseinyouthdetentionmaybeexplainedbyinefficienciesorlackofaccesstotransportation
Designimplementationandevaluation
49
Designimplementationandevaluation
50
Shareyourstoryandsuccess
Sharetheresults
Storytelling
AnintegralstepinyourstrategyissharingtheresultsofyourpilotorpolicychangewithyourcommunityThisincludesdevelopingmessagingaroundtheworkofyourorganizationandcreatingaplantodisseminateyourstoriesWhytellstoriesaboutyourwork
Informandorunderstandasubjectorsituation
Giveneededcontextofdataorbitsofinformation
Relatablestoriesarememorableforreadersviewers
ReadersorviewerswillseetheinformationasmorecompellingwhenthereisademonstratedandrelatablechangeorimpactonthemselvesortheirfriendsneighborsandcolleaguesStoriesareaboutpeopleandgovernmentisfundamentallyabouthelpingpeopleSotellthestoriesofthepeopleyouandyourcitycolleagueshelped
Keepthesetipsinmindasyoudevelopyourstorywithyourteam
WhatwereyourfindingsandwhyshouldanyonecareTellyouraudiencehowtheproblemwassolvedWhathastheimpactbeenonresidentsandcitystafftodateWhatwasthesuccessandweretheremultiplesuccessesalongthewayIfyouhavepreliminaryresultsordatashowingimprovementtosharethisisthetimetodoit
WhowasinvolvedandimpactedReviewthelistofstakeholdersHowwereeachofthemimpactedbytheprogramHumanizeyourstoryandimpactbyaddingadditionalvoicesThesevoicescanbeoneofthefollowing
Apersonorfamilywhowaspositivelyaffectedbythechangesuccess
Aninvolvedthird-partydescribingboththeproblemandthesolution(ieanorganizationnonprofitprovidersbusinesstradeassociationsportsteametc)whoispartofthecommunitythatthegovernmentworkedwithonthechangesuccess
Someonefromthegovernmentitselfwhocanspeakaboutthechangesuccess
TellmoreaboutyourcityDescribeyourcityincludingpopulationdemographicsindustriesetcWhatareitschallengesWhatisitknownforWhatmakesituniqueHowisitsimilarordifferentfromothercitiesintheUS
Shareyourstoryandsuccess
51
DescribeyourcityrsquoschallengeWriteoutthechallengeyourcityisfacing(asdefinedinyourproblemstatement)HowdoesitaffectresidentsHowdoesthisproblemaffecttheworkofcitystaffmembers
AddthedetailsProvidedetailsonyourappliedresearchstrategyTalkaboutwhatyourcityhasdonenthepasttoresolvetheissueDiscusshowyourstrategyhaschangedwhenyoubegantoadoptthisapproachWasmoneysavedorinvested
Usethisstorymapexercisetoorganizeyourthoughtsandguaranteethatyouincludealloftheelementsofatraditionalstory
AfteryoudevelopyourmessagingcreateacommunicationsmarketingandoutreachstrategydesignedtospreadyourmessageYourstrategyshouldwalkyouthroughthefollowingkeyactivities
1 IdentifyandprofileyouraudienceWhencreatinganddisseminatingyourmessageitisalwaysgoodtothinkaboutyouraudienceWhoexactlywouldbeinterestedinwhatyouhavetosayThegeneralpopulationofthecityPeoplewhoworkinnonprofitsBusinessownersItmattersbecausenoteveryonewholivesinacitymaybethetargetoftheinformationBusinessownersmaybemoreinterestedindatashowingeffectsonthebusinessdistrictthanothersandsoonWewillcoverthislaterbutthisstepwillalsohelpyoutothinkofthemosteffectivewaytoreachyouraudience
2 Defineyourcalltoaction(ifyouhaveone)SpendsometimeconsideringwhyyouareinterestedintellingyourstoryDoyousimplywanttoupdatethepubliconyourprogressorcelebrateyoursuccessesAreyouinterestedinyouraudiencechangingitsbehaviororsupportingaspecificinitiativebeinglaunchedbyyourorganizationIfyouwanttohaveyouraudienceactorbehaveinacertainwaybespecificandincorporateitinyourmessaging
Shareyourstoryandsuccess
52
3 SelectperformancemeasuresThinkabouthowyouwoulddetermineifyourcampaignissuccessfulandselectappropriateperformancemeasuresYouwillrevisitthesemeasuresattheendofyourcampaigncommunicationsplanimplementationtodeterminewhetheryouneedtomodifyyourcommunicationsandoutreachstrategyforthefuture
4 MethodstocommunicateyourstoryTherearemanywaystocommunicatesuccessofyourprojectHerearesomesuggestionsonreachingaudiencesItisimportanttodeterminewhichmethodsarethemosteffectiveforyourdesiredaudiencetoreceiveinformationDoyourresearch(PewResearchCenterhasexcellentstudiesonthereachofmediadigitalmediaandmarketing)ForexampleifseniorsareyourtargetpopulationandtheyarelessinclinedtouseInstagram(thatistrue)youmaywanttoconsiderFacebookorthenightlylocalnewsbothofwhichareusedmoreoftenbyseniorsBelowisalistofsomemethodsandtoolsyoucanuse
i Pressreleasestolocalnewspapersandtvstations
ii ShortsmartphonevideosAgreatwaytoengageanaudienceisthroughvideoAskonequestionofsomeonewhoispartofthestoryMaybethereisasocialworkerwhocandiscusshungerinyourcityThesevideoscanbereusedonmultipleplatforms
iii Long-formwriting
iv TellingindividualstoriesthroughaseriesofphotosandshortnarrativesforpublicationonFacebookorInstagram
v Op-edsopinionpiecesplacedinthelocalpaper
vi Emailupdatestoyourlist
vii Connectingwithappropriateorganizationsthatareinvestedinthetopicandworkwiththecommunityandaskthemtodisseminateinformationtotheirmembers
viii EmployingldquoWordofMouthrdquo(WOM)tacticsbycreatingtownhallsorfreeeventsforresidentstoattendintheirownneighborhoods
5 DeterminethetypeofcollateralUsuallycampaignsthattargetresidentsrequireseveralavenuestoreachthemPeoplearenotalwaysinthesameplaceorspaceHavingakeenunderstandingofyouraudiencewillassistyouwithyourchoicesonwhatneedstobedevelopedSometimesthatmeansimagesforsocialmediaaonepageflierthatgivespeopleinformationonattendinganeventoranin-depthbrochure
Shareyourstoryandsuccess
53
6 ImplementyourstrategyKeepinmindstrategiescanbefluidSometimessomethingmaynotturnoutasexpectedevenifduringplanningsessionsitwasldquosupposedtordquoItisokYoucananalyzewhathappenedduringadebriefsessionandonceyoursurveysandothermeasurementshavebeenfinalized
Visualizations
Withtherightvisualizationyoucanreviewandshareinformationinsightsanalysisperformanceandtrendsallofwhichisvaluabletoacross-sectionofthepublicButwhichvisualizationisrightforyouTochoosetherighttypeofchartgraphdeterminethetypeofquestionyouwanttoanswerSomecommoncharttypesarebelowandmoreinformationcanbefoundhere
Exploreadditionalanalyticaltechniques
Duringthegoal-settingphaseyouestablishedoutputoutcomeandimplementationmeasuresBasedontheresultsofyourpilotyoumayneedtoadjustyourmeasuresWiththepilotdatainhandyoumayhavesomeideasofhowfutureanalysescouldbedifferentandpotentiallyimprovethequalityofyourprogramTakesometimetobrainstormotherrelevantdatasetsthatyoudidnothavethetimetopursueearlierincludingmessydepartmentaldatathatneedsmoreattentionorsourcesofpublicdatafromthestateandfederalgovernmentnonprofitsandevenprivatecompaniesthatcouldsupplementyourunderstandingofanissueUsingmachinelearningtechniquesyoumightbeabletobetter
Shareyourstoryandsuccess
54
predictdemandforhomelessservicesusingNOAAweatherforecastspreemptwatermainbreaksusingsoilstratigraphyorinvestineconomicdevelopmentbymonitoringrealestatelistingdata
Advancedanalyticaltechniquescanalsohelpyourimplementationrunmoreeffectivelybyintegratingdata-informedtargetinganddecisiontoolsintotheprocessByautomatingroutineworkflowsandsurfacingsubtletrendsthesetoolscanhelpyouiterateinamoreprecisemannerandprovidekeystrategicguidancetofront-lineemployeesthatwouldotherwisebeunavailableThesesortsofdata-science-enhancedprocessescanbeclassifiedasoneofsixprojecttypesasexploredfurtherbytheNewOrleansLANOLAlyticsteamFormoreguidanceonscopingasimilarprojecttheGovExadvancedanalyticsscopingguidegivesyouastep-by-stepoverviewofhowtobesuccessful
Usethespacebelowtolistyourideasforpossibleadvancedanalyticsprojectsbytype
Typeofproject Ideas
Findingtheneedleinthehaystack
Prioritizingforimpact
Earlywarningtools
Betterquickerdecisions
Optimizingresourceallocation
Experimentingforwhatworks
Keepthefeedbackloopgoing
GatherinformationonpublicsentimentaboutpolicychangethroughfocusgroupssurveysandpublicmeetingsledbyadvisorygroupmembersNotewhethernewinformationisavailablethatcanhelpanalyzeorcontextualizeyourprogress(egfederalorstatedatareports)Sharetheresultsofyourpilotandthepolicychangeprocesswithothercitiesandresearchers
Shareyourstoryandsuccess
55
WorkingwitharangeofevidencePresentlythereisalotofattentionandinvestmentinevidence-basedpolicymakingincludingthefederalbipartisanCommissiononEvidence-BasedPolicymakingthePewCharitableTrustsreportsandtoolkitsonHowStatesEngageinEvidence-BasedPolicymakingandtheLauraandJohnArnoldFoundationsupportedEvidence-BasedPolicymakingCollaborativeManyoftheseinitiativesfocusonrigorousevaluation--meaningrandomizedcontrolledtrials(RCT)orotherexperimentalmodels--asthebestkindofevidencetouseHoweverthisbodyofevidenceisofteninsufficientforthespecifickindsofquestionsthatgovernmentscanhavewhenmakingadecisionaboutpolicyInthisguideweproposeanapproachtosupportgovernmentsinincorporatingadditionaltypesofevidenceintotheirdecisionmaking
OnewaytounderstandarangeofevidenceistouseNestarsquosStandardsofEvidencetoframeanddefineevidenceTheBritishnonprofitNestaworkstoinjectpositivechangeintothewayUKgovernmentpoliciesandprogramsfunctionOneoftheirkeyareasofworkhasbeendevelopingWhatWorksCentresorganizationsthatcollectcommissionsynthesizeanddisseminateevidenceaboutinterventioneffectivenessacrossarangeofsocialimpactareasNestadevelopedthefollowingStandardsofEvidencetoprovideausefulwaytodocumenthowdifferentkindsofinformationdemonstratedifferentlevelsofevidence
Workingwitharangeofevidence
56
Theimportanceofpotentiallybiasedinformation
Onechallengepresentedbyanevidenceclassificationthatacknowledgesthevalueofnon-scientificformsofevidenceistofigureouthowtobalanceallofthepiecestogetherwhenyouusethemTounderstandhowtodothiswehavetothinkabouthowtopositionpotentiallybiasedinformationindecisionmaking
WhenwelookatinformationwhichhasnotbeenrigorouslyevaluatedweacceptthatthisinformationmightbebiasedThewholereasonweconductrigorousevaluationistoremovebiasfromourobservationsWeconductstructuredtrialstoproduceresultsthatareconsistentandgeneralizableIfafindingisunbiasedthisaccuracymdashthislackofbiasmdashcanbeprovedbecauseanyindependenttestercanreproduceitwhentheyfollowanidenticalprocess
YoucanlookatpotentiallybiasedinformationasjustinformationthatisearlieroninitspathtorefinementreplicationandverifiedaccuracyApieceofinformationbasedonanyoneindividualrsquosobservationsislikelytobebiasedbecauseitiscomingfromjustonesourceBiasisreducedbygatheringtogetherlotsofindependentsourcesofinformationandfiguringoutitsmostcommonattributes
MaybethebiasedinformationmakesaclaimthatistoolargemdashitsayssomethingistrueunderallconditionswhenitisreallyonlytrueundercertainconditionsorthataneffectofaninterventionisbiggerthanitstrueaverageorthataneffectlastsovertimewhenitdoesnotPerhapschangingtheclaimtobemorespecificmdashcriticallyreviewingtheinformationtoseeifspecificconditionsmustbemettoachievetheoutcomesmdashwouldmaketheclaimmoreaccurate
IfyoutestedyourclaimfounditdidnotholdupinsomecaseschangedittesteditagainuntilyourclaimrepeatedlyheldupthenyoucoulddemonstratethattheclaimwasnolongerbiasedbutaccurateThisprocessoftestingimprovingtestingandvalidatingiswhatwegetfromstructuredscientificstudy
Withindividualpiecesofinformation--crediblehypothesescasestudiesexpertopinion--youcannotassumethatkindofaccuracyAtthesametimetheseearlyobservationsarefoundationaltoanyreplicablescientificobservationsWewillnevergetanywhereifwedonotstartwiththesepiecesofinformationeveniftheymightbebiased
Combininglevelsofevidencetogetthemostdetailedinformationforyourcity
ItisespeciallyimportanttoacceptdifferentformsofevidencewhenyoufacequestionsthathavenotyetbeentestedbyindependentstudiesSomeofthemostimportantquestionsrelatetothingsthatareveryspecifictoyoursituationYoufaceaparticularcombinationof
Workingwitharangeofevidence
57
issuesactorstrendsandresourcesthatisultimatelydifferentfromeveryotherplaceintheworld
ThisisimportantbecauseaspolicypractitionersandnotscientistsyoucaremoreaboutyourspecificcasethanahypotheticalaveragegeneralcaseYoucarelessaboutthetotallikelyrangeofimpactsfromapolicythanhowthissingleimplementationwillworkYouwantinformationaboutpoliciesthatwillprovideyouwiththebestpossiblepredictionabouthowtheywillworkinyourcontext
ByputtingdifferentkindsofevidenceintothesameframeworkNestarsquosStandardsofEvidenceimplythatwhenyoucannotgetrigorouslytestedexperimentallyverifiedinformationitisappropriatetoworkwithwhatwedohaveThisisrelevantforthemunicipalimplementationcontextbecausenostudypulledfromaliteraturereviewwillaccountforallofthevariablesyoufaceinyourowncontext
Intheabsenceofhavingastudythatpredictshowaninterventionworksinyourspecificcontextyouwillneednon-scientificinformationtotrytofillinthosegapsYouwillneedtogatherstoriesfromresidentsstaffmembersandotherstakeholdersthatsitonLevels1or2ofthestandardsofevidence
Someofthekindsofinformationyouwillprobablygetmoreeasilyfrompeoplethanstudiesinclude
howpeopleincitieslikeyoursreactedtoaninterventioninpublicforumshowpeoplewhoimplementedaninterventioninacitylikeyoursfiguredouthowtogetitpassedhowthecitycommunicatedpubliclyabouttheapparentbenefitsanddrawbacksoftheinterventionthepoliticalacceptabilityoftheintervention
Youcanremainsimultaneouslyawarethattheinformationyouaregatheringthroughindividualstoriesisbiased--notgeneralizableinformedbyspecificperspectives--whilealsorecognizingthathavingsomeinformationaboutsomethingimportantisbetterthannoinformationEvidencefromthefirsttwoevidencelevelsareessentialbuildingblocksofknowledgeandperfectlyusefulwhenyoucannotgetdetailedunbiasedknowledge
Youhavetostartsomewhereandyoushouldalwaysstartwithevidenceyouhaveaboutyourveryspecificconditionsthatyouknowareimportantforsuccess
Workingwitharangeofevidence
58
Thankyou
Interviewswithcities
GovExheldsemi-structuredinterviewswithsixstaffmembersinfourcitiestoestablishtheinitialstepsoftheRoadmapWeaskedrespondentstotalkaboutaninitiativeorpolicythattheyhavebeenresponsibleforimplementingintheircityRespondentsfromthreecitiesspokeabouttheireffortstoaddresshomelessnessandrespondentsfromonecityspokeaboutblightAfterrespondentsidentifiedtheproblemtheyspokeabouthowthecityprioritizedthisissuethevariousstepsinvolvedandbarrierspresent
GovExconductedtextanalysisoftheseinterviewswhichrevealedthatpeoplewasthemostcommonlyusedtermduringtheseconversationswhendiscussingthestepstochangeaprogramorpolicyThisisathemethroughoutthisRoadmapThefirstsectionbeginswithidentifyingtherelevantpeopleTheconversationsaroundevidencearefocusedonhumanbehaviorratherthanmethodologyorstrengthofevidence
Thankyou
59
OncewehadadraftoftheRoadmapwefollowedupwithcitiesonanindividualbasistolearnmoreaboutwhatwouldbehelpfulWearegratefultothefollowingcitystafffortheinterviewsandindividualconversations
MichaelBaskinChiefPolicyOfficerChattanoogaTN
MelissaBridgesPerformanceandInnovationCoordinatorLittleRockAR
VictoriaCarreoacutenAdministrativeOfficerLasVegasNV
TyroneGrandstrandCommunityDevelopmentPlannerFargoND
TibLaughlinDirectorofGeneralServicesKansasCityKS
TimMorelandDirectorofPerformanceManagementandOpenDataChattanoogaTN
CharlesRosendahlManagementAnalystDentonTX
Thankyou
60
MelissaSiebenAssistantCountyAdministratorKansasCityKS
GinaThompsonDirectorofStrategicServicesLewisvilleTX
JHUFacultycollaboration
GovExalsoconvenedthecitycontactsandfacultypartnersinaseriesofAdvisoryGroupmeetingsThesemeetingsservedtoconnectcitiesandfacultytounderstandoneanothersperspective
StefanBaralBloombergSchoolofPublicHealth
AshleyBernerJHUSchoolofEducation
MelissaDavey-RothwellBloombergSchoolofPublicHealth
JonLinksBloombergSchoolofPublicHealth
TamarMendelsonBloombergSchoolofPublicHealth
KristinMmariBloombergSchoolofPublicHealth
KeshiaPollackPorterBloombergSchoolofPublicHealth
JoshSharfsteinBloombergSchoolofPublicHealth
LizStuartBloombergSchoolofPublicHealth
LindsayThompsonCareyBusinessSchool
DanielWebsterBloombergSchoolofPublicHealth
Partnersindataandevidence-basedpolicy
Wealsospokewithpartnerswhoareexpertsinthefieldofevidence-basedpolicywhosharedtheirknowledgeandresourcesofcurrentandfuturethinkinginevidence-informedthinking
WendyBarrenoAssociateDirectorWilsonSheehanLabforEconomicOpportunitiesUniversityofNotreDame
TiffanyDavenportDirectorofDataPartnershipsBaltimoresPromise
TomiHiersDirectorBaltimoreCivicSiteAnnieECaseyFoundation
ChrisKingsleySeniorAssociateAnnieECaseyFoundation
KathrynOliverAssociateProfessorLondonSchoolofHygieneandTropicalMedicine
Thankyou
61
KristenWatkinsSeniorPolicyAssociateJ-PAL
GovExParticipants
GovExAnalystMiriamMcKinneyanalyzedtheinterviewtranscriptsandconductedtextanalysisSeniorImplementationAdvisorEmilyShawwillintegrateRoadmapactivitieswithadditionalservicedeliveryactivitiesatGovEx
Thankyou
62
ReferencesAbrahamKGampHaskinsR(2017)Thepromiseofevidence-basedpolicymakingReportofthecommissiononevidence-basedpolicymakingWashingtonDCCommissiononEvidence-basedPolicymakingBigmove2Extendingthegreen(nd)Retrievedfromhttpswwwcityoflewisvillecomdoing-businesslewisville-2025big-move-2-extending-the-greenBatesMAampGlennersterR(2017)ThegeneralizabilitypuzzleRetrievedfromhttpsssirorgarticlesentrythe_generalizability_puzzleBuildwithnotfor(2018January)RetrievedfromhttpsbuildwithnotforwordpresscomCairneyPampOliverK(2017)Evidence-basedpolicymakingisnotlikeevidence-basedmedicinesohowfarshouldyougotobridgethedividebetweenevidenceandpolicyHealthResearchPolicyandSystemsdoi101186s12961-017-0192-xCairneyPOliverKampWellsteadA(2016)TobridgethedividebetweenevidenceandpolicyReduceambiguityasmuchasuncertaintyPublicAdministrationReview76(3)399-402Retrievedfromhttpdxdoiorg101111puar12555Centreforpublicimpact(nd)RetrievedfromhttpswwwcentreforpublicimpactorgCernaL(2013)ThenatureofpolicychangeandimplementationAreviewofdifferenttheoreticalapproachesOrganisationforEconomicCo-OperationandDevelopmentRetrievedfromhttpswwwoecdorgeduceriTheNatureofPolicyChangeandImplementationpdfColoradoSprings(nd)Retrievedfromhttpscoloradospringsgovcommunicationspagewhat-expect-when-you-speakupContinuumofcare(coc)program(nd)RetrievedfromhttpswwwhudexchangeinfoprogramscocCouncilpresidentkicksoff$12millionyouthfund(nd)Retrievedfromhttpwwwbaltimorecitycouncilcomcontentcouncil-president-kicks-12-million-youth-fundDurlakJAampDuPreEP(2008)ImplementationmattersAreviewofresearchontheinfluenceofimplementationonprogramoutcomesandthefactorsaffectingimplementationAmericanJournalofCommunityPsychology41327-350Retrievedfromhttpswwwncbinlmnihgovpubmed18322790Fishbonediagram(ishikawa)(nd)Retrievedfromhttpasqorglearn-about-qualitycause-analysis-toolsoverviewfishbonehtmlFivewhysGettingtotherootofaproblemquickly(nd)RetrievedfromhttpswwwmindtoolscompagesarticlenewTMC_5Whtm
References
63
FligsteinNampMcAdamD(2011)TowardageneraltheoryofstrategicactionfieldsSociologicalTheory29(1)1-26RetrievedfromhttpsociologyberkeleyedusitesdefaultfilesfacultyfligsteinFligstein20McAdam20Soc20Theory20PaperpdHeinkelP(2018January12)CitymullspermitsforbikesharingcompaniesRetrievedfromhttpwwwdentonrccomnewsdenton20180112city-mulls-permits-bike-sharing-companiesHowisareamedianincomecalculated(2012October)Retrievedfromhttpswwwhudexchangeinfofaqs843how-is-area-median-income-calculatedLampkinLMampHatryHP(2003)KeystepsinoutcomemanagementUrbanInstituteRetrievedfromhttpswwwurbanorgresearchpublicationkey-steps-outcome-managementMarshDampSmithM(2000)UnderstandingpolicynetworksTowardsadialecticalapproachPoliticalStudies484-21Retrievedfromhttpjournalssagepubcomdoiabs1011111467-924800247McCannL(nd)RetrievedfromhttplaurenellencomMoultonSampSandfortJR(2016)ThestrategicactionfieldframeworkforpolicyimplementationresearchPolicyStudiesJournal45(1)Retrievedfromhttponlinelibrarywileycomstore101111psj12147assetpsj12147pdfv=1ampt=jdkhhcp4amps=e184c625e1855a6b1615ccb99ee27b1c0b3212d8PaynterB(2018January1)What100citiesarelearningfromeachotherbysharingtheirdataRetrievedfromhttpswwwfastcompanycom40518004what-100-cities-are-learning-from-each-other-by-sharing-their-dataPeakacademy(nd)Retrievedfromhttpswwwdenvergovorgcontentdenvergovenmayors-officeprograms-initiativespeak-performancepeak-academyhtmlPinnockHBarwickMCarpenterCREldridgeSGrandesGGriffithsCJRycroft-MaloneJMeissnerPMurrayEPatelASheikhAampTaylorSJC(2017)Standardsforreportingimplementationstudies(stari)ExplanationandelaborationdocumentBMJOpenRetrievedfromhttpbmjopenbmjcomcontentbmjopen74e013318fullpdfProctorESilmereHRaghavanRHovmandPAaronsGBungerAGriffeyRampHensleyM(2011)OutcomesforimplementationresearchConceptualdistinctionsmeasurementchallengesandresearchagendaAdministrationandPolicyinMentalHealth38(2)65-76doi101007s10488-010-0319-7TandonSD(2018)ProgressincommunityhealthpartnershipsResearcheducationandactionRetrievedfromhttpswwwpressjhuedujournalsprogress-community-health-partnerships-research-education-and-actionProvanKGampKenisP(2007)ModesofnetworkgovernanceStructuremanagementandeffectivenessJournalofPublicAdministrationResearchandTheory
References
64
18(2)229-252Retrievedfromhttpsacademicoupcomjpartarticle182229935895SabatierPA(1988)Anadvocacycoalitionframeworkofpolicychangeandtheroleofpolicy-orientedlearningthereinPolicySciences21(3)129-168Retrievedfromhttpwwwjstororgstable4532139SandstromAampCarlssonL(2008)TheperformanceofpolicynetworksTherelationbetweennetworkstructureandnetworkperformancePolicyStudiesJournal36497-524Retrievedfromhttponlinelibrarywileycomdoi101111j1541-0072200800281xabstractSchwartzMampWilsonE(nd)WhocanaffordtoliveinahomeAlookatdatafromthe2006americancommunitysurveyRetrievedfromhttpswwwcensusgovhousingcensuspublicationswho-can-affordpdfShearerJCAbelsonAKouyateBLavisJNampWaltG(2016)WhydopolicieschangeInstitutionsinterestsideasandnetworksinthreecasesofpolicyreformHealthPolicyandPlanning31(9)1200-1211Retrievedfromhttpsdoiorg101093heapolczw052ThelessonsofadministrativedataHigh-profilepolicy-relevantresearchpoweredbyadministrativedata(nd)Retrievedfromhttpswwwpovertyactionlaborgsitesdefaultfilesresources20161114-The-Lessons-of-Administrative-DatapdfToolsandotherneedsforqualitativeandquantitativemethods(2015February)AgencyforHealthcareResearchandQualityRetrievedfromhttpwwwahrqgovprofessionalsquality-patient-safetytalkingqualityassesstoolshtml
WeaverRCampBagchi-SenS(2013)SpatialanalysisofurbandeclineThegeographyofblightAppliedGeography4061-70
References
65
UniversityPartners
Name Title Affiliation Area(s)ofexpertise
Contactinformation
Examplesofworkingwithnon-academicdecisionmakers
WendyPBarreno
AssociateDirector
WilsonSheehanLabforEconomicOpportunitiesUniversityofNotreDame
LEOsfocusareasincludehealtheducationhousingchildwelfareeconomicopportunitycriminaljustice
wbarrenonduduorleondedu
MelissaDavey-RothwellPhD
AssociateScientist
JohnsHopkinsBloombergSchoolofPublicHealth
Health mdavey1jhuedu410-614-5854
BaltimoreCityHealthDepartmentLeadersMarylandDepartmentofHealthAddictionRecoveryServicesinDCExecutiveDirectorsofcommunityorganizationsacrosstheUSthatfocusonHIVSTIsubstanceabuseandimplementationscience
ElizabethStuart
AssociateDeanforEducationandProfessorDepartmentsofMentalHealthBiostatisticsandHealthPolicyand
JohnsHopkinsBloombergSchoolofPublicHealth
Educationsocialpolicystatisticalmethods
estuartjhuedu
Resources
66
Management
Design-ThinkingResources
EngagementGuideforCivicTech
Journeymap
PerformanceManagementResources
DenverPeakAcademy
Proxymeasuresguide
Talkingaboutperformancewithoutcomplicatingit(video)
SettingPerformanceTargetsguide
Datavisualizationguide
Performancemanagementgettingstartedguide
Exercises
Fishboneexercise
5Whys
AdministrativeData
TheLessonsofAdministrativeData(JPAL)
EvaluationResources
Quantitativeandqualitativemethods(USDepartmentofHealthandHumanServicesAgencyforHealthcareResearchandQuality)
KeyStepsinOutcomeManagement(UrbanInstitute)
RCT-YES
Sageresearchmethods
BestPracticesforMixedMethodsResearchintheHealthSciences
ACCORDChecklist
Pilotandimplementationplanning
Benchmarkingexamplesandproxymeasures
Seattleopendataplaybook
Resources
67
Datasciencescopingguide
Resources
68
AboutGovExGovExhelpsgovernmentsusedataandevidencetoimproveresidentsrsquolivesWesupportgovernmentsinbuildingtheirinternalcapacitytooperatesustainableanalyticspracticeswhichimprovecommunityoutcomesinfieldsfrompublicsafetytoeconomicdevelopmentGovExhasaproventrackrecordofworkinginconcertwithpartnerstoprovidetechnicalassistanceandtrainingatscale
SinceourlaunchinApril2015GovExhas
Trainedover2000governmentemployeesinpersonandonlineintopicsincludingdatamanagementperformancemanagementopendatacommunityengagementandanalytics
Providedtechnicalassistancetomorethan100localgovernmentsacrosstheglobeimprovingtheiruseofdatatoachieveefficienciesinservicedeliveryandbetteroutcomesforresidents
Assessedthecapacityofmorethan130governmentalorganizationstodeterminehowleadersstaffandexternalstakeholdersmanageandusedata
Localgovernmentsareincreasinglyonthefrontlinesofaddressinglargescaleproblemswithglobalimplicationssuchasclimatechangepublicsafetyandeconomicinequality
Whileevidence-basedinterventionstohelpsolvethoseproblemsareavailablelocalgovernmentstaffhavelimitedtimeandcapacitytofindandimplementresearch-basedsolutionsAsaresultcitiescontinuetoworkthewaytheyalwayshave--makingincrementalimprovementstoprogramsbutnotalwayscreatingchangeatasystem-level
ThroughresourcedevelopmentandcollaborationwithJHUfacultyGovExissupportingcityleadersinusingdataandevidencetomakedecisionsandcreatinglastingrelationshipswithresearchers
TheRoadmapwasdevelopedbyGovExaspartofitspartnershipwithBloombergPhilanthropiesWhatWorksCitiesinitiative
AboutGovEx
69
- Title page
- Introduction and purpose of the Roadmap
- Using the Roadmap
-
- Gather information about the problem
- Strategically engage key partners
- Review the evidence and collect details
- Decide on a policy to implement based on local factors
- Establish a framework to monitor progress
- Assess data accessibility quality and security
- Design implementation and evaluation
- Share your story and success
-
- Working with a range of evidence
- Thank you
- References
- Resources
- About GovEx
-
description
Programorpolicygoal (1sentencecapturingtheprincipalaimoftheintervention)
Existingimplementations
(WherehastheinterventionbeentriedListasmanysitesasyouareawareof)
Characteristicsofimplementationsites
(ForeachimplementationsiteofferbasicgeographicIDmdashcityandstateThenyouwillwanttocollectthedetailswhicharerelevanttoyourdecisionmakersrsquoconsiderationsaboutwhatmakesalocationldquosimilartordquoorldquodifferentfromrdquoyoursituationYouwillatleastwanttoincludeforeachimplementationsitePopulationnumberPopulationdensity(ruralurbansuburban)Socio-economiccharacteristicsBeforestartingaskyourdecisionmakersaboutanyparticularcharacteristicsofprogramimplementationsitestheywouldwanttoknowbeforeseekingtoreplicate)
Characteristicsofimplementationpartners
(Anyavailabledetailsaboutorganizationsinstitutionsorpeoplewhowereinvolvedwithimplementingtheintervention-egindependentchildcarecenterpublicelementaryschoolcitypublichealthnurses)
Characteristicsoftargetpopulationeligibleprogramparticipants
(Anyavailabledetailsaboutpeoplewhoarethebeneficiariesoftheprogramorotherwiseincludedoraffectedbytheintervention-egnumberofprogramparticipantscategoriesofpropertyownersaffected)
Policyorprogramduration
(HowlonghastheinterventionbeeninplaceineachimplementationsiteHasitendedandifsowhy)
Whatbenefitscamefromeachimplementation
(Anyobservedpositiveeffectsoftheinterventionincludingeffectsize)
Statisticalestimatesofinterventioneffectiveness
(Anyavailablestatisticalestimatesoftheinterventionrsquoseffects)
Whatdrawbackswereobserved
(Anyavailabledetailsaboutunintendedorunavoidablenegativeeffectsoftheintervention)
Programorpolicycosts
(Anyavailabledetailsaboutprogramcostsperimplementation)
Reviewtheevidenceandcollectdetails
28
Reviewtheevidenceandcollectdetails
29
DecideonapolicytoimplementbasedonlocalfactorsTogetthecriticalbuy-infromleadershiptoimplementanewpolicyorprogramtheprojectleadshouldscheduleameetingwithdecisionmakerstopresenttheevidencesurroundingthepolicyoptionscapturedinthetemplateDependingonyourinsightintoyourcityleadershipswillingnesstomakechangeyoumaywishtosharetheresearchyouhaveconductedontherootcausesoftheproblemandsomeofyouranalysisontheimpactofcurrenteffortstodate
Considertheapplicabilityofanapproach
AsdiscussedinthepreviouschapterpolicymakersoftenlookforevidenceofeffectiveprogramsinothercitiesbutoftenthequestionarisesofhowgeneralizableaprogramisfromonecitytoanotherTheAbdulLatifJameelPovertyActionLab(J-PAL)suggeststhatinreviewingevidencefromanotherlocationitishelpfultoidentifythespecificmechanismthatwasused(deliveryofprogramorserviceinformationsharingetc)identifythelocalconditionsthatwereinplaceobservethebehaviorchangethatresultedfromthemechanismandfinallytounderstandlocalcontextandconditions(legalorcostconstraintsculturalnormsetc)Thissetofpresentationelementscanbedrawnfromthelistofdetailscollectedintheresearchtemplate
EducationpolicymakersinChicagoILusedfindingsfromprogramsinKenyaandIndiainwhichchildrenwhohadfallenbehindtheirpeersweregiveneducationalsupporttargetedtotheirleveloflearningThemechanismusedwastargetedteachingandthereweresimilarconditionspresentinallthreecountriesAlthoughsomestudentshadfallenbehindinthecurriculumteacherswereincentivizedtoteachgrade-levelmaterialsConsiderationsofhowthelocalcontextinChicagomightaffectimplementationofatargetedteachingprogramincludedunderstandinghowtheneighborhoodsthatthestudentswerefrommayhaveaffectedtheireducationaloutcomesandthehighcostofteachinginsmallgroupsInChicagoMatchEducationprovidedtwo-on-onetutoringwhichmatchedstudentswithwell-educatedindividualswhohavenothadformalteachertrainingforalower-costmodelTheresultswerepositiveStudentswhoweretutoredperformedbetterinmathandweremorelikelytopasshighschoolclasses
ThegeneralizabilityframeworkproposedbyJ-PALisbelowUsethisasatemplateasyouconsiderpotentialpolicysolutions
Decideonapolicytoimplementbasedonlocalfactors
30
SourceTheGeneralizabilityPuzzle
OnceyouhavenarrowedthepolicyoptionsdiscussthequestionsbelowforeachoptionsyouareconsideringThisstepiscriticalasunderstandinglocalcontextissoimportantinsuccessfulpolicyimplementation
CommunityContextQuestionnaire
Localcontext
DoesthisoptionmakesenseinourcommunityIsthisaonesizefitsallapproachIsanyparticulargroupabsentfromthisapproachIfsoisthatok
Youranswers
Effectiveness
WhatwouldweexpecttoobserveifthepilotisaseffectivehereasinCityX
Youranswers
Timing
DoesthetimingmakesenseIsthistherighttimetoimplementthispolicychange
Decideonapolicytoimplementbasedonlocalfactors
31
WherearethepotentialroadblocksthatcouldaffectthescheduleIsthereanywaytomitigatethisHowlongwouldittaketoseeanimpact
Youranswers
Implementation
DowehavethestaffcapacitytoimplementthisArethereadditionalpartnersthatyoucanbringtothetabletosupportimplementationDoesthismakesensegivenourformofgovernment
Youranswers
Authority
DoesourcityhavetheauthoritytoenactthisIfsowhatistheappropriatemechanismAretherelegalbarriersArethereanyexistingpreemptionissuesfromthestateorfederalgovernmentCanthosebeaddressedthroughlegislationadvocacyIstherepoliticalsupportoroppositiontothis
Youranswers
Cost
HowmuchwillthiscostannuallyHowwillwepayforitCanthepolicybeimplementedwithexistingfunding
IfsoisthisfundingreliableHowwouldthispolicyoptionaffectourcitysbudgetintheshortandlongtermWhatotherfundingoptionscanweconsiderIsthereanalternativethatwouldcostlessandgiveussimilarresultsWillanyconstituencybeadverselyaffectediffundingforprogramsthatimpactthemisshiftedtoanotherprogrampolicyorinitiative
Youranswers
Decideonapolicytoimplementbasedonlocalfactors
32
EstablishaframeworktomonitorprogressNowthatyouhavereviewedtheevidenceconsideredlocalcontextandchosenyourpolicyitistimetodeterminehowyourcitywillmonitorprogressofthisnewapproachCitystaffandleadersneedtoestablishtheinfrastructureforongoingcontinuousdecisionmakingbasedonwhetherthepolicychangeisleadingtothedesiredoutcomesforresidentsandwhetherthechangeismeaningfulandrationaltofrontlineworkersSettingarbitrarygoalsatthelocallevelthatconflictwithgoalsatotherlevelsisconfusingtotheindividualsresponsibleforachievingthegoalsGoalsthataresettoohightoachievecanbedemoralizingtostaff
ThissectionincludesstepstosetupavenueinwhichCitystaffandleadershiproutinelyvisitkeyindicatorstodeterminewhatisorisnotworkingandtomakedecisionsinreal-timeFormorereadingonperformancemanagementpleasevisittheGovExPerformanceManagementGettingStartedGuide
InperformancemanagementtermssuchasindicatormeasureandmetricmaybeusedinterchangeablyInthisguidewewilluseindicatorInyourworkthemostimportantthingistobeconsistentinwhichevertermyouuseThisvideomaybehelpfulinunderstandingthisconceptandcommunicatingittoothers
Establishthefundamentalsandgatheryourteam
FirstdeterminewhetherthereisanestablishedvenueorregularmeetingscheduleduringwhichyoucandiscussprogressIfyourcityhasanexistingperformancemanagementstructureonarelatedtopicaddingadiscussionoftheprogressofpolicychangetoyouragendacanbeagreatideaIfthiswillbeastandalonemeetingdecidehowfrequentlytheprojectteamanddecisionmakerswillmeetandputthosemeetingsonthecalendarfortheentirelengthoftheproject
DeterminewhowillstaffandattendroutinemeetingsonCityprogressThesemeetingsmayincludetheMayorandorCityManagerdepartmentheadstaffwhoareimplementingthepolicychangeandanalysts
Identifyindicators
ItiscriticalthatindicatorsbeanimportantcityandcommunitypriorityaswellassomethingtheCityplaysamajorroleinaddressingTheremaybeinstancesinwhichtheCitywillneedtoreachouttoorcollaboratewithexternalpartnerstogetamorecompletepictureofprogressontheoverallpolicychange
Establishaframeworktomonitorprogress
33
Anoutcomeindicatordescribesthedesiredeffectofthetargetedpopulation
ExampleBellevuesfamilyfriendlyneighborhoodsindicatormeasuresthepercentofresidentswhobelievetheyliveinaneighborhoodthatsupportsfamiliesparticularlythosewithchildren
WithyourkeypartnersrevisityourproblemstatementcurrentstateandfuturestateexerciseandbegintoidentifytheoutcomeindicatorsthatwillillustratesuccessFornowchooseasingletop-levelkeyperformanceindicator(KPI)ItiscriticalthattheKPI-thepieceyouwillmeasure-beapriorityforthecommunityandsomethingtheCityplaysamajorroleinaddressing
Outputindicatorsmeasuretheprocessofprogramorservicedelivery
ExampleSanDiegoMayorKevinFaulconersgoaltorepair1000milesofstreetby2020
ReviewingtheadministrativedatasetinventorybegintoparedowntherelevantprogramsanddatasetsthatsupportyourKPIWhatarethemeasuresofeffectivenessforeachprogramTheseareyouroutputindicators
Thediagrambelowillustratestherelationshipbetweenoutputandoutcomeindicators
ThinkaboutimplementationindicatorsasmeasuringstepsintheprogramsrolloutthatarenecessarytoobtaindesiredchangesImplementationoutcomescanfocusontheinterventionstaffdeliveringtheinterventionanorganizationorsystemandcanaddressfeasibilityadaptabilitysustainabilityandcostTwokeyquestionstoconsiderwhendevelopingimplementationindicatorsare
1 Istheinterventionacceptableandappropriatetothecommunityand
2 IsitfeasibletoimplementinterventionXatthistimewiththeseresources
Establishaframeworktomonitorprogress
34
Amoreextensivelistwithdefinitionsofpotentialimplementationoutcomesislinkedhere
UsethetablebelowtochoosethedesiredimplementationoutcomesforyourpilotandtoplanyourmeasurementtechniqueThisshouldincludelevelofanalysis(individualproviderorconsumerorganizationorsetting)yourmeasurementtechnique(surveysobservationinterviewfocusgroupsadministrativedataanalysis)andthephasewhenyouwillcollectthisinformation(earlymidlate)Theremaybemorethanonelevelofanalysismeasurementtechniqueandphaseofdatacollection
Implementationoutcome
Definitionofoutcome
Levelofanalysisandmeasurement
Yourmeasurementtechnique
Phaseofpilotwhenyouwillcollectdata
Usethespacebelowtocaptureyourindicatorwishlist
Indicatorwishlist
Outcomes_____________________________________________________________________________
Outputs________________________________________________________________________________
Implementationoutcomes______________________________________________________________
Setapublichigh-levelgoal
Thehigh-levelgoaliscriticalinestablishingdirectiontogalvanizethepeoplewhowillbedoingtheworkWhendiscussingthegoalusetheMayorsbullypulpitLookforkeymilestonesinCitygovernmenttosharethegoalstatementsuchasthebeginningofaMayoraladministrationaStateoftheCityspeechorotherpublicremarks
GoalsshouldbeldquoSMARTrdquo
Specificaspossible(innamingthepopulationwhowillbebetterofftheactionthecitywilltake)
Measurable
Achievable
Establishaframeworktomonitorprogress
35
Results-focused
Time-bound
ThegoalstatementshouldincludeyouroutcomeindicatorlikeChattanoogasgoalldquoWewillendchronicveteranhomelessnessbyDecember2016rdquoAdiagramtohepyoubuildaSMARTgoalaroundyouroutcomeindicatorisbelow
Component Description Yourturn
Specific WhowillbebetteroffWhatactionthecitywilltake
Measurable Howwillyoumeasuresuccess
Achievable Isthegoalattainable
Results-focused Whatistheexpectedresult
Time-bound When
Setrealisticambitioustargetsandtimelinesforeachindicator
ToestablisharealisticandambitioustargetforyourgoalstatementusetheadministrativedatacollectedforyourchosenKPItoreviewactualCityperformanceinthemostrecentyearcomparedwiththethreeorfourmostrecentyearsworthofdataBegintocalculatesummarystatistics(egtotalminimumaveragemedianmaximum)toidentifytrendsCreatecumulativeandmonthlychartstoidentifytrendsandanomaliesBasedonyouranalysisofhistoricaldataignoretheanomaliesandassumetheproductivitytrendswillfollowthesameoverallpatternfrommonth-to-monthFinallyapplyanincreasedproductivityfactortotheperformanceeachmonthtoforecastfutureprogressBasedonthisanalysischooseatargetthatinspiresimprovementwhileaccountingforreality
Manycitiesthataremorematureintheirperformancemanagementpracticesrecognizetheimportanceoftakingacommunity-levelapproachtoimprovingoutcomesforresidentsThesecitiesarecreatinggoaltaxonomiesthatincludemeasuresrelatedtopublicsafetyhousingandevenrecreationCitiesdriveprogressonthesegoalsbyimplementingstrategiesthatengagemultipledepartmentsandcollaboratewithserviceprovidersinthecommunity
EventuallyyouwilldevelopgoalstatementsfortheoutputsinordertocapturetheefforttoachievetheoutcomecontainedinyourgoalstatementYoursetofindicatorsshouldbecomposedofshort-andinterim-termsupportivemeasureswithaccuratetargetsandrelevanttimelinesAnexamplegoaltaxonomyfromtheCityofLasVegassResultsVegasprogramisbelowUsethistaxonomyasamodelforyourownsuiteofindicatorsandgoalstatements
Establishaframeworktomonitorprogress
36
Yourturn
Thetemplatebelowismeanttohelpyoucraftagoalstatementforeachsupportingmeasure
Goalstatement
Departments
Outcome
Activities
KPI
Target
Timeframe
Conductanddiscussrelevantanalysis
ThrougharoutineseriesofmeetingscontinuallymonitorprogressofKPIssupportingindicatorsandanyotherdatacollectedsuchasqualitativedatatoidentifytrendsthatwillindicatewhetherthechosenpolicyishavingthedesiredimpactThesedatapointsshoulddrivethediscussionduringthesemeetingspointingtoopportunitiesforfurtherinquiryinto
Establishaframeworktomonitorprogress
37
whatisdrivingtheobservedtrendsAnalystsshouldbeencouragedandtrainedtoaskquestionsofthedepartmentsproducingthedataandconductsitevisitswhenpossibletodeepentheirunderstandingofthedataPresentationoftheanalysisshouldfocusononeideaatatimetokeeptheconversationfocusedonkeypointsandspecificopportunitiesforimprovementYoumayconsiderdevelopingapublicfacingdashboardtosharethisprogressmorebroadly
AgeneralcomparisonofthestatusquodatawithwhatwasobservedduringthepilotcanprovidedecisionmakerswithaquickunderstandingofthedifferencesbetweentheoldandnewapproachesalthoughthiswillnotshowcausalityOftenthesecomparisonswillincludeoperationalmetricssuchasdaysordollarssavedbutrememberthattheimpactonpeoplewillbemostimportantIfyouareworkingwithanevaluationpartnertheywillsupportthisstep
Usethespacebelowtosummarizetheresultsofyouranalysis
MeasureTargetvalue(seesectiononsettingtargetsabove)
Pilotvalue(whatdidyouobserveduringthepilot)
Historicalaverage(ifavailable)
Results(WhoisbetteroffIsanyonenegativelyimpacted1-2sentencesreassessingyourgoalstatement)
DocumentyourprocessandprogressincludingfeedbackfromtheparticipantsimplementationpartnersandtheprojectteamDidanythingunexpectedoccurduringthepilotArethereinternalandexternalfactorsthatmayaffectyouroutcomesThisiswheretheimplementationoutcomescanbecanhelpfulIftheinterventiondidnotgoasplannedconsiderwhetheritisbecausetheinterventionwasnotacceptableorfeasible
UsethetemplatebelowcreatedbyDenversPeakAcademytocapturethisinformationIfthe306090daytimelinedoesnotmakesenseforyourcityfeelfreetocustomizeit
CurrentState
FutureState
30Day
60Day
90day
Whatwentwell(+)
Whatdidnotgowellshouldbechanged(-)
IfaquestionordiscussionpointarisesthatcannotbeansweredinthemeetinganassignedstaffmembershouldcapturethequestionduringthemeetingAfterwardthefollowupitemsshouldbesenttorelevantpartnersandthenextmeetingshouldbeginwithadiscussionof
Establishaframeworktomonitorprogress
38
thoseitems
Takeaction
DiscussionsaboutwhetheryourindicatorsaremovingintherightdirectionandwhatmightbedrivingtrendsmayleadtoadiscoverythattherearenotenoughresourcesallocatedtothispolicyorissueTheremaybelimitedfundingforstaffprogrammingortechnologythatisdrivingthedatainthewrongdirectionIfCityleadershipisintheroomthisisagreattimetomakethecaseforadditionalfundingduringthebudgetseasonBycollectinganalyzinganddiscussingdataonthispolicyroutinelyyouarebuildingevidencetomakethecasefortherightleveloffundingtoreachtheestablishedtargets
Establishaframeworktomonitorprogress
39
Assessdataaccessibilityqualityandsecurity
Determineaccessibilityandqualityofdata
OnceyouhavecompiledyourwishlistofindicatorsitistimetodeterminetheaccessibilityandqualityofthedataTopreventcomplicationyoumaywishtoprioritizerelevantindicatorsforwhichyouarealreadycollectingdataAvoidthetemptationofadoptingatechnologysolutiontocollectnewdataatthispointsuchasaphotosurveytoolorotherappFornowyouaretryingtodeterminewhichdatasetsarerelevantaccessibleandhighqualityBesuretoworkwithyourkeyinternalpartnerswhileyouareidentifyingyourmeasuresDepartmentstaffwhoworkonprogramsarefamiliarwiththeindicatorsthatwillaccuratelysupporttheoutcomesanddatabaseadministratorswillbevaluablewhendiscussingaccessibilityandqualityofthedata
Abrieflistofconsiderationsfortheaccessibilityofyourdataisbelow
ArewecurrentlycollectingthisdatasetorwillwebecollectingitforthefirsttimeHowfarbackdoestherecordgo(timehorizon)WouldweneednewtechnologytocollectoranalyzethedataCanthedatabecollectedmanuallyorcanitbeautomatedHowfrequentlyisthedataproduced(continuouslyhourlydailyweeklymonthlyquarterlyyearlyetc)IsthedataavailablefortheentireCitypopulationorasubsetofthepopulationHowgranularisthedata(individuallevelCensusblockneighborhoodcitywideetc)DoesthedatacontainsensitiveinformationIsthereaproxymeasurethatwecanusefornowIsthedatamachinereadable
AsnotedbytheWilsonSheehanLabforEconomicOpportunities(LEO)attheUniversityofNotreDameleveragingadministrativedatacansignificantlyreducethecostsandtimelinesofrigorousevaluationCitiescanestablishaninternaldatasharingagreementacrossdepartmentsincludinglanguagethatenablesadministrativedatatobeusedforresearchpurposesCitiescanalsoidentifyexternaladministrativedata-holderssuchasacountyschooldistrictpolicedepartmenthospitaloranotherstatethatmayhaveoutcomedataimportantformeasuringprogramimpactCitiescanleverageexistingrelationshipsandorworkwithresearchinstitutionstodevelopstreamlineddatasharingprocessesandagreementswiththeseexternalentities
Assessdataaccessibilityqualityandsecurity
40
Evaluatethequalityofyourdata
QualityisasimportantasaccessibilityToevaluatedataqualityyoumaywishtocreatearubricliketheonebelow
Value ValueDescription
3 ExcellentDataQualityThedataiscompleteaccurateandupdatedregularly
2 AdequateDataQualityThedataissomewhatcompleteaccurateandupdatedfairlyregularly
1 PoorDataQualityThedataisnotcompleteaccurateorupdatedregularly
ThisisagreattimetobuildonthedatainventorythatyoustartedinthegatherinformationpageSuggestedadditionalfieldstoaddtoyourdatainventorytemplatearebelow
Program DatasetTitle
DatasetDescription DataSource
Departmentorexternalpartner
DoestheCityalreadycollectthisdata
Whatisthenameoftheprogram
Human-readablenameofthedataset
Shouldbeunderstandablebynon-technicalusers
Whatistheoriginalsystemapplicationfilethathousesit
Departmentorexternalpartnerthatmaintainsthedata
Isthisdataalreadycollected
Updatefrequency-ThefileformatforthedatasetTypicallycsvxlsxshapefiletxtetc
Format-Howoftenthedatasetisupdated
Timerange-Theperiodoftimecoveredinthisparticulardataset
Datause-Howisthisdatasetcommonlyused
Datausers-Whotypicallyusesthisdataset
Dataownerstitle-Titleofthedataowner
Dataownersposition-DataOwnersjobtitle
Dataownersemailaddress-DataOwnersemailaddressWhomanagesthedataandorisresponsibleforgrantingpermissiontoaccessthedataWhounderstandswhatthedatasetincludesandcananswerquestionsaboutit
Dataqualityconcerns-Describetheconcernsyoumayhaveaboutthequalityofthedataset
Assessdataaccessibilityqualityandsecurity
41
Division-DivisionwherethedataismaintainedIfmorethanonedivisionisresponsiblelisttheprimaryownerofthedataset
Sensitivedatacomments-DescribetheissuesthatwouldpreventthecityfromsharingthedatapubliclyCommonsensitivityconcernsincludeprivacyviolationssecurityissuesorhighcostorstaffdemandstopostthedata
Publishingstatus-Thistheplacetoputanycommentsregardingthecurrentstatusofthedataset
Publishedlink-Thisisthelinktothepublisheddatasetontheopendataportalorcitywebsite
Reviewdatasecurityandprivacyprotectionprotocols
OftenapolicychangealsoinvolvesachangeinthedatathatisbeingcollectedhowthatdataisbeinghandledwhohasaccesstoitandahostofotheradjustmentsAsyoutakestockofthedataassetsthatareavailableforyourinitiativeitisvitaltoalsothinkthroughwhatwillbedonetoprotectthatdataandtheprivacyofthepeopleitrepresentsAbigpartofgooddatasecurityinvolvesstrongdatagovernancepracticesthatdefineandidentifysensitivedataandconsciouslyworktobalancethevalueandriskofstoringsuchsensitivedataOnewayofevaluatingyourdatacomesfromcaselawrelatedtotheUSFreedomofInformationActwhichwehavetranslatedtothechecklistbelow
1 Woulddisclosureresultinasubstantialinvasionofprivacy2 Haveweconsideredtheextentorvalueofthepublicinterestandthepurposeorobject
oftheindividualsseekingdisclosure3 Istheinformationavailablefromothersources4 Wastheinformationgivenwithanexpectationofconfidentiality5 Isitpossibletomouldreliefsoastolimittheinvasionofindividualprivacy
TheseriskscouldbemitigatedusingavarietyofstrategiesfromaggregatingdatatoobfuscatingindividualstoredactingfieldsthatmayincludePersonallyIdentifiableInformation(PII)orre-evaluatingyourdatacollectionmethodstofindalternativewaysofgettingtheinformationyouneedThesestepsshouldbecodifiedandmanagedbyadatagovernancecommitteeifpossibleForthesensitivedatathatisdeemedessentialcreateaplanaroundpotentialdatabreachesinordertoquicklyandconfidentlyrespondtoemergencysituations
ProactivelypreparingforthesesituationsisoftenthebeststrategyfordatasecuritybutdonotforgetthetechnicalsideInvolveyourITadministratorsinconversationsaboutdatastorageandaccessensuringthattheywillbeprotectedbyenterprise-gradecybersecuritysystems
Assessdataaccessibilityqualityandsecurity
42
Assessdataaccessibilityqualityandsecurity
43
Designimplementationandevaluation
Designtheimplementation
NowthatyouhaveyouraccountabilityplanandkeyindicatorsitistimetogetspecificaboutyourplanforimplementationThetemplatebelowwillhelpyouthinkthroughthespecificmechanismsofimplementingpolicychange
UnderstandingorganizationsandsystemsiscriticaltounderstandingandmakingdecisionsaboutwhereandhowinterventionsshouldbeappliedApolicyimplementedinasinglesettingmayshowpromisingresultsbutimplementingmultiplepoliciesandprogramsatthesametimewithdifferentactorsresponsibleforeachaspectcanhelpensuresustainablebehaviorchange
InSomervilleMAShapeUpSomerville(SUS)isacitywidepublichealthprogramthatisimplementedacrossmultiplelevelsofinfluenceandsystemsthroughinterventionsbeforeduringandafterschooltoincludechangestobuiltenvironmentsuchasbikelanesregionalmasstransportationandphysicaleducationequipmentinschoolsandgymstoincreasephysicalactivitywellnesspoliciesandschoolgardensnutritioneducationduringandafterschoolandhealthyrestaurantprogramstosupporthealthyfoodandhealthyeatinghabits
Usethetemplatebelowtoplanyourimplementation
Policyimplementationplanningtemplate
ListthespecificdesiredoutcomesInthepreviousstepyouworkedwithdepartmentstafftoestablishspecificmeasurableoutcomesandinterimindicatorsPrioritizetheindicatorsthatwilldemonstrateprogressUsethespacebelowtocapturethedesiredoutcomesofthepolicychange
ExampleShapeupSomervillebeganwiththegoalofpreventingobesityin1stthrough3rdgraders
Yourturn
____________________________________________________________________________
____________________________________________________________________________
Designimplementationandevaluation
44
SpecifythetargetpopulationAreyoudesigninganinterventionthattargetsonlycertainsitesAreyouplanningonimplementingaprogramorinitiativeforaspecifictargetaudienceClearlyspecifythepopulationyouplantotarget
IfyouplantoconductarigorousimpactevaluationasnotedbytheWilsonSheehanLabforEconomicOpportunitiesitisimportanttoestablishconcreteobjectiveeligibilitycriteriaandensurethatthesecriteriaareappliedconsistentlybyallstaffduringscreeningandintakeItisalsovitaltoidentifytheindividualswhoareeligibleforaparticularprogramorserviceevenifeveryoneeligiblecannotbeservedbyexistingresourcesThishelpscitiesidentifyanyunmetneedsandguidefutureresourceallocationwhilealsocreatingthepossibilityofarigorousimpactevaluationbyidentifyingapotentialcomparisongroupFinallywhenconductingscreeningandintakeitisimperativetocollectandmaintainbasicinformation(egnamedateofbirthgender)aboutallprogram-eligibleindividualsnotjustthosewhoparticipateintheprogram
ExampleIntheShapeUpSomervilleexampletheprogrambeganbytargeting1stthrough3rdgraders
Yourturn
____________________________________________________________________________
____________________________________________________________________________
DefinethelocationWherewillthepolicychangetakeplaceWillimplementationtakeplaceatonesiteormultiplesites
ExampleShapeUpSomervilleinterventionsweredeliveredinschools
Yourturn
____________________________________________________________________________
____________________________________________________________________________
ClarifyimplementationpartnersandarticulatetheirrolesWhatarethekeyorganizationsinstitutionsorpeopleresponsibleforimplementationTheprojectteamshouldbeasclearandspecificaspossibleinprovidingadescriptionoftheconcreteactivitiesthatimplementationpartnersaremeanttocarryoutPracticeprofilesmaybeahelpfultooltohelpdefinewhatisrequiredoftheactorswhoareresponsibleforimplementingtheintervention
Designimplementationandevaluation
45
ExampleInSomervillemayoralleadershipandengagementofmultiplecityagencies(publichealthprimarycareschoolsCambridgeHealthAllianceetc)streamlinedandconnectedthevariousShapeUpSomervilleinitiativesandgrantswhichhelpedtokeepthetalentandenergyfocusedonhealthyeatingandactivelivingforallsectorsofSomervilleSUSpartnersultimatelyincludedschoolsnonprofitsCityandstateagenciesandnationalfoundations
Yourturn
____________________________________________________________________________
____________________________________________________________________________
DescribetheinterventionWhatarethestepsactivitiesorprocessesRememberthatthemechanismthespecificstepsorprocessthatleadtobehaviorchangeiskeytounderstandingtheeffectivenessofthepolicychangeandsharingtheresultswithothers
ExampleGoalsofShapeUpSomervilleweretoinfluenceeverypartofanearlyelementaryschoolchildrsquosdayndashbeforeduringandafter-schooltopromotehealthyeatingphysicalactivityandtheirweightstatusInterventionincludedfoodserviceenhancementstoimprovebreakfastandlunchandstaffprofessionaldevelopmentwalktoschoolactivitiesSUSclassroomcurriculumandprofessionaldevelopmentschoolwellnesspolicydevelopmentSUSafter-schoolcurriculumandprofessionaldevelopmentoutreachandeducationtohomethroughmaterialsforumseventsldquoSUSapprovedrdquorestaurantscommunityoutreachandcapacitybuildingthroughpolicydevelopmenttrainingsmediaplacements
Yourturn
____________________________________________________________________________
____________________________________________________________________________
ClarifytimelineandmilestonesfordeliveryandevaluationHowfrequentlywilltheservicesinterventionbedeliveredHowlongwilltheprogramrunWhatisthetimetoexpectedimpacts(eg12monthsafterprogramcompletion)
Designimplementationandevaluation
46
ExampleShapeUpSomervilleledbyTuftsUniversityFriedmanSchoolofNutritionScienceandPolicyconductedaquasi-experimental(nonrandomized)interventionstudyamong1stthrough3rdgradestudentsinSomerville(intervention)andtwocomparisoncommunitiesfor9-monthsduringoneschoolyear(fall2002ndashspring2003)ThestudyshowedpositiveassociationoftheinterventiononchangeinBMISUSreleasedreportssummarizingevaluationsprogresstodateplansforadditionalprojectsandimpactstenandfifteenyearsintotheinitiative
Yourturn
____________________________________________________________________________
____________________________________________________________________________
PlanyouranalysisTheprevioussectionaddressedestablishinganaccountabilityframeworktomonitorprogressandimplementationDeterminewhichindicatorswillbediscussthroughthatprocesswhowillsharethedataandanalysisandhowfeedbackfromthoseconversationswillbeincorporatedbytheprojectteam
ExampleTheMayorrsquosofficeimplementedadata-drivendecisionmakingapproachcalledSomerStatwhichwassynergisticwiththeapproachestakenbytheresearchandevaluationteamsdeployedforthespecificgrants
Yourturn
____________________________________________________________________________
____________________________________________________________________________
PlanforsustainabilityIfthepolicyandprogramchangeissuccessfulhowwilltheprojectteamsustainandexpandtoadditionalsitesorpopulations
ExampleAlthoughmostofthepartnershipsforgedintheearlyyearsweremaintainednewgrantsandcollaborationsexpandedattentiontoolderpublicschoolstudentsactivetransportationandtheGreenLinesubwayextensionthebuiltenvironmentfarmerrsquosmarketeffortssuchasldquoGrowninSomervillerdquowithUnionMainStreeteconomicdevelopmentandhealthdisparitiesaswellasdatacollectionmonitoringandsurveillance
Yourturn
Designimplementationandevaluation
47
____________________________________________________________________________
____________________________________________________________________________
Pilotplanningtemplate
IfyouplantoconductapilotfirstdeterminethepurposeofthepilotDoyouwanttoobservewhetherthepolicychangeresultsinthedesiredbehaviorchangeAreyouconcernedaboutfeasibilityscalabilityorcostDoyouwanttotestwhetheryourdesiredresultscanbeachievedthroughthispolicyapproachClarifythegoalofthepilotandthequestionsyouhopetoanswerThiswillhelpyouidentifythekeyindicatorsthatyouwillusetomeasurewhetherthepilotissuccessfulBeforebeginningestablishabaselineofcurrentperformanceandsetyourtargetsaccordinglybasedonwhatyouhopetoaccomplishthroughthepilot
NextestablishthepilotplanforyourprojectteamincludingmilestonesforteamcheckinstodeterminewhethertomoveaheadorchangecourseThisisseparatefromthemeetingsthatwilloccuraspartoftheaccountabilityframeworkUsethespacebelowtocreateyourtimelineanddevelopyourmeetingschedule
Task Personresponsible Startdate Checkindate Deliverable
Planyourevaluation
WhileyouaredesigningtheimplementationitisalsotimetobegintodevelopyourplanforanalysisAsnotedbytheWilsonSheehanLabforEconomicOpportunitiesdecisionsaboutprogramsetup-includingoutreachscreeningintakedatatrackingandenrollment-candeterminewhetherornotarigorousimpactevaluationwillbepossibleAsdiscussedcityleadersmayfindithelpfultoincludeevaluationpartnersinearlyconversationsaboutprogramsetupDuringpreviousstagesyoumayhaveworkedwitharesearchertoplanyourproposedpolicychangeYoushouldbeclearaboutthepurposeoftheevaluationanddevelopanevaluationplanBelowareadditionalconsiderations
Causationvstrendanalysis
IntheEstablishaframeworktomonitorprogressstagetherewasguidanceontrackingtrendsovertimeItisimportanttonoteherethatmonitoringdataisdifferentfromanalyzingacausaleffectTrendanalysiswillillustratethatchangesareoccurringandmayleadtoquestionsthatcouldbeansweredbyyourevaluationbuttrendanalysisalonedoesnotprovideevidenceaboutwhatisdrivingthosechangesYoumayseechangesinyour
Designimplementationandevaluation
48
outcomeindicatorsbasedonvariablesoutsideoftheCityscontrolForinstanceachangeinfederaltradepolicycanhaveanimpactemploymentratesinyourCityoutsideofyourCitysnewjobtrainingandplacementprogram
ToaddresscausalityyoumustcomparetheoutcomesofindividualswhoreceivedtheinterventionwiththosewhodidnotWithoutacomparisongroupofsimilarindividualswhowerenotexposedtotheinterventionitisimpossibletoknowwhichfactorsareresponsiblefortheoutcomesRandomizedControlTrials(RCTs)inwhichindividualsarerandomlyassignedtotreatmentandcontrolgroupsarethegoldstandardinprovingcausalityIfyoucandemonstratethatoutcomesimprovedforthosewhoreceivedtheinterventionanddidnotimproveatallorimprovedlessforacomparisongroupyoucanbesomewhatcertainthattheinterventionwasthecause
Examplesofothernon-causalimpactevaluationsthatarerigorousandreliableincludecertainquasi-experimentalstudiesusingcomparisongroupdesignssmartinterruptedtimeseriesanalysesthatmodeltrendsovertimeinwaysthatallowestimationofinterventioneffectswelldesignedpre-posttestanalysesneedsassessmentsimplementationstudiesetcFormoreinformationondifferenttypesofevaluationsseetheSageResearchMethodswebsite
Lookfordifferencesingroups
BuildintimetoalsoexaminethepilotdatatoseeifgroupsareexperiencingdifferentresultsReviewingperformancebasedonoutcomemeasuresforapopulationinaggregatemayhidethesubtlewaysthatsomepeoplemaybeaffecteddifferentlysoitisnecessarytocarefullyanalyzethedatafordisparateoutcomesForinstancetheMovingtoOpportunity(MTO)studythatexaminedtheimpactofhousingvouchersonfamilieswholivedinhigh-povertyneighborhoodsfoundthatchildrenwhomovedbeforetheageof13hadbettereducationandhigherincomesasadultsHoweveradultsdidnotseeimprovededucationandemploymentoutcomes
Disaggregatingdataintomorespecificcategoriesofraceandethnicitycanhelpunderstandtrendsanddisparitiesinamoregranularwaywhichcanhelpdecisionmakerstargetpoliciesandinterventionsinamoreequitableway
EarlierintheRoadmapweworkedtoidentifystakeholdersandwhomightbepositivelyandnegativelyimpactedbyachangeinpolicyUsetheresultsofthatexercisetoinvolvethecommunityininterpretingdataanalysisandtogatherqualitativefeedbackfromthecommunitytocallattentiontoissuesForinstanceanincreaseinyouthdetentionmaybeexplainedbyinefficienciesorlackofaccesstotransportation
Designimplementationandevaluation
49
Designimplementationandevaluation
50
Shareyourstoryandsuccess
Sharetheresults
Storytelling
AnintegralstepinyourstrategyissharingtheresultsofyourpilotorpolicychangewithyourcommunityThisincludesdevelopingmessagingaroundtheworkofyourorganizationandcreatingaplantodisseminateyourstoriesWhytellstoriesaboutyourwork
Informandorunderstandasubjectorsituation
Giveneededcontextofdataorbitsofinformation
Relatablestoriesarememorableforreadersviewers
ReadersorviewerswillseetheinformationasmorecompellingwhenthereisademonstratedandrelatablechangeorimpactonthemselvesortheirfriendsneighborsandcolleaguesStoriesareaboutpeopleandgovernmentisfundamentallyabouthelpingpeopleSotellthestoriesofthepeopleyouandyourcitycolleagueshelped
Keepthesetipsinmindasyoudevelopyourstorywithyourteam
WhatwereyourfindingsandwhyshouldanyonecareTellyouraudiencehowtheproblemwassolvedWhathastheimpactbeenonresidentsandcitystafftodateWhatwasthesuccessandweretheremultiplesuccessesalongthewayIfyouhavepreliminaryresultsordatashowingimprovementtosharethisisthetimetodoit
WhowasinvolvedandimpactedReviewthelistofstakeholdersHowwereeachofthemimpactedbytheprogramHumanizeyourstoryandimpactbyaddingadditionalvoicesThesevoicescanbeoneofthefollowing
Apersonorfamilywhowaspositivelyaffectedbythechangesuccess
Aninvolvedthird-partydescribingboththeproblemandthesolution(ieanorganizationnonprofitprovidersbusinesstradeassociationsportsteametc)whoispartofthecommunitythatthegovernmentworkedwithonthechangesuccess
Someonefromthegovernmentitselfwhocanspeakaboutthechangesuccess
TellmoreaboutyourcityDescribeyourcityincludingpopulationdemographicsindustriesetcWhatareitschallengesWhatisitknownforWhatmakesituniqueHowisitsimilarordifferentfromothercitiesintheUS
Shareyourstoryandsuccess
51
DescribeyourcityrsquoschallengeWriteoutthechallengeyourcityisfacing(asdefinedinyourproblemstatement)HowdoesitaffectresidentsHowdoesthisproblemaffecttheworkofcitystaffmembers
AddthedetailsProvidedetailsonyourappliedresearchstrategyTalkaboutwhatyourcityhasdonenthepasttoresolvetheissueDiscusshowyourstrategyhaschangedwhenyoubegantoadoptthisapproachWasmoneysavedorinvested
Usethisstorymapexercisetoorganizeyourthoughtsandguaranteethatyouincludealloftheelementsofatraditionalstory
AfteryoudevelopyourmessagingcreateacommunicationsmarketingandoutreachstrategydesignedtospreadyourmessageYourstrategyshouldwalkyouthroughthefollowingkeyactivities
1 IdentifyandprofileyouraudienceWhencreatinganddisseminatingyourmessageitisalwaysgoodtothinkaboutyouraudienceWhoexactlywouldbeinterestedinwhatyouhavetosayThegeneralpopulationofthecityPeoplewhoworkinnonprofitsBusinessownersItmattersbecausenoteveryonewholivesinacitymaybethetargetoftheinformationBusinessownersmaybemoreinterestedindatashowingeffectsonthebusinessdistrictthanothersandsoonWewillcoverthislaterbutthisstepwillalsohelpyoutothinkofthemosteffectivewaytoreachyouraudience
2 Defineyourcalltoaction(ifyouhaveone)SpendsometimeconsideringwhyyouareinterestedintellingyourstoryDoyousimplywanttoupdatethepubliconyourprogressorcelebrateyoursuccessesAreyouinterestedinyouraudiencechangingitsbehaviororsupportingaspecificinitiativebeinglaunchedbyyourorganizationIfyouwanttohaveyouraudienceactorbehaveinacertainwaybespecificandincorporateitinyourmessaging
Shareyourstoryandsuccess
52
3 SelectperformancemeasuresThinkabouthowyouwoulddetermineifyourcampaignissuccessfulandselectappropriateperformancemeasuresYouwillrevisitthesemeasuresattheendofyourcampaigncommunicationsplanimplementationtodeterminewhetheryouneedtomodifyyourcommunicationsandoutreachstrategyforthefuture
4 MethodstocommunicateyourstoryTherearemanywaystocommunicatesuccessofyourprojectHerearesomesuggestionsonreachingaudiencesItisimportanttodeterminewhichmethodsarethemosteffectiveforyourdesiredaudiencetoreceiveinformationDoyourresearch(PewResearchCenterhasexcellentstudiesonthereachofmediadigitalmediaandmarketing)ForexampleifseniorsareyourtargetpopulationandtheyarelessinclinedtouseInstagram(thatistrue)youmaywanttoconsiderFacebookorthenightlylocalnewsbothofwhichareusedmoreoftenbyseniorsBelowisalistofsomemethodsandtoolsyoucanuse
i Pressreleasestolocalnewspapersandtvstations
ii ShortsmartphonevideosAgreatwaytoengageanaudienceisthroughvideoAskonequestionofsomeonewhoispartofthestoryMaybethereisasocialworkerwhocandiscusshungerinyourcityThesevideoscanbereusedonmultipleplatforms
iii Long-formwriting
iv TellingindividualstoriesthroughaseriesofphotosandshortnarrativesforpublicationonFacebookorInstagram
v Op-edsopinionpiecesplacedinthelocalpaper
vi Emailupdatestoyourlist
vii Connectingwithappropriateorganizationsthatareinvestedinthetopicandworkwiththecommunityandaskthemtodisseminateinformationtotheirmembers
viii EmployingldquoWordofMouthrdquo(WOM)tacticsbycreatingtownhallsorfreeeventsforresidentstoattendintheirownneighborhoods
5 DeterminethetypeofcollateralUsuallycampaignsthattargetresidentsrequireseveralavenuestoreachthemPeoplearenotalwaysinthesameplaceorspaceHavingakeenunderstandingofyouraudiencewillassistyouwithyourchoicesonwhatneedstobedevelopedSometimesthatmeansimagesforsocialmediaaonepageflierthatgivespeopleinformationonattendinganeventoranin-depthbrochure
Shareyourstoryandsuccess
53
6 ImplementyourstrategyKeepinmindstrategiescanbefluidSometimessomethingmaynotturnoutasexpectedevenifduringplanningsessionsitwasldquosupposedtordquoItisokYoucananalyzewhathappenedduringadebriefsessionandonceyoursurveysandothermeasurementshavebeenfinalized
Visualizations
Withtherightvisualizationyoucanreviewandshareinformationinsightsanalysisperformanceandtrendsallofwhichisvaluabletoacross-sectionofthepublicButwhichvisualizationisrightforyouTochoosetherighttypeofchartgraphdeterminethetypeofquestionyouwanttoanswerSomecommoncharttypesarebelowandmoreinformationcanbefoundhere
Exploreadditionalanalyticaltechniques
Duringthegoal-settingphaseyouestablishedoutputoutcomeandimplementationmeasuresBasedontheresultsofyourpilotyoumayneedtoadjustyourmeasuresWiththepilotdatainhandyoumayhavesomeideasofhowfutureanalysescouldbedifferentandpotentiallyimprovethequalityofyourprogramTakesometimetobrainstormotherrelevantdatasetsthatyoudidnothavethetimetopursueearlierincludingmessydepartmentaldatathatneedsmoreattentionorsourcesofpublicdatafromthestateandfederalgovernmentnonprofitsandevenprivatecompaniesthatcouldsupplementyourunderstandingofanissueUsingmachinelearningtechniquesyoumightbeabletobetter
Shareyourstoryandsuccess
54
predictdemandforhomelessservicesusingNOAAweatherforecastspreemptwatermainbreaksusingsoilstratigraphyorinvestineconomicdevelopmentbymonitoringrealestatelistingdata
Advancedanalyticaltechniquescanalsohelpyourimplementationrunmoreeffectivelybyintegratingdata-informedtargetinganddecisiontoolsintotheprocessByautomatingroutineworkflowsandsurfacingsubtletrendsthesetoolscanhelpyouiterateinamoreprecisemannerandprovidekeystrategicguidancetofront-lineemployeesthatwouldotherwisebeunavailableThesesortsofdata-science-enhancedprocessescanbeclassifiedasoneofsixprojecttypesasexploredfurtherbytheNewOrleansLANOLAlyticsteamFormoreguidanceonscopingasimilarprojecttheGovExadvancedanalyticsscopingguidegivesyouastep-by-stepoverviewofhowtobesuccessful
Usethespacebelowtolistyourideasforpossibleadvancedanalyticsprojectsbytype
Typeofproject Ideas
Findingtheneedleinthehaystack
Prioritizingforimpact
Earlywarningtools
Betterquickerdecisions
Optimizingresourceallocation
Experimentingforwhatworks
Keepthefeedbackloopgoing
GatherinformationonpublicsentimentaboutpolicychangethroughfocusgroupssurveysandpublicmeetingsledbyadvisorygroupmembersNotewhethernewinformationisavailablethatcanhelpanalyzeorcontextualizeyourprogress(egfederalorstatedatareports)Sharetheresultsofyourpilotandthepolicychangeprocesswithothercitiesandresearchers
Shareyourstoryandsuccess
55
WorkingwitharangeofevidencePresentlythereisalotofattentionandinvestmentinevidence-basedpolicymakingincludingthefederalbipartisanCommissiononEvidence-BasedPolicymakingthePewCharitableTrustsreportsandtoolkitsonHowStatesEngageinEvidence-BasedPolicymakingandtheLauraandJohnArnoldFoundationsupportedEvidence-BasedPolicymakingCollaborativeManyoftheseinitiativesfocusonrigorousevaluation--meaningrandomizedcontrolledtrials(RCT)orotherexperimentalmodels--asthebestkindofevidencetouseHoweverthisbodyofevidenceisofteninsufficientforthespecifickindsofquestionsthatgovernmentscanhavewhenmakingadecisionaboutpolicyInthisguideweproposeanapproachtosupportgovernmentsinincorporatingadditionaltypesofevidenceintotheirdecisionmaking
OnewaytounderstandarangeofevidenceistouseNestarsquosStandardsofEvidencetoframeanddefineevidenceTheBritishnonprofitNestaworkstoinjectpositivechangeintothewayUKgovernmentpoliciesandprogramsfunctionOneoftheirkeyareasofworkhasbeendevelopingWhatWorksCentresorganizationsthatcollectcommissionsynthesizeanddisseminateevidenceaboutinterventioneffectivenessacrossarangeofsocialimpactareasNestadevelopedthefollowingStandardsofEvidencetoprovideausefulwaytodocumenthowdifferentkindsofinformationdemonstratedifferentlevelsofevidence
Workingwitharangeofevidence
56
Theimportanceofpotentiallybiasedinformation
Onechallengepresentedbyanevidenceclassificationthatacknowledgesthevalueofnon-scientificformsofevidenceistofigureouthowtobalanceallofthepiecestogetherwhenyouusethemTounderstandhowtodothiswehavetothinkabouthowtopositionpotentiallybiasedinformationindecisionmaking
WhenwelookatinformationwhichhasnotbeenrigorouslyevaluatedweacceptthatthisinformationmightbebiasedThewholereasonweconductrigorousevaluationistoremovebiasfromourobservationsWeconductstructuredtrialstoproduceresultsthatareconsistentandgeneralizableIfafindingisunbiasedthisaccuracymdashthislackofbiasmdashcanbeprovedbecauseanyindependenttestercanreproduceitwhentheyfollowanidenticalprocess
YoucanlookatpotentiallybiasedinformationasjustinformationthatisearlieroninitspathtorefinementreplicationandverifiedaccuracyApieceofinformationbasedonanyoneindividualrsquosobservationsislikelytobebiasedbecauseitiscomingfromjustonesourceBiasisreducedbygatheringtogetherlotsofindependentsourcesofinformationandfiguringoutitsmostcommonattributes
MaybethebiasedinformationmakesaclaimthatistoolargemdashitsayssomethingistrueunderallconditionswhenitisreallyonlytrueundercertainconditionsorthataneffectofaninterventionisbiggerthanitstrueaverageorthataneffectlastsovertimewhenitdoesnotPerhapschangingtheclaimtobemorespecificmdashcriticallyreviewingtheinformationtoseeifspecificconditionsmustbemettoachievetheoutcomesmdashwouldmaketheclaimmoreaccurate
IfyoutestedyourclaimfounditdidnotholdupinsomecaseschangedittesteditagainuntilyourclaimrepeatedlyheldupthenyoucoulddemonstratethattheclaimwasnolongerbiasedbutaccurateThisprocessoftestingimprovingtestingandvalidatingiswhatwegetfromstructuredscientificstudy
Withindividualpiecesofinformation--crediblehypothesescasestudiesexpertopinion--youcannotassumethatkindofaccuracyAtthesametimetheseearlyobservationsarefoundationaltoanyreplicablescientificobservationsWewillnevergetanywhereifwedonotstartwiththesepiecesofinformationeveniftheymightbebiased
Combininglevelsofevidencetogetthemostdetailedinformationforyourcity
ItisespeciallyimportanttoacceptdifferentformsofevidencewhenyoufacequestionsthathavenotyetbeentestedbyindependentstudiesSomeofthemostimportantquestionsrelatetothingsthatareveryspecifictoyoursituationYoufaceaparticularcombinationof
Workingwitharangeofevidence
57
issuesactorstrendsandresourcesthatisultimatelydifferentfromeveryotherplaceintheworld
ThisisimportantbecauseaspolicypractitionersandnotscientistsyoucaremoreaboutyourspecificcasethanahypotheticalaveragegeneralcaseYoucarelessaboutthetotallikelyrangeofimpactsfromapolicythanhowthissingleimplementationwillworkYouwantinformationaboutpoliciesthatwillprovideyouwiththebestpossiblepredictionabouthowtheywillworkinyourcontext
ByputtingdifferentkindsofevidenceintothesameframeworkNestarsquosStandardsofEvidenceimplythatwhenyoucannotgetrigorouslytestedexperimentallyverifiedinformationitisappropriatetoworkwithwhatwedohaveThisisrelevantforthemunicipalimplementationcontextbecausenostudypulledfromaliteraturereviewwillaccountforallofthevariablesyoufaceinyourowncontext
Intheabsenceofhavingastudythatpredictshowaninterventionworksinyourspecificcontextyouwillneednon-scientificinformationtotrytofillinthosegapsYouwillneedtogatherstoriesfromresidentsstaffmembersandotherstakeholdersthatsitonLevels1or2ofthestandardsofevidence
Someofthekindsofinformationyouwillprobablygetmoreeasilyfrompeoplethanstudiesinclude
howpeopleincitieslikeyoursreactedtoaninterventioninpublicforumshowpeoplewhoimplementedaninterventioninacitylikeyoursfiguredouthowtogetitpassedhowthecitycommunicatedpubliclyabouttheapparentbenefitsanddrawbacksoftheinterventionthepoliticalacceptabilityoftheintervention
Youcanremainsimultaneouslyawarethattheinformationyouaregatheringthroughindividualstoriesisbiased--notgeneralizableinformedbyspecificperspectives--whilealsorecognizingthathavingsomeinformationaboutsomethingimportantisbetterthannoinformationEvidencefromthefirsttwoevidencelevelsareessentialbuildingblocksofknowledgeandperfectlyusefulwhenyoucannotgetdetailedunbiasedknowledge
Youhavetostartsomewhereandyoushouldalwaysstartwithevidenceyouhaveaboutyourveryspecificconditionsthatyouknowareimportantforsuccess
Workingwitharangeofevidence
58
Thankyou
Interviewswithcities
GovExheldsemi-structuredinterviewswithsixstaffmembersinfourcitiestoestablishtheinitialstepsoftheRoadmapWeaskedrespondentstotalkaboutaninitiativeorpolicythattheyhavebeenresponsibleforimplementingintheircityRespondentsfromthreecitiesspokeabouttheireffortstoaddresshomelessnessandrespondentsfromonecityspokeaboutblightAfterrespondentsidentifiedtheproblemtheyspokeabouthowthecityprioritizedthisissuethevariousstepsinvolvedandbarrierspresent
GovExconductedtextanalysisoftheseinterviewswhichrevealedthatpeoplewasthemostcommonlyusedtermduringtheseconversationswhendiscussingthestepstochangeaprogramorpolicyThisisathemethroughoutthisRoadmapThefirstsectionbeginswithidentifyingtherelevantpeopleTheconversationsaroundevidencearefocusedonhumanbehaviorratherthanmethodologyorstrengthofevidence
Thankyou
59
OncewehadadraftoftheRoadmapwefollowedupwithcitiesonanindividualbasistolearnmoreaboutwhatwouldbehelpfulWearegratefultothefollowingcitystafffortheinterviewsandindividualconversations
MichaelBaskinChiefPolicyOfficerChattanoogaTN
MelissaBridgesPerformanceandInnovationCoordinatorLittleRockAR
VictoriaCarreoacutenAdministrativeOfficerLasVegasNV
TyroneGrandstrandCommunityDevelopmentPlannerFargoND
TibLaughlinDirectorofGeneralServicesKansasCityKS
TimMorelandDirectorofPerformanceManagementandOpenDataChattanoogaTN
CharlesRosendahlManagementAnalystDentonTX
Thankyou
60
MelissaSiebenAssistantCountyAdministratorKansasCityKS
GinaThompsonDirectorofStrategicServicesLewisvilleTX
JHUFacultycollaboration
GovExalsoconvenedthecitycontactsandfacultypartnersinaseriesofAdvisoryGroupmeetingsThesemeetingsservedtoconnectcitiesandfacultytounderstandoneanothersperspective
StefanBaralBloombergSchoolofPublicHealth
AshleyBernerJHUSchoolofEducation
MelissaDavey-RothwellBloombergSchoolofPublicHealth
JonLinksBloombergSchoolofPublicHealth
TamarMendelsonBloombergSchoolofPublicHealth
KristinMmariBloombergSchoolofPublicHealth
KeshiaPollackPorterBloombergSchoolofPublicHealth
JoshSharfsteinBloombergSchoolofPublicHealth
LizStuartBloombergSchoolofPublicHealth
LindsayThompsonCareyBusinessSchool
DanielWebsterBloombergSchoolofPublicHealth
Partnersindataandevidence-basedpolicy
Wealsospokewithpartnerswhoareexpertsinthefieldofevidence-basedpolicywhosharedtheirknowledgeandresourcesofcurrentandfuturethinkinginevidence-informedthinking
WendyBarrenoAssociateDirectorWilsonSheehanLabforEconomicOpportunitiesUniversityofNotreDame
TiffanyDavenportDirectorofDataPartnershipsBaltimoresPromise
TomiHiersDirectorBaltimoreCivicSiteAnnieECaseyFoundation
ChrisKingsleySeniorAssociateAnnieECaseyFoundation
KathrynOliverAssociateProfessorLondonSchoolofHygieneandTropicalMedicine
Thankyou
61
KristenWatkinsSeniorPolicyAssociateJ-PAL
GovExParticipants
GovExAnalystMiriamMcKinneyanalyzedtheinterviewtranscriptsandconductedtextanalysisSeniorImplementationAdvisorEmilyShawwillintegrateRoadmapactivitieswithadditionalservicedeliveryactivitiesatGovEx
Thankyou
62
ReferencesAbrahamKGampHaskinsR(2017)Thepromiseofevidence-basedpolicymakingReportofthecommissiononevidence-basedpolicymakingWashingtonDCCommissiononEvidence-basedPolicymakingBigmove2Extendingthegreen(nd)Retrievedfromhttpswwwcityoflewisvillecomdoing-businesslewisville-2025big-move-2-extending-the-greenBatesMAampGlennersterR(2017)ThegeneralizabilitypuzzleRetrievedfromhttpsssirorgarticlesentrythe_generalizability_puzzleBuildwithnotfor(2018January)RetrievedfromhttpsbuildwithnotforwordpresscomCairneyPampOliverK(2017)Evidence-basedpolicymakingisnotlikeevidence-basedmedicinesohowfarshouldyougotobridgethedividebetweenevidenceandpolicyHealthResearchPolicyandSystemsdoi101186s12961-017-0192-xCairneyPOliverKampWellsteadA(2016)TobridgethedividebetweenevidenceandpolicyReduceambiguityasmuchasuncertaintyPublicAdministrationReview76(3)399-402Retrievedfromhttpdxdoiorg101111puar12555Centreforpublicimpact(nd)RetrievedfromhttpswwwcentreforpublicimpactorgCernaL(2013)ThenatureofpolicychangeandimplementationAreviewofdifferenttheoreticalapproachesOrganisationforEconomicCo-OperationandDevelopmentRetrievedfromhttpswwwoecdorgeduceriTheNatureofPolicyChangeandImplementationpdfColoradoSprings(nd)Retrievedfromhttpscoloradospringsgovcommunicationspagewhat-expect-when-you-speakupContinuumofcare(coc)program(nd)RetrievedfromhttpswwwhudexchangeinfoprogramscocCouncilpresidentkicksoff$12millionyouthfund(nd)Retrievedfromhttpwwwbaltimorecitycouncilcomcontentcouncil-president-kicks-12-million-youth-fundDurlakJAampDuPreEP(2008)ImplementationmattersAreviewofresearchontheinfluenceofimplementationonprogramoutcomesandthefactorsaffectingimplementationAmericanJournalofCommunityPsychology41327-350Retrievedfromhttpswwwncbinlmnihgovpubmed18322790Fishbonediagram(ishikawa)(nd)Retrievedfromhttpasqorglearn-about-qualitycause-analysis-toolsoverviewfishbonehtmlFivewhysGettingtotherootofaproblemquickly(nd)RetrievedfromhttpswwwmindtoolscompagesarticlenewTMC_5Whtm
References
63
FligsteinNampMcAdamD(2011)TowardageneraltheoryofstrategicactionfieldsSociologicalTheory29(1)1-26RetrievedfromhttpsociologyberkeleyedusitesdefaultfilesfacultyfligsteinFligstein20McAdam20Soc20Theory20PaperpdHeinkelP(2018January12)CitymullspermitsforbikesharingcompaniesRetrievedfromhttpwwwdentonrccomnewsdenton20180112city-mulls-permits-bike-sharing-companiesHowisareamedianincomecalculated(2012October)Retrievedfromhttpswwwhudexchangeinfofaqs843how-is-area-median-income-calculatedLampkinLMampHatryHP(2003)KeystepsinoutcomemanagementUrbanInstituteRetrievedfromhttpswwwurbanorgresearchpublicationkey-steps-outcome-managementMarshDampSmithM(2000)UnderstandingpolicynetworksTowardsadialecticalapproachPoliticalStudies484-21Retrievedfromhttpjournalssagepubcomdoiabs1011111467-924800247McCannL(nd)RetrievedfromhttplaurenellencomMoultonSampSandfortJR(2016)ThestrategicactionfieldframeworkforpolicyimplementationresearchPolicyStudiesJournal45(1)Retrievedfromhttponlinelibrarywileycomstore101111psj12147assetpsj12147pdfv=1ampt=jdkhhcp4amps=e184c625e1855a6b1615ccb99ee27b1c0b3212d8PaynterB(2018January1)What100citiesarelearningfromeachotherbysharingtheirdataRetrievedfromhttpswwwfastcompanycom40518004what-100-cities-are-learning-from-each-other-by-sharing-their-dataPeakacademy(nd)Retrievedfromhttpswwwdenvergovorgcontentdenvergovenmayors-officeprograms-initiativespeak-performancepeak-academyhtmlPinnockHBarwickMCarpenterCREldridgeSGrandesGGriffithsCJRycroft-MaloneJMeissnerPMurrayEPatelASheikhAampTaylorSJC(2017)Standardsforreportingimplementationstudies(stari)ExplanationandelaborationdocumentBMJOpenRetrievedfromhttpbmjopenbmjcomcontentbmjopen74e013318fullpdfProctorESilmereHRaghavanRHovmandPAaronsGBungerAGriffeyRampHensleyM(2011)OutcomesforimplementationresearchConceptualdistinctionsmeasurementchallengesandresearchagendaAdministrationandPolicyinMentalHealth38(2)65-76doi101007s10488-010-0319-7TandonSD(2018)ProgressincommunityhealthpartnershipsResearcheducationandactionRetrievedfromhttpswwwpressjhuedujournalsprogress-community-health-partnerships-research-education-and-actionProvanKGampKenisP(2007)ModesofnetworkgovernanceStructuremanagementandeffectivenessJournalofPublicAdministrationResearchandTheory
References
64
18(2)229-252Retrievedfromhttpsacademicoupcomjpartarticle182229935895SabatierPA(1988)Anadvocacycoalitionframeworkofpolicychangeandtheroleofpolicy-orientedlearningthereinPolicySciences21(3)129-168Retrievedfromhttpwwwjstororgstable4532139SandstromAampCarlssonL(2008)TheperformanceofpolicynetworksTherelationbetweennetworkstructureandnetworkperformancePolicyStudiesJournal36497-524Retrievedfromhttponlinelibrarywileycomdoi101111j1541-0072200800281xabstractSchwartzMampWilsonE(nd)WhocanaffordtoliveinahomeAlookatdatafromthe2006americancommunitysurveyRetrievedfromhttpswwwcensusgovhousingcensuspublicationswho-can-affordpdfShearerJCAbelsonAKouyateBLavisJNampWaltG(2016)WhydopolicieschangeInstitutionsinterestsideasandnetworksinthreecasesofpolicyreformHealthPolicyandPlanning31(9)1200-1211Retrievedfromhttpsdoiorg101093heapolczw052ThelessonsofadministrativedataHigh-profilepolicy-relevantresearchpoweredbyadministrativedata(nd)Retrievedfromhttpswwwpovertyactionlaborgsitesdefaultfilesresources20161114-The-Lessons-of-Administrative-DatapdfToolsandotherneedsforqualitativeandquantitativemethods(2015February)AgencyforHealthcareResearchandQualityRetrievedfromhttpwwwahrqgovprofessionalsquality-patient-safetytalkingqualityassesstoolshtml
WeaverRCampBagchi-SenS(2013)SpatialanalysisofurbandeclineThegeographyofblightAppliedGeography4061-70
References
65
UniversityPartners
Name Title Affiliation Area(s)ofexpertise
Contactinformation
Examplesofworkingwithnon-academicdecisionmakers
WendyPBarreno
AssociateDirector
WilsonSheehanLabforEconomicOpportunitiesUniversityofNotreDame
LEOsfocusareasincludehealtheducationhousingchildwelfareeconomicopportunitycriminaljustice
wbarrenonduduorleondedu
MelissaDavey-RothwellPhD
AssociateScientist
JohnsHopkinsBloombergSchoolofPublicHealth
Health mdavey1jhuedu410-614-5854
BaltimoreCityHealthDepartmentLeadersMarylandDepartmentofHealthAddictionRecoveryServicesinDCExecutiveDirectorsofcommunityorganizationsacrosstheUSthatfocusonHIVSTIsubstanceabuseandimplementationscience
ElizabethStuart
AssociateDeanforEducationandProfessorDepartmentsofMentalHealthBiostatisticsandHealthPolicyand
JohnsHopkinsBloombergSchoolofPublicHealth
Educationsocialpolicystatisticalmethods
estuartjhuedu
Resources
66
Management
Design-ThinkingResources
EngagementGuideforCivicTech
Journeymap
PerformanceManagementResources
DenverPeakAcademy
Proxymeasuresguide
Talkingaboutperformancewithoutcomplicatingit(video)
SettingPerformanceTargetsguide
Datavisualizationguide
Performancemanagementgettingstartedguide
Exercises
Fishboneexercise
5Whys
AdministrativeData
TheLessonsofAdministrativeData(JPAL)
EvaluationResources
Quantitativeandqualitativemethods(USDepartmentofHealthandHumanServicesAgencyforHealthcareResearchandQuality)
KeyStepsinOutcomeManagement(UrbanInstitute)
RCT-YES
Sageresearchmethods
BestPracticesforMixedMethodsResearchintheHealthSciences
ACCORDChecklist
Pilotandimplementationplanning
Benchmarkingexamplesandproxymeasures
Seattleopendataplaybook
Resources
67
Datasciencescopingguide
Resources
68
AboutGovExGovExhelpsgovernmentsusedataandevidencetoimproveresidentsrsquolivesWesupportgovernmentsinbuildingtheirinternalcapacitytooperatesustainableanalyticspracticeswhichimprovecommunityoutcomesinfieldsfrompublicsafetytoeconomicdevelopmentGovExhasaproventrackrecordofworkinginconcertwithpartnerstoprovidetechnicalassistanceandtrainingatscale
SinceourlaunchinApril2015GovExhas
Trainedover2000governmentemployeesinpersonandonlineintopicsincludingdatamanagementperformancemanagementopendatacommunityengagementandanalytics
Providedtechnicalassistancetomorethan100localgovernmentsacrosstheglobeimprovingtheiruseofdatatoachieveefficienciesinservicedeliveryandbetteroutcomesforresidents
Assessedthecapacityofmorethan130governmentalorganizationstodeterminehowleadersstaffandexternalstakeholdersmanageandusedata
Localgovernmentsareincreasinglyonthefrontlinesofaddressinglargescaleproblemswithglobalimplicationssuchasclimatechangepublicsafetyandeconomicinequality
Whileevidence-basedinterventionstohelpsolvethoseproblemsareavailablelocalgovernmentstaffhavelimitedtimeandcapacitytofindandimplementresearch-basedsolutionsAsaresultcitiescontinuetoworkthewaytheyalwayshave--makingincrementalimprovementstoprogramsbutnotalwayscreatingchangeatasystem-level
ThroughresourcedevelopmentandcollaborationwithJHUfacultyGovExissupportingcityleadersinusingdataandevidencetomakedecisionsandcreatinglastingrelationshipswithresearchers
TheRoadmapwasdevelopedbyGovExaspartofitspartnershipwithBloombergPhilanthropiesWhatWorksCitiesinitiative
AboutGovEx
69
- Title page
- Introduction and purpose of the Roadmap
- Using the Roadmap
-
- Gather information about the problem
- Strategically engage key partners
- Review the evidence and collect details
- Decide on a policy to implement based on local factors
- Establish a framework to monitor progress
- Assess data accessibility quality and security
- Design implementation and evaluation
- Share your story and success
-
- Working with a range of evidence
- Thank you
- References
- Resources
- About GovEx
-
Reviewtheevidenceandcollectdetails
29
DecideonapolicytoimplementbasedonlocalfactorsTogetthecriticalbuy-infromleadershiptoimplementanewpolicyorprogramtheprojectleadshouldscheduleameetingwithdecisionmakerstopresenttheevidencesurroundingthepolicyoptionscapturedinthetemplateDependingonyourinsightintoyourcityleadershipswillingnesstomakechangeyoumaywishtosharetheresearchyouhaveconductedontherootcausesoftheproblemandsomeofyouranalysisontheimpactofcurrenteffortstodate
Considertheapplicabilityofanapproach
AsdiscussedinthepreviouschapterpolicymakersoftenlookforevidenceofeffectiveprogramsinothercitiesbutoftenthequestionarisesofhowgeneralizableaprogramisfromonecitytoanotherTheAbdulLatifJameelPovertyActionLab(J-PAL)suggeststhatinreviewingevidencefromanotherlocationitishelpfultoidentifythespecificmechanismthatwasused(deliveryofprogramorserviceinformationsharingetc)identifythelocalconditionsthatwereinplaceobservethebehaviorchangethatresultedfromthemechanismandfinallytounderstandlocalcontextandconditions(legalorcostconstraintsculturalnormsetc)Thissetofpresentationelementscanbedrawnfromthelistofdetailscollectedintheresearchtemplate
EducationpolicymakersinChicagoILusedfindingsfromprogramsinKenyaandIndiainwhichchildrenwhohadfallenbehindtheirpeersweregiveneducationalsupporttargetedtotheirleveloflearningThemechanismusedwastargetedteachingandthereweresimilarconditionspresentinallthreecountriesAlthoughsomestudentshadfallenbehindinthecurriculumteacherswereincentivizedtoteachgrade-levelmaterialsConsiderationsofhowthelocalcontextinChicagomightaffectimplementationofatargetedteachingprogramincludedunderstandinghowtheneighborhoodsthatthestudentswerefrommayhaveaffectedtheireducationaloutcomesandthehighcostofteachinginsmallgroupsInChicagoMatchEducationprovidedtwo-on-onetutoringwhichmatchedstudentswithwell-educatedindividualswhohavenothadformalteachertrainingforalower-costmodelTheresultswerepositiveStudentswhoweretutoredperformedbetterinmathandweremorelikelytopasshighschoolclasses
ThegeneralizabilityframeworkproposedbyJ-PALisbelowUsethisasatemplateasyouconsiderpotentialpolicysolutions
Decideonapolicytoimplementbasedonlocalfactors
30
SourceTheGeneralizabilityPuzzle
OnceyouhavenarrowedthepolicyoptionsdiscussthequestionsbelowforeachoptionsyouareconsideringThisstepiscriticalasunderstandinglocalcontextissoimportantinsuccessfulpolicyimplementation
CommunityContextQuestionnaire
Localcontext
DoesthisoptionmakesenseinourcommunityIsthisaonesizefitsallapproachIsanyparticulargroupabsentfromthisapproachIfsoisthatok
Youranswers
Effectiveness
WhatwouldweexpecttoobserveifthepilotisaseffectivehereasinCityX
Youranswers
Timing
DoesthetimingmakesenseIsthistherighttimetoimplementthispolicychange
Decideonapolicytoimplementbasedonlocalfactors
31
WherearethepotentialroadblocksthatcouldaffectthescheduleIsthereanywaytomitigatethisHowlongwouldittaketoseeanimpact
Youranswers
Implementation
DowehavethestaffcapacitytoimplementthisArethereadditionalpartnersthatyoucanbringtothetabletosupportimplementationDoesthismakesensegivenourformofgovernment
Youranswers
Authority
DoesourcityhavetheauthoritytoenactthisIfsowhatistheappropriatemechanismAretherelegalbarriersArethereanyexistingpreemptionissuesfromthestateorfederalgovernmentCanthosebeaddressedthroughlegislationadvocacyIstherepoliticalsupportoroppositiontothis
Youranswers
Cost
HowmuchwillthiscostannuallyHowwillwepayforitCanthepolicybeimplementedwithexistingfunding
IfsoisthisfundingreliableHowwouldthispolicyoptionaffectourcitysbudgetintheshortandlongtermWhatotherfundingoptionscanweconsiderIsthereanalternativethatwouldcostlessandgiveussimilarresultsWillanyconstituencybeadverselyaffectediffundingforprogramsthatimpactthemisshiftedtoanotherprogrampolicyorinitiative
Youranswers
Decideonapolicytoimplementbasedonlocalfactors
32
EstablishaframeworktomonitorprogressNowthatyouhavereviewedtheevidenceconsideredlocalcontextandchosenyourpolicyitistimetodeterminehowyourcitywillmonitorprogressofthisnewapproachCitystaffandleadersneedtoestablishtheinfrastructureforongoingcontinuousdecisionmakingbasedonwhetherthepolicychangeisleadingtothedesiredoutcomesforresidentsandwhetherthechangeismeaningfulandrationaltofrontlineworkersSettingarbitrarygoalsatthelocallevelthatconflictwithgoalsatotherlevelsisconfusingtotheindividualsresponsibleforachievingthegoalsGoalsthataresettoohightoachievecanbedemoralizingtostaff
ThissectionincludesstepstosetupavenueinwhichCitystaffandleadershiproutinelyvisitkeyindicatorstodeterminewhatisorisnotworkingandtomakedecisionsinreal-timeFormorereadingonperformancemanagementpleasevisittheGovExPerformanceManagementGettingStartedGuide
InperformancemanagementtermssuchasindicatormeasureandmetricmaybeusedinterchangeablyInthisguidewewilluseindicatorInyourworkthemostimportantthingistobeconsistentinwhichevertermyouuseThisvideomaybehelpfulinunderstandingthisconceptandcommunicatingittoothers
Establishthefundamentalsandgatheryourteam
FirstdeterminewhetherthereisanestablishedvenueorregularmeetingscheduleduringwhichyoucandiscussprogressIfyourcityhasanexistingperformancemanagementstructureonarelatedtopicaddingadiscussionoftheprogressofpolicychangetoyouragendacanbeagreatideaIfthiswillbeastandalonemeetingdecidehowfrequentlytheprojectteamanddecisionmakerswillmeetandputthosemeetingsonthecalendarfortheentirelengthoftheproject
DeterminewhowillstaffandattendroutinemeetingsonCityprogressThesemeetingsmayincludetheMayorandorCityManagerdepartmentheadstaffwhoareimplementingthepolicychangeandanalysts
Identifyindicators
ItiscriticalthatindicatorsbeanimportantcityandcommunitypriorityaswellassomethingtheCityplaysamajorroleinaddressingTheremaybeinstancesinwhichtheCitywillneedtoreachouttoorcollaboratewithexternalpartnerstogetamorecompletepictureofprogressontheoverallpolicychange
Establishaframeworktomonitorprogress
33
Anoutcomeindicatordescribesthedesiredeffectofthetargetedpopulation
ExampleBellevuesfamilyfriendlyneighborhoodsindicatormeasuresthepercentofresidentswhobelievetheyliveinaneighborhoodthatsupportsfamiliesparticularlythosewithchildren
WithyourkeypartnersrevisityourproblemstatementcurrentstateandfuturestateexerciseandbegintoidentifytheoutcomeindicatorsthatwillillustratesuccessFornowchooseasingletop-levelkeyperformanceindicator(KPI)ItiscriticalthattheKPI-thepieceyouwillmeasure-beapriorityforthecommunityandsomethingtheCityplaysamajorroleinaddressing
Outputindicatorsmeasuretheprocessofprogramorservicedelivery
ExampleSanDiegoMayorKevinFaulconersgoaltorepair1000milesofstreetby2020
ReviewingtheadministrativedatasetinventorybegintoparedowntherelevantprogramsanddatasetsthatsupportyourKPIWhatarethemeasuresofeffectivenessforeachprogramTheseareyouroutputindicators
Thediagrambelowillustratestherelationshipbetweenoutputandoutcomeindicators
ThinkaboutimplementationindicatorsasmeasuringstepsintheprogramsrolloutthatarenecessarytoobtaindesiredchangesImplementationoutcomescanfocusontheinterventionstaffdeliveringtheinterventionanorganizationorsystemandcanaddressfeasibilityadaptabilitysustainabilityandcostTwokeyquestionstoconsiderwhendevelopingimplementationindicatorsare
1 Istheinterventionacceptableandappropriatetothecommunityand
2 IsitfeasibletoimplementinterventionXatthistimewiththeseresources
Establishaframeworktomonitorprogress
34
Amoreextensivelistwithdefinitionsofpotentialimplementationoutcomesislinkedhere
UsethetablebelowtochoosethedesiredimplementationoutcomesforyourpilotandtoplanyourmeasurementtechniqueThisshouldincludelevelofanalysis(individualproviderorconsumerorganizationorsetting)yourmeasurementtechnique(surveysobservationinterviewfocusgroupsadministrativedataanalysis)andthephasewhenyouwillcollectthisinformation(earlymidlate)Theremaybemorethanonelevelofanalysismeasurementtechniqueandphaseofdatacollection
Implementationoutcome
Definitionofoutcome
Levelofanalysisandmeasurement
Yourmeasurementtechnique
Phaseofpilotwhenyouwillcollectdata
Usethespacebelowtocaptureyourindicatorwishlist
Indicatorwishlist
Outcomes_____________________________________________________________________________
Outputs________________________________________________________________________________
Implementationoutcomes______________________________________________________________
Setapublichigh-levelgoal
Thehigh-levelgoaliscriticalinestablishingdirectiontogalvanizethepeoplewhowillbedoingtheworkWhendiscussingthegoalusetheMayorsbullypulpitLookforkeymilestonesinCitygovernmenttosharethegoalstatementsuchasthebeginningofaMayoraladministrationaStateoftheCityspeechorotherpublicremarks
GoalsshouldbeldquoSMARTrdquo
Specificaspossible(innamingthepopulationwhowillbebetterofftheactionthecitywilltake)
Measurable
Achievable
Establishaframeworktomonitorprogress
35
Results-focused
Time-bound
ThegoalstatementshouldincludeyouroutcomeindicatorlikeChattanoogasgoalldquoWewillendchronicveteranhomelessnessbyDecember2016rdquoAdiagramtohepyoubuildaSMARTgoalaroundyouroutcomeindicatorisbelow
Component Description Yourturn
Specific WhowillbebetteroffWhatactionthecitywilltake
Measurable Howwillyoumeasuresuccess
Achievable Isthegoalattainable
Results-focused Whatistheexpectedresult
Time-bound When
Setrealisticambitioustargetsandtimelinesforeachindicator
ToestablisharealisticandambitioustargetforyourgoalstatementusetheadministrativedatacollectedforyourchosenKPItoreviewactualCityperformanceinthemostrecentyearcomparedwiththethreeorfourmostrecentyearsworthofdataBegintocalculatesummarystatistics(egtotalminimumaveragemedianmaximum)toidentifytrendsCreatecumulativeandmonthlychartstoidentifytrendsandanomaliesBasedonyouranalysisofhistoricaldataignoretheanomaliesandassumetheproductivitytrendswillfollowthesameoverallpatternfrommonth-to-monthFinallyapplyanincreasedproductivityfactortotheperformanceeachmonthtoforecastfutureprogressBasedonthisanalysischooseatargetthatinspiresimprovementwhileaccountingforreality
Manycitiesthataremorematureintheirperformancemanagementpracticesrecognizetheimportanceoftakingacommunity-levelapproachtoimprovingoutcomesforresidentsThesecitiesarecreatinggoaltaxonomiesthatincludemeasuresrelatedtopublicsafetyhousingandevenrecreationCitiesdriveprogressonthesegoalsbyimplementingstrategiesthatengagemultipledepartmentsandcollaboratewithserviceprovidersinthecommunity
EventuallyyouwilldevelopgoalstatementsfortheoutputsinordertocapturetheefforttoachievetheoutcomecontainedinyourgoalstatementYoursetofindicatorsshouldbecomposedofshort-andinterim-termsupportivemeasureswithaccuratetargetsandrelevanttimelinesAnexamplegoaltaxonomyfromtheCityofLasVegassResultsVegasprogramisbelowUsethistaxonomyasamodelforyourownsuiteofindicatorsandgoalstatements
Establishaframeworktomonitorprogress
36
Yourturn
Thetemplatebelowismeanttohelpyoucraftagoalstatementforeachsupportingmeasure
Goalstatement
Departments
Outcome
Activities
KPI
Target
Timeframe
Conductanddiscussrelevantanalysis
ThrougharoutineseriesofmeetingscontinuallymonitorprogressofKPIssupportingindicatorsandanyotherdatacollectedsuchasqualitativedatatoidentifytrendsthatwillindicatewhetherthechosenpolicyishavingthedesiredimpactThesedatapointsshoulddrivethediscussionduringthesemeetingspointingtoopportunitiesforfurtherinquiryinto
Establishaframeworktomonitorprogress
37
whatisdrivingtheobservedtrendsAnalystsshouldbeencouragedandtrainedtoaskquestionsofthedepartmentsproducingthedataandconductsitevisitswhenpossibletodeepentheirunderstandingofthedataPresentationoftheanalysisshouldfocusononeideaatatimetokeeptheconversationfocusedonkeypointsandspecificopportunitiesforimprovementYoumayconsiderdevelopingapublicfacingdashboardtosharethisprogressmorebroadly
AgeneralcomparisonofthestatusquodatawithwhatwasobservedduringthepilotcanprovidedecisionmakerswithaquickunderstandingofthedifferencesbetweentheoldandnewapproachesalthoughthiswillnotshowcausalityOftenthesecomparisonswillincludeoperationalmetricssuchasdaysordollarssavedbutrememberthattheimpactonpeoplewillbemostimportantIfyouareworkingwithanevaluationpartnertheywillsupportthisstep
Usethespacebelowtosummarizetheresultsofyouranalysis
MeasureTargetvalue(seesectiononsettingtargetsabove)
Pilotvalue(whatdidyouobserveduringthepilot)
Historicalaverage(ifavailable)
Results(WhoisbetteroffIsanyonenegativelyimpacted1-2sentencesreassessingyourgoalstatement)
DocumentyourprocessandprogressincludingfeedbackfromtheparticipantsimplementationpartnersandtheprojectteamDidanythingunexpectedoccurduringthepilotArethereinternalandexternalfactorsthatmayaffectyouroutcomesThisiswheretheimplementationoutcomescanbecanhelpfulIftheinterventiondidnotgoasplannedconsiderwhetheritisbecausetheinterventionwasnotacceptableorfeasible
UsethetemplatebelowcreatedbyDenversPeakAcademytocapturethisinformationIfthe306090daytimelinedoesnotmakesenseforyourcityfeelfreetocustomizeit
CurrentState
FutureState
30Day
60Day
90day
Whatwentwell(+)
Whatdidnotgowellshouldbechanged(-)
IfaquestionordiscussionpointarisesthatcannotbeansweredinthemeetinganassignedstaffmembershouldcapturethequestionduringthemeetingAfterwardthefollowupitemsshouldbesenttorelevantpartnersandthenextmeetingshouldbeginwithadiscussionof
Establishaframeworktomonitorprogress
38
thoseitems
Takeaction
DiscussionsaboutwhetheryourindicatorsaremovingintherightdirectionandwhatmightbedrivingtrendsmayleadtoadiscoverythattherearenotenoughresourcesallocatedtothispolicyorissueTheremaybelimitedfundingforstaffprogrammingortechnologythatisdrivingthedatainthewrongdirectionIfCityleadershipisintheroomthisisagreattimetomakethecaseforadditionalfundingduringthebudgetseasonBycollectinganalyzinganddiscussingdataonthispolicyroutinelyyouarebuildingevidencetomakethecasefortherightleveloffundingtoreachtheestablishedtargets
Establishaframeworktomonitorprogress
39
Assessdataaccessibilityqualityandsecurity
Determineaccessibilityandqualityofdata
OnceyouhavecompiledyourwishlistofindicatorsitistimetodeterminetheaccessibilityandqualityofthedataTopreventcomplicationyoumaywishtoprioritizerelevantindicatorsforwhichyouarealreadycollectingdataAvoidthetemptationofadoptingatechnologysolutiontocollectnewdataatthispointsuchasaphotosurveytoolorotherappFornowyouaretryingtodeterminewhichdatasetsarerelevantaccessibleandhighqualityBesuretoworkwithyourkeyinternalpartnerswhileyouareidentifyingyourmeasuresDepartmentstaffwhoworkonprogramsarefamiliarwiththeindicatorsthatwillaccuratelysupporttheoutcomesanddatabaseadministratorswillbevaluablewhendiscussingaccessibilityandqualityofthedata
Abrieflistofconsiderationsfortheaccessibilityofyourdataisbelow
ArewecurrentlycollectingthisdatasetorwillwebecollectingitforthefirsttimeHowfarbackdoestherecordgo(timehorizon)WouldweneednewtechnologytocollectoranalyzethedataCanthedatabecollectedmanuallyorcanitbeautomatedHowfrequentlyisthedataproduced(continuouslyhourlydailyweeklymonthlyquarterlyyearlyetc)IsthedataavailablefortheentireCitypopulationorasubsetofthepopulationHowgranularisthedata(individuallevelCensusblockneighborhoodcitywideetc)DoesthedatacontainsensitiveinformationIsthereaproxymeasurethatwecanusefornowIsthedatamachinereadable
AsnotedbytheWilsonSheehanLabforEconomicOpportunities(LEO)attheUniversityofNotreDameleveragingadministrativedatacansignificantlyreducethecostsandtimelinesofrigorousevaluationCitiescanestablishaninternaldatasharingagreementacrossdepartmentsincludinglanguagethatenablesadministrativedatatobeusedforresearchpurposesCitiescanalsoidentifyexternaladministrativedata-holderssuchasacountyschooldistrictpolicedepartmenthospitaloranotherstatethatmayhaveoutcomedataimportantformeasuringprogramimpactCitiescanleverageexistingrelationshipsandorworkwithresearchinstitutionstodevelopstreamlineddatasharingprocessesandagreementswiththeseexternalentities
Assessdataaccessibilityqualityandsecurity
40
Evaluatethequalityofyourdata
QualityisasimportantasaccessibilityToevaluatedataqualityyoumaywishtocreatearubricliketheonebelow
Value ValueDescription
3 ExcellentDataQualityThedataiscompleteaccurateandupdatedregularly
2 AdequateDataQualityThedataissomewhatcompleteaccurateandupdatedfairlyregularly
1 PoorDataQualityThedataisnotcompleteaccurateorupdatedregularly
ThisisagreattimetobuildonthedatainventorythatyoustartedinthegatherinformationpageSuggestedadditionalfieldstoaddtoyourdatainventorytemplatearebelow
Program DatasetTitle
DatasetDescription DataSource
Departmentorexternalpartner
DoestheCityalreadycollectthisdata
Whatisthenameoftheprogram
Human-readablenameofthedataset
Shouldbeunderstandablebynon-technicalusers
Whatistheoriginalsystemapplicationfilethathousesit
Departmentorexternalpartnerthatmaintainsthedata
Isthisdataalreadycollected
Updatefrequency-ThefileformatforthedatasetTypicallycsvxlsxshapefiletxtetc
Format-Howoftenthedatasetisupdated
Timerange-Theperiodoftimecoveredinthisparticulardataset
Datause-Howisthisdatasetcommonlyused
Datausers-Whotypicallyusesthisdataset
Dataownerstitle-Titleofthedataowner
Dataownersposition-DataOwnersjobtitle
Dataownersemailaddress-DataOwnersemailaddressWhomanagesthedataandorisresponsibleforgrantingpermissiontoaccessthedataWhounderstandswhatthedatasetincludesandcananswerquestionsaboutit
Dataqualityconcerns-Describetheconcernsyoumayhaveaboutthequalityofthedataset
Assessdataaccessibilityqualityandsecurity
41
Division-DivisionwherethedataismaintainedIfmorethanonedivisionisresponsiblelisttheprimaryownerofthedataset
Sensitivedatacomments-DescribetheissuesthatwouldpreventthecityfromsharingthedatapubliclyCommonsensitivityconcernsincludeprivacyviolationssecurityissuesorhighcostorstaffdemandstopostthedata
Publishingstatus-Thistheplacetoputanycommentsregardingthecurrentstatusofthedataset
Publishedlink-Thisisthelinktothepublisheddatasetontheopendataportalorcitywebsite
Reviewdatasecurityandprivacyprotectionprotocols
OftenapolicychangealsoinvolvesachangeinthedatathatisbeingcollectedhowthatdataisbeinghandledwhohasaccesstoitandahostofotheradjustmentsAsyoutakestockofthedataassetsthatareavailableforyourinitiativeitisvitaltoalsothinkthroughwhatwillbedonetoprotectthatdataandtheprivacyofthepeopleitrepresentsAbigpartofgooddatasecurityinvolvesstrongdatagovernancepracticesthatdefineandidentifysensitivedataandconsciouslyworktobalancethevalueandriskofstoringsuchsensitivedataOnewayofevaluatingyourdatacomesfromcaselawrelatedtotheUSFreedomofInformationActwhichwehavetranslatedtothechecklistbelow
1 Woulddisclosureresultinasubstantialinvasionofprivacy2 Haveweconsideredtheextentorvalueofthepublicinterestandthepurposeorobject
oftheindividualsseekingdisclosure3 Istheinformationavailablefromothersources4 Wastheinformationgivenwithanexpectationofconfidentiality5 Isitpossibletomouldreliefsoastolimittheinvasionofindividualprivacy
TheseriskscouldbemitigatedusingavarietyofstrategiesfromaggregatingdatatoobfuscatingindividualstoredactingfieldsthatmayincludePersonallyIdentifiableInformation(PII)orre-evaluatingyourdatacollectionmethodstofindalternativewaysofgettingtheinformationyouneedThesestepsshouldbecodifiedandmanagedbyadatagovernancecommitteeifpossibleForthesensitivedatathatisdeemedessentialcreateaplanaroundpotentialdatabreachesinordertoquicklyandconfidentlyrespondtoemergencysituations
ProactivelypreparingforthesesituationsisoftenthebeststrategyfordatasecuritybutdonotforgetthetechnicalsideInvolveyourITadministratorsinconversationsaboutdatastorageandaccessensuringthattheywillbeprotectedbyenterprise-gradecybersecuritysystems
Assessdataaccessibilityqualityandsecurity
42
Assessdataaccessibilityqualityandsecurity
43
Designimplementationandevaluation
Designtheimplementation
NowthatyouhaveyouraccountabilityplanandkeyindicatorsitistimetogetspecificaboutyourplanforimplementationThetemplatebelowwillhelpyouthinkthroughthespecificmechanismsofimplementingpolicychange
UnderstandingorganizationsandsystemsiscriticaltounderstandingandmakingdecisionsaboutwhereandhowinterventionsshouldbeappliedApolicyimplementedinasinglesettingmayshowpromisingresultsbutimplementingmultiplepoliciesandprogramsatthesametimewithdifferentactorsresponsibleforeachaspectcanhelpensuresustainablebehaviorchange
InSomervilleMAShapeUpSomerville(SUS)isacitywidepublichealthprogramthatisimplementedacrossmultiplelevelsofinfluenceandsystemsthroughinterventionsbeforeduringandafterschooltoincludechangestobuiltenvironmentsuchasbikelanesregionalmasstransportationandphysicaleducationequipmentinschoolsandgymstoincreasephysicalactivitywellnesspoliciesandschoolgardensnutritioneducationduringandafterschoolandhealthyrestaurantprogramstosupporthealthyfoodandhealthyeatinghabits
Usethetemplatebelowtoplanyourimplementation
Policyimplementationplanningtemplate
ListthespecificdesiredoutcomesInthepreviousstepyouworkedwithdepartmentstafftoestablishspecificmeasurableoutcomesandinterimindicatorsPrioritizetheindicatorsthatwilldemonstrateprogressUsethespacebelowtocapturethedesiredoutcomesofthepolicychange
ExampleShapeupSomervillebeganwiththegoalofpreventingobesityin1stthrough3rdgraders
Yourturn
____________________________________________________________________________
____________________________________________________________________________
Designimplementationandevaluation
44
SpecifythetargetpopulationAreyoudesigninganinterventionthattargetsonlycertainsitesAreyouplanningonimplementingaprogramorinitiativeforaspecifictargetaudienceClearlyspecifythepopulationyouplantotarget
IfyouplantoconductarigorousimpactevaluationasnotedbytheWilsonSheehanLabforEconomicOpportunitiesitisimportanttoestablishconcreteobjectiveeligibilitycriteriaandensurethatthesecriteriaareappliedconsistentlybyallstaffduringscreeningandintakeItisalsovitaltoidentifytheindividualswhoareeligibleforaparticularprogramorserviceevenifeveryoneeligiblecannotbeservedbyexistingresourcesThishelpscitiesidentifyanyunmetneedsandguidefutureresourceallocationwhilealsocreatingthepossibilityofarigorousimpactevaluationbyidentifyingapotentialcomparisongroupFinallywhenconductingscreeningandintakeitisimperativetocollectandmaintainbasicinformation(egnamedateofbirthgender)aboutallprogram-eligibleindividualsnotjustthosewhoparticipateintheprogram
ExampleIntheShapeUpSomervilleexampletheprogrambeganbytargeting1stthrough3rdgraders
Yourturn
____________________________________________________________________________
____________________________________________________________________________
DefinethelocationWherewillthepolicychangetakeplaceWillimplementationtakeplaceatonesiteormultiplesites
ExampleShapeUpSomervilleinterventionsweredeliveredinschools
Yourturn
____________________________________________________________________________
____________________________________________________________________________
ClarifyimplementationpartnersandarticulatetheirrolesWhatarethekeyorganizationsinstitutionsorpeopleresponsibleforimplementationTheprojectteamshouldbeasclearandspecificaspossibleinprovidingadescriptionoftheconcreteactivitiesthatimplementationpartnersaremeanttocarryoutPracticeprofilesmaybeahelpfultooltohelpdefinewhatisrequiredoftheactorswhoareresponsibleforimplementingtheintervention
Designimplementationandevaluation
45
ExampleInSomervillemayoralleadershipandengagementofmultiplecityagencies(publichealthprimarycareschoolsCambridgeHealthAllianceetc)streamlinedandconnectedthevariousShapeUpSomervilleinitiativesandgrantswhichhelpedtokeepthetalentandenergyfocusedonhealthyeatingandactivelivingforallsectorsofSomervilleSUSpartnersultimatelyincludedschoolsnonprofitsCityandstateagenciesandnationalfoundations
Yourturn
____________________________________________________________________________
____________________________________________________________________________
DescribetheinterventionWhatarethestepsactivitiesorprocessesRememberthatthemechanismthespecificstepsorprocessthatleadtobehaviorchangeiskeytounderstandingtheeffectivenessofthepolicychangeandsharingtheresultswithothers
ExampleGoalsofShapeUpSomervilleweretoinfluenceeverypartofanearlyelementaryschoolchildrsquosdayndashbeforeduringandafter-schooltopromotehealthyeatingphysicalactivityandtheirweightstatusInterventionincludedfoodserviceenhancementstoimprovebreakfastandlunchandstaffprofessionaldevelopmentwalktoschoolactivitiesSUSclassroomcurriculumandprofessionaldevelopmentschoolwellnesspolicydevelopmentSUSafter-schoolcurriculumandprofessionaldevelopmentoutreachandeducationtohomethroughmaterialsforumseventsldquoSUSapprovedrdquorestaurantscommunityoutreachandcapacitybuildingthroughpolicydevelopmenttrainingsmediaplacements
Yourturn
____________________________________________________________________________
____________________________________________________________________________
ClarifytimelineandmilestonesfordeliveryandevaluationHowfrequentlywilltheservicesinterventionbedeliveredHowlongwilltheprogramrunWhatisthetimetoexpectedimpacts(eg12monthsafterprogramcompletion)
Designimplementationandevaluation
46
ExampleShapeUpSomervilleledbyTuftsUniversityFriedmanSchoolofNutritionScienceandPolicyconductedaquasi-experimental(nonrandomized)interventionstudyamong1stthrough3rdgradestudentsinSomerville(intervention)andtwocomparisoncommunitiesfor9-monthsduringoneschoolyear(fall2002ndashspring2003)ThestudyshowedpositiveassociationoftheinterventiononchangeinBMISUSreleasedreportssummarizingevaluationsprogresstodateplansforadditionalprojectsandimpactstenandfifteenyearsintotheinitiative
Yourturn
____________________________________________________________________________
____________________________________________________________________________
PlanyouranalysisTheprevioussectionaddressedestablishinganaccountabilityframeworktomonitorprogressandimplementationDeterminewhichindicatorswillbediscussthroughthatprocesswhowillsharethedataandanalysisandhowfeedbackfromthoseconversationswillbeincorporatedbytheprojectteam
ExampleTheMayorrsquosofficeimplementedadata-drivendecisionmakingapproachcalledSomerStatwhichwassynergisticwiththeapproachestakenbytheresearchandevaluationteamsdeployedforthespecificgrants
Yourturn
____________________________________________________________________________
____________________________________________________________________________
PlanforsustainabilityIfthepolicyandprogramchangeissuccessfulhowwilltheprojectteamsustainandexpandtoadditionalsitesorpopulations
ExampleAlthoughmostofthepartnershipsforgedintheearlyyearsweremaintainednewgrantsandcollaborationsexpandedattentiontoolderpublicschoolstudentsactivetransportationandtheGreenLinesubwayextensionthebuiltenvironmentfarmerrsquosmarketeffortssuchasldquoGrowninSomervillerdquowithUnionMainStreeteconomicdevelopmentandhealthdisparitiesaswellasdatacollectionmonitoringandsurveillance
Yourturn
Designimplementationandevaluation
47
____________________________________________________________________________
____________________________________________________________________________
Pilotplanningtemplate
IfyouplantoconductapilotfirstdeterminethepurposeofthepilotDoyouwanttoobservewhetherthepolicychangeresultsinthedesiredbehaviorchangeAreyouconcernedaboutfeasibilityscalabilityorcostDoyouwanttotestwhetheryourdesiredresultscanbeachievedthroughthispolicyapproachClarifythegoalofthepilotandthequestionsyouhopetoanswerThiswillhelpyouidentifythekeyindicatorsthatyouwillusetomeasurewhetherthepilotissuccessfulBeforebeginningestablishabaselineofcurrentperformanceandsetyourtargetsaccordinglybasedonwhatyouhopetoaccomplishthroughthepilot
NextestablishthepilotplanforyourprojectteamincludingmilestonesforteamcheckinstodeterminewhethertomoveaheadorchangecourseThisisseparatefromthemeetingsthatwilloccuraspartoftheaccountabilityframeworkUsethespacebelowtocreateyourtimelineanddevelopyourmeetingschedule
Task Personresponsible Startdate Checkindate Deliverable
Planyourevaluation
WhileyouaredesigningtheimplementationitisalsotimetobegintodevelopyourplanforanalysisAsnotedbytheWilsonSheehanLabforEconomicOpportunitiesdecisionsaboutprogramsetup-includingoutreachscreeningintakedatatrackingandenrollment-candeterminewhetherornotarigorousimpactevaluationwillbepossibleAsdiscussedcityleadersmayfindithelpfultoincludeevaluationpartnersinearlyconversationsaboutprogramsetupDuringpreviousstagesyoumayhaveworkedwitharesearchertoplanyourproposedpolicychangeYoushouldbeclearaboutthepurposeoftheevaluationanddevelopanevaluationplanBelowareadditionalconsiderations
Causationvstrendanalysis
IntheEstablishaframeworktomonitorprogressstagetherewasguidanceontrackingtrendsovertimeItisimportanttonoteherethatmonitoringdataisdifferentfromanalyzingacausaleffectTrendanalysiswillillustratethatchangesareoccurringandmayleadtoquestionsthatcouldbeansweredbyyourevaluationbuttrendanalysisalonedoesnotprovideevidenceaboutwhatisdrivingthosechangesYoumayseechangesinyour
Designimplementationandevaluation
48
outcomeindicatorsbasedonvariablesoutsideoftheCityscontrolForinstanceachangeinfederaltradepolicycanhaveanimpactemploymentratesinyourCityoutsideofyourCitysnewjobtrainingandplacementprogram
ToaddresscausalityyoumustcomparetheoutcomesofindividualswhoreceivedtheinterventionwiththosewhodidnotWithoutacomparisongroupofsimilarindividualswhowerenotexposedtotheinterventionitisimpossibletoknowwhichfactorsareresponsiblefortheoutcomesRandomizedControlTrials(RCTs)inwhichindividualsarerandomlyassignedtotreatmentandcontrolgroupsarethegoldstandardinprovingcausalityIfyoucandemonstratethatoutcomesimprovedforthosewhoreceivedtheinterventionanddidnotimproveatallorimprovedlessforacomparisongroupyoucanbesomewhatcertainthattheinterventionwasthecause
Examplesofothernon-causalimpactevaluationsthatarerigorousandreliableincludecertainquasi-experimentalstudiesusingcomparisongroupdesignssmartinterruptedtimeseriesanalysesthatmodeltrendsovertimeinwaysthatallowestimationofinterventioneffectswelldesignedpre-posttestanalysesneedsassessmentsimplementationstudiesetcFormoreinformationondifferenttypesofevaluationsseetheSageResearchMethodswebsite
Lookfordifferencesingroups
BuildintimetoalsoexaminethepilotdatatoseeifgroupsareexperiencingdifferentresultsReviewingperformancebasedonoutcomemeasuresforapopulationinaggregatemayhidethesubtlewaysthatsomepeoplemaybeaffecteddifferentlysoitisnecessarytocarefullyanalyzethedatafordisparateoutcomesForinstancetheMovingtoOpportunity(MTO)studythatexaminedtheimpactofhousingvouchersonfamilieswholivedinhigh-povertyneighborhoodsfoundthatchildrenwhomovedbeforetheageof13hadbettereducationandhigherincomesasadultsHoweveradultsdidnotseeimprovededucationandemploymentoutcomes
Disaggregatingdataintomorespecificcategoriesofraceandethnicitycanhelpunderstandtrendsanddisparitiesinamoregranularwaywhichcanhelpdecisionmakerstargetpoliciesandinterventionsinamoreequitableway
EarlierintheRoadmapweworkedtoidentifystakeholdersandwhomightbepositivelyandnegativelyimpactedbyachangeinpolicyUsetheresultsofthatexercisetoinvolvethecommunityininterpretingdataanalysisandtogatherqualitativefeedbackfromthecommunitytocallattentiontoissuesForinstanceanincreaseinyouthdetentionmaybeexplainedbyinefficienciesorlackofaccesstotransportation
Designimplementationandevaluation
49
Designimplementationandevaluation
50
Shareyourstoryandsuccess
Sharetheresults
Storytelling
AnintegralstepinyourstrategyissharingtheresultsofyourpilotorpolicychangewithyourcommunityThisincludesdevelopingmessagingaroundtheworkofyourorganizationandcreatingaplantodisseminateyourstoriesWhytellstoriesaboutyourwork
Informandorunderstandasubjectorsituation
Giveneededcontextofdataorbitsofinformation
Relatablestoriesarememorableforreadersviewers
ReadersorviewerswillseetheinformationasmorecompellingwhenthereisademonstratedandrelatablechangeorimpactonthemselvesortheirfriendsneighborsandcolleaguesStoriesareaboutpeopleandgovernmentisfundamentallyabouthelpingpeopleSotellthestoriesofthepeopleyouandyourcitycolleagueshelped
Keepthesetipsinmindasyoudevelopyourstorywithyourteam
WhatwereyourfindingsandwhyshouldanyonecareTellyouraudiencehowtheproblemwassolvedWhathastheimpactbeenonresidentsandcitystafftodateWhatwasthesuccessandweretheremultiplesuccessesalongthewayIfyouhavepreliminaryresultsordatashowingimprovementtosharethisisthetimetodoit
WhowasinvolvedandimpactedReviewthelistofstakeholdersHowwereeachofthemimpactedbytheprogramHumanizeyourstoryandimpactbyaddingadditionalvoicesThesevoicescanbeoneofthefollowing
Apersonorfamilywhowaspositivelyaffectedbythechangesuccess
Aninvolvedthird-partydescribingboththeproblemandthesolution(ieanorganizationnonprofitprovidersbusinesstradeassociationsportsteametc)whoispartofthecommunitythatthegovernmentworkedwithonthechangesuccess
Someonefromthegovernmentitselfwhocanspeakaboutthechangesuccess
TellmoreaboutyourcityDescribeyourcityincludingpopulationdemographicsindustriesetcWhatareitschallengesWhatisitknownforWhatmakesituniqueHowisitsimilarordifferentfromothercitiesintheUS
Shareyourstoryandsuccess
51
DescribeyourcityrsquoschallengeWriteoutthechallengeyourcityisfacing(asdefinedinyourproblemstatement)HowdoesitaffectresidentsHowdoesthisproblemaffecttheworkofcitystaffmembers
AddthedetailsProvidedetailsonyourappliedresearchstrategyTalkaboutwhatyourcityhasdonenthepasttoresolvetheissueDiscusshowyourstrategyhaschangedwhenyoubegantoadoptthisapproachWasmoneysavedorinvested
Usethisstorymapexercisetoorganizeyourthoughtsandguaranteethatyouincludealloftheelementsofatraditionalstory
AfteryoudevelopyourmessagingcreateacommunicationsmarketingandoutreachstrategydesignedtospreadyourmessageYourstrategyshouldwalkyouthroughthefollowingkeyactivities
1 IdentifyandprofileyouraudienceWhencreatinganddisseminatingyourmessageitisalwaysgoodtothinkaboutyouraudienceWhoexactlywouldbeinterestedinwhatyouhavetosayThegeneralpopulationofthecityPeoplewhoworkinnonprofitsBusinessownersItmattersbecausenoteveryonewholivesinacitymaybethetargetoftheinformationBusinessownersmaybemoreinterestedindatashowingeffectsonthebusinessdistrictthanothersandsoonWewillcoverthislaterbutthisstepwillalsohelpyoutothinkofthemosteffectivewaytoreachyouraudience
2 Defineyourcalltoaction(ifyouhaveone)SpendsometimeconsideringwhyyouareinterestedintellingyourstoryDoyousimplywanttoupdatethepubliconyourprogressorcelebrateyoursuccessesAreyouinterestedinyouraudiencechangingitsbehaviororsupportingaspecificinitiativebeinglaunchedbyyourorganizationIfyouwanttohaveyouraudienceactorbehaveinacertainwaybespecificandincorporateitinyourmessaging
Shareyourstoryandsuccess
52
3 SelectperformancemeasuresThinkabouthowyouwoulddetermineifyourcampaignissuccessfulandselectappropriateperformancemeasuresYouwillrevisitthesemeasuresattheendofyourcampaigncommunicationsplanimplementationtodeterminewhetheryouneedtomodifyyourcommunicationsandoutreachstrategyforthefuture
4 MethodstocommunicateyourstoryTherearemanywaystocommunicatesuccessofyourprojectHerearesomesuggestionsonreachingaudiencesItisimportanttodeterminewhichmethodsarethemosteffectiveforyourdesiredaudiencetoreceiveinformationDoyourresearch(PewResearchCenterhasexcellentstudiesonthereachofmediadigitalmediaandmarketing)ForexampleifseniorsareyourtargetpopulationandtheyarelessinclinedtouseInstagram(thatistrue)youmaywanttoconsiderFacebookorthenightlylocalnewsbothofwhichareusedmoreoftenbyseniorsBelowisalistofsomemethodsandtoolsyoucanuse
i Pressreleasestolocalnewspapersandtvstations
ii ShortsmartphonevideosAgreatwaytoengageanaudienceisthroughvideoAskonequestionofsomeonewhoispartofthestoryMaybethereisasocialworkerwhocandiscusshungerinyourcityThesevideoscanbereusedonmultipleplatforms
iii Long-formwriting
iv TellingindividualstoriesthroughaseriesofphotosandshortnarrativesforpublicationonFacebookorInstagram
v Op-edsopinionpiecesplacedinthelocalpaper
vi Emailupdatestoyourlist
vii Connectingwithappropriateorganizationsthatareinvestedinthetopicandworkwiththecommunityandaskthemtodisseminateinformationtotheirmembers
viii EmployingldquoWordofMouthrdquo(WOM)tacticsbycreatingtownhallsorfreeeventsforresidentstoattendintheirownneighborhoods
5 DeterminethetypeofcollateralUsuallycampaignsthattargetresidentsrequireseveralavenuestoreachthemPeoplearenotalwaysinthesameplaceorspaceHavingakeenunderstandingofyouraudiencewillassistyouwithyourchoicesonwhatneedstobedevelopedSometimesthatmeansimagesforsocialmediaaonepageflierthatgivespeopleinformationonattendinganeventoranin-depthbrochure
Shareyourstoryandsuccess
53
6 ImplementyourstrategyKeepinmindstrategiescanbefluidSometimessomethingmaynotturnoutasexpectedevenifduringplanningsessionsitwasldquosupposedtordquoItisokYoucananalyzewhathappenedduringadebriefsessionandonceyoursurveysandothermeasurementshavebeenfinalized
Visualizations
Withtherightvisualizationyoucanreviewandshareinformationinsightsanalysisperformanceandtrendsallofwhichisvaluabletoacross-sectionofthepublicButwhichvisualizationisrightforyouTochoosetherighttypeofchartgraphdeterminethetypeofquestionyouwanttoanswerSomecommoncharttypesarebelowandmoreinformationcanbefoundhere
Exploreadditionalanalyticaltechniques
Duringthegoal-settingphaseyouestablishedoutputoutcomeandimplementationmeasuresBasedontheresultsofyourpilotyoumayneedtoadjustyourmeasuresWiththepilotdatainhandyoumayhavesomeideasofhowfutureanalysescouldbedifferentandpotentiallyimprovethequalityofyourprogramTakesometimetobrainstormotherrelevantdatasetsthatyoudidnothavethetimetopursueearlierincludingmessydepartmentaldatathatneedsmoreattentionorsourcesofpublicdatafromthestateandfederalgovernmentnonprofitsandevenprivatecompaniesthatcouldsupplementyourunderstandingofanissueUsingmachinelearningtechniquesyoumightbeabletobetter
Shareyourstoryandsuccess
54
predictdemandforhomelessservicesusingNOAAweatherforecastspreemptwatermainbreaksusingsoilstratigraphyorinvestineconomicdevelopmentbymonitoringrealestatelistingdata
Advancedanalyticaltechniquescanalsohelpyourimplementationrunmoreeffectivelybyintegratingdata-informedtargetinganddecisiontoolsintotheprocessByautomatingroutineworkflowsandsurfacingsubtletrendsthesetoolscanhelpyouiterateinamoreprecisemannerandprovidekeystrategicguidancetofront-lineemployeesthatwouldotherwisebeunavailableThesesortsofdata-science-enhancedprocessescanbeclassifiedasoneofsixprojecttypesasexploredfurtherbytheNewOrleansLANOLAlyticsteamFormoreguidanceonscopingasimilarprojecttheGovExadvancedanalyticsscopingguidegivesyouastep-by-stepoverviewofhowtobesuccessful
Usethespacebelowtolistyourideasforpossibleadvancedanalyticsprojectsbytype
Typeofproject Ideas
Findingtheneedleinthehaystack
Prioritizingforimpact
Earlywarningtools
Betterquickerdecisions
Optimizingresourceallocation
Experimentingforwhatworks
Keepthefeedbackloopgoing
GatherinformationonpublicsentimentaboutpolicychangethroughfocusgroupssurveysandpublicmeetingsledbyadvisorygroupmembersNotewhethernewinformationisavailablethatcanhelpanalyzeorcontextualizeyourprogress(egfederalorstatedatareports)Sharetheresultsofyourpilotandthepolicychangeprocesswithothercitiesandresearchers
Shareyourstoryandsuccess
55
WorkingwitharangeofevidencePresentlythereisalotofattentionandinvestmentinevidence-basedpolicymakingincludingthefederalbipartisanCommissiononEvidence-BasedPolicymakingthePewCharitableTrustsreportsandtoolkitsonHowStatesEngageinEvidence-BasedPolicymakingandtheLauraandJohnArnoldFoundationsupportedEvidence-BasedPolicymakingCollaborativeManyoftheseinitiativesfocusonrigorousevaluation--meaningrandomizedcontrolledtrials(RCT)orotherexperimentalmodels--asthebestkindofevidencetouseHoweverthisbodyofevidenceisofteninsufficientforthespecifickindsofquestionsthatgovernmentscanhavewhenmakingadecisionaboutpolicyInthisguideweproposeanapproachtosupportgovernmentsinincorporatingadditionaltypesofevidenceintotheirdecisionmaking
OnewaytounderstandarangeofevidenceistouseNestarsquosStandardsofEvidencetoframeanddefineevidenceTheBritishnonprofitNestaworkstoinjectpositivechangeintothewayUKgovernmentpoliciesandprogramsfunctionOneoftheirkeyareasofworkhasbeendevelopingWhatWorksCentresorganizationsthatcollectcommissionsynthesizeanddisseminateevidenceaboutinterventioneffectivenessacrossarangeofsocialimpactareasNestadevelopedthefollowingStandardsofEvidencetoprovideausefulwaytodocumenthowdifferentkindsofinformationdemonstratedifferentlevelsofevidence
Workingwitharangeofevidence
56
Theimportanceofpotentiallybiasedinformation
Onechallengepresentedbyanevidenceclassificationthatacknowledgesthevalueofnon-scientificformsofevidenceistofigureouthowtobalanceallofthepiecestogetherwhenyouusethemTounderstandhowtodothiswehavetothinkabouthowtopositionpotentiallybiasedinformationindecisionmaking
WhenwelookatinformationwhichhasnotbeenrigorouslyevaluatedweacceptthatthisinformationmightbebiasedThewholereasonweconductrigorousevaluationistoremovebiasfromourobservationsWeconductstructuredtrialstoproduceresultsthatareconsistentandgeneralizableIfafindingisunbiasedthisaccuracymdashthislackofbiasmdashcanbeprovedbecauseanyindependenttestercanreproduceitwhentheyfollowanidenticalprocess
YoucanlookatpotentiallybiasedinformationasjustinformationthatisearlieroninitspathtorefinementreplicationandverifiedaccuracyApieceofinformationbasedonanyoneindividualrsquosobservationsislikelytobebiasedbecauseitiscomingfromjustonesourceBiasisreducedbygatheringtogetherlotsofindependentsourcesofinformationandfiguringoutitsmostcommonattributes
MaybethebiasedinformationmakesaclaimthatistoolargemdashitsayssomethingistrueunderallconditionswhenitisreallyonlytrueundercertainconditionsorthataneffectofaninterventionisbiggerthanitstrueaverageorthataneffectlastsovertimewhenitdoesnotPerhapschangingtheclaimtobemorespecificmdashcriticallyreviewingtheinformationtoseeifspecificconditionsmustbemettoachievetheoutcomesmdashwouldmaketheclaimmoreaccurate
IfyoutestedyourclaimfounditdidnotholdupinsomecaseschangedittesteditagainuntilyourclaimrepeatedlyheldupthenyoucoulddemonstratethattheclaimwasnolongerbiasedbutaccurateThisprocessoftestingimprovingtestingandvalidatingiswhatwegetfromstructuredscientificstudy
Withindividualpiecesofinformation--crediblehypothesescasestudiesexpertopinion--youcannotassumethatkindofaccuracyAtthesametimetheseearlyobservationsarefoundationaltoanyreplicablescientificobservationsWewillnevergetanywhereifwedonotstartwiththesepiecesofinformationeveniftheymightbebiased
Combininglevelsofevidencetogetthemostdetailedinformationforyourcity
ItisespeciallyimportanttoacceptdifferentformsofevidencewhenyoufacequestionsthathavenotyetbeentestedbyindependentstudiesSomeofthemostimportantquestionsrelatetothingsthatareveryspecifictoyoursituationYoufaceaparticularcombinationof
Workingwitharangeofevidence
57
issuesactorstrendsandresourcesthatisultimatelydifferentfromeveryotherplaceintheworld
ThisisimportantbecauseaspolicypractitionersandnotscientistsyoucaremoreaboutyourspecificcasethanahypotheticalaveragegeneralcaseYoucarelessaboutthetotallikelyrangeofimpactsfromapolicythanhowthissingleimplementationwillworkYouwantinformationaboutpoliciesthatwillprovideyouwiththebestpossiblepredictionabouthowtheywillworkinyourcontext
ByputtingdifferentkindsofevidenceintothesameframeworkNestarsquosStandardsofEvidenceimplythatwhenyoucannotgetrigorouslytestedexperimentallyverifiedinformationitisappropriatetoworkwithwhatwedohaveThisisrelevantforthemunicipalimplementationcontextbecausenostudypulledfromaliteraturereviewwillaccountforallofthevariablesyoufaceinyourowncontext
Intheabsenceofhavingastudythatpredictshowaninterventionworksinyourspecificcontextyouwillneednon-scientificinformationtotrytofillinthosegapsYouwillneedtogatherstoriesfromresidentsstaffmembersandotherstakeholdersthatsitonLevels1or2ofthestandardsofevidence
Someofthekindsofinformationyouwillprobablygetmoreeasilyfrompeoplethanstudiesinclude
howpeopleincitieslikeyoursreactedtoaninterventioninpublicforumshowpeoplewhoimplementedaninterventioninacitylikeyoursfiguredouthowtogetitpassedhowthecitycommunicatedpubliclyabouttheapparentbenefitsanddrawbacksoftheinterventionthepoliticalacceptabilityoftheintervention
Youcanremainsimultaneouslyawarethattheinformationyouaregatheringthroughindividualstoriesisbiased--notgeneralizableinformedbyspecificperspectives--whilealsorecognizingthathavingsomeinformationaboutsomethingimportantisbetterthannoinformationEvidencefromthefirsttwoevidencelevelsareessentialbuildingblocksofknowledgeandperfectlyusefulwhenyoucannotgetdetailedunbiasedknowledge
Youhavetostartsomewhereandyoushouldalwaysstartwithevidenceyouhaveaboutyourveryspecificconditionsthatyouknowareimportantforsuccess
Workingwitharangeofevidence
58
Thankyou
Interviewswithcities
GovExheldsemi-structuredinterviewswithsixstaffmembersinfourcitiestoestablishtheinitialstepsoftheRoadmapWeaskedrespondentstotalkaboutaninitiativeorpolicythattheyhavebeenresponsibleforimplementingintheircityRespondentsfromthreecitiesspokeabouttheireffortstoaddresshomelessnessandrespondentsfromonecityspokeaboutblightAfterrespondentsidentifiedtheproblemtheyspokeabouthowthecityprioritizedthisissuethevariousstepsinvolvedandbarrierspresent
GovExconductedtextanalysisoftheseinterviewswhichrevealedthatpeoplewasthemostcommonlyusedtermduringtheseconversationswhendiscussingthestepstochangeaprogramorpolicyThisisathemethroughoutthisRoadmapThefirstsectionbeginswithidentifyingtherelevantpeopleTheconversationsaroundevidencearefocusedonhumanbehaviorratherthanmethodologyorstrengthofevidence
Thankyou
59
OncewehadadraftoftheRoadmapwefollowedupwithcitiesonanindividualbasistolearnmoreaboutwhatwouldbehelpfulWearegratefultothefollowingcitystafffortheinterviewsandindividualconversations
MichaelBaskinChiefPolicyOfficerChattanoogaTN
MelissaBridgesPerformanceandInnovationCoordinatorLittleRockAR
VictoriaCarreoacutenAdministrativeOfficerLasVegasNV
TyroneGrandstrandCommunityDevelopmentPlannerFargoND
TibLaughlinDirectorofGeneralServicesKansasCityKS
TimMorelandDirectorofPerformanceManagementandOpenDataChattanoogaTN
CharlesRosendahlManagementAnalystDentonTX
Thankyou
60
MelissaSiebenAssistantCountyAdministratorKansasCityKS
GinaThompsonDirectorofStrategicServicesLewisvilleTX
JHUFacultycollaboration
GovExalsoconvenedthecitycontactsandfacultypartnersinaseriesofAdvisoryGroupmeetingsThesemeetingsservedtoconnectcitiesandfacultytounderstandoneanothersperspective
StefanBaralBloombergSchoolofPublicHealth
AshleyBernerJHUSchoolofEducation
MelissaDavey-RothwellBloombergSchoolofPublicHealth
JonLinksBloombergSchoolofPublicHealth
TamarMendelsonBloombergSchoolofPublicHealth
KristinMmariBloombergSchoolofPublicHealth
KeshiaPollackPorterBloombergSchoolofPublicHealth
JoshSharfsteinBloombergSchoolofPublicHealth
LizStuartBloombergSchoolofPublicHealth
LindsayThompsonCareyBusinessSchool
DanielWebsterBloombergSchoolofPublicHealth
Partnersindataandevidence-basedpolicy
Wealsospokewithpartnerswhoareexpertsinthefieldofevidence-basedpolicywhosharedtheirknowledgeandresourcesofcurrentandfuturethinkinginevidence-informedthinking
WendyBarrenoAssociateDirectorWilsonSheehanLabforEconomicOpportunitiesUniversityofNotreDame
TiffanyDavenportDirectorofDataPartnershipsBaltimoresPromise
TomiHiersDirectorBaltimoreCivicSiteAnnieECaseyFoundation
ChrisKingsleySeniorAssociateAnnieECaseyFoundation
KathrynOliverAssociateProfessorLondonSchoolofHygieneandTropicalMedicine
Thankyou
61
KristenWatkinsSeniorPolicyAssociateJ-PAL
GovExParticipants
GovExAnalystMiriamMcKinneyanalyzedtheinterviewtranscriptsandconductedtextanalysisSeniorImplementationAdvisorEmilyShawwillintegrateRoadmapactivitieswithadditionalservicedeliveryactivitiesatGovEx
Thankyou
62
ReferencesAbrahamKGampHaskinsR(2017)Thepromiseofevidence-basedpolicymakingReportofthecommissiononevidence-basedpolicymakingWashingtonDCCommissiononEvidence-basedPolicymakingBigmove2Extendingthegreen(nd)Retrievedfromhttpswwwcityoflewisvillecomdoing-businesslewisville-2025big-move-2-extending-the-greenBatesMAampGlennersterR(2017)ThegeneralizabilitypuzzleRetrievedfromhttpsssirorgarticlesentrythe_generalizability_puzzleBuildwithnotfor(2018January)RetrievedfromhttpsbuildwithnotforwordpresscomCairneyPampOliverK(2017)Evidence-basedpolicymakingisnotlikeevidence-basedmedicinesohowfarshouldyougotobridgethedividebetweenevidenceandpolicyHealthResearchPolicyandSystemsdoi101186s12961-017-0192-xCairneyPOliverKampWellsteadA(2016)TobridgethedividebetweenevidenceandpolicyReduceambiguityasmuchasuncertaintyPublicAdministrationReview76(3)399-402Retrievedfromhttpdxdoiorg101111puar12555Centreforpublicimpact(nd)RetrievedfromhttpswwwcentreforpublicimpactorgCernaL(2013)ThenatureofpolicychangeandimplementationAreviewofdifferenttheoreticalapproachesOrganisationforEconomicCo-OperationandDevelopmentRetrievedfromhttpswwwoecdorgeduceriTheNatureofPolicyChangeandImplementationpdfColoradoSprings(nd)Retrievedfromhttpscoloradospringsgovcommunicationspagewhat-expect-when-you-speakupContinuumofcare(coc)program(nd)RetrievedfromhttpswwwhudexchangeinfoprogramscocCouncilpresidentkicksoff$12millionyouthfund(nd)Retrievedfromhttpwwwbaltimorecitycouncilcomcontentcouncil-president-kicks-12-million-youth-fundDurlakJAampDuPreEP(2008)ImplementationmattersAreviewofresearchontheinfluenceofimplementationonprogramoutcomesandthefactorsaffectingimplementationAmericanJournalofCommunityPsychology41327-350Retrievedfromhttpswwwncbinlmnihgovpubmed18322790Fishbonediagram(ishikawa)(nd)Retrievedfromhttpasqorglearn-about-qualitycause-analysis-toolsoverviewfishbonehtmlFivewhysGettingtotherootofaproblemquickly(nd)RetrievedfromhttpswwwmindtoolscompagesarticlenewTMC_5Whtm
References
63
FligsteinNampMcAdamD(2011)TowardageneraltheoryofstrategicactionfieldsSociologicalTheory29(1)1-26RetrievedfromhttpsociologyberkeleyedusitesdefaultfilesfacultyfligsteinFligstein20McAdam20Soc20Theory20PaperpdHeinkelP(2018January12)CitymullspermitsforbikesharingcompaniesRetrievedfromhttpwwwdentonrccomnewsdenton20180112city-mulls-permits-bike-sharing-companiesHowisareamedianincomecalculated(2012October)Retrievedfromhttpswwwhudexchangeinfofaqs843how-is-area-median-income-calculatedLampkinLMampHatryHP(2003)KeystepsinoutcomemanagementUrbanInstituteRetrievedfromhttpswwwurbanorgresearchpublicationkey-steps-outcome-managementMarshDampSmithM(2000)UnderstandingpolicynetworksTowardsadialecticalapproachPoliticalStudies484-21Retrievedfromhttpjournalssagepubcomdoiabs1011111467-924800247McCannL(nd)RetrievedfromhttplaurenellencomMoultonSampSandfortJR(2016)ThestrategicactionfieldframeworkforpolicyimplementationresearchPolicyStudiesJournal45(1)Retrievedfromhttponlinelibrarywileycomstore101111psj12147assetpsj12147pdfv=1ampt=jdkhhcp4amps=e184c625e1855a6b1615ccb99ee27b1c0b3212d8PaynterB(2018January1)What100citiesarelearningfromeachotherbysharingtheirdataRetrievedfromhttpswwwfastcompanycom40518004what-100-cities-are-learning-from-each-other-by-sharing-their-dataPeakacademy(nd)Retrievedfromhttpswwwdenvergovorgcontentdenvergovenmayors-officeprograms-initiativespeak-performancepeak-academyhtmlPinnockHBarwickMCarpenterCREldridgeSGrandesGGriffithsCJRycroft-MaloneJMeissnerPMurrayEPatelASheikhAampTaylorSJC(2017)Standardsforreportingimplementationstudies(stari)ExplanationandelaborationdocumentBMJOpenRetrievedfromhttpbmjopenbmjcomcontentbmjopen74e013318fullpdfProctorESilmereHRaghavanRHovmandPAaronsGBungerAGriffeyRampHensleyM(2011)OutcomesforimplementationresearchConceptualdistinctionsmeasurementchallengesandresearchagendaAdministrationandPolicyinMentalHealth38(2)65-76doi101007s10488-010-0319-7TandonSD(2018)ProgressincommunityhealthpartnershipsResearcheducationandactionRetrievedfromhttpswwwpressjhuedujournalsprogress-community-health-partnerships-research-education-and-actionProvanKGampKenisP(2007)ModesofnetworkgovernanceStructuremanagementandeffectivenessJournalofPublicAdministrationResearchandTheory
References
64
18(2)229-252Retrievedfromhttpsacademicoupcomjpartarticle182229935895SabatierPA(1988)Anadvocacycoalitionframeworkofpolicychangeandtheroleofpolicy-orientedlearningthereinPolicySciences21(3)129-168Retrievedfromhttpwwwjstororgstable4532139SandstromAampCarlssonL(2008)TheperformanceofpolicynetworksTherelationbetweennetworkstructureandnetworkperformancePolicyStudiesJournal36497-524Retrievedfromhttponlinelibrarywileycomdoi101111j1541-0072200800281xabstractSchwartzMampWilsonE(nd)WhocanaffordtoliveinahomeAlookatdatafromthe2006americancommunitysurveyRetrievedfromhttpswwwcensusgovhousingcensuspublicationswho-can-affordpdfShearerJCAbelsonAKouyateBLavisJNampWaltG(2016)WhydopolicieschangeInstitutionsinterestsideasandnetworksinthreecasesofpolicyreformHealthPolicyandPlanning31(9)1200-1211Retrievedfromhttpsdoiorg101093heapolczw052ThelessonsofadministrativedataHigh-profilepolicy-relevantresearchpoweredbyadministrativedata(nd)Retrievedfromhttpswwwpovertyactionlaborgsitesdefaultfilesresources20161114-The-Lessons-of-Administrative-DatapdfToolsandotherneedsforqualitativeandquantitativemethods(2015February)AgencyforHealthcareResearchandQualityRetrievedfromhttpwwwahrqgovprofessionalsquality-patient-safetytalkingqualityassesstoolshtml
WeaverRCampBagchi-SenS(2013)SpatialanalysisofurbandeclineThegeographyofblightAppliedGeography4061-70
References
65
UniversityPartners
Name Title Affiliation Area(s)ofexpertise
Contactinformation
Examplesofworkingwithnon-academicdecisionmakers
WendyPBarreno
AssociateDirector
WilsonSheehanLabforEconomicOpportunitiesUniversityofNotreDame
LEOsfocusareasincludehealtheducationhousingchildwelfareeconomicopportunitycriminaljustice
wbarrenonduduorleondedu
MelissaDavey-RothwellPhD
AssociateScientist
JohnsHopkinsBloombergSchoolofPublicHealth
Health mdavey1jhuedu410-614-5854
BaltimoreCityHealthDepartmentLeadersMarylandDepartmentofHealthAddictionRecoveryServicesinDCExecutiveDirectorsofcommunityorganizationsacrosstheUSthatfocusonHIVSTIsubstanceabuseandimplementationscience
ElizabethStuart
AssociateDeanforEducationandProfessorDepartmentsofMentalHealthBiostatisticsandHealthPolicyand
JohnsHopkinsBloombergSchoolofPublicHealth
Educationsocialpolicystatisticalmethods
estuartjhuedu
Resources
66
Management
Design-ThinkingResources
EngagementGuideforCivicTech
Journeymap
PerformanceManagementResources
DenverPeakAcademy
Proxymeasuresguide
Talkingaboutperformancewithoutcomplicatingit(video)
SettingPerformanceTargetsguide
Datavisualizationguide
Performancemanagementgettingstartedguide
Exercises
Fishboneexercise
5Whys
AdministrativeData
TheLessonsofAdministrativeData(JPAL)
EvaluationResources
Quantitativeandqualitativemethods(USDepartmentofHealthandHumanServicesAgencyforHealthcareResearchandQuality)
KeyStepsinOutcomeManagement(UrbanInstitute)
RCT-YES
Sageresearchmethods
BestPracticesforMixedMethodsResearchintheHealthSciences
ACCORDChecklist
Pilotandimplementationplanning
Benchmarkingexamplesandproxymeasures
Seattleopendataplaybook
Resources
67
Datasciencescopingguide
Resources
68
AboutGovExGovExhelpsgovernmentsusedataandevidencetoimproveresidentsrsquolivesWesupportgovernmentsinbuildingtheirinternalcapacitytooperatesustainableanalyticspracticeswhichimprovecommunityoutcomesinfieldsfrompublicsafetytoeconomicdevelopmentGovExhasaproventrackrecordofworkinginconcertwithpartnerstoprovidetechnicalassistanceandtrainingatscale
SinceourlaunchinApril2015GovExhas
Trainedover2000governmentemployeesinpersonandonlineintopicsincludingdatamanagementperformancemanagementopendatacommunityengagementandanalytics
Providedtechnicalassistancetomorethan100localgovernmentsacrosstheglobeimprovingtheiruseofdatatoachieveefficienciesinservicedeliveryandbetteroutcomesforresidents
Assessedthecapacityofmorethan130governmentalorganizationstodeterminehowleadersstaffandexternalstakeholdersmanageandusedata
Localgovernmentsareincreasinglyonthefrontlinesofaddressinglargescaleproblemswithglobalimplicationssuchasclimatechangepublicsafetyandeconomicinequality
Whileevidence-basedinterventionstohelpsolvethoseproblemsareavailablelocalgovernmentstaffhavelimitedtimeandcapacitytofindandimplementresearch-basedsolutionsAsaresultcitiescontinuetoworkthewaytheyalwayshave--makingincrementalimprovementstoprogramsbutnotalwayscreatingchangeatasystem-level
ThroughresourcedevelopmentandcollaborationwithJHUfacultyGovExissupportingcityleadersinusingdataandevidencetomakedecisionsandcreatinglastingrelationshipswithresearchers
TheRoadmapwasdevelopedbyGovExaspartofitspartnershipwithBloombergPhilanthropiesWhatWorksCitiesinitiative
AboutGovEx
69
- Title page
- Introduction and purpose of the Roadmap
- Using the Roadmap
-
- Gather information about the problem
- Strategically engage key partners
- Review the evidence and collect details
- Decide on a policy to implement based on local factors
- Establish a framework to monitor progress
- Assess data accessibility quality and security
- Design implementation and evaluation
- Share your story and success
-
- Working with a range of evidence
- Thank you
- References
- Resources
- About GovEx
-
DecideonapolicytoimplementbasedonlocalfactorsTogetthecriticalbuy-infromleadershiptoimplementanewpolicyorprogramtheprojectleadshouldscheduleameetingwithdecisionmakerstopresenttheevidencesurroundingthepolicyoptionscapturedinthetemplateDependingonyourinsightintoyourcityleadershipswillingnesstomakechangeyoumaywishtosharetheresearchyouhaveconductedontherootcausesoftheproblemandsomeofyouranalysisontheimpactofcurrenteffortstodate
Considertheapplicabilityofanapproach
AsdiscussedinthepreviouschapterpolicymakersoftenlookforevidenceofeffectiveprogramsinothercitiesbutoftenthequestionarisesofhowgeneralizableaprogramisfromonecitytoanotherTheAbdulLatifJameelPovertyActionLab(J-PAL)suggeststhatinreviewingevidencefromanotherlocationitishelpfultoidentifythespecificmechanismthatwasused(deliveryofprogramorserviceinformationsharingetc)identifythelocalconditionsthatwereinplaceobservethebehaviorchangethatresultedfromthemechanismandfinallytounderstandlocalcontextandconditions(legalorcostconstraintsculturalnormsetc)Thissetofpresentationelementscanbedrawnfromthelistofdetailscollectedintheresearchtemplate
EducationpolicymakersinChicagoILusedfindingsfromprogramsinKenyaandIndiainwhichchildrenwhohadfallenbehindtheirpeersweregiveneducationalsupporttargetedtotheirleveloflearningThemechanismusedwastargetedteachingandthereweresimilarconditionspresentinallthreecountriesAlthoughsomestudentshadfallenbehindinthecurriculumteacherswereincentivizedtoteachgrade-levelmaterialsConsiderationsofhowthelocalcontextinChicagomightaffectimplementationofatargetedteachingprogramincludedunderstandinghowtheneighborhoodsthatthestudentswerefrommayhaveaffectedtheireducationaloutcomesandthehighcostofteachinginsmallgroupsInChicagoMatchEducationprovidedtwo-on-onetutoringwhichmatchedstudentswithwell-educatedindividualswhohavenothadformalteachertrainingforalower-costmodelTheresultswerepositiveStudentswhoweretutoredperformedbetterinmathandweremorelikelytopasshighschoolclasses
ThegeneralizabilityframeworkproposedbyJ-PALisbelowUsethisasatemplateasyouconsiderpotentialpolicysolutions
Decideonapolicytoimplementbasedonlocalfactors
30
SourceTheGeneralizabilityPuzzle
OnceyouhavenarrowedthepolicyoptionsdiscussthequestionsbelowforeachoptionsyouareconsideringThisstepiscriticalasunderstandinglocalcontextissoimportantinsuccessfulpolicyimplementation
CommunityContextQuestionnaire
Localcontext
DoesthisoptionmakesenseinourcommunityIsthisaonesizefitsallapproachIsanyparticulargroupabsentfromthisapproachIfsoisthatok
Youranswers
Effectiveness
WhatwouldweexpecttoobserveifthepilotisaseffectivehereasinCityX
Youranswers
Timing
DoesthetimingmakesenseIsthistherighttimetoimplementthispolicychange
Decideonapolicytoimplementbasedonlocalfactors
31
WherearethepotentialroadblocksthatcouldaffectthescheduleIsthereanywaytomitigatethisHowlongwouldittaketoseeanimpact
Youranswers
Implementation
DowehavethestaffcapacitytoimplementthisArethereadditionalpartnersthatyoucanbringtothetabletosupportimplementationDoesthismakesensegivenourformofgovernment
Youranswers
Authority
DoesourcityhavetheauthoritytoenactthisIfsowhatistheappropriatemechanismAretherelegalbarriersArethereanyexistingpreemptionissuesfromthestateorfederalgovernmentCanthosebeaddressedthroughlegislationadvocacyIstherepoliticalsupportoroppositiontothis
Youranswers
Cost
HowmuchwillthiscostannuallyHowwillwepayforitCanthepolicybeimplementedwithexistingfunding
IfsoisthisfundingreliableHowwouldthispolicyoptionaffectourcitysbudgetintheshortandlongtermWhatotherfundingoptionscanweconsiderIsthereanalternativethatwouldcostlessandgiveussimilarresultsWillanyconstituencybeadverselyaffectediffundingforprogramsthatimpactthemisshiftedtoanotherprogrampolicyorinitiative
Youranswers
Decideonapolicytoimplementbasedonlocalfactors
32
EstablishaframeworktomonitorprogressNowthatyouhavereviewedtheevidenceconsideredlocalcontextandchosenyourpolicyitistimetodeterminehowyourcitywillmonitorprogressofthisnewapproachCitystaffandleadersneedtoestablishtheinfrastructureforongoingcontinuousdecisionmakingbasedonwhetherthepolicychangeisleadingtothedesiredoutcomesforresidentsandwhetherthechangeismeaningfulandrationaltofrontlineworkersSettingarbitrarygoalsatthelocallevelthatconflictwithgoalsatotherlevelsisconfusingtotheindividualsresponsibleforachievingthegoalsGoalsthataresettoohightoachievecanbedemoralizingtostaff
ThissectionincludesstepstosetupavenueinwhichCitystaffandleadershiproutinelyvisitkeyindicatorstodeterminewhatisorisnotworkingandtomakedecisionsinreal-timeFormorereadingonperformancemanagementpleasevisittheGovExPerformanceManagementGettingStartedGuide
InperformancemanagementtermssuchasindicatormeasureandmetricmaybeusedinterchangeablyInthisguidewewilluseindicatorInyourworkthemostimportantthingistobeconsistentinwhichevertermyouuseThisvideomaybehelpfulinunderstandingthisconceptandcommunicatingittoothers
Establishthefundamentalsandgatheryourteam
FirstdeterminewhetherthereisanestablishedvenueorregularmeetingscheduleduringwhichyoucandiscussprogressIfyourcityhasanexistingperformancemanagementstructureonarelatedtopicaddingadiscussionoftheprogressofpolicychangetoyouragendacanbeagreatideaIfthiswillbeastandalonemeetingdecidehowfrequentlytheprojectteamanddecisionmakerswillmeetandputthosemeetingsonthecalendarfortheentirelengthoftheproject
DeterminewhowillstaffandattendroutinemeetingsonCityprogressThesemeetingsmayincludetheMayorandorCityManagerdepartmentheadstaffwhoareimplementingthepolicychangeandanalysts
Identifyindicators
ItiscriticalthatindicatorsbeanimportantcityandcommunitypriorityaswellassomethingtheCityplaysamajorroleinaddressingTheremaybeinstancesinwhichtheCitywillneedtoreachouttoorcollaboratewithexternalpartnerstogetamorecompletepictureofprogressontheoverallpolicychange
Establishaframeworktomonitorprogress
33
Anoutcomeindicatordescribesthedesiredeffectofthetargetedpopulation
ExampleBellevuesfamilyfriendlyneighborhoodsindicatormeasuresthepercentofresidentswhobelievetheyliveinaneighborhoodthatsupportsfamiliesparticularlythosewithchildren
WithyourkeypartnersrevisityourproblemstatementcurrentstateandfuturestateexerciseandbegintoidentifytheoutcomeindicatorsthatwillillustratesuccessFornowchooseasingletop-levelkeyperformanceindicator(KPI)ItiscriticalthattheKPI-thepieceyouwillmeasure-beapriorityforthecommunityandsomethingtheCityplaysamajorroleinaddressing
Outputindicatorsmeasuretheprocessofprogramorservicedelivery
ExampleSanDiegoMayorKevinFaulconersgoaltorepair1000milesofstreetby2020
ReviewingtheadministrativedatasetinventorybegintoparedowntherelevantprogramsanddatasetsthatsupportyourKPIWhatarethemeasuresofeffectivenessforeachprogramTheseareyouroutputindicators
Thediagrambelowillustratestherelationshipbetweenoutputandoutcomeindicators
ThinkaboutimplementationindicatorsasmeasuringstepsintheprogramsrolloutthatarenecessarytoobtaindesiredchangesImplementationoutcomescanfocusontheinterventionstaffdeliveringtheinterventionanorganizationorsystemandcanaddressfeasibilityadaptabilitysustainabilityandcostTwokeyquestionstoconsiderwhendevelopingimplementationindicatorsare
1 Istheinterventionacceptableandappropriatetothecommunityand
2 IsitfeasibletoimplementinterventionXatthistimewiththeseresources
Establishaframeworktomonitorprogress
34
Amoreextensivelistwithdefinitionsofpotentialimplementationoutcomesislinkedhere
UsethetablebelowtochoosethedesiredimplementationoutcomesforyourpilotandtoplanyourmeasurementtechniqueThisshouldincludelevelofanalysis(individualproviderorconsumerorganizationorsetting)yourmeasurementtechnique(surveysobservationinterviewfocusgroupsadministrativedataanalysis)andthephasewhenyouwillcollectthisinformation(earlymidlate)Theremaybemorethanonelevelofanalysismeasurementtechniqueandphaseofdatacollection
Implementationoutcome
Definitionofoutcome
Levelofanalysisandmeasurement
Yourmeasurementtechnique
Phaseofpilotwhenyouwillcollectdata
Usethespacebelowtocaptureyourindicatorwishlist
Indicatorwishlist
Outcomes_____________________________________________________________________________
Outputs________________________________________________________________________________
Implementationoutcomes______________________________________________________________
Setapublichigh-levelgoal
Thehigh-levelgoaliscriticalinestablishingdirectiontogalvanizethepeoplewhowillbedoingtheworkWhendiscussingthegoalusetheMayorsbullypulpitLookforkeymilestonesinCitygovernmenttosharethegoalstatementsuchasthebeginningofaMayoraladministrationaStateoftheCityspeechorotherpublicremarks
GoalsshouldbeldquoSMARTrdquo
Specificaspossible(innamingthepopulationwhowillbebetterofftheactionthecitywilltake)
Measurable
Achievable
Establishaframeworktomonitorprogress
35
Results-focused
Time-bound
ThegoalstatementshouldincludeyouroutcomeindicatorlikeChattanoogasgoalldquoWewillendchronicveteranhomelessnessbyDecember2016rdquoAdiagramtohepyoubuildaSMARTgoalaroundyouroutcomeindicatorisbelow
Component Description Yourturn
Specific WhowillbebetteroffWhatactionthecitywilltake
Measurable Howwillyoumeasuresuccess
Achievable Isthegoalattainable
Results-focused Whatistheexpectedresult
Time-bound When
Setrealisticambitioustargetsandtimelinesforeachindicator
ToestablisharealisticandambitioustargetforyourgoalstatementusetheadministrativedatacollectedforyourchosenKPItoreviewactualCityperformanceinthemostrecentyearcomparedwiththethreeorfourmostrecentyearsworthofdataBegintocalculatesummarystatistics(egtotalminimumaveragemedianmaximum)toidentifytrendsCreatecumulativeandmonthlychartstoidentifytrendsandanomaliesBasedonyouranalysisofhistoricaldataignoretheanomaliesandassumetheproductivitytrendswillfollowthesameoverallpatternfrommonth-to-monthFinallyapplyanincreasedproductivityfactortotheperformanceeachmonthtoforecastfutureprogressBasedonthisanalysischooseatargetthatinspiresimprovementwhileaccountingforreality
Manycitiesthataremorematureintheirperformancemanagementpracticesrecognizetheimportanceoftakingacommunity-levelapproachtoimprovingoutcomesforresidentsThesecitiesarecreatinggoaltaxonomiesthatincludemeasuresrelatedtopublicsafetyhousingandevenrecreationCitiesdriveprogressonthesegoalsbyimplementingstrategiesthatengagemultipledepartmentsandcollaboratewithserviceprovidersinthecommunity
EventuallyyouwilldevelopgoalstatementsfortheoutputsinordertocapturetheefforttoachievetheoutcomecontainedinyourgoalstatementYoursetofindicatorsshouldbecomposedofshort-andinterim-termsupportivemeasureswithaccuratetargetsandrelevanttimelinesAnexamplegoaltaxonomyfromtheCityofLasVegassResultsVegasprogramisbelowUsethistaxonomyasamodelforyourownsuiteofindicatorsandgoalstatements
Establishaframeworktomonitorprogress
36
Yourturn
Thetemplatebelowismeanttohelpyoucraftagoalstatementforeachsupportingmeasure
Goalstatement
Departments
Outcome
Activities
KPI
Target
Timeframe
Conductanddiscussrelevantanalysis
ThrougharoutineseriesofmeetingscontinuallymonitorprogressofKPIssupportingindicatorsandanyotherdatacollectedsuchasqualitativedatatoidentifytrendsthatwillindicatewhetherthechosenpolicyishavingthedesiredimpactThesedatapointsshoulddrivethediscussionduringthesemeetingspointingtoopportunitiesforfurtherinquiryinto
Establishaframeworktomonitorprogress
37
whatisdrivingtheobservedtrendsAnalystsshouldbeencouragedandtrainedtoaskquestionsofthedepartmentsproducingthedataandconductsitevisitswhenpossibletodeepentheirunderstandingofthedataPresentationoftheanalysisshouldfocusononeideaatatimetokeeptheconversationfocusedonkeypointsandspecificopportunitiesforimprovementYoumayconsiderdevelopingapublicfacingdashboardtosharethisprogressmorebroadly
AgeneralcomparisonofthestatusquodatawithwhatwasobservedduringthepilotcanprovidedecisionmakerswithaquickunderstandingofthedifferencesbetweentheoldandnewapproachesalthoughthiswillnotshowcausalityOftenthesecomparisonswillincludeoperationalmetricssuchasdaysordollarssavedbutrememberthattheimpactonpeoplewillbemostimportantIfyouareworkingwithanevaluationpartnertheywillsupportthisstep
Usethespacebelowtosummarizetheresultsofyouranalysis
MeasureTargetvalue(seesectiononsettingtargetsabove)
Pilotvalue(whatdidyouobserveduringthepilot)
Historicalaverage(ifavailable)
Results(WhoisbetteroffIsanyonenegativelyimpacted1-2sentencesreassessingyourgoalstatement)
DocumentyourprocessandprogressincludingfeedbackfromtheparticipantsimplementationpartnersandtheprojectteamDidanythingunexpectedoccurduringthepilotArethereinternalandexternalfactorsthatmayaffectyouroutcomesThisiswheretheimplementationoutcomescanbecanhelpfulIftheinterventiondidnotgoasplannedconsiderwhetheritisbecausetheinterventionwasnotacceptableorfeasible
UsethetemplatebelowcreatedbyDenversPeakAcademytocapturethisinformationIfthe306090daytimelinedoesnotmakesenseforyourcityfeelfreetocustomizeit
CurrentState
FutureState
30Day
60Day
90day
Whatwentwell(+)
Whatdidnotgowellshouldbechanged(-)
IfaquestionordiscussionpointarisesthatcannotbeansweredinthemeetinganassignedstaffmembershouldcapturethequestionduringthemeetingAfterwardthefollowupitemsshouldbesenttorelevantpartnersandthenextmeetingshouldbeginwithadiscussionof
Establishaframeworktomonitorprogress
38
thoseitems
Takeaction
DiscussionsaboutwhetheryourindicatorsaremovingintherightdirectionandwhatmightbedrivingtrendsmayleadtoadiscoverythattherearenotenoughresourcesallocatedtothispolicyorissueTheremaybelimitedfundingforstaffprogrammingortechnologythatisdrivingthedatainthewrongdirectionIfCityleadershipisintheroomthisisagreattimetomakethecaseforadditionalfundingduringthebudgetseasonBycollectinganalyzinganddiscussingdataonthispolicyroutinelyyouarebuildingevidencetomakethecasefortherightleveloffundingtoreachtheestablishedtargets
Establishaframeworktomonitorprogress
39
Assessdataaccessibilityqualityandsecurity
Determineaccessibilityandqualityofdata
OnceyouhavecompiledyourwishlistofindicatorsitistimetodeterminetheaccessibilityandqualityofthedataTopreventcomplicationyoumaywishtoprioritizerelevantindicatorsforwhichyouarealreadycollectingdataAvoidthetemptationofadoptingatechnologysolutiontocollectnewdataatthispointsuchasaphotosurveytoolorotherappFornowyouaretryingtodeterminewhichdatasetsarerelevantaccessibleandhighqualityBesuretoworkwithyourkeyinternalpartnerswhileyouareidentifyingyourmeasuresDepartmentstaffwhoworkonprogramsarefamiliarwiththeindicatorsthatwillaccuratelysupporttheoutcomesanddatabaseadministratorswillbevaluablewhendiscussingaccessibilityandqualityofthedata
Abrieflistofconsiderationsfortheaccessibilityofyourdataisbelow
ArewecurrentlycollectingthisdatasetorwillwebecollectingitforthefirsttimeHowfarbackdoestherecordgo(timehorizon)WouldweneednewtechnologytocollectoranalyzethedataCanthedatabecollectedmanuallyorcanitbeautomatedHowfrequentlyisthedataproduced(continuouslyhourlydailyweeklymonthlyquarterlyyearlyetc)IsthedataavailablefortheentireCitypopulationorasubsetofthepopulationHowgranularisthedata(individuallevelCensusblockneighborhoodcitywideetc)DoesthedatacontainsensitiveinformationIsthereaproxymeasurethatwecanusefornowIsthedatamachinereadable
AsnotedbytheWilsonSheehanLabforEconomicOpportunities(LEO)attheUniversityofNotreDameleveragingadministrativedatacansignificantlyreducethecostsandtimelinesofrigorousevaluationCitiescanestablishaninternaldatasharingagreementacrossdepartmentsincludinglanguagethatenablesadministrativedatatobeusedforresearchpurposesCitiescanalsoidentifyexternaladministrativedata-holderssuchasacountyschooldistrictpolicedepartmenthospitaloranotherstatethatmayhaveoutcomedataimportantformeasuringprogramimpactCitiescanleverageexistingrelationshipsandorworkwithresearchinstitutionstodevelopstreamlineddatasharingprocessesandagreementswiththeseexternalentities
Assessdataaccessibilityqualityandsecurity
40
Evaluatethequalityofyourdata
QualityisasimportantasaccessibilityToevaluatedataqualityyoumaywishtocreatearubricliketheonebelow
Value ValueDescription
3 ExcellentDataQualityThedataiscompleteaccurateandupdatedregularly
2 AdequateDataQualityThedataissomewhatcompleteaccurateandupdatedfairlyregularly
1 PoorDataQualityThedataisnotcompleteaccurateorupdatedregularly
ThisisagreattimetobuildonthedatainventorythatyoustartedinthegatherinformationpageSuggestedadditionalfieldstoaddtoyourdatainventorytemplatearebelow
Program DatasetTitle
DatasetDescription DataSource
Departmentorexternalpartner
DoestheCityalreadycollectthisdata
Whatisthenameoftheprogram
Human-readablenameofthedataset
Shouldbeunderstandablebynon-technicalusers
Whatistheoriginalsystemapplicationfilethathousesit
Departmentorexternalpartnerthatmaintainsthedata
Isthisdataalreadycollected
Updatefrequency-ThefileformatforthedatasetTypicallycsvxlsxshapefiletxtetc
Format-Howoftenthedatasetisupdated
Timerange-Theperiodoftimecoveredinthisparticulardataset
Datause-Howisthisdatasetcommonlyused
Datausers-Whotypicallyusesthisdataset
Dataownerstitle-Titleofthedataowner
Dataownersposition-DataOwnersjobtitle
Dataownersemailaddress-DataOwnersemailaddressWhomanagesthedataandorisresponsibleforgrantingpermissiontoaccessthedataWhounderstandswhatthedatasetincludesandcananswerquestionsaboutit
Dataqualityconcerns-Describetheconcernsyoumayhaveaboutthequalityofthedataset
Assessdataaccessibilityqualityandsecurity
41
Division-DivisionwherethedataismaintainedIfmorethanonedivisionisresponsiblelisttheprimaryownerofthedataset
Sensitivedatacomments-DescribetheissuesthatwouldpreventthecityfromsharingthedatapubliclyCommonsensitivityconcernsincludeprivacyviolationssecurityissuesorhighcostorstaffdemandstopostthedata
Publishingstatus-Thistheplacetoputanycommentsregardingthecurrentstatusofthedataset
Publishedlink-Thisisthelinktothepublisheddatasetontheopendataportalorcitywebsite
Reviewdatasecurityandprivacyprotectionprotocols
OftenapolicychangealsoinvolvesachangeinthedatathatisbeingcollectedhowthatdataisbeinghandledwhohasaccesstoitandahostofotheradjustmentsAsyoutakestockofthedataassetsthatareavailableforyourinitiativeitisvitaltoalsothinkthroughwhatwillbedonetoprotectthatdataandtheprivacyofthepeopleitrepresentsAbigpartofgooddatasecurityinvolvesstrongdatagovernancepracticesthatdefineandidentifysensitivedataandconsciouslyworktobalancethevalueandriskofstoringsuchsensitivedataOnewayofevaluatingyourdatacomesfromcaselawrelatedtotheUSFreedomofInformationActwhichwehavetranslatedtothechecklistbelow
1 Woulddisclosureresultinasubstantialinvasionofprivacy2 Haveweconsideredtheextentorvalueofthepublicinterestandthepurposeorobject
oftheindividualsseekingdisclosure3 Istheinformationavailablefromothersources4 Wastheinformationgivenwithanexpectationofconfidentiality5 Isitpossibletomouldreliefsoastolimittheinvasionofindividualprivacy
TheseriskscouldbemitigatedusingavarietyofstrategiesfromaggregatingdatatoobfuscatingindividualstoredactingfieldsthatmayincludePersonallyIdentifiableInformation(PII)orre-evaluatingyourdatacollectionmethodstofindalternativewaysofgettingtheinformationyouneedThesestepsshouldbecodifiedandmanagedbyadatagovernancecommitteeifpossibleForthesensitivedatathatisdeemedessentialcreateaplanaroundpotentialdatabreachesinordertoquicklyandconfidentlyrespondtoemergencysituations
ProactivelypreparingforthesesituationsisoftenthebeststrategyfordatasecuritybutdonotforgetthetechnicalsideInvolveyourITadministratorsinconversationsaboutdatastorageandaccessensuringthattheywillbeprotectedbyenterprise-gradecybersecuritysystems
Assessdataaccessibilityqualityandsecurity
42
Assessdataaccessibilityqualityandsecurity
43
Designimplementationandevaluation
Designtheimplementation
NowthatyouhaveyouraccountabilityplanandkeyindicatorsitistimetogetspecificaboutyourplanforimplementationThetemplatebelowwillhelpyouthinkthroughthespecificmechanismsofimplementingpolicychange
UnderstandingorganizationsandsystemsiscriticaltounderstandingandmakingdecisionsaboutwhereandhowinterventionsshouldbeappliedApolicyimplementedinasinglesettingmayshowpromisingresultsbutimplementingmultiplepoliciesandprogramsatthesametimewithdifferentactorsresponsibleforeachaspectcanhelpensuresustainablebehaviorchange
InSomervilleMAShapeUpSomerville(SUS)isacitywidepublichealthprogramthatisimplementedacrossmultiplelevelsofinfluenceandsystemsthroughinterventionsbeforeduringandafterschooltoincludechangestobuiltenvironmentsuchasbikelanesregionalmasstransportationandphysicaleducationequipmentinschoolsandgymstoincreasephysicalactivitywellnesspoliciesandschoolgardensnutritioneducationduringandafterschoolandhealthyrestaurantprogramstosupporthealthyfoodandhealthyeatinghabits
Usethetemplatebelowtoplanyourimplementation
Policyimplementationplanningtemplate
ListthespecificdesiredoutcomesInthepreviousstepyouworkedwithdepartmentstafftoestablishspecificmeasurableoutcomesandinterimindicatorsPrioritizetheindicatorsthatwilldemonstrateprogressUsethespacebelowtocapturethedesiredoutcomesofthepolicychange
ExampleShapeupSomervillebeganwiththegoalofpreventingobesityin1stthrough3rdgraders
Yourturn
____________________________________________________________________________
____________________________________________________________________________
Designimplementationandevaluation
44
SpecifythetargetpopulationAreyoudesigninganinterventionthattargetsonlycertainsitesAreyouplanningonimplementingaprogramorinitiativeforaspecifictargetaudienceClearlyspecifythepopulationyouplantotarget
IfyouplantoconductarigorousimpactevaluationasnotedbytheWilsonSheehanLabforEconomicOpportunitiesitisimportanttoestablishconcreteobjectiveeligibilitycriteriaandensurethatthesecriteriaareappliedconsistentlybyallstaffduringscreeningandintakeItisalsovitaltoidentifytheindividualswhoareeligibleforaparticularprogramorserviceevenifeveryoneeligiblecannotbeservedbyexistingresourcesThishelpscitiesidentifyanyunmetneedsandguidefutureresourceallocationwhilealsocreatingthepossibilityofarigorousimpactevaluationbyidentifyingapotentialcomparisongroupFinallywhenconductingscreeningandintakeitisimperativetocollectandmaintainbasicinformation(egnamedateofbirthgender)aboutallprogram-eligibleindividualsnotjustthosewhoparticipateintheprogram
ExampleIntheShapeUpSomervilleexampletheprogrambeganbytargeting1stthrough3rdgraders
Yourturn
____________________________________________________________________________
____________________________________________________________________________
DefinethelocationWherewillthepolicychangetakeplaceWillimplementationtakeplaceatonesiteormultiplesites
ExampleShapeUpSomervilleinterventionsweredeliveredinschools
Yourturn
____________________________________________________________________________
____________________________________________________________________________
ClarifyimplementationpartnersandarticulatetheirrolesWhatarethekeyorganizationsinstitutionsorpeopleresponsibleforimplementationTheprojectteamshouldbeasclearandspecificaspossibleinprovidingadescriptionoftheconcreteactivitiesthatimplementationpartnersaremeanttocarryoutPracticeprofilesmaybeahelpfultooltohelpdefinewhatisrequiredoftheactorswhoareresponsibleforimplementingtheintervention
Designimplementationandevaluation
45
ExampleInSomervillemayoralleadershipandengagementofmultiplecityagencies(publichealthprimarycareschoolsCambridgeHealthAllianceetc)streamlinedandconnectedthevariousShapeUpSomervilleinitiativesandgrantswhichhelpedtokeepthetalentandenergyfocusedonhealthyeatingandactivelivingforallsectorsofSomervilleSUSpartnersultimatelyincludedschoolsnonprofitsCityandstateagenciesandnationalfoundations
Yourturn
____________________________________________________________________________
____________________________________________________________________________
DescribetheinterventionWhatarethestepsactivitiesorprocessesRememberthatthemechanismthespecificstepsorprocessthatleadtobehaviorchangeiskeytounderstandingtheeffectivenessofthepolicychangeandsharingtheresultswithothers
ExampleGoalsofShapeUpSomervilleweretoinfluenceeverypartofanearlyelementaryschoolchildrsquosdayndashbeforeduringandafter-schooltopromotehealthyeatingphysicalactivityandtheirweightstatusInterventionincludedfoodserviceenhancementstoimprovebreakfastandlunchandstaffprofessionaldevelopmentwalktoschoolactivitiesSUSclassroomcurriculumandprofessionaldevelopmentschoolwellnesspolicydevelopmentSUSafter-schoolcurriculumandprofessionaldevelopmentoutreachandeducationtohomethroughmaterialsforumseventsldquoSUSapprovedrdquorestaurantscommunityoutreachandcapacitybuildingthroughpolicydevelopmenttrainingsmediaplacements
Yourturn
____________________________________________________________________________
____________________________________________________________________________
ClarifytimelineandmilestonesfordeliveryandevaluationHowfrequentlywilltheservicesinterventionbedeliveredHowlongwilltheprogramrunWhatisthetimetoexpectedimpacts(eg12monthsafterprogramcompletion)
Designimplementationandevaluation
46
ExampleShapeUpSomervilleledbyTuftsUniversityFriedmanSchoolofNutritionScienceandPolicyconductedaquasi-experimental(nonrandomized)interventionstudyamong1stthrough3rdgradestudentsinSomerville(intervention)andtwocomparisoncommunitiesfor9-monthsduringoneschoolyear(fall2002ndashspring2003)ThestudyshowedpositiveassociationoftheinterventiononchangeinBMISUSreleasedreportssummarizingevaluationsprogresstodateplansforadditionalprojectsandimpactstenandfifteenyearsintotheinitiative
Yourturn
____________________________________________________________________________
____________________________________________________________________________
PlanyouranalysisTheprevioussectionaddressedestablishinganaccountabilityframeworktomonitorprogressandimplementationDeterminewhichindicatorswillbediscussthroughthatprocesswhowillsharethedataandanalysisandhowfeedbackfromthoseconversationswillbeincorporatedbytheprojectteam
ExampleTheMayorrsquosofficeimplementedadata-drivendecisionmakingapproachcalledSomerStatwhichwassynergisticwiththeapproachestakenbytheresearchandevaluationteamsdeployedforthespecificgrants
Yourturn
____________________________________________________________________________
____________________________________________________________________________
PlanforsustainabilityIfthepolicyandprogramchangeissuccessfulhowwilltheprojectteamsustainandexpandtoadditionalsitesorpopulations
ExampleAlthoughmostofthepartnershipsforgedintheearlyyearsweremaintainednewgrantsandcollaborationsexpandedattentiontoolderpublicschoolstudentsactivetransportationandtheGreenLinesubwayextensionthebuiltenvironmentfarmerrsquosmarketeffortssuchasldquoGrowninSomervillerdquowithUnionMainStreeteconomicdevelopmentandhealthdisparitiesaswellasdatacollectionmonitoringandsurveillance
Yourturn
Designimplementationandevaluation
47
____________________________________________________________________________
____________________________________________________________________________
Pilotplanningtemplate
IfyouplantoconductapilotfirstdeterminethepurposeofthepilotDoyouwanttoobservewhetherthepolicychangeresultsinthedesiredbehaviorchangeAreyouconcernedaboutfeasibilityscalabilityorcostDoyouwanttotestwhetheryourdesiredresultscanbeachievedthroughthispolicyapproachClarifythegoalofthepilotandthequestionsyouhopetoanswerThiswillhelpyouidentifythekeyindicatorsthatyouwillusetomeasurewhetherthepilotissuccessfulBeforebeginningestablishabaselineofcurrentperformanceandsetyourtargetsaccordinglybasedonwhatyouhopetoaccomplishthroughthepilot
NextestablishthepilotplanforyourprojectteamincludingmilestonesforteamcheckinstodeterminewhethertomoveaheadorchangecourseThisisseparatefromthemeetingsthatwilloccuraspartoftheaccountabilityframeworkUsethespacebelowtocreateyourtimelineanddevelopyourmeetingschedule
Task Personresponsible Startdate Checkindate Deliverable
Planyourevaluation
WhileyouaredesigningtheimplementationitisalsotimetobegintodevelopyourplanforanalysisAsnotedbytheWilsonSheehanLabforEconomicOpportunitiesdecisionsaboutprogramsetup-includingoutreachscreeningintakedatatrackingandenrollment-candeterminewhetherornotarigorousimpactevaluationwillbepossibleAsdiscussedcityleadersmayfindithelpfultoincludeevaluationpartnersinearlyconversationsaboutprogramsetupDuringpreviousstagesyoumayhaveworkedwitharesearchertoplanyourproposedpolicychangeYoushouldbeclearaboutthepurposeoftheevaluationanddevelopanevaluationplanBelowareadditionalconsiderations
Causationvstrendanalysis
IntheEstablishaframeworktomonitorprogressstagetherewasguidanceontrackingtrendsovertimeItisimportanttonoteherethatmonitoringdataisdifferentfromanalyzingacausaleffectTrendanalysiswillillustratethatchangesareoccurringandmayleadtoquestionsthatcouldbeansweredbyyourevaluationbuttrendanalysisalonedoesnotprovideevidenceaboutwhatisdrivingthosechangesYoumayseechangesinyour
Designimplementationandevaluation
48
outcomeindicatorsbasedonvariablesoutsideoftheCityscontrolForinstanceachangeinfederaltradepolicycanhaveanimpactemploymentratesinyourCityoutsideofyourCitysnewjobtrainingandplacementprogram
ToaddresscausalityyoumustcomparetheoutcomesofindividualswhoreceivedtheinterventionwiththosewhodidnotWithoutacomparisongroupofsimilarindividualswhowerenotexposedtotheinterventionitisimpossibletoknowwhichfactorsareresponsiblefortheoutcomesRandomizedControlTrials(RCTs)inwhichindividualsarerandomlyassignedtotreatmentandcontrolgroupsarethegoldstandardinprovingcausalityIfyoucandemonstratethatoutcomesimprovedforthosewhoreceivedtheinterventionanddidnotimproveatallorimprovedlessforacomparisongroupyoucanbesomewhatcertainthattheinterventionwasthecause
Examplesofothernon-causalimpactevaluationsthatarerigorousandreliableincludecertainquasi-experimentalstudiesusingcomparisongroupdesignssmartinterruptedtimeseriesanalysesthatmodeltrendsovertimeinwaysthatallowestimationofinterventioneffectswelldesignedpre-posttestanalysesneedsassessmentsimplementationstudiesetcFormoreinformationondifferenttypesofevaluationsseetheSageResearchMethodswebsite
Lookfordifferencesingroups
BuildintimetoalsoexaminethepilotdatatoseeifgroupsareexperiencingdifferentresultsReviewingperformancebasedonoutcomemeasuresforapopulationinaggregatemayhidethesubtlewaysthatsomepeoplemaybeaffecteddifferentlysoitisnecessarytocarefullyanalyzethedatafordisparateoutcomesForinstancetheMovingtoOpportunity(MTO)studythatexaminedtheimpactofhousingvouchersonfamilieswholivedinhigh-povertyneighborhoodsfoundthatchildrenwhomovedbeforetheageof13hadbettereducationandhigherincomesasadultsHoweveradultsdidnotseeimprovededucationandemploymentoutcomes
Disaggregatingdataintomorespecificcategoriesofraceandethnicitycanhelpunderstandtrendsanddisparitiesinamoregranularwaywhichcanhelpdecisionmakerstargetpoliciesandinterventionsinamoreequitableway
EarlierintheRoadmapweworkedtoidentifystakeholdersandwhomightbepositivelyandnegativelyimpactedbyachangeinpolicyUsetheresultsofthatexercisetoinvolvethecommunityininterpretingdataanalysisandtogatherqualitativefeedbackfromthecommunitytocallattentiontoissuesForinstanceanincreaseinyouthdetentionmaybeexplainedbyinefficienciesorlackofaccesstotransportation
Designimplementationandevaluation
49
Designimplementationandevaluation
50
Shareyourstoryandsuccess
Sharetheresults
Storytelling
AnintegralstepinyourstrategyissharingtheresultsofyourpilotorpolicychangewithyourcommunityThisincludesdevelopingmessagingaroundtheworkofyourorganizationandcreatingaplantodisseminateyourstoriesWhytellstoriesaboutyourwork
Informandorunderstandasubjectorsituation
Giveneededcontextofdataorbitsofinformation
Relatablestoriesarememorableforreadersviewers
ReadersorviewerswillseetheinformationasmorecompellingwhenthereisademonstratedandrelatablechangeorimpactonthemselvesortheirfriendsneighborsandcolleaguesStoriesareaboutpeopleandgovernmentisfundamentallyabouthelpingpeopleSotellthestoriesofthepeopleyouandyourcitycolleagueshelped
Keepthesetipsinmindasyoudevelopyourstorywithyourteam
WhatwereyourfindingsandwhyshouldanyonecareTellyouraudiencehowtheproblemwassolvedWhathastheimpactbeenonresidentsandcitystafftodateWhatwasthesuccessandweretheremultiplesuccessesalongthewayIfyouhavepreliminaryresultsordatashowingimprovementtosharethisisthetimetodoit
WhowasinvolvedandimpactedReviewthelistofstakeholdersHowwereeachofthemimpactedbytheprogramHumanizeyourstoryandimpactbyaddingadditionalvoicesThesevoicescanbeoneofthefollowing
Apersonorfamilywhowaspositivelyaffectedbythechangesuccess
Aninvolvedthird-partydescribingboththeproblemandthesolution(ieanorganizationnonprofitprovidersbusinesstradeassociationsportsteametc)whoispartofthecommunitythatthegovernmentworkedwithonthechangesuccess
Someonefromthegovernmentitselfwhocanspeakaboutthechangesuccess
TellmoreaboutyourcityDescribeyourcityincludingpopulationdemographicsindustriesetcWhatareitschallengesWhatisitknownforWhatmakesituniqueHowisitsimilarordifferentfromothercitiesintheUS
Shareyourstoryandsuccess
51
DescribeyourcityrsquoschallengeWriteoutthechallengeyourcityisfacing(asdefinedinyourproblemstatement)HowdoesitaffectresidentsHowdoesthisproblemaffecttheworkofcitystaffmembers
AddthedetailsProvidedetailsonyourappliedresearchstrategyTalkaboutwhatyourcityhasdonenthepasttoresolvetheissueDiscusshowyourstrategyhaschangedwhenyoubegantoadoptthisapproachWasmoneysavedorinvested
Usethisstorymapexercisetoorganizeyourthoughtsandguaranteethatyouincludealloftheelementsofatraditionalstory
AfteryoudevelopyourmessagingcreateacommunicationsmarketingandoutreachstrategydesignedtospreadyourmessageYourstrategyshouldwalkyouthroughthefollowingkeyactivities
1 IdentifyandprofileyouraudienceWhencreatinganddisseminatingyourmessageitisalwaysgoodtothinkaboutyouraudienceWhoexactlywouldbeinterestedinwhatyouhavetosayThegeneralpopulationofthecityPeoplewhoworkinnonprofitsBusinessownersItmattersbecausenoteveryonewholivesinacitymaybethetargetoftheinformationBusinessownersmaybemoreinterestedindatashowingeffectsonthebusinessdistrictthanothersandsoonWewillcoverthislaterbutthisstepwillalsohelpyoutothinkofthemosteffectivewaytoreachyouraudience
2 Defineyourcalltoaction(ifyouhaveone)SpendsometimeconsideringwhyyouareinterestedintellingyourstoryDoyousimplywanttoupdatethepubliconyourprogressorcelebrateyoursuccessesAreyouinterestedinyouraudiencechangingitsbehaviororsupportingaspecificinitiativebeinglaunchedbyyourorganizationIfyouwanttohaveyouraudienceactorbehaveinacertainwaybespecificandincorporateitinyourmessaging
Shareyourstoryandsuccess
52
3 SelectperformancemeasuresThinkabouthowyouwoulddetermineifyourcampaignissuccessfulandselectappropriateperformancemeasuresYouwillrevisitthesemeasuresattheendofyourcampaigncommunicationsplanimplementationtodeterminewhetheryouneedtomodifyyourcommunicationsandoutreachstrategyforthefuture
4 MethodstocommunicateyourstoryTherearemanywaystocommunicatesuccessofyourprojectHerearesomesuggestionsonreachingaudiencesItisimportanttodeterminewhichmethodsarethemosteffectiveforyourdesiredaudiencetoreceiveinformationDoyourresearch(PewResearchCenterhasexcellentstudiesonthereachofmediadigitalmediaandmarketing)ForexampleifseniorsareyourtargetpopulationandtheyarelessinclinedtouseInstagram(thatistrue)youmaywanttoconsiderFacebookorthenightlylocalnewsbothofwhichareusedmoreoftenbyseniorsBelowisalistofsomemethodsandtoolsyoucanuse
i Pressreleasestolocalnewspapersandtvstations
ii ShortsmartphonevideosAgreatwaytoengageanaudienceisthroughvideoAskonequestionofsomeonewhoispartofthestoryMaybethereisasocialworkerwhocandiscusshungerinyourcityThesevideoscanbereusedonmultipleplatforms
iii Long-formwriting
iv TellingindividualstoriesthroughaseriesofphotosandshortnarrativesforpublicationonFacebookorInstagram
v Op-edsopinionpiecesplacedinthelocalpaper
vi Emailupdatestoyourlist
vii Connectingwithappropriateorganizationsthatareinvestedinthetopicandworkwiththecommunityandaskthemtodisseminateinformationtotheirmembers
viii EmployingldquoWordofMouthrdquo(WOM)tacticsbycreatingtownhallsorfreeeventsforresidentstoattendintheirownneighborhoods
5 DeterminethetypeofcollateralUsuallycampaignsthattargetresidentsrequireseveralavenuestoreachthemPeoplearenotalwaysinthesameplaceorspaceHavingakeenunderstandingofyouraudiencewillassistyouwithyourchoicesonwhatneedstobedevelopedSometimesthatmeansimagesforsocialmediaaonepageflierthatgivespeopleinformationonattendinganeventoranin-depthbrochure
Shareyourstoryandsuccess
53
6 ImplementyourstrategyKeepinmindstrategiescanbefluidSometimessomethingmaynotturnoutasexpectedevenifduringplanningsessionsitwasldquosupposedtordquoItisokYoucananalyzewhathappenedduringadebriefsessionandonceyoursurveysandothermeasurementshavebeenfinalized
Visualizations
Withtherightvisualizationyoucanreviewandshareinformationinsightsanalysisperformanceandtrendsallofwhichisvaluabletoacross-sectionofthepublicButwhichvisualizationisrightforyouTochoosetherighttypeofchartgraphdeterminethetypeofquestionyouwanttoanswerSomecommoncharttypesarebelowandmoreinformationcanbefoundhere
Exploreadditionalanalyticaltechniques
Duringthegoal-settingphaseyouestablishedoutputoutcomeandimplementationmeasuresBasedontheresultsofyourpilotyoumayneedtoadjustyourmeasuresWiththepilotdatainhandyoumayhavesomeideasofhowfutureanalysescouldbedifferentandpotentiallyimprovethequalityofyourprogramTakesometimetobrainstormotherrelevantdatasetsthatyoudidnothavethetimetopursueearlierincludingmessydepartmentaldatathatneedsmoreattentionorsourcesofpublicdatafromthestateandfederalgovernmentnonprofitsandevenprivatecompaniesthatcouldsupplementyourunderstandingofanissueUsingmachinelearningtechniquesyoumightbeabletobetter
Shareyourstoryandsuccess
54
predictdemandforhomelessservicesusingNOAAweatherforecastspreemptwatermainbreaksusingsoilstratigraphyorinvestineconomicdevelopmentbymonitoringrealestatelistingdata
Advancedanalyticaltechniquescanalsohelpyourimplementationrunmoreeffectivelybyintegratingdata-informedtargetinganddecisiontoolsintotheprocessByautomatingroutineworkflowsandsurfacingsubtletrendsthesetoolscanhelpyouiterateinamoreprecisemannerandprovidekeystrategicguidancetofront-lineemployeesthatwouldotherwisebeunavailableThesesortsofdata-science-enhancedprocessescanbeclassifiedasoneofsixprojecttypesasexploredfurtherbytheNewOrleansLANOLAlyticsteamFormoreguidanceonscopingasimilarprojecttheGovExadvancedanalyticsscopingguidegivesyouastep-by-stepoverviewofhowtobesuccessful
Usethespacebelowtolistyourideasforpossibleadvancedanalyticsprojectsbytype
Typeofproject Ideas
Findingtheneedleinthehaystack
Prioritizingforimpact
Earlywarningtools
Betterquickerdecisions
Optimizingresourceallocation
Experimentingforwhatworks
Keepthefeedbackloopgoing
GatherinformationonpublicsentimentaboutpolicychangethroughfocusgroupssurveysandpublicmeetingsledbyadvisorygroupmembersNotewhethernewinformationisavailablethatcanhelpanalyzeorcontextualizeyourprogress(egfederalorstatedatareports)Sharetheresultsofyourpilotandthepolicychangeprocesswithothercitiesandresearchers
Shareyourstoryandsuccess
55
WorkingwitharangeofevidencePresentlythereisalotofattentionandinvestmentinevidence-basedpolicymakingincludingthefederalbipartisanCommissiononEvidence-BasedPolicymakingthePewCharitableTrustsreportsandtoolkitsonHowStatesEngageinEvidence-BasedPolicymakingandtheLauraandJohnArnoldFoundationsupportedEvidence-BasedPolicymakingCollaborativeManyoftheseinitiativesfocusonrigorousevaluation--meaningrandomizedcontrolledtrials(RCT)orotherexperimentalmodels--asthebestkindofevidencetouseHoweverthisbodyofevidenceisofteninsufficientforthespecifickindsofquestionsthatgovernmentscanhavewhenmakingadecisionaboutpolicyInthisguideweproposeanapproachtosupportgovernmentsinincorporatingadditionaltypesofevidenceintotheirdecisionmaking
OnewaytounderstandarangeofevidenceistouseNestarsquosStandardsofEvidencetoframeanddefineevidenceTheBritishnonprofitNestaworkstoinjectpositivechangeintothewayUKgovernmentpoliciesandprogramsfunctionOneoftheirkeyareasofworkhasbeendevelopingWhatWorksCentresorganizationsthatcollectcommissionsynthesizeanddisseminateevidenceaboutinterventioneffectivenessacrossarangeofsocialimpactareasNestadevelopedthefollowingStandardsofEvidencetoprovideausefulwaytodocumenthowdifferentkindsofinformationdemonstratedifferentlevelsofevidence
Workingwitharangeofevidence
56
Theimportanceofpotentiallybiasedinformation
Onechallengepresentedbyanevidenceclassificationthatacknowledgesthevalueofnon-scientificformsofevidenceistofigureouthowtobalanceallofthepiecestogetherwhenyouusethemTounderstandhowtodothiswehavetothinkabouthowtopositionpotentiallybiasedinformationindecisionmaking
WhenwelookatinformationwhichhasnotbeenrigorouslyevaluatedweacceptthatthisinformationmightbebiasedThewholereasonweconductrigorousevaluationistoremovebiasfromourobservationsWeconductstructuredtrialstoproduceresultsthatareconsistentandgeneralizableIfafindingisunbiasedthisaccuracymdashthislackofbiasmdashcanbeprovedbecauseanyindependenttestercanreproduceitwhentheyfollowanidenticalprocess
YoucanlookatpotentiallybiasedinformationasjustinformationthatisearlieroninitspathtorefinementreplicationandverifiedaccuracyApieceofinformationbasedonanyoneindividualrsquosobservationsislikelytobebiasedbecauseitiscomingfromjustonesourceBiasisreducedbygatheringtogetherlotsofindependentsourcesofinformationandfiguringoutitsmostcommonattributes
MaybethebiasedinformationmakesaclaimthatistoolargemdashitsayssomethingistrueunderallconditionswhenitisreallyonlytrueundercertainconditionsorthataneffectofaninterventionisbiggerthanitstrueaverageorthataneffectlastsovertimewhenitdoesnotPerhapschangingtheclaimtobemorespecificmdashcriticallyreviewingtheinformationtoseeifspecificconditionsmustbemettoachievetheoutcomesmdashwouldmaketheclaimmoreaccurate
IfyoutestedyourclaimfounditdidnotholdupinsomecaseschangedittesteditagainuntilyourclaimrepeatedlyheldupthenyoucoulddemonstratethattheclaimwasnolongerbiasedbutaccurateThisprocessoftestingimprovingtestingandvalidatingiswhatwegetfromstructuredscientificstudy
Withindividualpiecesofinformation--crediblehypothesescasestudiesexpertopinion--youcannotassumethatkindofaccuracyAtthesametimetheseearlyobservationsarefoundationaltoanyreplicablescientificobservationsWewillnevergetanywhereifwedonotstartwiththesepiecesofinformationeveniftheymightbebiased
Combininglevelsofevidencetogetthemostdetailedinformationforyourcity
ItisespeciallyimportanttoacceptdifferentformsofevidencewhenyoufacequestionsthathavenotyetbeentestedbyindependentstudiesSomeofthemostimportantquestionsrelatetothingsthatareveryspecifictoyoursituationYoufaceaparticularcombinationof
Workingwitharangeofevidence
57
issuesactorstrendsandresourcesthatisultimatelydifferentfromeveryotherplaceintheworld
ThisisimportantbecauseaspolicypractitionersandnotscientistsyoucaremoreaboutyourspecificcasethanahypotheticalaveragegeneralcaseYoucarelessaboutthetotallikelyrangeofimpactsfromapolicythanhowthissingleimplementationwillworkYouwantinformationaboutpoliciesthatwillprovideyouwiththebestpossiblepredictionabouthowtheywillworkinyourcontext
ByputtingdifferentkindsofevidenceintothesameframeworkNestarsquosStandardsofEvidenceimplythatwhenyoucannotgetrigorouslytestedexperimentallyverifiedinformationitisappropriatetoworkwithwhatwedohaveThisisrelevantforthemunicipalimplementationcontextbecausenostudypulledfromaliteraturereviewwillaccountforallofthevariablesyoufaceinyourowncontext
Intheabsenceofhavingastudythatpredictshowaninterventionworksinyourspecificcontextyouwillneednon-scientificinformationtotrytofillinthosegapsYouwillneedtogatherstoriesfromresidentsstaffmembersandotherstakeholdersthatsitonLevels1or2ofthestandardsofevidence
Someofthekindsofinformationyouwillprobablygetmoreeasilyfrompeoplethanstudiesinclude
howpeopleincitieslikeyoursreactedtoaninterventioninpublicforumshowpeoplewhoimplementedaninterventioninacitylikeyoursfiguredouthowtogetitpassedhowthecitycommunicatedpubliclyabouttheapparentbenefitsanddrawbacksoftheinterventionthepoliticalacceptabilityoftheintervention
Youcanremainsimultaneouslyawarethattheinformationyouaregatheringthroughindividualstoriesisbiased--notgeneralizableinformedbyspecificperspectives--whilealsorecognizingthathavingsomeinformationaboutsomethingimportantisbetterthannoinformationEvidencefromthefirsttwoevidencelevelsareessentialbuildingblocksofknowledgeandperfectlyusefulwhenyoucannotgetdetailedunbiasedknowledge
Youhavetostartsomewhereandyoushouldalwaysstartwithevidenceyouhaveaboutyourveryspecificconditionsthatyouknowareimportantforsuccess
Workingwitharangeofevidence
58
Thankyou
Interviewswithcities
GovExheldsemi-structuredinterviewswithsixstaffmembersinfourcitiestoestablishtheinitialstepsoftheRoadmapWeaskedrespondentstotalkaboutaninitiativeorpolicythattheyhavebeenresponsibleforimplementingintheircityRespondentsfromthreecitiesspokeabouttheireffortstoaddresshomelessnessandrespondentsfromonecityspokeaboutblightAfterrespondentsidentifiedtheproblemtheyspokeabouthowthecityprioritizedthisissuethevariousstepsinvolvedandbarrierspresent
GovExconductedtextanalysisoftheseinterviewswhichrevealedthatpeoplewasthemostcommonlyusedtermduringtheseconversationswhendiscussingthestepstochangeaprogramorpolicyThisisathemethroughoutthisRoadmapThefirstsectionbeginswithidentifyingtherelevantpeopleTheconversationsaroundevidencearefocusedonhumanbehaviorratherthanmethodologyorstrengthofevidence
Thankyou
59
OncewehadadraftoftheRoadmapwefollowedupwithcitiesonanindividualbasistolearnmoreaboutwhatwouldbehelpfulWearegratefultothefollowingcitystafffortheinterviewsandindividualconversations
MichaelBaskinChiefPolicyOfficerChattanoogaTN
MelissaBridgesPerformanceandInnovationCoordinatorLittleRockAR
VictoriaCarreoacutenAdministrativeOfficerLasVegasNV
TyroneGrandstrandCommunityDevelopmentPlannerFargoND
TibLaughlinDirectorofGeneralServicesKansasCityKS
TimMorelandDirectorofPerformanceManagementandOpenDataChattanoogaTN
CharlesRosendahlManagementAnalystDentonTX
Thankyou
60
MelissaSiebenAssistantCountyAdministratorKansasCityKS
GinaThompsonDirectorofStrategicServicesLewisvilleTX
JHUFacultycollaboration
GovExalsoconvenedthecitycontactsandfacultypartnersinaseriesofAdvisoryGroupmeetingsThesemeetingsservedtoconnectcitiesandfacultytounderstandoneanothersperspective
StefanBaralBloombergSchoolofPublicHealth
AshleyBernerJHUSchoolofEducation
MelissaDavey-RothwellBloombergSchoolofPublicHealth
JonLinksBloombergSchoolofPublicHealth
TamarMendelsonBloombergSchoolofPublicHealth
KristinMmariBloombergSchoolofPublicHealth
KeshiaPollackPorterBloombergSchoolofPublicHealth
JoshSharfsteinBloombergSchoolofPublicHealth
LizStuartBloombergSchoolofPublicHealth
LindsayThompsonCareyBusinessSchool
DanielWebsterBloombergSchoolofPublicHealth
Partnersindataandevidence-basedpolicy
Wealsospokewithpartnerswhoareexpertsinthefieldofevidence-basedpolicywhosharedtheirknowledgeandresourcesofcurrentandfuturethinkinginevidence-informedthinking
WendyBarrenoAssociateDirectorWilsonSheehanLabforEconomicOpportunitiesUniversityofNotreDame
TiffanyDavenportDirectorofDataPartnershipsBaltimoresPromise
TomiHiersDirectorBaltimoreCivicSiteAnnieECaseyFoundation
ChrisKingsleySeniorAssociateAnnieECaseyFoundation
KathrynOliverAssociateProfessorLondonSchoolofHygieneandTropicalMedicine
Thankyou
61
KristenWatkinsSeniorPolicyAssociateJ-PAL
GovExParticipants
GovExAnalystMiriamMcKinneyanalyzedtheinterviewtranscriptsandconductedtextanalysisSeniorImplementationAdvisorEmilyShawwillintegrateRoadmapactivitieswithadditionalservicedeliveryactivitiesatGovEx
Thankyou
62
ReferencesAbrahamKGampHaskinsR(2017)Thepromiseofevidence-basedpolicymakingReportofthecommissiononevidence-basedpolicymakingWashingtonDCCommissiononEvidence-basedPolicymakingBigmove2Extendingthegreen(nd)Retrievedfromhttpswwwcityoflewisvillecomdoing-businesslewisville-2025big-move-2-extending-the-greenBatesMAampGlennersterR(2017)ThegeneralizabilitypuzzleRetrievedfromhttpsssirorgarticlesentrythe_generalizability_puzzleBuildwithnotfor(2018January)RetrievedfromhttpsbuildwithnotforwordpresscomCairneyPampOliverK(2017)Evidence-basedpolicymakingisnotlikeevidence-basedmedicinesohowfarshouldyougotobridgethedividebetweenevidenceandpolicyHealthResearchPolicyandSystemsdoi101186s12961-017-0192-xCairneyPOliverKampWellsteadA(2016)TobridgethedividebetweenevidenceandpolicyReduceambiguityasmuchasuncertaintyPublicAdministrationReview76(3)399-402Retrievedfromhttpdxdoiorg101111puar12555Centreforpublicimpact(nd)RetrievedfromhttpswwwcentreforpublicimpactorgCernaL(2013)ThenatureofpolicychangeandimplementationAreviewofdifferenttheoreticalapproachesOrganisationforEconomicCo-OperationandDevelopmentRetrievedfromhttpswwwoecdorgeduceriTheNatureofPolicyChangeandImplementationpdfColoradoSprings(nd)Retrievedfromhttpscoloradospringsgovcommunicationspagewhat-expect-when-you-speakupContinuumofcare(coc)program(nd)RetrievedfromhttpswwwhudexchangeinfoprogramscocCouncilpresidentkicksoff$12millionyouthfund(nd)Retrievedfromhttpwwwbaltimorecitycouncilcomcontentcouncil-president-kicks-12-million-youth-fundDurlakJAampDuPreEP(2008)ImplementationmattersAreviewofresearchontheinfluenceofimplementationonprogramoutcomesandthefactorsaffectingimplementationAmericanJournalofCommunityPsychology41327-350Retrievedfromhttpswwwncbinlmnihgovpubmed18322790Fishbonediagram(ishikawa)(nd)Retrievedfromhttpasqorglearn-about-qualitycause-analysis-toolsoverviewfishbonehtmlFivewhysGettingtotherootofaproblemquickly(nd)RetrievedfromhttpswwwmindtoolscompagesarticlenewTMC_5Whtm
References
63
FligsteinNampMcAdamD(2011)TowardageneraltheoryofstrategicactionfieldsSociologicalTheory29(1)1-26RetrievedfromhttpsociologyberkeleyedusitesdefaultfilesfacultyfligsteinFligstein20McAdam20Soc20Theory20PaperpdHeinkelP(2018January12)CitymullspermitsforbikesharingcompaniesRetrievedfromhttpwwwdentonrccomnewsdenton20180112city-mulls-permits-bike-sharing-companiesHowisareamedianincomecalculated(2012October)Retrievedfromhttpswwwhudexchangeinfofaqs843how-is-area-median-income-calculatedLampkinLMampHatryHP(2003)KeystepsinoutcomemanagementUrbanInstituteRetrievedfromhttpswwwurbanorgresearchpublicationkey-steps-outcome-managementMarshDampSmithM(2000)UnderstandingpolicynetworksTowardsadialecticalapproachPoliticalStudies484-21Retrievedfromhttpjournalssagepubcomdoiabs1011111467-924800247McCannL(nd)RetrievedfromhttplaurenellencomMoultonSampSandfortJR(2016)ThestrategicactionfieldframeworkforpolicyimplementationresearchPolicyStudiesJournal45(1)Retrievedfromhttponlinelibrarywileycomstore101111psj12147assetpsj12147pdfv=1ampt=jdkhhcp4amps=e184c625e1855a6b1615ccb99ee27b1c0b3212d8PaynterB(2018January1)What100citiesarelearningfromeachotherbysharingtheirdataRetrievedfromhttpswwwfastcompanycom40518004what-100-cities-are-learning-from-each-other-by-sharing-their-dataPeakacademy(nd)Retrievedfromhttpswwwdenvergovorgcontentdenvergovenmayors-officeprograms-initiativespeak-performancepeak-academyhtmlPinnockHBarwickMCarpenterCREldridgeSGrandesGGriffithsCJRycroft-MaloneJMeissnerPMurrayEPatelASheikhAampTaylorSJC(2017)Standardsforreportingimplementationstudies(stari)ExplanationandelaborationdocumentBMJOpenRetrievedfromhttpbmjopenbmjcomcontentbmjopen74e013318fullpdfProctorESilmereHRaghavanRHovmandPAaronsGBungerAGriffeyRampHensleyM(2011)OutcomesforimplementationresearchConceptualdistinctionsmeasurementchallengesandresearchagendaAdministrationandPolicyinMentalHealth38(2)65-76doi101007s10488-010-0319-7TandonSD(2018)ProgressincommunityhealthpartnershipsResearcheducationandactionRetrievedfromhttpswwwpressjhuedujournalsprogress-community-health-partnerships-research-education-and-actionProvanKGampKenisP(2007)ModesofnetworkgovernanceStructuremanagementandeffectivenessJournalofPublicAdministrationResearchandTheory
References
64
18(2)229-252Retrievedfromhttpsacademicoupcomjpartarticle182229935895SabatierPA(1988)Anadvocacycoalitionframeworkofpolicychangeandtheroleofpolicy-orientedlearningthereinPolicySciences21(3)129-168Retrievedfromhttpwwwjstororgstable4532139SandstromAampCarlssonL(2008)TheperformanceofpolicynetworksTherelationbetweennetworkstructureandnetworkperformancePolicyStudiesJournal36497-524Retrievedfromhttponlinelibrarywileycomdoi101111j1541-0072200800281xabstractSchwartzMampWilsonE(nd)WhocanaffordtoliveinahomeAlookatdatafromthe2006americancommunitysurveyRetrievedfromhttpswwwcensusgovhousingcensuspublicationswho-can-affordpdfShearerJCAbelsonAKouyateBLavisJNampWaltG(2016)WhydopolicieschangeInstitutionsinterestsideasandnetworksinthreecasesofpolicyreformHealthPolicyandPlanning31(9)1200-1211Retrievedfromhttpsdoiorg101093heapolczw052ThelessonsofadministrativedataHigh-profilepolicy-relevantresearchpoweredbyadministrativedata(nd)Retrievedfromhttpswwwpovertyactionlaborgsitesdefaultfilesresources20161114-The-Lessons-of-Administrative-DatapdfToolsandotherneedsforqualitativeandquantitativemethods(2015February)AgencyforHealthcareResearchandQualityRetrievedfromhttpwwwahrqgovprofessionalsquality-patient-safetytalkingqualityassesstoolshtml
WeaverRCampBagchi-SenS(2013)SpatialanalysisofurbandeclineThegeographyofblightAppliedGeography4061-70
References
65
UniversityPartners
Name Title Affiliation Area(s)ofexpertise
Contactinformation
Examplesofworkingwithnon-academicdecisionmakers
WendyPBarreno
AssociateDirector
WilsonSheehanLabforEconomicOpportunitiesUniversityofNotreDame
LEOsfocusareasincludehealtheducationhousingchildwelfareeconomicopportunitycriminaljustice
wbarrenonduduorleondedu
MelissaDavey-RothwellPhD
AssociateScientist
JohnsHopkinsBloombergSchoolofPublicHealth
Health mdavey1jhuedu410-614-5854
BaltimoreCityHealthDepartmentLeadersMarylandDepartmentofHealthAddictionRecoveryServicesinDCExecutiveDirectorsofcommunityorganizationsacrosstheUSthatfocusonHIVSTIsubstanceabuseandimplementationscience
ElizabethStuart
AssociateDeanforEducationandProfessorDepartmentsofMentalHealthBiostatisticsandHealthPolicyand
JohnsHopkinsBloombergSchoolofPublicHealth
Educationsocialpolicystatisticalmethods
estuartjhuedu
Resources
66
Management
Design-ThinkingResources
EngagementGuideforCivicTech
Journeymap
PerformanceManagementResources
DenverPeakAcademy
Proxymeasuresguide
Talkingaboutperformancewithoutcomplicatingit(video)
SettingPerformanceTargetsguide
Datavisualizationguide
Performancemanagementgettingstartedguide
Exercises
Fishboneexercise
5Whys
AdministrativeData
TheLessonsofAdministrativeData(JPAL)
EvaluationResources
Quantitativeandqualitativemethods(USDepartmentofHealthandHumanServicesAgencyforHealthcareResearchandQuality)
KeyStepsinOutcomeManagement(UrbanInstitute)
RCT-YES
Sageresearchmethods
BestPracticesforMixedMethodsResearchintheHealthSciences
ACCORDChecklist
Pilotandimplementationplanning
Benchmarkingexamplesandproxymeasures
Seattleopendataplaybook
Resources
67
Datasciencescopingguide
Resources
68
AboutGovExGovExhelpsgovernmentsusedataandevidencetoimproveresidentsrsquolivesWesupportgovernmentsinbuildingtheirinternalcapacitytooperatesustainableanalyticspracticeswhichimprovecommunityoutcomesinfieldsfrompublicsafetytoeconomicdevelopmentGovExhasaproventrackrecordofworkinginconcertwithpartnerstoprovidetechnicalassistanceandtrainingatscale
SinceourlaunchinApril2015GovExhas
Trainedover2000governmentemployeesinpersonandonlineintopicsincludingdatamanagementperformancemanagementopendatacommunityengagementandanalytics
Providedtechnicalassistancetomorethan100localgovernmentsacrosstheglobeimprovingtheiruseofdatatoachieveefficienciesinservicedeliveryandbetteroutcomesforresidents
Assessedthecapacityofmorethan130governmentalorganizationstodeterminehowleadersstaffandexternalstakeholdersmanageandusedata
Localgovernmentsareincreasinglyonthefrontlinesofaddressinglargescaleproblemswithglobalimplicationssuchasclimatechangepublicsafetyandeconomicinequality
Whileevidence-basedinterventionstohelpsolvethoseproblemsareavailablelocalgovernmentstaffhavelimitedtimeandcapacitytofindandimplementresearch-basedsolutionsAsaresultcitiescontinuetoworkthewaytheyalwayshave--makingincrementalimprovementstoprogramsbutnotalwayscreatingchangeatasystem-level
ThroughresourcedevelopmentandcollaborationwithJHUfacultyGovExissupportingcityleadersinusingdataandevidencetomakedecisionsandcreatinglastingrelationshipswithresearchers
TheRoadmapwasdevelopedbyGovExaspartofitspartnershipwithBloombergPhilanthropiesWhatWorksCitiesinitiative
AboutGovEx
69
- Title page
- Introduction and purpose of the Roadmap
- Using the Roadmap
-
- Gather information about the problem
- Strategically engage key partners
- Review the evidence and collect details
- Decide on a policy to implement based on local factors
- Establish a framework to monitor progress
- Assess data accessibility quality and security
- Design implementation and evaluation
- Share your story and success
-
- Working with a range of evidence
- Thank you
- References
- Resources
- About GovEx
-
SourceTheGeneralizabilityPuzzle
OnceyouhavenarrowedthepolicyoptionsdiscussthequestionsbelowforeachoptionsyouareconsideringThisstepiscriticalasunderstandinglocalcontextissoimportantinsuccessfulpolicyimplementation
CommunityContextQuestionnaire
Localcontext
DoesthisoptionmakesenseinourcommunityIsthisaonesizefitsallapproachIsanyparticulargroupabsentfromthisapproachIfsoisthatok
Youranswers
Effectiveness
WhatwouldweexpecttoobserveifthepilotisaseffectivehereasinCityX
Youranswers
Timing
DoesthetimingmakesenseIsthistherighttimetoimplementthispolicychange
Decideonapolicytoimplementbasedonlocalfactors
31
WherearethepotentialroadblocksthatcouldaffectthescheduleIsthereanywaytomitigatethisHowlongwouldittaketoseeanimpact
Youranswers
Implementation
DowehavethestaffcapacitytoimplementthisArethereadditionalpartnersthatyoucanbringtothetabletosupportimplementationDoesthismakesensegivenourformofgovernment
Youranswers
Authority
DoesourcityhavetheauthoritytoenactthisIfsowhatistheappropriatemechanismAretherelegalbarriersArethereanyexistingpreemptionissuesfromthestateorfederalgovernmentCanthosebeaddressedthroughlegislationadvocacyIstherepoliticalsupportoroppositiontothis
Youranswers
Cost
HowmuchwillthiscostannuallyHowwillwepayforitCanthepolicybeimplementedwithexistingfunding
IfsoisthisfundingreliableHowwouldthispolicyoptionaffectourcitysbudgetintheshortandlongtermWhatotherfundingoptionscanweconsiderIsthereanalternativethatwouldcostlessandgiveussimilarresultsWillanyconstituencybeadverselyaffectediffundingforprogramsthatimpactthemisshiftedtoanotherprogrampolicyorinitiative
Youranswers
Decideonapolicytoimplementbasedonlocalfactors
32
EstablishaframeworktomonitorprogressNowthatyouhavereviewedtheevidenceconsideredlocalcontextandchosenyourpolicyitistimetodeterminehowyourcitywillmonitorprogressofthisnewapproachCitystaffandleadersneedtoestablishtheinfrastructureforongoingcontinuousdecisionmakingbasedonwhetherthepolicychangeisleadingtothedesiredoutcomesforresidentsandwhetherthechangeismeaningfulandrationaltofrontlineworkersSettingarbitrarygoalsatthelocallevelthatconflictwithgoalsatotherlevelsisconfusingtotheindividualsresponsibleforachievingthegoalsGoalsthataresettoohightoachievecanbedemoralizingtostaff
ThissectionincludesstepstosetupavenueinwhichCitystaffandleadershiproutinelyvisitkeyindicatorstodeterminewhatisorisnotworkingandtomakedecisionsinreal-timeFormorereadingonperformancemanagementpleasevisittheGovExPerformanceManagementGettingStartedGuide
InperformancemanagementtermssuchasindicatormeasureandmetricmaybeusedinterchangeablyInthisguidewewilluseindicatorInyourworkthemostimportantthingistobeconsistentinwhichevertermyouuseThisvideomaybehelpfulinunderstandingthisconceptandcommunicatingittoothers
Establishthefundamentalsandgatheryourteam
FirstdeterminewhetherthereisanestablishedvenueorregularmeetingscheduleduringwhichyoucandiscussprogressIfyourcityhasanexistingperformancemanagementstructureonarelatedtopicaddingadiscussionoftheprogressofpolicychangetoyouragendacanbeagreatideaIfthiswillbeastandalonemeetingdecidehowfrequentlytheprojectteamanddecisionmakerswillmeetandputthosemeetingsonthecalendarfortheentirelengthoftheproject
DeterminewhowillstaffandattendroutinemeetingsonCityprogressThesemeetingsmayincludetheMayorandorCityManagerdepartmentheadstaffwhoareimplementingthepolicychangeandanalysts
Identifyindicators
ItiscriticalthatindicatorsbeanimportantcityandcommunitypriorityaswellassomethingtheCityplaysamajorroleinaddressingTheremaybeinstancesinwhichtheCitywillneedtoreachouttoorcollaboratewithexternalpartnerstogetamorecompletepictureofprogressontheoverallpolicychange
Establishaframeworktomonitorprogress
33
Anoutcomeindicatordescribesthedesiredeffectofthetargetedpopulation
ExampleBellevuesfamilyfriendlyneighborhoodsindicatormeasuresthepercentofresidentswhobelievetheyliveinaneighborhoodthatsupportsfamiliesparticularlythosewithchildren
WithyourkeypartnersrevisityourproblemstatementcurrentstateandfuturestateexerciseandbegintoidentifytheoutcomeindicatorsthatwillillustratesuccessFornowchooseasingletop-levelkeyperformanceindicator(KPI)ItiscriticalthattheKPI-thepieceyouwillmeasure-beapriorityforthecommunityandsomethingtheCityplaysamajorroleinaddressing
Outputindicatorsmeasuretheprocessofprogramorservicedelivery
ExampleSanDiegoMayorKevinFaulconersgoaltorepair1000milesofstreetby2020
ReviewingtheadministrativedatasetinventorybegintoparedowntherelevantprogramsanddatasetsthatsupportyourKPIWhatarethemeasuresofeffectivenessforeachprogramTheseareyouroutputindicators
Thediagrambelowillustratestherelationshipbetweenoutputandoutcomeindicators
ThinkaboutimplementationindicatorsasmeasuringstepsintheprogramsrolloutthatarenecessarytoobtaindesiredchangesImplementationoutcomescanfocusontheinterventionstaffdeliveringtheinterventionanorganizationorsystemandcanaddressfeasibilityadaptabilitysustainabilityandcostTwokeyquestionstoconsiderwhendevelopingimplementationindicatorsare
1 Istheinterventionacceptableandappropriatetothecommunityand
2 IsitfeasibletoimplementinterventionXatthistimewiththeseresources
Establishaframeworktomonitorprogress
34
Amoreextensivelistwithdefinitionsofpotentialimplementationoutcomesislinkedhere
UsethetablebelowtochoosethedesiredimplementationoutcomesforyourpilotandtoplanyourmeasurementtechniqueThisshouldincludelevelofanalysis(individualproviderorconsumerorganizationorsetting)yourmeasurementtechnique(surveysobservationinterviewfocusgroupsadministrativedataanalysis)andthephasewhenyouwillcollectthisinformation(earlymidlate)Theremaybemorethanonelevelofanalysismeasurementtechniqueandphaseofdatacollection
Implementationoutcome
Definitionofoutcome
Levelofanalysisandmeasurement
Yourmeasurementtechnique
Phaseofpilotwhenyouwillcollectdata
Usethespacebelowtocaptureyourindicatorwishlist
Indicatorwishlist
Outcomes_____________________________________________________________________________
Outputs________________________________________________________________________________
Implementationoutcomes______________________________________________________________
Setapublichigh-levelgoal
Thehigh-levelgoaliscriticalinestablishingdirectiontogalvanizethepeoplewhowillbedoingtheworkWhendiscussingthegoalusetheMayorsbullypulpitLookforkeymilestonesinCitygovernmenttosharethegoalstatementsuchasthebeginningofaMayoraladministrationaStateoftheCityspeechorotherpublicremarks
GoalsshouldbeldquoSMARTrdquo
Specificaspossible(innamingthepopulationwhowillbebetterofftheactionthecitywilltake)
Measurable
Achievable
Establishaframeworktomonitorprogress
35
Results-focused
Time-bound
ThegoalstatementshouldincludeyouroutcomeindicatorlikeChattanoogasgoalldquoWewillendchronicveteranhomelessnessbyDecember2016rdquoAdiagramtohepyoubuildaSMARTgoalaroundyouroutcomeindicatorisbelow
Component Description Yourturn
Specific WhowillbebetteroffWhatactionthecitywilltake
Measurable Howwillyoumeasuresuccess
Achievable Isthegoalattainable
Results-focused Whatistheexpectedresult
Time-bound When
Setrealisticambitioustargetsandtimelinesforeachindicator
ToestablisharealisticandambitioustargetforyourgoalstatementusetheadministrativedatacollectedforyourchosenKPItoreviewactualCityperformanceinthemostrecentyearcomparedwiththethreeorfourmostrecentyearsworthofdataBegintocalculatesummarystatistics(egtotalminimumaveragemedianmaximum)toidentifytrendsCreatecumulativeandmonthlychartstoidentifytrendsandanomaliesBasedonyouranalysisofhistoricaldataignoretheanomaliesandassumetheproductivitytrendswillfollowthesameoverallpatternfrommonth-to-monthFinallyapplyanincreasedproductivityfactortotheperformanceeachmonthtoforecastfutureprogressBasedonthisanalysischooseatargetthatinspiresimprovementwhileaccountingforreality
Manycitiesthataremorematureintheirperformancemanagementpracticesrecognizetheimportanceoftakingacommunity-levelapproachtoimprovingoutcomesforresidentsThesecitiesarecreatinggoaltaxonomiesthatincludemeasuresrelatedtopublicsafetyhousingandevenrecreationCitiesdriveprogressonthesegoalsbyimplementingstrategiesthatengagemultipledepartmentsandcollaboratewithserviceprovidersinthecommunity
EventuallyyouwilldevelopgoalstatementsfortheoutputsinordertocapturetheefforttoachievetheoutcomecontainedinyourgoalstatementYoursetofindicatorsshouldbecomposedofshort-andinterim-termsupportivemeasureswithaccuratetargetsandrelevanttimelinesAnexamplegoaltaxonomyfromtheCityofLasVegassResultsVegasprogramisbelowUsethistaxonomyasamodelforyourownsuiteofindicatorsandgoalstatements
Establishaframeworktomonitorprogress
36
Yourturn
Thetemplatebelowismeanttohelpyoucraftagoalstatementforeachsupportingmeasure
Goalstatement
Departments
Outcome
Activities
KPI
Target
Timeframe
Conductanddiscussrelevantanalysis
ThrougharoutineseriesofmeetingscontinuallymonitorprogressofKPIssupportingindicatorsandanyotherdatacollectedsuchasqualitativedatatoidentifytrendsthatwillindicatewhetherthechosenpolicyishavingthedesiredimpactThesedatapointsshoulddrivethediscussionduringthesemeetingspointingtoopportunitiesforfurtherinquiryinto
Establishaframeworktomonitorprogress
37
whatisdrivingtheobservedtrendsAnalystsshouldbeencouragedandtrainedtoaskquestionsofthedepartmentsproducingthedataandconductsitevisitswhenpossibletodeepentheirunderstandingofthedataPresentationoftheanalysisshouldfocusononeideaatatimetokeeptheconversationfocusedonkeypointsandspecificopportunitiesforimprovementYoumayconsiderdevelopingapublicfacingdashboardtosharethisprogressmorebroadly
AgeneralcomparisonofthestatusquodatawithwhatwasobservedduringthepilotcanprovidedecisionmakerswithaquickunderstandingofthedifferencesbetweentheoldandnewapproachesalthoughthiswillnotshowcausalityOftenthesecomparisonswillincludeoperationalmetricssuchasdaysordollarssavedbutrememberthattheimpactonpeoplewillbemostimportantIfyouareworkingwithanevaluationpartnertheywillsupportthisstep
Usethespacebelowtosummarizetheresultsofyouranalysis
MeasureTargetvalue(seesectiononsettingtargetsabove)
Pilotvalue(whatdidyouobserveduringthepilot)
Historicalaverage(ifavailable)
Results(WhoisbetteroffIsanyonenegativelyimpacted1-2sentencesreassessingyourgoalstatement)
DocumentyourprocessandprogressincludingfeedbackfromtheparticipantsimplementationpartnersandtheprojectteamDidanythingunexpectedoccurduringthepilotArethereinternalandexternalfactorsthatmayaffectyouroutcomesThisiswheretheimplementationoutcomescanbecanhelpfulIftheinterventiondidnotgoasplannedconsiderwhetheritisbecausetheinterventionwasnotacceptableorfeasible
UsethetemplatebelowcreatedbyDenversPeakAcademytocapturethisinformationIfthe306090daytimelinedoesnotmakesenseforyourcityfeelfreetocustomizeit
CurrentState
FutureState
30Day
60Day
90day
Whatwentwell(+)
Whatdidnotgowellshouldbechanged(-)
IfaquestionordiscussionpointarisesthatcannotbeansweredinthemeetinganassignedstaffmembershouldcapturethequestionduringthemeetingAfterwardthefollowupitemsshouldbesenttorelevantpartnersandthenextmeetingshouldbeginwithadiscussionof
Establishaframeworktomonitorprogress
38
thoseitems
Takeaction
DiscussionsaboutwhetheryourindicatorsaremovingintherightdirectionandwhatmightbedrivingtrendsmayleadtoadiscoverythattherearenotenoughresourcesallocatedtothispolicyorissueTheremaybelimitedfundingforstaffprogrammingortechnologythatisdrivingthedatainthewrongdirectionIfCityleadershipisintheroomthisisagreattimetomakethecaseforadditionalfundingduringthebudgetseasonBycollectinganalyzinganddiscussingdataonthispolicyroutinelyyouarebuildingevidencetomakethecasefortherightleveloffundingtoreachtheestablishedtargets
Establishaframeworktomonitorprogress
39
Assessdataaccessibilityqualityandsecurity
Determineaccessibilityandqualityofdata
OnceyouhavecompiledyourwishlistofindicatorsitistimetodeterminetheaccessibilityandqualityofthedataTopreventcomplicationyoumaywishtoprioritizerelevantindicatorsforwhichyouarealreadycollectingdataAvoidthetemptationofadoptingatechnologysolutiontocollectnewdataatthispointsuchasaphotosurveytoolorotherappFornowyouaretryingtodeterminewhichdatasetsarerelevantaccessibleandhighqualityBesuretoworkwithyourkeyinternalpartnerswhileyouareidentifyingyourmeasuresDepartmentstaffwhoworkonprogramsarefamiliarwiththeindicatorsthatwillaccuratelysupporttheoutcomesanddatabaseadministratorswillbevaluablewhendiscussingaccessibilityandqualityofthedata
Abrieflistofconsiderationsfortheaccessibilityofyourdataisbelow
ArewecurrentlycollectingthisdatasetorwillwebecollectingitforthefirsttimeHowfarbackdoestherecordgo(timehorizon)WouldweneednewtechnologytocollectoranalyzethedataCanthedatabecollectedmanuallyorcanitbeautomatedHowfrequentlyisthedataproduced(continuouslyhourlydailyweeklymonthlyquarterlyyearlyetc)IsthedataavailablefortheentireCitypopulationorasubsetofthepopulationHowgranularisthedata(individuallevelCensusblockneighborhoodcitywideetc)DoesthedatacontainsensitiveinformationIsthereaproxymeasurethatwecanusefornowIsthedatamachinereadable
AsnotedbytheWilsonSheehanLabforEconomicOpportunities(LEO)attheUniversityofNotreDameleveragingadministrativedatacansignificantlyreducethecostsandtimelinesofrigorousevaluationCitiescanestablishaninternaldatasharingagreementacrossdepartmentsincludinglanguagethatenablesadministrativedatatobeusedforresearchpurposesCitiescanalsoidentifyexternaladministrativedata-holderssuchasacountyschooldistrictpolicedepartmenthospitaloranotherstatethatmayhaveoutcomedataimportantformeasuringprogramimpactCitiescanleverageexistingrelationshipsandorworkwithresearchinstitutionstodevelopstreamlineddatasharingprocessesandagreementswiththeseexternalentities
Assessdataaccessibilityqualityandsecurity
40
Evaluatethequalityofyourdata
QualityisasimportantasaccessibilityToevaluatedataqualityyoumaywishtocreatearubricliketheonebelow
Value ValueDescription
3 ExcellentDataQualityThedataiscompleteaccurateandupdatedregularly
2 AdequateDataQualityThedataissomewhatcompleteaccurateandupdatedfairlyregularly
1 PoorDataQualityThedataisnotcompleteaccurateorupdatedregularly
ThisisagreattimetobuildonthedatainventorythatyoustartedinthegatherinformationpageSuggestedadditionalfieldstoaddtoyourdatainventorytemplatearebelow
Program DatasetTitle
DatasetDescription DataSource
Departmentorexternalpartner
DoestheCityalreadycollectthisdata
Whatisthenameoftheprogram
Human-readablenameofthedataset
Shouldbeunderstandablebynon-technicalusers
Whatistheoriginalsystemapplicationfilethathousesit
Departmentorexternalpartnerthatmaintainsthedata
Isthisdataalreadycollected
Updatefrequency-ThefileformatforthedatasetTypicallycsvxlsxshapefiletxtetc
Format-Howoftenthedatasetisupdated
Timerange-Theperiodoftimecoveredinthisparticulardataset
Datause-Howisthisdatasetcommonlyused
Datausers-Whotypicallyusesthisdataset
Dataownerstitle-Titleofthedataowner
Dataownersposition-DataOwnersjobtitle
Dataownersemailaddress-DataOwnersemailaddressWhomanagesthedataandorisresponsibleforgrantingpermissiontoaccessthedataWhounderstandswhatthedatasetincludesandcananswerquestionsaboutit
Dataqualityconcerns-Describetheconcernsyoumayhaveaboutthequalityofthedataset
Assessdataaccessibilityqualityandsecurity
41
Division-DivisionwherethedataismaintainedIfmorethanonedivisionisresponsiblelisttheprimaryownerofthedataset
Sensitivedatacomments-DescribetheissuesthatwouldpreventthecityfromsharingthedatapubliclyCommonsensitivityconcernsincludeprivacyviolationssecurityissuesorhighcostorstaffdemandstopostthedata
Publishingstatus-Thistheplacetoputanycommentsregardingthecurrentstatusofthedataset
Publishedlink-Thisisthelinktothepublisheddatasetontheopendataportalorcitywebsite
Reviewdatasecurityandprivacyprotectionprotocols
OftenapolicychangealsoinvolvesachangeinthedatathatisbeingcollectedhowthatdataisbeinghandledwhohasaccesstoitandahostofotheradjustmentsAsyoutakestockofthedataassetsthatareavailableforyourinitiativeitisvitaltoalsothinkthroughwhatwillbedonetoprotectthatdataandtheprivacyofthepeopleitrepresentsAbigpartofgooddatasecurityinvolvesstrongdatagovernancepracticesthatdefineandidentifysensitivedataandconsciouslyworktobalancethevalueandriskofstoringsuchsensitivedataOnewayofevaluatingyourdatacomesfromcaselawrelatedtotheUSFreedomofInformationActwhichwehavetranslatedtothechecklistbelow
1 Woulddisclosureresultinasubstantialinvasionofprivacy2 Haveweconsideredtheextentorvalueofthepublicinterestandthepurposeorobject
oftheindividualsseekingdisclosure3 Istheinformationavailablefromothersources4 Wastheinformationgivenwithanexpectationofconfidentiality5 Isitpossibletomouldreliefsoastolimittheinvasionofindividualprivacy
TheseriskscouldbemitigatedusingavarietyofstrategiesfromaggregatingdatatoobfuscatingindividualstoredactingfieldsthatmayincludePersonallyIdentifiableInformation(PII)orre-evaluatingyourdatacollectionmethodstofindalternativewaysofgettingtheinformationyouneedThesestepsshouldbecodifiedandmanagedbyadatagovernancecommitteeifpossibleForthesensitivedatathatisdeemedessentialcreateaplanaroundpotentialdatabreachesinordertoquicklyandconfidentlyrespondtoemergencysituations
ProactivelypreparingforthesesituationsisoftenthebeststrategyfordatasecuritybutdonotforgetthetechnicalsideInvolveyourITadministratorsinconversationsaboutdatastorageandaccessensuringthattheywillbeprotectedbyenterprise-gradecybersecuritysystems
Assessdataaccessibilityqualityandsecurity
42
Assessdataaccessibilityqualityandsecurity
43
Designimplementationandevaluation
Designtheimplementation
NowthatyouhaveyouraccountabilityplanandkeyindicatorsitistimetogetspecificaboutyourplanforimplementationThetemplatebelowwillhelpyouthinkthroughthespecificmechanismsofimplementingpolicychange
UnderstandingorganizationsandsystemsiscriticaltounderstandingandmakingdecisionsaboutwhereandhowinterventionsshouldbeappliedApolicyimplementedinasinglesettingmayshowpromisingresultsbutimplementingmultiplepoliciesandprogramsatthesametimewithdifferentactorsresponsibleforeachaspectcanhelpensuresustainablebehaviorchange
InSomervilleMAShapeUpSomerville(SUS)isacitywidepublichealthprogramthatisimplementedacrossmultiplelevelsofinfluenceandsystemsthroughinterventionsbeforeduringandafterschooltoincludechangestobuiltenvironmentsuchasbikelanesregionalmasstransportationandphysicaleducationequipmentinschoolsandgymstoincreasephysicalactivitywellnesspoliciesandschoolgardensnutritioneducationduringandafterschoolandhealthyrestaurantprogramstosupporthealthyfoodandhealthyeatinghabits
Usethetemplatebelowtoplanyourimplementation
Policyimplementationplanningtemplate
ListthespecificdesiredoutcomesInthepreviousstepyouworkedwithdepartmentstafftoestablishspecificmeasurableoutcomesandinterimindicatorsPrioritizetheindicatorsthatwilldemonstrateprogressUsethespacebelowtocapturethedesiredoutcomesofthepolicychange
ExampleShapeupSomervillebeganwiththegoalofpreventingobesityin1stthrough3rdgraders
Yourturn
____________________________________________________________________________
____________________________________________________________________________
Designimplementationandevaluation
44
SpecifythetargetpopulationAreyoudesigninganinterventionthattargetsonlycertainsitesAreyouplanningonimplementingaprogramorinitiativeforaspecifictargetaudienceClearlyspecifythepopulationyouplantotarget
IfyouplantoconductarigorousimpactevaluationasnotedbytheWilsonSheehanLabforEconomicOpportunitiesitisimportanttoestablishconcreteobjectiveeligibilitycriteriaandensurethatthesecriteriaareappliedconsistentlybyallstaffduringscreeningandintakeItisalsovitaltoidentifytheindividualswhoareeligibleforaparticularprogramorserviceevenifeveryoneeligiblecannotbeservedbyexistingresourcesThishelpscitiesidentifyanyunmetneedsandguidefutureresourceallocationwhilealsocreatingthepossibilityofarigorousimpactevaluationbyidentifyingapotentialcomparisongroupFinallywhenconductingscreeningandintakeitisimperativetocollectandmaintainbasicinformation(egnamedateofbirthgender)aboutallprogram-eligibleindividualsnotjustthosewhoparticipateintheprogram
ExampleIntheShapeUpSomervilleexampletheprogrambeganbytargeting1stthrough3rdgraders
Yourturn
____________________________________________________________________________
____________________________________________________________________________
DefinethelocationWherewillthepolicychangetakeplaceWillimplementationtakeplaceatonesiteormultiplesites
ExampleShapeUpSomervilleinterventionsweredeliveredinschools
Yourturn
____________________________________________________________________________
____________________________________________________________________________
ClarifyimplementationpartnersandarticulatetheirrolesWhatarethekeyorganizationsinstitutionsorpeopleresponsibleforimplementationTheprojectteamshouldbeasclearandspecificaspossibleinprovidingadescriptionoftheconcreteactivitiesthatimplementationpartnersaremeanttocarryoutPracticeprofilesmaybeahelpfultooltohelpdefinewhatisrequiredoftheactorswhoareresponsibleforimplementingtheintervention
Designimplementationandevaluation
45
ExampleInSomervillemayoralleadershipandengagementofmultiplecityagencies(publichealthprimarycareschoolsCambridgeHealthAllianceetc)streamlinedandconnectedthevariousShapeUpSomervilleinitiativesandgrantswhichhelpedtokeepthetalentandenergyfocusedonhealthyeatingandactivelivingforallsectorsofSomervilleSUSpartnersultimatelyincludedschoolsnonprofitsCityandstateagenciesandnationalfoundations
Yourturn
____________________________________________________________________________
____________________________________________________________________________
DescribetheinterventionWhatarethestepsactivitiesorprocessesRememberthatthemechanismthespecificstepsorprocessthatleadtobehaviorchangeiskeytounderstandingtheeffectivenessofthepolicychangeandsharingtheresultswithothers
ExampleGoalsofShapeUpSomervilleweretoinfluenceeverypartofanearlyelementaryschoolchildrsquosdayndashbeforeduringandafter-schooltopromotehealthyeatingphysicalactivityandtheirweightstatusInterventionincludedfoodserviceenhancementstoimprovebreakfastandlunchandstaffprofessionaldevelopmentwalktoschoolactivitiesSUSclassroomcurriculumandprofessionaldevelopmentschoolwellnesspolicydevelopmentSUSafter-schoolcurriculumandprofessionaldevelopmentoutreachandeducationtohomethroughmaterialsforumseventsldquoSUSapprovedrdquorestaurantscommunityoutreachandcapacitybuildingthroughpolicydevelopmenttrainingsmediaplacements
Yourturn
____________________________________________________________________________
____________________________________________________________________________
ClarifytimelineandmilestonesfordeliveryandevaluationHowfrequentlywilltheservicesinterventionbedeliveredHowlongwilltheprogramrunWhatisthetimetoexpectedimpacts(eg12monthsafterprogramcompletion)
Designimplementationandevaluation
46
ExampleShapeUpSomervilleledbyTuftsUniversityFriedmanSchoolofNutritionScienceandPolicyconductedaquasi-experimental(nonrandomized)interventionstudyamong1stthrough3rdgradestudentsinSomerville(intervention)andtwocomparisoncommunitiesfor9-monthsduringoneschoolyear(fall2002ndashspring2003)ThestudyshowedpositiveassociationoftheinterventiononchangeinBMISUSreleasedreportssummarizingevaluationsprogresstodateplansforadditionalprojectsandimpactstenandfifteenyearsintotheinitiative
Yourturn
____________________________________________________________________________
____________________________________________________________________________
PlanyouranalysisTheprevioussectionaddressedestablishinganaccountabilityframeworktomonitorprogressandimplementationDeterminewhichindicatorswillbediscussthroughthatprocesswhowillsharethedataandanalysisandhowfeedbackfromthoseconversationswillbeincorporatedbytheprojectteam
ExampleTheMayorrsquosofficeimplementedadata-drivendecisionmakingapproachcalledSomerStatwhichwassynergisticwiththeapproachestakenbytheresearchandevaluationteamsdeployedforthespecificgrants
Yourturn
____________________________________________________________________________
____________________________________________________________________________
PlanforsustainabilityIfthepolicyandprogramchangeissuccessfulhowwilltheprojectteamsustainandexpandtoadditionalsitesorpopulations
ExampleAlthoughmostofthepartnershipsforgedintheearlyyearsweremaintainednewgrantsandcollaborationsexpandedattentiontoolderpublicschoolstudentsactivetransportationandtheGreenLinesubwayextensionthebuiltenvironmentfarmerrsquosmarketeffortssuchasldquoGrowninSomervillerdquowithUnionMainStreeteconomicdevelopmentandhealthdisparitiesaswellasdatacollectionmonitoringandsurveillance
Yourturn
Designimplementationandevaluation
47
____________________________________________________________________________
____________________________________________________________________________
Pilotplanningtemplate
IfyouplantoconductapilotfirstdeterminethepurposeofthepilotDoyouwanttoobservewhetherthepolicychangeresultsinthedesiredbehaviorchangeAreyouconcernedaboutfeasibilityscalabilityorcostDoyouwanttotestwhetheryourdesiredresultscanbeachievedthroughthispolicyapproachClarifythegoalofthepilotandthequestionsyouhopetoanswerThiswillhelpyouidentifythekeyindicatorsthatyouwillusetomeasurewhetherthepilotissuccessfulBeforebeginningestablishabaselineofcurrentperformanceandsetyourtargetsaccordinglybasedonwhatyouhopetoaccomplishthroughthepilot
NextestablishthepilotplanforyourprojectteamincludingmilestonesforteamcheckinstodeterminewhethertomoveaheadorchangecourseThisisseparatefromthemeetingsthatwilloccuraspartoftheaccountabilityframeworkUsethespacebelowtocreateyourtimelineanddevelopyourmeetingschedule
Task Personresponsible Startdate Checkindate Deliverable
Planyourevaluation
WhileyouaredesigningtheimplementationitisalsotimetobegintodevelopyourplanforanalysisAsnotedbytheWilsonSheehanLabforEconomicOpportunitiesdecisionsaboutprogramsetup-includingoutreachscreeningintakedatatrackingandenrollment-candeterminewhetherornotarigorousimpactevaluationwillbepossibleAsdiscussedcityleadersmayfindithelpfultoincludeevaluationpartnersinearlyconversationsaboutprogramsetupDuringpreviousstagesyoumayhaveworkedwitharesearchertoplanyourproposedpolicychangeYoushouldbeclearaboutthepurposeoftheevaluationanddevelopanevaluationplanBelowareadditionalconsiderations
Causationvstrendanalysis
IntheEstablishaframeworktomonitorprogressstagetherewasguidanceontrackingtrendsovertimeItisimportanttonoteherethatmonitoringdataisdifferentfromanalyzingacausaleffectTrendanalysiswillillustratethatchangesareoccurringandmayleadtoquestionsthatcouldbeansweredbyyourevaluationbuttrendanalysisalonedoesnotprovideevidenceaboutwhatisdrivingthosechangesYoumayseechangesinyour
Designimplementationandevaluation
48
outcomeindicatorsbasedonvariablesoutsideoftheCityscontrolForinstanceachangeinfederaltradepolicycanhaveanimpactemploymentratesinyourCityoutsideofyourCitysnewjobtrainingandplacementprogram
ToaddresscausalityyoumustcomparetheoutcomesofindividualswhoreceivedtheinterventionwiththosewhodidnotWithoutacomparisongroupofsimilarindividualswhowerenotexposedtotheinterventionitisimpossibletoknowwhichfactorsareresponsiblefortheoutcomesRandomizedControlTrials(RCTs)inwhichindividualsarerandomlyassignedtotreatmentandcontrolgroupsarethegoldstandardinprovingcausalityIfyoucandemonstratethatoutcomesimprovedforthosewhoreceivedtheinterventionanddidnotimproveatallorimprovedlessforacomparisongroupyoucanbesomewhatcertainthattheinterventionwasthecause
Examplesofothernon-causalimpactevaluationsthatarerigorousandreliableincludecertainquasi-experimentalstudiesusingcomparisongroupdesignssmartinterruptedtimeseriesanalysesthatmodeltrendsovertimeinwaysthatallowestimationofinterventioneffectswelldesignedpre-posttestanalysesneedsassessmentsimplementationstudiesetcFormoreinformationondifferenttypesofevaluationsseetheSageResearchMethodswebsite
Lookfordifferencesingroups
BuildintimetoalsoexaminethepilotdatatoseeifgroupsareexperiencingdifferentresultsReviewingperformancebasedonoutcomemeasuresforapopulationinaggregatemayhidethesubtlewaysthatsomepeoplemaybeaffecteddifferentlysoitisnecessarytocarefullyanalyzethedatafordisparateoutcomesForinstancetheMovingtoOpportunity(MTO)studythatexaminedtheimpactofhousingvouchersonfamilieswholivedinhigh-povertyneighborhoodsfoundthatchildrenwhomovedbeforetheageof13hadbettereducationandhigherincomesasadultsHoweveradultsdidnotseeimprovededucationandemploymentoutcomes
Disaggregatingdataintomorespecificcategoriesofraceandethnicitycanhelpunderstandtrendsanddisparitiesinamoregranularwaywhichcanhelpdecisionmakerstargetpoliciesandinterventionsinamoreequitableway
EarlierintheRoadmapweworkedtoidentifystakeholdersandwhomightbepositivelyandnegativelyimpactedbyachangeinpolicyUsetheresultsofthatexercisetoinvolvethecommunityininterpretingdataanalysisandtogatherqualitativefeedbackfromthecommunitytocallattentiontoissuesForinstanceanincreaseinyouthdetentionmaybeexplainedbyinefficienciesorlackofaccesstotransportation
Designimplementationandevaluation
49
Designimplementationandevaluation
50
Shareyourstoryandsuccess
Sharetheresults
Storytelling
AnintegralstepinyourstrategyissharingtheresultsofyourpilotorpolicychangewithyourcommunityThisincludesdevelopingmessagingaroundtheworkofyourorganizationandcreatingaplantodisseminateyourstoriesWhytellstoriesaboutyourwork
Informandorunderstandasubjectorsituation
Giveneededcontextofdataorbitsofinformation
Relatablestoriesarememorableforreadersviewers
ReadersorviewerswillseetheinformationasmorecompellingwhenthereisademonstratedandrelatablechangeorimpactonthemselvesortheirfriendsneighborsandcolleaguesStoriesareaboutpeopleandgovernmentisfundamentallyabouthelpingpeopleSotellthestoriesofthepeopleyouandyourcitycolleagueshelped
Keepthesetipsinmindasyoudevelopyourstorywithyourteam
WhatwereyourfindingsandwhyshouldanyonecareTellyouraudiencehowtheproblemwassolvedWhathastheimpactbeenonresidentsandcitystafftodateWhatwasthesuccessandweretheremultiplesuccessesalongthewayIfyouhavepreliminaryresultsordatashowingimprovementtosharethisisthetimetodoit
WhowasinvolvedandimpactedReviewthelistofstakeholdersHowwereeachofthemimpactedbytheprogramHumanizeyourstoryandimpactbyaddingadditionalvoicesThesevoicescanbeoneofthefollowing
Apersonorfamilywhowaspositivelyaffectedbythechangesuccess
Aninvolvedthird-partydescribingboththeproblemandthesolution(ieanorganizationnonprofitprovidersbusinesstradeassociationsportsteametc)whoispartofthecommunitythatthegovernmentworkedwithonthechangesuccess
Someonefromthegovernmentitselfwhocanspeakaboutthechangesuccess
TellmoreaboutyourcityDescribeyourcityincludingpopulationdemographicsindustriesetcWhatareitschallengesWhatisitknownforWhatmakesituniqueHowisitsimilarordifferentfromothercitiesintheUS
Shareyourstoryandsuccess
51
DescribeyourcityrsquoschallengeWriteoutthechallengeyourcityisfacing(asdefinedinyourproblemstatement)HowdoesitaffectresidentsHowdoesthisproblemaffecttheworkofcitystaffmembers
AddthedetailsProvidedetailsonyourappliedresearchstrategyTalkaboutwhatyourcityhasdonenthepasttoresolvetheissueDiscusshowyourstrategyhaschangedwhenyoubegantoadoptthisapproachWasmoneysavedorinvested
Usethisstorymapexercisetoorganizeyourthoughtsandguaranteethatyouincludealloftheelementsofatraditionalstory
AfteryoudevelopyourmessagingcreateacommunicationsmarketingandoutreachstrategydesignedtospreadyourmessageYourstrategyshouldwalkyouthroughthefollowingkeyactivities
1 IdentifyandprofileyouraudienceWhencreatinganddisseminatingyourmessageitisalwaysgoodtothinkaboutyouraudienceWhoexactlywouldbeinterestedinwhatyouhavetosayThegeneralpopulationofthecityPeoplewhoworkinnonprofitsBusinessownersItmattersbecausenoteveryonewholivesinacitymaybethetargetoftheinformationBusinessownersmaybemoreinterestedindatashowingeffectsonthebusinessdistrictthanothersandsoonWewillcoverthislaterbutthisstepwillalsohelpyoutothinkofthemosteffectivewaytoreachyouraudience
2 Defineyourcalltoaction(ifyouhaveone)SpendsometimeconsideringwhyyouareinterestedintellingyourstoryDoyousimplywanttoupdatethepubliconyourprogressorcelebrateyoursuccessesAreyouinterestedinyouraudiencechangingitsbehaviororsupportingaspecificinitiativebeinglaunchedbyyourorganizationIfyouwanttohaveyouraudienceactorbehaveinacertainwaybespecificandincorporateitinyourmessaging
Shareyourstoryandsuccess
52
3 SelectperformancemeasuresThinkabouthowyouwoulddetermineifyourcampaignissuccessfulandselectappropriateperformancemeasuresYouwillrevisitthesemeasuresattheendofyourcampaigncommunicationsplanimplementationtodeterminewhetheryouneedtomodifyyourcommunicationsandoutreachstrategyforthefuture
4 MethodstocommunicateyourstoryTherearemanywaystocommunicatesuccessofyourprojectHerearesomesuggestionsonreachingaudiencesItisimportanttodeterminewhichmethodsarethemosteffectiveforyourdesiredaudiencetoreceiveinformationDoyourresearch(PewResearchCenterhasexcellentstudiesonthereachofmediadigitalmediaandmarketing)ForexampleifseniorsareyourtargetpopulationandtheyarelessinclinedtouseInstagram(thatistrue)youmaywanttoconsiderFacebookorthenightlylocalnewsbothofwhichareusedmoreoftenbyseniorsBelowisalistofsomemethodsandtoolsyoucanuse
i Pressreleasestolocalnewspapersandtvstations
ii ShortsmartphonevideosAgreatwaytoengageanaudienceisthroughvideoAskonequestionofsomeonewhoispartofthestoryMaybethereisasocialworkerwhocandiscusshungerinyourcityThesevideoscanbereusedonmultipleplatforms
iii Long-formwriting
iv TellingindividualstoriesthroughaseriesofphotosandshortnarrativesforpublicationonFacebookorInstagram
v Op-edsopinionpiecesplacedinthelocalpaper
vi Emailupdatestoyourlist
vii Connectingwithappropriateorganizationsthatareinvestedinthetopicandworkwiththecommunityandaskthemtodisseminateinformationtotheirmembers
viii EmployingldquoWordofMouthrdquo(WOM)tacticsbycreatingtownhallsorfreeeventsforresidentstoattendintheirownneighborhoods
5 DeterminethetypeofcollateralUsuallycampaignsthattargetresidentsrequireseveralavenuestoreachthemPeoplearenotalwaysinthesameplaceorspaceHavingakeenunderstandingofyouraudiencewillassistyouwithyourchoicesonwhatneedstobedevelopedSometimesthatmeansimagesforsocialmediaaonepageflierthatgivespeopleinformationonattendinganeventoranin-depthbrochure
Shareyourstoryandsuccess
53
6 ImplementyourstrategyKeepinmindstrategiescanbefluidSometimessomethingmaynotturnoutasexpectedevenifduringplanningsessionsitwasldquosupposedtordquoItisokYoucananalyzewhathappenedduringadebriefsessionandonceyoursurveysandothermeasurementshavebeenfinalized
Visualizations
Withtherightvisualizationyoucanreviewandshareinformationinsightsanalysisperformanceandtrendsallofwhichisvaluabletoacross-sectionofthepublicButwhichvisualizationisrightforyouTochoosetherighttypeofchartgraphdeterminethetypeofquestionyouwanttoanswerSomecommoncharttypesarebelowandmoreinformationcanbefoundhere
Exploreadditionalanalyticaltechniques
Duringthegoal-settingphaseyouestablishedoutputoutcomeandimplementationmeasuresBasedontheresultsofyourpilotyoumayneedtoadjustyourmeasuresWiththepilotdatainhandyoumayhavesomeideasofhowfutureanalysescouldbedifferentandpotentiallyimprovethequalityofyourprogramTakesometimetobrainstormotherrelevantdatasetsthatyoudidnothavethetimetopursueearlierincludingmessydepartmentaldatathatneedsmoreattentionorsourcesofpublicdatafromthestateandfederalgovernmentnonprofitsandevenprivatecompaniesthatcouldsupplementyourunderstandingofanissueUsingmachinelearningtechniquesyoumightbeabletobetter
Shareyourstoryandsuccess
54
predictdemandforhomelessservicesusingNOAAweatherforecastspreemptwatermainbreaksusingsoilstratigraphyorinvestineconomicdevelopmentbymonitoringrealestatelistingdata
Advancedanalyticaltechniquescanalsohelpyourimplementationrunmoreeffectivelybyintegratingdata-informedtargetinganddecisiontoolsintotheprocessByautomatingroutineworkflowsandsurfacingsubtletrendsthesetoolscanhelpyouiterateinamoreprecisemannerandprovidekeystrategicguidancetofront-lineemployeesthatwouldotherwisebeunavailableThesesortsofdata-science-enhancedprocessescanbeclassifiedasoneofsixprojecttypesasexploredfurtherbytheNewOrleansLANOLAlyticsteamFormoreguidanceonscopingasimilarprojecttheGovExadvancedanalyticsscopingguidegivesyouastep-by-stepoverviewofhowtobesuccessful
Usethespacebelowtolistyourideasforpossibleadvancedanalyticsprojectsbytype
Typeofproject Ideas
Findingtheneedleinthehaystack
Prioritizingforimpact
Earlywarningtools
Betterquickerdecisions
Optimizingresourceallocation
Experimentingforwhatworks
Keepthefeedbackloopgoing
GatherinformationonpublicsentimentaboutpolicychangethroughfocusgroupssurveysandpublicmeetingsledbyadvisorygroupmembersNotewhethernewinformationisavailablethatcanhelpanalyzeorcontextualizeyourprogress(egfederalorstatedatareports)Sharetheresultsofyourpilotandthepolicychangeprocesswithothercitiesandresearchers
Shareyourstoryandsuccess
55
WorkingwitharangeofevidencePresentlythereisalotofattentionandinvestmentinevidence-basedpolicymakingincludingthefederalbipartisanCommissiononEvidence-BasedPolicymakingthePewCharitableTrustsreportsandtoolkitsonHowStatesEngageinEvidence-BasedPolicymakingandtheLauraandJohnArnoldFoundationsupportedEvidence-BasedPolicymakingCollaborativeManyoftheseinitiativesfocusonrigorousevaluation--meaningrandomizedcontrolledtrials(RCT)orotherexperimentalmodels--asthebestkindofevidencetouseHoweverthisbodyofevidenceisofteninsufficientforthespecifickindsofquestionsthatgovernmentscanhavewhenmakingadecisionaboutpolicyInthisguideweproposeanapproachtosupportgovernmentsinincorporatingadditionaltypesofevidenceintotheirdecisionmaking
OnewaytounderstandarangeofevidenceistouseNestarsquosStandardsofEvidencetoframeanddefineevidenceTheBritishnonprofitNestaworkstoinjectpositivechangeintothewayUKgovernmentpoliciesandprogramsfunctionOneoftheirkeyareasofworkhasbeendevelopingWhatWorksCentresorganizationsthatcollectcommissionsynthesizeanddisseminateevidenceaboutinterventioneffectivenessacrossarangeofsocialimpactareasNestadevelopedthefollowingStandardsofEvidencetoprovideausefulwaytodocumenthowdifferentkindsofinformationdemonstratedifferentlevelsofevidence
Workingwitharangeofevidence
56
Theimportanceofpotentiallybiasedinformation
Onechallengepresentedbyanevidenceclassificationthatacknowledgesthevalueofnon-scientificformsofevidenceistofigureouthowtobalanceallofthepiecestogetherwhenyouusethemTounderstandhowtodothiswehavetothinkabouthowtopositionpotentiallybiasedinformationindecisionmaking
WhenwelookatinformationwhichhasnotbeenrigorouslyevaluatedweacceptthatthisinformationmightbebiasedThewholereasonweconductrigorousevaluationistoremovebiasfromourobservationsWeconductstructuredtrialstoproduceresultsthatareconsistentandgeneralizableIfafindingisunbiasedthisaccuracymdashthislackofbiasmdashcanbeprovedbecauseanyindependenttestercanreproduceitwhentheyfollowanidenticalprocess
YoucanlookatpotentiallybiasedinformationasjustinformationthatisearlieroninitspathtorefinementreplicationandverifiedaccuracyApieceofinformationbasedonanyoneindividualrsquosobservationsislikelytobebiasedbecauseitiscomingfromjustonesourceBiasisreducedbygatheringtogetherlotsofindependentsourcesofinformationandfiguringoutitsmostcommonattributes
MaybethebiasedinformationmakesaclaimthatistoolargemdashitsayssomethingistrueunderallconditionswhenitisreallyonlytrueundercertainconditionsorthataneffectofaninterventionisbiggerthanitstrueaverageorthataneffectlastsovertimewhenitdoesnotPerhapschangingtheclaimtobemorespecificmdashcriticallyreviewingtheinformationtoseeifspecificconditionsmustbemettoachievetheoutcomesmdashwouldmaketheclaimmoreaccurate
IfyoutestedyourclaimfounditdidnotholdupinsomecaseschangedittesteditagainuntilyourclaimrepeatedlyheldupthenyoucoulddemonstratethattheclaimwasnolongerbiasedbutaccurateThisprocessoftestingimprovingtestingandvalidatingiswhatwegetfromstructuredscientificstudy
Withindividualpiecesofinformation--crediblehypothesescasestudiesexpertopinion--youcannotassumethatkindofaccuracyAtthesametimetheseearlyobservationsarefoundationaltoanyreplicablescientificobservationsWewillnevergetanywhereifwedonotstartwiththesepiecesofinformationeveniftheymightbebiased
Combininglevelsofevidencetogetthemostdetailedinformationforyourcity
ItisespeciallyimportanttoacceptdifferentformsofevidencewhenyoufacequestionsthathavenotyetbeentestedbyindependentstudiesSomeofthemostimportantquestionsrelatetothingsthatareveryspecifictoyoursituationYoufaceaparticularcombinationof
Workingwitharangeofevidence
57
issuesactorstrendsandresourcesthatisultimatelydifferentfromeveryotherplaceintheworld
ThisisimportantbecauseaspolicypractitionersandnotscientistsyoucaremoreaboutyourspecificcasethanahypotheticalaveragegeneralcaseYoucarelessaboutthetotallikelyrangeofimpactsfromapolicythanhowthissingleimplementationwillworkYouwantinformationaboutpoliciesthatwillprovideyouwiththebestpossiblepredictionabouthowtheywillworkinyourcontext
ByputtingdifferentkindsofevidenceintothesameframeworkNestarsquosStandardsofEvidenceimplythatwhenyoucannotgetrigorouslytestedexperimentallyverifiedinformationitisappropriatetoworkwithwhatwedohaveThisisrelevantforthemunicipalimplementationcontextbecausenostudypulledfromaliteraturereviewwillaccountforallofthevariablesyoufaceinyourowncontext
Intheabsenceofhavingastudythatpredictshowaninterventionworksinyourspecificcontextyouwillneednon-scientificinformationtotrytofillinthosegapsYouwillneedtogatherstoriesfromresidentsstaffmembersandotherstakeholdersthatsitonLevels1or2ofthestandardsofevidence
Someofthekindsofinformationyouwillprobablygetmoreeasilyfrompeoplethanstudiesinclude
howpeopleincitieslikeyoursreactedtoaninterventioninpublicforumshowpeoplewhoimplementedaninterventioninacitylikeyoursfiguredouthowtogetitpassedhowthecitycommunicatedpubliclyabouttheapparentbenefitsanddrawbacksoftheinterventionthepoliticalacceptabilityoftheintervention
Youcanremainsimultaneouslyawarethattheinformationyouaregatheringthroughindividualstoriesisbiased--notgeneralizableinformedbyspecificperspectives--whilealsorecognizingthathavingsomeinformationaboutsomethingimportantisbetterthannoinformationEvidencefromthefirsttwoevidencelevelsareessentialbuildingblocksofknowledgeandperfectlyusefulwhenyoucannotgetdetailedunbiasedknowledge
Youhavetostartsomewhereandyoushouldalwaysstartwithevidenceyouhaveaboutyourveryspecificconditionsthatyouknowareimportantforsuccess
Workingwitharangeofevidence
58
Thankyou
Interviewswithcities
GovExheldsemi-structuredinterviewswithsixstaffmembersinfourcitiestoestablishtheinitialstepsoftheRoadmapWeaskedrespondentstotalkaboutaninitiativeorpolicythattheyhavebeenresponsibleforimplementingintheircityRespondentsfromthreecitiesspokeabouttheireffortstoaddresshomelessnessandrespondentsfromonecityspokeaboutblightAfterrespondentsidentifiedtheproblemtheyspokeabouthowthecityprioritizedthisissuethevariousstepsinvolvedandbarrierspresent
GovExconductedtextanalysisoftheseinterviewswhichrevealedthatpeoplewasthemostcommonlyusedtermduringtheseconversationswhendiscussingthestepstochangeaprogramorpolicyThisisathemethroughoutthisRoadmapThefirstsectionbeginswithidentifyingtherelevantpeopleTheconversationsaroundevidencearefocusedonhumanbehaviorratherthanmethodologyorstrengthofevidence
Thankyou
59
OncewehadadraftoftheRoadmapwefollowedupwithcitiesonanindividualbasistolearnmoreaboutwhatwouldbehelpfulWearegratefultothefollowingcitystafffortheinterviewsandindividualconversations
MichaelBaskinChiefPolicyOfficerChattanoogaTN
MelissaBridgesPerformanceandInnovationCoordinatorLittleRockAR
VictoriaCarreoacutenAdministrativeOfficerLasVegasNV
TyroneGrandstrandCommunityDevelopmentPlannerFargoND
TibLaughlinDirectorofGeneralServicesKansasCityKS
TimMorelandDirectorofPerformanceManagementandOpenDataChattanoogaTN
CharlesRosendahlManagementAnalystDentonTX
Thankyou
60
MelissaSiebenAssistantCountyAdministratorKansasCityKS
GinaThompsonDirectorofStrategicServicesLewisvilleTX
JHUFacultycollaboration
GovExalsoconvenedthecitycontactsandfacultypartnersinaseriesofAdvisoryGroupmeetingsThesemeetingsservedtoconnectcitiesandfacultytounderstandoneanothersperspective
StefanBaralBloombergSchoolofPublicHealth
AshleyBernerJHUSchoolofEducation
MelissaDavey-RothwellBloombergSchoolofPublicHealth
JonLinksBloombergSchoolofPublicHealth
TamarMendelsonBloombergSchoolofPublicHealth
KristinMmariBloombergSchoolofPublicHealth
KeshiaPollackPorterBloombergSchoolofPublicHealth
JoshSharfsteinBloombergSchoolofPublicHealth
LizStuartBloombergSchoolofPublicHealth
LindsayThompsonCareyBusinessSchool
DanielWebsterBloombergSchoolofPublicHealth
Partnersindataandevidence-basedpolicy
Wealsospokewithpartnerswhoareexpertsinthefieldofevidence-basedpolicywhosharedtheirknowledgeandresourcesofcurrentandfuturethinkinginevidence-informedthinking
WendyBarrenoAssociateDirectorWilsonSheehanLabforEconomicOpportunitiesUniversityofNotreDame
TiffanyDavenportDirectorofDataPartnershipsBaltimoresPromise
TomiHiersDirectorBaltimoreCivicSiteAnnieECaseyFoundation
ChrisKingsleySeniorAssociateAnnieECaseyFoundation
KathrynOliverAssociateProfessorLondonSchoolofHygieneandTropicalMedicine
Thankyou
61
KristenWatkinsSeniorPolicyAssociateJ-PAL
GovExParticipants
GovExAnalystMiriamMcKinneyanalyzedtheinterviewtranscriptsandconductedtextanalysisSeniorImplementationAdvisorEmilyShawwillintegrateRoadmapactivitieswithadditionalservicedeliveryactivitiesatGovEx
Thankyou
62
ReferencesAbrahamKGampHaskinsR(2017)Thepromiseofevidence-basedpolicymakingReportofthecommissiononevidence-basedpolicymakingWashingtonDCCommissiononEvidence-basedPolicymakingBigmove2Extendingthegreen(nd)Retrievedfromhttpswwwcityoflewisvillecomdoing-businesslewisville-2025big-move-2-extending-the-greenBatesMAampGlennersterR(2017)ThegeneralizabilitypuzzleRetrievedfromhttpsssirorgarticlesentrythe_generalizability_puzzleBuildwithnotfor(2018January)RetrievedfromhttpsbuildwithnotforwordpresscomCairneyPampOliverK(2017)Evidence-basedpolicymakingisnotlikeevidence-basedmedicinesohowfarshouldyougotobridgethedividebetweenevidenceandpolicyHealthResearchPolicyandSystemsdoi101186s12961-017-0192-xCairneyPOliverKampWellsteadA(2016)TobridgethedividebetweenevidenceandpolicyReduceambiguityasmuchasuncertaintyPublicAdministrationReview76(3)399-402Retrievedfromhttpdxdoiorg101111puar12555Centreforpublicimpact(nd)RetrievedfromhttpswwwcentreforpublicimpactorgCernaL(2013)ThenatureofpolicychangeandimplementationAreviewofdifferenttheoreticalapproachesOrganisationforEconomicCo-OperationandDevelopmentRetrievedfromhttpswwwoecdorgeduceriTheNatureofPolicyChangeandImplementationpdfColoradoSprings(nd)Retrievedfromhttpscoloradospringsgovcommunicationspagewhat-expect-when-you-speakupContinuumofcare(coc)program(nd)RetrievedfromhttpswwwhudexchangeinfoprogramscocCouncilpresidentkicksoff$12millionyouthfund(nd)Retrievedfromhttpwwwbaltimorecitycouncilcomcontentcouncil-president-kicks-12-million-youth-fundDurlakJAampDuPreEP(2008)ImplementationmattersAreviewofresearchontheinfluenceofimplementationonprogramoutcomesandthefactorsaffectingimplementationAmericanJournalofCommunityPsychology41327-350Retrievedfromhttpswwwncbinlmnihgovpubmed18322790Fishbonediagram(ishikawa)(nd)Retrievedfromhttpasqorglearn-about-qualitycause-analysis-toolsoverviewfishbonehtmlFivewhysGettingtotherootofaproblemquickly(nd)RetrievedfromhttpswwwmindtoolscompagesarticlenewTMC_5Whtm
References
63
FligsteinNampMcAdamD(2011)TowardageneraltheoryofstrategicactionfieldsSociologicalTheory29(1)1-26RetrievedfromhttpsociologyberkeleyedusitesdefaultfilesfacultyfligsteinFligstein20McAdam20Soc20Theory20PaperpdHeinkelP(2018January12)CitymullspermitsforbikesharingcompaniesRetrievedfromhttpwwwdentonrccomnewsdenton20180112city-mulls-permits-bike-sharing-companiesHowisareamedianincomecalculated(2012October)Retrievedfromhttpswwwhudexchangeinfofaqs843how-is-area-median-income-calculatedLampkinLMampHatryHP(2003)KeystepsinoutcomemanagementUrbanInstituteRetrievedfromhttpswwwurbanorgresearchpublicationkey-steps-outcome-managementMarshDampSmithM(2000)UnderstandingpolicynetworksTowardsadialecticalapproachPoliticalStudies484-21Retrievedfromhttpjournalssagepubcomdoiabs1011111467-924800247McCannL(nd)RetrievedfromhttplaurenellencomMoultonSampSandfortJR(2016)ThestrategicactionfieldframeworkforpolicyimplementationresearchPolicyStudiesJournal45(1)Retrievedfromhttponlinelibrarywileycomstore101111psj12147assetpsj12147pdfv=1ampt=jdkhhcp4amps=e184c625e1855a6b1615ccb99ee27b1c0b3212d8PaynterB(2018January1)What100citiesarelearningfromeachotherbysharingtheirdataRetrievedfromhttpswwwfastcompanycom40518004what-100-cities-are-learning-from-each-other-by-sharing-their-dataPeakacademy(nd)Retrievedfromhttpswwwdenvergovorgcontentdenvergovenmayors-officeprograms-initiativespeak-performancepeak-academyhtmlPinnockHBarwickMCarpenterCREldridgeSGrandesGGriffithsCJRycroft-MaloneJMeissnerPMurrayEPatelASheikhAampTaylorSJC(2017)Standardsforreportingimplementationstudies(stari)ExplanationandelaborationdocumentBMJOpenRetrievedfromhttpbmjopenbmjcomcontentbmjopen74e013318fullpdfProctorESilmereHRaghavanRHovmandPAaronsGBungerAGriffeyRampHensleyM(2011)OutcomesforimplementationresearchConceptualdistinctionsmeasurementchallengesandresearchagendaAdministrationandPolicyinMentalHealth38(2)65-76doi101007s10488-010-0319-7TandonSD(2018)ProgressincommunityhealthpartnershipsResearcheducationandactionRetrievedfromhttpswwwpressjhuedujournalsprogress-community-health-partnerships-research-education-and-actionProvanKGampKenisP(2007)ModesofnetworkgovernanceStructuremanagementandeffectivenessJournalofPublicAdministrationResearchandTheory
References
64
18(2)229-252Retrievedfromhttpsacademicoupcomjpartarticle182229935895SabatierPA(1988)Anadvocacycoalitionframeworkofpolicychangeandtheroleofpolicy-orientedlearningthereinPolicySciences21(3)129-168Retrievedfromhttpwwwjstororgstable4532139SandstromAampCarlssonL(2008)TheperformanceofpolicynetworksTherelationbetweennetworkstructureandnetworkperformancePolicyStudiesJournal36497-524Retrievedfromhttponlinelibrarywileycomdoi101111j1541-0072200800281xabstractSchwartzMampWilsonE(nd)WhocanaffordtoliveinahomeAlookatdatafromthe2006americancommunitysurveyRetrievedfromhttpswwwcensusgovhousingcensuspublicationswho-can-affordpdfShearerJCAbelsonAKouyateBLavisJNampWaltG(2016)WhydopolicieschangeInstitutionsinterestsideasandnetworksinthreecasesofpolicyreformHealthPolicyandPlanning31(9)1200-1211Retrievedfromhttpsdoiorg101093heapolczw052ThelessonsofadministrativedataHigh-profilepolicy-relevantresearchpoweredbyadministrativedata(nd)Retrievedfromhttpswwwpovertyactionlaborgsitesdefaultfilesresources20161114-The-Lessons-of-Administrative-DatapdfToolsandotherneedsforqualitativeandquantitativemethods(2015February)AgencyforHealthcareResearchandQualityRetrievedfromhttpwwwahrqgovprofessionalsquality-patient-safetytalkingqualityassesstoolshtml
WeaverRCampBagchi-SenS(2013)SpatialanalysisofurbandeclineThegeographyofblightAppliedGeography4061-70
References
65
UniversityPartners
Name Title Affiliation Area(s)ofexpertise
Contactinformation
Examplesofworkingwithnon-academicdecisionmakers
WendyPBarreno
AssociateDirector
WilsonSheehanLabforEconomicOpportunitiesUniversityofNotreDame
LEOsfocusareasincludehealtheducationhousingchildwelfareeconomicopportunitycriminaljustice
wbarrenonduduorleondedu
MelissaDavey-RothwellPhD
AssociateScientist
JohnsHopkinsBloombergSchoolofPublicHealth
Health mdavey1jhuedu410-614-5854
BaltimoreCityHealthDepartmentLeadersMarylandDepartmentofHealthAddictionRecoveryServicesinDCExecutiveDirectorsofcommunityorganizationsacrosstheUSthatfocusonHIVSTIsubstanceabuseandimplementationscience
ElizabethStuart
AssociateDeanforEducationandProfessorDepartmentsofMentalHealthBiostatisticsandHealthPolicyand
JohnsHopkinsBloombergSchoolofPublicHealth
Educationsocialpolicystatisticalmethods
estuartjhuedu
Resources
66
Management
Design-ThinkingResources
EngagementGuideforCivicTech
Journeymap
PerformanceManagementResources
DenverPeakAcademy
Proxymeasuresguide
Talkingaboutperformancewithoutcomplicatingit(video)
SettingPerformanceTargetsguide
Datavisualizationguide
Performancemanagementgettingstartedguide
Exercises
Fishboneexercise
5Whys
AdministrativeData
TheLessonsofAdministrativeData(JPAL)
EvaluationResources
Quantitativeandqualitativemethods(USDepartmentofHealthandHumanServicesAgencyforHealthcareResearchandQuality)
KeyStepsinOutcomeManagement(UrbanInstitute)
RCT-YES
Sageresearchmethods
BestPracticesforMixedMethodsResearchintheHealthSciences
ACCORDChecklist
Pilotandimplementationplanning
Benchmarkingexamplesandproxymeasures
Seattleopendataplaybook
Resources
67
Datasciencescopingguide
Resources
68
AboutGovExGovExhelpsgovernmentsusedataandevidencetoimproveresidentsrsquolivesWesupportgovernmentsinbuildingtheirinternalcapacitytooperatesustainableanalyticspracticeswhichimprovecommunityoutcomesinfieldsfrompublicsafetytoeconomicdevelopmentGovExhasaproventrackrecordofworkinginconcertwithpartnerstoprovidetechnicalassistanceandtrainingatscale
SinceourlaunchinApril2015GovExhas
Trainedover2000governmentemployeesinpersonandonlineintopicsincludingdatamanagementperformancemanagementopendatacommunityengagementandanalytics
Providedtechnicalassistancetomorethan100localgovernmentsacrosstheglobeimprovingtheiruseofdatatoachieveefficienciesinservicedeliveryandbetteroutcomesforresidents
Assessedthecapacityofmorethan130governmentalorganizationstodeterminehowleadersstaffandexternalstakeholdersmanageandusedata
Localgovernmentsareincreasinglyonthefrontlinesofaddressinglargescaleproblemswithglobalimplicationssuchasclimatechangepublicsafetyandeconomicinequality
Whileevidence-basedinterventionstohelpsolvethoseproblemsareavailablelocalgovernmentstaffhavelimitedtimeandcapacitytofindandimplementresearch-basedsolutionsAsaresultcitiescontinuetoworkthewaytheyalwayshave--makingincrementalimprovementstoprogramsbutnotalwayscreatingchangeatasystem-level
ThroughresourcedevelopmentandcollaborationwithJHUfacultyGovExissupportingcityleadersinusingdataandevidencetomakedecisionsandcreatinglastingrelationshipswithresearchers
TheRoadmapwasdevelopedbyGovExaspartofitspartnershipwithBloombergPhilanthropiesWhatWorksCitiesinitiative
AboutGovEx
69
- Title page
- Introduction and purpose of the Roadmap
- Using the Roadmap
-
- Gather information about the problem
- Strategically engage key partners
- Review the evidence and collect details
- Decide on a policy to implement based on local factors
- Establish a framework to monitor progress
- Assess data accessibility quality and security
- Design implementation and evaluation
- Share your story and success
-
- Working with a range of evidence
- Thank you
- References
- Resources
- About GovEx
-
WherearethepotentialroadblocksthatcouldaffectthescheduleIsthereanywaytomitigatethisHowlongwouldittaketoseeanimpact
Youranswers
Implementation
DowehavethestaffcapacitytoimplementthisArethereadditionalpartnersthatyoucanbringtothetabletosupportimplementationDoesthismakesensegivenourformofgovernment
Youranswers
Authority
DoesourcityhavetheauthoritytoenactthisIfsowhatistheappropriatemechanismAretherelegalbarriersArethereanyexistingpreemptionissuesfromthestateorfederalgovernmentCanthosebeaddressedthroughlegislationadvocacyIstherepoliticalsupportoroppositiontothis
Youranswers
Cost
HowmuchwillthiscostannuallyHowwillwepayforitCanthepolicybeimplementedwithexistingfunding
IfsoisthisfundingreliableHowwouldthispolicyoptionaffectourcitysbudgetintheshortandlongtermWhatotherfundingoptionscanweconsiderIsthereanalternativethatwouldcostlessandgiveussimilarresultsWillanyconstituencybeadverselyaffectediffundingforprogramsthatimpactthemisshiftedtoanotherprogrampolicyorinitiative
Youranswers
Decideonapolicytoimplementbasedonlocalfactors
32
EstablishaframeworktomonitorprogressNowthatyouhavereviewedtheevidenceconsideredlocalcontextandchosenyourpolicyitistimetodeterminehowyourcitywillmonitorprogressofthisnewapproachCitystaffandleadersneedtoestablishtheinfrastructureforongoingcontinuousdecisionmakingbasedonwhetherthepolicychangeisleadingtothedesiredoutcomesforresidentsandwhetherthechangeismeaningfulandrationaltofrontlineworkersSettingarbitrarygoalsatthelocallevelthatconflictwithgoalsatotherlevelsisconfusingtotheindividualsresponsibleforachievingthegoalsGoalsthataresettoohightoachievecanbedemoralizingtostaff
ThissectionincludesstepstosetupavenueinwhichCitystaffandleadershiproutinelyvisitkeyindicatorstodeterminewhatisorisnotworkingandtomakedecisionsinreal-timeFormorereadingonperformancemanagementpleasevisittheGovExPerformanceManagementGettingStartedGuide
InperformancemanagementtermssuchasindicatormeasureandmetricmaybeusedinterchangeablyInthisguidewewilluseindicatorInyourworkthemostimportantthingistobeconsistentinwhichevertermyouuseThisvideomaybehelpfulinunderstandingthisconceptandcommunicatingittoothers
Establishthefundamentalsandgatheryourteam
FirstdeterminewhetherthereisanestablishedvenueorregularmeetingscheduleduringwhichyoucandiscussprogressIfyourcityhasanexistingperformancemanagementstructureonarelatedtopicaddingadiscussionoftheprogressofpolicychangetoyouragendacanbeagreatideaIfthiswillbeastandalonemeetingdecidehowfrequentlytheprojectteamanddecisionmakerswillmeetandputthosemeetingsonthecalendarfortheentirelengthoftheproject
DeterminewhowillstaffandattendroutinemeetingsonCityprogressThesemeetingsmayincludetheMayorandorCityManagerdepartmentheadstaffwhoareimplementingthepolicychangeandanalysts
Identifyindicators
ItiscriticalthatindicatorsbeanimportantcityandcommunitypriorityaswellassomethingtheCityplaysamajorroleinaddressingTheremaybeinstancesinwhichtheCitywillneedtoreachouttoorcollaboratewithexternalpartnerstogetamorecompletepictureofprogressontheoverallpolicychange
Establishaframeworktomonitorprogress
33
Anoutcomeindicatordescribesthedesiredeffectofthetargetedpopulation
ExampleBellevuesfamilyfriendlyneighborhoodsindicatormeasuresthepercentofresidentswhobelievetheyliveinaneighborhoodthatsupportsfamiliesparticularlythosewithchildren
WithyourkeypartnersrevisityourproblemstatementcurrentstateandfuturestateexerciseandbegintoidentifytheoutcomeindicatorsthatwillillustratesuccessFornowchooseasingletop-levelkeyperformanceindicator(KPI)ItiscriticalthattheKPI-thepieceyouwillmeasure-beapriorityforthecommunityandsomethingtheCityplaysamajorroleinaddressing
Outputindicatorsmeasuretheprocessofprogramorservicedelivery
ExampleSanDiegoMayorKevinFaulconersgoaltorepair1000milesofstreetby2020
ReviewingtheadministrativedatasetinventorybegintoparedowntherelevantprogramsanddatasetsthatsupportyourKPIWhatarethemeasuresofeffectivenessforeachprogramTheseareyouroutputindicators
Thediagrambelowillustratestherelationshipbetweenoutputandoutcomeindicators
ThinkaboutimplementationindicatorsasmeasuringstepsintheprogramsrolloutthatarenecessarytoobtaindesiredchangesImplementationoutcomescanfocusontheinterventionstaffdeliveringtheinterventionanorganizationorsystemandcanaddressfeasibilityadaptabilitysustainabilityandcostTwokeyquestionstoconsiderwhendevelopingimplementationindicatorsare
1 Istheinterventionacceptableandappropriatetothecommunityand
2 IsitfeasibletoimplementinterventionXatthistimewiththeseresources
Establishaframeworktomonitorprogress
34
Amoreextensivelistwithdefinitionsofpotentialimplementationoutcomesislinkedhere
UsethetablebelowtochoosethedesiredimplementationoutcomesforyourpilotandtoplanyourmeasurementtechniqueThisshouldincludelevelofanalysis(individualproviderorconsumerorganizationorsetting)yourmeasurementtechnique(surveysobservationinterviewfocusgroupsadministrativedataanalysis)andthephasewhenyouwillcollectthisinformation(earlymidlate)Theremaybemorethanonelevelofanalysismeasurementtechniqueandphaseofdatacollection
Implementationoutcome
Definitionofoutcome
Levelofanalysisandmeasurement
Yourmeasurementtechnique
Phaseofpilotwhenyouwillcollectdata
Usethespacebelowtocaptureyourindicatorwishlist
Indicatorwishlist
Outcomes_____________________________________________________________________________
Outputs________________________________________________________________________________
Implementationoutcomes______________________________________________________________
Setapublichigh-levelgoal
Thehigh-levelgoaliscriticalinestablishingdirectiontogalvanizethepeoplewhowillbedoingtheworkWhendiscussingthegoalusetheMayorsbullypulpitLookforkeymilestonesinCitygovernmenttosharethegoalstatementsuchasthebeginningofaMayoraladministrationaStateoftheCityspeechorotherpublicremarks
GoalsshouldbeldquoSMARTrdquo
Specificaspossible(innamingthepopulationwhowillbebetterofftheactionthecitywilltake)
Measurable
Achievable
Establishaframeworktomonitorprogress
35
Results-focused
Time-bound
ThegoalstatementshouldincludeyouroutcomeindicatorlikeChattanoogasgoalldquoWewillendchronicveteranhomelessnessbyDecember2016rdquoAdiagramtohepyoubuildaSMARTgoalaroundyouroutcomeindicatorisbelow
Component Description Yourturn
Specific WhowillbebetteroffWhatactionthecitywilltake
Measurable Howwillyoumeasuresuccess
Achievable Isthegoalattainable
Results-focused Whatistheexpectedresult
Time-bound When
Setrealisticambitioustargetsandtimelinesforeachindicator
ToestablisharealisticandambitioustargetforyourgoalstatementusetheadministrativedatacollectedforyourchosenKPItoreviewactualCityperformanceinthemostrecentyearcomparedwiththethreeorfourmostrecentyearsworthofdataBegintocalculatesummarystatistics(egtotalminimumaveragemedianmaximum)toidentifytrendsCreatecumulativeandmonthlychartstoidentifytrendsandanomaliesBasedonyouranalysisofhistoricaldataignoretheanomaliesandassumetheproductivitytrendswillfollowthesameoverallpatternfrommonth-to-monthFinallyapplyanincreasedproductivityfactortotheperformanceeachmonthtoforecastfutureprogressBasedonthisanalysischooseatargetthatinspiresimprovementwhileaccountingforreality
Manycitiesthataremorematureintheirperformancemanagementpracticesrecognizetheimportanceoftakingacommunity-levelapproachtoimprovingoutcomesforresidentsThesecitiesarecreatinggoaltaxonomiesthatincludemeasuresrelatedtopublicsafetyhousingandevenrecreationCitiesdriveprogressonthesegoalsbyimplementingstrategiesthatengagemultipledepartmentsandcollaboratewithserviceprovidersinthecommunity
EventuallyyouwilldevelopgoalstatementsfortheoutputsinordertocapturetheefforttoachievetheoutcomecontainedinyourgoalstatementYoursetofindicatorsshouldbecomposedofshort-andinterim-termsupportivemeasureswithaccuratetargetsandrelevanttimelinesAnexamplegoaltaxonomyfromtheCityofLasVegassResultsVegasprogramisbelowUsethistaxonomyasamodelforyourownsuiteofindicatorsandgoalstatements
Establishaframeworktomonitorprogress
36
Yourturn
Thetemplatebelowismeanttohelpyoucraftagoalstatementforeachsupportingmeasure
Goalstatement
Departments
Outcome
Activities
KPI
Target
Timeframe
Conductanddiscussrelevantanalysis
ThrougharoutineseriesofmeetingscontinuallymonitorprogressofKPIssupportingindicatorsandanyotherdatacollectedsuchasqualitativedatatoidentifytrendsthatwillindicatewhetherthechosenpolicyishavingthedesiredimpactThesedatapointsshoulddrivethediscussionduringthesemeetingspointingtoopportunitiesforfurtherinquiryinto
Establishaframeworktomonitorprogress
37
whatisdrivingtheobservedtrendsAnalystsshouldbeencouragedandtrainedtoaskquestionsofthedepartmentsproducingthedataandconductsitevisitswhenpossibletodeepentheirunderstandingofthedataPresentationoftheanalysisshouldfocusononeideaatatimetokeeptheconversationfocusedonkeypointsandspecificopportunitiesforimprovementYoumayconsiderdevelopingapublicfacingdashboardtosharethisprogressmorebroadly
AgeneralcomparisonofthestatusquodatawithwhatwasobservedduringthepilotcanprovidedecisionmakerswithaquickunderstandingofthedifferencesbetweentheoldandnewapproachesalthoughthiswillnotshowcausalityOftenthesecomparisonswillincludeoperationalmetricssuchasdaysordollarssavedbutrememberthattheimpactonpeoplewillbemostimportantIfyouareworkingwithanevaluationpartnertheywillsupportthisstep
Usethespacebelowtosummarizetheresultsofyouranalysis
MeasureTargetvalue(seesectiononsettingtargetsabove)
Pilotvalue(whatdidyouobserveduringthepilot)
Historicalaverage(ifavailable)
Results(WhoisbetteroffIsanyonenegativelyimpacted1-2sentencesreassessingyourgoalstatement)
DocumentyourprocessandprogressincludingfeedbackfromtheparticipantsimplementationpartnersandtheprojectteamDidanythingunexpectedoccurduringthepilotArethereinternalandexternalfactorsthatmayaffectyouroutcomesThisiswheretheimplementationoutcomescanbecanhelpfulIftheinterventiondidnotgoasplannedconsiderwhetheritisbecausetheinterventionwasnotacceptableorfeasible
UsethetemplatebelowcreatedbyDenversPeakAcademytocapturethisinformationIfthe306090daytimelinedoesnotmakesenseforyourcityfeelfreetocustomizeit
CurrentState
FutureState
30Day
60Day
90day
Whatwentwell(+)
Whatdidnotgowellshouldbechanged(-)
IfaquestionordiscussionpointarisesthatcannotbeansweredinthemeetinganassignedstaffmembershouldcapturethequestionduringthemeetingAfterwardthefollowupitemsshouldbesenttorelevantpartnersandthenextmeetingshouldbeginwithadiscussionof
Establishaframeworktomonitorprogress
38
thoseitems
Takeaction
DiscussionsaboutwhetheryourindicatorsaremovingintherightdirectionandwhatmightbedrivingtrendsmayleadtoadiscoverythattherearenotenoughresourcesallocatedtothispolicyorissueTheremaybelimitedfundingforstaffprogrammingortechnologythatisdrivingthedatainthewrongdirectionIfCityleadershipisintheroomthisisagreattimetomakethecaseforadditionalfundingduringthebudgetseasonBycollectinganalyzinganddiscussingdataonthispolicyroutinelyyouarebuildingevidencetomakethecasefortherightleveloffundingtoreachtheestablishedtargets
Establishaframeworktomonitorprogress
39
Assessdataaccessibilityqualityandsecurity
Determineaccessibilityandqualityofdata
OnceyouhavecompiledyourwishlistofindicatorsitistimetodeterminetheaccessibilityandqualityofthedataTopreventcomplicationyoumaywishtoprioritizerelevantindicatorsforwhichyouarealreadycollectingdataAvoidthetemptationofadoptingatechnologysolutiontocollectnewdataatthispointsuchasaphotosurveytoolorotherappFornowyouaretryingtodeterminewhichdatasetsarerelevantaccessibleandhighqualityBesuretoworkwithyourkeyinternalpartnerswhileyouareidentifyingyourmeasuresDepartmentstaffwhoworkonprogramsarefamiliarwiththeindicatorsthatwillaccuratelysupporttheoutcomesanddatabaseadministratorswillbevaluablewhendiscussingaccessibilityandqualityofthedata
Abrieflistofconsiderationsfortheaccessibilityofyourdataisbelow
ArewecurrentlycollectingthisdatasetorwillwebecollectingitforthefirsttimeHowfarbackdoestherecordgo(timehorizon)WouldweneednewtechnologytocollectoranalyzethedataCanthedatabecollectedmanuallyorcanitbeautomatedHowfrequentlyisthedataproduced(continuouslyhourlydailyweeklymonthlyquarterlyyearlyetc)IsthedataavailablefortheentireCitypopulationorasubsetofthepopulationHowgranularisthedata(individuallevelCensusblockneighborhoodcitywideetc)DoesthedatacontainsensitiveinformationIsthereaproxymeasurethatwecanusefornowIsthedatamachinereadable
AsnotedbytheWilsonSheehanLabforEconomicOpportunities(LEO)attheUniversityofNotreDameleveragingadministrativedatacansignificantlyreducethecostsandtimelinesofrigorousevaluationCitiescanestablishaninternaldatasharingagreementacrossdepartmentsincludinglanguagethatenablesadministrativedatatobeusedforresearchpurposesCitiescanalsoidentifyexternaladministrativedata-holderssuchasacountyschooldistrictpolicedepartmenthospitaloranotherstatethatmayhaveoutcomedataimportantformeasuringprogramimpactCitiescanleverageexistingrelationshipsandorworkwithresearchinstitutionstodevelopstreamlineddatasharingprocessesandagreementswiththeseexternalentities
Assessdataaccessibilityqualityandsecurity
40
Evaluatethequalityofyourdata
QualityisasimportantasaccessibilityToevaluatedataqualityyoumaywishtocreatearubricliketheonebelow
Value ValueDescription
3 ExcellentDataQualityThedataiscompleteaccurateandupdatedregularly
2 AdequateDataQualityThedataissomewhatcompleteaccurateandupdatedfairlyregularly
1 PoorDataQualityThedataisnotcompleteaccurateorupdatedregularly
ThisisagreattimetobuildonthedatainventorythatyoustartedinthegatherinformationpageSuggestedadditionalfieldstoaddtoyourdatainventorytemplatearebelow
Program DatasetTitle
DatasetDescription DataSource
Departmentorexternalpartner
DoestheCityalreadycollectthisdata
Whatisthenameoftheprogram
Human-readablenameofthedataset
Shouldbeunderstandablebynon-technicalusers
Whatistheoriginalsystemapplicationfilethathousesit
Departmentorexternalpartnerthatmaintainsthedata
Isthisdataalreadycollected
Updatefrequency-ThefileformatforthedatasetTypicallycsvxlsxshapefiletxtetc
Format-Howoftenthedatasetisupdated
Timerange-Theperiodoftimecoveredinthisparticulardataset
Datause-Howisthisdatasetcommonlyused
Datausers-Whotypicallyusesthisdataset
Dataownerstitle-Titleofthedataowner
Dataownersposition-DataOwnersjobtitle
Dataownersemailaddress-DataOwnersemailaddressWhomanagesthedataandorisresponsibleforgrantingpermissiontoaccessthedataWhounderstandswhatthedatasetincludesandcananswerquestionsaboutit
Dataqualityconcerns-Describetheconcernsyoumayhaveaboutthequalityofthedataset
Assessdataaccessibilityqualityandsecurity
41
Division-DivisionwherethedataismaintainedIfmorethanonedivisionisresponsiblelisttheprimaryownerofthedataset
Sensitivedatacomments-DescribetheissuesthatwouldpreventthecityfromsharingthedatapubliclyCommonsensitivityconcernsincludeprivacyviolationssecurityissuesorhighcostorstaffdemandstopostthedata
Publishingstatus-Thistheplacetoputanycommentsregardingthecurrentstatusofthedataset
Publishedlink-Thisisthelinktothepublisheddatasetontheopendataportalorcitywebsite
Reviewdatasecurityandprivacyprotectionprotocols
OftenapolicychangealsoinvolvesachangeinthedatathatisbeingcollectedhowthatdataisbeinghandledwhohasaccesstoitandahostofotheradjustmentsAsyoutakestockofthedataassetsthatareavailableforyourinitiativeitisvitaltoalsothinkthroughwhatwillbedonetoprotectthatdataandtheprivacyofthepeopleitrepresentsAbigpartofgooddatasecurityinvolvesstrongdatagovernancepracticesthatdefineandidentifysensitivedataandconsciouslyworktobalancethevalueandriskofstoringsuchsensitivedataOnewayofevaluatingyourdatacomesfromcaselawrelatedtotheUSFreedomofInformationActwhichwehavetranslatedtothechecklistbelow
1 Woulddisclosureresultinasubstantialinvasionofprivacy2 Haveweconsideredtheextentorvalueofthepublicinterestandthepurposeorobject
oftheindividualsseekingdisclosure3 Istheinformationavailablefromothersources4 Wastheinformationgivenwithanexpectationofconfidentiality5 Isitpossibletomouldreliefsoastolimittheinvasionofindividualprivacy
TheseriskscouldbemitigatedusingavarietyofstrategiesfromaggregatingdatatoobfuscatingindividualstoredactingfieldsthatmayincludePersonallyIdentifiableInformation(PII)orre-evaluatingyourdatacollectionmethodstofindalternativewaysofgettingtheinformationyouneedThesestepsshouldbecodifiedandmanagedbyadatagovernancecommitteeifpossibleForthesensitivedatathatisdeemedessentialcreateaplanaroundpotentialdatabreachesinordertoquicklyandconfidentlyrespondtoemergencysituations
ProactivelypreparingforthesesituationsisoftenthebeststrategyfordatasecuritybutdonotforgetthetechnicalsideInvolveyourITadministratorsinconversationsaboutdatastorageandaccessensuringthattheywillbeprotectedbyenterprise-gradecybersecuritysystems
Assessdataaccessibilityqualityandsecurity
42
Assessdataaccessibilityqualityandsecurity
43
Designimplementationandevaluation
Designtheimplementation
NowthatyouhaveyouraccountabilityplanandkeyindicatorsitistimetogetspecificaboutyourplanforimplementationThetemplatebelowwillhelpyouthinkthroughthespecificmechanismsofimplementingpolicychange
UnderstandingorganizationsandsystemsiscriticaltounderstandingandmakingdecisionsaboutwhereandhowinterventionsshouldbeappliedApolicyimplementedinasinglesettingmayshowpromisingresultsbutimplementingmultiplepoliciesandprogramsatthesametimewithdifferentactorsresponsibleforeachaspectcanhelpensuresustainablebehaviorchange
InSomervilleMAShapeUpSomerville(SUS)isacitywidepublichealthprogramthatisimplementedacrossmultiplelevelsofinfluenceandsystemsthroughinterventionsbeforeduringandafterschooltoincludechangestobuiltenvironmentsuchasbikelanesregionalmasstransportationandphysicaleducationequipmentinschoolsandgymstoincreasephysicalactivitywellnesspoliciesandschoolgardensnutritioneducationduringandafterschoolandhealthyrestaurantprogramstosupporthealthyfoodandhealthyeatinghabits
Usethetemplatebelowtoplanyourimplementation
Policyimplementationplanningtemplate
ListthespecificdesiredoutcomesInthepreviousstepyouworkedwithdepartmentstafftoestablishspecificmeasurableoutcomesandinterimindicatorsPrioritizetheindicatorsthatwilldemonstrateprogressUsethespacebelowtocapturethedesiredoutcomesofthepolicychange
ExampleShapeupSomervillebeganwiththegoalofpreventingobesityin1stthrough3rdgraders
Yourturn
____________________________________________________________________________
____________________________________________________________________________
Designimplementationandevaluation
44
SpecifythetargetpopulationAreyoudesigninganinterventionthattargetsonlycertainsitesAreyouplanningonimplementingaprogramorinitiativeforaspecifictargetaudienceClearlyspecifythepopulationyouplantotarget
IfyouplantoconductarigorousimpactevaluationasnotedbytheWilsonSheehanLabforEconomicOpportunitiesitisimportanttoestablishconcreteobjectiveeligibilitycriteriaandensurethatthesecriteriaareappliedconsistentlybyallstaffduringscreeningandintakeItisalsovitaltoidentifytheindividualswhoareeligibleforaparticularprogramorserviceevenifeveryoneeligiblecannotbeservedbyexistingresourcesThishelpscitiesidentifyanyunmetneedsandguidefutureresourceallocationwhilealsocreatingthepossibilityofarigorousimpactevaluationbyidentifyingapotentialcomparisongroupFinallywhenconductingscreeningandintakeitisimperativetocollectandmaintainbasicinformation(egnamedateofbirthgender)aboutallprogram-eligibleindividualsnotjustthosewhoparticipateintheprogram
ExampleIntheShapeUpSomervilleexampletheprogrambeganbytargeting1stthrough3rdgraders
Yourturn
____________________________________________________________________________
____________________________________________________________________________
DefinethelocationWherewillthepolicychangetakeplaceWillimplementationtakeplaceatonesiteormultiplesites
ExampleShapeUpSomervilleinterventionsweredeliveredinschools
Yourturn
____________________________________________________________________________
____________________________________________________________________________
ClarifyimplementationpartnersandarticulatetheirrolesWhatarethekeyorganizationsinstitutionsorpeopleresponsibleforimplementationTheprojectteamshouldbeasclearandspecificaspossibleinprovidingadescriptionoftheconcreteactivitiesthatimplementationpartnersaremeanttocarryoutPracticeprofilesmaybeahelpfultooltohelpdefinewhatisrequiredoftheactorswhoareresponsibleforimplementingtheintervention
Designimplementationandevaluation
45
ExampleInSomervillemayoralleadershipandengagementofmultiplecityagencies(publichealthprimarycareschoolsCambridgeHealthAllianceetc)streamlinedandconnectedthevariousShapeUpSomervilleinitiativesandgrantswhichhelpedtokeepthetalentandenergyfocusedonhealthyeatingandactivelivingforallsectorsofSomervilleSUSpartnersultimatelyincludedschoolsnonprofitsCityandstateagenciesandnationalfoundations
Yourturn
____________________________________________________________________________
____________________________________________________________________________
DescribetheinterventionWhatarethestepsactivitiesorprocessesRememberthatthemechanismthespecificstepsorprocessthatleadtobehaviorchangeiskeytounderstandingtheeffectivenessofthepolicychangeandsharingtheresultswithothers
ExampleGoalsofShapeUpSomervilleweretoinfluenceeverypartofanearlyelementaryschoolchildrsquosdayndashbeforeduringandafter-schooltopromotehealthyeatingphysicalactivityandtheirweightstatusInterventionincludedfoodserviceenhancementstoimprovebreakfastandlunchandstaffprofessionaldevelopmentwalktoschoolactivitiesSUSclassroomcurriculumandprofessionaldevelopmentschoolwellnesspolicydevelopmentSUSafter-schoolcurriculumandprofessionaldevelopmentoutreachandeducationtohomethroughmaterialsforumseventsldquoSUSapprovedrdquorestaurantscommunityoutreachandcapacitybuildingthroughpolicydevelopmenttrainingsmediaplacements
Yourturn
____________________________________________________________________________
____________________________________________________________________________
ClarifytimelineandmilestonesfordeliveryandevaluationHowfrequentlywilltheservicesinterventionbedeliveredHowlongwilltheprogramrunWhatisthetimetoexpectedimpacts(eg12monthsafterprogramcompletion)
Designimplementationandevaluation
46
ExampleShapeUpSomervilleledbyTuftsUniversityFriedmanSchoolofNutritionScienceandPolicyconductedaquasi-experimental(nonrandomized)interventionstudyamong1stthrough3rdgradestudentsinSomerville(intervention)andtwocomparisoncommunitiesfor9-monthsduringoneschoolyear(fall2002ndashspring2003)ThestudyshowedpositiveassociationoftheinterventiononchangeinBMISUSreleasedreportssummarizingevaluationsprogresstodateplansforadditionalprojectsandimpactstenandfifteenyearsintotheinitiative
Yourturn
____________________________________________________________________________
____________________________________________________________________________
PlanyouranalysisTheprevioussectionaddressedestablishinganaccountabilityframeworktomonitorprogressandimplementationDeterminewhichindicatorswillbediscussthroughthatprocesswhowillsharethedataandanalysisandhowfeedbackfromthoseconversationswillbeincorporatedbytheprojectteam
ExampleTheMayorrsquosofficeimplementedadata-drivendecisionmakingapproachcalledSomerStatwhichwassynergisticwiththeapproachestakenbytheresearchandevaluationteamsdeployedforthespecificgrants
Yourturn
____________________________________________________________________________
____________________________________________________________________________
PlanforsustainabilityIfthepolicyandprogramchangeissuccessfulhowwilltheprojectteamsustainandexpandtoadditionalsitesorpopulations
ExampleAlthoughmostofthepartnershipsforgedintheearlyyearsweremaintainednewgrantsandcollaborationsexpandedattentiontoolderpublicschoolstudentsactivetransportationandtheGreenLinesubwayextensionthebuiltenvironmentfarmerrsquosmarketeffortssuchasldquoGrowninSomervillerdquowithUnionMainStreeteconomicdevelopmentandhealthdisparitiesaswellasdatacollectionmonitoringandsurveillance
Yourturn
Designimplementationandevaluation
47
____________________________________________________________________________
____________________________________________________________________________
Pilotplanningtemplate
IfyouplantoconductapilotfirstdeterminethepurposeofthepilotDoyouwanttoobservewhetherthepolicychangeresultsinthedesiredbehaviorchangeAreyouconcernedaboutfeasibilityscalabilityorcostDoyouwanttotestwhetheryourdesiredresultscanbeachievedthroughthispolicyapproachClarifythegoalofthepilotandthequestionsyouhopetoanswerThiswillhelpyouidentifythekeyindicatorsthatyouwillusetomeasurewhetherthepilotissuccessfulBeforebeginningestablishabaselineofcurrentperformanceandsetyourtargetsaccordinglybasedonwhatyouhopetoaccomplishthroughthepilot
NextestablishthepilotplanforyourprojectteamincludingmilestonesforteamcheckinstodeterminewhethertomoveaheadorchangecourseThisisseparatefromthemeetingsthatwilloccuraspartoftheaccountabilityframeworkUsethespacebelowtocreateyourtimelineanddevelopyourmeetingschedule
Task Personresponsible Startdate Checkindate Deliverable
Planyourevaluation
WhileyouaredesigningtheimplementationitisalsotimetobegintodevelopyourplanforanalysisAsnotedbytheWilsonSheehanLabforEconomicOpportunitiesdecisionsaboutprogramsetup-includingoutreachscreeningintakedatatrackingandenrollment-candeterminewhetherornotarigorousimpactevaluationwillbepossibleAsdiscussedcityleadersmayfindithelpfultoincludeevaluationpartnersinearlyconversationsaboutprogramsetupDuringpreviousstagesyoumayhaveworkedwitharesearchertoplanyourproposedpolicychangeYoushouldbeclearaboutthepurposeoftheevaluationanddevelopanevaluationplanBelowareadditionalconsiderations
Causationvstrendanalysis
IntheEstablishaframeworktomonitorprogressstagetherewasguidanceontrackingtrendsovertimeItisimportanttonoteherethatmonitoringdataisdifferentfromanalyzingacausaleffectTrendanalysiswillillustratethatchangesareoccurringandmayleadtoquestionsthatcouldbeansweredbyyourevaluationbuttrendanalysisalonedoesnotprovideevidenceaboutwhatisdrivingthosechangesYoumayseechangesinyour
Designimplementationandevaluation
48
outcomeindicatorsbasedonvariablesoutsideoftheCityscontrolForinstanceachangeinfederaltradepolicycanhaveanimpactemploymentratesinyourCityoutsideofyourCitysnewjobtrainingandplacementprogram
ToaddresscausalityyoumustcomparetheoutcomesofindividualswhoreceivedtheinterventionwiththosewhodidnotWithoutacomparisongroupofsimilarindividualswhowerenotexposedtotheinterventionitisimpossibletoknowwhichfactorsareresponsiblefortheoutcomesRandomizedControlTrials(RCTs)inwhichindividualsarerandomlyassignedtotreatmentandcontrolgroupsarethegoldstandardinprovingcausalityIfyoucandemonstratethatoutcomesimprovedforthosewhoreceivedtheinterventionanddidnotimproveatallorimprovedlessforacomparisongroupyoucanbesomewhatcertainthattheinterventionwasthecause
Examplesofothernon-causalimpactevaluationsthatarerigorousandreliableincludecertainquasi-experimentalstudiesusingcomparisongroupdesignssmartinterruptedtimeseriesanalysesthatmodeltrendsovertimeinwaysthatallowestimationofinterventioneffectswelldesignedpre-posttestanalysesneedsassessmentsimplementationstudiesetcFormoreinformationondifferenttypesofevaluationsseetheSageResearchMethodswebsite
Lookfordifferencesingroups
BuildintimetoalsoexaminethepilotdatatoseeifgroupsareexperiencingdifferentresultsReviewingperformancebasedonoutcomemeasuresforapopulationinaggregatemayhidethesubtlewaysthatsomepeoplemaybeaffecteddifferentlysoitisnecessarytocarefullyanalyzethedatafordisparateoutcomesForinstancetheMovingtoOpportunity(MTO)studythatexaminedtheimpactofhousingvouchersonfamilieswholivedinhigh-povertyneighborhoodsfoundthatchildrenwhomovedbeforetheageof13hadbettereducationandhigherincomesasadultsHoweveradultsdidnotseeimprovededucationandemploymentoutcomes
Disaggregatingdataintomorespecificcategoriesofraceandethnicitycanhelpunderstandtrendsanddisparitiesinamoregranularwaywhichcanhelpdecisionmakerstargetpoliciesandinterventionsinamoreequitableway
EarlierintheRoadmapweworkedtoidentifystakeholdersandwhomightbepositivelyandnegativelyimpactedbyachangeinpolicyUsetheresultsofthatexercisetoinvolvethecommunityininterpretingdataanalysisandtogatherqualitativefeedbackfromthecommunitytocallattentiontoissuesForinstanceanincreaseinyouthdetentionmaybeexplainedbyinefficienciesorlackofaccesstotransportation
Designimplementationandevaluation
49
Designimplementationandevaluation
50
Shareyourstoryandsuccess
Sharetheresults
Storytelling
AnintegralstepinyourstrategyissharingtheresultsofyourpilotorpolicychangewithyourcommunityThisincludesdevelopingmessagingaroundtheworkofyourorganizationandcreatingaplantodisseminateyourstoriesWhytellstoriesaboutyourwork
Informandorunderstandasubjectorsituation
Giveneededcontextofdataorbitsofinformation
Relatablestoriesarememorableforreadersviewers
ReadersorviewerswillseetheinformationasmorecompellingwhenthereisademonstratedandrelatablechangeorimpactonthemselvesortheirfriendsneighborsandcolleaguesStoriesareaboutpeopleandgovernmentisfundamentallyabouthelpingpeopleSotellthestoriesofthepeopleyouandyourcitycolleagueshelped
Keepthesetipsinmindasyoudevelopyourstorywithyourteam
WhatwereyourfindingsandwhyshouldanyonecareTellyouraudiencehowtheproblemwassolvedWhathastheimpactbeenonresidentsandcitystafftodateWhatwasthesuccessandweretheremultiplesuccessesalongthewayIfyouhavepreliminaryresultsordatashowingimprovementtosharethisisthetimetodoit
WhowasinvolvedandimpactedReviewthelistofstakeholdersHowwereeachofthemimpactedbytheprogramHumanizeyourstoryandimpactbyaddingadditionalvoicesThesevoicescanbeoneofthefollowing
Apersonorfamilywhowaspositivelyaffectedbythechangesuccess
Aninvolvedthird-partydescribingboththeproblemandthesolution(ieanorganizationnonprofitprovidersbusinesstradeassociationsportsteametc)whoispartofthecommunitythatthegovernmentworkedwithonthechangesuccess
Someonefromthegovernmentitselfwhocanspeakaboutthechangesuccess
TellmoreaboutyourcityDescribeyourcityincludingpopulationdemographicsindustriesetcWhatareitschallengesWhatisitknownforWhatmakesituniqueHowisitsimilarordifferentfromothercitiesintheUS
Shareyourstoryandsuccess
51
DescribeyourcityrsquoschallengeWriteoutthechallengeyourcityisfacing(asdefinedinyourproblemstatement)HowdoesitaffectresidentsHowdoesthisproblemaffecttheworkofcitystaffmembers
AddthedetailsProvidedetailsonyourappliedresearchstrategyTalkaboutwhatyourcityhasdonenthepasttoresolvetheissueDiscusshowyourstrategyhaschangedwhenyoubegantoadoptthisapproachWasmoneysavedorinvested
Usethisstorymapexercisetoorganizeyourthoughtsandguaranteethatyouincludealloftheelementsofatraditionalstory
AfteryoudevelopyourmessagingcreateacommunicationsmarketingandoutreachstrategydesignedtospreadyourmessageYourstrategyshouldwalkyouthroughthefollowingkeyactivities
1 IdentifyandprofileyouraudienceWhencreatinganddisseminatingyourmessageitisalwaysgoodtothinkaboutyouraudienceWhoexactlywouldbeinterestedinwhatyouhavetosayThegeneralpopulationofthecityPeoplewhoworkinnonprofitsBusinessownersItmattersbecausenoteveryonewholivesinacitymaybethetargetoftheinformationBusinessownersmaybemoreinterestedindatashowingeffectsonthebusinessdistrictthanothersandsoonWewillcoverthislaterbutthisstepwillalsohelpyoutothinkofthemosteffectivewaytoreachyouraudience
2 Defineyourcalltoaction(ifyouhaveone)SpendsometimeconsideringwhyyouareinterestedintellingyourstoryDoyousimplywanttoupdatethepubliconyourprogressorcelebrateyoursuccessesAreyouinterestedinyouraudiencechangingitsbehaviororsupportingaspecificinitiativebeinglaunchedbyyourorganizationIfyouwanttohaveyouraudienceactorbehaveinacertainwaybespecificandincorporateitinyourmessaging
Shareyourstoryandsuccess
52
3 SelectperformancemeasuresThinkabouthowyouwoulddetermineifyourcampaignissuccessfulandselectappropriateperformancemeasuresYouwillrevisitthesemeasuresattheendofyourcampaigncommunicationsplanimplementationtodeterminewhetheryouneedtomodifyyourcommunicationsandoutreachstrategyforthefuture
4 MethodstocommunicateyourstoryTherearemanywaystocommunicatesuccessofyourprojectHerearesomesuggestionsonreachingaudiencesItisimportanttodeterminewhichmethodsarethemosteffectiveforyourdesiredaudiencetoreceiveinformationDoyourresearch(PewResearchCenterhasexcellentstudiesonthereachofmediadigitalmediaandmarketing)ForexampleifseniorsareyourtargetpopulationandtheyarelessinclinedtouseInstagram(thatistrue)youmaywanttoconsiderFacebookorthenightlylocalnewsbothofwhichareusedmoreoftenbyseniorsBelowisalistofsomemethodsandtoolsyoucanuse
i Pressreleasestolocalnewspapersandtvstations
ii ShortsmartphonevideosAgreatwaytoengageanaudienceisthroughvideoAskonequestionofsomeonewhoispartofthestoryMaybethereisasocialworkerwhocandiscusshungerinyourcityThesevideoscanbereusedonmultipleplatforms
iii Long-formwriting
iv TellingindividualstoriesthroughaseriesofphotosandshortnarrativesforpublicationonFacebookorInstagram
v Op-edsopinionpiecesplacedinthelocalpaper
vi Emailupdatestoyourlist
vii Connectingwithappropriateorganizationsthatareinvestedinthetopicandworkwiththecommunityandaskthemtodisseminateinformationtotheirmembers
viii EmployingldquoWordofMouthrdquo(WOM)tacticsbycreatingtownhallsorfreeeventsforresidentstoattendintheirownneighborhoods
5 DeterminethetypeofcollateralUsuallycampaignsthattargetresidentsrequireseveralavenuestoreachthemPeoplearenotalwaysinthesameplaceorspaceHavingakeenunderstandingofyouraudiencewillassistyouwithyourchoicesonwhatneedstobedevelopedSometimesthatmeansimagesforsocialmediaaonepageflierthatgivespeopleinformationonattendinganeventoranin-depthbrochure
Shareyourstoryandsuccess
53
6 ImplementyourstrategyKeepinmindstrategiescanbefluidSometimessomethingmaynotturnoutasexpectedevenifduringplanningsessionsitwasldquosupposedtordquoItisokYoucananalyzewhathappenedduringadebriefsessionandonceyoursurveysandothermeasurementshavebeenfinalized
Visualizations
Withtherightvisualizationyoucanreviewandshareinformationinsightsanalysisperformanceandtrendsallofwhichisvaluabletoacross-sectionofthepublicButwhichvisualizationisrightforyouTochoosetherighttypeofchartgraphdeterminethetypeofquestionyouwanttoanswerSomecommoncharttypesarebelowandmoreinformationcanbefoundhere
Exploreadditionalanalyticaltechniques
Duringthegoal-settingphaseyouestablishedoutputoutcomeandimplementationmeasuresBasedontheresultsofyourpilotyoumayneedtoadjustyourmeasuresWiththepilotdatainhandyoumayhavesomeideasofhowfutureanalysescouldbedifferentandpotentiallyimprovethequalityofyourprogramTakesometimetobrainstormotherrelevantdatasetsthatyoudidnothavethetimetopursueearlierincludingmessydepartmentaldatathatneedsmoreattentionorsourcesofpublicdatafromthestateandfederalgovernmentnonprofitsandevenprivatecompaniesthatcouldsupplementyourunderstandingofanissueUsingmachinelearningtechniquesyoumightbeabletobetter
Shareyourstoryandsuccess
54
predictdemandforhomelessservicesusingNOAAweatherforecastspreemptwatermainbreaksusingsoilstratigraphyorinvestineconomicdevelopmentbymonitoringrealestatelistingdata
Advancedanalyticaltechniquescanalsohelpyourimplementationrunmoreeffectivelybyintegratingdata-informedtargetinganddecisiontoolsintotheprocessByautomatingroutineworkflowsandsurfacingsubtletrendsthesetoolscanhelpyouiterateinamoreprecisemannerandprovidekeystrategicguidancetofront-lineemployeesthatwouldotherwisebeunavailableThesesortsofdata-science-enhancedprocessescanbeclassifiedasoneofsixprojecttypesasexploredfurtherbytheNewOrleansLANOLAlyticsteamFormoreguidanceonscopingasimilarprojecttheGovExadvancedanalyticsscopingguidegivesyouastep-by-stepoverviewofhowtobesuccessful
Usethespacebelowtolistyourideasforpossibleadvancedanalyticsprojectsbytype
Typeofproject Ideas
Findingtheneedleinthehaystack
Prioritizingforimpact
Earlywarningtools
Betterquickerdecisions
Optimizingresourceallocation
Experimentingforwhatworks
Keepthefeedbackloopgoing
GatherinformationonpublicsentimentaboutpolicychangethroughfocusgroupssurveysandpublicmeetingsledbyadvisorygroupmembersNotewhethernewinformationisavailablethatcanhelpanalyzeorcontextualizeyourprogress(egfederalorstatedatareports)Sharetheresultsofyourpilotandthepolicychangeprocesswithothercitiesandresearchers
Shareyourstoryandsuccess
55
WorkingwitharangeofevidencePresentlythereisalotofattentionandinvestmentinevidence-basedpolicymakingincludingthefederalbipartisanCommissiononEvidence-BasedPolicymakingthePewCharitableTrustsreportsandtoolkitsonHowStatesEngageinEvidence-BasedPolicymakingandtheLauraandJohnArnoldFoundationsupportedEvidence-BasedPolicymakingCollaborativeManyoftheseinitiativesfocusonrigorousevaluation--meaningrandomizedcontrolledtrials(RCT)orotherexperimentalmodels--asthebestkindofevidencetouseHoweverthisbodyofevidenceisofteninsufficientforthespecifickindsofquestionsthatgovernmentscanhavewhenmakingadecisionaboutpolicyInthisguideweproposeanapproachtosupportgovernmentsinincorporatingadditionaltypesofevidenceintotheirdecisionmaking
OnewaytounderstandarangeofevidenceistouseNestarsquosStandardsofEvidencetoframeanddefineevidenceTheBritishnonprofitNestaworkstoinjectpositivechangeintothewayUKgovernmentpoliciesandprogramsfunctionOneoftheirkeyareasofworkhasbeendevelopingWhatWorksCentresorganizationsthatcollectcommissionsynthesizeanddisseminateevidenceaboutinterventioneffectivenessacrossarangeofsocialimpactareasNestadevelopedthefollowingStandardsofEvidencetoprovideausefulwaytodocumenthowdifferentkindsofinformationdemonstratedifferentlevelsofevidence
Workingwitharangeofevidence
56
Theimportanceofpotentiallybiasedinformation
Onechallengepresentedbyanevidenceclassificationthatacknowledgesthevalueofnon-scientificformsofevidenceistofigureouthowtobalanceallofthepiecestogetherwhenyouusethemTounderstandhowtodothiswehavetothinkabouthowtopositionpotentiallybiasedinformationindecisionmaking
WhenwelookatinformationwhichhasnotbeenrigorouslyevaluatedweacceptthatthisinformationmightbebiasedThewholereasonweconductrigorousevaluationistoremovebiasfromourobservationsWeconductstructuredtrialstoproduceresultsthatareconsistentandgeneralizableIfafindingisunbiasedthisaccuracymdashthislackofbiasmdashcanbeprovedbecauseanyindependenttestercanreproduceitwhentheyfollowanidenticalprocess
YoucanlookatpotentiallybiasedinformationasjustinformationthatisearlieroninitspathtorefinementreplicationandverifiedaccuracyApieceofinformationbasedonanyoneindividualrsquosobservationsislikelytobebiasedbecauseitiscomingfromjustonesourceBiasisreducedbygatheringtogetherlotsofindependentsourcesofinformationandfiguringoutitsmostcommonattributes
MaybethebiasedinformationmakesaclaimthatistoolargemdashitsayssomethingistrueunderallconditionswhenitisreallyonlytrueundercertainconditionsorthataneffectofaninterventionisbiggerthanitstrueaverageorthataneffectlastsovertimewhenitdoesnotPerhapschangingtheclaimtobemorespecificmdashcriticallyreviewingtheinformationtoseeifspecificconditionsmustbemettoachievetheoutcomesmdashwouldmaketheclaimmoreaccurate
IfyoutestedyourclaimfounditdidnotholdupinsomecaseschangedittesteditagainuntilyourclaimrepeatedlyheldupthenyoucoulddemonstratethattheclaimwasnolongerbiasedbutaccurateThisprocessoftestingimprovingtestingandvalidatingiswhatwegetfromstructuredscientificstudy
Withindividualpiecesofinformation--crediblehypothesescasestudiesexpertopinion--youcannotassumethatkindofaccuracyAtthesametimetheseearlyobservationsarefoundationaltoanyreplicablescientificobservationsWewillnevergetanywhereifwedonotstartwiththesepiecesofinformationeveniftheymightbebiased
Combininglevelsofevidencetogetthemostdetailedinformationforyourcity
ItisespeciallyimportanttoacceptdifferentformsofevidencewhenyoufacequestionsthathavenotyetbeentestedbyindependentstudiesSomeofthemostimportantquestionsrelatetothingsthatareveryspecifictoyoursituationYoufaceaparticularcombinationof
Workingwitharangeofevidence
57
issuesactorstrendsandresourcesthatisultimatelydifferentfromeveryotherplaceintheworld
ThisisimportantbecauseaspolicypractitionersandnotscientistsyoucaremoreaboutyourspecificcasethanahypotheticalaveragegeneralcaseYoucarelessaboutthetotallikelyrangeofimpactsfromapolicythanhowthissingleimplementationwillworkYouwantinformationaboutpoliciesthatwillprovideyouwiththebestpossiblepredictionabouthowtheywillworkinyourcontext
ByputtingdifferentkindsofevidenceintothesameframeworkNestarsquosStandardsofEvidenceimplythatwhenyoucannotgetrigorouslytestedexperimentallyverifiedinformationitisappropriatetoworkwithwhatwedohaveThisisrelevantforthemunicipalimplementationcontextbecausenostudypulledfromaliteraturereviewwillaccountforallofthevariablesyoufaceinyourowncontext
Intheabsenceofhavingastudythatpredictshowaninterventionworksinyourspecificcontextyouwillneednon-scientificinformationtotrytofillinthosegapsYouwillneedtogatherstoriesfromresidentsstaffmembersandotherstakeholdersthatsitonLevels1or2ofthestandardsofevidence
Someofthekindsofinformationyouwillprobablygetmoreeasilyfrompeoplethanstudiesinclude
howpeopleincitieslikeyoursreactedtoaninterventioninpublicforumshowpeoplewhoimplementedaninterventioninacitylikeyoursfiguredouthowtogetitpassedhowthecitycommunicatedpubliclyabouttheapparentbenefitsanddrawbacksoftheinterventionthepoliticalacceptabilityoftheintervention
Youcanremainsimultaneouslyawarethattheinformationyouaregatheringthroughindividualstoriesisbiased--notgeneralizableinformedbyspecificperspectives--whilealsorecognizingthathavingsomeinformationaboutsomethingimportantisbetterthannoinformationEvidencefromthefirsttwoevidencelevelsareessentialbuildingblocksofknowledgeandperfectlyusefulwhenyoucannotgetdetailedunbiasedknowledge
Youhavetostartsomewhereandyoushouldalwaysstartwithevidenceyouhaveaboutyourveryspecificconditionsthatyouknowareimportantforsuccess
Workingwitharangeofevidence
58
Thankyou
Interviewswithcities
GovExheldsemi-structuredinterviewswithsixstaffmembersinfourcitiestoestablishtheinitialstepsoftheRoadmapWeaskedrespondentstotalkaboutaninitiativeorpolicythattheyhavebeenresponsibleforimplementingintheircityRespondentsfromthreecitiesspokeabouttheireffortstoaddresshomelessnessandrespondentsfromonecityspokeaboutblightAfterrespondentsidentifiedtheproblemtheyspokeabouthowthecityprioritizedthisissuethevariousstepsinvolvedandbarrierspresent
GovExconductedtextanalysisoftheseinterviewswhichrevealedthatpeoplewasthemostcommonlyusedtermduringtheseconversationswhendiscussingthestepstochangeaprogramorpolicyThisisathemethroughoutthisRoadmapThefirstsectionbeginswithidentifyingtherelevantpeopleTheconversationsaroundevidencearefocusedonhumanbehaviorratherthanmethodologyorstrengthofevidence
Thankyou
59
OncewehadadraftoftheRoadmapwefollowedupwithcitiesonanindividualbasistolearnmoreaboutwhatwouldbehelpfulWearegratefultothefollowingcitystafffortheinterviewsandindividualconversations
MichaelBaskinChiefPolicyOfficerChattanoogaTN
MelissaBridgesPerformanceandInnovationCoordinatorLittleRockAR
VictoriaCarreoacutenAdministrativeOfficerLasVegasNV
TyroneGrandstrandCommunityDevelopmentPlannerFargoND
TibLaughlinDirectorofGeneralServicesKansasCityKS
TimMorelandDirectorofPerformanceManagementandOpenDataChattanoogaTN
CharlesRosendahlManagementAnalystDentonTX
Thankyou
60
MelissaSiebenAssistantCountyAdministratorKansasCityKS
GinaThompsonDirectorofStrategicServicesLewisvilleTX
JHUFacultycollaboration
GovExalsoconvenedthecitycontactsandfacultypartnersinaseriesofAdvisoryGroupmeetingsThesemeetingsservedtoconnectcitiesandfacultytounderstandoneanothersperspective
StefanBaralBloombergSchoolofPublicHealth
AshleyBernerJHUSchoolofEducation
MelissaDavey-RothwellBloombergSchoolofPublicHealth
JonLinksBloombergSchoolofPublicHealth
TamarMendelsonBloombergSchoolofPublicHealth
KristinMmariBloombergSchoolofPublicHealth
KeshiaPollackPorterBloombergSchoolofPublicHealth
JoshSharfsteinBloombergSchoolofPublicHealth
LizStuartBloombergSchoolofPublicHealth
LindsayThompsonCareyBusinessSchool
DanielWebsterBloombergSchoolofPublicHealth
Partnersindataandevidence-basedpolicy
Wealsospokewithpartnerswhoareexpertsinthefieldofevidence-basedpolicywhosharedtheirknowledgeandresourcesofcurrentandfuturethinkinginevidence-informedthinking
WendyBarrenoAssociateDirectorWilsonSheehanLabforEconomicOpportunitiesUniversityofNotreDame
TiffanyDavenportDirectorofDataPartnershipsBaltimoresPromise
TomiHiersDirectorBaltimoreCivicSiteAnnieECaseyFoundation
ChrisKingsleySeniorAssociateAnnieECaseyFoundation
KathrynOliverAssociateProfessorLondonSchoolofHygieneandTropicalMedicine
Thankyou
61
KristenWatkinsSeniorPolicyAssociateJ-PAL
GovExParticipants
GovExAnalystMiriamMcKinneyanalyzedtheinterviewtranscriptsandconductedtextanalysisSeniorImplementationAdvisorEmilyShawwillintegrateRoadmapactivitieswithadditionalservicedeliveryactivitiesatGovEx
Thankyou
62
ReferencesAbrahamKGampHaskinsR(2017)Thepromiseofevidence-basedpolicymakingReportofthecommissiononevidence-basedpolicymakingWashingtonDCCommissiononEvidence-basedPolicymakingBigmove2Extendingthegreen(nd)Retrievedfromhttpswwwcityoflewisvillecomdoing-businesslewisville-2025big-move-2-extending-the-greenBatesMAampGlennersterR(2017)ThegeneralizabilitypuzzleRetrievedfromhttpsssirorgarticlesentrythe_generalizability_puzzleBuildwithnotfor(2018January)RetrievedfromhttpsbuildwithnotforwordpresscomCairneyPampOliverK(2017)Evidence-basedpolicymakingisnotlikeevidence-basedmedicinesohowfarshouldyougotobridgethedividebetweenevidenceandpolicyHealthResearchPolicyandSystemsdoi101186s12961-017-0192-xCairneyPOliverKampWellsteadA(2016)TobridgethedividebetweenevidenceandpolicyReduceambiguityasmuchasuncertaintyPublicAdministrationReview76(3)399-402Retrievedfromhttpdxdoiorg101111puar12555Centreforpublicimpact(nd)RetrievedfromhttpswwwcentreforpublicimpactorgCernaL(2013)ThenatureofpolicychangeandimplementationAreviewofdifferenttheoreticalapproachesOrganisationforEconomicCo-OperationandDevelopmentRetrievedfromhttpswwwoecdorgeduceriTheNatureofPolicyChangeandImplementationpdfColoradoSprings(nd)Retrievedfromhttpscoloradospringsgovcommunicationspagewhat-expect-when-you-speakupContinuumofcare(coc)program(nd)RetrievedfromhttpswwwhudexchangeinfoprogramscocCouncilpresidentkicksoff$12millionyouthfund(nd)Retrievedfromhttpwwwbaltimorecitycouncilcomcontentcouncil-president-kicks-12-million-youth-fundDurlakJAampDuPreEP(2008)ImplementationmattersAreviewofresearchontheinfluenceofimplementationonprogramoutcomesandthefactorsaffectingimplementationAmericanJournalofCommunityPsychology41327-350Retrievedfromhttpswwwncbinlmnihgovpubmed18322790Fishbonediagram(ishikawa)(nd)Retrievedfromhttpasqorglearn-about-qualitycause-analysis-toolsoverviewfishbonehtmlFivewhysGettingtotherootofaproblemquickly(nd)RetrievedfromhttpswwwmindtoolscompagesarticlenewTMC_5Whtm
References
63
FligsteinNampMcAdamD(2011)TowardageneraltheoryofstrategicactionfieldsSociologicalTheory29(1)1-26RetrievedfromhttpsociologyberkeleyedusitesdefaultfilesfacultyfligsteinFligstein20McAdam20Soc20Theory20PaperpdHeinkelP(2018January12)CitymullspermitsforbikesharingcompaniesRetrievedfromhttpwwwdentonrccomnewsdenton20180112city-mulls-permits-bike-sharing-companiesHowisareamedianincomecalculated(2012October)Retrievedfromhttpswwwhudexchangeinfofaqs843how-is-area-median-income-calculatedLampkinLMampHatryHP(2003)KeystepsinoutcomemanagementUrbanInstituteRetrievedfromhttpswwwurbanorgresearchpublicationkey-steps-outcome-managementMarshDampSmithM(2000)UnderstandingpolicynetworksTowardsadialecticalapproachPoliticalStudies484-21Retrievedfromhttpjournalssagepubcomdoiabs1011111467-924800247McCannL(nd)RetrievedfromhttplaurenellencomMoultonSampSandfortJR(2016)ThestrategicactionfieldframeworkforpolicyimplementationresearchPolicyStudiesJournal45(1)Retrievedfromhttponlinelibrarywileycomstore101111psj12147assetpsj12147pdfv=1ampt=jdkhhcp4amps=e184c625e1855a6b1615ccb99ee27b1c0b3212d8PaynterB(2018January1)What100citiesarelearningfromeachotherbysharingtheirdataRetrievedfromhttpswwwfastcompanycom40518004what-100-cities-are-learning-from-each-other-by-sharing-their-dataPeakacademy(nd)Retrievedfromhttpswwwdenvergovorgcontentdenvergovenmayors-officeprograms-initiativespeak-performancepeak-academyhtmlPinnockHBarwickMCarpenterCREldridgeSGrandesGGriffithsCJRycroft-MaloneJMeissnerPMurrayEPatelASheikhAampTaylorSJC(2017)Standardsforreportingimplementationstudies(stari)ExplanationandelaborationdocumentBMJOpenRetrievedfromhttpbmjopenbmjcomcontentbmjopen74e013318fullpdfProctorESilmereHRaghavanRHovmandPAaronsGBungerAGriffeyRampHensleyM(2011)OutcomesforimplementationresearchConceptualdistinctionsmeasurementchallengesandresearchagendaAdministrationandPolicyinMentalHealth38(2)65-76doi101007s10488-010-0319-7TandonSD(2018)ProgressincommunityhealthpartnershipsResearcheducationandactionRetrievedfromhttpswwwpressjhuedujournalsprogress-community-health-partnerships-research-education-and-actionProvanKGampKenisP(2007)ModesofnetworkgovernanceStructuremanagementandeffectivenessJournalofPublicAdministrationResearchandTheory
References
64
18(2)229-252Retrievedfromhttpsacademicoupcomjpartarticle182229935895SabatierPA(1988)Anadvocacycoalitionframeworkofpolicychangeandtheroleofpolicy-orientedlearningthereinPolicySciences21(3)129-168Retrievedfromhttpwwwjstororgstable4532139SandstromAampCarlssonL(2008)TheperformanceofpolicynetworksTherelationbetweennetworkstructureandnetworkperformancePolicyStudiesJournal36497-524Retrievedfromhttponlinelibrarywileycomdoi101111j1541-0072200800281xabstractSchwartzMampWilsonE(nd)WhocanaffordtoliveinahomeAlookatdatafromthe2006americancommunitysurveyRetrievedfromhttpswwwcensusgovhousingcensuspublicationswho-can-affordpdfShearerJCAbelsonAKouyateBLavisJNampWaltG(2016)WhydopolicieschangeInstitutionsinterestsideasandnetworksinthreecasesofpolicyreformHealthPolicyandPlanning31(9)1200-1211Retrievedfromhttpsdoiorg101093heapolczw052ThelessonsofadministrativedataHigh-profilepolicy-relevantresearchpoweredbyadministrativedata(nd)Retrievedfromhttpswwwpovertyactionlaborgsitesdefaultfilesresources20161114-The-Lessons-of-Administrative-DatapdfToolsandotherneedsforqualitativeandquantitativemethods(2015February)AgencyforHealthcareResearchandQualityRetrievedfromhttpwwwahrqgovprofessionalsquality-patient-safetytalkingqualityassesstoolshtml
WeaverRCampBagchi-SenS(2013)SpatialanalysisofurbandeclineThegeographyofblightAppliedGeography4061-70
References
65
UniversityPartners
Name Title Affiliation Area(s)ofexpertise
Contactinformation
Examplesofworkingwithnon-academicdecisionmakers
WendyPBarreno
AssociateDirector
WilsonSheehanLabforEconomicOpportunitiesUniversityofNotreDame
LEOsfocusareasincludehealtheducationhousingchildwelfareeconomicopportunitycriminaljustice
wbarrenonduduorleondedu
MelissaDavey-RothwellPhD
AssociateScientist
JohnsHopkinsBloombergSchoolofPublicHealth
Health mdavey1jhuedu410-614-5854
BaltimoreCityHealthDepartmentLeadersMarylandDepartmentofHealthAddictionRecoveryServicesinDCExecutiveDirectorsofcommunityorganizationsacrosstheUSthatfocusonHIVSTIsubstanceabuseandimplementationscience
ElizabethStuart
AssociateDeanforEducationandProfessorDepartmentsofMentalHealthBiostatisticsandHealthPolicyand
JohnsHopkinsBloombergSchoolofPublicHealth
Educationsocialpolicystatisticalmethods
estuartjhuedu
Resources
66
Management
Design-ThinkingResources
EngagementGuideforCivicTech
Journeymap
PerformanceManagementResources
DenverPeakAcademy
Proxymeasuresguide
Talkingaboutperformancewithoutcomplicatingit(video)
SettingPerformanceTargetsguide
Datavisualizationguide
Performancemanagementgettingstartedguide
Exercises
Fishboneexercise
5Whys
AdministrativeData
TheLessonsofAdministrativeData(JPAL)
EvaluationResources
Quantitativeandqualitativemethods(USDepartmentofHealthandHumanServicesAgencyforHealthcareResearchandQuality)
KeyStepsinOutcomeManagement(UrbanInstitute)
RCT-YES
Sageresearchmethods
BestPracticesforMixedMethodsResearchintheHealthSciences
ACCORDChecklist
Pilotandimplementationplanning
Benchmarkingexamplesandproxymeasures
Seattleopendataplaybook
Resources
67
Datasciencescopingguide
Resources
68
AboutGovExGovExhelpsgovernmentsusedataandevidencetoimproveresidentsrsquolivesWesupportgovernmentsinbuildingtheirinternalcapacitytooperatesustainableanalyticspracticeswhichimprovecommunityoutcomesinfieldsfrompublicsafetytoeconomicdevelopmentGovExhasaproventrackrecordofworkinginconcertwithpartnerstoprovidetechnicalassistanceandtrainingatscale
SinceourlaunchinApril2015GovExhas
Trainedover2000governmentemployeesinpersonandonlineintopicsincludingdatamanagementperformancemanagementopendatacommunityengagementandanalytics
Providedtechnicalassistancetomorethan100localgovernmentsacrosstheglobeimprovingtheiruseofdatatoachieveefficienciesinservicedeliveryandbetteroutcomesforresidents
Assessedthecapacityofmorethan130governmentalorganizationstodeterminehowleadersstaffandexternalstakeholdersmanageandusedata
Localgovernmentsareincreasinglyonthefrontlinesofaddressinglargescaleproblemswithglobalimplicationssuchasclimatechangepublicsafetyandeconomicinequality
Whileevidence-basedinterventionstohelpsolvethoseproblemsareavailablelocalgovernmentstaffhavelimitedtimeandcapacitytofindandimplementresearch-basedsolutionsAsaresultcitiescontinuetoworkthewaytheyalwayshave--makingincrementalimprovementstoprogramsbutnotalwayscreatingchangeatasystem-level
ThroughresourcedevelopmentandcollaborationwithJHUfacultyGovExissupportingcityleadersinusingdataandevidencetomakedecisionsandcreatinglastingrelationshipswithresearchers
TheRoadmapwasdevelopedbyGovExaspartofitspartnershipwithBloombergPhilanthropiesWhatWorksCitiesinitiative
AboutGovEx
69
- Title page
- Introduction and purpose of the Roadmap
- Using the Roadmap
-
- Gather information about the problem
- Strategically engage key partners
- Review the evidence and collect details
- Decide on a policy to implement based on local factors
- Establish a framework to monitor progress
- Assess data accessibility quality and security
- Design implementation and evaluation
- Share your story and success
-
- Working with a range of evidence
- Thank you
- References
- Resources
- About GovEx
-
EstablishaframeworktomonitorprogressNowthatyouhavereviewedtheevidenceconsideredlocalcontextandchosenyourpolicyitistimetodeterminehowyourcitywillmonitorprogressofthisnewapproachCitystaffandleadersneedtoestablishtheinfrastructureforongoingcontinuousdecisionmakingbasedonwhetherthepolicychangeisleadingtothedesiredoutcomesforresidentsandwhetherthechangeismeaningfulandrationaltofrontlineworkersSettingarbitrarygoalsatthelocallevelthatconflictwithgoalsatotherlevelsisconfusingtotheindividualsresponsibleforachievingthegoalsGoalsthataresettoohightoachievecanbedemoralizingtostaff
ThissectionincludesstepstosetupavenueinwhichCitystaffandleadershiproutinelyvisitkeyindicatorstodeterminewhatisorisnotworkingandtomakedecisionsinreal-timeFormorereadingonperformancemanagementpleasevisittheGovExPerformanceManagementGettingStartedGuide
InperformancemanagementtermssuchasindicatormeasureandmetricmaybeusedinterchangeablyInthisguidewewilluseindicatorInyourworkthemostimportantthingistobeconsistentinwhichevertermyouuseThisvideomaybehelpfulinunderstandingthisconceptandcommunicatingittoothers
Establishthefundamentalsandgatheryourteam
FirstdeterminewhetherthereisanestablishedvenueorregularmeetingscheduleduringwhichyoucandiscussprogressIfyourcityhasanexistingperformancemanagementstructureonarelatedtopicaddingadiscussionoftheprogressofpolicychangetoyouragendacanbeagreatideaIfthiswillbeastandalonemeetingdecidehowfrequentlytheprojectteamanddecisionmakerswillmeetandputthosemeetingsonthecalendarfortheentirelengthoftheproject
DeterminewhowillstaffandattendroutinemeetingsonCityprogressThesemeetingsmayincludetheMayorandorCityManagerdepartmentheadstaffwhoareimplementingthepolicychangeandanalysts
Identifyindicators
ItiscriticalthatindicatorsbeanimportantcityandcommunitypriorityaswellassomethingtheCityplaysamajorroleinaddressingTheremaybeinstancesinwhichtheCitywillneedtoreachouttoorcollaboratewithexternalpartnerstogetamorecompletepictureofprogressontheoverallpolicychange
Establishaframeworktomonitorprogress
33
Anoutcomeindicatordescribesthedesiredeffectofthetargetedpopulation
ExampleBellevuesfamilyfriendlyneighborhoodsindicatormeasuresthepercentofresidentswhobelievetheyliveinaneighborhoodthatsupportsfamiliesparticularlythosewithchildren
WithyourkeypartnersrevisityourproblemstatementcurrentstateandfuturestateexerciseandbegintoidentifytheoutcomeindicatorsthatwillillustratesuccessFornowchooseasingletop-levelkeyperformanceindicator(KPI)ItiscriticalthattheKPI-thepieceyouwillmeasure-beapriorityforthecommunityandsomethingtheCityplaysamajorroleinaddressing
Outputindicatorsmeasuretheprocessofprogramorservicedelivery
ExampleSanDiegoMayorKevinFaulconersgoaltorepair1000milesofstreetby2020
ReviewingtheadministrativedatasetinventorybegintoparedowntherelevantprogramsanddatasetsthatsupportyourKPIWhatarethemeasuresofeffectivenessforeachprogramTheseareyouroutputindicators
Thediagrambelowillustratestherelationshipbetweenoutputandoutcomeindicators
ThinkaboutimplementationindicatorsasmeasuringstepsintheprogramsrolloutthatarenecessarytoobtaindesiredchangesImplementationoutcomescanfocusontheinterventionstaffdeliveringtheinterventionanorganizationorsystemandcanaddressfeasibilityadaptabilitysustainabilityandcostTwokeyquestionstoconsiderwhendevelopingimplementationindicatorsare
1 Istheinterventionacceptableandappropriatetothecommunityand
2 IsitfeasibletoimplementinterventionXatthistimewiththeseresources
Establishaframeworktomonitorprogress
34
Amoreextensivelistwithdefinitionsofpotentialimplementationoutcomesislinkedhere
UsethetablebelowtochoosethedesiredimplementationoutcomesforyourpilotandtoplanyourmeasurementtechniqueThisshouldincludelevelofanalysis(individualproviderorconsumerorganizationorsetting)yourmeasurementtechnique(surveysobservationinterviewfocusgroupsadministrativedataanalysis)andthephasewhenyouwillcollectthisinformation(earlymidlate)Theremaybemorethanonelevelofanalysismeasurementtechniqueandphaseofdatacollection
Implementationoutcome
Definitionofoutcome
Levelofanalysisandmeasurement
Yourmeasurementtechnique
Phaseofpilotwhenyouwillcollectdata
Usethespacebelowtocaptureyourindicatorwishlist
Indicatorwishlist
Outcomes_____________________________________________________________________________
Outputs________________________________________________________________________________
Implementationoutcomes______________________________________________________________
Setapublichigh-levelgoal
Thehigh-levelgoaliscriticalinestablishingdirectiontogalvanizethepeoplewhowillbedoingtheworkWhendiscussingthegoalusetheMayorsbullypulpitLookforkeymilestonesinCitygovernmenttosharethegoalstatementsuchasthebeginningofaMayoraladministrationaStateoftheCityspeechorotherpublicremarks
GoalsshouldbeldquoSMARTrdquo
Specificaspossible(innamingthepopulationwhowillbebetterofftheactionthecitywilltake)
Measurable
Achievable
Establishaframeworktomonitorprogress
35
Results-focused
Time-bound
ThegoalstatementshouldincludeyouroutcomeindicatorlikeChattanoogasgoalldquoWewillendchronicveteranhomelessnessbyDecember2016rdquoAdiagramtohepyoubuildaSMARTgoalaroundyouroutcomeindicatorisbelow
Component Description Yourturn
Specific WhowillbebetteroffWhatactionthecitywilltake
Measurable Howwillyoumeasuresuccess
Achievable Isthegoalattainable
Results-focused Whatistheexpectedresult
Time-bound When
Setrealisticambitioustargetsandtimelinesforeachindicator
ToestablisharealisticandambitioustargetforyourgoalstatementusetheadministrativedatacollectedforyourchosenKPItoreviewactualCityperformanceinthemostrecentyearcomparedwiththethreeorfourmostrecentyearsworthofdataBegintocalculatesummarystatistics(egtotalminimumaveragemedianmaximum)toidentifytrendsCreatecumulativeandmonthlychartstoidentifytrendsandanomaliesBasedonyouranalysisofhistoricaldataignoretheanomaliesandassumetheproductivitytrendswillfollowthesameoverallpatternfrommonth-to-monthFinallyapplyanincreasedproductivityfactortotheperformanceeachmonthtoforecastfutureprogressBasedonthisanalysischooseatargetthatinspiresimprovementwhileaccountingforreality
Manycitiesthataremorematureintheirperformancemanagementpracticesrecognizetheimportanceoftakingacommunity-levelapproachtoimprovingoutcomesforresidentsThesecitiesarecreatinggoaltaxonomiesthatincludemeasuresrelatedtopublicsafetyhousingandevenrecreationCitiesdriveprogressonthesegoalsbyimplementingstrategiesthatengagemultipledepartmentsandcollaboratewithserviceprovidersinthecommunity
EventuallyyouwilldevelopgoalstatementsfortheoutputsinordertocapturetheefforttoachievetheoutcomecontainedinyourgoalstatementYoursetofindicatorsshouldbecomposedofshort-andinterim-termsupportivemeasureswithaccuratetargetsandrelevanttimelinesAnexamplegoaltaxonomyfromtheCityofLasVegassResultsVegasprogramisbelowUsethistaxonomyasamodelforyourownsuiteofindicatorsandgoalstatements
Establishaframeworktomonitorprogress
36
Yourturn
Thetemplatebelowismeanttohelpyoucraftagoalstatementforeachsupportingmeasure
Goalstatement
Departments
Outcome
Activities
KPI
Target
Timeframe
Conductanddiscussrelevantanalysis
ThrougharoutineseriesofmeetingscontinuallymonitorprogressofKPIssupportingindicatorsandanyotherdatacollectedsuchasqualitativedatatoidentifytrendsthatwillindicatewhetherthechosenpolicyishavingthedesiredimpactThesedatapointsshoulddrivethediscussionduringthesemeetingspointingtoopportunitiesforfurtherinquiryinto
Establishaframeworktomonitorprogress
37
whatisdrivingtheobservedtrendsAnalystsshouldbeencouragedandtrainedtoaskquestionsofthedepartmentsproducingthedataandconductsitevisitswhenpossibletodeepentheirunderstandingofthedataPresentationoftheanalysisshouldfocusononeideaatatimetokeeptheconversationfocusedonkeypointsandspecificopportunitiesforimprovementYoumayconsiderdevelopingapublicfacingdashboardtosharethisprogressmorebroadly
AgeneralcomparisonofthestatusquodatawithwhatwasobservedduringthepilotcanprovidedecisionmakerswithaquickunderstandingofthedifferencesbetweentheoldandnewapproachesalthoughthiswillnotshowcausalityOftenthesecomparisonswillincludeoperationalmetricssuchasdaysordollarssavedbutrememberthattheimpactonpeoplewillbemostimportantIfyouareworkingwithanevaluationpartnertheywillsupportthisstep
Usethespacebelowtosummarizetheresultsofyouranalysis
MeasureTargetvalue(seesectiononsettingtargetsabove)
Pilotvalue(whatdidyouobserveduringthepilot)
Historicalaverage(ifavailable)
Results(WhoisbetteroffIsanyonenegativelyimpacted1-2sentencesreassessingyourgoalstatement)
DocumentyourprocessandprogressincludingfeedbackfromtheparticipantsimplementationpartnersandtheprojectteamDidanythingunexpectedoccurduringthepilotArethereinternalandexternalfactorsthatmayaffectyouroutcomesThisiswheretheimplementationoutcomescanbecanhelpfulIftheinterventiondidnotgoasplannedconsiderwhetheritisbecausetheinterventionwasnotacceptableorfeasible
UsethetemplatebelowcreatedbyDenversPeakAcademytocapturethisinformationIfthe306090daytimelinedoesnotmakesenseforyourcityfeelfreetocustomizeit
CurrentState
FutureState
30Day
60Day
90day
Whatwentwell(+)
Whatdidnotgowellshouldbechanged(-)
IfaquestionordiscussionpointarisesthatcannotbeansweredinthemeetinganassignedstaffmembershouldcapturethequestionduringthemeetingAfterwardthefollowupitemsshouldbesenttorelevantpartnersandthenextmeetingshouldbeginwithadiscussionof
Establishaframeworktomonitorprogress
38
thoseitems
Takeaction
DiscussionsaboutwhetheryourindicatorsaremovingintherightdirectionandwhatmightbedrivingtrendsmayleadtoadiscoverythattherearenotenoughresourcesallocatedtothispolicyorissueTheremaybelimitedfundingforstaffprogrammingortechnologythatisdrivingthedatainthewrongdirectionIfCityleadershipisintheroomthisisagreattimetomakethecaseforadditionalfundingduringthebudgetseasonBycollectinganalyzinganddiscussingdataonthispolicyroutinelyyouarebuildingevidencetomakethecasefortherightleveloffundingtoreachtheestablishedtargets
Establishaframeworktomonitorprogress
39
Assessdataaccessibilityqualityandsecurity
Determineaccessibilityandqualityofdata
OnceyouhavecompiledyourwishlistofindicatorsitistimetodeterminetheaccessibilityandqualityofthedataTopreventcomplicationyoumaywishtoprioritizerelevantindicatorsforwhichyouarealreadycollectingdataAvoidthetemptationofadoptingatechnologysolutiontocollectnewdataatthispointsuchasaphotosurveytoolorotherappFornowyouaretryingtodeterminewhichdatasetsarerelevantaccessibleandhighqualityBesuretoworkwithyourkeyinternalpartnerswhileyouareidentifyingyourmeasuresDepartmentstaffwhoworkonprogramsarefamiliarwiththeindicatorsthatwillaccuratelysupporttheoutcomesanddatabaseadministratorswillbevaluablewhendiscussingaccessibilityandqualityofthedata
Abrieflistofconsiderationsfortheaccessibilityofyourdataisbelow
ArewecurrentlycollectingthisdatasetorwillwebecollectingitforthefirsttimeHowfarbackdoestherecordgo(timehorizon)WouldweneednewtechnologytocollectoranalyzethedataCanthedatabecollectedmanuallyorcanitbeautomatedHowfrequentlyisthedataproduced(continuouslyhourlydailyweeklymonthlyquarterlyyearlyetc)IsthedataavailablefortheentireCitypopulationorasubsetofthepopulationHowgranularisthedata(individuallevelCensusblockneighborhoodcitywideetc)DoesthedatacontainsensitiveinformationIsthereaproxymeasurethatwecanusefornowIsthedatamachinereadable
AsnotedbytheWilsonSheehanLabforEconomicOpportunities(LEO)attheUniversityofNotreDameleveragingadministrativedatacansignificantlyreducethecostsandtimelinesofrigorousevaluationCitiescanestablishaninternaldatasharingagreementacrossdepartmentsincludinglanguagethatenablesadministrativedatatobeusedforresearchpurposesCitiescanalsoidentifyexternaladministrativedata-holderssuchasacountyschooldistrictpolicedepartmenthospitaloranotherstatethatmayhaveoutcomedataimportantformeasuringprogramimpactCitiescanleverageexistingrelationshipsandorworkwithresearchinstitutionstodevelopstreamlineddatasharingprocessesandagreementswiththeseexternalentities
Assessdataaccessibilityqualityandsecurity
40
Evaluatethequalityofyourdata
QualityisasimportantasaccessibilityToevaluatedataqualityyoumaywishtocreatearubricliketheonebelow
Value ValueDescription
3 ExcellentDataQualityThedataiscompleteaccurateandupdatedregularly
2 AdequateDataQualityThedataissomewhatcompleteaccurateandupdatedfairlyregularly
1 PoorDataQualityThedataisnotcompleteaccurateorupdatedregularly
ThisisagreattimetobuildonthedatainventorythatyoustartedinthegatherinformationpageSuggestedadditionalfieldstoaddtoyourdatainventorytemplatearebelow
Program DatasetTitle
DatasetDescription DataSource
Departmentorexternalpartner
DoestheCityalreadycollectthisdata
Whatisthenameoftheprogram
Human-readablenameofthedataset
Shouldbeunderstandablebynon-technicalusers
Whatistheoriginalsystemapplicationfilethathousesit
Departmentorexternalpartnerthatmaintainsthedata
Isthisdataalreadycollected
Updatefrequency-ThefileformatforthedatasetTypicallycsvxlsxshapefiletxtetc
Format-Howoftenthedatasetisupdated
Timerange-Theperiodoftimecoveredinthisparticulardataset
Datause-Howisthisdatasetcommonlyused
Datausers-Whotypicallyusesthisdataset
Dataownerstitle-Titleofthedataowner
Dataownersposition-DataOwnersjobtitle
Dataownersemailaddress-DataOwnersemailaddressWhomanagesthedataandorisresponsibleforgrantingpermissiontoaccessthedataWhounderstandswhatthedatasetincludesandcananswerquestionsaboutit
Dataqualityconcerns-Describetheconcernsyoumayhaveaboutthequalityofthedataset
Assessdataaccessibilityqualityandsecurity
41
Division-DivisionwherethedataismaintainedIfmorethanonedivisionisresponsiblelisttheprimaryownerofthedataset
Sensitivedatacomments-DescribetheissuesthatwouldpreventthecityfromsharingthedatapubliclyCommonsensitivityconcernsincludeprivacyviolationssecurityissuesorhighcostorstaffdemandstopostthedata
Publishingstatus-Thistheplacetoputanycommentsregardingthecurrentstatusofthedataset
Publishedlink-Thisisthelinktothepublisheddatasetontheopendataportalorcitywebsite
Reviewdatasecurityandprivacyprotectionprotocols
OftenapolicychangealsoinvolvesachangeinthedatathatisbeingcollectedhowthatdataisbeinghandledwhohasaccesstoitandahostofotheradjustmentsAsyoutakestockofthedataassetsthatareavailableforyourinitiativeitisvitaltoalsothinkthroughwhatwillbedonetoprotectthatdataandtheprivacyofthepeopleitrepresentsAbigpartofgooddatasecurityinvolvesstrongdatagovernancepracticesthatdefineandidentifysensitivedataandconsciouslyworktobalancethevalueandriskofstoringsuchsensitivedataOnewayofevaluatingyourdatacomesfromcaselawrelatedtotheUSFreedomofInformationActwhichwehavetranslatedtothechecklistbelow
1 Woulddisclosureresultinasubstantialinvasionofprivacy2 Haveweconsideredtheextentorvalueofthepublicinterestandthepurposeorobject
oftheindividualsseekingdisclosure3 Istheinformationavailablefromothersources4 Wastheinformationgivenwithanexpectationofconfidentiality5 Isitpossibletomouldreliefsoastolimittheinvasionofindividualprivacy
TheseriskscouldbemitigatedusingavarietyofstrategiesfromaggregatingdatatoobfuscatingindividualstoredactingfieldsthatmayincludePersonallyIdentifiableInformation(PII)orre-evaluatingyourdatacollectionmethodstofindalternativewaysofgettingtheinformationyouneedThesestepsshouldbecodifiedandmanagedbyadatagovernancecommitteeifpossibleForthesensitivedatathatisdeemedessentialcreateaplanaroundpotentialdatabreachesinordertoquicklyandconfidentlyrespondtoemergencysituations
ProactivelypreparingforthesesituationsisoftenthebeststrategyfordatasecuritybutdonotforgetthetechnicalsideInvolveyourITadministratorsinconversationsaboutdatastorageandaccessensuringthattheywillbeprotectedbyenterprise-gradecybersecuritysystems
Assessdataaccessibilityqualityandsecurity
42
Assessdataaccessibilityqualityandsecurity
43
Designimplementationandevaluation
Designtheimplementation
NowthatyouhaveyouraccountabilityplanandkeyindicatorsitistimetogetspecificaboutyourplanforimplementationThetemplatebelowwillhelpyouthinkthroughthespecificmechanismsofimplementingpolicychange
UnderstandingorganizationsandsystemsiscriticaltounderstandingandmakingdecisionsaboutwhereandhowinterventionsshouldbeappliedApolicyimplementedinasinglesettingmayshowpromisingresultsbutimplementingmultiplepoliciesandprogramsatthesametimewithdifferentactorsresponsibleforeachaspectcanhelpensuresustainablebehaviorchange
InSomervilleMAShapeUpSomerville(SUS)isacitywidepublichealthprogramthatisimplementedacrossmultiplelevelsofinfluenceandsystemsthroughinterventionsbeforeduringandafterschooltoincludechangestobuiltenvironmentsuchasbikelanesregionalmasstransportationandphysicaleducationequipmentinschoolsandgymstoincreasephysicalactivitywellnesspoliciesandschoolgardensnutritioneducationduringandafterschoolandhealthyrestaurantprogramstosupporthealthyfoodandhealthyeatinghabits
Usethetemplatebelowtoplanyourimplementation
Policyimplementationplanningtemplate
ListthespecificdesiredoutcomesInthepreviousstepyouworkedwithdepartmentstafftoestablishspecificmeasurableoutcomesandinterimindicatorsPrioritizetheindicatorsthatwilldemonstrateprogressUsethespacebelowtocapturethedesiredoutcomesofthepolicychange
ExampleShapeupSomervillebeganwiththegoalofpreventingobesityin1stthrough3rdgraders
Yourturn
____________________________________________________________________________
____________________________________________________________________________
Designimplementationandevaluation
44
SpecifythetargetpopulationAreyoudesigninganinterventionthattargetsonlycertainsitesAreyouplanningonimplementingaprogramorinitiativeforaspecifictargetaudienceClearlyspecifythepopulationyouplantotarget
IfyouplantoconductarigorousimpactevaluationasnotedbytheWilsonSheehanLabforEconomicOpportunitiesitisimportanttoestablishconcreteobjectiveeligibilitycriteriaandensurethatthesecriteriaareappliedconsistentlybyallstaffduringscreeningandintakeItisalsovitaltoidentifytheindividualswhoareeligibleforaparticularprogramorserviceevenifeveryoneeligiblecannotbeservedbyexistingresourcesThishelpscitiesidentifyanyunmetneedsandguidefutureresourceallocationwhilealsocreatingthepossibilityofarigorousimpactevaluationbyidentifyingapotentialcomparisongroupFinallywhenconductingscreeningandintakeitisimperativetocollectandmaintainbasicinformation(egnamedateofbirthgender)aboutallprogram-eligibleindividualsnotjustthosewhoparticipateintheprogram
ExampleIntheShapeUpSomervilleexampletheprogrambeganbytargeting1stthrough3rdgraders
Yourturn
____________________________________________________________________________
____________________________________________________________________________
DefinethelocationWherewillthepolicychangetakeplaceWillimplementationtakeplaceatonesiteormultiplesites
ExampleShapeUpSomervilleinterventionsweredeliveredinschools
Yourturn
____________________________________________________________________________
____________________________________________________________________________
ClarifyimplementationpartnersandarticulatetheirrolesWhatarethekeyorganizationsinstitutionsorpeopleresponsibleforimplementationTheprojectteamshouldbeasclearandspecificaspossibleinprovidingadescriptionoftheconcreteactivitiesthatimplementationpartnersaremeanttocarryoutPracticeprofilesmaybeahelpfultooltohelpdefinewhatisrequiredoftheactorswhoareresponsibleforimplementingtheintervention
Designimplementationandevaluation
45
ExampleInSomervillemayoralleadershipandengagementofmultiplecityagencies(publichealthprimarycareschoolsCambridgeHealthAllianceetc)streamlinedandconnectedthevariousShapeUpSomervilleinitiativesandgrantswhichhelpedtokeepthetalentandenergyfocusedonhealthyeatingandactivelivingforallsectorsofSomervilleSUSpartnersultimatelyincludedschoolsnonprofitsCityandstateagenciesandnationalfoundations
Yourturn
____________________________________________________________________________
____________________________________________________________________________
DescribetheinterventionWhatarethestepsactivitiesorprocessesRememberthatthemechanismthespecificstepsorprocessthatleadtobehaviorchangeiskeytounderstandingtheeffectivenessofthepolicychangeandsharingtheresultswithothers
ExampleGoalsofShapeUpSomervilleweretoinfluenceeverypartofanearlyelementaryschoolchildrsquosdayndashbeforeduringandafter-schooltopromotehealthyeatingphysicalactivityandtheirweightstatusInterventionincludedfoodserviceenhancementstoimprovebreakfastandlunchandstaffprofessionaldevelopmentwalktoschoolactivitiesSUSclassroomcurriculumandprofessionaldevelopmentschoolwellnesspolicydevelopmentSUSafter-schoolcurriculumandprofessionaldevelopmentoutreachandeducationtohomethroughmaterialsforumseventsldquoSUSapprovedrdquorestaurantscommunityoutreachandcapacitybuildingthroughpolicydevelopmenttrainingsmediaplacements
Yourturn
____________________________________________________________________________
____________________________________________________________________________
ClarifytimelineandmilestonesfordeliveryandevaluationHowfrequentlywilltheservicesinterventionbedeliveredHowlongwilltheprogramrunWhatisthetimetoexpectedimpacts(eg12monthsafterprogramcompletion)
Designimplementationandevaluation
46
ExampleShapeUpSomervilleledbyTuftsUniversityFriedmanSchoolofNutritionScienceandPolicyconductedaquasi-experimental(nonrandomized)interventionstudyamong1stthrough3rdgradestudentsinSomerville(intervention)andtwocomparisoncommunitiesfor9-monthsduringoneschoolyear(fall2002ndashspring2003)ThestudyshowedpositiveassociationoftheinterventiononchangeinBMISUSreleasedreportssummarizingevaluationsprogresstodateplansforadditionalprojectsandimpactstenandfifteenyearsintotheinitiative
Yourturn
____________________________________________________________________________
____________________________________________________________________________
PlanyouranalysisTheprevioussectionaddressedestablishinganaccountabilityframeworktomonitorprogressandimplementationDeterminewhichindicatorswillbediscussthroughthatprocesswhowillsharethedataandanalysisandhowfeedbackfromthoseconversationswillbeincorporatedbytheprojectteam
ExampleTheMayorrsquosofficeimplementedadata-drivendecisionmakingapproachcalledSomerStatwhichwassynergisticwiththeapproachestakenbytheresearchandevaluationteamsdeployedforthespecificgrants
Yourturn
____________________________________________________________________________
____________________________________________________________________________
PlanforsustainabilityIfthepolicyandprogramchangeissuccessfulhowwilltheprojectteamsustainandexpandtoadditionalsitesorpopulations
ExampleAlthoughmostofthepartnershipsforgedintheearlyyearsweremaintainednewgrantsandcollaborationsexpandedattentiontoolderpublicschoolstudentsactivetransportationandtheGreenLinesubwayextensionthebuiltenvironmentfarmerrsquosmarketeffortssuchasldquoGrowninSomervillerdquowithUnionMainStreeteconomicdevelopmentandhealthdisparitiesaswellasdatacollectionmonitoringandsurveillance
Yourturn
Designimplementationandevaluation
47
____________________________________________________________________________
____________________________________________________________________________
Pilotplanningtemplate
IfyouplantoconductapilotfirstdeterminethepurposeofthepilotDoyouwanttoobservewhetherthepolicychangeresultsinthedesiredbehaviorchangeAreyouconcernedaboutfeasibilityscalabilityorcostDoyouwanttotestwhetheryourdesiredresultscanbeachievedthroughthispolicyapproachClarifythegoalofthepilotandthequestionsyouhopetoanswerThiswillhelpyouidentifythekeyindicatorsthatyouwillusetomeasurewhetherthepilotissuccessfulBeforebeginningestablishabaselineofcurrentperformanceandsetyourtargetsaccordinglybasedonwhatyouhopetoaccomplishthroughthepilot
NextestablishthepilotplanforyourprojectteamincludingmilestonesforteamcheckinstodeterminewhethertomoveaheadorchangecourseThisisseparatefromthemeetingsthatwilloccuraspartoftheaccountabilityframeworkUsethespacebelowtocreateyourtimelineanddevelopyourmeetingschedule
Task Personresponsible Startdate Checkindate Deliverable
Planyourevaluation
WhileyouaredesigningtheimplementationitisalsotimetobegintodevelopyourplanforanalysisAsnotedbytheWilsonSheehanLabforEconomicOpportunitiesdecisionsaboutprogramsetup-includingoutreachscreeningintakedatatrackingandenrollment-candeterminewhetherornotarigorousimpactevaluationwillbepossibleAsdiscussedcityleadersmayfindithelpfultoincludeevaluationpartnersinearlyconversationsaboutprogramsetupDuringpreviousstagesyoumayhaveworkedwitharesearchertoplanyourproposedpolicychangeYoushouldbeclearaboutthepurposeoftheevaluationanddevelopanevaluationplanBelowareadditionalconsiderations
Causationvstrendanalysis
IntheEstablishaframeworktomonitorprogressstagetherewasguidanceontrackingtrendsovertimeItisimportanttonoteherethatmonitoringdataisdifferentfromanalyzingacausaleffectTrendanalysiswillillustratethatchangesareoccurringandmayleadtoquestionsthatcouldbeansweredbyyourevaluationbuttrendanalysisalonedoesnotprovideevidenceaboutwhatisdrivingthosechangesYoumayseechangesinyour
Designimplementationandevaluation
48
outcomeindicatorsbasedonvariablesoutsideoftheCityscontrolForinstanceachangeinfederaltradepolicycanhaveanimpactemploymentratesinyourCityoutsideofyourCitysnewjobtrainingandplacementprogram
ToaddresscausalityyoumustcomparetheoutcomesofindividualswhoreceivedtheinterventionwiththosewhodidnotWithoutacomparisongroupofsimilarindividualswhowerenotexposedtotheinterventionitisimpossibletoknowwhichfactorsareresponsiblefortheoutcomesRandomizedControlTrials(RCTs)inwhichindividualsarerandomlyassignedtotreatmentandcontrolgroupsarethegoldstandardinprovingcausalityIfyoucandemonstratethatoutcomesimprovedforthosewhoreceivedtheinterventionanddidnotimproveatallorimprovedlessforacomparisongroupyoucanbesomewhatcertainthattheinterventionwasthecause
Examplesofothernon-causalimpactevaluationsthatarerigorousandreliableincludecertainquasi-experimentalstudiesusingcomparisongroupdesignssmartinterruptedtimeseriesanalysesthatmodeltrendsovertimeinwaysthatallowestimationofinterventioneffectswelldesignedpre-posttestanalysesneedsassessmentsimplementationstudiesetcFormoreinformationondifferenttypesofevaluationsseetheSageResearchMethodswebsite
Lookfordifferencesingroups
BuildintimetoalsoexaminethepilotdatatoseeifgroupsareexperiencingdifferentresultsReviewingperformancebasedonoutcomemeasuresforapopulationinaggregatemayhidethesubtlewaysthatsomepeoplemaybeaffecteddifferentlysoitisnecessarytocarefullyanalyzethedatafordisparateoutcomesForinstancetheMovingtoOpportunity(MTO)studythatexaminedtheimpactofhousingvouchersonfamilieswholivedinhigh-povertyneighborhoodsfoundthatchildrenwhomovedbeforetheageof13hadbettereducationandhigherincomesasadultsHoweveradultsdidnotseeimprovededucationandemploymentoutcomes
Disaggregatingdataintomorespecificcategoriesofraceandethnicitycanhelpunderstandtrendsanddisparitiesinamoregranularwaywhichcanhelpdecisionmakerstargetpoliciesandinterventionsinamoreequitableway
EarlierintheRoadmapweworkedtoidentifystakeholdersandwhomightbepositivelyandnegativelyimpactedbyachangeinpolicyUsetheresultsofthatexercisetoinvolvethecommunityininterpretingdataanalysisandtogatherqualitativefeedbackfromthecommunitytocallattentiontoissuesForinstanceanincreaseinyouthdetentionmaybeexplainedbyinefficienciesorlackofaccesstotransportation
Designimplementationandevaluation
49
Designimplementationandevaluation
50
Shareyourstoryandsuccess
Sharetheresults
Storytelling
AnintegralstepinyourstrategyissharingtheresultsofyourpilotorpolicychangewithyourcommunityThisincludesdevelopingmessagingaroundtheworkofyourorganizationandcreatingaplantodisseminateyourstoriesWhytellstoriesaboutyourwork
Informandorunderstandasubjectorsituation
Giveneededcontextofdataorbitsofinformation
Relatablestoriesarememorableforreadersviewers
ReadersorviewerswillseetheinformationasmorecompellingwhenthereisademonstratedandrelatablechangeorimpactonthemselvesortheirfriendsneighborsandcolleaguesStoriesareaboutpeopleandgovernmentisfundamentallyabouthelpingpeopleSotellthestoriesofthepeopleyouandyourcitycolleagueshelped
Keepthesetipsinmindasyoudevelopyourstorywithyourteam
WhatwereyourfindingsandwhyshouldanyonecareTellyouraudiencehowtheproblemwassolvedWhathastheimpactbeenonresidentsandcitystafftodateWhatwasthesuccessandweretheremultiplesuccessesalongthewayIfyouhavepreliminaryresultsordatashowingimprovementtosharethisisthetimetodoit
WhowasinvolvedandimpactedReviewthelistofstakeholdersHowwereeachofthemimpactedbytheprogramHumanizeyourstoryandimpactbyaddingadditionalvoicesThesevoicescanbeoneofthefollowing
Apersonorfamilywhowaspositivelyaffectedbythechangesuccess
Aninvolvedthird-partydescribingboththeproblemandthesolution(ieanorganizationnonprofitprovidersbusinesstradeassociationsportsteametc)whoispartofthecommunitythatthegovernmentworkedwithonthechangesuccess
Someonefromthegovernmentitselfwhocanspeakaboutthechangesuccess
TellmoreaboutyourcityDescribeyourcityincludingpopulationdemographicsindustriesetcWhatareitschallengesWhatisitknownforWhatmakesituniqueHowisitsimilarordifferentfromothercitiesintheUS
Shareyourstoryandsuccess
51
DescribeyourcityrsquoschallengeWriteoutthechallengeyourcityisfacing(asdefinedinyourproblemstatement)HowdoesitaffectresidentsHowdoesthisproblemaffecttheworkofcitystaffmembers
AddthedetailsProvidedetailsonyourappliedresearchstrategyTalkaboutwhatyourcityhasdonenthepasttoresolvetheissueDiscusshowyourstrategyhaschangedwhenyoubegantoadoptthisapproachWasmoneysavedorinvested
Usethisstorymapexercisetoorganizeyourthoughtsandguaranteethatyouincludealloftheelementsofatraditionalstory
AfteryoudevelopyourmessagingcreateacommunicationsmarketingandoutreachstrategydesignedtospreadyourmessageYourstrategyshouldwalkyouthroughthefollowingkeyactivities
1 IdentifyandprofileyouraudienceWhencreatinganddisseminatingyourmessageitisalwaysgoodtothinkaboutyouraudienceWhoexactlywouldbeinterestedinwhatyouhavetosayThegeneralpopulationofthecityPeoplewhoworkinnonprofitsBusinessownersItmattersbecausenoteveryonewholivesinacitymaybethetargetoftheinformationBusinessownersmaybemoreinterestedindatashowingeffectsonthebusinessdistrictthanothersandsoonWewillcoverthislaterbutthisstepwillalsohelpyoutothinkofthemosteffectivewaytoreachyouraudience
2 Defineyourcalltoaction(ifyouhaveone)SpendsometimeconsideringwhyyouareinterestedintellingyourstoryDoyousimplywanttoupdatethepubliconyourprogressorcelebrateyoursuccessesAreyouinterestedinyouraudiencechangingitsbehaviororsupportingaspecificinitiativebeinglaunchedbyyourorganizationIfyouwanttohaveyouraudienceactorbehaveinacertainwaybespecificandincorporateitinyourmessaging
Shareyourstoryandsuccess
52
3 SelectperformancemeasuresThinkabouthowyouwoulddetermineifyourcampaignissuccessfulandselectappropriateperformancemeasuresYouwillrevisitthesemeasuresattheendofyourcampaigncommunicationsplanimplementationtodeterminewhetheryouneedtomodifyyourcommunicationsandoutreachstrategyforthefuture
4 MethodstocommunicateyourstoryTherearemanywaystocommunicatesuccessofyourprojectHerearesomesuggestionsonreachingaudiencesItisimportanttodeterminewhichmethodsarethemosteffectiveforyourdesiredaudiencetoreceiveinformationDoyourresearch(PewResearchCenterhasexcellentstudiesonthereachofmediadigitalmediaandmarketing)ForexampleifseniorsareyourtargetpopulationandtheyarelessinclinedtouseInstagram(thatistrue)youmaywanttoconsiderFacebookorthenightlylocalnewsbothofwhichareusedmoreoftenbyseniorsBelowisalistofsomemethodsandtoolsyoucanuse
i Pressreleasestolocalnewspapersandtvstations
ii ShortsmartphonevideosAgreatwaytoengageanaudienceisthroughvideoAskonequestionofsomeonewhoispartofthestoryMaybethereisasocialworkerwhocandiscusshungerinyourcityThesevideoscanbereusedonmultipleplatforms
iii Long-formwriting
iv TellingindividualstoriesthroughaseriesofphotosandshortnarrativesforpublicationonFacebookorInstagram
v Op-edsopinionpiecesplacedinthelocalpaper
vi Emailupdatestoyourlist
vii Connectingwithappropriateorganizationsthatareinvestedinthetopicandworkwiththecommunityandaskthemtodisseminateinformationtotheirmembers
viii EmployingldquoWordofMouthrdquo(WOM)tacticsbycreatingtownhallsorfreeeventsforresidentstoattendintheirownneighborhoods
5 DeterminethetypeofcollateralUsuallycampaignsthattargetresidentsrequireseveralavenuestoreachthemPeoplearenotalwaysinthesameplaceorspaceHavingakeenunderstandingofyouraudiencewillassistyouwithyourchoicesonwhatneedstobedevelopedSometimesthatmeansimagesforsocialmediaaonepageflierthatgivespeopleinformationonattendinganeventoranin-depthbrochure
Shareyourstoryandsuccess
53
6 ImplementyourstrategyKeepinmindstrategiescanbefluidSometimessomethingmaynotturnoutasexpectedevenifduringplanningsessionsitwasldquosupposedtordquoItisokYoucananalyzewhathappenedduringadebriefsessionandonceyoursurveysandothermeasurementshavebeenfinalized
Visualizations
Withtherightvisualizationyoucanreviewandshareinformationinsightsanalysisperformanceandtrendsallofwhichisvaluabletoacross-sectionofthepublicButwhichvisualizationisrightforyouTochoosetherighttypeofchartgraphdeterminethetypeofquestionyouwanttoanswerSomecommoncharttypesarebelowandmoreinformationcanbefoundhere
Exploreadditionalanalyticaltechniques
Duringthegoal-settingphaseyouestablishedoutputoutcomeandimplementationmeasuresBasedontheresultsofyourpilotyoumayneedtoadjustyourmeasuresWiththepilotdatainhandyoumayhavesomeideasofhowfutureanalysescouldbedifferentandpotentiallyimprovethequalityofyourprogramTakesometimetobrainstormotherrelevantdatasetsthatyoudidnothavethetimetopursueearlierincludingmessydepartmentaldatathatneedsmoreattentionorsourcesofpublicdatafromthestateandfederalgovernmentnonprofitsandevenprivatecompaniesthatcouldsupplementyourunderstandingofanissueUsingmachinelearningtechniquesyoumightbeabletobetter
Shareyourstoryandsuccess
54
predictdemandforhomelessservicesusingNOAAweatherforecastspreemptwatermainbreaksusingsoilstratigraphyorinvestineconomicdevelopmentbymonitoringrealestatelistingdata
Advancedanalyticaltechniquescanalsohelpyourimplementationrunmoreeffectivelybyintegratingdata-informedtargetinganddecisiontoolsintotheprocessByautomatingroutineworkflowsandsurfacingsubtletrendsthesetoolscanhelpyouiterateinamoreprecisemannerandprovidekeystrategicguidancetofront-lineemployeesthatwouldotherwisebeunavailableThesesortsofdata-science-enhancedprocessescanbeclassifiedasoneofsixprojecttypesasexploredfurtherbytheNewOrleansLANOLAlyticsteamFormoreguidanceonscopingasimilarprojecttheGovExadvancedanalyticsscopingguidegivesyouastep-by-stepoverviewofhowtobesuccessful
Usethespacebelowtolistyourideasforpossibleadvancedanalyticsprojectsbytype
Typeofproject Ideas
Findingtheneedleinthehaystack
Prioritizingforimpact
Earlywarningtools
Betterquickerdecisions
Optimizingresourceallocation
Experimentingforwhatworks
Keepthefeedbackloopgoing
GatherinformationonpublicsentimentaboutpolicychangethroughfocusgroupssurveysandpublicmeetingsledbyadvisorygroupmembersNotewhethernewinformationisavailablethatcanhelpanalyzeorcontextualizeyourprogress(egfederalorstatedatareports)Sharetheresultsofyourpilotandthepolicychangeprocesswithothercitiesandresearchers
Shareyourstoryandsuccess
55
WorkingwitharangeofevidencePresentlythereisalotofattentionandinvestmentinevidence-basedpolicymakingincludingthefederalbipartisanCommissiononEvidence-BasedPolicymakingthePewCharitableTrustsreportsandtoolkitsonHowStatesEngageinEvidence-BasedPolicymakingandtheLauraandJohnArnoldFoundationsupportedEvidence-BasedPolicymakingCollaborativeManyoftheseinitiativesfocusonrigorousevaluation--meaningrandomizedcontrolledtrials(RCT)orotherexperimentalmodels--asthebestkindofevidencetouseHoweverthisbodyofevidenceisofteninsufficientforthespecifickindsofquestionsthatgovernmentscanhavewhenmakingadecisionaboutpolicyInthisguideweproposeanapproachtosupportgovernmentsinincorporatingadditionaltypesofevidenceintotheirdecisionmaking
OnewaytounderstandarangeofevidenceistouseNestarsquosStandardsofEvidencetoframeanddefineevidenceTheBritishnonprofitNestaworkstoinjectpositivechangeintothewayUKgovernmentpoliciesandprogramsfunctionOneoftheirkeyareasofworkhasbeendevelopingWhatWorksCentresorganizationsthatcollectcommissionsynthesizeanddisseminateevidenceaboutinterventioneffectivenessacrossarangeofsocialimpactareasNestadevelopedthefollowingStandardsofEvidencetoprovideausefulwaytodocumenthowdifferentkindsofinformationdemonstratedifferentlevelsofevidence
Workingwitharangeofevidence
56
Theimportanceofpotentiallybiasedinformation
Onechallengepresentedbyanevidenceclassificationthatacknowledgesthevalueofnon-scientificformsofevidenceistofigureouthowtobalanceallofthepiecestogetherwhenyouusethemTounderstandhowtodothiswehavetothinkabouthowtopositionpotentiallybiasedinformationindecisionmaking
WhenwelookatinformationwhichhasnotbeenrigorouslyevaluatedweacceptthatthisinformationmightbebiasedThewholereasonweconductrigorousevaluationistoremovebiasfromourobservationsWeconductstructuredtrialstoproduceresultsthatareconsistentandgeneralizableIfafindingisunbiasedthisaccuracymdashthislackofbiasmdashcanbeprovedbecauseanyindependenttestercanreproduceitwhentheyfollowanidenticalprocess
YoucanlookatpotentiallybiasedinformationasjustinformationthatisearlieroninitspathtorefinementreplicationandverifiedaccuracyApieceofinformationbasedonanyoneindividualrsquosobservationsislikelytobebiasedbecauseitiscomingfromjustonesourceBiasisreducedbygatheringtogetherlotsofindependentsourcesofinformationandfiguringoutitsmostcommonattributes
MaybethebiasedinformationmakesaclaimthatistoolargemdashitsayssomethingistrueunderallconditionswhenitisreallyonlytrueundercertainconditionsorthataneffectofaninterventionisbiggerthanitstrueaverageorthataneffectlastsovertimewhenitdoesnotPerhapschangingtheclaimtobemorespecificmdashcriticallyreviewingtheinformationtoseeifspecificconditionsmustbemettoachievetheoutcomesmdashwouldmaketheclaimmoreaccurate
IfyoutestedyourclaimfounditdidnotholdupinsomecaseschangedittesteditagainuntilyourclaimrepeatedlyheldupthenyoucoulddemonstratethattheclaimwasnolongerbiasedbutaccurateThisprocessoftestingimprovingtestingandvalidatingiswhatwegetfromstructuredscientificstudy
Withindividualpiecesofinformation--crediblehypothesescasestudiesexpertopinion--youcannotassumethatkindofaccuracyAtthesametimetheseearlyobservationsarefoundationaltoanyreplicablescientificobservationsWewillnevergetanywhereifwedonotstartwiththesepiecesofinformationeveniftheymightbebiased
Combininglevelsofevidencetogetthemostdetailedinformationforyourcity
ItisespeciallyimportanttoacceptdifferentformsofevidencewhenyoufacequestionsthathavenotyetbeentestedbyindependentstudiesSomeofthemostimportantquestionsrelatetothingsthatareveryspecifictoyoursituationYoufaceaparticularcombinationof
Workingwitharangeofevidence
57
issuesactorstrendsandresourcesthatisultimatelydifferentfromeveryotherplaceintheworld
ThisisimportantbecauseaspolicypractitionersandnotscientistsyoucaremoreaboutyourspecificcasethanahypotheticalaveragegeneralcaseYoucarelessaboutthetotallikelyrangeofimpactsfromapolicythanhowthissingleimplementationwillworkYouwantinformationaboutpoliciesthatwillprovideyouwiththebestpossiblepredictionabouthowtheywillworkinyourcontext
ByputtingdifferentkindsofevidenceintothesameframeworkNestarsquosStandardsofEvidenceimplythatwhenyoucannotgetrigorouslytestedexperimentallyverifiedinformationitisappropriatetoworkwithwhatwedohaveThisisrelevantforthemunicipalimplementationcontextbecausenostudypulledfromaliteraturereviewwillaccountforallofthevariablesyoufaceinyourowncontext
Intheabsenceofhavingastudythatpredictshowaninterventionworksinyourspecificcontextyouwillneednon-scientificinformationtotrytofillinthosegapsYouwillneedtogatherstoriesfromresidentsstaffmembersandotherstakeholdersthatsitonLevels1or2ofthestandardsofevidence
Someofthekindsofinformationyouwillprobablygetmoreeasilyfrompeoplethanstudiesinclude
howpeopleincitieslikeyoursreactedtoaninterventioninpublicforumshowpeoplewhoimplementedaninterventioninacitylikeyoursfiguredouthowtogetitpassedhowthecitycommunicatedpubliclyabouttheapparentbenefitsanddrawbacksoftheinterventionthepoliticalacceptabilityoftheintervention
Youcanremainsimultaneouslyawarethattheinformationyouaregatheringthroughindividualstoriesisbiased--notgeneralizableinformedbyspecificperspectives--whilealsorecognizingthathavingsomeinformationaboutsomethingimportantisbetterthannoinformationEvidencefromthefirsttwoevidencelevelsareessentialbuildingblocksofknowledgeandperfectlyusefulwhenyoucannotgetdetailedunbiasedknowledge
Youhavetostartsomewhereandyoushouldalwaysstartwithevidenceyouhaveaboutyourveryspecificconditionsthatyouknowareimportantforsuccess
Workingwitharangeofevidence
58
Thankyou
Interviewswithcities
GovExheldsemi-structuredinterviewswithsixstaffmembersinfourcitiestoestablishtheinitialstepsoftheRoadmapWeaskedrespondentstotalkaboutaninitiativeorpolicythattheyhavebeenresponsibleforimplementingintheircityRespondentsfromthreecitiesspokeabouttheireffortstoaddresshomelessnessandrespondentsfromonecityspokeaboutblightAfterrespondentsidentifiedtheproblemtheyspokeabouthowthecityprioritizedthisissuethevariousstepsinvolvedandbarrierspresent
GovExconductedtextanalysisoftheseinterviewswhichrevealedthatpeoplewasthemostcommonlyusedtermduringtheseconversationswhendiscussingthestepstochangeaprogramorpolicyThisisathemethroughoutthisRoadmapThefirstsectionbeginswithidentifyingtherelevantpeopleTheconversationsaroundevidencearefocusedonhumanbehaviorratherthanmethodologyorstrengthofevidence
Thankyou
59
OncewehadadraftoftheRoadmapwefollowedupwithcitiesonanindividualbasistolearnmoreaboutwhatwouldbehelpfulWearegratefultothefollowingcitystafffortheinterviewsandindividualconversations
MichaelBaskinChiefPolicyOfficerChattanoogaTN
MelissaBridgesPerformanceandInnovationCoordinatorLittleRockAR
VictoriaCarreoacutenAdministrativeOfficerLasVegasNV
TyroneGrandstrandCommunityDevelopmentPlannerFargoND
TibLaughlinDirectorofGeneralServicesKansasCityKS
TimMorelandDirectorofPerformanceManagementandOpenDataChattanoogaTN
CharlesRosendahlManagementAnalystDentonTX
Thankyou
60
MelissaSiebenAssistantCountyAdministratorKansasCityKS
GinaThompsonDirectorofStrategicServicesLewisvilleTX
JHUFacultycollaboration
GovExalsoconvenedthecitycontactsandfacultypartnersinaseriesofAdvisoryGroupmeetingsThesemeetingsservedtoconnectcitiesandfacultytounderstandoneanothersperspective
StefanBaralBloombergSchoolofPublicHealth
AshleyBernerJHUSchoolofEducation
MelissaDavey-RothwellBloombergSchoolofPublicHealth
JonLinksBloombergSchoolofPublicHealth
TamarMendelsonBloombergSchoolofPublicHealth
KristinMmariBloombergSchoolofPublicHealth
KeshiaPollackPorterBloombergSchoolofPublicHealth
JoshSharfsteinBloombergSchoolofPublicHealth
LizStuartBloombergSchoolofPublicHealth
LindsayThompsonCareyBusinessSchool
DanielWebsterBloombergSchoolofPublicHealth
Partnersindataandevidence-basedpolicy
Wealsospokewithpartnerswhoareexpertsinthefieldofevidence-basedpolicywhosharedtheirknowledgeandresourcesofcurrentandfuturethinkinginevidence-informedthinking
WendyBarrenoAssociateDirectorWilsonSheehanLabforEconomicOpportunitiesUniversityofNotreDame
TiffanyDavenportDirectorofDataPartnershipsBaltimoresPromise
TomiHiersDirectorBaltimoreCivicSiteAnnieECaseyFoundation
ChrisKingsleySeniorAssociateAnnieECaseyFoundation
KathrynOliverAssociateProfessorLondonSchoolofHygieneandTropicalMedicine
Thankyou
61
KristenWatkinsSeniorPolicyAssociateJ-PAL
GovExParticipants
GovExAnalystMiriamMcKinneyanalyzedtheinterviewtranscriptsandconductedtextanalysisSeniorImplementationAdvisorEmilyShawwillintegrateRoadmapactivitieswithadditionalservicedeliveryactivitiesatGovEx
Thankyou
62
ReferencesAbrahamKGampHaskinsR(2017)Thepromiseofevidence-basedpolicymakingReportofthecommissiononevidence-basedpolicymakingWashingtonDCCommissiononEvidence-basedPolicymakingBigmove2Extendingthegreen(nd)Retrievedfromhttpswwwcityoflewisvillecomdoing-businesslewisville-2025big-move-2-extending-the-greenBatesMAampGlennersterR(2017)ThegeneralizabilitypuzzleRetrievedfromhttpsssirorgarticlesentrythe_generalizability_puzzleBuildwithnotfor(2018January)RetrievedfromhttpsbuildwithnotforwordpresscomCairneyPampOliverK(2017)Evidence-basedpolicymakingisnotlikeevidence-basedmedicinesohowfarshouldyougotobridgethedividebetweenevidenceandpolicyHealthResearchPolicyandSystemsdoi101186s12961-017-0192-xCairneyPOliverKampWellsteadA(2016)TobridgethedividebetweenevidenceandpolicyReduceambiguityasmuchasuncertaintyPublicAdministrationReview76(3)399-402Retrievedfromhttpdxdoiorg101111puar12555Centreforpublicimpact(nd)RetrievedfromhttpswwwcentreforpublicimpactorgCernaL(2013)ThenatureofpolicychangeandimplementationAreviewofdifferenttheoreticalapproachesOrganisationforEconomicCo-OperationandDevelopmentRetrievedfromhttpswwwoecdorgeduceriTheNatureofPolicyChangeandImplementationpdfColoradoSprings(nd)Retrievedfromhttpscoloradospringsgovcommunicationspagewhat-expect-when-you-speakupContinuumofcare(coc)program(nd)RetrievedfromhttpswwwhudexchangeinfoprogramscocCouncilpresidentkicksoff$12millionyouthfund(nd)Retrievedfromhttpwwwbaltimorecitycouncilcomcontentcouncil-president-kicks-12-million-youth-fundDurlakJAampDuPreEP(2008)ImplementationmattersAreviewofresearchontheinfluenceofimplementationonprogramoutcomesandthefactorsaffectingimplementationAmericanJournalofCommunityPsychology41327-350Retrievedfromhttpswwwncbinlmnihgovpubmed18322790Fishbonediagram(ishikawa)(nd)Retrievedfromhttpasqorglearn-about-qualitycause-analysis-toolsoverviewfishbonehtmlFivewhysGettingtotherootofaproblemquickly(nd)RetrievedfromhttpswwwmindtoolscompagesarticlenewTMC_5Whtm
References
63
FligsteinNampMcAdamD(2011)TowardageneraltheoryofstrategicactionfieldsSociologicalTheory29(1)1-26RetrievedfromhttpsociologyberkeleyedusitesdefaultfilesfacultyfligsteinFligstein20McAdam20Soc20Theory20PaperpdHeinkelP(2018January12)CitymullspermitsforbikesharingcompaniesRetrievedfromhttpwwwdentonrccomnewsdenton20180112city-mulls-permits-bike-sharing-companiesHowisareamedianincomecalculated(2012October)Retrievedfromhttpswwwhudexchangeinfofaqs843how-is-area-median-income-calculatedLampkinLMampHatryHP(2003)KeystepsinoutcomemanagementUrbanInstituteRetrievedfromhttpswwwurbanorgresearchpublicationkey-steps-outcome-managementMarshDampSmithM(2000)UnderstandingpolicynetworksTowardsadialecticalapproachPoliticalStudies484-21Retrievedfromhttpjournalssagepubcomdoiabs1011111467-924800247McCannL(nd)RetrievedfromhttplaurenellencomMoultonSampSandfortJR(2016)ThestrategicactionfieldframeworkforpolicyimplementationresearchPolicyStudiesJournal45(1)Retrievedfromhttponlinelibrarywileycomstore101111psj12147assetpsj12147pdfv=1ampt=jdkhhcp4amps=e184c625e1855a6b1615ccb99ee27b1c0b3212d8PaynterB(2018January1)What100citiesarelearningfromeachotherbysharingtheirdataRetrievedfromhttpswwwfastcompanycom40518004what-100-cities-are-learning-from-each-other-by-sharing-their-dataPeakacademy(nd)Retrievedfromhttpswwwdenvergovorgcontentdenvergovenmayors-officeprograms-initiativespeak-performancepeak-academyhtmlPinnockHBarwickMCarpenterCREldridgeSGrandesGGriffithsCJRycroft-MaloneJMeissnerPMurrayEPatelASheikhAampTaylorSJC(2017)Standardsforreportingimplementationstudies(stari)ExplanationandelaborationdocumentBMJOpenRetrievedfromhttpbmjopenbmjcomcontentbmjopen74e013318fullpdfProctorESilmereHRaghavanRHovmandPAaronsGBungerAGriffeyRampHensleyM(2011)OutcomesforimplementationresearchConceptualdistinctionsmeasurementchallengesandresearchagendaAdministrationandPolicyinMentalHealth38(2)65-76doi101007s10488-010-0319-7TandonSD(2018)ProgressincommunityhealthpartnershipsResearcheducationandactionRetrievedfromhttpswwwpressjhuedujournalsprogress-community-health-partnerships-research-education-and-actionProvanKGampKenisP(2007)ModesofnetworkgovernanceStructuremanagementandeffectivenessJournalofPublicAdministrationResearchandTheory
References
64
18(2)229-252Retrievedfromhttpsacademicoupcomjpartarticle182229935895SabatierPA(1988)Anadvocacycoalitionframeworkofpolicychangeandtheroleofpolicy-orientedlearningthereinPolicySciences21(3)129-168Retrievedfromhttpwwwjstororgstable4532139SandstromAampCarlssonL(2008)TheperformanceofpolicynetworksTherelationbetweennetworkstructureandnetworkperformancePolicyStudiesJournal36497-524Retrievedfromhttponlinelibrarywileycomdoi101111j1541-0072200800281xabstractSchwartzMampWilsonE(nd)WhocanaffordtoliveinahomeAlookatdatafromthe2006americancommunitysurveyRetrievedfromhttpswwwcensusgovhousingcensuspublicationswho-can-affordpdfShearerJCAbelsonAKouyateBLavisJNampWaltG(2016)WhydopolicieschangeInstitutionsinterestsideasandnetworksinthreecasesofpolicyreformHealthPolicyandPlanning31(9)1200-1211Retrievedfromhttpsdoiorg101093heapolczw052ThelessonsofadministrativedataHigh-profilepolicy-relevantresearchpoweredbyadministrativedata(nd)Retrievedfromhttpswwwpovertyactionlaborgsitesdefaultfilesresources20161114-The-Lessons-of-Administrative-DatapdfToolsandotherneedsforqualitativeandquantitativemethods(2015February)AgencyforHealthcareResearchandQualityRetrievedfromhttpwwwahrqgovprofessionalsquality-patient-safetytalkingqualityassesstoolshtml
WeaverRCampBagchi-SenS(2013)SpatialanalysisofurbandeclineThegeographyofblightAppliedGeography4061-70
References
65
UniversityPartners
Name Title Affiliation Area(s)ofexpertise
Contactinformation
Examplesofworkingwithnon-academicdecisionmakers
WendyPBarreno
AssociateDirector
WilsonSheehanLabforEconomicOpportunitiesUniversityofNotreDame
LEOsfocusareasincludehealtheducationhousingchildwelfareeconomicopportunitycriminaljustice
wbarrenonduduorleondedu
MelissaDavey-RothwellPhD
AssociateScientist
JohnsHopkinsBloombergSchoolofPublicHealth
Health mdavey1jhuedu410-614-5854
BaltimoreCityHealthDepartmentLeadersMarylandDepartmentofHealthAddictionRecoveryServicesinDCExecutiveDirectorsofcommunityorganizationsacrosstheUSthatfocusonHIVSTIsubstanceabuseandimplementationscience
ElizabethStuart
AssociateDeanforEducationandProfessorDepartmentsofMentalHealthBiostatisticsandHealthPolicyand
JohnsHopkinsBloombergSchoolofPublicHealth
Educationsocialpolicystatisticalmethods
estuartjhuedu
Resources
66
Management
Design-ThinkingResources
EngagementGuideforCivicTech
Journeymap
PerformanceManagementResources
DenverPeakAcademy
Proxymeasuresguide
Talkingaboutperformancewithoutcomplicatingit(video)
SettingPerformanceTargetsguide
Datavisualizationguide
Performancemanagementgettingstartedguide
Exercises
Fishboneexercise
5Whys
AdministrativeData
TheLessonsofAdministrativeData(JPAL)
EvaluationResources
Quantitativeandqualitativemethods(USDepartmentofHealthandHumanServicesAgencyforHealthcareResearchandQuality)
KeyStepsinOutcomeManagement(UrbanInstitute)
RCT-YES
Sageresearchmethods
BestPracticesforMixedMethodsResearchintheHealthSciences
ACCORDChecklist
Pilotandimplementationplanning
Benchmarkingexamplesandproxymeasures
Seattleopendataplaybook
Resources
67
Datasciencescopingguide
Resources
68
AboutGovExGovExhelpsgovernmentsusedataandevidencetoimproveresidentsrsquolivesWesupportgovernmentsinbuildingtheirinternalcapacitytooperatesustainableanalyticspracticeswhichimprovecommunityoutcomesinfieldsfrompublicsafetytoeconomicdevelopmentGovExhasaproventrackrecordofworkinginconcertwithpartnerstoprovidetechnicalassistanceandtrainingatscale
SinceourlaunchinApril2015GovExhas
Trainedover2000governmentemployeesinpersonandonlineintopicsincludingdatamanagementperformancemanagementopendatacommunityengagementandanalytics
Providedtechnicalassistancetomorethan100localgovernmentsacrosstheglobeimprovingtheiruseofdatatoachieveefficienciesinservicedeliveryandbetteroutcomesforresidents
Assessedthecapacityofmorethan130governmentalorganizationstodeterminehowleadersstaffandexternalstakeholdersmanageandusedata
Localgovernmentsareincreasinglyonthefrontlinesofaddressinglargescaleproblemswithglobalimplicationssuchasclimatechangepublicsafetyandeconomicinequality
Whileevidence-basedinterventionstohelpsolvethoseproblemsareavailablelocalgovernmentstaffhavelimitedtimeandcapacitytofindandimplementresearch-basedsolutionsAsaresultcitiescontinuetoworkthewaytheyalwayshave--makingincrementalimprovementstoprogramsbutnotalwayscreatingchangeatasystem-level
ThroughresourcedevelopmentandcollaborationwithJHUfacultyGovExissupportingcityleadersinusingdataandevidencetomakedecisionsandcreatinglastingrelationshipswithresearchers
TheRoadmapwasdevelopedbyGovExaspartofitspartnershipwithBloombergPhilanthropiesWhatWorksCitiesinitiative
AboutGovEx
69
- Title page
- Introduction and purpose of the Roadmap
- Using the Roadmap
-
- Gather information about the problem
- Strategically engage key partners
- Review the evidence and collect details
- Decide on a policy to implement based on local factors
- Establish a framework to monitor progress
- Assess data accessibility quality and security
- Design implementation and evaluation
- Share your story and success
-
- Working with a range of evidence
- Thank you
- References
- Resources
- About GovEx
-
Anoutcomeindicatordescribesthedesiredeffectofthetargetedpopulation
ExampleBellevuesfamilyfriendlyneighborhoodsindicatormeasuresthepercentofresidentswhobelievetheyliveinaneighborhoodthatsupportsfamiliesparticularlythosewithchildren
WithyourkeypartnersrevisityourproblemstatementcurrentstateandfuturestateexerciseandbegintoidentifytheoutcomeindicatorsthatwillillustratesuccessFornowchooseasingletop-levelkeyperformanceindicator(KPI)ItiscriticalthattheKPI-thepieceyouwillmeasure-beapriorityforthecommunityandsomethingtheCityplaysamajorroleinaddressing
Outputindicatorsmeasuretheprocessofprogramorservicedelivery
ExampleSanDiegoMayorKevinFaulconersgoaltorepair1000milesofstreetby2020
ReviewingtheadministrativedatasetinventorybegintoparedowntherelevantprogramsanddatasetsthatsupportyourKPIWhatarethemeasuresofeffectivenessforeachprogramTheseareyouroutputindicators
Thediagrambelowillustratestherelationshipbetweenoutputandoutcomeindicators
ThinkaboutimplementationindicatorsasmeasuringstepsintheprogramsrolloutthatarenecessarytoobtaindesiredchangesImplementationoutcomescanfocusontheinterventionstaffdeliveringtheinterventionanorganizationorsystemandcanaddressfeasibilityadaptabilitysustainabilityandcostTwokeyquestionstoconsiderwhendevelopingimplementationindicatorsare
1 Istheinterventionacceptableandappropriatetothecommunityand
2 IsitfeasibletoimplementinterventionXatthistimewiththeseresources
Establishaframeworktomonitorprogress
34
Amoreextensivelistwithdefinitionsofpotentialimplementationoutcomesislinkedhere
UsethetablebelowtochoosethedesiredimplementationoutcomesforyourpilotandtoplanyourmeasurementtechniqueThisshouldincludelevelofanalysis(individualproviderorconsumerorganizationorsetting)yourmeasurementtechnique(surveysobservationinterviewfocusgroupsadministrativedataanalysis)andthephasewhenyouwillcollectthisinformation(earlymidlate)Theremaybemorethanonelevelofanalysismeasurementtechniqueandphaseofdatacollection
Implementationoutcome
Definitionofoutcome
Levelofanalysisandmeasurement
Yourmeasurementtechnique
Phaseofpilotwhenyouwillcollectdata
Usethespacebelowtocaptureyourindicatorwishlist
Indicatorwishlist
Outcomes_____________________________________________________________________________
Outputs________________________________________________________________________________
Implementationoutcomes______________________________________________________________
Setapublichigh-levelgoal
Thehigh-levelgoaliscriticalinestablishingdirectiontogalvanizethepeoplewhowillbedoingtheworkWhendiscussingthegoalusetheMayorsbullypulpitLookforkeymilestonesinCitygovernmenttosharethegoalstatementsuchasthebeginningofaMayoraladministrationaStateoftheCityspeechorotherpublicremarks
GoalsshouldbeldquoSMARTrdquo
Specificaspossible(innamingthepopulationwhowillbebetterofftheactionthecitywilltake)
Measurable
Achievable
Establishaframeworktomonitorprogress
35
Results-focused
Time-bound
ThegoalstatementshouldincludeyouroutcomeindicatorlikeChattanoogasgoalldquoWewillendchronicveteranhomelessnessbyDecember2016rdquoAdiagramtohepyoubuildaSMARTgoalaroundyouroutcomeindicatorisbelow
Component Description Yourturn
Specific WhowillbebetteroffWhatactionthecitywilltake
Measurable Howwillyoumeasuresuccess
Achievable Isthegoalattainable
Results-focused Whatistheexpectedresult
Time-bound When
Setrealisticambitioustargetsandtimelinesforeachindicator
ToestablisharealisticandambitioustargetforyourgoalstatementusetheadministrativedatacollectedforyourchosenKPItoreviewactualCityperformanceinthemostrecentyearcomparedwiththethreeorfourmostrecentyearsworthofdataBegintocalculatesummarystatistics(egtotalminimumaveragemedianmaximum)toidentifytrendsCreatecumulativeandmonthlychartstoidentifytrendsandanomaliesBasedonyouranalysisofhistoricaldataignoretheanomaliesandassumetheproductivitytrendswillfollowthesameoverallpatternfrommonth-to-monthFinallyapplyanincreasedproductivityfactortotheperformanceeachmonthtoforecastfutureprogressBasedonthisanalysischooseatargetthatinspiresimprovementwhileaccountingforreality
Manycitiesthataremorematureintheirperformancemanagementpracticesrecognizetheimportanceoftakingacommunity-levelapproachtoimprovingoutcomesforresidentsThesecitiesarecreatinggoaltaxonomiesthatincludemeasuresrelatedtopublicsafetyhousingandevenrecreationCitiesdriveprogressonthesegoalsbyimplementingstrategiesthatengagemultipledepartmentsandcollaboratewithserviceprovidersinthecommunity
EventuallyyouwilldevelopgoalstatementsfortheoutputsinordertocapturetheefforttoachievetheoutcomecontainedinyourgoalstatementYoursetofindicatorsshouldbecomposedofshort-andinterim-termsupportivemeasureswithaccuratetargetsandrelevanttimelinesAnexamplegoaltaxonomyfromtheCityofLasVegassResultsVegasprogramisbelowUsethistaxonomyasamodelforyourownsuiteofindicatorsandgoalstatements
Establishaframeworktomonitorprogress
36
Yourturn
Thetemplatebelowismeanttohelpyoucraftagoalstatementforeachsupportingmeasure
Goalstatement
Departments
Outcome
Activities
KPI
Target
Timeframe
Conductanddiscussrelevantanalysis
ThrougharoutineseriesofmeetingscontinuallymonitorprogressofKPIssupportingindicatorsandanyotherdatacollectedsuchasqualitativedatatoidentifytrendsthatwillindicatewhetherthechosenpolicyishavingthedesiredimpactThesedatapointsshoulddrivethediscussionduringthesemeetingspointingtoopportunitiesforfurtherinquiryinto
Establishaframeworktomonitorprogress
37
whatisdrivingtheobservedtrendsAnalystsshouldbeencouragedandtrainedtoaskquestionsofthedepartmentsproducingthedataandconductsitevisitswhenpossibletodeepentheirunderstandingofthedataPresentationoftheanalysisshouldfocusononeideaatatimetokeeptheconversationfocusedonkeypointsandspecificopportunitiesforimprovementYoumayconsiderdevelopingapublicfacingdashboardtosharethisprogressmorebroadly
AgeneralcomparisonofthestatusquodatawithwhatwasobservedduringthepilotcanprovidedecisionmakerswithaquickunderstandingofthedifferencesbetweentheoldandnewapproachesalthoughthiswillnotshowcausalityOftenthesecomparisonswillincludeoperationalmetricssuchasdaysordollarssavedbutrememberthattheimpactonpeoplewillbemostimportantIfyouareworkingwithanevaluationpartnertheywillsupportthisstep
Usethespacebelowtosummarizetheresultsofyouranalysis
MeasureTargetvalue(seesectiononsettingtargetsabove)
Pilotvalue(whatdidyouobserveduringthepilot)
Historicalaverage(ifavailable)
Results(WhoisbetteroffIsanyonenegativelyimpacted1-2sentencesreassessingyourgoalstatement)
DocumentyourprocessandprogressincludingfeedbackfromtheparticipantsimplementationpartnersandtheprojectteamDidanythingunexpectedoccurduringthepilotArethereinternalandexternalfactorsthatmayaffectyouroutcomesThisiswheretheimplementationoutcomescanbecanhelpfulIftheinterventiondidnotgoasplannedconsiderwhetheritisbecausetheinterventionwasnotacceptableorfeasible
UsethetemplatebelowcreatedbyDenversPeakAcademytocapturethisinformationIfthe306090daytimelinedoesnotmakesenseforyourcityfeelfreetocustomizeit
CurrentState
FutureState
30Day
60Day
90day
Whatwentwell(+)
Whatdidnotgowellshouldbechanged(-)
IfaquestionordiscussionpointarisesthatcannotbeansweredinthemeetinganassignedstaffmembershouldcapturethequestionduringthemeetingAfterwardthefollowupitemsshouldbesenttorelevantpartnersandthenextmeetingshouldbeginwithadiscussionof
Establishaframeworktomonitorprogress
38
thoseitems
Takeaction
DiscussionsaboutwhetheryourindicatorsaremovingintherightdirectionandwhatmightbedrivingtrendsmayleadtoadiscoverythattherearenotenoughresourcesallocatedtothispolicyorissueTheremaybelimitedfundingforstaffprogrammingortechnologythatisdrivingthedatainthewrongdirectionIfCityleadershipisintheroomthisisagreattimetomakethecaseforadditionalfundingduringthebudgetseasonBycollectinganalyzinganddiscussingdataonthispolicyroutinelyyouarebuildingevidencetomakethecasefortherightleveloffundingtoreachtheestablishedtargets
Establishaframeworktomonitorprogress
39
Assessdataaccessibilityqualityandsecurity
Determineaccessibilityandqualityofdata
OnceyouhavecompiledyourwishlistofindicatorsitistimetodeterminetheaccessibilityandqualityofthedataTopreventcomplicationyoumaywishtoprioritizerelevantindicatorsforwhichyouarealreadycollectingdataAvoidthetemptationofadoptingatechnologysolutiontocollectnewdataatthispointsuchasaphotosurveytoolorotherappFornowyouaretryingtodeterminewhichdatasetsarerelevantaccessibleandhighqualityBesuretoworkwithyourkeyinternalpartnerswhileyouareidentifyingyourmeasuresDepartmentstaffwhoworkonprogramsarefamiliarwiththeindicatorsthatwillaccuratelysupporttheoutcomesanddatabaseadministratorswillbevaluablewhendiscussingaccessibilityandqualityofthedata
Abrieflistofconsiderationsfortheaccessibilityofyourdataisbelow
ArewecurrentlycollectingthisdatasetorwillwebecollectingitforthefirsttimeHowfarbackdoestherecordgo(timehorizon)WouldweneednewtechnologytocollectoranalyzethedataCanthedatabecollectedmanuallyorcanitbeautomatedHowfrequentlyisthedataproduced(continuouslyhourlydailyweeklymonthlyquarterlyyearlyetc)IsthedataavailablefortheentireCitypopulationorasubsetofthepopulationHowgranularisthedata(individuallevelCensusblockneighborhoodcitywideetc)DoesthedatacontainsensitiveinformationIsthereaproxymeasurethatwecanusefornowIsthedatamachinereadable
AsnotedbytheWilsonSheehanLabforEconomicOpportunities(LEO)attheUniversityofNotreDameleveragingadministrativedatacansignificantlyreducethecostsandtimelinesofrigorousevaluationCitiescanestablishaninternaldatasharingagreementacrossdepartmentsincludinglanguagethatenablesadministrativedatatobeusedforresearchpurposesCitiescanalsoidentifyexternaladministrativedata-holderssuchasacountyschooldistrictpolicedepartmenthospitaloranotherstatethatmayhaveoutcomedataimportantformeasuringprogramimpactCitiescanleverageexistingrelationshipsandorworkwithresearchinstitutionstodevelopstreamlineddatasharingprocessesandagreementswiththeseexternalentities
Assessdataaccessibilityqualityandsecurity
40
Evaluatethequalityofyourdata
QualityisasimportantasaccessibilityToevaluatedataqualityyoumaywishtocreatearubricliketheonebelow
Value ValueDescription
3 ExcellentDataQualityThedataiscompleteaccurateandupdatedregularly
2 AdequateDataQualityThedataissomewhatcompleteaccurateandupdatedfairlyregularly
1 PoorDataQualityThedataisnotcompleteaccurateorupdatedregularly
ThisisagreattimetobuildonthedatainventorythatyoustartedinthegatherinformationpageSuggestedadditionalfieldstoaddtoyourdatainventorytemplatearebelow
Program DatasetTitle
DatasetDescription DataSource
Departmentorexternalpartner
DoestheCityalreadycollectthisdata
Whatisthenameoftheprogram
Human-readablenameofthedataset
Shouldbeunderstandablebynon-technicalusers
Whatistheoriginalsystemapplicationfilethathousesit
Departmentorexternalpartnerthatmaintainsthedata
Isthisdataalreadycollected
Updatefrequency-ThefileformatforthedatasetTypicallycsvxlsxshapefiletxtetc
Format-Howoftenthedatasetisupdated
Timerange-Theperiodoftimecoveredinthisparticulardataset
Datause-Howisthisdatasetcommonlyused
Datausers-Whotypicallyusesthisdataset
Dataownerstitle-Titleofthedataowner
Dataownersposition-DataOwnersjobtitle
Dataownersemailaddress-DataOwnersemailaddressWhomanagesthedataandorisresponsibleforgrantingpermissiontoaccessthedataWhounderstandswhatthedatasetincludesandcananswerquestionsaboutit
Dataqualityconcerns-Describetheconcernsyoumayhaveaboutthequalityofthedataset
Assessdataaccessibilityqualityandsecurity
41
Division-DivisionwherethedataismaintainedIfmorethanonedivisionisresponsiblelisttheprimaryownerofthedataset
Sensitivedatacomments-DescribetheissuesthatwouldpreventthecityfromsharingthedatapubliclyCommonsensitivityconcernsincludeprivacyviolationssecurityissuesorhighcostorstaffdemandstopostthedata
Publishingstatus-Thistheplacetoputanycommentsregardingthecurrentstatusofthedataset
Publishedlink-Thisisthelinktothepublisheddatasetontheopendataportalorcitywebsite
Reviewdatasecurityandprivacyprotectionprotocols
OftenapolicychangealsoinvolvesachangeinthedatathatisbeingcollectedhowthatdataisbeinghandledwhohasaccesstoitandahostofotheradjustmentsAsyoutakestockofthedataassetsthatareavailableforyourinitiativeitisvitaltoalsothinkthroughwhatwillbedonetoprotectthatdataandtheprivacyofthepeopleitrepresentsAbigpartofgooddatasecurityinvolvesstrongdatagovernancepracticesthatdefineandidentifysensitivedataandconsciouslyworktobalancethevalueandriskofstoringsuchsensitivedataOnewayofevaluatingyourdatacomesfromcaselawrelatedtotheUSFreedomofInformationActwhichwehavetranslatedtothechecklistbelow
1 Woulddisclosureresultinasubstantialinvasionofprivacy2 Haveweconsideredtheextentorvalueofthepublicinterestandthepurposeorobject
oftheindividualsseekingdisclosure3 Istheinformationavailablefromothersources4 Wastheinformationgivenwithanexpectationofconfidentiality5 Isitpossibletomouldreliefsoastolimittheinvasionofindividualprivacy
TheseriskscouldbemitigatedusingavarietyofstrategiesfromaggregatingdatatoobfuscatingindividualstoredactingfieldsthatmayincludePersonallyIdentifiableInformation(PII)orre-evaluatingyourdatacollectionmethodstofindalternativewaysofgettingtheinformationyouneedThesestepsshouldbecodifiedandmanagedbyadatagovernancecommitteeifpossibleForthesensitivedatathatisdeemedessentialcreateaplanaroundpotentialdatabreachesinordertoquicklyandconfidentlyrespondtoemergencysituations
ProactivelypreparingforthesesituationsisoftenthebeststrategyfordatasecuritybutdonotforgetthetechnicalsideInvolveyourITadministratorsinconversationsaboutdatastorageandaccessensuringthattheywillbeprotectedbyenterprise-gradecybersecuritysystems
Assessdataaccessibilityqualityandsecurity
42
Assessdataaccessibilityqualityandsecurity
43
Designimplementationandevaluation
Designtheimplementation
NowthatyouhaveyouraccountabilityplanandkeyindicatorsitistimetogetspecificaboutyourplanforimplementationThetemplatebelowwillhelpyouthinkthroughthespecificmechanismsofimplementingpolicychange
UnderstandingorganizationsandsystemsiscriticaltounderstandingandmakingdecisionsaboutwhereandhowinterventionsshouldbeappliedApolicyimplementedinasinglesettingmayshowpromisingresultsbutimplementingmultiplepoliciesandprogramsatthesametimewithdifferentactorsresponsibleforeachaspectcanhelpensuresustainablebehaviorchange
InSomervilleMAShapeUpSomerville(SUS)isacitywidepublichealthprogramthatisimplementedacrossmultiplelevelsofinfluenceandsystemsthroughinterventionsbeforeduringandafterschooltoincludechangestobuiltenvironmentsuchasbikelanesregionalmasstransportationandphysicaleducationequipmentinschoolsandgymstoincreasephysicalactivitywellnesspoliciesandschoolgardensnutritioneducationduringandafterschoolandhealthyrestaurantprogramstosupporthealthyfoodandhealthyeatinghabits
Usethetemplatebelowtoplanyourimplementation
Policyimplementationplanningtemplate
ListthespecificdesiredoutcomesInthepreviousstepyouworkedwithdepartmentstafftoestablishspecificmeasurableoutcomesandinterimindicatorsPrioritizetheindicatorsthatwilldemonstrateprogressUsethespacebelowtocapturethedesiredoutcomesofthepolicychange
ExampleShapeupSomervillebeganwiththegoalofpreventingobesityin1stthrough3rdgraders
Yourturn
____________________________________________________________________________
____________________________________________________________________________
Designimplementationandevaluation
44
SpecifythetargetpopulationAreyoudesigninganinterventionthattargetsonlycertainsitesAreyouplanningonimplementingaprogramorinitiativeforaspecifictargetaudienceClearlyspecifythepopulationyouplantotarget
IfyouplantoconductarigorousimpactevaluationasnotedbytheWilsonSheehanLabforEconomicOpportunitiesitisimportanttoestablishconcreteobjectiveeligibilitycriteriaandensurethatthesecriteriaareappliedconsistentlybyallstaffduringscreeningandintakeItisalsovitaltoidentifytheindividualswhoareeligibleforaparticularprogramorserviceevenifeveryoneeligiblecannotbeservedbyexistingresourcesThishelpscitiesidentifyanyunmetneedsandguidefutureresourceallocationwhilealsocreatingthepossibilityofarigorousimpactevaluationbyidentifyingapotentialcomparisongroupFinallywhenconductingscreeningandintakeitisimperativetocollectandmaintainbasicinformation(egnamedateofbirthgender)aboutallprogram-eligibleindividualsnotjustthosewhoparticipateintheprogram
ExampleIntheShapeUpSomervilleexampletheprogrambeganbytargeting1stthrough3rdgraders
Yourturn
____________________________________________________________________________
____________________________________________________________________________
DefinethelocationWherewillthepolicychangetakeplaceWillimplementationtakeplaceatonesiteormultiplesites
ExampleShapeUpSomervilleinterventionsweredeliveredinschools
Yourturn
____________________________________________________________________________
____________________________________________________________________________
ClarifyimplementationpartnersandarticulatetheirrolesWhatarethekeyorganizationsinstitutionsorpeopleresponsibleforimplementationTheprojectteamshouldbeasclearandspecificaspossibleinprovidingadescriptionoftheconcreteactivitiesthatimplementationpartnersaremeanttocarryoutPracticeprofilesmaybeahelpfultooltohelpdefinewhatisrequiredoftheactorswhoareresponsibleforimplementingtheintervention
Designimplementationandevaluation
45
ExampleInSomervillemayoralleadershipandengagementofmultiplecityagencies(publichealthprimarycareschoolsCambridgeHealthAllianceetc)streamlinedandconnectedthevariousShapeUpSomervilleinitiativesandgrantswhichhelpedtokeepthetalentandenergyfocusedonhealthyeatingandactivelivingforallsectorsofSomervilleSUSpartnersultimatelyincludedschoolsnonprofitsCityandstateagenciesandnationalfoundations
Yourturn
____________________________________________________________________________
____________________________________________________________________________
DescribetheinterventionWhatarethestepsactivitiesorprocessesRememberthatthemechanismthespecificstepsorprocessthatleadtobehaviorchangeiskeytounderstandingtheeffectivenessofthepolicychangeandsharingtheresultswithothers
ExampleGoalsofShapeUpSomervilleweretoinfluenceeverypartofanearlyelementaryschoolchildrsquosdayndashbeforeduringandafter-schooltopromotehealthyeatingphysicalactivityandtheirweightstatusInterventionincludedfoodserviceenhancementstoimprovebreakfastandlunchandstaffprofessionaldevelopmentwalktoschoolactivitiesSUSclassroomcurriculumandprofessionaldevelopmentschoolwellnesspolicydevelopmentSUSafter-schoolcurriculumandprofessionaldevelopmentoutreachandeducationtohomethroughmaterialsforumseventsldquoSUSapprovedrdquorestaurantscommunityoutreachandcapacitybuildingthroughpolicydevelopmenttrainingsmediaplacements
Yourturn
____________________________________________________________________________
____________________________________________________________________________
ClarifytimelineandmilestonesfordeliveryandevaluationHowfrequentlywilltheservicesinterventionbedeliveredHowlongwilltheprogramrunWhatisthetimetoexpectedimpacts(eg12monthsafterprogramcompletion)
Designimplementationandevaluation
46
ExampleShapeUpSomervilleledbyTuftsUniversityFriedmanSchoolofNutritionScienceandPolicyconductedaquasi-experimental(nonrandomized)interventionstudyamong1stthrough3rdgradestudentsinSomerville(intervention)andtwocomparisoncommunitiesfor9-monthsduringoneschoolyear(fall2002ndashspring2003)ThestudyshowedpositiveassociationoftheinterventiononchangeinBMISUSreleasedreportssummarizingevaluationsprogresstodateplansforadditionalprojectsandimpactstenandfifteenyearsintotheinitiative
Yourturn
____________________________________________________________________________
____________________________________________________________________________
PlanyouranalysisTheprevioussectionaddressedestablishinganaccountabilityframeworktomonitorprogressandimplementationDeterminewhichindicatorswillbediscussthroughthatprocesswhowillsharethedataandanalysisandhowfeedbackfromthoseconversationswillbeincorporatedbytheprojectteam
ExampleTheMayorrsquosofficeimplementedadata-drivendecisionmakingapproachcalledSomerStatwhichwassynergisticwiththeapproachestakenbytheresearchandevaluationteamsdeployedforthespecificgrants
Yourturn
____________________________________________________________________________
____________________________________________________________________________
PlanforsustainabilityIfthepolicyandprogramchangeissuccessfulhowwilltheprojectteamsustainandexpandtoadditionalsitesorpopulations
ExampleAlthoughmostofthepartnershipsforgedintheearlyyearsweremaintainednewgrantsandcollaborationsexpandedattentiontoolderpublicschoolstudentsactivetransportationandtheGreenLinesubwayextensionthebuiltenvironmentfarmerrsquosmarketeffortssuchasldquoGrowninSomervillerdquowithUnionMainStreeteconomicdevelopmentandhealthdisparitiesaswellasdatacollectionmonitoringandsurveillance
Yourturn
Designimplementationandevaluation
47
____________________________________________________________________________
____________________________________________________________________________
Pilotplanningtemplate
IfyouplantoconductapilotfirstdeterminethepurposeofthepilotDoyouwanttoobservewhetherthepolicychangeresultsinthedesiredbehaviorchangeAreyouconcernedaboutfeasibilityscalabilityorcostDoyouwanttotestwhetheryourdesiredresultscanbeachievedthroughthispolicyapproachClarifythegoalofthepilotandthequestionsyouhopetoanswerThiswillhelpyouidentifythekeyindicatorsthatyouwillusetomeasurewhetherthepilotissuccessfulBeforebeginningestablishabaselineofcurrentperformanceandsetyourtargetsaccordinglybasedonwhatyouhopetoaccomplishthroughthepilot
NextestablishthepilotplanforyourprojectteamincludingmilestonesforteamcheckinstodeterminewhethertomoveaheadorchangecourseThisisseparatefromthemeetingsthatwilloccuraspartoftheaccountabilityframeworkUsethespacebelowtocreateyourtimelineanddevelopyourmeetingschedule
Task Personresponsible Startdate Checkindate Deliverable
Planyourevaluation
WhileyouaredesigningtheimplementationitisalsotimetobegintodevelopyourplanforanalysisAsnotedbytheWilsonSheehanLabforEconomicOpportunitiesdecisionsaboutprogramsetup-includingoutreachscreeningintakedatatrackingandenrollment-candeterminewhetherornotarigorousimpactevaluationwillbepossibleAsdiscussedcityleadersmayfindithelpfultoincludeevaluationpartnersinearlyconversationsaboutprogramsetupDuringpreviousstagesyoumayhaveworkedwitharesearchertoplanyourproposedpolicychangeYoushouldbeclearaboutthepurposeoftheevaluationanddevelopanevaluationplanBelowareadditionalconsiderations
Causationvstrendanalysis
IntheEstablishaframeworktomonitorprogressstagetherewasguidanceontrackingtrendsovertimeItisimportanttonoteherethatmonitoringdataisdifferentfromanalyzingacausaleffectTrendanalysiswillillustratethatchangesareoccurringandmayleadtoquestionsthatcouldbeansweredbyyourevaluationbuttrendanalysisalonedoesnotprovideevidenceaboutwhatisdrivingthosechangesYoumayseechangesinyour
Designimplementationandevaluation
48
outcomeindicatorsbasedonvariablesoutsideoftheCityscontrolForinstanceachangeinfederaltradepolicycanhaveanimpactemploymentratesinyourCityoutsideofyourCitysnewjobtrainingandplacementprogram
ToaddresscausalityyoumustcomparetheoutcomesofindividualswhoreceivedtheinterventionwiththosewhodidnotWithoutacomparisongroupofsimilarindividualswhowerenotexposedtotheinterventionitisimpossibletoknowwhichfactorsareresponsiblefortheoutcomesRandomizedControlTrials(RCTs)inwhichindividualsarerandomlyassignedtotreatmentandcontrolgroupsarethegoldstandardinprovingcausalityIfyoucandemonstratethatoutcomesimprovedforthosewhoreceivedtheinterventionanddidnotimproveatallorimprovedlessforacomparisongroupyoucanbesomewhatcertainthattheinterventionwasthecause
Examplesofothernon-causalimpactevaluationsthatarerigorousandreliableincludecertainquasi-experimentalstudiesusingcomparisongroupdesignssmartinterruptedtimeseriesanalysesthatmodeltrendsovertimeinwaysthatallowestimationofinterventioneffectswelldesignedpre-posttestanalysesneedsassessmentsimplementationstudiesetcFormoreinformationondifferenttypesofevaluationsseetheSageResearchMethodswebsite
Lookfordifferencesingroups
BuildintimetoalsoexaminethepilotdatatoseeifgroupsareexperiencingdifferentresultsReviewingperformancebasedonoutcomemeasuresforapopulationinaggregatemayhidethesubtlewaysthatsomepeoplemaybeaffecteddifferentlysoitisnecessarytocarefullyanalyzethedatafordisparateoutcomesForinstancetheMovingtoOpportunity(MTO)studythatexaminedtheimpactofhousingvouchersonfamilieswholivedinhigh-povertyneighborhoodsfoundthatchildrenwhomovedbeforetheageof13hadbettereducationandhigherincomesasadultsHoweveradultsdidnotseeimprovededucationandemploymentoutcomes
Disaggregatingdataintomorespecificcategoriesofraceandethnicitycanhelpunderstandtrendsanddisparitiesinamoregranularwaywhichcanhelpdecisionmakerstargetpoliciesandinterventionsinamoreequitableway
EarlierintheRoadmapweworkedtoidentifystakeholdersandwhomightbepositivelyandnegativelyimpactedbyachangeinpolicyUsetheresultsofthatexercisetoinvolvethecommunityininterpretingdataanalysisandtogatherqualitativefeedbackfromthecommunitytocallattentiontoissuesForinstanceanincreaseinyouthdetentionmaybeexplainedbyinefficienciesorlackofaccesstotransportation
Designimplementationandevaluation
49
Designimplementationandevaluation
50
Shareyourstoryandsuccess
Sharetheresults
Storytelling
AnintegralstepinyourstrategyissharingtheresultsofyourpilotorpolicychangewithyourcommunityThisincludesdevelopingmessagingaroundtheworkofyourorganizationandcreatingaplantodisseminateyourstoriesWhytellstoriesaboutyourwork
Informandorunderstandasubjectorsituation
Giveneededcontextofdataorbitsofinformation
Relatablestoriesarememorableforreadersviewers
ReadersorviewerswillseetheinformationasmorecompellingwhenthereisademonstratedandrelatablechangeorimpactonthemselvesortheirfriendsneighborsandcolleaguesStoriesareaboutpeopleandgovernmentisfundamentallyabouthelpingpeopleSotellthestoriesofthepeopleyouandyourcitycolleagueshelped
Keepthesetipsinmindasyoudevelopyourstorywithyourteam
WhatwereyourfindingsandwhyshouldanyonecareTellyouraudiencehowtheproblemwassolvedWhathastheimpactbeenonresidentsandcitystafftodateWhatwasthesuccessandweretheremultiplesuccessesalongthewayIfyouhavepreliminaryresultsordatashowingimprovementtosharethisisthetimetodoit
WhowasinvolvedandimpactedReviewthelistofstakeholdersHowwereeachofthemimpactedbytheprogramHumanizeyourstoryandimpactbyaddingadditionalvoicesThesevoicescanbeoneofthefollowing
Apersonorfamilywhowaspositivelyaffectedbythechangesuccess
Aninvolvedthird-partydescribingboththeproblemandthesolution(ieanorganizationnonprofitprovidersbusinesstradeassociationsportsteametc)whoispartofthecommunitythatthegovernmentworkedwithonthechangesuccess
Someonefromthegovernmentitselfwhocanspeakaboutthechangesuccess
TellmoreaboutyourcityDescribeyourcityincludingpopulationdemographicsindustriesetcWhatareitschallengesWhatisitknownforWhatmakesituniqueHowisitsimilarordifferentfromothercitiesintheUS
Shareyourstoryandsuccess
51
DescribeyourcityrsquoschallengeWriteoutthechallengeyourcityisfacing(asdefinedinyourproblemstatement)HowdoesitaffectresidentsHowdoesthisproblemaffecttheworkofcitystaffmembers
AddthedetailsProvidedetailsonyourappliedresearchstrategyTalkaboutwhatyourcityhasdonenthepasttoresolvetheissueDiscusshowyourstrategyhaschangedwhenyoubegantoadoptthisapproachWasmoneysavedorinvested
Usethisstorymapexercisetoorganizeyourthoughtsandguaranteethatyouincludealloftheelementsofatraditionalstory
AfteryoudevelopyourmessagingcreateacommunicationsmarketingandoutreachstrategydesignedtospreadyourmessageYourstrategyshouldwalkyouthroughthefollowingkeyactivities
1 IdentifyandprofileyouraudienceWhencreatinganddisseminatingyourmessageitisalwaysgoodtothinkaboutyouraudienceWhoexactlywouldbeinterestedinwhatyouhavetosayThegeneralpopulationofthecityPeoplewhoworkinnonprofitsBusinessownersItmattersbecausenoteveryonewholivesinacitymaybethetargetoftheinformationBusinessownersmaybemoreinterestedindatashowingeffectsonthebusinessdistrictthanothersandsoonWewillcoverthislaterbutthisstepwillalsohelpyoutothinkofthemosteffectivewaytoreachyouraudience
2 Defineyourcalltoaction(ifyouhaveone)SpendsometimeconsideringwhyyouareinterestedintellingyourstoryDoyousimplywanttoupdatethepubliconyourprogressorcelebrateyoursuccessesAreyouinterestedinyouraudiencechangingitsbehaviororsupportingaspecificinitiativebeinglaunchedbyyourorganizationIfyouwanttohaveyouraudienceactorbehaveinacertainwaybespecificandincorporateitinyourmessaging
Shareyourstoryandsuccess
52
3 SelectperformancemeasuresThinkabouthowyouwoulddetermineifyourcampaignissuccessfulandselectappropriateperformancemeasuresYouwillrevisitthesemeasuresattheendofyourcampaigncommunicationsplanimplementationtodeterminewhetheryouneedtomodifyyourcommunicationsandoutreachstrategyforthefuture
4 MethodstocommunicateyourstoryTherearemanywaystocommunicatesuccessofyourprojectHerearesomesuggestionsonreachingaudiencesItisimportanttodeterminewhichmethodsarethemosteffectiveforyourdesiredaudiencetoreceiveinformationDoyourresearch(PewResearchCenterhasexcellentstudiesonthereachofmediadigitalmediaandmarketing)ForexampleifseniorsareyourtargetpopulationandtheyarelessinclinedtouseInstagram(thatistrue)youmaywanttoconsiderFacebookorthenightlylocalnewsbothofwhichareusedmoreoftenbyseniorsBelowisalistofsomemethodsandtoolsyoucanuse
i Pressreleasestolocalnewspapersandtvstations
ii ShortsmartphonevideosAgreatwaytoengageanaudienceisthroughvideoAskonequestionofsomeonewhoispartofthestoryMaybethereisasocialworkerwhocandiscusshungerinyourcityThesevideoscanbereusedonmultipleplatforms
iii Long-formwriting
iv TellingindividualstoriesthroughaseriesofphotosandshortnarrativesforpublicationonFacebookorInstagram
v Op-edsopinionpiecesplacedinthelocalpaper
vi Emailupdatestoyourlist
vii Connectingwithappropriateorganizationsthatareinvestedinthetopicandworkwiththecommunityandaskthemtodisseminateinformationtotheirmembers
viii EmployingldquoWordofMouthrdquo(WOM)tacticsbycreatingtownhallsorfreeeventsforresidentstoattendintheirownneighborhoods
5 DeterminethetypeofcollateralUsuallycampaignsthattargetresidentsrequireseveralavenuestoreachthemPeoplearenotalwaysinthesameplaceorspaceHavingakeenunderstandingofyouraudiencewillassistyouwithyourchoicesonwhatneedstobedevelopedSometimesthatmeansimagesforsocialmediaaonepageflierthatgivespeopleinformationonattendinganeventoranin-depthbrochure
Shareyourstoryandsuccess
53
6 ImplementyourstrategyKeepinmindstrategiescanbefluidSometimessomethingmaynotturnoutasexpectedevenifduringplanningsessionsitwasldquosupposedtordquoItisokYoucananalyzewhathappenedduringadebriefsessionandonceyoursurveysandothermeasurementshavebeenfinalized
Visualizations
Withtherightvisualizationyoucanreviewandshareinformationinsightsanalysisperformanceandtrendsallofwhichisvaluabletoacross-sectionofthepublicButwhichvisualizationisrightforyouTochoosetherighttypeofchartgraphdeterminethetypeofquestionyouwanttoanswerSomecommoncharttypesarebelowandmoreinformationcanbefoundhere
Exploreadditionalanalyticaltechniques
Duringthegoal-settingphaseyouestablishedoutputoutcomeandimplementationmeasuresBasedontheresultsofyourpilotyoumayneedtoadjustyourmeasuresWiththepilotdatainhandyoumayhavesomeideasofhowfutureanalysescouldbedifferentandpotentiallyimprovethequalityofyourprogramTakesometimetobrainstormotherrelevantdatasetsthatyoudidnothavethetimetopursueearlierincludingmessydepartmentaldatathatneedsmoreattentionorsourcesofpublicdatafromthestateandfederalgovernmentnonprofitsandevenprivatecompaniesthatcouldsupplementyourunderstandingofanissueUsingmachinelearningtechniquesyoumightbeabletobetter
Shareyourstoryandsuccess
54
predictdemandforhomelessservicesusingNOAAweatherforecastspreemptwatermainbreaksusingsoilstratigraphyorinvestineconomicdevelopmentbymonitoringrealestatelistingdata
Advancedanalyticaltechniquescanalsohelpyourimplementationrunmoreeffectivelybyintegratingdata-informedtargetinganddecisiontoolsintotheprocessByautomatingroutineworkflowsandsurfacingsubtletrendsthesetoolscanhelpyouiterateinamoreprecisemannerandprovidekeystrategicguidancetofront-lineemployeesthatwouldotherwisebeunavailableThesesortsofdata-science-enhancedprocessescanbeclassifiedasoneofsixprojecttypesasexploredfurtherbytheNewOrleansLANOLAlyticsteamFormoreguidanceonscopingasimilarprojecttheGovExadvancedanalyticsscopingguidegivesyouastep-by-stepoverviewofhowtobesuccessful
Usethespacebelowtolistyourideasforpossibleadvancedanalyticsprojectsbytype
Typeofproject Ideas
Findingtheneedleinthehaystack
Prioritizingforimpact
Earlywarningtools
Betterquickerdecisions
Optimizingresourceallocation
Experimentingforwhatworks
Keepthefeedbackloopgoing
GatherinformationonpublicsentimentaboutpolicychangethroughfocusgroupssurveysandpublicmeetingsledbyadvisorygroupmembersNotewhethernewinformationisavailablethatcanhelpanalyzeorcontextualizeyourprogress(egfederalorstatedatareports)Sharetheresultsofyourpilotandthepolicychangeprocesswithothercitiesandresearchers
Shareyourstoryandsuccess
55
WorkingwitharangeofevidencePresentlythereisalotofattentionandinvestmentinevidence-basedpolicymakingincludingthefederalbipartisanCommissiononEvidence-BasedPolicymakingthePewCharitableTrustsreportsandtoolkitsonHowStatesEngageinEvidence-BasedPolicymakingandtheLauraandJohnArnoldFoundationsupportedEvidence-BasedPolicymakingCollaborativeManyoftheseinitiativesfocusonrigorousevaluation--meaningrandomizedcontrolledtrials(RCT)orotherexperimentalmodels--asthebestkindofevidencetouseHoweverthisbodyofevidenceisofteninsufficientforthespecifickindsofquestionsthatgovernmentscanhavewhenmakingadecisionaboutpolicyInthisguideweproposeanapproachtosupportgovernmentsinincorporatingadditionaltypesofevidenceintotheirdecisionmaking
OnewaytounderstandarangeofevidenceistouseNestarsquosStandardsofEvidencetoframeanddefineevidenceTheBritishnonprofitNestaworkstoinjectpositivechangeintothewayUKgovernmentpoliciesandprogramsfunctionOneoftheirkeyareasofworkhasbeendevelopingWhatWorksCentresorganizationsthatcollectcommissionsynthesizeanddisseminateevidenceaboutinterventioneffectivenessacrossarangeofsocialimpactareasNestadevelopedthefollowingStandardsofEvidencetoprovideausefulwaytodocumenthowdifferentkindsofinformationdemonstratedifferentlevelsofevidence
Workingwitharangeofevidence
56
Theimportanceofpotentiallybiasedinformation
Onechallengepresentedbyanevidenceclassificationthatacknowledgesthevalueofnon-scientificformsofevidenceistofigureouthowtobalanceallofthepiecestogetherwhenyouusethemTounderstandhowtodothiswehavetothinkabouthowtopositionpotentiallybiasedinformationindecisionmaking
WhenwelookatinformationwhichhasnotbeenrigorouslyevaluatedweacceptthatthisinformationmightbebiasedThewholereasonweconductrigorousevaluationistoremovebiasfromourobservationsWeconductstructuredtrialstoproduceresultsthatareconsistentandgeneralizableIfafindingisunbiasedthisaccuracymdashthislackofbiasmdashcanbeprovedbecauseanyindependenttestercanreproduceitwhentheyfollowanidenticalprocess
YoucanlookatpotentiallybiasedinformationasjustinformationthatisearlieroninitspathtorefinementreplicationandverifiedaccuracyApieceofinformationbasedonanyoneindividualrsquosobservationsislikelytobebiasedbecauseitiscomingfromjustonesourceBiasisreducedbygatheringtogetherlotsofindependentsourcesofinformationandfiguringoutitsmostcommonattributes
MaybethebiasedinformationmakesaclaimthatistoolargemdashitsayssomethingistrueunderallconditionswhenitisreallyonlytrueundercertainconditionsorthataneffectofaninterventionisbiggerthanitstrueaverageorthataneffectlastsovertimewhenitdoesnotPerhapschangingtheclaimtobemorespecificmdashcriticallyreviewingtheinformationtoseeifspecificconditionsmustbemettoachievetheoutcomesmdashwouldmaketheclaimmoreaccurate
IfyoutestedyourclaimfounditdidnotholdupinsomecaseschangedittesteditagainuntilyourclaimrepeatedlyheldupthenyoucoulddemonstratethattheclaimwasnolongerbiasedbutaccurateThisprocessoftestingimprovingtestingandvalidatingiswhatwegetfromstructuredscientificstudy
Withindividualpiecesofinformation--crediblehypothesescasestudiesexpertopinion--youcannotassumethatkindofaccuracyAtthesametimetheseearlyobservationsarefoundationaltoanyreplicablescientificobservationsWewillnevergetanywhereifwedonotstartwiththesepiecesofinformationeveniftheymightbebiased
Combininglevelsofevidencetogetthemostdetailedinformationforyourcity
ItisespeciallyimportanttoacceptdifferentformsofevidencewhenyoufacequestionsthathavenotyetbeentestedbyindependentstudiesSomeofthemostimportantquestionsrelatetothingsthatareveryspecifictoyoursituationYoufaceaparticularcombinationof
Workingwitharangeofevidence
57
issuesactorstrendsandresourcesthatisultimatelydifferentfromeveryotherplaceintheworld
ThisisimportantbecauseaspolicypractitionersandnotscientistsyoucaremoreaboutyourspecificcasethanahypotheticalaveragegeneralcaseYoucarelessaboutthetotallikelyrangeofimpactsfromapolicythanhowthissingleimplementationwillworkYouwantinformationaboutpoliciesthatwillprovideyouwiththebestpossiblepredictionabouthowtheywillworkinyourcontext
ByputtingdifferentkindsofevidenceintothesameframeworkNestarsquosStandardsofEvidenceimplythatwhenyoucannotgetrigorouslytestedexperimentallyverifiedinformationitisappropriatetoworkwithwhatwedohaveThisisrelevantforthemunicipalimplementationcontextbecausenostudypulledfromaliteraturereviewwillaccountforallofthevariablesyoufaceinyourowncontext
Intheabsenceofhavingastudythatpredictshowaninterventionworksinyourspecificcontextyouwillneednon-scientificinformationtotrytofillinthosegapsYouwillneedtogatherstoriesfromresidentsstaffmembersandotherstakeholdersthatsitonLevels1or2ofthestandardsofevidence
Someofthekindsofinformationyouwillprobablygetmoreeasilyfrompeoplethanstudiesinclude
howpeopleincitieslikeyoursreactedtoaninterventioninpublicforumshowpeoplewhoimplementedaninterventioninacitylikeyoursfiguredouthowtogetitpassedhowthecitycommunicatedpubliclyabouttheapparentbenefitsanddrawbacksoftheinterventionthepoliticalacceptabilityoftheintervention
Youcanremainsimultaneouslyawarethattheinformationyouaregatheringthroughindividualstoriesisbiased--notgeneralizableinformedbyspecificperspectives--whilealsorecognizingthathavingsomeinformationaboutsomethingimportantisbetterthannoinformationEvidencefromthefirsttwoevidencelevelsareessentialbuildingblocksofknowledgeandperfectlyusefulwhenyoucannotgetdetailedunbiasedknowledge
Youhavetostartsomewhereandyoushouldalwaysstartwithevidenceyouhaveaboutyourveryspecificconditionsthatyouknowareimportantforsuccess
Workingwitharangeofevidence
58
Thankyou
Interviewswithcities
GovExheldsemi-structuredinterviewswithsixstaffmembersinfourcitiestoestablishtheinitialstepsoftheRoadmapWeaskedrespondentstotalkaboutaninitiativeorpolicythattheyhavebeenresponsibleforimplementingintheircityRespondentsfromthreecitiesspokeabouttheireffortstoaddresshomelessnessandrespondentsfromonecityspokeaboutblightAfterrespondentsidentifiedtheproblemtheyspokeabouthowthecityprioritizedthisissuethevariousstepsinvolvedandbarrierspresent
GovExconductedtextanalysisoftheseinterviewswhichrevealedthatpeoplewasthemostcommonlyusedtermduringtheseconversationswhendiscussingthestepstochangeaprogramorpolicyThisisathemethroughoutthisRoadmapThefirstsectionbeginswithidentifyingtherelevantpeopleTheconversationsaroundevidencearefocusedonhumanbehaviorratherthanmethodologyorstrengthofevidence
Thankyou
59
OncewehadadraftoftheRoadmapwefollowedupwithcitiesonanindividualbasistolearnmoreaboutwhatwouldbehelpfulWearegratefultothefollowingcitystafffortheinterviewsandindividualconversations
MichaelBaskinChiefPolicyOfficerChattanoogaTN
MelissaBridgesPerformanceandInnovationCoordinatorLittleRockAR
VictoriaCarreoacutenAdministrativeOfficerLasVegasNV
TyroneGrandstrandCommunityDevelopmentPlannerFargoND
TibLaughlinDirectorofGeneralServicesKansasCityKS
TimMorelandDirectorofPerformanceManagementandOpenDataChattanoogaTN
CharlesRosendahlManagementAnalystDentonTX
Thankyou
60
MelissaSiebenAssistantCountyAdministratorKansasCityKS
GinaThompsonDirectorofStrategicServicesLewisvilleTX
JHUFacultycollaboration
GovExalsoconvenedthecitycontactsandfacultypartnersinaseriesofAdvisoryGroupmeetingsThesemeetingsservedtoconnectcitiesandfacultytounderstandoneanothersperspective
StefanBaralBloombergSchoolofPublicHealth
AshleyBernerJHUSchoolofEducation
MelissaDavey-RothwellBloombergSchoolofPublicHealth
JonLinksBloombergSchoolofPublicHealth
TamarMendelsonBloombergSchoolofPublicHealth
KristinMmariBloombergSchoolofPublicHealth
KeshiaPollackPorterBloombergSchoolofPublicHealth
JoshSharfsteinBloombergSchoolofPublicHealth
LizStuartBloombergSchoolofPublicHealth
LindsayThompsonCareyBusinessSchool
DanielWebsterBloombergSchoolofPublicHealth
Partnersindataandevidence-basedpolicy
Wealsospokewithpartnerswhoareexpertsinthefieldofevidence-basedpolicywhosharedtheirknowledgeandresourcesofcurrentandfuturethinkinginevidence-informedthinking
WendyBarrenoAssociateDirectorWilsonSheehanLabforEconomicOpportunitiesUniversityofNotreDame
TiffanyDavenportDirectorofDataPartnershipsBaltimoresPromise
TomiHiersDirectorBaltimoreCivicSiteAnnieECaseyFoundation
ChrisKingsleySeniorAssociateAnnieECaseyFoundation
KathrynOliverAssociateProfessorLondonSchoolofHygieneandTropicalMedicine
Thankyou
61
KristenWatkinsSeniorPolicyAssociateJ-PAL
GovExParticipants
GovExAnalystMiriamMcKinneyanalyzedtheinterviewtranscriptsandconductedtextanalysisSeniorImplementationAdvisorEmilyShawwillintegrateRoadmapactivitieswithadditionalservicedeliveryactivitiesatGovEx
Thankyou
62
ReferencesAbrahamKGampHaskinsR(2017)Thepromiseofevidence-basedpolicymakingReportofthecommissiononevidence-basedpolicymakingWashingtonDCCommissiononEvidence-basedPolicymakingBigmove2Extendingthegreen(nd)Retrievedfromhttpswwwcityoflewisvillecomdoing-businesslewisville-2025big-move-2-extending-the-greenBatesMAampGlennersterR(2017)ThegeneralizabilitypuzzleRetrievedfromhttpsssirorgarticlesentrythe_generalizability_puzzleBuildwithnotfor(2018January)RetrievedfromhttpsbuildwithnotforwordpresscomCairneyPampOliverK(2017)Evidence-basedpolicymakingisnotlikeevidence-basedmedicinesohowfarshouldyougotobridgethedividebetweenevidenceandpolicyHealthResearchPolicyandSystemsdoi101186s12961-017-0192-xCairneyPOliverKampWellsteadA(2016)TobridgethedividebetweenevidenceandpolicyReduceambiguityasmuchasuncertaintyPublicAdministrationReview76(3)399-402Retrievedfromhttpdxdoiorg101111puar12555Centreforpublicimpact(nd)RetrievedfromhttpswwwcentreforpublicimpactorgCernaL(2013)ThenatureofpolicychangeandimplementationAreviewofdifferenttheoreticalapproachesOrganisationforEconomicCo-OperationandDevelopmentRetrievedfromhttpswwwoecdorgeduceriTheNatureofPolicyChangeandImplementationpdfColoradoSprings(nd)Retrievedfromhttpscoloradospringsgovcommunicationspagewhat-expect-when-you-speakupContinuumofcare(coc)program(nd)RetrievedfromhttpswwwhudexchangeinfoprogramscocCouncilpresidentkicksoff$12millionyouthfund(nd)Retrievedfromhttpwwwbaltimorecitycouncilcomcontentcouncil-president-kicks-12-million-youth-fundDurlakJAampDuPreEP(2008)ImplementationmattersAreviewofresearchontheinfluenceofimplementationonprogramoutcomesandthefactorsaffectingimplementationAmericanJournalofCommunityPsychology41327-350Retrievedfromhttpswwwncbinlmnihgovpubmed18322790Fishbonediagram(ishikawa)(nd)Retrievedfromhttpasqorglearn-about-qualitycause-analysis-toolsoverviewfishbonehtmlFivewhysGettingtotherootofaproblemquickly(nd)RetrievedfromhttpswwwmindtoolscompagesarticlenewTMC_5Whtm
References
63
FligsteinNampMcAdamD(2011)TowardageneraltheoryofstrategicactionfieldsSociologicalTheory29(1)1-26RetrievedfromhttpsociologyberkeleyedusitesdefaultfilesfacultyfligsteinFligstein20McAdam20Soc20Theory20PaperpdHeinkelP(2018January12)CitymullspermitsforbikesharingcompaniesRetrievedfromhttpwwwdentonrccomnewsdenton20180112city-mulls-permits-bike-sharing-companiesHowisareamedianincomecalculated(2012October)Retrievedfromhttpswwwhudexchangeinfofaqs843how-is-area-median-income-calculatedLampkinLMampHatryHP(2003)KeystepsinoutcomemanagementUrbanInstituteRetrievedfromhttpswwwurbanorgresearchpublicationkey-steps-outcome-managementMarshDampSmithM(2000)UnderstandingpolicynetworksTowardsadialecticalapproachPoliticalStudies484-21Retrievedfromhttpjournalssagepubcomdoiabs1011111467-924800247McCannL(nd)RetrievedfromhttplaurenellencomMoultonSampSandfortJR(2016)ThestrategicactionfieldframeworkforpolicyimplementationresearchPolicyStudiesJournal45(1)Retrievedfromhttponlinelibrarywileycomstore101111psj12147assetpsj12147pdfv=1ampt=jdkhhcp4amps=e184c625e1855a6b1615ccb99ee27b1c0b3212d8PaynterB(2018January1)What100citiesarelearningfromeachotherbysharingtheirdataRetrievedfromhttpswwwfastcompanycom40518004what-100-cities-are-learning-from-each-other-by-sharing-their-dataPeakacademy(nd)Retrievedfromhttpswwwdenvergovorgcontentdenvergovenmayors-officeprograms-initiativespeak-performancepeak-academyhtmlPinnockHBarwickMCarpenterCREldridgeSGrandesGGriffithsCJRycroft-MaloneJMeissnerPMurrayEPatelASheikhAampTaylorSJC(2017)Standardsforreportingimplementationstudies(stari)ExplanationandelaborationdocumentBMJOpenRetrievedfromhttpbmjopenbmjcomcontentbmjopen74e013318fullpdfProctorESilmereHRaghavanRHovmandPAaronsGBungerAGriffeyRampHensleyM(2011)OutcomesforimplementationresearchConceptualdistinctionsmeasurementchallengesandresearchagendaAdministrationandPolicyinMentalHealth38(2)65-76doi101007s10488-010-0319-7TandonSD(2018)ProgressincommunityhealthpartnershipsResearcheducationandactionRetrievedfromhttpswwwpressjhuedujournalsprogress-community-health-partnerships-research-education-and-actionProvanKGampKenisP(2007)ModesofnetworkgovernanceStructuremanagementandeffectivenessJournalofPublicAdministrationResearchandTheory
References
64
18(2)229-252Retrievedfromhttpsacademicoupcomjpartarticle182229935895SabatierPA(1988)Anadvocacycoalitionframeworkofpolicychangeandtheroleofpolicy-orientedlearningthereinPolicySciences21(3)129-168Retrievedfromhttpwwwjstororgstable4532139SandstromAampCarlssonL(2008)TheperformanceofpolicynetworksTherelationbetweennetworkstructureandnetworkperformancePolicyStudiesJournal36497-524Retrievedfromhttponlinelibrarywileycomdoi101111j1541-0072200800281xabstractSchwartzMampWilsonE(nd)WhocanaffordtoliveinahomeAlookatdatafromthe2006americancommunitysurveyRetrievedfromhttpswwwcensusgovhousingcensuspublicationswho-can-affordpdfShearerJCAbelsonAKouyateBLavisJNampWaltG(2016)WhydopolicieschangeInstitutionsinterestsideasandnetworksinthreecasesofpolicyreformHealthPolicyandPlanning31(9)1200-1211Retrievedfromhttpsdoiorg101093heapolczw052ThelessonsofadministrativedataHigh-profilepolicy-relevantresearchpoweredbyadministrativedata(nd)Retrievedfromhttpswwwpovertyactionlaborgsitesdefaultfilesresources20161114-The-Lessons-of-Administrative-DatapdfToolsandotherneedsforqualitativeandquantitativemethods(2015February)AgencyforHealthcareResearchandQualityRetrievedfromhttpwwwahrqgovprofessionalsquality-patient-safetytalkingqualityassesstoolshtml
WeaverRCampBagchi-SenS(2013)SpatialanalysisofurbandeclineThegeographyofblightAppliedGeography4061-70
References
65
UniversityPartners
Name Title Affiliation Area(s)ofexpertise
Contactinformation
Examplesofworkingwithnon-academicdecisionmakers
WendyPBarreno
AssociateDirector
WilsonSheehanLabforEconomicOpportunitiesUniversityofNotreDame
LEOsfocusareasincludehealtheducationhousingchildwelfareeconomicopportunitycriminaljustice
wbarrenonduduorleondedu
MelissaDavey-RothwellPhD
AssociateScientist
JohnsHopkinsBloombergSchoolofPublicHealth
Health mdavey1jhuedu410-614-5854
BaltimoreCityHealthDepartmentLeadersMarylandDepartmentofHealthAddictionRecoveryServicesinDCExecutiveDirectorsofcommunityorganizationsacrosstheUSthatfocusonHIVSTIsubstanceabuseandimplementationscience
ElizabethStuart
AssociateDeanforEducationandProfessorDepartmentsofMentalHealthBiostatisticsandHealthPolicyand
JohnsHopkinsBloombergSchoolofPublicHealth
Educationsocialpolicystatisticalmethods
estuartjhuedu
Resources
66
Management
Design-ThinkingResources
EngagementGuideforCivicTech
Journeymap
PerformanceManagementResources
DenverPeakAcademy
Proxymeasuresguide
Talkingaboutperformancewithoutcomplicatingit(video)
SettingPerformanceTargetsguide
Datavisualizationguide
Performancemanagementgettingstartedguide
Exercises
Fishboneexercise
5Whys
AdministrativeData
TheLessonsofAdministrativeData(JPAL)
EvaluationResources
Quantitativeandqualitativemethods(USDepartmentofHealthandHumanServicesAgencyforHealthcareResearchandQuality)
KeyStepsinOutcomeManagement(UrbanInstitute)
RCT-YES
Sageresearchmethods
BestPracticesforMixedMethodsResearchintheHealthSciences
ACCORDChecklist
Pilotandimplementationplanning
Benchmarkingexamplesandproxymeasures
Seattleopendataplaybook
Resources
67
Datasciencescopingguide
Resources
68
AboutGovExGovExhelpsgovernmentsusedataandevidencetoimproveresidentsrsquolivesWesupportgovernmentsinbuildingtheirinternalcapacitytooperatesustainableanalyticspracticeswhichimprovecommunityoutcomesinfieldsfrompublicsafetytoeconomicdevelopmentGovExhasaproventrackrecordofworkinginconcertwithpartnerstoprovidetechnicalassistanceandtrainingatscale
SinceourlaunchinApril2015GovExhas
Trainedover2000governmentemployeesinpersonandonlineintopicsincludingdatamanagementperformancemanagementopendatacommunityengagementandanalytics
Providedtechnicalassistancetomorethan100localgovernmentsacrosstheglobeimprovingtheiruseofdatatoachieveefficienciesinservicedeliveryandbetteroutcomesforresidents
Assessedthecapacityofmorethan130governmentalorganizationstodeterminehowleadersstaffandexternalstakeholdersmanageandusedata
Localgovernmentsareincreasinglyonthefrontlinesofaddressinglargescaleproblemswithglobalimplicationssuchasclimatechangepublicsafetyandeconomicinequality
Whileevidence-basedinterventionstohelpsolvethoseproblemsareavailablelocalgovernmentstaffhavelimitedtimeandcapacitytofindandimplementresearch-basedsolutionsAsaresultcitiescontinuetoworkthewaytheyalwayshave--makingincrementalimprovementstoprogramsbutnotalwayscreatingchangeatasystem-level
ThroughresourcedevelopmentandcollaborationwithJHUfacultyGovExissupportingcityleadersinusingdataandevidencetomakedecisionsandcreatinglastingrelationshipswithresearchers
TheRoadmapwasdevelopedbyGovExaspartofitspartnershipwithBloombergPhilanthropiesWhatWorksCitiesinitiative
AboutGovEx
69
- Title page
- Introduction and purpose of the Roadmap
- Using the Roadmap
-
- Gather information about the problem
- Strategically engage key partners
- Review the evidence and collect details
- Decide on a policy to implement based on local factors
- Establish a framework to monitor progress
- Assess data accessibility quality and security
- Design implementation and evaluation
- Share your story and success
-
- Working with a range of evidence
- Thank you
- References
- Resources
- About GovEx
-
Amoreextensivelistwithdefinitionsofpotentialimplementationoutcomesislinkedhere
UsethetablebelowtochoosethedesiredimplementationoutcomesforyourpilotandtoplanyourmeasurementtechniqueThisshouldincludelevelofanalysis(individualproviderorconsumerorganizationorsetting)yourmeasurementtechnique(surveysobservationinterviewfocusgroupsadministrativedataanalysis)andthephasewhenyouwillcollectthisinformation(earlymidlate)Theremaybemorethanonelevelofanalysismeasurementtechniqueandphaseofdatacollection
Implementationoutcome
Definitionofoutcome
Levelofanalysisandmeasurement
Yourmeasurementtechnique
Phaseofpilotwhenyouwillcollectdata
Usethespacebelowtocaptureyourindicatorwishlist
Indicatorwishlist
Outcomes_____________________________________________________________________________
Outputs________________________________________________________________________________
Implementationoutcomes______________________________________________________________
Setapublichigh-levelgoal
Thehigh-levelgoaliscriticalinestablishingdirectiontogalvanizethepeoplewhowillbedoingtheworkWhendiscussingthegoalusetheMayorsbullypulpitLookforkeymilestonesinCitygovernmenttosharethegoalstatementsuchasthebeginningofaMayoraladministrationaStateoftheCityspeechorotherpublicremarks
GoalsshouldbeldquoSMARTrdquo
Specificaspossible(innamingthepopulationwhowillbebetterofftheactionthecitywilltake)
Measurable
Achievable
Establishaframeworktomonitorprogress
35
Results-focused
Time-bound
ThegoalstatementshouldincludeyouroutcomeindicatorlikeChattanoogasgoalldquoWewillendchronicveteranhomelessnessbyDecember2016rdquoAdiagramtohepyoubuildaSMARTgoalaroundyouroutcomeindicatorisbelow
Component Description Yourturn
Specific WhowillbebetteroffWhatactionthecitywilltake
Measurable Howwillyoumeasuresuccess
Achievable Isthegoalattainable
Results-focused Whatistheexpectedresult
Time-bound When
Setrealisticambitioustargetsandtimelinesforeachindicator
ToestablisharealisticandambitioustargetforyourgoalstatementusetheadministrativedatacollectedforyourchosenKPItoreviewactualCityperformanceinthemostrecentyearcomparedwiththethreeorfourmostrecentyearsworthofdataBegintocalculatesummarystatistics(egtotalminimumaveragemedianmaximum)toidentifytrendsCreatecumulativeandmonthlychartstoidentifytrendsandanomaliesBasedonyouranalysisofhistoricaldataignoretheanomaliesandassumetheproductivitytrendswillfollowthesameoverallpatternfrommonth-to-monthFinallyapplyanincreasedproductivityfactortotheperformanceeachmonthtoforecastfutureprogressBasedonthisanalysischooseatargetthatinspiresimprovementwhileaccountingforreality
Manycitiesthataremorematureintheirperformancemanagementpracticesrecognizetheimportanceoftakingacommunity-levelapproachtoimprovingoutcomesforresidentsThesecitiesarecreatinggoaltaxonomiesthatincludemeasuresrelatedtopublicsafetyhousingandevenrecreationCitiesdriveprogressonthesegoalsbyimplementingstrategiesthatengagemultipledepartmentsandcollaboratewithserviceprovidersinthecommunity
EventuallyyouwilldevelopgoalstatementsfortheoutputsinordertocapturetheefforttoachievetheoutcomecontainedinyourgoalstatementYoursetofindicatorsshouldbecomposedofshort-andinterim-termsupportivemeasureswithaccuratetargetsandrelevanttimelinesAnexamplegoaltaxonomyfromtheCityofLasVegassResultsVegasprogramisbelowUsethistaxonomyasamodelforyourownsuiteofindicatorsandgoalstatements
Establishaframeworktomonitorprogress
36
Yourturn
Thetemplatebelowismeanttohelpyoucraftagoalstatementforeachsupportingmeasure
Goalstatement
Departments
Outcome
Activities
KPI
Target
Timeframe
Conductanddiscussrelevantanalysis
ThrougharoutineseriesofmeetingscontinuallymonitorprogressofKPIssupportingindicatorsandanyotherdatacollectedsuchasqualitativedatatoidentifytrendsthatwillindicatewhetherthechosenpolicyishavingthedesiredimpactThesedatapointsshoulddrivethediscussionduringthesemeetingspointingtoopportunitiesforfurtherinquiryinto
Establishaframeworktomonitorprogress
37
whatisdrivingtheobservedtrendsAnalystsshouldbeencouragedandtrainedtoaskquestionsofthedepartmentsproducingthedataandconductsitevisitswhenpossibletodeepentheirunderstandingofthedataPresentationoftheanalysisshouldfocusononeideaatatimetokeeptheconversationfocusedonkeypointsandspecificopportunitiesforimprovementYoumayconsiderdevelopingapublicfacingdashboardtosharethisprogressmorebroadly
AgeneralcomparisonofthestatusquodatawithwhatwasobservedduringthepilotcanprovidedecisionmakerswithaquickunderstandingofthedifferencesbetweentheoldandnewapproachesalthoughthiswillnotshowcausalityOftenthesecomparisonswillincludeoperationalmetricssuchasdaysordollarssavedbutrememberthattheimpactonpeoplewillbemostimportantIfyouareworkingwithanevaluationpartnertheywillsupportthisstep
Usethespacebelowtosummarizetheresultsofyouranalysis
MeasureTargetvalue(seesectiononsettingtargetsabove)
Pilotvalue(whatdidyouobserveduringthepilot)
Historicalaverage(ifavailable)
Results(WhoisbetteroffIsanyonenegativelyimpacted1-2sentencesreassessingyourgoalstatement)
DocumentyourprocessandprogressincludingfeedbackfromtheparticipantsimplementationpartnersandtheprojectteamDidanythingunexpectedoccurduringthepilotArethereinternalandexternalfactorsthatmayaffectyouroutcomesThisiswheretheimplementationoutcomescanbecanhelpfulIftheinterventiondidnotgoasplannedconsiderwhetheritisbecausetheinterventionwasnotacceptableorfeasible
UsethetemplatebelowcreatedbyDenversPeakAcademytocapturethisinformationIfthe306090daytimelinedoesnotmakesenseforyourcityfeelfreetocustomizeit
CurrentState
FutureState
30Day
60Day
90day
Whatwentwell(+)
Whatdidnotgowellshouldbechanged(-)
IfaquestionordiscussionpointarisesthatcannotbeansweredinthemeetinganassignedstaffmembershouldcapturethequestionduringthemeetingAfterwardthefollowupitemsshouldbesenttorelevantpartnersandthenextmeetingshouldbeginwithadiscussionof
Establishaframeworktomonitorprogress
38
thoseitems
Takeaction
DiscussionsaboutwhetheryourindicatorsaremovingintherightdirectionandwhatmightbedrivingtrendsmayleadtoadiscoverythattherearenotenoughresourcesallocatedtothispolicyorissueTheremaybelimitedfundingforstaffprogrammingortechnologythatisdrivingthedatainthewrongdirectionIfCityleadershipisintheroomthisisagreattimetomakethecaseforadditionalfundingduringthebudgetseasonBycollectinganalyzinganddiscussingdataonthispolicyroutinelyyouarebuildingevidencetomakethecasefortherightleveloffundingtoreachtheestablishedtargets
Establishaframeworktomonitorprogress
39
Assessdataaccessibilityqualityandsecurity
Determineaccessibilityandqualityofdata
OnceyouhavecompiledyourwishlistofindicatorsitistimetodeterminetheaccessibilityandqualityofthedataTopreventcomplicationyoumaywishtoprioritizerelevantindicatorsforwhichyouarealreadycollectingdataAvoidthetemptationofadoptingatechnologysolutiontocollectnewdataatthispointsuchasaphotosurveytoolorotherappFornowyouaretryingtodeterminewhichdatasetsarerelevantaccessibleandhighqualityBesuretoworkwithyourkeyinternalpartnerswhileyouareidentifyingyourmeasuresDepartmentstaffwhoworkonprogramsarefamiliarwiththeindicatorsthatwillaccuratelysupporttheoutcomesanddatabaseadministratorswillbevaluablewhendiscussingaccessibilityandqualityofthedata
Abrieflistofconsiderationsfortheaccessibilityofyourdataisbelow
ArewecurrentlycollectingthisdatasetorwillwebecollectingitforthefirsttimeHowfarbackdoestherecordgo(timehorizon)WouldweneednewtechnologytocollectoranalyzethedataCanthedatabecollectedmanuallyorcanitbeautomatedHowfrequentlyisthedataproduced(continuouslyhourlydailyweeklymonthlyquarterlyyearlyetc)IsthedataavailablefortheentireCitypopulationorasubsetofthepopulationHowgranularisthedata(individuallevelCensusblockneighborhoodcitywideetc)DoesthedatacontainsensitiveinformationIsthereaproxymeasurethatwecanusefornowIsthedatamachinereadable
AsnotedbytheWilsonSheehanLabforEconomicOpportunities(LEO)attheUniversityofNotreDameleveragingadministrativedatacansignificantlyreducethecostsandtimelinesofrigorousevaluationCitiescanestablishaninternaldatasharingagreementacrossdepartmentsincludinglanguagethatenablesadministrativedatatobeusedforresearchpurposesCitiescanalsoidentifyexternaladministrativedata-holderssuchasacountyschooldistrictpolicedepartmenthospitaloranotherstatethatmayhaveoutcomedataimportantformeasuringprogramimpactCitiescanleverageexistingrelationshipsandorworkwithresearchinstitutionstodevelopstreamlineddatasharingprocessesandagreementswiththeseexternalentities
Assessdataaccessibilityqualityandsecurity
40
Evaluatethequalityofyourdata
QualityisasimportantasaccessibilityToevaluatedataqualityyoumaywishtocreatearubricliketheonebelow
Value ValueDescription
3 ExcellentDataQualityThedataiscompleteaccurateandupdatedregularly
2 AdequateDataQualityThedataissomewhatcompleteaccurateandupdatedfairlyregularly
1 PoorDataQualityThedataisnotcompleteaccurateorupdatedregularly
ThisisagreattimetobuildonthedatainventorythatyoustartedinthegatherinformationpageSuggestedadditionalfieldstoaddtoyourdatainventorytemplatearebelow
Program DatasetTitle
DatasetDescription DataSource
Departmentorexternalpartner
DoestheCityalreadycollectthisdata
Whatisthenameoftheprogram
Human-readablenameofthedataset
Shouldbeunderstandablebynon-technicalusers
Whatistheoriginalsystemapplicationfilethathousesit
Departmentorexternalpartnerthatmaintainsthedata
Isthisdataalreadycollected
Updatefrequency-ThefileformatforthedatasetTypicallycsvxlsxshapefiletxtetc
Format-Howoftenthedatasetisupdated
Timerange-Theperiodoftimecoveredinthisparticulardataset
Datause-Howisthisdatasetcommonlyused
Datausers-Whotypicallyusesthisdataset
Dataownerstitle-Titleofthedataowner
Dataownersposition-DataOwnersjobtitle
Dataownersemailaddress-DataOwnersemailaddressWhomanagesthedataandorisresponsibleforgrantingpermissiontoaccessthedataWhounderstandswhatthedatasetincludesandcananswerquestionsaboutit
Dataqualityconcerns-Describetheconcernsyoumayhaveaboutthequalityofthedataset
Assessdataaccessibilityqualityandsecurity
41
Division-DivisionwherethedataismaintainedIfmorethanonedivisionisresponsiblelisttheprimaryownerofthedataset
Sensitivedatacomments-DescribetheissuesthatwouldpreventthecityfromsharingthedatapubliclyCommonsensitivityconcernsincludeprivacyviolationssecurityissuesorhighcostorstaffdemandstopostthedata
Publishingstatus-Thistheplacetoputanycommentsregardingthecurrentstatusofthedataset
Publishedlink-Thisisthelinktothepublisheddatasetontheopendataportalorcitywebsite
Reviewdatasecurityandprivacyprotectionprotocols
OftenapolicychangealsoinvolvesachangeinthedatathatisbeingcollectedhowthatdataisbeinghandledwhohasaccesstoitandahostofotheradjustmentsAsyoutakestockofthedataassetsthatareavailableforyourinitiativeitisvitaltoalsothinkthroughwhatwillbedonetoprotectthatdataandtheprivacyofthepeopleitrepresentsAbigpartofgooddatasecurityinvolvesstrongdatagovernancepracticesthatdefineandidentifysensitivedataandconsciouslyworktobalancethevalueandriskofstoringsuchsensitivedataOnewayofevaluatingyourdatacomesfromcaselawrelatedtotheUSFreedomofInformationActwhichwehavetranslatedtothechecklistbelow
1 Woulddisclosureresultinasubstantialinvasionofprivacy2 Haveweconsideredtheextentorvalueofthepublicinterestandthepurposeorobject
oftheindividualsseekingdisclosure3 Istheinformationavailablefromothersources4 Wastheinformationgivenwithanexpectationofconfidentiality5 Isitpossibletomouldreliefsoastolimittheinvasionofindividualprivacy
TheseriskscouldbemitigatedusingavarietyofstrategiesfromaggregatingdatatoobfuscatingindividualstoredactingfieldsthatmayincludePersonallyIdentifiableInformation(PII)orre-evaluatingyourdatacollectionmethodstofindalternativewaysofgettingtheinformationyouneedThesestepsshouldbecodifiedandmanagedbyadatagovernancecommitteeifpossibleForthesensitivedatathatisdeemedessentialcreateaplanaroundpotentialdatabreachesinordertoquicklyandconfidentlyrespondtoemergencysituations
ProactivelypreparingforthesesituationsisoftenthebeststrategyfordatasecuritybutdonotforgetthetechnicalsideInvolveyourITadministratorsinconversationsaboutdatastorageandaccessensuringthattheywillbeprotectedbyenterprise-gradecybersecuritysystems
Assessdataaccessibilityqualityandsecurity
42
Assessdataaccessibilityqualityandsecurity
43
Designimplementationandevaluation
Designtheimplementation
NowthatyouhaveyouraccountabilityplanandkeyindicatorsitistimetogetspecificaboutyourplanforimplementationThetemplatebelowwillhelpyouthinkthroughthespecificmechanismsofimplementingpolicychange
UnderstandingorganizationsandsystemsiscriticaltounderstandingandmakingdecisionsaboutwhereandhowinterventionsshouldbeappliedApolicyimplementedinasinglesettingmayshowpromisingresultsbutimplementingmultiplepoliciesandprogramsatthesametimewithdifferentactorsresponsibleforeachaspectcanhelpensuresustainablebehaviorchange
InSomervilleMAShapeUpSomerville(SUS)isacitywidepublichealthprogramthatisimplementedacrossmultiplelevelsofinfluenceandsystemsthroughinterventionsbeforeduringandafterschooltoincludechangestobuiltenvironmentsuchasbikelanesregionalmasstransportationandphysicaleducationequipmentinschoolsandgymstoincreasephysicalactivitywellnesspoliciesandschoolgardensnutritioneducationduringandafterschoolandhealthyrestaurantprogramstosupporthealthyfoodandhealthyeatinghabits
Usethetemplatebelowtoplanyourimplementation
Policyimplementationplanningtemplate
ListthespecificdesiredoutcomesInthepreviousstepyouworkedwithdepartmentstafftoestablishspecificmeasurableoutcomesandinterimindicatorsPrioritizetheindicatorsthatwilldemonstrateprogressUsethespacebelowtocapturethedesiredoutcomesofthepolicychange
ExampleShapeupSomervillebeganwiththegoalofpreventingobesityin1stthrough3rdgraders
Yourturn
____________________________________________________________________________
____________________________________________________________________________
Designimplementationandevaluation
44
SpecifythetargetpopulationAreyoudesigninganinterventionthattargetsonlycertainsitesAreyouplanningonimplementingaprogramorinitiativeforaspecifictargetaudienceClearlyspecifythepopulationyouplantotarget
IfyouplantoconductarigorousimpactevaluationasnotedbytheWilsonSheehanLabforEconomicOpportunitiesitisimportanttoestablishconcreteobjectiveeligibilitycriteriaandensurethatthesecriteriaareappliedconsistentlybyallstaffduringscreeningandintakeItisalsovitaltoidentifytheindividualswhoareeligibleforaparticularprogramorserviceevenifeveryoneeligiblecannotbeservedbyexistingresourcesThishelpscitiesidentifyanyunmetneedsandguidefutureresourceallocationwhilealsocreatingthepossibilityofarigorousimpactevaluationbyidentifyingapotentialcomparisongroupFinallywhenconductingscreeningandintakeitisimperativetocollectandmaintainbasicinformation(egnamedateofbirthgender)aboutallprogram-eligibleindividualsnotjustthosewhoparticipateintheprogram
ExampleIntheShapeUpSomervilleexampletheprogrambeganbytargeting1stthrough3rdgraders
Yourturn
____________________________________________________________________________
____________________________________________________________________________
DefinethelocationWherewillthepolicychangetakeplaceWillimplementationtakeplaceatonesiteormultiplesites
ExampleShapeUpSomervilleinterventionsweredeliveredinschools
Yourturn
____________________________________________________________________________
____________________________________________________________________________
ClarifyimplementationpartnersandarticulatetheirrolesWhatarethekeyorganizationsinstitutionsorpeopleresponsibleforimplementationTheprojectteamshouldbeasclearandspecificaspossibleinprovidingadescriptionoftheconcreteactivitiesthatimplementationpartnersaremeanttocarryoutPracticeprofilesmaybeahelpfultooltohelpdefinewhatisrequiredoftheactorswhoareresponsibleforimplementingtheintervention
Designimplementationandevaluation
45
ExampleInSomervillemayoralleadershipandengagementofmultiplecityagencies(publichealthprimarycareschoolsCambridgeHealthAllianceetc)streamlinedandconnectedthevariousShapeUpSomervilleinitiativesandgrantswhichhelpedtokeepthetalentandenergyfocusedonhealthyeatingandactivelivingforallsectorsofSomervilleSUSpartnersultimatelyincludedschoolsnonprofitsCityandstateagenciesandnationalfoundations
Yourturn
____________________________________________________________________________
____________________________________________________________________________
DescribetheinterventionWhatarethestepsactivitiesorprocessesRememberthatthemechanismthespecificstepsorprocessthatleadtobehaviorchangeiskeytounderstandingtheeffectivenessofthepolicychangeandsharingtheresultswithothers
ExampleGoalsofShapeUpSomervilleweretoinfluenceeverypartofanearlyelementaryschoolchildrsquosdayndashbeforeduringandafter-schooltopromotehealthyeatingphysicalactivityandtheirweightstatusInterventionincludedfoodserviceenhancementstoimprovebreakfastandlunchandstaffprofessionaldevelopmentwalktoschoolactivitiesSUSclassroomcurriculumandprofessionaldevelopmentschoolwellnesspolicydevelopmentSUSafter-schoolcurriculumandprofessionaldevelopmentoutreachandeducationtohomethroughmaterialsforumseventsldquoSUSapprovedrdquorestaurantscommunityoutreachandcapacitybuildingthroughpolicydevelopmenttrainingsmediaplacements
Yourturn
____________________________________________________________________________
____________________________________________________________________________
ClarifytimelineandmilestonesfordeliveryandevaluationHowfrequentlywilltheservicesinterventionbedeliveredHowlongwilltheprogramrunWhatisthetimetoexpectedimpacts(eg12monthsafterprogramcompletion)
Designimplementationandevaluation
46
ExampleShapeUpSomervilleledbyTuftsUniversityFriedmanSchoolofNutritionScienceandPolicyconductedaquasi-experimental(nonrandomized)interventionstudyamong1stthrough3rdgradestudentsinSomerville(intervention)andtwocomparisoncommunitiesfor9-monthsduringoneschoolyear(fall2002ndashspring2003)ThestudyshowedpositiveassociationoftheinterventiononchangeinBMISUSreleasedreportssummarizingevaluationsprogresstodateplansforadditionalprojectsandimpactstenandfifteenyearsintotheinitiative
Yourturn
____________________________________________________________________________
____________________________________________________________________________
PlanyouranalysisTheprevioussectionaddressedestablishinganaccountabilityframeworktomonitorprogressandimplementationDeterminewhichindicatorswillbediscussthroughthatprocesswhowillsharethedataandanalysisandhowfeedbackfromthoseconversationswillbeincorporatedbytheprojectteam
ExampleTheMayorrsquosofficeimplementedadata-drivendecisionmakingapproachcalledSomerStatwhichwassynergisticwiththeapproachestakenbytheresearchandevaluationteamsdeployedforthespecificgrants
Yourturn
____________________________________________________________________________
____________________________________________________________________________
PlanforsustainabilityIfthepolicyandprogramchangeissuccessfulhowwilltheprojectteamsustainandexpandtoadditionalsitesorpopulations
ExampleAlthoughmostofthepartnershipsforgedintheearlyyearsweremaintainednewgrantsandcollaborationsexpandedattentiontoolderpublicschoolstudentsactivetransportationandtheGreenLinesubwayextensionthebuiltenvironmentfarmerrsquosmarketeffortssuchasldquoGrowninSomervillerdquowithUnionMainStreeteconomicdevelopmentandhealthdisparitiesaswellasdatacollectionmonitoringandsurveillance
Yourturn
Designimplementationandevaluation
47
____________________________________________________________________________
____________________________________________________________________________
Pilotplanningtemplate
IfyouplantoconductapilotfirstdeterminethepurposeofthepilotDoyouwanttoobservewhetherthepolicychangeresultsinthedesiredbehaviorchangeAreyouconcernedaboutfeasibilityscalabilityorcostDoyouwanttotestwhetheryourdesiredresultscanbeachievedthroughthispolicyapproachClarifythegoalofthepilotandthequestionsyouhopetoanswerThiswillhelpyouidentifythekeyindicatorsthatyouwillusetomeasurewhetherthepilotissuccessfulBeforebeginningestablishabaselineofcurrentperformanceandsetyourtargetsaccordinglybasedonwhatyouhopetoaccomplishthroughthepilot
NextestablishthepilotplanforyourprojectteamincludingmilestonesforteamcheckinstodeterminewhethertomoveaheadorchangecourseThisisseparatefromthemeetingsthatwilloccuraspartoftheaccountabilityframeworkUsethespacebelowtocreateyourtimelineanddevelopyourmeetingschedule
Task Personresponsible Startdate Checkindate Deliverable
Planyourevaluation
WhileyouaredesigningtheimplementationitisalsotimetobegintodevelopyourplanforanalysisAsnotedbytheWilsonSheehanLabforEconomicOpportunitiesdecisionsaboutprogramsetup-includingoutreachscreeningintakedatatrackingandenrollment-candeterminewhetherornotarigorousimpactevaluationwillbepossibleAsdiscussedcityleadersmayfindithelpfultoincludeevaluationpartnersinearlyconversationsaboutprogramsetupDuringpreviousstagesyoumayhaveworkedwitharesearchertoplanyourproposedpolicychangeYoushouldbeclearaboutthepurposeoftheevaluationanddevelopanevaluationplanBelowareadditionalconsiderations
Causationvstrendanalysis
IntheEstablishaframeworktomonitorprogressstagetherewasguidanceontrackingtrendsovertimeItisimportanttonoteherethatmonitoringdataisdifferentfromanalyzingacausaleffectTrendanalysiswillillustratethatchangesareoccurringandmayleadtoquestionsthatcouldbeansweredbyyourevaluationbuttrendanalysisalonedoesnotprovideevidenceaboutwhatisdrivingthosechangesYoumayseechangesinyour
Designimplementationandevaluation
48
outcomeindicatorsbasedonvariablesoutsideoftheCityscontrolForinstanceachangeinfederaltradepolicycanhaveanimpactemploymentratesinyourCityoutsideofyourCitysnewjobtrainingandplacementprogram
ToaddresscausalityyoumustcomparetheoutcomesofindividualswhoreceivedtheinterventionwiththosewhodidnotWithoutacomparisongroupofsimilarindividualswhowerenotexposedtotheinterventionitisimpossibletoknowwhichfactorsareresponsiblefortheoutcomesRandomizedControlTrials(RCTs)inwhichindividualsarerandomlyassignedtotreatmentandcontrolgroupsarethegoldstandardinprovingcausalityIfyoucandemonstratethatoutcomesimprovedforthosewhoreceivedtheinterventionanddidnotimproveatallorimprovedlessforacomparisongroupyoucanbesomewhatcertainthattheinterventionwasthecause
Examplesofothernon-causalimpactevaluationsthatarerigorousandreliableincludecertainquasi-experimentalstudiesusingcomparisongroupdesignssmartinterruptedtimeseriesanalysesthatmodeltrendsovertimeinwaysthatallowestimationofinterventioneffectswelldesignedpre-posttestanalysesneedsassessmentsimplementationstudiesetcFormoreinformationondifferenttypesofevaluationsseetheSageResearchMethodswebsite
Lookfordifferencesingroups
BuildintimetoalsoexaminethepilotdatatoseeifgroupsareexperiencingdifferentresultsReviewingperformancebasedonoutcomemeasuresforapopulationinaggregatemayhidethesubtlewaysthatsomepeoplemaybeaffecteddifferentlysoitisnecessarytocarefullyanalyzethedatafordisparateoutcomesForinstancetheMovingtoOpportunity(MTO)studythatexaminedtheimpactofhousingvouchersonfamilieswholivedinhigh-povertyneighborhoodsfoundthatchildrenwhomovedbeforetheageof13hadbettereducationandhigherincomesasadultsHoweveradultsdidnotseeimprovededucationandemploymentoutcomes
Disaggregatingdataintomorespecificcategoriesofraceandethnicitycanhelpunderstandtrendsanddisparitiesinamoregranularwaywhichcanhelpdecisionmakerstargetpoliciesandinterventionsinamoreequitableway
EarlierintheRoadmapweworkedtoidentifystakeholdersandwhomightbepositivelyandnegativelyimpactedbyachangeinpolicyUsetheresultsofthatexercisetoinvolvethecommunityininterpretingdataanalysisandtogatherqualitativefeedbackfromthecommunitytocallattentiontoissuesForinstanceanincreaseinyouthdetentionmaybeexplainedbyinefficienciesorlackofaccesstotransportation
Designimplementationandevaluation
49
Designimplementationandevaluation
50
Shareyourstoryandsuccess
Sharetheresults
Storytelling
AnintegralstepinyourstrategyissharingtheresultsofyourpilotorpolicychangewithyourcommunityThisincludesdevelopingmessagingaroundtheworkofyourorganizationandcreatingaplantodisseminateyourstoriesWhytellstoriesaboutyourwork
Informandorunderstandasubjectorsituation
Giveneededcontextofdataorbitsofinformation
Relatablestoriesarememorableforreadersviewers
ReadersorviewerswillseetheinformationasmorecompellingwhenthereisademonstratedandrelatablechangeorimpactonthemselvesortheirfriendsneighborsandcolleaguesStoriesareaboutpeopleandgovernmentisfundamentallyabouthelpingpeopleSotellthestoriesofthepeopleyouandyourcitycolleagueshelped
Keepthesetipsinmindasyoudevelopyourstorywithyourteam
WhatwereyourfindingsandwhyshouldanyonecareTellyouraudiencehowtheproblemwassolvedWhathastheimpactbeenonresidentsandcitystafftodateWhatwasthesuccessandweretheremultiplesuccessesalongthewayIfyouhavepreliminaryresultsordatashowingimprovementtosharethisisthetimetodoit
WhowasinvolvedandimpactedReviewthelistofstakeholdersHowwereeachofthemimpactedbytheprogramHumanizeyourstoryandimpactbyaddingadditionalvoicesThesevoicescanbeoneofthefollowing
Apersonorfamilywhowaspositivelyaffectedbythechangesuccess
Aninvolvedthird-partydescribingboththeproblemandthesolution(ieanorganizationnonprofitprovidersbusinesstradeassociationsportsteametc)whoispartofthecommunitythatthegovernmentworkedwithonthechangesuccess
Someonefromthegovernmentitselfwhocanspeakaboutthechangesuccess
TellmoreaboutyourcityDescribeyourcityincludingpopulationdemographicsindustriesetcWhatareitschallengesWhatisitknownforWhatmakesituniqueHowisitsimilarordifferentfromothercitiesintheUS
Shareyourstoryandsuccess
51
DescribeyourcityrsquoschallengeWriteoutthechallengeyourcityisfacing(asdefinedinyourproblemstatement)HowdoesitaffectresidentsHowdoesthisproblemaffecttheworkofcitystaffmembers
AddthedetailsProvidedetailsonyourappliedresearchstrategyTalkaboutwhatyourcityhasdonenthepasttoresolvetheissueDiscusshowyourstrategyhaschangedwhenyoubegantoadoptthisapproachWasmoneysavedorinvested
Usethisstorymapexercisetoorganizeyourthoughtsandguaranteethatyouincludealloftheelementsofatraditionalstory
AfteryoudevelopyourmessagingcreateacommunicationsmarketingandoutreachstrategydesignedtospreadyourmessageYourstrategyshouldwalkyouthroughthefollowingkeyactivities
1 IdentifyandprofileyouraudienceWhencreatinganddisseminatingyourmessageitisalwaysgoodtothinkaboutyouraudienceWhoexactlywouldbeinterestedinwhatyouhavetosayThegeneralpopulationofthecityPeoplewhoworkinnonprofitsBusinessownersItmattersbecausenoteveryonewholivesinacitymaybethetargetoftheinformationBusinessownersmaybemoreinterestedindatashowingeffectsonthebusinessdistrictthanothersandsoonWewillcoverthislaterbutthisstepwillalsohelpyoutothinkofthemosteffectivewaytoreachyouraudience
2 Defineyourcalltoaction(ifyouhaveone)SpendsometimeconsideringwhyyouareinterestedintellingyourstoryDoyousimplywanttoupdatethepubliconyourprogressorcelebrateyoursuccessesAreyouinterestedinyouraudiencechangingitsbehaviororsupportingaspecificinitiativebeinglaunchedbyyourorganizationIfyouwanttohaveyouraudienceactorbehaveinacertainwaybespecificandincorporateitinyourmessaging
Shareyourstoryandsuccess
52
3 SelectperformancemeasuresThinkabouthowyouwoulddetermineifyourcampaignissuccessfulandselectappropriateperformancemeasuresYouwillrevisitthesemeasuresattheendofyourcampaigncommunicationsplanimplementationtodeterminewhetheryouneedtomodifyyourcommunicationsandoutreachstrategyforthefuture
4 MethodstocommunicateyourstoryTherearemanywaystocommunicatesuccessofyourprojectHerearesomesuggestionsonreachingaudiencesItisimportanttodeterminewhichmethodsarethemosteffectiveforyourdesiredaudiencetoreceiveinformationDoyourresearch(PewResearchCenterhasexcellentstudiesonthereachofmediadigitalmediaandmarketing)ForexampleifseniorsareyourtargetpopulationandtheyarelessinclinedtouseInstagram(thatistrue)youmaywanttoconsiderFacebookorthenightlylocalnewsbothofwhichareusedmoreoftenbyseniorsBelowisalistofsomemethodsandtoolsyoucanuse
i Pressreleasestolocalnewspapersandtvstations
ii ShortsmartphonevideosAgreatwaytoengageanaudienceisthroughvideoAskonequestionofsomeonewhoispartofthestoryMaybethereisasocialworkerwhocandiscusshungerinyourcityThesevideoscanbereusedonmultipleplatforms
iii Long-formwriting
iv TellingindividualstoriesthroughaseriesofphotosandshortnarrativesforpublicationonFacebookorInstagram
v Op-edsopinionpiecesplacedinthelocalpaper
vi Emailupdatestoyourlist
vii Connectingwithappropriateorganizationsthatareinvestedinthetopicandworkwiththecommunityandaskthemtodisseminateinformationtotheirmembers
viii EmployingldquoWordofMouthrdquo(WOM)tacticsbycreatingtownhallsorfreeeventsforresidentstoattendintheirownneighborhoods
5 DeterminethetypeofcollateralUsuallycampaignsthattargetresidentsrequireseveralavenuestoreachthemPeoplearenotalwaysinthesameplaceorspaceHavingakeenunderstandingofyouraudiencewillassistyouwithyourchoicesonwhatneedstobedevelopedSometimesthatmeansimagesforsocialmediaaonepageflierthatgivespeopleinformationonattendinganeventoranin-depthbrochure
Shareyourstoryandsuccess
53
6 ImplementyourstrategyKeepinmindstrategiescanbefluidSometimessomethingmaynotturnoutasexpectedevenifduringplanningsessionsitwasldquosupposedtordquoItisokYoucananalyzewhathappenedduringadebriefsessionandonceyoursurveysandothermeasurementshavebeenfinalized
Visualizations
Withtherightvisualizationyoucanreviewandshareinformationinsightsanalysisperformanceandtrendsallofwhichisvaluabletoacross-sectionofthepublicButwhichvisualizationisrightforyouTochoosetherighttypeofchartgraphdeterminethetypeofquestionyouwanttoanswerSomecommoncharttypesarebelowandmoreinformationcanbefoundhere
Exploreadditionalanalyticaltechniques
Duringthegoal-settingphaseyouestablishedoutputoutcomeandimplementationmeasuresBasedontheresultsofyourpilotyoumayneedtoadjustyourmeasuresWiththepilotdatainhandyoumayhavesomeideasofhowfutureanalysescouldbedifferentandpotentiallyimprovethequalityofyourprogramTakesometimetobrainstormotherrelevantdatasetsthatyoudidnothavethetimetopursueearlierincludingmessydepartmentaldatathatneedsmoreattentionorsourcesofpublicdatafromthestateandfederalgovernmentnonprofitsandevenprivatecompaniesthatcouldsupplementyourunderstandingofanissueUsingmachinelearningtechniquesyoumightbeabletobetter
Shareyourstoryandsuccess
54
predictdemandforhomelessservicesusingNOAAweatherforecastspreemptwatermainbreaksusingsoilstratigraphyorinvestineconomicdevelopmentbymonitoringrealestatelistingdata
Advancedanalyticaltechniquescanalsohelpyourimplementationrunmoreeffectivelybyintegratingdata-informedtargetinganddecisiontoolsintotheprocessByautomatingroutineworkflowsandsurfacingsubtletrendsthesetoolscanhelpyouiterateinamoreprecisemannerandprovidekeystrategicguidancetofront-lineemployeesthatwouldotherwisebeunavailableThesesortsofdata-science-enhancedprocessescanbeclassifiedasoneofsixprojecttypesasexploredfurtherbytheNewOrleansLANOLAlyticsteamFormoreguidanceonscopingasimilarprojecttheGovExadvancedanalyticsscopingguidegivesyouastep-by-stepoverviewofhowtobesuccessful
Usethespacebelowtolistyourideasforpossibleadvancedanalyticsprojectsbytype
Typeofproject Ideas
Findingtheneedleinthehaystack
Prioritizingforimpact
Earlywarningtools
Betterquickerdecisions
Optimizingresourceallocation
Experimentingforwhatworks
Keepthefeedbackloopgoing
GatherinformationonpublicsentimentaboutpolicychangethroughfocusgroupssurveysandpublicmeetingsledbyadvisorygroupmembersNotewhethernewinformationisavailablethatcanhelpanalyzeorcontextualizeyourprogress(egfederalorstatedatareports)Sharetheresultsofyourpilotandthepolicychangeprocesswithothercitiesandresearchers
Shareyourstoryandsuccess
55
WorkingwitharangeofevidencePresentlythereisalotofattentionandinvestmentinevidence-basedpolicymakingincludingthefederalbipartisanCommissiononEvidence-BasedPolicymakingthePewCharitableTrustsreportsandtoolkitsonHowStatesEngageinEvidence-BasedPolicymakingandtheLauraandJohnArnoldFoundationsupportedEvidence-BasedPolicymakingCollaborativeManyoftheseinitiativesfocusonrigorousevaluation--meaningrandomizedcontrolledtrials(RCT)orotherexperimentalmodels--asthebestkindofevidencetouseHoweverthisbodyofevidenceisofteninsufficientforthespecifickindsofquestionsthatgovernmentscanhavewhenmakingadecisionaboutpolicyInthisguideweproposeanapproachtosupportgovernmentsinincorporatingadditionaltypesofevidenceintotheirdecisionmaking
OnewaytounderstandarangeofevidenceistouseNestarsquosStandardsofEvidencetoframeanddefineevidenceTheBritishnonprofitNestaworkstoinjectpositivechangeintothewayUKgovernmentpoliciesandprogramsfunctionOneoftheirkeyareasofworkhasbeendevelopingWhatWorksCentresorganizationsthatcollectcommissionsynthesizeanddisseminateevidenceaboutinterventioneffectivenessacrossarangeofsocialimpactareasNestadevelopedthefollowingStandardsofEvidencetoprovideausefulwaytodocumenthowdifferentkindsofinformationdemonstratedifferentlevelsofevidence
Workingwitharangeofevidence
56
Theimportanceofpotentiallybiasedinformation
Onechallengepresentedbyanevidenceclassificationthatacknowledgesthevalueofnon-scientificformsofevidenceistofigureouthowtobalanceallofthepiecestogetherwhenyouusethemTounderstandhowtodothiswehavetothinkabouthowtopositionpotentiallybiasedinformationindecisionmaking
WhenwelookatinformationwhichhasnotbeenrigorouslyevaluatedweacceptthatthisinformationmightbebiasedThewholereasonweconductrigorousevaluationistoremovebiasfromourobservationsWeconductstructuredtrialstoproduceresultsthatareconsistentandgeneralizableIfafindingisunbiasedthisaccuracymdashthislackofbiasmdashcanbeprovedbecauseanyindependenttestercanreproduceitwhentheyfollowanidenticalprocess
YoucanlookatpotentiallybiasedinformationasjustinformationthatisearlieroninitspathtorefinementreplicationandverifiedaccuracyApieceofinformationbasedonanyoneindividualrsquosobservationsislikelytobebiasedbecauseitiscomingfromjustonesourceBiasisreducedbygatheringtogetherlotsofindependentsourcesofinformationandfiguringoutitsmostcommonattributes
MaybethebiasedinformationmakesaclaimthatistoolargemdashitsayssomethingistrueunderallconditionswhenitisreallyonlytrueundercertainconditionsorthataneffectofaninterventionisbiggerthanitstrueaverageorthataneffectlastsovertimewhenitdoesnotPerhapschangingtheclaimtobemorespecificmdashcriticallyreviewingtheinformationtoseeifspecificconditionsmustbemettoachievetheoutcomesmdashwouldmaketheclaimmoreaccurate
IfyoutestedyourclaimfounditdidnotholdupinsomecaseschangedittesteditagainuntilyourclaimrepeatedlyheldupthenyoucoulddemonstratethattheclaimwasnolongerbiasedbutaccurateThisprocessoftestingimprovingtestingandvalidatingiswhatwegetfromstructuredscientificstudy
Withindividualpiecesofinformation--crediblehypothesescasestudiesexpertopinion--youcannotassumethatkindofaccuracyAtthesametimetheseearlyobservationsarefoundationaltoanyreplicablescientificobservationsWewillnevergetanywhereifwedonotstartwiththesepiecesofinformationeveniftheymightbebiased
Combininglevelsofevidencetogetthemostdetailedinformationforyourcity
ItisespeciallyimportanttoacceptdifferentformsofevidencewhenyoufacequestionsthathavenotyetbeentestedbyindependentstudiesSomeofthemostimportantquestionsrelatetothingsthatareveryspecifictoyoursituationYoufaceaparticularcombinationof
Workingwitharangeofevidence
57
issuesactorstrendsandresourcesthatisultimatelydifferentfromeveryotherplaceintheworld
ThisisimportantbecauseaspolicypractitionersandnotscientistsyoucaremoreaboutyourspecificcasethanahypotheticalaveragegeneralcaseYoucarelessaboutthetotallikelyrangeofimpactsfromapolicythanhowthissingleimplementationwillworkYouwantinformationaboutpoliciesthatwillprovideyouwiththebestpossiblepredictionabouthowtheywillworkinyourcontext
ByputtingdifferentkindsofevidenceintothesameframeworkNestarsquosStandardsofEvidenceimplythatwhenyoucannotgetrigorouslytestedexperimentallyverifiedinformationitisappropriatetoworkwithwhatwedohaveThisisrelevantforthemunicipalimplementationcontextbecausenostudypulledfromaliteraturereviewwillaccountforallofthevariablesyoufaceinyourowncontext
Intheabsenceofhavingastudythatpredictshowaninterventionworksinyourspecificcontextyouwillneednon-scientificinformationtotrytofillinthosegapsYouwillneedtogatherstoriesfromresidentsstaffmembersandotherstakeholdersthatsitonLevels1or2ofthestandardsofevidence
Someofthekindsofinformationyouwillprobablygetmoreeasilyfrompeoplethanstudiesinclude
howpeopleincitieslikeyoursreactedtoaninterventioninpublicforumshowpeoplewhoimplementedaninterventioninacitylikeyoursfiguredouthowtogetitpassedhowthecitycommunicatedpubliclyabouttheapparentbenefitsanddrawbacksoftheinterventionthepoliticalacceptabilityoftheintervention
Youcanremainsimultaneouslyawarethattheinformationyouaregatheringthroughindividualstoriesisbiased--notgeneralizableinformedbyspecificperspectives--whilealsorecognizingthathavingsomeinformationaboutsomethingimportantisbetterthannoinformationEvidencefromthefirsttwoevidencelevelsareessentialbuildingblocksofknowledgeandperfectlyusefulwhenyoucannotgetdetailedunbiasedknowledge
Youhavetostartsomewhereandyoushouldalwaysstartwithevidenceyouhaveaboutyourveryspecificconditionsthatyouknowareimportantforsuccess
Workingwitharangeofevidence
58
Thankyou
Interviewswithcities
GovExheldsemi-structuredinterviewswithsixstaffmembersinfourcitiestoestablishtheinitialstepsoftheRoadmapWeaskedrespondentstotalkaboutaninitiativeorpolicythattheyhavebeenresponsibleforimplementingintheircityRespondentsfromthreecitiesspokeabouttheireffortstoaddresshomelessnessandrespondentsfromonecityspokeaboutblightAfterrespondentsidentifiedtheproblemtheyspokeabouthowthecityprioritizedthisissuethevariousstepsinvolvedandbarrierspresent
GovExconductedtextanalysisoftheseinterviewswhichrevealedthatpeoplewasthemostcommonlyusedtermduringtheseconversationswhendiscussingthestepstochangeaprogramorpolicyThisisathemethroughoutthisRoadmapThefirstsectionbeginswithidentifyingtherelevantpeopleTheconversationsaroundevidencearefocusedonhumanbehaviorratherthanmethodologyorstrengthofevidence
Thankyou
59
OncewehadadraftoftheRoadmapwefollowedupwithcitiesonanindividualbasistolearnmoreaboutwhatwouldbehelpfulWearegratefultothefollowingcitystafffortheinterviewsandindividualconversations
MichaelBaskinChiefPolicyOfficerChattanoogaTN
MelissaBridgesPerformanceandInnovationCoordinatorLittleRockAR
VictoriaCarreoacutenAdministrativeOfficerLasVegasNV
TyroneGrandstrandCommunityDevelopmentPlannerFargoND
TibLaughlinDirectorofGeneralServicesKansasCityKS
TimMorelandDirectorofPerformanceManagementandOpenDataChattanoogaTN
CharlesRosendahlManagementAnalystDentonTX
Thankyou
60
MelissaSiebenAssistantCountyAdministratorKansasCityKS
GinaThompsonDirectorofStrategicServicesLewisvilleTX
JHUFacultycollaboration
GovExalsoconvenedthecitycontactsandfacultypartnersinaseriesofAdvisoryGroupmeetingsThesemeetingsservedtoconnectcitiesandfacultytounderstandoneanothersperspective
StefanBaralBloombergSchoolofPublicHealth
AshleyBernerJHUSchoolofEducation
MelissaDavey-RothwellBloombergSchoolofPublicHealth
JonLinksBloombergSchoolofPublicHealth
TamarMendelsonBloombergSchoolofPublicHealth
KristinMmariBloombergSchoolofPublicHealth
KeshiaPollackPorterBloombergSchoolofPublicHealth
JoshSharfsteinBloombergSchoolofPublicHealth
LizStuartBloombergSchoolofPublicHealth
LindsayThompsonCareyBusinessSchool
DanielWebsterBloombergSchoolofPublicHealth
Partnersindataandevidence-basedpolicy
Wealsospokewithpartnerswhoareexpertsinthefieldofevidence-basedpolicywhosharedtheirknowledgeandresourcesofcurrentandfuturethinkinginevidence-informedthinking
WendyBarrenoAssociateDirectorWilsonSheehanLabforEconomicOpportunitiesUniversityofNotreDame
TiffanyDavenportDirectorofDataPartnershipsBaltimoresPromise
TomiHiersDirectorBaltimoreCivicSiteAnnieECaseyFoundation
ChrisKingsleySeniorAssociateAnnieECaseyFoundation
KathrynOliverAssociateProfessorLondonSchoolofHygieneandTropicalMedicine
Thankyou
61
KristenWatkinsSeniorPolicyAssociateJ-PAL
GovExParticipants
GovExAnalystMiriamMcKinneyanalyzedtheinterviewtranscriptsandconductedtextanalysisSeniorImplementationAdvisorEmilyShawwillintegrateRoadmapactivitieswithadditionalservicedeliveryactivitiesatGovEx
Thankyou
62
ReferencesAbrahamKGampHaskinsR(2017)Thepromiseofevidence-basedpolicymakingReportofthecommissiononevidence-basedpolicymakingWashingtonDCCommissiononEvidence-basedPolicymakingBigmove2Extendingthegreen(nd)Retrievedfromhttpswwwcityoflewisvillecomdoing-businesslewisville-2025big-move-2-extending-the-greenBatesMAampGlennersterR(2017)ThegeneralizabilitypuzzleRetrievedfromhttpsssirorgarticlesentrythe_generalizability_puzzleBuildwithnotfor(2018January)RetrievedfromhttpsbuildwithnotforwordpresscomCairneyPampOliverK(2017)Evidence-basedpolicymakingisnotlikeevidence-basedmedicinesohowfarshouldyougotobridgethedividebetweenevidenceandpolicyHealthResearchPolicyandSystemsdoi101186s12961-017-0192-xCairneyPOliverKampWellsteadA(2016)TobridgethedividebetweenevidenceandpolicyReduceambiguityasmuchasuncertaintyPublicAdministrationReview76(3)399-402Retrievedfromhttpdxdoiorg101111puar12555Centreforpublicimpact(nd)RetrievedfromhttpswwwcentreforpublicimpactorgCernaL(2013)ThenatureofpolicychangeandimplementationAreviewofdifferenttheoreticalapproachesOrganisationforEconomicCo-OperationandDevelopmentRetrievedfromhttpswwwoecdorgeduceriTheNatureofPolicyChangeandImplementationpdfColoradoSprings(nd)Retrievedfromhttpscoloradospringsgovcommunicationspagewhat-expect-when-you-speakupContinuumofcare(coc)program(nd)RetrievedfromhttpswwwhudexchangeinfoprogramscocCouncilpresidentkicksoff$12millionyouthfund(nd)Retrievedfromhttpwwwbaltimorecitycouncilcomcontentcouncil-president-kicks-12-million-youth-fundDurlakJAampDuPreEP(2008)ImplementationmattersAreviewofresearchontheinfluenceofimplementationonprogramoutcomesandthefactorsaffectingimplementationAmericanJournalofCommunityPsychology41327-350Retrievedfromhttpswwwncbinlmnihgovpubmed18322790Fishbonediagram(ishikawa)(nd)Retrievedfromhttpasqorglearn-about-qualitycause-analysis-toolsoverviewfishbonehtmlFivewhysGettingtotherootofaproblemquickly(nd)RetrievedfromhttpswwwmindtoolscompagesarticlenewTMC_5Whtm
References
63
FligsteinNampMcAdamD(2011)TowardageneraltheoryofstrategicactionfieldsSociologicalTheory29(1)1-26RetrievedfromhttpsociologyberkeleyedusitesdefaultfilesfacultyfligsteinFligstein20McAdam20Soc20Theory20PaperpdHeinkelP(2018January12)CitymullspermitsforbikesharingcompaniesRetrievedfromhttpwwwdentonrccomnewsdenton20180112city-mulls-permits-bike-sharing-companiesHowisareamedianincomecalculated(2012October)Retrievedfromhttpswwwhudexchangeinfofaqs843how-is-area-median-income-calculatedLampkinLMampHatryHP(2003)KeystepsinoutcomemanagementUrbanInstituteRetrievedfromhttpswwwurbanorgresearchpublicationkey-steps-outcome-managementMarshDampSmithM(2000)UnderstandingpolicynetworksTowardsadialecticalapproachPoliticalStudies484-21Retrievedfromhttpjournalssagepubcomdoiabs1011111467-924800247McCannL(nd)RetrievedfromhttplaurenellencomMoultonSampSandfortJR(2016)ThestrategicactionfieldframeworkforpolicyimplementationresearchPolicyStudiesJournal45(1)Retrievedfromhttponlinelibrarywileycomstore101111psj12147assetpsj12147pdfv=1ampt=jdkhhcp4amps=e184c625e1855a6b1615ccb99ee27b1c0b3212d8PaynterB(2018January1)What100citiesarelearningfromeachotherbysharingtheirdataRetrievedfromhttpswwwfastcompanycom40518004what-100-cities-are-learning-from-each-other-by-sharing-their-dataPeakacademy(nd)Retrievedfromhttpswwwdenvergovorgcontentdenvergovenmayors-officeprograms-initiativespeak-performancepeak-academyhtmlPinnockHBarwickMCarpenterCREldridgeSGrandesGGriffithsCJRycroft-MaloneJMeissnerPMurrayEPatelASheikhAampTaylorSJC(2017)Standardsforreportingimplementationstudies(stari)ExplanationandelaborationdocumentBMJOpenRetrievedfromhttpbmjopenbmjcomcontentbmjopen74e013318fullpdfProctorESilmereHRaghavanRHovmandPAaronsGBungerAGriffeyRampHensleyM(2011)OutcomesforimplementationresearchConceptualdistinctionsmeasurementchallengesandresearchagendaAdministrationandPolicyinMentalHealth38(2)65-76doi101007s10488-010-0319-7TandonSD(2018)ProgressincommunityhealthpartnershipsResearcheducationandactionRetrievedfromhttpswwwpressjhuedujournalsprogress-community-health-partnerships-research-education-and-actionProvanKGampKenisP(2007)ModesofnetworkgovernanceStructuremanagementandeffectivenessJournalofPublicAdministrationResearchandTheory
References
64
18(2)229-252Retrievedfromhttpsacademicoupcomjpartarticle182229935895SabatierPA(1988)Anadvocacycoalitionframeworkofpolicychangeandtheroleofpolicy-orientedlearningthereinPolicySciences21(3)129-168Retrievedfromhttpwwwjstororgstable4532139SandstromAampCarlssonL(2008)TheperformanceofpolicynetworksTherelationbetweennetworkstructureandnetworkperformancePolicyStudiesJournal36497-524Retrievedfromhttponlinelibrarywileycomdoi101111j1541-0072200800281xabstractSchwartzMampWilsonE(nd)WhocanaffordtoliveinahomeAlookatdatafromthe2006americancommunitysurveyRetrievedfromhttpswwwcensusgovhousingcensuspublicationswho-can-affordpdfShearerJCAbelsonAKouyateBLavisJNampWaltG(2016)WhydopolicieschangeInstitutionsinterestsideasandnetworksinthreecasesofpolicyreformHealthPolicyandPlanning31(9)1200-1211Retrievedfromhttpsdoiorg101093heapolczw052ThelessonsofadministrativedataHigh-profilepolicy-relevantresearchpoweredbyadministrativedata(nd)Retrievedfromhttpswwwpovertyactionlaborgsitesdefaultfilesresources20161114-The-Lessons-of-Administrative-DatapdfToolsandotherneedsforqualitativeandquantitativemethods(2015February)AgencyforHealthcareResearchandQualityRetrievedfromhttpwwwahrqgovprofessionalsquality-patient-safetytalkingqualityassesstoolshtml
WeaverRCampBagchi-SenS(2013)SpatialanalysisofurbandeclineThegeographyofblightAppliedGeography4061-70
References
65
UniversityPartners
Name Title Affiliation Area(s)ofexpertise
Contactinformation
Examplesofworkingwithnon-academicdecisionmakers
WendyPBarreno
AssociateDirector
WilsonSheehanLabforEconomicOpportunitiesUniversityofNotreDame
LEOsfocusareasincludehealtheducationhousingchildwelfareeconomicopportunitycriminaljustice
wbarrenonduduorleondedu
MelissaDavey-RothwellPhD
AssociateScientist
JohnsHopkinsBloombergSchoolofPublicHealth
Health mdavey1jhuedu410-614-5854
BaltimoreCityHealthDepartmentLeadersMarylandDepartmentofHealthAddictionRecoveryServicesinDCExecutiveDirectorsofcommunityorganizationsacrosstheUSthatfocusonHIVSTIsubstanceabuseandimplementationscience
ElizabethStuart
AssociateDeanforEducationandProfessorDepartmentsofMentalHealthBiostatisticsandHealthPolicyand
JohnsHopkinsBloombergSchoolofPublicHealth
Educationsocialpolicystatisticalmethods
estuartjhuedu
Resources
66
Management
Design-ThinkingResources
EngagementGuideforCivicTech
Journeymap
PerformanceManagementResources
DenverPeakAcademy
Proxymeasuresguide
Talkingaboutperformancewithoutcomplicatingit(video)
SettingPerformanceTargetsguide
Datavisualizationguide
Performancemanagementgettingstartedguide
Exercises
Fishboneexercise
5Whys
AdministrativeData
TheLessonsofAdministrativeData(JPAL)
EvaluationResources
Quantitativeandqualitativemethods(USDepartmentofHealthandHumanServicesAgencyforHealthcareResearchandQuality)
KeyStepsinOutcomeManagement(UrbanInstitute)
RCT-YES
Sageresearchmethods
BestPracticesforMixedMethodsResearchintheHealthSciences
ACCORDChecklist
Pilotandimplementationplanning
Benchmarkingexamplesandproxymeasures
Seattleopendataplaybook
Resources
67
Datasciencescopingguide
Resources
68
AboutGovExGovExhelpsgovernmentsusedataandevidencetoimproveresidentsrsquolivesWesupportgovernmentsinbuildingtheirinternalcapacitytooperatesustainableanalyticspracticeswhichimprovecommunityoutcomesinfieldsfrompublicsafetytoeconomicdevelopmentGovExhasaproventrackrecordofworkinginconcertwithpartnerstoprovidetechnicalassistanceandtrainingatscale
SinceourlaunchinApril2015GovExhas
Trainedover2000governmentemployeesinpersonandonlineintopicsincludingdatamanagementperformancemanagementopendatacommunityengagementandanalytics
Providedtechnicalassistancetomorethan100localgovernmentsacrosstheglobeimprovingtheiruseofdatatoachieveefficienciesinservicedeliveryandbetteroutcomesforresidents
Assessedthecapacityofmorethan130governmentalorganizationstodeterminehowleadersstaffandexternalstakeholdersmanageandusedata
Localgovernmentsareincreasinglyonthefrontlinesofaddressinglargescaleproblemswithglobalimplicationssuchasclimatechangepublicsafetyandeconomicinequality
Whileevidence-basedinterventionstohelpsolvethoseproblemsareavailablelocalgovernmentstaffhavelimitedtimeandcapacitytofindandimplementresearch-basedsolutionsAsaresultcitiescontinuetoworkthewaytheyalwayshave--makingincrementalimprovementstoprogramsbutnotalwayscreatingchangeatasystem-level
ThroughresourcedevelopmentandcollaborationwithJHUfacultyGovExissupportingcityleadersinusingdataandevidencetomakedecisionsandcreatinglastingrelationshipswithresearchers
TheRoadmapwasdevelopedbyGovExaspartofitspartnershipwithBloombergPhilanthropiesWhatWorksCitiesinitiative
AboutGovEx
69
- Title page
- Introduction and purpose of the Roadmap
- Using the Roadmap
-
- Gather information about the problem
- Strategically engage key partners
- Review the evidence and collect details
- Decide on a policy to implement based on local factors
- Establish a framework to monitor progress
- Assess data accessibility quality and security
- Design implementation and evaluation
- Share your story and success
-
- Working with a range of evidence
- Thank you
- References
- Resources
- About GovEx
-
Results-focused
Time-bound
ThegoalstatementshouldincludeyouroutcomeindicatorlikeChattanoogasgoalldquoWewillendchronicveteranhomelessnessbyDecember2016rdquoAdiagramtohepyoubuildaSMARTgoalaroundyouroutcomeindicatorisbelow
Component Description Yourturn
Specific WhowillbebetteroffWhatactionthecitywilltake
Measurable Howwillyoumeasuresuccess
Achievable Isthegoalattainable
Results-focused Whatistheexpectedresult
Time-bound When
Setrealisticambitioustargetsandtimelinesforeachindicator
ToestablisharealisticandambitioustargetforyourgoalstatementusetheadministrativedatacollectedforyourchosenKPItoreviewactualCityperformanceinthemostrecentyearcomparedwiththethreeorfourmostrecentyearsworthofdataBegintocalculatesummarystatistics(egtotalminimumaveragemedianmaximum)toidentifytrendsCreatecumulativeandmonthlychartstoidentifytrendsandanomaliesBasedonyouranalysisofhistoricaldataignoretheanomaliesandassumetheproductivitytrendswillfollowthesameoverallpatternfrommonth-to-monthFinallyapplyanincreasedproductivityfactortotheperformanceeachmonthtoforecastfutureprogressBasedonthisanalysischooseatargetthatinspiresimprovementwhileaccountingforreality
Manycitiesthataremorematureintheirperformancemanagementpracticesrecognizetheimportanceoftakingacommunity-levelapproachtoimprovingoutcomesforresidentsThesecitiesarecreatinggoaltaxonomiesthatincludemeasuresrelatedtopublicsafetyhousingandevenrecreationCitiesdriveprogressonthesegoalsbyimplementingstrategiesthatengagemultipledepartmentsandcollaboratewithserviceprovidersinthecommunity
EventuallyyouwilldevelopgoalstatementsfortheoutputsinordertocapturetheefforttoachievetheoutcomecontainedinyourgoalstatementYoursetofindicatorsshouldbecomposedofshort-andinterim-termsupportivemeasureswithaccuratetargetsandrelevanttimelinesAnexamplegoaltaxonomyfromtheCityofLasVegassResultsVegasprogramisbelowUsethistaxonomyasamodelforyourownsuiteofindicatorsandgoalstatements
Establishaframeworktomonitorprogress
36
Yourturn
Thetemplatebelowismeanttohelpyoucraftagoalstatementforeachsupportingmeasure
Goalstatement
Departments
Outcome
Activities
KPI
Target
Timeframe
Conductanddiscussrelevantanalysis
ThrougharoutineseriesofmeetingscontinuallymonitorprogressofKPIssupportingindicatorsandanyotherdatacollectedsuchasqualitativedatatoidentifytrendsthatwillindicatewhetherthechosenpolicyishavingthedesiredimpactThesedatapointsshoulddrivethediscussionduringthesemeetingspointingtoopportunitiesforfurtherinquiryinto
Establishaframeworktomonitorprogress
37
whatisdrivingtheobservedtrendsAnalystsshouldbeencouragedandtrainedtoaskquestionsofthedepartmentsproducingthedataandconductsitevisitswhenpossibletodeepentheirunderstandingofthedataPresentationoftheanalysisshouldfocusononeideaatatimetokeeptheconversationfocusedonkeypointsandspecificopportunitiesforimprovementYoumayconsiderdevelopingapublicfacingdashboardtosharethisprogressmorebroadly
AgeneralcomparisonofthestatusquodatawithwhatwasobservedduringthepilotcanprovidedecisionmakerswithaquickunderstandingofthedifferencesbetweentheoldandnewapproachesalthoughthiswillnotshowcausalityOftenthesecomparisonswillincludeoperationalmetricssuchasdaysordollarssavedbutrememberthattheimpactonpeoplewillbemostimportantIfyouareworkingwithanevaluationpartnertheywillsupportthisstep
Usethespacebelowtosummarizetheresultsofyouranalysis
MeasureTargetvalue(seesectiononsettingtargetsabove)
Pilotvalue(whatdidyouobserveduringthepilot)
Historicalaverage(ifavailable)
Results(WhoisbetteroffIsanyonenegativelyimpacted1-2sentencesreassessingyourgoalstatement)
DocumentyourprocessandprogressincludingfeedbackfromtheparticipantsimplementationpartnersandtheprojectteamDidanythingunexpectedoccurduringthepilotArethereinternalandexternalfactorsthatmayaffectyouroutcomesThisiswheretheimplementationoutcomescanbecanhelpfulIftheinterventiondidnotgoasplannedconsiderwhetheritisbecausetheinterventionwasnotacceptableorfeasible
UsethetemplatebelowcreatedbyDenversPeakAcademytocapturethisinformationIfthe306090daytimelinedoesnotmakesenseforyourcityfeelfreetocustomizeit
CurrentState
FutureState
30Day
60Day
90day
Whatwentwell(+)
Whatdidnotgowellshouldbechanged(-)
IfaquestionordiscussionpointarisesthatcannotbeansweredinthemeetinganassignedstaffmembershouldcapturethequestionduringthemeetingAfterwardthefollowupitemsshouldbesenttorelevantpartnersandthenextmeetingshouldbeginwithadiscussionof
Establishaframeworktomonitorprogress
38
thoseitems
Takeaction
DiscussionsaboutwhetheryourindicatorsaremovingintherightdirectionandwhatmightbedrivingtrendsmayleadtoadiscoverythattherearenotenoughresourcesallocatedtothispolicyorissueTheremaybelimitedfundingforstaffprogrammingortechnologythatisdrivingthedatainthewrongdirectionIfCityleadershipisintheroomthisisagreattimetomakethecaseforadditionalfundingduringthebudgetseasonBycollectinganalyzinganddiscussingdataonthispolicyroutinelyyouarebuildingevidencetomakethecasefortherightleveloffundingtoreachtheestablishedtargets
Establishaframeworktomonitorprogress
39
Assessdataaccessibilityqualityandsecurity
Determineaccessibilityandqualityofdata
OnceyouhavecompiledyourwishlistofindicatorsitistimetodeterminetheaccessibilityandqualityofthedataTopreventcomplicationyoumaywishtoprioritizerelevantindicatorsforwhichyouarealreadycollectingdataAvoidthetemptationofadoptingatechnologysolutiontocollectnewdataatthispointsuchasaphotosurveytoolorotherappFornowyouaretryingtodeterminewhichdatasetsarerelevantaccessibleandhighqualityBesuretoworkwithyourkeyinternalpartnerswhileyouareidentifyingyourmeasuresDepartmentstaffwhoworkonprogramsarefamiliarwiththeindicatorsthatwillaccuratelysupporttheoutcomesanddatabaseadministratorswillbevaluablewhendiscussingaccessibilityandqualityofthedata
Abrieflistofconsiderationsfortheaccessibilityofyourdataisbelow
ArewecurrentlycollectingthisdatasetorwillwebecollectingitforthefirsttimeHowfarbackdoestherecordgo(timehorizon)WouldweneednewtechnologytocollectoranalyzethedataCanthedatabecollectedmanuallyorcanitbeautomatedHowfrequentlyisthedataproduced(continuouslyhourlydailyweeklymonthlyquarterlyyearlyetc)IsthedataavailablefortheentireCitypopulationorasubsetofthepopulationHowgranularisthedata(individuallevelCensusblockneighborhoodcitywideetc)DoesthedatacontainsensitiveinformationIsthereaproxymeasurethatwecanusefornowIsthedatamachinereadable
AsnotedbytheWilsonSheehanLabforEconomicOpportunities(LEO)attheUniversityofNotreDameleveragingadministrativedatacansignificantlyreducethecostsandtimelinesofrigorousevaluationCitiescanestablishaninternaldatasharingagreementacrossdepartmentsincludinglanguagethatenablesadministrativedatatobeusedforresearchpurposesCitiescanalsoidentifyexternaladministrativedata-holderssuchasacountyschooldistrictpolicedepartmenthospitaloranotherstatethatmayhaveoutcomedataimportantformeasuringprogramimpactCitiescanleverageexistingrelationshipsandorworkwithresearchinstitutionstodevelopstreamlineddatasharingprocessesandagreementswiththeseexternalentities
Assessdataaccessibilityqualityandsecurity
40
Evaluatethequalityofyourdata
QualityisasimportantasaccessibilityToevaluatedataqualityyoumaywishtocreatearubricliketheonebelow
Value ValueDescription
3 ExcellentDataQualityThedataiscompleteaccurateandupdatedregularly
2 AdequateDataQualityThedataissomewhatcompleteaccurateandupdatedfairlyregularly
1 PoorDataQualityThedataisnotcompleteaccurateorupdatedregularly
ThisisagreattimetobuildonthedatainventorythatyoustartedinthegatherinformationpageSuggestedadditionalfieldstoaddtoyourdatainventorytemplatearebelow
Program DatasetTitle
DatasetDescription DataSource
Departmentorexternalpartner
DoestheCityalreadycollectthisdata
Whatisthenameoftheprogram
Human-readablenameofthedataset
Shouldbeunderstandablebynon-technicalusers
Whatistheoriginalsystemapplicationfilethathousesit
Departmentorexternalpartnerthatmaintainsthedata
Isthisdataalreadycollected
Updatefrequency-ThefileformatforthedatasetTypicallycsvxlsxshapefiletxtetc
Format-Howoftenthedatasetisupdated
Timerange-Theperiodoftimecoveredinthisparticulardataset
Datause-Howisthisdatasetcommonlyused
Datausers-Whotypicallyusesthisdataset
Dataownerstitle-Titleofthedataowner
Dataownersposition-DataOwnersjobtitle
Dataownersemailaddress-DataOwnersemailaddressWhomanagesthedataandorisresponsibleforgrantingpermissiontoaccessthedataWhounderstandswhatthedatasetincludesandcananswerquestionsaboutit
Dataqualityconcerns-Describetheconcernsyoumayhaveaboutthequalityofthedataset
Assessdataaccessibilityqualityandsecurity
41
Division-DivisionwherethedataismaintainedIfmorethanonedivisionisresponsiblelisttheprimaryownerofthedataset
Sensitivedatacomments-DescribetheissuesthatwouldpreventthecityfromsharingthedatapubliclyCommonsensitivityconcernsincludeprivacyviolationssecurityissuesorhighcostorstaffdemandstopostthedata
Publishingstatus-Thistheplacetoputanycommentsregardingthecurrentstatusofthedataset
Publishedlink-Thisisthelinktothepublisheddatasetontheopendataportalorcitywebsite
Reviewdatasecurityandprivacyprotectionprotocols
OftenapolicychangealsoinvolvesachangeinthedatathatisbeingcollectedhowthatdataisbeinghandledwhohasaccesstoitandahostofotheradjustmentsAsyoutakestockofthedataassetsthatareavailableforyourinitiativeitisvitaltoalsothinkthroughwhatwillbedonetoprotectthatdataandtheprivacyofthepeopleitrepresentsAbigpartofgooddatasecurityinvolvesstrongdatagovernancepracticesthatdefineandidentifysensitivedataandconsciouslyworktobalancethevalueandriskofstoringsuchsensitivedataOnewayofevaluatingyourdatacomesfromcaselawrelatedtotheUSFreedomofInformationActwhichwehavetranslatedtothechecklistbelow
1 Woulddisclosureresultinasubstantialinvasionofprivacy2 Haveweconsideredtheextentorvalueofthepublicinterestandthepurposeorobject
oftheindividualsseekingdisclosure3 Istheinformationavailablefromothersources4 Wastheinformationgivenwithanexpectationofconfidentiality5 Isitpossibletomouldreliefsoastolimittheinvasionofindividualprivacy
TheseriskscouldbemitigatedusingavarietyofstrategiesfromaggregatingdatatoobfuscatingindividualstoredactingfieldsthatmayincludePersonallyIdentifiableInformation(PII)orre-evaluatingyourdatacollectionmethodstofindalternativewaysofgettingtheinformationyouneedThesestepsshouldbecodifiedandmanagedbyadatagovernancecommitteeifpossibleForthesensitivedatathatisdeemedessentialcreateaplanaroundpotentialdatabreachesinordertoquicklyandconfidentlyrespondtoemergencysituations
ProactivelypreparingforthesesituationsisoftenthebeststrategyfordatasecuritybutdonotforgetthetechnicalsideInvolveyourITadministratorsinconversationsaboutdatastorageandaccessensuringthattheywillbeprotectedbyenterprise-gradecybersecuritysystems
Assessdataaccessibilityqualityandsecurity
42
Assessdataaccessibilityqualityandsecurity
43
Designimplementationandevaluation
Designtheimplementation
NowthatyouhaveyouraccountabilityplanandkeyindicatorsitistimetogetspecificaboutyourplanforimplementationThetemplatebelowwillhelpyouthinkthroughthespecificmechanismsofimplementingpolicychange
UnderstandingorganizationsandsystemsiscriticaltounderstandingandmakingdecisionsaboutwhereandhowinterventionsshouldbeappliedApolicyimplementedinasinglesettingmayshowpromisingresultsbutimplementingmultiplepoliciesandprogramsatthesametimewithdifferentactorsresponsibleforeachaspectcanhelpensuresustainablebehaviorchange
InSomervilleMAShapeUpSomerville(SUS)isacitywidepublichealthprogramthatisimplementedacrossmultiplelevelsofinfluenceandsystemsthroughinterventionsbeforeduringandafterschooltoincludechangestobuiltenvironmentsuchasbikelanesregionalmasstransportationandphysicaleducationequipmentinschoolsandgymstoincreasephysicalactivitywellnesspoliciesandschoolgardensnutritioneducationduringandafterschoolandhealthyrestaurantprogramstosupporthealthyfoodandhealthyeatinghabits
Usethetemplatebelowtoplanyourimplementation
Policyimplementationplanningtemplate
ListthespecificdesiredoutcomesInthepreviousstepyouworkedwithdepartmentstafftoestablishspecificmeasurableoutcomesandinterimindicatorsPrioritizetheindicatorsthatwilldemonstrateprogressUsethespacebelowtocapturethedesiredoutcomesofthepolicychange
ExampleShapeupSomervillebeganwiththegoalofpreventingobesityin1stthrough3rdgraders
Yourturn
____________________________________________________________________________
____________________________________________________________________________
Designimplementationandevaluation
44
SpecifythetargetpopulationAreyoudesigninganinterventionthattargetsonlycertainsitesAreyouplanningonimplementingaprogramorinitiativeforaspecifictargetaudienceClearlyspecifythepopulationyouplantotarget
IfyouplantoconductarigorousimpactevaluationasnotedbytheWilsonSheehanLabforEconomicOpportunitiesitisimportanttoestablishconcreteobjectiveeligibilitycriteriaandensurethatthesecriteriaareappliedconsistentlybyallstaffduringscreeningandintakeItisalsovitaltoidentifytheindividualswhoareeligibleforaparticularprogramorserviceevenifeveryoneeligiblecannotbeservedbyexistingresourcesThishelpscitiesidentifyanyunmetneedsandguidefutureresourceallocationwhilealsocreatingthepossibilityofarigorousimpactevaluationbyidentifyingapotentialcomparisongroupFinallywhenconductingscreeningandintakeitisimperativetocollectandmaintainbasicinformation(egnamedateofbirthgender)aboutallprogram-eligibleindividualsnotjustthosewhoparticipateintheprogram
ExampleIntheShapeUpSomervilleexampletheprogrambeganbytargeting1stthrough3rdgraders
Yourturn
____________________________________________________________________________
____________________________________________________________________________
DefinethelocationWherewillthepolicychangetakeplaceWillimplementationtakeplaceatonesiteormultiplesites
ExampleShapeUpSomervilleinterventionsweredeliveredinschools
Yourturn
____________________________________________________________________________
____________________________________________________________________________
ClarifyimplementationpartnersandarticulatetheirrolesWhatarethekeyorganizationsinstitutionsorpeopleresponsibleforimplementationTheprojectteamshouldbeasclearandspecificaspossibleinprovidingadescriptionoftheconcreteactivitiesthatimplementationpartnersaremeanttocarryoutPracticeprofilesmaybeahelpfultooltohelpdefinewhatisrequiredoftheactorswhoareresponsibleforimplementingtheintervention
Designimplementationandevaluation
45
ExampleInSomervillemayoralleadershipandengagementofmultiplecityagencies(publichealthprimarycareschoolsCambridgeHealthAllianceetc)streamlinedandconnectedthevariousShapeUpSomervilleinitiativesandgrantswhichhelpedtokeepthetalentandenergyfocusedonhealthyeatingandactivelivingforallsectorsofSomervilleSUSpartnersultimatelyincludedschoolsnonprofitsCityandstateagenciesandnationalfoundations
Yourturn
____________________________________________________________________________
____________________________________________________________________________
DescribetheinterventionWhatarethestepsactivitiesorprocessesRememberthatthemechanismthespecificstepsorprocessthatleadtobehaviorchangeiskeytounderstandingtheeffectivenessofthepolicychangeandsharingtheresultswithothers
ExampleGoalsofShapeUpSomervilleweretoinfluenceeverypartofanearlyelementaryschoolchildrsquosdayndashbeforeduringandafter-schooltopromotehealthyeatingphysicalactivityandtheirweightstatusInterventionincludedfoodserviceenhancementstoimprovebreakfastandlunchandstaffprofessionaldevelopmentwalktoschoolactivitiesSUSclassroomcurriculumandprofessionaldevelopmentschoolwellnesspolicydevelopmentSUSafter-schoolcurriculumandprofessionaldevelopmentoutreachandeducationtohomethroughmaterialsforumseventsldquoSUSapprovedrdquorestaurantscommunityoutreachandcapacitybuildingthroughpolicydevelopmenttrainingsmediaplacements
Yourturn
____________________________________________________________________________
____________________________________________________________________________
ClarifytimelineandmilestonesfordeliveryandevaluationHowfrequentlywilltheservicesinterventionbedeliveredHowlongwilltheprogramrunWhatisthetimetoexpectedimpacts(eg12monthsafterprogramcompletion)
Designimplementationandevaluation
46
ExampleShapeUpSomervilleledbyTuftsUniversityFriedmanSchoolofNutritionScienceandPolicyconductedaquasi-experimental(nonrandomized)interventionstudyamong1stthrough3rdgradestudentsinSomerville(intervention)andtwocomparisoncommunitiesfor9-monthsduringoneschoolyear(fall2002ndashspring2003)ThestudyshowedpositiveassociationoftheinterventiononchangeinBMISUSreleasedreportssummarizingevaluationsprogresstodateplansforadditionalprojectsandimpactstenandfifteenyearsintotheinitiative
Yourturn
____________________________________________________________________________
____________________________________________________________________________
PlanyouranalysisTheprevioussectionaddressedestablishinganaccountabilityframeworktomonitorprogressandimplementationDeterminewhichindicatorswillbediscussthroughthatprocesswhowillsharethedataandanalysisandhowfeedbackfromthoseconversationswillbeincorporatedbytheprojectteam
ExampleTheMayorrsquosofficeimplementedadata-drivendecisionmakingapproachcalledSomerStatwhichwassynergisticwiththeapproachestakenbytheresearchandevaluationteamsdeployedforthespecificgrants
Yourturn
____________________________________________________________________________
____________________________________________________________________________
PlanforsustainabilityIfthepolicyandprogramchangeissuccessfulhowwilltheprojectteamsustainandexpandtoadditionalsitesorpopulations
ExampleAlthoughmostofthepartnershipsforgedintheearlyyearsweremaintainednewgrantsandcollaborationsexpandedattentiontoolderpublicschoolstudentsactivetransportationandtheGreenLinesubwayextensionthebuiltenvironmentfarmerrsquosmarketeffortssuchasldquoGrowninSomervillerdquowithUnionMainStreeteconomicdevelopmentandhealthdisparitiesaswellasdatacollectionmonitoringandsurveillance
Yourturn
Designimplementationandevaluation
47
____________________________________________________________________________
____________________________________________________________________________
Pilotplanningtemplate
IfyouplantoconductapilotfirstdeterminethepurposeofthepilotDoyouwanttoobservewhetherthepolicychangeresultsinthedesiredbehaviorchangeAreyouconcernedaboutfeasibilityscalabilityorcostDoyouwanttotestwhetheryourdesiredresultscanbeachievedthroughthispolicyapproachClarifythegoalofthepilotandthequestionsyouhopetoanswerThiswillhelpyouidentifythekeyindicatorsthatyouwillusetomeasurewhetherthepilotissuccessfulBeforebeginningestablishabaselineofcurrentperformanceandsetyourtargetsaccordinglybasedonwhatyouhopetoaccomplishthroughthepilot
NextestablishthepilotplanforyourprojectteamincludingmilestonesforteamcheckinstodeterminewhethertomoveaheadorchangecourseThisisseparatefromthemeetingsthatwilloccuraspartoftheaccountabilityframeworkUsethespacebelowtocreateyourtimelineanddevelopyourmeetingschedule
Task Personresponsible Startdate Checkindate Deliverable
Planyourevaluation
WhileyouaredesigningtheimplementationitisalsotimetobegintodevelopyourplanforanalysisAsnotedbytheWilsonSheehanLabforEconomicOpportunitiesdecisionsaboutprogramsetup-includingoutreachscreeningintakedatatrackingandenrollment-candeterminewhetherornotarigorousimpactevaluationwillbepossibleAsdiscussedcityleadersmayfindithelpfultoincludeevaluationpartnersinearlyconversationsaboutprogramsetupDuringpreviousstagesyoumayhaveworkedwitharesearchertoplanyourproposedpolicychangeYoushouldbeclearaboutthepurposeoftheevaluationanddevelopanevaluationplanBelowareadditionalconsiderations
Causationvstrendanalysis
IntheEstablishaframeworktomonitorprogressstagetherewasguidanceontrackingtrendsovertimeItisimportanttonoteherethatmonitoringdataisdifferentfromanalyzingacausaleffectTrendanalysiswillillustratethatchangesareoccurringandmayleadtoquestionsthatcouldbeansweredbyyourevaluationbuttrendanalysisalonedoesnotprovideevidenceaboutwhatisdrivingthosechangesYoumayseechangesinyour
Designimplementationandevaluation
48
outcomeindicatorsbasedonvariablesoutsideoftheCityscontrolForinstanceachangeinfederaltradepolicycanhaveanimpactemploymentratesinyourCityoutsideofyourCitysnewjobtrainingandplacementprogram
ToaddresscausalityyoumustcomparetheoutcomesofindividualswhoreceivedtheinterventionwiththosewhodidnotWithoutacomparisongroupofsimilarindividualswhowerenotexposedtotheinterventionitisimpossibletoknowwhichfactorsareresponsiblefortheoutcomesRandomizedControlTrials(RCTs)inwhichindividualsarerandomlyassignedtotreatmentandcontrolgroupsarethegoldstandardinprovingcausalityIfyoucandemonstratethatoutcomesimprovedforthosewhoreceivedtheinterventionanddidnotimproveatallorimprovedlessforacomparisongroupyoucanbesomewhatcertainthattheinterventionwasthecause
Examplesofothernon-causalimpactevaluationsthatarerigorousandreliableincludecertainquasi-experimentalstudiesusingcomparisongroupdesignssmartinterruptedtimeseriesanalysesthatmodeltrendsovertimeinwaysthatallowestimationofinterventioneffectswelldesignedpre-posttestanalysesneedsassessmentsimplementationstudiesetcFormoreinformationondifferenttypesofevaluationsseetheSageResearchMethodswebsite
Lookfordifferencesingroups
BuildintimetoalsoexaminethepilotdatatoseeifgroupsareexperiencingdifferentresultsReviewingperformancebasedonoutcomemeasuresforapopulationinaggregatemayhidethesubtlewaysthatsomepeoplemaybeaffecteddifferentlysoitisnecessarytocarefullyanalyzethedatafordisparateoutcomesForinstancetheMovingtoOpportunity(MTO)studythatexaminedtheimpactofhousingvouchersonfamilieswholivedinhigh-povertyneighborhoodsfoundthatchildrenwhomovedbeforetheageof13hadbettereducationandhigherincomesasadultsHoweveradultsdidnotseeimprovededucationandemploymentoutcomes
Disaggregatingdataintomorespecificcategoriesofraceandethnicitycanhelpunderstandtrendsanddisparitiesinamoregranularwaywhichcanhelpdecisionmakerstargetpoliciesandinterventionsinamoreequitableway
EarlierintheRoadmapweworkedtoidentifystakeholdersandwhomightbepositivelyandnegativelyimpactedbyachangeinpolicyUsetheresultsofthatexercisetoinvolvethecommunityininterpretingdataanalysisandtogatherqualitativefeedbackfromthecommunitytocallattentiontoissuesForinstanceanincreaseinyouthdetentionmaybeexplainedbyinefficienciesorlackofaccesstotransportation
Designimplementationandevaluation
49
Designimplementationandevaluation
50
Shareyourstoryandsuccess
Sharetheresults
Storytelling
AnintegralstepinyourstrategyissharingtheresultsofyourpilotorpolicychangewithyourcommunityThisincludesdevelopingmessagingaroundtheworkofyourorganizationandcreatingaplantodisseminateyourstoriesWhytellstoriesaboutyourwork
Informandorunderstandasubjectorsituation
Giveneededcontextofdataorbitsofinformation
Relatablestoriesarememorableforreadersviewers
ReadersorviewerswillseetheinformationasmorecompellingwhenthereisademonstratedandrelatablechangeorimpactonthemselvesortheirfriendsneighborsandcolleaguesStoriesareaboutpeopleandgovernmentisfundamentallyabouthelpingpeopleSotellthestoriesofthepeopleyouandyourcitycolleagueshelped
Keepthesetipsinmindasyoudevelopyourstorywithyourteam
WhatwereyourfindingsandwhyshouldanyonecareTellyouraudiencehowtheproblemwassolvedWhathastheimpactbeenonresidentsandcitystafftodateWhatwasthesuccessandweretheremultiplesuccessesalongthewayIfyouhavepreliminaryresultsordatashowingimprovementtosharethisisthetimetodoit
WhowasinvolvedandimpactedReviewthelistofstakeholdersHowwereeachofthemimpactedbytheprogramHumanizeyourstoryandimpactbyaddingadditionalvoicesThesevoicescanbeoneofthefollowing
Apersonorfamilywhowaspositivelyaffectedbythechangesuccess
Aninvolvedthird-partydescribingboththeproblemandthesolution(ieanorganizationnonprofitprovidersbusinesstradeassociationsportsteametc)whoispartofthecommunitythatthegovernmentworkedwithonthechangesuccess
Someonefromthegovernmentitselfwhocanspeakaboutthechangesuccess
TellmoreaboutyourcityDescribeyourcityincludingpopulationdemographicsindustriesetcWhatareitschallengesWhatisitknownforWhatmakesituniqueHowisitsimilarordifferentfromothercitiesintheUS
Shareyourstoryandsuccess
51
DescribeyourcityrsquoschallengeWriteoutthechallengeyourcityisfacing(asdefinedinyourproblemstatement)HowdoesitaffectresidentsHowdoesthisproblemaffecttheworkofcitystaffmembers
AddthedetailsProvidedetailsonyourappliedresearchstrategyTalkaboutwhatyourcityhasdonenthepasttoresolvetheissueDiscusshowyourstrategyhaschangedwhenyoubegantoadoptthisapproachWasmoneysavedorinvested
Usethisstorymapexercisetoorganizeyourthoughtsandguaranteethatyouincludealloftheelementsofatraditionalstory
AfteryoudevelopyourmessagingcreateacommunicationsmarketingandoutreachstrategydesignedtospreadyourmessageYourstrategyshouldwalkyouthroughthefollowingkeyactivities
1 IdentifyandprofileyouraudienceWhencreatinganddisseminatingyourmessageitisalwaysgoodtothinkaboutyouraudienceWhoexactlywouldbeinterestedinwhatyouhavetosayThegeneralpopulationofthecityPeoplewhoworkinnonprofitsBusinessownersItmattersbecausenoteveryonewholivesinacitymaybethetargetoftheinformationBusinessownersmaybemoreinterestedindatashowingeffectsonthebusinessdistrictthanothersandsoonWewillcoverthislaterbutthisstepwillalsohelpyoutothinkofthemosteffectivewaytoreachyouraudience
2 Defineyourcalltoaction(ifyouhaveone)SpendsometimeconsideringwhyyouareinterestedintellingyourstoryDoyousimplywanttoupdatethepubliconyourprogressorcelebrateyoursuccessesAreyouinterestedinyouraudiencechangingitsbehaviororsupportingaspecificinitiativebeinglaunchedbyyourorganizationIfyouwanttohaveyouraudienceactorbehaveinacertainwaybespecificandincorporateitinyourmessaging
Shareyourstoryandsuccess
52
3 SelectperformancemeasuresThinkabouthowyouwoulddetermineifyourcampaignissuccessfulandselectappropriateperformancemeasuresYouwillrevisitthesemeasuresattheendofyourcampaigncommunicationsplanimplementationtodeterminewhetheryouneedtomodifyyourcommunicationsandoutreachstrategyforthefuture
4 MethodstocommunicateyourstoryTherearemanywaystocommunicatesuccessofyourprojectHerearesomesuggestionsonreachingaudiencesItisimportanttodeterminewhichmethodsarethemosteffectiveforyourdesiredaudiencetoreceiveinformationDoyourresearch(PewResearchCenterhasexcellentstudiesonthereachofmediadigitalmediaandmarketing)ForexampleifseniorsareyourtargetpopulationandtheyarelessinclinedtouseInstagram(thatistrue)youmaywanttoconsiderFacebookorthenightlylocalnewsbothofwhichareusedmoreoftenbyseniorsBelowisalistofsomemethodsandtoolsyoucanuse
i Pressreleasestolocalnewspapersandtvstations
ii ShortsmartphonevideosAgreatwaytoengageanaudienceisthroughvideoAskonequestionofsomeonewhoispartofthestoryMaybethereisasocialworkerwhocandiscusshungerinyourcityThesevideoscanbereusedonmultipleplatforms
iii Long-formwriting
iv TellingindividualstoriesthroughaseriesofphotosandshortnarrativesforpublicationonFacebookorInstagram
v Op-edsopinionpiecesplacedinthelocalpaper
vi Emailupdatestoyourlist
vii Connectingwithappropriateorganizationsthatareinvestedinthetopicandworkwiththecommunityandaskthemtodisseminateinformationtotheirmembers
viii EmployingldquoWordofMouthrdquo(WOM)tacticsbycreatingtownhallsorfreeeventsforresidentstoattendintheirownneighborhoods
5 DeterminethetypeofcollateralUsuallycampaignsthattargetresidentsrequireseveralavenuestoreachthemPeoplearenotalwaysinthesameplaceorspaceHavingakeenunderstandingofyouraudiencewillassistyouwithyourchoicesonwhatneedstobedevelopedSometimesthatmeansimagesforsocialmediaaonepageflierthatgivespeopleinformationonattendinganeventoranin-depthbrochure
Shareyourstoryandsuccess
53
6 ImplementyourstrategyKeepinmindstrategiescanbefluidSometimessomethingmaynotturnoutasexpectedevenifduringplanningsessionsitwasldquosupposedtordquoItisokYoucananalyzewhathappenedduringadebriefsessionandonceyoursurveysandothermeasurementshavebeenfinalized
Visualizations
Withtherightvisualizationyoucanreviewandshareinformationinsightsanalysisperformanceandtrendsallofwhichisvaluabletoacross-sectionofthepublicButwhichvisualizationisrightforyouTochoosetherighttypeofchartgraphdeterminethetypeofquestionyouwanttoanswerSomecommoncharttypesarebelowandmoreinformationcanbefoundhere
Exploreadditionalanalyticaltechniques
Duringthegoal-settingphaseyouestablishedoutputoutcomeandimplementationmeasuresBasedontheresultsofyourpilotyoumayneedtoadjustyourmeasuresWiththepilotdatainhandyoumayhavesomeideasofhowfutureanalysescouldbedifferentandpotentiallyimprovethequalityofyourprogramTakesometimetobrainstormotherrelevantdatasetsthatyoudidnothavethetimetopursueearlierincludingmessydepartmentaldatathatneedsmoreattentionorsourcesofpublicdatafromthestateandfederalgovernmentnonprofitsandevenprivatecompaniesthatcouldsupplementyourunderstandingofanissueUsingmachinelearningtechniquesyoumightbeabletobetter
Shareyourstoryandsuccess
54
predictdemandforhomelessservicesusingNOAAweatherforecastspreemptwatermainbreaksusingsoilstratigraphyorinvestineconomicdevelopmentbymonitoringrealestatelistingdata
Advancedanalyticaltechniquescanalsohelpyourimplementationrunmoreeffectivelybyintegratingdata-informedtargetinganddecisiontoolsintotheprocessByautomatingroutineworkflowsandsurfacingsubtletrendsthesetoolscanhelpyouiterateinamoreprecisemannerandprovidekeystrategicguidancetofront-lineemployeesthatwouldotherwisebeunavailableThesesortsofdata-science-enhancedprocessescanbeclassifiedasoneofsixprojecttypesasexploredfurtherbytheNewOrleansLANOLAlyticsteamFormoreguidanceonscopingasimilarprojecttheGovExadvancedanalyticsscopingguidegivesyouastep-by-stepoverviewofhowtobesuccessful
Usethespacebelowtolistyourideasforpossibleadvancedanalyticsprojectsbytype
Typeofproject Ideas
Findingtheneedleinthehaystack
Prioritizingforimpact
Earlywarningtools
Betterquickerdecisions
Optimizingresourceallocation
Experimentingforwhatworks
Keepthefeedbackloopgoing
GatherinformationonpublicsentimentaboutpolicychangethroughfocusgroupssurveysandpublicmeetingsledbyadvisorygroupmembersNotewhethernewinformationisavailablethatcanhelpanalyzeorcontextualizeyourprogress(egfederalorstatedatareports)Sharetheresultsofyourpilotandthepolicychangeprocesswithothercitiesandresearchers
Shareyourstoryandsuccess
55
WorkingwitharangeofevidencePresentlythereisalotofattentionandinvestmentinevidence-basedpolicymakingincludingthefederalbipartisanCommissiononEvidence-BasedPolicymakingthePewCharitableTrustsreportsandtoolkitsonHowStatesEngageinEvidence-BasedPolicymakingandtheLauraandJohnArnoldFoundationsupportedEvidence-BasedPolicymakingCollaborativeManyoftheseinitiativesfocusonrigorousevaluation--meaningrandomizedcontrolledtrials(RCT)orotherexperimentalmodels--asthebestkindofevidencetouseHoweverthisbodyofevidenceisofteninsufficientforthespecifickindsofquestionsthatgovernmentscanhavewhenmakingadecisionaboutpolicyInthisguideweproposeanapproachtosupportgovernmentsinincorporatingadditionaltypesofevidenceintotheirdecisionmaking
OnewaytounderstandarangeofevidenceistouseNestarsquosStandardsofEvidencetoframeanddefineevidenceTheBritishnonprofitNestaworkstoinjectpositivechangeintothewayUKgovernmentpoliciesandprogramsfunctionOneoftheirkeyareasofworkhasbeendevelopingWhatWorksCentresorganizationsthatcollectcommissionsynthesizeanddisseminateevidenceaboutinterventioneffectivenessacrossarangeofsocialimpactareasNestadevelopedthefollowingStandardsofEvidencetoprovideausefulwaytodocumenthowdifferentkindsofinformationdemonstratedifferentlevelsofevidence
Workingwitharangeofevidence
56
Theimportanceofpotentiallybiasedinformation
Onechallengepresentedbyanevidenceclassificationthatacknowledgesthevalueofnon-scientificformsofevidenceistofigureouthowtobalanceallofthepiecestogetherwhenyouusethemTounderstandhowtodothiswehavetothinkabouthowtopositionpotentiallybiasedinformationindecisionmaking
WhenwelookatinformationwhichhasnotbeenrigorouslyevaluatedweacceptthatthisinformationmightbebiasedThewholereasonweconductrigorousevaluationistoremovebiasfromourobservationsWeconductstructuredtrialstoproduceresultsthatareconsistentandgeneralizableIfafindingisunbiasedthisaccuracymdashthislackofbiasmdashcanbeprovedbecauseanyindependenttestercanreproduceitwhentheyfollowanidenticalprocess
YoucanlookatpotentiallybiasedinformationasjustinformationthatisearlieroninitspathtorefinementreplicationandverifiedaccuracyApieceofinformationbasedonanyoneindividualrsquosobservationsislikelytobebiasedbecauseitiscomingfromjustonesourceBiasisreducedbygatheringtogetherlotsofindependentsourcesofinformationandfiguringoutitsmostcommonattributes
MaybethebiasedinformationmakesaclaimthatistoolargemdashitsayssomethingistrueunderallconditionswhenitisreallyonlytrueundercertainconditionsorthataneffectofaninterventionisbiggerthanitstrueaverageorthataneffectlastsovertimewhenitdoesnotPerhapschangingtheclaimtobemorespecificmdashcriticallyreviewingtheinformationtoseeifspecificconditionsmustbemettoachievetheoutcomesmdashwouldmaketheclaimmoreaccurate
IfyoutestedyourclaimfounditdidnotholdupinsomecaseschangedittesteditagainuntilyourclaimrepeatedlyheldupthenyoucoulddemonstratethattheclaimwasnolongerbiasedbutaccurateThisprocessoftestingimprovingtestingandvalidatingiswhatwegetfromstructuredscientificstudy
Withindividualpiecesofinformation--crediblehypothesescasestudiesexpertopinion--youcannotassumethatkindofaccuracyAtthesametimetheseearlyobservationsarefoundationaltoanyreplicablescientificobservationsWewillnevergetanywhereifwedonotstartwiththesepiecesofinformationeveniftheymightbebiased
Combininglevelsofevidencetogetthemostdetailedinformationforyourcity
ItisespeciallyimportanttoacceptdifferentformsofevidencewhenyoufacequestionsthathavenotyetbeentestedbyindependentstudiesSomeofthemostimportantquestionsrelatetothingsthatareveryspecifictoyoursituationYoufaceaparticularcombinationof
Workingwitharangeofevidence
57
issuesactorstrendsandresourcesthatisultimatelydifferentfromeveryotherplaceintheworld
ThisisimportantbecauseaspolicypractitionersandnotscientistsyoucaremoreaboutyourspecificcasethanahypotheticalaveragegeneralcaseYoucarelessaboutthetotallikelyrangeofimpactsfromapolicythanhowthissingleimplementationwillworkYouwantinformationaboutpoliciesthatwillprovideyouwiththebestpossiblepredictionabouthowtheywillworkinyourcontext
ByputtingdifferentkindsofevidenceintothesameframeworkNestarsquosStandardsofEvidenceimplythatwhenyoucannotgetrigorouslytestedexperimentallyverifiedinformationitisappropriatetoworkwithwhatwedohaveThisisrelevantforthemunicipalimplementationcontextbecausenostudypulledfromaliteraturereviewwillaccountforallofthevariablesyoufaceinyourowncontext
Intheabsenceofhavingastudythatpredictshowaninterventionworksinyourspecificcontextyouwillneednon-scientificinformationtotrytofillinthosegapsYouwillneedtogatherstoriesfromresidentsstaffmembersandotherstakeholdersthatsitonLevels1or2ofthestandardsofevidence
Someofthekindsofinformationyouwillprobablygetmoreeasilyfrompeoplethanstudiesinclude
howpeopleincitieslikeyoursreactedtoaninterventioninpublicforumshowpeoplewhoimplementedaninterventioninacitylikeyoursfiguredouthowtogetitpassedhowthecitycommunicatedpubliclyabouttheapparentbenefitsanddrawbacksoftheinterventionthepoliticalacceptabilityoftheintervention
Youcanremainsimultaneouslyawarethattheinformationyouaregatheringthroughindividualstoriesisbiased--notgeneralizableinformedbyspecificperspectives--whilealsorecognizingthathavingsomeinformationaboutsomethingimportantisbetterthannoinformationEvidencefromthefirsttwoevidencelevelsareessentialbuildingblocksofknowledgeandperfectlyusefulwhenyoucannotgetdetailedunbiasedknowledge
Youhavetostartsomewhereandyoushouldalwaysstartwithevidenceyouhaveaboutyourveryspecificconditionsthatyouknowareimportantforsuccess
Workingwitharangeofevidence
58
Thankyou
Interviewswithcities
GovExheldsemi-structuredinterviewswithsixstaffmembersinfourcitiestoestablishtheinitialstepsoftheRoadmapWeaskedrespondentstotalkaboutaninitiativeorpolicythattheyhavebeenresponsibleforimplementingintheircityRespondentsfromthreecitiesspokeabouttheireffortstoaddresshomelessnessandrespondentsfromonecityspokeaboutblightAfterrespondentsidentifiedtheproblemtheyspokeabouthowthecityprioritizedthisissuethevariousstepsinvolvedandbarrierspresent
GovExconductedtextanalysisoftheseinterviewswhichrevealedthatpeoplewasthemostcommonlyusedtermduringtheseconversationswhendiscussingthestepstochangeaprogramorpolicyThisisathemethroughoutthisRoadmapThefirstsectionbeginswithidentifyingtherelevantpeopleTheconversationsaroundevidencearefocusedonhumanbehaviorratherthanmethodologyorstrengthofevidence
Thankyou
59
OncewehadadraftoftheRoadmapwefollowedupwithcitiesonanindividualbasistolearnmoreaboutwhatwouldbehelpfulWearegratefultothefollowingcitystafffortheinterviewsandindividualconversations
MichaelBaskinChiefPolicyOfficerChattanoogaTN
MelissaBridgesPerformanceandInnovationCoordinatorLittleRockAR
VictoriaCarreoacutenAdministrativeOfficerLasVegasNV
TyroneGrandstrandCommunityDevelopmentPlannerFargoND
TibLaughlinDirectorofGeneralServicesKansasCityKS
TimMorelandDirectorofPerformanceManagementandOpenDataChattanoogaTN
CharlesRosendahlManagementAnalystDentonTX
Thankyou
60
MelissaSiebenAssistantCountyAdministratorKansasCityKS
GinaThompsonDirectorofStrategicServicesLewisvilleTX
JHUFacultycollaboration
GovExalsoconvenedthecitycontactsandfacultypartnersinaseriesofAdvisoryGroupmeetingsThesemeetingsservedtoconnectcitiesandfacultytounderstandoneanothersperspective
StefanBaralBloombergSchoolofPublicHealth
AshleyBernerJHUSchoolofEducation
MelissaDavey-RothwellBloombergSchoolofPublicHealth
JonLinksBloombergSchoolofPublicHealth
TamarMendelsonBloombergSchoolofPublicHealth
KristinMmariBloombergSchoolofPublicHealth
KeshiaPollackPorterBloombergSchoolofPublicHealth
JoshSharfsteinBloombergSchoolofPublicHealth
LizStuartBloombergSchoolofPublicHealth
LindsayThompsonCareyBusinessSchool
DanielWebsterBloombergSchoolofPublicHealth
Partnersindataandevidence-basedpolicy
Wealsospokewithpartnerswhoareexpertsinthefieldofevidence-basedpolicywhosharedtheirknowledgeandresourcesofcurrentandfuturethinkinginevidence-informedthinking
WendyBarrenoAssociateDirectorWilsonSheehanLabforEconomicOpportunitiesUniversityofNotreDame
TiffanyDavenportDirectorofDataPartnershipsBaltimoresPromise
TomiHiersDirectorBaltimoreCivicSiteAnnieECaseyFoundation
ChrisKingsleySeniorAssociateAnnieECaseyFoundation
KathrynOliverAssociateProfessorLondonSchoolofHygieneandTropicalMedicine
Thankyou
61
KristenWatkinsSeniorPolicyAssociateJ-PAL
GovExParticipants
GovExAnalystMiriamMcKinneyanalyzedtheinterviewtranscriptsandconductedtextanalysisSeniorImplementationAdvisorEmilyShawwillintegrateRoadmapactivitieswithadditionalservicedeliveryactivitiesatGovEx
Thankyou
62
ReferencesAbrahamKGampHaskinsR(2017)Thepromiseofevidence-basedpolicymakingReportofthecommissiononevidence-basedpolicymakingWashingtonDCCommissiononEvidence-basedPolicymakingBigmove2Extendingthegreen(nd)Retrievedfromhttpswwwcityoflewisvillecomdoing-businesslewisville-2025big-move-2-extending-the-greenBatesMAampGlennersterR(2017)ThegeneralizabilitypuzzleRetrievedfromhttpsssirorgarticlesentrythe_generalizability_puzzleBuildwithnotfor(2018January)RetrievedfromhttpsbuildwithnotforwordpresscomCairneyPampOliverK(2017)Evidence-basedpolicymakingisnotlikeevidence-basedmedicinesohowfarshouldyougotobridgethedividebetweenevidenceandpolicyHealthResearchPolicyandSystemsdoi101186s12961-017-0192-xCairneyPOliverKampWellsteadA(2016)TobridgethedividebetweenevidenceandpolicyReduceambiguityasmuchasuncertaintyPublicAdministrationReview76(3)399-402Retrievedfromhttpdxdoiorg101111puar12555Centreforpublicimpact(nd)RetrievedfromhttpswwwcentreforpublicimpactorgCernaL(2013)ThenatureofpolicychangeandimplementationAreviewofdifferenttheoreticalapproachesOrganisationforEconomicCo-OperationandDevelopmentRetrievedfromhttpswwwoecdorgeduceriTheNatureofPolicyChangeandImplementationpdfColoradoSprings(nd)Retrievedfromhttpscoloradospringsgovcommunicationspagewhat-expect-when-you-speakupContinuumofcare(coc)program(nd)RetrievedfromhttpswwwhudexchangeinfoprogramscocCouncilpresidentkicksoff$12millionyouthfund(nd)Retrievedfromhttpwwwbaltimorecitycouncilcomcontentcouncil-president-kicks-12-million-youth-fundDurlakJAampDuPreEP(2008)ImplementationmattersAreviewofresearchontheinfluenceofimplementationonprogramoutcomesandthefactorsaffectingimplementationAmericanJournalofCommunityPsychology41327-350Retrievedfromhttpswwwncbinlmnihgovpubmed18322790Fishbonediagram(ishikawa)(nd)Retrievedfromhttpasqorglearn-about-qualitycause-analysis-toolsoverviewfishbonehtmlFivewhysGettingtotherootofaproblemquickly(nd)RetrievedfromhttpswwwmindtoolscompagesarticlenewTMC_5Whtm
References
63
FligsteinNampMcAdamD(2011)TowardageneraltheoryofstrategicactionfieldsSociologicalTheory29(1)1-26RetrievedfromhttpsociologyberkeleyedusitesdefaultfilesfacultyfligsteinFligstein20McAdam20Soc20Theory20PaperpdHeinkelP(2018January12)CitymullspermitsforbikesharingcompaniesRetrievedfromhttpwwwdentonrccomnewsdenton20180112city-mulls-permits-bike-sharing-companiesHowisareamedianincomecalculated(2012October)Retrievedfromhttpswwwhudexchangeinfofaqs843how-is-area-median-income-calculatedLampkinLMampHatryHP(2003)KeystepsinoutcomemanagementUrbanInstituteRetrievedfromhttpswwwurbanorgresearchpublicationkey-steps-outcome-managementMarshDampSmithM(2000)UnderstandingpolicynetworksTowardsadialecticalapproachPoliticalStudies484-21Retrievedfromhttpjournalssagepubcomdoiabs1011111467-924800247McCannL(nd)RetrievedfromhttplaurenellencomMoultonSampSandfortJR(2016)ThestrategicactionfieldframeworkforpolicyimplementationresearchPolicyStudiesJournal45(1)Retrievedfromhttponlinelibrarywileycomstore101111psj12147assetpsj12147pdfv=1ampt=jdkhhcp4amps=e184c625e1855a6b1615ccb99ee27b1c0b3212d8PaynterB(2018January1)What100citiesarelearningfromeachotherbysharingtheirdataRetrievedfromhttpswwwfastcompanycom40518004what-100-cities-are-learning-from-each-other-by-sharing-their-dataPeakacademy(nd)Retrievedfromhttpswwwdenvergovorgcontentdenvergovenmayors-officeprograms-initiativespeak-performancepeak-academyhtmlPinnockHBarwickMCarpenterCREldridgeSGrandesGGriffithsCJRycroft-MaloneJMeissnerPMurrayEPatelASheikhAampTaylorSJC(2017)Standardsforreportingimplementationstudies(stari)ExplanationandelaborationdocumentBMJOpenRetrievedfromhttpbmjopenbmjcomcontentbmjopen74e013318fullpdfProctorESilmereHRaghavanRHovmandPAaronsGBungerAGriffeyRampHensleyM(2011)OutcomesforimplementationresearchConceptualdistinctionsmeasurementchallengesandresearchagendaAdministrationandPolicyinMentalHealth38(2)65-76doi101007s10488-010-0319-7TandonSD(2018)ProgressincommunityhealthpartnershipsResearcheducationandactionRetrievedfromhttpswwwpressjhuedujournalsprogress-community-health-partnerships-research-education-and-actionProvanKGampKenisP(2007)ModesofnetworkgovernanceStructuremanagementandeffectivenessJournalofPublicAdministrationResearchandTheory
References
64
18(2)229-252Retrievedfromhttpsacademicoupcomjpartarticle182229935895SabatierPA(1988)Anadvocacycoalitionframeworkofpolicychangeandtheroleofpolicy-orientedlearningthereinPolicySciences21(3)129-168Retrievedfromhttpwwwjstororgstable4532139SandstromAampCarlssonL(2008)TheperformanceofpolicynetworksTherelationbetweennetworkstructureandnetworkperformancePolicyStudiesJournal36497-524Retrievedfromhttponlinelibrarywileycomdoi101111j1541-0072200800281xabstractSchwartzMampWilsonE(nd)WhocanaffordtoliveinahomeAlookatdatafromthe2006americancommunitysurveyRetrievedfromhttpswwwcensusgovhousingcensuspublicationswho-can-affordpdfShearerJCAbelsonAKouyateBLavisJNampWaltG(2016)WhydopolicieschangeInstitutionsinterestsideasandnetworksinthreecasesofpolicyreformHealthPolicyandPlanning31(9)1200-1211Retrievedfromhttpsdoiorg101093heapolczw052ThelessonsofadministrativedataHigh-profilepolicy-relevantresearchpoweredbyadministrativedata(nd)Retrievedfromhttpswwwpovertyactionlaborgsitesdefaultfilesresources20161114-The-Lessons-of-Administrative-DatapdfToolsandotherneedsforqualitativeandquantitativemethods(2015February)AgencyforHealthcareResearchandQualityRetrievedfromhttpwwwahrqgovprofessionalsquality-patient-safetytalkingqualityassesstoolshtml
WeaverRCampBagchi-SenS(2013)SpatialanalysisofurbandeclineThegeographyofblightAppliedGeography4061-70
References
65
UniversityPartners
Name Title Affiliation Area(s)ofexpertise
Contactinformation
Examplesofworkingwithnon-academicdecisionmakers
WendyPBarreno
AssociateDirector
WilsonSheehanLabforEconomicOpportunitiesUniversityofNotreDame
LEOsfocusareasincludehealtheducationhousingchildwelfareeconomicopportunitycriminaljustice
wbarrenonduduorleondedu
MelissaDavey-RothwellPhD
AssociateScientist
JohnsHopkinsBloombergSchoolofPublicHealth
Health mdavey1jhuedu410-614-5854
BaltimoreCityHealthDepartmentLeadersMarylandDepartmentofHealthAddictionRecoveryServicesinDCExecutiveDirectorsofcommunityorganizationsacrosstheUSthatfocusonHIVSTIsubstanceabuseandimplementationscience
ElizabethStuart
AssociateDeanforEducationandProfessorDepartmentsofMentalHealthBiostatisticsandHealthPolicyand
JohnsHopkinsBloombergSchoolofPublicHealth
Educationsocialpolicystatisticalmethods
estuartjhuedu
Resources
66
Management
Design-ThinkingResources
EngagementGuideforCivicTech
Journeymap
PerformanceManagementResources
DenverPeakAcademy
Proxymeasuresguide
Talkingaboutperformancewithoutcomplicatingit(video)
SettingPerformanceTargetsguide
Datavisualizationguide
Performancemanagementgettingstartedguide
Exercises
Fishboneexercise
5Whys
AdministrativeData
TheLessonsofAdministrativeData(JPAL)
EvaluationResources
Quantitativeandqualitativemethods(USDepartmentofHealthandHumanServicesAgencyforHealthcareResearchandQuality)
KeyStepsinOutcomeManagement(UrbanInstitute)
RCT-YES
Sageresearchmethods
BestPracticesforMixedMethodsResearchintheHealthSciences
ACCORDChecklist
Pilotandimplementationplanning
Benchmarkingexamplesandproxymeasures
Seattleopendataplaybook
Resources
67
Datasciencescopingguide
Resources
68
AboutGovExGovExhelpsgovernmentsusedataandevidencetoimproveresidentsrsquolivesWesupportgovernmentsinbuildingtheirinternalcapacitytooperatesustainableanalyticspracticeswhichimprovecommunityoutcomesinfieldsfrompublicsafetytoeconomicdevelopmentGovExhasaproventrackrecordofworkinginconcertwithpartnerstoprovidetechnicalassistanceandtrainingatscale
SinceourlaunchinApril2015GovExhas
Trainedover2000governmentemployeesinpersonandonlineintopicsincludingdatamanagementperformancemanagementopendatacommunityengagementandanalytics
Providedtechnicalassistancetomorethan100localgovernmentsacrosstheglobeimprovingtheiruseofdatatoachieveefficienciesinservicedeliveryandbetteroutcomesforresidents
Assessedthecapacityofmorethan130governmentalorganizationstodeterminehowleadersstaffandexternalstakeholdersmanageandusedata
Localgovernmentsareincreasinglyonthefrontlinesofaddressinglargescaleproblemswithglobalimplicationssuchasclimatechangepublicsafetyandeconomicinequality
Whileevidence-basedinterventionstohelpsolvethoseproblemsareavailablelocalgovernmentstaffhavelimitedtimeandcapacitytofindandimplementresearch-basedsolutionsAsaresultcitiescontinuetoworkthewaytheyalwayshave--makingincrementalimprovementstoprogramsbutnotalwayscreatingchangeatasystem-level
ThroughresourcedevelopmentandcollaborationwithJHUfacultyGovExissupportingcityleadersinusingdataandevidencetomakedecisionsandcreatinglastingrelationshipswithresearchers
TheRoadmapwasdevelopedbyGovExaspartofitspartnershipwithBloombergPhilanthropiesWhatWorksCitiesinitiative
AboutGovEx
69
- Title page
- Introduction and purpose of the Roadmap
- Using the Roadmap
-
- Gather information about the problem
- Strategically engage key partners
- Review the evidence and collect details
- Decide on a policy to implement based on local factors
- Establish a framework to monitor progress
- Assess data accessibility quality and security
- Design implementation and evaluation
- Share your story and success
-
- Working with a range of evidence
- Thank you
- References
- Resources
- About GovEx
-
Yourturn
Thetemplatebelowismeanttohelpyoucraftagoalstatementforeachsupportingmeasure
Goalstatement
Departments
Outcome
Activities
KPI
Target
Timeframe
Conductanddiscussrelevantanalysis
ThrougharoutineseriesofmeetingscontinuallymonitorprogressofKPIssupportingindicatorsandanyotherdatacollectedsuchasqualitativedatatoidentifytrendsthatwillindicatewhetherthechosenpolicyishavingthedesiredimpactThesedatapointsshoulddrivethediscussionduringthesemeetingspointingtoopportunitiesforfurtherinquiryinto
Establishaframeworktomonitorprogress
37
whatisdrivingtheobservedtrendsAnalystsshouldbeencouragedandtrainedtoaskquestionsofthedepartmentsproducingthedataandconductsitevisitswhenpossibletodeepentheirunderstandingofthedataPresentationoftheanalysisshouldfocusononeideaatatimetokeeptheconversationfocusedonkeypointsandspecificopportunitiesforimprovementYoumayconsiderdevelopingapublicfacingdashboardtosharethisprogressmorebroadly
AgeneralcomparisonofthestatusquodatawithwhatwasobservedduringthepilotcanprovidedecisionmakerswithaquickunderstandingofthedifferencesbetweentheoldandnewapproachesalthoughthiswillnotshowcausalityOftenthesecomparisonswillincludeoperationalmetricssuchasdaysordollarssavedbutrememberthattheimpactonpeoplewillbemostimportantIfyouareworkingwithanevaluationpartnertheywillsupportthisstep
Usethespacebelowtosummarizetheresultsofyouranalysis
MeasureTargetvalue(seesectiononsettingtargetsabove)
Pilotvalue(whatdidyouobserveduringthepilot)
Historicalaverage(ifavailable)
Results(WhoisbetteroffIsanyonenegativelyimpacted1-2sentencesreassessingyourgoalstatement)
DocumentyourprocessandprogressincludingfeedbackfromtheparticipantsimplementationpartnersandtheprojectteamDidanythingunexpectedoccurduringthepilotArethereinternalandexternalfactorsthatmayaffectyouroutcomesThisiswheretheimplementationoutcomescanbecanhelpfulIftheinterventiondidnotgoasplannedconsiderwhetheritisbecausetheinterventionwasnotacceptableorfeasible
UsethetemplatebelowcreatedbyDenversPeakAcademytocapturethisinformationIfthe306090daytimelinedoesnotmakesenseforyourcityfeelfreetocustomizeit
CurrentState
FutureState
30Day
60Day
90day
Whatwentwell(+)
Whatdidnotgowellshouldbechanged(-)
IfaquestionordiscussionpointarisesthatcannotbeansweredinthemeetinganassignedstaffmembershouldcapturethequestionduringthemeetingAfterwardthefollowupitemsshouldbesenttorelevantpartnersandthenextmeetingshouldbeginwithadiscussionof
Establishaframeworktomonitorprogress
38
thoseitems
Takeaction
DiscussionsaboutwhetheryourindicatorsaremovingintherightdirectionandwhatmightbedrivingtrendsmayleadtoadiscoverythattherearenotenoughresourcesallocatedtothispolicyorissueTheremaybelimitedfundingforstaffprogrammingortechnologythatisdrivingthedatainthewrongdirectionIfCityleadershipisintheroomthisisagreattimetomakethecaseforadditionalfundingduringthebudgetseasonBycollectinganalyzinganddiscussingdataonthispolicyroutinelyyouarebuildingevidencetomakethecasefortherightleveloffundingtoreachtheestablishedtargets
Establishaframeworktomonitorprogress
39
Assessdataaccessibilityqualityandsecurity
Determineaccessibilityandqualityofdata
OnceyouhavecompiledyourwishlistofindicatorsitistimetodeterminetheaccessibilityandqualityofthedataTopreventcomplicationyoumaywishtoprioritizerelevantindicatorsforwhichyouarealreadycollectingdataAvoidthetemptationofadoptingatechnologysolutiontocollectnewdataatthispointsuchasaphotosurveytoolorotherappFornowyouaretryingtodeterminewhichdatasetsarerelevantaccessibleandhighqualityBesuretoworkwithyourkeyinternalpartnerswhileyouareidentifyingyourmeasuresDepartmentstaffwhoworkonprogramsarefamiliarwiththeindicatorsthatwillaccuratelysupporttheoutcomesanddatabaseadministratorswillbevaluablewhendiscussingaccessibilityandqualityofthedata
Abrieflistofconsiderationsfortheaccessibilityofyourdataisbelow
ArewecurrentlycollectingthisdatasetorwillwebecollectingitforthefirsttimeHowfarbackdoestherecordgo(timehorizon)WouldweneednewtechnologytocollectoranalyzethedataCanthedatabecollectedmanuallyorcanitbeautomatedHowfrequentlyisthedataproduced(continuouslyhourlydailyweeklymonthlyquarterlyyearlyetc)IsthedataavailablefortheentireCitypopulationorasubsetofthepopulationHowgranularisthedata(individuallevelCensusblockneighborhoodcitywideetc)DoesthedatacontainsensitiveinformationIsthereaproxymeasurethatwecanusefornowIsthedatamachinereadable
AsnotedbytheWilsonSheehanLabforEconomicOpportunities(LEO)attheUniversityofNotreDameleveragingadministrativedatacansignificantlyreducethecostsandtimelinesofrigorousevaluationCitiescanestablishaninternaldatasharingagreementacrossdepartmentsincludinglanguagethatenablesadministrativedatatobeusedforresearchpurposesCitiescanalsoidentifyexternaladministrativedata-holderssuchasacountyschooldistrictpolicedepartmenthospitaloranotherstatethatmayhaveoutcomedataimportantformeasuringprogramimpactCitiescanleverageexistingrelationshipsandorworkwithresearchinstitutionstodevelopstreamlineddatasharingprocessesandagreementswiththeseexternalentities
Assessdataaccessibilityqualityandsecurity
40
Evaluatethequalityofyourdata
QualityisasimportantasaccessibilityToevaluatedataqualityyoumaywishtocreatearubricliketheonebelow
Value ValueDescription
3 ExcellentDataQualityThedataiscompleteaccurateandupdatedregularly
2 AdequateDataQualityThedataissomewhatcompleteaccurateandupdatedfairlyregularly
1 PoorDataQualityThedataisnotcompleteaccurateorupdatedregularly
ThisisagreattimetobuildonthedatainventorythatyoustartedinthegatherinformationpageSuggestedadditionalfieldstoaddtoyourdatainventorytemplatearebelow
Program DatasetTitle
DatasetDescription DataSource
Departmentorexternalpartner
DoestheCityalreadycollectthisdata
Whatisthenameoftheprogram
Human-readablenameofthedataset
Shouldbeunderstandablebynon-technicalusers
Whatistheoriginalsystemapplicationfilethathousesit
Departmentorexternalpartnerthatmaintainsthedata
Isthisdataalreadycollected
Updatefrequency-ThefileformatforthedatasetTypicallycsvxlsxshapefiletxtetc
Format-Howoftenthedatasetisupdated
Timerange-Theperiodoftimecoveredinthisparticulardataset
Datause-Howisthisdatasetcommonlyused
Datausers-Whotypicallyusesthisdataset
Dataownerstitle-Titleofthedataowner
Dataownersposition-DataOwnersjobtitle
Dataownersemailaddress-DataOwnersemailaddressWhomanagesthedataandorisresponsibleforgrantingpermissiontoaccessthedataWhounderstandswhatthedatasetincludesandcananswerquestionsaboutit
Dataqualityconcerns-Describetheconcernsyoumayhaveaboutthequalityofthedataset
Assessdataaccessibilityqualityandsecurity
41
Division-DivisionwherethedataismaintainedIfmorethanonedivisionisresponsiblelisttheprimaryownerofthedataset
Sensitivedatacomments-DescribetheissuesthatwouldpreventthecityfromsharingthedatapubliclyCommonsensitivityconcernsincludeprivacyviolationssecurityissuesorhighcostorstaffdemandstopostthedata
Publishingstatus-Thistheplacetoputanycommentsregardingthecurrentstatusofthedataset
Publishedlink-Thisisthelinktothepublisheddatasetontheopendataportalorcitywebsite
Reviewdatasecurityandprivacyprotectionprotocols
OftenapolicychangealsoinvolvesachangeinthedatathatisbeingcollectedhowthatdataisbeinghandledwhohasaccesstoitandahostofotheradjustmentsAsyoutakestockofthedataassetsthatareavailableforyourinitiativeitisvitaltoalsothinkthroughwhatwillbedonetoprotectthatdataandtheprivacyofthepeopleitrepresentsAbigpartofgooddatasecurityinvolvesstrongdatagovernancepracticesthatdefineandidentifysensitivedataandconsciouslyworktobalancethevalueandriskofstoringsuchsensitivedataOnewayofevaluatingyourdatacomesfromcaselawrelatedtotheUSFreedomofInformationActwhichwehavetranslatedtothechecklistbelow
1 Woulddisclosureresultinasubstantialinvasionofprivacy2 Haveweconsideredtheextentorvalueofthepublicinterestandthepurposeorobject
oftheindividualsseekingdisclosure3 Istheinformationavailablefromothersources4 Wastheinformationgivenwithanexpectationofconfidentiality5 Isitpossibletomouldreliefsoastolimittheinvasionofindividualprivacy
TheseriskscouldbemitigatedusingavarietyofstrategiesfromaggregatingdatatoobfuscatingindividualstoredactingfieldsthatmayincludePersonallyIdentifiableInformation(PII)orre-evaluatingyourdatacollectionmethodstofindalternativewaysofgettingtheinformationyouneedThesestepsshouldbecodifiedandmanagedbyadatagovernancecommitteeifpossibleForthesensitivedatathatisdeemedessentialcreateaplanaroundpotentialdatabreachesinordertoquicklyandconfidentlyrespondtoemergencysituations
ProactivelypreparingforthesesituationsisoftenthebeststrategyfordatasecuritybutdonotforgetthetechnicalsideInvolveyourITadministratorsinconversationsaboutdatastorageandaccessensuringthattheywillbeprotectedbyenterprise-gradecybersecuritysystems
Assessdataaccessibilityqualityandsecurity
42
Assessdataaccessibilityqualityandsecurity
43
Designimplementationandevaluation
Designtheimplementation
NowthatyouhaveyouraccountabilityplanandkeyindicatorsitistimetogetspecificaboutyourplanforimplementationThetemplatebelowwillhelpyouthinkthroughthespecificmechanismsofimplementingpolicychange
UnderstandingorganizationsandsystemsiscriticaltounderstandingandmakingdecisionsaboutwhereandhowinterventionsshouldbeappliedApolicyimplementedinasinglesettingmayshowpromisingresultsbutimplementingmultiplepoliciesandprogramsatthesametimewithdifferentactorsresponsibleforeachaspectcanhelpensuresustainablebehaviorchange
InSomervilleMAShapeUpSomerville(SUS)isacitywidepublichealthprogramthatisimplementedacrossmultiplelevelsofinfluenceandsystemsthroughinterventionsbeforeduringandafterschooltoincludechangestobuiltenvironmentsuchasbikelanesregionalmasstransportationandphysicaleducationequipmentinschoolsandgymstoincreasephysicalactivitywellnesspoliciesandschoolgardensnutritioneducationduringandafterschoolandhealthyrestaurantprogramstosupporthealthyfoodandhealthyeatinghabits
Usethetemplatebelowtoplanyourimplementation
Policyimplementationplanningtemplate
ListthespecificdesiredoutcomesInthepreviousstepyouworkedwithdepartmentstafftoestablishspecificmeasurableoutcomesandinterimindicatorsPrioritizetheindicatorsthatwilldemonstrateprogressUsethespacebelowtocapturethedesiredoutcomesofthepolicychange
ExampleShapeupSomervillebeganwiththegoalofpreventingobesityin1stthrough3rdgraders
Yourturn
____________________________________________________________________________
____________________________________________________________________________
Designimplementationandevaluation
44
SpecifythetargetpopulationAreyoudesigninganinterventionthattargetsonlycertainsitesAreyouplanningonimplementingaprogramorinitiativeforaspecifictargetaudienceClearlyspecifythepopulationyouplantotarget
IfyouplantoconductarigorousimpactevaluationasnotedbytheWilsonSheehanLabforEconomicOpportunitiesitisimportanttoestablishconcreteobjectiveeligibilitycriteriaandensurethatthesecriteriaareappliedconsistentlybyallstaffduringscreeningandintakeItisalsovitaltoidentifytheindividualswhoareeligibleforaparticularprogramorserviceevenifeveryoneeligiblecannotbeservedbyexistingresourcesThishelpscitiesidentifyanyunmetneedsandguidefutureresourceallocationwhilealsocreatingthepossibilityofarigorousimpactevaluationbyidentifyingapotentialcomparisongroupFinallywhenconductingscreeningandintakeitisimperativetocollectandmaintainbasicinformation(egnamedateofbirthgender)aboutallprogram-eligibleindividualsnotjustthosewhoparticipateintheprogram
ExampleIntheShapeUpSomervilleexampletheprogrambeganbytargeting1stthrough3rdgraders
Yourturn
____________________________________________________________________________
____________________________________________________________________________
DefinethelocationWherewillthepolicychangetakeplaceWillimplementationtakeplaceatonesiteormultiplesites
ExampleShapeUpSomervilleinterventionsweredeliveredinschools
Yourturn
____________________________________________________________________________
____________________________________________________________________________
ClarifyimplementationpartnersandarticulatetheirrolesWhatarethekeyorganizationsinstitutionsorpeopleresponsibleforimplementationTheprojectteamshouldbeasclearandspecificaspossibleinprovidingadescriptionoftheconcreteactivitiesthatimplementationpartnersaremeanttocarryoutPracticeprofilesmaybeahelpfultooltohelpdefinewhatisrequiredoftheactorswhoareresponsibleforimplementingtheintervention
Designimplementationandevaluation
45
ExampleInSomervillemayoralleadershipandengagementofmultiplecityagencies(publichealthprimarycareschoolsCambridgeHealthAllianceetc)streamlinedandconnectedthevariousShapeUpSomervilleinitiativesandgrantswhichhelpedtokeepthetalentandenergyfocusedonhealthyeatingandactivelivingforallsectorsofSomervilleSUSpartnersultimatelyincludedschoolsnonprofitsCityandstateagenciesandnationalfoundations
Yourturn
____________________________________________________________________________
____________________________________________________________________________
DescribetheinterventionWhatarethestepsactivitiesorprocessesRememberthatthemechanismthespecificstepsorprocessthatleadtobehaviorchangeiskeytounderstandingtheeffectivenessofthepolicychangeandsharingtheresultswithothers
ExampleGoalsofShapeUpSomervilleweretoinfluenceeverypartofanearlyelementaryschoolchildrsquosdayndashbeforeduringandafter-schooltopromotehealthyeatingphysicalactivityandtheirweightstatusInterventionincludedfoodserviceenhancementstoimprovebreakfastandlunchandstaffprofessionaldevelopmentwalktoschoolactivitiesSUSclassroomcurriculumandprofessionaldevelopmentschoolwellnesspolicydevelopmentSUSafter-schoolcurriculumandprofessionaldevelopmentoutreachandeducationtohomethroughmaterialsforumseventsldquoSUSapprovedrdquorestaurantscommunityoutreachandcapacitybuildingthroughpolicydevelopmenttrainingsmediaplacements
Yourturn
____________________________________________________________________________
____________________________________________________________________________
ClarifytimelineandmilestonesfordeliveryandevaluationHowfrequentlywilltheservicesinterventionbedeliveredHowlongwilltheprogramrunWhatisthetimetoexpectedimpacts(eg12monthsafterprogramcompletion)
Designimplementationandevaluation
46
ExampleShapeUpSomervilleledbyTuftsUniversityFriedmanSchoolofNutritionScienceandPolicyconductedaquasi-experimental(nonrandomized)interventionstudyamong1stthrough3rdgradestudentsinSomerville(intervention)andtwocomparisoncommunitiesfor9-monthsduringoneschoolyear(fall2002ndashspring2003)ThestudyshowedpositiveassociationoftheinterventiononchangeinBMISUSreleasedreportssummarizingevaluationsprogresstodateplansforadditionalprojectsandimpactstenandfifteenyearsintotheinitiative
Yourturn
____________________________________________________________________________
____________________________________________________________________________
PlanyouranalysisTheprevioussectionaddressedestablishinganaccountabilityframeworktomonitorprogressandimplementationDeterminewhichindicatorswillbediscussthroughthatprocesswhowillsharethedataandanalysisandhowfeedbackfromthoseconversationswillbeincorporatedbytheprojectteam
ExampleTheMayorrsquosofficeimplementedadata-drivendecisionmakingapproachcalledSomerStatwhichwassynergisticwiththeapproachestakenbytheresearchandevaluationteamsdeployedforthespecificgrants
Yourturn
____________________________________________________________________________
____________________________________________________________________________
PlanforsustainabilityIfthepolicyandprogramchangeissuccessfulhowwilltheprojectteamsustainandexpandtoadditionalsitesorpopulations
ExampleAlthoughmostofthepartnershipsforgedintheearlyyearsweremaintainednewgrantsandcollaborationsexpandedattentiontoolderpublicschoolstudentsactivetransportationandtheGreenLinesubwayextensionthebuiltenvironmentfarmerrsquosmarketeffortssuchasldquoGrowninSomervillerdquowithUnionMainStreeteconomicdevelopmentandhealthdisparitiesaswellasdatacollectionmonitoringandsurveillance
Yourturn
Designimplementationandevaluation
47
____________________________________________________________________________
____________________________________________________________________________
Pilotplanningtemplate
IfyouplantoconductapilotfirstdeterminethepurposeofthepilotDoyouwanttoobservewhetherthepolicychangeresultsinthedesiredbehaviorchangeAreyouconcernedaboutfeasibilityscalabilityorcostDoyouwanttotestwhetheryourdesiredresultscanbeachievedthroughthispolicyapproachClarifythegoalofthepilotandthequestionsyouhopetoanswerThiswillhelpyouidentifythekeyindicatorsthatyouwillusetomeasurewhetherthepilotissuccessfulBeforebeginningestablishabaselineofcurrentperformanceandsetyourtargetsaccordinglybasedonwhatyouhopetoaccomplishthroughthepilot
NextestablishthepilotplanforyourprojectteamincludingmilestonesforteamcheckinstodeterminewhethertomoveaheadorchangecourseThisisseparatefromthemeetingsthatwilloccuraspartoftheaccountabilityframeworkUsethespacebelowtocreateyourtimelineanddevelopyourmeetingschedule
Task Personresponsible Startdate Checkindate Deliverable
Planyourevaluation
WhileyouaredesigningtheimplementationitisalsotimetobegintodevelopyourplanforanalysisAsnotedbytheWilsonSheehanLabforEconomicOpportunitiesdecisionsaboutprogramsetup-includingoutreachscreeningintakedatatrackingandenrollment-candeterminewhetherornotarigorousimpactevaluationwillbepossibleAsdiscussedcityleadersmayfindithelpfultoincludeevaluationpartnersinearlyconversationsaboutprogramsetupDuringpreviousstagesyoumayhaveworkedwitharesearchertoplanyourproposedpolicychangeYoushouldbeclearaboutthepurposeoftheevaluationanddevelopanevaluationplanBelowareadditionalconsiderations
Causationvstrendanalysis
IntheEstablishaframeworktomonitorprogressstagetherewasguidanceontrackingtrendsovertimeItisimportanttonoteherethatmonitoringdataisdifferentfromanalyzingacausaleffectTrendanalysiswillillustratethatchangesareoccurringandmayleadtoquestionsthatcouldbeansweredbyyourevaluationbuttrendanalysisalonedoesnotprovideevidenceaboutwhatisdrivingthosechangesYoumayseechangesinyour
Designimplementationandevaluation
48
outcomeindicatorsbasedonvariablesoutsideoftheCityscontrolForinstanceachangeinfederaltradepolicycanhaveanimpactemploymentratesinyourCityoutsideofyourCitysnewjobtrainingandplacementprogram
ToaddresscausalityyoumustcomparetheoutcomesofindividualswhoreceivedtheinterventionwiththosewhodidnotWithoutacomparisongroupofsimilarindividualswhowerenotexposedtotheinterventionitisimpossibletoknowwhichfactorsareresponsiblefortheoutcomesRandomizedControlTrials(RCTs)inwhichindividualsarerandomlyassignedtotreatmentandcontrolgroupsarethegoldstandardinprovingcausalityIfyoucandemonstratethatoutcomesimprovedforthosewhoreceivedtheinterventionanddidnotimproveatallorimprovedlessforacomparisongroupyoucanbesomewhatcertainthattheinterventionwasthecause
Examplesofothernon-causalimpactevaluationsthatarerigorousandreliableincludecertainquasi-experimentalstudiesusingcomparisongroupdesignssmartinterruptedtimeseriesanalysesthatmodeltrendsovertimeinwaysthatallowestimationofinterventioneffectswelldesignedpre-posttestanalysesneedsassessmentsimplementationstudiesetcFormoreinformationondifferenttypesofevaluationsseetheSageResearchMethodswebsite
Lookfordifferencesingroups
BuildintimetoalsoexaminethepilotdatatoseeifgroupsareexperiencingdifferentresultsReviewingperformancebasedonoutcomemeasuresforapopulationinaggregatemayhidethesubtlewaysthatsomepeoplemaybeaffecteddifferentlysoitisnecessarytocarefullyanalyzethedatafordisparateoutcomesForinstancetheMovingtoOpportunity(MTO)studythatexaminedtheimpactofhousingvouchersonfamilieswholivedinhigh-povertyneighborhoodsfoundthatchildrenwhomovedbeforetheageof13hadbettereducationandhigherincomesasadultsHoweveradultsdidnotseeimprovededucationandemploymentoutcomes
Disaggregatingdataintomorespecificcategoriesofraceandethnicitycanhelpunderstandtrendsanddisparitiesinamoregranularwaywhichcanhelpdecisionmakerstargetpoliciesandinterventionsinamoreequitableway
EarlierintheRoadmapweworkedtoidentifystakeholdersandwhomightbepositivelyandnegativelyimpactedbyachangeinpolicyUsetheresultsofthatexercisetoinvolvethecommunityininterpretingdataanalysisandtogatherqualitativefeedbackfromthecommunitytocallattentiontoissuesForinstanceanincreaseinyouthdetentionmaybeexplainedbyinefficienciesorlackofaccesstotransportation
Designimplementationandevaluation
49
Designimplementationandevaluation
50
Shareyourstoryandsuccess
Sharetheresults
Storytelling
AnintegralstepinyourstrategyissharingtheresultsofyourpilotorpolicychangewithyourcommunityThisincludesdevelopingmessagingaroundtheworkofyourorganizationandcreatingaplantodisseminateyourstoriesWhytellstoriesaboutyourwork
Informandorunderstandasubjectorsituation
Giveneededcontextofdataorbitsofinformation
Relatablestoriesarememorableforreadersviewers
ReadersorviewerswillseetheinformationasmorecompellingwhenthereisademonstratedandrelatablechangeorimpactonthemselvesortheirfriendsneighborsandcolleaguesStoriesareaboutpeopleandgovernmentisfundamentallyabouthelpingpeopleSotellthestoriesofthepeopleyouandyourcitycolleagueshelped
Keepthesetipsinmindasyoudevelopyourstorywithyourteam
WhatwereyourfindingsandwhyshouldanyonecareTellyouraudiencehowtheproblemwassolvedWhathastheimpactbeenonresidentsandcitystafftodateWhatwasthesuccessandweretheremultiplesuccessesalongthewayIfyouhavepreliminaryresultsordatashowingimprovementtosharethisisthetimetodoit
WhowasinvolvedandimpactedReviewthelistofstakeholdersHowwereeachofthemimpactedbytheprogramHumanizeyourstoryandimpactbyaddingadditionalvoicesThesevoicescanbeoneofthefollowing
Apersonorfamilywhowaspositivelyaffectedbythechangesuccess
Aninvolvedthird-partydescribingboththeproblemandthesolution(ieanorganizationnonprofitprovidersbusinesstradeassociationsportsteametc)whoispartofthecommunitythatthegovernmentworkedwithonthechangesuccess
Someonefromthegovernmentitselfwhocanspeakaboutthechangesuccess
TellmoreaboutyourcityDescribeyourcityincludingpopulationdemographicsindustriesetcWhatareitschallengesWhatisitknownforWhatmakesituniqueHowisitsimilarordifferentfromothercitiesintheUS
Shareyourstoryandsuccess
51
DescribeyourcityrsquoschallengeWriteoutthechallengeyourcityisfacing(asdefinedinyourproblemstatement)HowdoesitaffectresidentsHowdoesthisproblemaffecttheworkofcitystaffmembers
AddthedetailsProvidedetailsonyourappliedresearchstrategyTalkaboutwhatyourcityhasdonenthepasttoresolvetheissueDiscusshowyourstrategyhaschangedwhenyoubegantoadoptthisapproachWasmoneysavedorinvested
Usethisstorymapexercisetoorganizeyourthoughtsandguaranteethatyouincludealloftheelementsofatraditionalstory
AfteryoudevelopyourmessagingcreateacommunicationsmarketingandoutreachstrategydesignedtospreadyourmessageYourstrategyshouldwalkyouthroughthefollowingkeyactivities
1 IdentifyandprofileyouraudienceWhencreatinganddisseminatingyourmessageitisalwaysgoodtothinkaboutyouraudienceWhoexactlywouldbeinterestedinwhatyouhavetosayThegeneralpopulationofthecityPeoplewhoworkinnonprofitsBusinessownersItmattersbecausenoteveryonewholivesinacitymaybethetargetoftheinformationBusinessownersmaybemoreinterestedindatashowingeffectsonthebusinessdistrictthanothersandsoonWewillcoverthislaterbutthisstepwillalsohelpyoutothinkofthemosteffectivewaytoreachyouraudience
2 Defineyourcalltoaction(ifyouhaveone)SpendsometimeconsideringwhyyouareinterestedintellingyourstoryDoyousimplywanttoupdatethepubliconyourprogressorcelebrateyoursuccessesAreyouinterestedinyouraudiencechangingitsbehaviororsupportingaspecificinitiativebeinglaunchedbyyourorganizationIfyouwanttohaveyouraudienceactorbehaveinacertainwaybespecificandincorporateitinyourmessaging
Shareyourstoryandsuccess
52
3 SelectperformancemeasuresThinkabouthowyouwoulddetermineifyourcampaignissuccessfulandselectappropriateperformancemeasuresYouwillrevisitthesemeasuresattheendofyourcampaigncommunicationsplanimplementationtodeterminewhetheryouneedtomodifyyourcommunicationsandoutreachstrategyforthefuture
4 MethodstocommunicateyourstoryTherearemanywaystocommunicatesuccessofyourprojectHerearesomesuggestionsonreachingaudiencesItisimportanttodeterminewhichmethodsarethemosteffectiveforyourdesiredaudiencetoreceiveinformationDoyourresearch(PewResearchCenterhasexcellentstudiesonthereachofmediadigitalmediaandmarketing)ForexampleifseniorsareyourtargetpopulationandtheyarelessinclinedtouseInstagram(thatistrue)youmaywanttoconsiderFacebookorthenightlylocalnewsbothofwhichareusedmoreoftenbyseniorsBelowisalistofsomemethodsandtoolsyoucanuse
i Pressreleasestolocalnewspapersandtvstations
ii ShortsmartphonevideosAgreatwaytoengageanaudienceisthroughvideoAskonequestionofsomeonewhoispartofthestoryMaybethereisasocialworkerwhocandiscusshungerinyourcityThesevideoscanbereusedonmultipleplatforms
iii Long-formwriting
iv TellingindividualstoriesthroughaseriesofphotosandshortnarrativesforpublicationonFacebookorInstagram
v Op-edsopinionpiecesplacedinthelocalpaper
vi Emailupdatestoyourlist
vii Connectingwithappropriateorganizationsthatareinvestedinthetopicandworkwiththecommunityandaskthemtodisseminateinformationtotheirmembers
viii EmployingldquoWordofMouthrdquo(WOM)tacticsbycreatingtownhallsorfreeeventsforresidentstoattendintheirownneighborhoods
5 DeterminethetypeofcollateralUsuallycampaignsthattargetresidentsrequireseveralavenuestoreachthemPeoplearenotalwaysinthesameplaceorspaceHavingakeenunderstandingofyouraudiencewillassistyouwithyourchoicesonwhatneedstobedevelopedSometimesthatmeansimagesforsocialmediaaonepageflierthatgivespeopleinformationonattendinganeventoranin-depthbrochure
Shareyourstoryandsuccess
53
6 ImplementyourstrategyKeepinmindstrategiescanbefluidSometimessomethingmaynotturnoutasexpectedevenifduringplanningsessionsitwasldquosupposedtordquoItisokYoucananalyzewhathappenedduringadebriefsessionandonceyoursurveysandothermeasurementshavebeenfinalized
Visualizations
Withtherightvisualizationyoucanreviewandshareinformationinsightsanalysisperformanceandtrendsallofwhichisvaluabletoacross-sectionofthepublicButwhichvisualizationisrightforyouTochoosetherighttypeofchartgraphdeterminethetypeofquestionyouwanttoanswerSomecommoncharttypesarebelowandmoreinformationcanbefoundhere
Exploreadditionalanalyticaltechniques
Duringthegoal-settingphaseyouestablishedoutputoutcomeandimplementationmeasuresBasedontheresultsofyourpilotyoumayneedtoadjustyourmeasuresWiththepilotdatainhandyoumayhavesomeideasofhowfutureanalysescouldbedifferentandpotentiallyimprovethequalityofyourprogramTakesometimetobrainstormotherrelevantdatasetsthatyoudidnothavethetimetopursueearlierincludingmessydepartmentaldatathatneedsmoreattentionorsourcesofpublicdatafromthestateandfederalgovernmentnonprofitsandevenprivatecompaniesthatcouldsupplementyourunderstandingofanissueUsingmachinelearningtechniquesyoumightbeabletobetter
Shareyourstoryandsuccess
54
predictdemandforhomelessservicesusingNOAAweatherforecastspreemptwatermainbreaksusingsoilstratigraphyorinvestineconomicdevelopmentbymonitoringrealestatelistingdata
Advancedanalyticaltechniquescanalsohelpyourimplementationrunmoreeffectivelybyintegratingdata-informedtargetinganddecisiontoolsintotheprocessByautomatingroutineworkflowsandsurfacingsubtletrendsthesetoolscanhelpyouiterateinamoreprecisemannerandprovidekeystrategicguidancetofront-lineemployeesthatwouldotherwisebeunavailableThesesortsofdata-science-enhancedprocessescanbeclassifiedasoneofsixprojecttypesasexploredfurtherbytheNewOrleansLANOLAlyticsteamFormoreguidanceonscopingasimilarprojecttheGovExadvancedanalyticsscopingguidegivesyouastep-by-stepoverviewofhowtobesuccessful
Usethespacebelowtolistyourideasforpossibleadvancedanalyticsprojectsbytype
Typeofproject Ideas
Findingtheneedleinthehaystack
Prioritizingforimpact
Earlywarningtools
Betterquickerdecisions
Optimizingresourceallocation
Experimentingforwhatworks
Keepthefeedbackloopgoing
GatherinformationonpublicsentimentaboutpolicychangethroughfocusgroupssurveysandpublicmeetingsledbyadvisorygroupmembersNotewhethernewinformationisavailablethatcanhelpanalyzeorcontextualizeyourprogress(egfederalorstatedatareports)Sharetheresultsofyourpilotandthepolicychangeprocesswithothercitiesandresearchers
Shareyourstoryandsuccess
55
WorkingwitharangeofevidencePresentlythereisalotofattentionandinvestmentinevidence-basedpolicymakingincludingthefederalbipartisanCommissiononEvidence-BasedPolicymakingthePewCharitableTrustsreportsandtoolkitsonHowStatesEngageinEvidence-BasedPolicymakingandtheLauraandJohnArnoldFoundationsupportedEvidence-BasedPolicymakingCollaborativeManyoftheseinitiativesfocusonrigorousevaluation--meaningrandomizedcontrolledtrials(RCT)orotherexperimentalmodels--asthebestkindofevidencetouseHoweverthisbodyofevidenceisofteninsufficientforthespecifickindsofquestionsthatgovernmentscanhavewhenmakingadecisionaboutpolicyInthisguideweproposeanapproachtosupportgovernmentsinincorporatingadditionaltypesofevidenceintotheirdecisionmaking
OnewaytounderstandarangeofevidenceistouseNestarsquosStandardsofEvidencetoframeanddefineevidenceTheBritishnonprofitNestaworkstoinjectpositivechangeintothewayUKgovernmentpoliciesandprogramsfunctionOneoftheirkeyareasofworkhasbeendevelopingWhatWorksCentresorganizationsthatcollectcommissionsynthesizeanddisseminateevidenceaboutinterventioneffectivenessacrossarangeofsocialimpactareasNestadevelopedthefollowingStandardsofEvidencetoprovideausefulwaytodocumenthowdifferentkindsofinformationdemonstratedifferentlevelsofevidence
Workingwitharangeofevidence
56
Theimportanceofpotentiallybiasedinformation
Onechallengepresentedbyanevidenceclassificationthatacknowledgesthevalueofnon-scientificformsofevidenceistofigureouthowtobalanceallofthepiecestogetherwhenyouusethemTounderstandhowtodothiswehavetothinkabouthowtopositionpotentiallybiasedinformationindecisionmaking
WhenwelookatinformationwhichhasnotbeenrigorouslyevaluatedweacceptthatthisinformationmightbebiasedThewholereasonweconductrigorousevaluationistoremovebiasfromourobservationsWeconductstructuredtrialstoproduceresultsthatareconsistentandgeneralizableIfafindingisunbiasedthisaccuracymdashthislackofbiasmdashcanbeprovedbecauseanyindependenttestercanreproduceitwhentheyfollowanidenticalprocess
YoucanlookatpotentiallybiasedinformationasjustinformationthatisearlieroninitspathtorefinementreplicationandverifiedaccuracyApieceofinformationbasedonanyoneindividualrsquosobservationsislikelytobebiasedbecauseitiscomingfromjustonesourceBiasisreducedbygatheringtogetherlotsofindependentsourcesofinformationandfiguringoutitsmostcommonattributes
MaybethebiasedinformationmakesaclaimthatistoolargemdashitsayssomethingistrueunderallconditionswhenitisreallyonlytrueundercertainconditionsorthataneffectofaninterventionisbiggerthanitstrueaverageorthataneffectlastsovertimewhenitdoesnotPerhapschangingtheclaimtobemorespecificmdashcriticallyreviewingtheinformationtoseeifspecificconditionsmustbemettoachievetheoutcomesmdashwouldmaketheclaimmoreaccurate
IfyoutestedyourclaimfounditdidnotholdupinsomecaseschangedittesteditagainuntilyourclaimrepeatedlyheldupthenyoucoulddemonstratethattheclaimwasnolongerbiasedbutaccurateThisprocessoftestingimprovingtestingandvalidatingiswhatwegetfromstructuredscientificstudy
Withindividualpiecesofinformation--crediblehypothesescasestudiesexpertopinion--youcannotassumethatkindofaccuracyAtthesametimetheseearlyobservationsarefoundationaltoanyreplicablescientificobservationsWewillnevergetanywhereifwedonotstartwiththesepiecesofinformationeveniftheymightbebiased
Combininglevelsofevidencetogetthemostdetailedinformationforyourcity
ItisespeciallyimportanttoacceptdifferentformsofevidencewhenyoufacequestionsthathavenotyetbeentestedbyindependentstudiesSomeofthemostimportantquestionsrelatetothingsthatareveryspecifictoyoursituationYoufaceaparticularcombinationof
Workingwitharangeofevidence
57
issuesactorstrendsandresourcesthatisultimatelydifferentfromeveryotherplaceintheworld
ThisisimportantbecauseaspolicypractitionersandnotscientistsyoucaremoreaboutyourspecificcasethanahypotheticalaveragegeneralcaseYoucarelessaboutthetotallikelyrangeofimpactsfromapolicythanhowthissingleimplementationwillworkYouwantinformationaboutpoliciesthatwillprovideyouwiththebestpossiblepredictionabouthowtheywillworkinyourcontext
ByputtingdifferentkindsofevidenceintothesameframeworkNestarsquosStandardsofEvidenceimplythatwhenyoucannotgetrigorouslytestedexperimentallyverifiedinformationitisappropriatetoworkwithwhatwedohaveThisisrelevantforthemunicipalimplementationcontextbecausenostudypulledfromaliteraturereviewwillaccountforallofthevariablesyoufaceinyourowncontext
Intheabsenceofhavingastudythatpredictshowaninterventionworksinyourspecificcontextyouwillneednon-scientificinformationtotrytofillinthosegapsYouwillneedtogatherstoriesfromresidentsstaffmembersandotherstakeholdersthatsitonLevels1or2ofthestandardsofevidence
Someofthekindsofinformationyouwillprobablygetmoreeasilyfrompeoplethanstudiesinclude
howpeopleincitieslikeyoursreactedtoaninterventioninpublicforumshowpeoplewhoimplementedaninterventioninacitylikeyoursfiguredouthowtogetitpassedhowthecitycommunicatedpubliclyabouttheapparentbenefitsanddrawbacksoftheinterventionthepoliticalacceptabilityoftheintervention
Youcanremainsimultaneouslyawarethattheinformationyouaregatheringthroughindividualstoriesisbiased--notgeneralizableinformedbyspecificperspectives--whilealsorecognizingthathavingsomeinformationaboutsomethingimportantisbetterthannoinformationEvidencefromthefirsttwoevidencelevelsareessentialbuildingblocksofknowledgeandperfectlyusefulwhenyoucannotgetdetailedunbiasedknowledge
Youhavetostartsomewhereandyoushouldalwaysstartwithevidenceyouhaveaboutyourveryspecificconditionsthatyouknowareimportantforsuccess
Workingwitharangeofevidence
58
Thankyou
Interviewswithcities
GovExheldsemi-structuredinterviewswithsixstaffmembersinfourcitiestoestablishtheinitialstepsoftheRoadmapWeaskedrespondentstotalkaboutaninitiativeorpolicythattheyhavebeenresponsibleforimplementingintheircityRespondentsfromthreecitiesspokeabouttheireffortstoaddresshomelessnessandrespondentsfromonecityspokeaboutblightAfterrespondentsidentifiedtheproblemtheyspokeabouthowthecityprioritizedthisissuethevariousstepsinvolvedandbarrierspresent
GovExconductedtextanalysisoftheseinterviewswhichrevealedthatpeoplewasthemostcommonlyusedtermduringtheseconversationswhendiscussingthestepstochangeaprogramorpolicyThisisathemethroughoutthisRoadmapThefirstsectionbeginswithidentifyingtherelevantpeopleTheconversationsaroundevidencearefocusedonhumanbehaviorratherthanmethodologyorstrengthofevidence
Thankyou
59
OncewehadadraftoftheRoadmapwefollowedupwithcitiesonanindividualbasistolearnmoreaboutwhatwouldbehelpfulWearegratefultothefollowingcitystafffortheinterviewsandindividualconversations
MichaelBaskinChiefPolicyOfficerChattanoogaTN
MelissaBridgesPerformanceandInnovationCoordinatorLittleRockAR
VictoriaCarreoacutenAdministrativeOfficerLasVegasNV
TyroneGrandstrandCommunityDevelopmentPlannerFargoND
TibLaughlinDirectorofGeneralServicesKansasCityKS
TimMorelandDirectorofPerformanceManagementandOpenDataChattanoogaTN
CharlesRosendahlManagementAnalystDentonTX
Thankyou
60
MelissaSiebenAssistantCountyAdministratorKansasCityKS
GinaThompsonDirectorofStrategicServicesLewisvilleTX
JHUFacultycollaboration
GovExalsoconvenedthecitycontactsandfacultypartnersinaseriesofAdvisoryGroupmeetingsThesemeetingsservedtoconnectcitiesandfacultytounderstandoneanothersperspective
StefanBaralBloombergSchoolofPublicHealth
AshleyBernerJHUSchoolofEducation
MelissaDavey-RothwellBloombergSchoolofPublicHealth
JonLinksBloombergSchoolofPublicHealth
TamarMendelsonBloombergSchoolofPublicHealth
KristinMmariBloombergSchoolofPublicHealth
KeshiaPollackPorterBloombergSchoolofPublicHealth
JoshSharfsteinBloombergSchoolofPublicHealth
LizStuartBloombergSchoolofPublicHealth
LindsayThompsonCareyBusinessSchool
DanielWebsterBloombergSchoolofPublicHealth
Partnersindataandevidence-basedpolicy
Wealsospokewithpartnerswhoareexpertsinthefieldofevidence-basedpolicywhosharedtheirknowledgeandresourcesofcurrentandfuturethinkinginevidence-informedthinking
WendyBarrenoAssociateDirectorWilsonSheehanLabforEconomicOpportunitiesUniversityofNotreDame
TiffanyDavenportDirectorofDataPartnershipsBaltimoresPromise
TomiHiersDirectorBaltimoreCivicSiteAnnieECaseyFoundation
ChrisKingsleySeniorAssociateAnnieECaseyFoundation
KathrynOliverAssociateProfessorLondonSchoolofHygieneandTropicalMedicine
Thankyou
61
KristenWatkinsSeniorPolicyAssociateJ-PAL
GovExParticipants
GovExAnalystMiriamMcKinneyanalyzedtheinterviewtranscriptsandconductedtextanalysisSeniorImplementationAdvisorEmilyShawwillintegrateRoadmapactivitieswithadditionalservicedeliveryactivitiesatGovEx
Thankyou
62
ReferencesAbrahamKGampHaskinsR(2017)Thepromiseofevidence-basedpolicymakingReportofthecommissiononevidence-basedpolicymakingWashingtonDCCommissiononEvidence-basedPolicymakingBigmove2Extendingthegreen(nd)Retrievedfromhttpswwwcityoflewisvillecomdoing-businesslewisville-2025big-move-2-extending-the-greenBatesMAampGlennersterR(2017)ThegeneralizabilitypuzzleRetrievedfromhttpsssirorgarticlesentrythe_generalizability_puzzleBuildwithnotfor(2018January)RetrievedfromhttpsbuildwithnotforwordpresscomCairneyPampOliverK(2017)Evidence-basedpolicymakingisnotlikeevidence-basedmedicinesohowfarshouldyougotobridgethedividebetweenevidenceandpolicyHealthResearchPolicyandSystemsdoi101186s12961-017-0192-xCairneyPOliverKampWellsteadA(2016)TobridgethedividebetweenevidenceandpolicyReduceambiguityasmuchasuncertaintyPublicAdministrationReview76(3)399-402Retrievedfromhttpdxdoiorg101111puar12555Centreforpublicimpact(nd)RetrievedfromhttpswwwcentreforpublicimpactorgCernaL(2013)ThenatureofpolicychangeandimplementationAreviewofdifferenttheoreticalapproachesOrganisationforEconomicCo-OperationandDevelopmentRetrievedfromhttpswwwoecdorgeduceriTheNatureofPolicyChangeandImplementationpdfColoradoSprings(nd)Retrievedfromhttpscoloradospringsgovcommunicationspagewhat-expect-when-you-speakupContinuumofcare(coc)program(nd)RetrievedfromhttpswwwhudexchangeinfoprogramscocCouncilpresidentkicksoff$12millionyouthfund(nd)Retrievedfromhttpwwwbaltimorecitycouncilcomcontentcouncil-president-kicks-12-million-youth-fundDurlakJAampDuPreEP(2008)ImplementationmattersAreviewofresearchontheinfluenceofimplementationonprogramoutcomesandthefactorsaffectingimplementationAmericanJournalofCommunityPsychology41327-350Retrievedfromhttpswwwncbinlmnihgovpubmed18322790Fishbonediagram(ishikawa)(nd)Retrievedfromhttpasqorglearn-about-qualitycause-analysis-toolsoverviewfishbonehtmlFivewhysGettingtotherootofaproblemquickly(nd)RetrievedfromhttpswwwmindtoolscompagesarticlenewTMC_5Whtm
References
63
FligsteinNampMcAdamD(2011)TowardageneraltheoryofstrategicactionfieldsSociologicalTheory29(1)1-26RetrievedfromhttpsociologyberkeleyedusitesdefaultfilesfacultyfligsteinFligstein20McAdam20Soc20Theory20PaperpdHeinkelP(2018January12)CitymullspermitsforbikesharingcompaniesRetrievedfromhttpwwwdentonrccomnewsdenton20180112city-mulls-permits-bike-sharing-companiesHowisareamedianincomecalculated(2012October)Retrievedfromhttpswwwhudexchangeinfofaqs843how-is-area-median-income-calculatedLampkinLMampHatryHP(2003)KeystepsinoutcomemanagementUrbanInstituteRetrievedfromhttpswwwurbanorgresearchpublicationkey-steps-outcome-managementMarshDampSmithM(2000)UnderstandingpolicynetworksTowardsadialecticalapproachPoliticalStudies484-21Retrievedfromhttpjournalssagepubcomdoiabs1011111467-924800247McCannL(nd)RetrievedfromhttplaurenellencomMoultonSampSandfortJR(2016)ThestrategicactionfieldframeworkforpolicyimplementationresearchPolicyStudiesJournal45(1)Retrievedfromhttponlinelibrarywileycomstore101111psj12147assetpsj12147pdfv=1ampt=jdkhhcp4amps=e184c625e1855a6b1615ccb99ee27b1c0b3212d8PaynterB(2018January1)What100citiesarelearningfromeachotherbysharingtheirdataRetrievedfromhttpswwwfastcompanycom40518004what-100-cities-are-learning-from-each-other-by-sharing-their-dataPeakacademy(nd)Retrievedfromhttpswwwdenvergovorgcontentdenvergovenmayors-officeprograms-initiativespeak-performancepeak-academyhtmlPinnockHBarwickMCarpenterCREldridgeSGrandesGGriffithsCJRycroft-MaloneJMeissnerPMurrayEPatelASheikhAampTaylorSJC(2017)Standardsforreportingimplementationstudies(stari)ExplanationandelaborationdocumentBMJOpenRetrievedfromhttpbmjopenbmjcomcontentbmjopen74e013318fullpdfProctorESilmereHRaghavanRHovmandPAaronsGBungerAGriffeyRampHensleyM(2011)OutcomesforimplementationresearchConceptualdistinctionsmeasurementchallengesandresearchagendaAdministrationandPolicyinMentalHealth38(2)65-76doi101007s10488-010-0319-7TandonSD(2018)ProgressincommunityhealthpartnershipsResearcheducationandactionRetrievedfromhttpswwwpressjhuedujournalsprogress-community-health-partnerships-research-education-and-actionProvanKGampKenisP(2007)ModesofnetworkgovernanceStructuremanagementandeffectivenessJournalofPublicAdministrationResearchandTheory
References
64
18(2)229-252Retrievedfromhttpsacademicoupcomjpartarticle182229935895SabatierPA(1988)Anadvocacycoalitionframeworkofpolicychangeandtheroleofpolicy-orientedlearningthereinPolicySciences21(3)129-168Retrievedfromhttpwwwjstororgstable4532139SandstromAampCarlssonL(2008)TheperformanceofpolicynetworksTherelationbetweennetworkstructureandnetworkperformancePolicyStudiesJournal36497-524Retrievedfromhttponlinelibrarywileycomdoi101111j1541-0072200800281xabstractSchwartzMampWilsonE(nd)WhocanaffordtoliveinahomeAlookatdatafromthe2006americancommunitysurveyRetrievedfromhttpswwwcensusgovhousingcensuspublicationswho-can-affordpdfShearerJCAbelsonAKouyateBLavisJNampWaltG(2016)WhydopolicieschangeInstitutionsinterestsideasandnetworksinthreecasesofpolicyreformHealthPolicyandPlanning31(9)1200-1211Retrievedfromhttpsdoiorg101093heapolczw052ThelessonsofadministrativedataHigh-profilepolicy-relevantresearchpoweredbyadministrativedata(nd)Retrievedfromhttpswwwpovertyactionlaborgsitesdefaultfilesresources20161114-The-Lessons-of-Administrative-DatapdfToolsandotherneedsforqualitativeandquantitativemethods(2015February)AgencyforHealthcareResearchandQualityRetrievedfromhttpwwwahrqgovprofessionalsquality-patient-safetytalkingqualityassesstoolshtml
WeaverRCampBagchi-SenS(2013)SpatialanalysisofurbandeclineThegeographyofblightAppliedGeography4061-70
References
65
UniversityPartners
Name Title Affiliation Area(s)ofexpertise
Contactinformation
Examplesofworkingwithnon-academicdecisionmakers
WendyPBarreno
AssociateDirector
WilsonSheehanLabforEconomicOpportunitiesUniversityofNotreDame
LEOsfocusareasincludehealtheducationhousingchildwelfareeconomicopportunitycriminaljustice
wbarrenonduduorleondedu
MelissaDavey-RothwellPhD
AssociateScientist
JohnsHopkinsBloombergSchoolofPublicHealth
Health mdavey1jhuedu410-614-5854
BaltimoreCityHealthDepartmentLeadersMarylandDepartmentofHealthAddictionRecoveryServicesinDCExecutiveDirectorsofcommunityorganizationsacrosstheUSthatfocusonHIVSTIsubstanceabuseandimplementationscience
ElizabethStuart
AssociateDeanforEducationandProfessorDepartmentsofMentalHealthBiostatisticsandHealthPolicyand
JohnsHopkinsBloombergSchoolofPublicHealth
Educationsocialpolicystatisticalmethods
estuartjhuedu
Resources
66
Management
Design-ThinkingResources
EngagementGuideforCivicTech
Journeymap
PerformanceManagementResources
DenverPeakAcademy
Proxymeasuresguide
Talkingaboutperformancewithoutcomplicatingit(video)
SettingPerformanceTargetsguide
Datavisualizationguide
Performancemanagementgettingstartedguide
Exercises
Fishboneexercise
5Whys
AdministrativeData
TheLessonsofAdministrativeData(JPAL)
EvaluationResources
Quantitativeandqualitativemethods(USDepartmentofHealthandHumanServicesAgencyforHealthcareResearchandQuality)
KeyStepsinOutcomeManagement(UrbanInstitute)
RCT-YES
Sageresearchmethods
BestPracticesforMixedMethodsResearchintheHealthSciences
ACCORDChecklist
Pilotandimplementationplanning
Benchmarkingexamplesandproxymeasures
Seattleopendataplaybook
Resources
67
Datasciencescopingguide
Resources
68
AboutGovExGovExhelpsgovernmentsusedataandevidencetoimproveresidentsrsquolivesWesupportgovernmentsinbuildingtheirinternalcapacitytooperatesustainableanalyticspracticeswhichimprovecommunityoutcomesinfieldsfrompublicsafetytoeconomicdevelopmentGovExhasaproventrackrecordofworkinginconcertwithpartnerstoprovidetechnicalassistanceandtrainingatscale
SinceourlaunchinApril2015GovExhas
Trainedover2000governmentemployeesinpersonandonlineintopicsincludingdatamanagementperformancemanagementopendatacommunityengagementandanalytics
Providedtechnicalassistancetomorethan100localgovernmentsacrosstheglobeimprovingtheiruseofdatatoachieveefficienciesinservicedeliveryandbetteroutcomesforresidents
Assessedthecapacityofmorethan130governmentalorganizationstodeterminehowleadersstaffandexternalstakeholdersmanageandusedata
Localgovernmentsareincreasinglyonthefrontlinesofaddressinglargescaleproblemswithglobalimplicationssuchasclimatechangepublicsafetyandeconomicinequality
Whileevidence-basedinterventionstohelpsolvethoseproblemsareavailablelocalgovernmentstaffhavelimitedtimeandcapacitytofindandimplementresearch-basedsolutionsAsaresultcitiescontinuetoworkthewaytheyalwayshave--makingincrementalimprovementstoprogramsbutnotalwayscreatingchangeatasystem-level
ThroughresourcedevelopmentandcollaborationwithJHUfacultyGovExissupportingcityleadersinusingdataandevidencetomakedecisionsandcreatinglastingrelationshipswithresearchers
TheRoadmapwasdevelopedbyGovExaspartofitspartnershipwithBloombergPhilanthropiesWhatWorksCitiesinitiative
AboutGovEx
69
- Title page
- Introduction and purpose of the Roadmap
- Using the Roadmap
-
- Gather information about the problem
- Strategically engage key partners
- Review the evidence and collect details
- Decide on a policy to implement based on local factors
- Establish a framework to monitor progress
- Assess data accessibility quality and security
- Design implementation and evaluation
- Share your story and success
-
- Working with a range of evidence
- Thank you
- References
- Resources
- About GovEx
-
whatisdrivingtheobservedtrendsAnalystsshouldbeencouragedandtrainedtoaskquestionsofthedepartmentsproducingthedataandconductsitevisitswhenpossibletodeepentheirunderstandingofthedataPresentationoftheanalysisshouldfocusononeideaatatimetokeeptheconversationfocusedonkeypointsandspecificopportunitiesforimprovementYoumayconsiderdevelopingapublicfacingdashboardtosharethisprogressmorebroadly
AgeneralcomparisonofthestatusquodatawithwhatwasobservedduringthepilotcanprovidedecisionmakerswithaquickunderstandingofthedifferencesbetweentheoldandnewapproachesalthoughthiswillnotshowcausalityOftenthesecomparisonswillincludeoperationalmetricssuchasdaysordollarssavedbutrememberthattheimpactonpeoplewillbemostimportantIfyouareworkingwithanevaluationpartnertheywillsupportthisstep
Usethespacebelowtosummarizetheresultsofyouranalysis
MeasureTargetvalue(seesectiononsettingtargetsabove)
Pilotvalue(whatdidyouobserveduringthepilot)
Historicalaverage(ifavailable)
Results(WhoisbetteroffIsanyonenegativelyimpacted1-2sentencesreassessingyourgoalstatement)
DocumentyourprocessandprogressincludingfeedbackfromtheparticipantsimplementationpartnersandtheprojectteamDidanythingunexpectedoccurduringthepilotArethereinternalandexternalfactorsthatmayaffectyouroutcomesThisiswheretheimplementationoutcomescanbecanhelpfulIftheinterventiondidnotgoasplannedconsiderwhetheritisbecausetheinterventionwasnotacceptableorfeasible
UsethetemplatebelowcreatedbyDenversPeakAcademytocapturethisinformationIfthe306090daytimelinedoesnotmakesenseforyourcityfeelfreetocustomizeit
CurrentState
FutureState
30Day
60Day
90day
Whatwentwell(+)
Whatdidnotgowellshouldbechanged(-)
IfaquestionordiscussionpointarisesthatcannotbeansweredinthemeetinganassignedstaffmembershouldcapturethequestionduringthemeetingAfterwardthefollowupitemsshouldbesenttorelevantpartnersandthenextmeetingshouldbeginwithadiscussionof
Establishaframeworktomonitorprogress
38
thoseitems
Takeaction
DiscussionsaboutwhetheryourindicatorsaremovingintherightdirectionandwhatmightbedrivingtrendsmayleadtoadiscoverythattherearenotenoughresourcesallocatedtothispolicyorissueTheremaybelimitedfundingforstaffprogrammingortechnologythatisdrivingthedatainthewrongdirectionIfCityleadershipisintheroomthisisagreattimetomakethecaseforadditionalfundingduringthebudgetseasonBycollectinganalyzinganddiscussingdataonthispolicyroutinelyyouarebuildingevidencetomakethecasefortherightleveloffundingtoreachtheestablishedtargets
Establishaframeworktomonitorprogress
39
Assessdataaccessibilityqualityandsecurity
Determineaccessibilityandqualityofdata
OnceyouhavecompiledyourwishlistofindicatorsitistimetodeterminetheaccessibilityandqualityofthedataTopreventcomplicationyoumaywishtoprioritizerelevantindicatorsforwhichyouarealreadycollectingdataAvoidthetemptationofadoptingatechnologysolutiontocollectnewdataatthispointsuchasaphotosurveytoolorotherappFornowyouaretryingtodeterminewhichdatasetsarerelevantaccessibleandhighqualityBesuretoworkwithyourkeyinternalpartnerswhileyouareidentifyingyourmeasuresDepartmentstaffwhoworkonprogramsarefamiliarwiththeindicatorsthatwillaccuratelysupporttheoutcomesanddatabaseadministratorswillbevaluablewhendiscussingaccessibilityandqualityofthedata
Abrieflistofconsiderationsfortheaccessibilityofyourdataisbelow
ArewecurrentlycollectingthisdatasetorwillwebecollectingitforthefirsttimeHowfarbackdoestherecordgo(timehorizon)WouldweneednewtechnologytocollectoranalyzethedataCanthedatabecollectedmanuallyorcanitbeautomatedHowfrequentlyisthedataproduced(continuouslyhourlydailyweeklymonthlyquarterlyyearlyetc)IsthedataavailablefortheentireCitypopulationorasubsetofthepopulationHowgranularisthedata(individuallevelCensusblockneighborhoodcitywideetc)DoesthedatacontainsensitiveinformationIsthereaproxymeasurethatwecanusefornowIsthedatamachinereadable
AsnotedbytheWilsonSheehanLabforEconomicOpportunities(LEO)attheUniversityofNotreDameleveragingadministrativedatacansignificantlyreducethecostsandtimelinesofrigorousevaluationCitiescanestablishaninternaldatasharingagreementacrossdepartmentsincludinglanguagethatenablesadministrativedatatobeusedforresearchpurposesCitiescanalsoidentifyexternaladministrativedata-holderssuchasacountyschooldistrictpolicedepartmenthospitaloranotherstatethatmayhaveoutcomedataimportantformeasuringprogramimpactCitiescanleverageexistingrelationshipsandorworkwithresearchinstitutionstodevelopstreamlineddatasharingprocessesandagreementswiththeseexternalentities
Assessdataaccessibilityqualityandsecurity
40
Evaluatethequalityofyourdata
QualityisasimportantasaccessibilityToevaluatedataqualityyoumaywishtocreatearubricliketheonebelow
Value ValueDescription
3 ExcellentDataQualityThedataiscompleteaccurateandupdatedregularly
2 AdequateDataQualityThedataissomewhatcompleteaccurateandupdatedfairlyregularly
1 PoorDataQualityThedataisnotcompleteaccurateorupdatedregularly
ThisisagreattimetobuildonthedatainventorythatyoustartedinthegatherinformationpageSuggestedadditionalfieldstoaddtoyourdatainventorytemplatearebelow
Program DatasetTitle
DatasetDescription DataSource
Departmentorexternalpartner
DoestheCityalreadycollectthisdata
Whatisthenameoftheprogram
Human-readablenameofthedataset
Shouldbeunderstandablebynon-technicalusers
Whatistheoriginalsystemapplicationfilethathousesit
Departmentorexternalpartnerthatmaintainsthedata
Isthisdataalreadycollected
Updatefrequency-ThefileformatforthedatasetTypicallycsvxlsxshapefiletxtetc
Format-Howoftenthedatasetisupdated
Timerange-Theperiodoftimecoveredinthisparticulardataset
Datause-Howisthisdatasetcommonlyused
Datausers-Whotypicallyusesthisdataset
Dataownerstitle-Titleofthedataowner
Dataownersposition-DataOwnersjobtitle
Dataownersemailaddress-DataOwnersemailaddressWhomanagesthedataandorisresponsibleforgrantingpermissiontoaccessthedataWhounderstandswhatthedatasetincludesandcananswerquestionsaboutit
Dataqualityconcerns-Describetheconcernsyoumayhaveaboutthequalityofthedataset
Assessdataaccessibilityqualityandsecurity
41
Division-DivisionwherethedataismaintainedIfmorethanonedivisionisresponsiblelisttheprimaryownerofthedataset
Sensitivedatacomments-DescribetheissuesthatwouldpreventthecityfromsharingthedatapubliclyCommonsensitivityconcernsincludeprivacyviolationssecurityissuesorhighcostorstaffdemandstopostthedata
Publishingstatus-Thistheplacetoputanycommentsregardingthecurrentstatusofthedataset
Publishedlink-Thisisthelinktothepublisheddatasetontheopendataportalorcitywebsite
Reviewdatasecurityandprivacyprotectionprotocols
OftenapolicychangealsoinvolvesachangeinthedatathatisbeingcollectedhowthatdataisbeinghandledwhohasaccesstoitandahostofotheradjustmentsAsyoutakestockofthedataassetsthatareavailableforyourinitiativeitisvitaltoalsothinkthroughwhatwillbedonetoprotectthatdataandtheprivacyofthepeopleitrepresentsAbigpartofgooddatasecurityinvolvesstrongdatagovernancepracticesthatdefineandidentifysensitivedataandconsciouslyworktobalancethevalueandriskofstoringsuchsensitivedataOnewayofevaluatingyourdatacomesfromcaselawrelatedtotheUSFreedomofInformationActwhichwehavetranslatedtothechecklistbelow
1 Woulddisclosureresultinasubstantialinvasionofprivacy2 Haveweconsideredtheextentorvalueofthepublicinterestandthepurposeorobject
oftheindividualsseekingdisclosure3 Istheinformationavailablefromothersources4 Wastheinformationgivenwithanexpectationofconfidentiality5 Isitpossibletomouldreliefsoastolimittheinvasionofindividualprivacy
TheseriskscouldbemitigatedusingavarietyofstrategiesfromaggregatingdatatoobfuscatingindividualstoredactingfieldsthatmayincludePersonallyIdentifiableInformation(PII)orre-evaluatingyourdatacollectionmethodstofindalternativewaysofgettingtheinformationyouneedThesestepsshouldbecodifiedandmanagedbyadatagovernancecommitteeifpossibleForthesensitivedatathatisdeemedessentialcreateaplanaroundpotentialdatabreachesinordertoquicklyandconfidentlyrespondtoemergencysituations
ProactivelypreparingforthesesituationsisoftenthebeststrategyfordatasecuritybutdonotforgetthetechnicalsideInvolveyourITadministratorsinconversationsaboutdatastorageandaccessensuringthattheywillbeprotectedbyenterprise-gradecybersecuritysystems
Assessdataaccessibilityqualityandsecurity
42
Assessdataaccessibilityqualityandsecurity
43
Designimplementationandevaluation
Designtheimplementation
NowthatyouhaveyouraccountabilityplanandkeyindicatorsitistimetogetspecificaboutyourplanforimplementationThetemplatebelowwillhelpyouthinkthroughthespecificmechanismsofimplementingpolicychange
UnderstandingorganizationsandsystemsiscriticaltounderstandingandmakingdecisionsaboutwhereandhowinterventionsshouldbeappliedApolicyimplementedinasinglesettingmayshowpromisingresultsbutimplementingmultiplepoliciesandprogramsatthesametimewithdifferentactorsresponsibleforeachaspectcanhelpensuresustainablebehaviorchange
InSomervilleMAShapeUpSomerville(SUS)isacitywidepublichealthprogramthatisimplementedacrossmultiplelevelsofinfluenceandsystemsthroughinterventionsbeforeduringandafterschooltoincludechangestobuiltenvironmentsuchasbikelanesregionalmasstransportationandphysicaleducationequipmentinschoolsandgymstoincreasephysicalactivitywellnesspoliciesandschoolgardensnutritioneducationduringandafterschoolandhealthyrestaurantprogramstosupporthealthyfoodandhealthyeatinghabits
Usethetemplatebelowtoplanyourimplementation
Policyimplementationplanningtemplate
ListthespecificdesiredoutcomesInthepreviousstepyouworkedwithdepartmentstafftoestablishspecificmeasurableoutcomesandinterimindicatorsPrioritizetheindicatorsthatwilldemonstrateprogressUsethespacebelowtocapturethedesiredoutcomesofthepolicychange
ExampleShapeupSomervillebeganwiththegoalofpreventingobesityin1stthrough3rdgraders
Yourturn
____________________________________________________________________________
____________________________________________________________________________
Designimplementationandevaluation
44
SpecifythetargetpopulationAreyoudesigninganinterventionthattargetsonlycertainsitesAreyouplanningonimplementingaprogramorinitiativeforaspecifictargetaudienceClearlyspecifythepopulationyouplantotarget
IfyouplantoconductarigorousimpactevaluationasnotedbytheWilsonSheehanLabforEconomicOpportunitiesitisimportanttoestablishconcreteobjectiveeligibilitycriteriaandensurethatthesecriteriaareappliedconsistentlybyallstaffduringscreeningandintakeItisalsovitaltoidentifytheindividualswhoareeligibleforaparticularprogramorserviceevenifeveryoneeligiblecannotbeservedbyexistingresourcesThishelpscitiesidentifyanyunmetneedsandguidefutureresourceallocationwhilealsocreatingthepossibilityofarigorousimpactevaluationbyidentifyingapotentialcomparisongroupFinallywhenconductingscreeningandintakeitisimperativetocollectandmaintainbasicinformation(egnamedateofbirthgender)aboutallprogram-eligibleindividualsnotjustthosewhoparticipateintheprogram
ExampleIntheShapeUpSomervilleexampletheprogrambeganbytargeting1stthrough3rdgraders
Yourturn
____________________________________________________________________________
____________________________________________________________________________
DefinethelocationWherewillthepolicychangetakeplaceWillimplementationtakeplaceatonesiteormultiplesites
ExampleShapeUpSomervilleinterventionsweredeliveredinschools
Yourturn
____________________________________________________________________________
____________________________________________________________________________
ClarifyimplementationpartnersandarticulatetheirrolesWhatarethekeyorganizationsinstitutionsorpeopleresponsibleforimplementationTheprojectteamshouldbeasclearandspecificaspossibleinprovidingadescriptionoftheconcreteactivitiesthatimplementationpartnersaremeanttocarryoutPracticeprofilesmaybeahelpfultooltohelpdefinewhatisrequiredoftheactorswhoareresponsibleforimplementingtheintervention
Designimplementationandevaluation
45
ExampleInSomervillemayoralleadershipandengagementofmultiplecityagencies(publichealthprimarycareschoolsCambridgeHealthAllianceetc)streamlinedandconnectedthevariousShapeUpSomervilleinitiativesandgrantswhichhelpedtokeepthetalentandenergyfocusedonhealthyeatingandactivelivingforallsectorsofSomervilleSUSpartnersultimatelyincludedschoolsnonprofitsCityandstateagenciesandnationalfoundations
Yourturn
____________________________________________________________________________
____________________________________________________________________________
DescribetheinterventionWhatarethestepsactivitiesorprocessesRememberthatthemechanismthespecificstepsorprocessthatleadtobehaviorchangeiskeytounderstandingtheeffectivenessofthepolicychangeandsharingtheresultswithothers
ExampleGoalsofShapeUpSomervilleweretoinfluenceeverypartofanearlyelementaryschoolchildrsquosdayndashbeforeduringandafter-schooltopromotehealthyeatingphysicalactivityandtheirweightstatusInterventionincludedfoodserviceenhancementstoimprovebreakfastandlunchandstaffprofessionaldevelopmentwalktoschoolactivitiesSUSclassroomcurriculumandprofessionaldevelopmentschoolwellnesspolicydevelopmentSUSafter-schoolcurriculumandprofessionaldevelopmentoutreachandeducationtohomethroughmaterialsforumseventsldquoSUSapprovedrdquorestaurantscommunityoutreachandcapacitybuildingthroughpolicydevelopmenttrainingsmediaplacements
Yourturn
____________________________________________________________________________
____________________________________________________________________________
ClarifytimelineandmilestonesfordeliveryandevaluationHowfrequentlywilltheservicesinterventionbedeliveredHowlongwilltheprogramrunWhatisthetimetoexpectedimpacts(eg12monthsafterprogramcompletion)
Designimplementationandevaluation
46
ExampleShapeUpSomervilleledbyTuftsUniversityFriedmanSchoolofNutritionScienceandPolicyconductedaquasi-experimental(nonrandomized)interventionstudyamong1stthrough3rdgradestudentsinSomerville(intervention)andtwocomparisoncommunitiesfor9-monthsduringoneschoolyear(fall2002ndashspring2003)ThestudyshowedpositiveassociationoftheinterventiononchangeinBMISUSreleasedreportssummarizingevaluationsprogresstodateplansforadditionalprojectsandimpactstenandfifteenyearsintotheinitiative
Yourturn
____________________________________________________________________________
____________________________________________________________________________
PlanyouranalysisTheprevioussectionaddressedestablishinganaccountabilityframeworktomonitorprogressandimplementationDeterminewhichindicatorswillbediscussthroughthatprocesswhowillsharethedataandanalysisandhowfeedbackfromthoseconversationswillbeincorporatedbytheprojectteam
ExampleTheMayorrsquosofficeimplementedadata-drivendecisionmakingapproachcalledSomerStatwhichwassynergisticwiththeapproachestakenbytheresearchandevaluationteamsdeployedforthespecificgrants
Yourturn
____________________________________________________________________________
____________________________________________________________________________
PlanforsustainabilityIfthepolicyandprogramchangeissuccessfulhowwilltheprojectteamsustainandexpandtoadditionalsitesorpopulations
ExampleAlthoughmostofthepartnershipsforgedintheearlyyearsweremaintainednewgrantsandcollaborationsexpandedattentiontoolderpublicschoolstudentsactivetransportationandtheGreenLinesubwayextensionthebuiltenvironmentfarmerrsquosmarketeffortssuchasldquoGrowninSomervillerdquowithUnionMainStreeteconomicdevelopmentandhealthdisparitiesaswellasdatacollectionmonitoringandsurveillance
Yourturn
Designimplementationandevaluation
47
____________________________________________________________________________
____________________________________________________________________________
Pilotplanningtemplate
IfyouplantoconductapilotfirstdeterminethepurposeofthepilotDoyouwanttoobservewhetherthepolicychangeresultsinthedesiredbehaviorchangeAreyouconcernedaboutfeasibilityscalabilityorcostDoyouwanttotestwhetheryourdesiredresultscanbeachievedthroughthispolicyapproachClarifythegoalofthepilotandthequestionsyouhopetoanswerThiswillhelpyouidentifythekeyindicatorsthatyouwillusetomeasurewhetherthepilotissuccessfulBeforebeginningestablishabaselineofcurrentperformanceandsetyourtargetsaccordinglybasedonwhatyouhopetoaccomplishthroughthepilot
NextestablishthepilotplanforyourprojectteamincludingmilestonesforteamcheckinstodeterminewhethertomoveaheadorchangecourseThisisseparatefromthemeetingsthatwilloccuraspartoftheaccountabilityframeworkUsethespacebelowtocreateyourtimelineanddevelopyourmeetingschedule
Task Personresponsible Startdate Checkindate Deliverable
Planyourevaluation
WhileyouaredesigningtheimplementationitisalsotimetobegintodevelopyourplanforanalysisAsnotedbytheWilsonSheehanLabforEconomicOpportunitiesdecisionsaboutprogramsetup-includingoutreachscreeningintakedatatrackingandenrollment-candeterminewhetherornotarigorousimpactevaluationwillbepossibleAsdiscussedcityleadersmayfindithelpfultoincludeevaluationpartnersinearlyconversationsaboutprogramsetupDuringpreviousstagesyoumayhaveworkedwitharesearchertoplanyourproposedpolicychangeYoushouldbeclearaboutthepurposeoftheevaluationanddevelopanevaluationplanBelowareadditionalconsiderations
Causationvstrendanalysis
IntheEstablishaframeworktomonitorprogressstagetherewasguidanceontrackingtrendsovertimeItisimportanttonoteherethatmonitoringdataisdifferentfromanalyzingacausaleffectTrendanalysiswillillustratethatchangesareoccurringandmayleadtoquestionsthatcouldbeansweredbyyourevaluationbuttrendanalysisalonedoesnotprovideevidenceaboutwhatisdrivingthosechangesYoumayseechangesinyour
Designimplementationandevaluation
48
outcomeindicatorsbasedonvariablesoutsideoftheCityscontrolForinstanceachangeinfederaltradepolicycanhaveanimpactemploymentratesinyourCityoutsideofyourCitysnewjobtrainingandplacementprogram
ToaddresscausalityyoumustcomparetheoutcomesofindividualswhoreceivedtheinterventionwiththosewhodidnotWithoutacomparisongroupofsimilarindividualswhowerenotexposedtotheinterventionitisimpossibletoknowwhichfactorsareresponsiblefortheoutcomesRandomizedControlTrials(RCTs)inwhichindividualsarerandomlyassignedtotreatmentandcontrolgroupsarethegoldstandardinprovingcausalityIfyoucandemonstratethatoutcomesimprovedforthosewhoreceivedtheinterventionanddidnotimproveatallorimprovedlessforacomparisongroupyoucanbesomewhatcertainthattheinterventionwasthecause
Examplesofothernon-causalimpactevaluationsthatarerigorousandreliableincludecertainquasi-experimentalstudiesusingcomparisongroupdesignssmartinterruptedtimeseriesanalysesthatmodeltrendsovertimeinwaysthatallowestimationofinterventioneffectswelldesignedpre-posttestanalysesneedsassessmentsimplementationstudiesetcFormoreinformationondifferenttypesofevaluationsseetheSageResearchMethodswebsite
Lookfordifferencesingroups
BuildintimetoalsoexaminethepilotdatatoseeifgroupsareexperiencingdifferentresultsReviewingperformancebasedonoutcomemeasuresforapopulationinaggregatemayhidethesubtlewaysthatsomepeoplemaybeaffecteddifferentlysoitisnecessarytocarefullyanalyzethedatafordisparateoutcomesForinstancetheMovingtoOpportunity(MTO)studythatexaminedtheimpactofhousingvouchersonfamilieswholivedinhigh-povertyneighborhoodsfoundthatchildrenwhomovedbeforetheageof13hadbettereducationandhigherincomesasadultsHoweveradultsdidnotseeimprovededucationandemploymentoutcomes
Disaggregatingdataintomorespecificcategoriesofraceandethnicitycanhelpunderstandtrendsanddisparitiesinamoregranularwaywhichcanhelpdecisionmakerstargetpoliciesandinterventionsinamoreequitableway
EarlierintheRoadmapweworkedtoidentifystakeholdersandwhomightbepositivelyandnegativelyimpactedbyachangeinpolicyUsetheresultsofthatexercisetoinvolvethecommunityininterpretingdataanalysisandtogatherqualitativefeedbackfromthecommunitytocallattentiontoissuesForinstanceanincreaseinyouthdetentionmaybeexplainedbyinefficienciesorlackofaccesstotransportation
Designimplementationandevaluation
49
Designimplementationandevaluation
50
Shareyourstoryandsuccess
Sharetheresults
Storytelling
AnintegralstepinyourstrategyissharingtheresultsofyourpilotorpolicychangewithyourcommunityThisincludesdevelopingmessagingaroundtheworkofyourorganizationandcreatingaplantodisseminateyourstoriesWhytellstoriesaboutyourwork
Informandorunderstandasubjectorsituation
Giveneededcontextofdataorbitsofinformation
Relatablestoriesarememorableforreadersviewers
ReadersorviewerswillseetheinformationasmorecompellingwhenthereisademonstratedandrelatablechangeorimpactonthemselvesortheirfriendsneighborsandcolleaguesStoriesareaboutpeopleandgovernmentisfundamentallyabouthelpingpeopleSotellthestoriesofthepeopleyouandyourcitycolleagueshelped
Keepthesetipsinmindasyoudevelopyourstorywithyourteam
WhatwereyourfindingsandwhyshouldanyonecareTellyouraudiencehowtheproblemwassolvedWhathastheimpactbeenonresidentsandcitystafftodateWhatwasthesuccessandweretheremultiplesuccessesalongthewayIfyouhavepreliminaryresultsordatashowingimprovementtosharethisisthetimetodoit
WhowasinvolvedandimpactedReviewthelistofstakeholdersHowwereeachofthemimpactedbytheprogramHumanizeyourstoryandimpactbyaddingadditionalvoicesThesevoicescanbeoneofthefollowing
Apersonorfamilywhowaspositivelyaffectedbythechangesuccess
Aninvolvedthird-partydescribingboththeproblemandthesolution(ieanorganizationnonprofitprovidersbusinesstradeassociationsportsteametc)whoispartofthecommunitythatthegovernmentworkedwithonthechangesuccess
Someonefromthegovernmentitselfwhocanspeakaboutthechangesuccess
TellmoreaboutyourcityDescribeyourcityincludingpopulationdemographicsindustriesetcWhatareitschallengesWhatisitknownforWhatmakesituniqueHowisitsimilarordifferentfromothercitiesintheUS
Shareyourstoryandsuccess
51
DescribeyourcityrsquoschallengeWriteoutthechallengeyourcityisfacing(asdefinedinyourproblemstatement)HowdoesitaffectresidentsHowdoesthisproblemaffecttheworkofcitystaffmembers
AddthedetailsProvidedetailsonyourappliedresearchstrategyTalkaboutwhatyourcityhasdonenthepasttoresolvetheissueDiscusshowyourstrategyhaschangedwhenyoubegantoadoptthisapproachWasmoneysavedorinvested
Usethisstorymapexercisetoorganizeyourthoughtsandguaranteethatyouincludealloftheelementsofatraditionalstory
AfteryoudevelopyourmessagingcreateacommunicationsmarketingandoutreachstrategydesignedtospreadyourmessageYourstrategyshouldwalkyouthroughthefollowingkeyactivities
1 IdentifyandprofileyouraudienceWhencreatinganddisseminatingyourmessageitisalwaysgoodtothinkaboutyouraudienceWhoexactlywouldbeinterestedinwhatyouhavetosayThegeneralpopulationofthecityPeoplewhoworkinnonprofitsBusinessownersItmattersbecausenoteveryonewholivesinacitymaybethetargetoftheinformationBusinessownersmaybemoreinterestedindatashowingeffectsonthebusinessdistrictthanothersandsoonWewillcoverthislaterbutthisstepwillalsohelpyoutothinkofthemosteffectivewaytoreachyouraudience
2 Defineyourcalltoaction(ifyouhaveone)SpendsometimeconsideringwhyyouareinterestedintellingyourstoryDoyousimplywanttoupdatethepubliconyourprogressorcelebrateyoursuccessesAreyouinterestedinyouraudiencechangingitsbehaviororsupportingaspecificinitiativebeinglaunchedbyyourorganizationIfyouwanttohaveyouraudienceactorbehaveinacertainwaybespecificandincorporateitinyourmessaging
Shareyourstoryandsuccess
52
3 SelectperformancemeasuresThinkabouthowyouwoulddetermineifyourcampaignissuccessfulandselectappropriateperformancemeasuresYouwillrevisitthesemeasuresattheendofyourcampaigncommunicationsplanimplementationtodeterminewhetheryouneedtomodifyyourcommunicationsandoutreachstrategyforthefuture
4 MethodstocommunicateyourstoryTherearemanywaystocommunicatesuccessofyourprojectHerearesomesuggestionsonreachingaudiencesItisimportanttodeterminewhichmethodsarethemosteffectiveforyourdesiredaudiencetoreceiveinformationDoyourresearch(PewResearchCenterhasexcellentstudiesonthereachofmediadigitalmediaandmarketing)ForexampleifseniorsareyourtargetpopulationandtheyarelessinclinedtouseInstagram(thatistrue)youmaywanttoconsiderFacebookorthenightlylocalnewsbothofwhichareusedmoreoftenbyseniorsBelowisalistofsomemethodsandtoolsyoucanuse
i Pressreleasestolocalnewspapersandtvstations
ii ShortsmartphonevideosAgreatwaytoengageanaudienceisthroughvideoAskonequestionofsomeonewhoispartofthestoryMaybethereisasocialworkerwhocandiscusshungerinyourcityThesevideoscanbereusedonmultipleplatforms
iii Long-formwriting
iv TellingindividualstoriesthroughaseriesofphotosandshortnarrativesforpublicationonFacebookorInstagram
v Op-edsopinionpiecesplacedinthelocalpaper
vi Emailupdatestoyourlist
vii Connectingwithappropriateorganizationsthatareinvestedinthetopicandworkwiththecommunityandaskthemtodisseminateinformationtotheirmembers
viii EmployingldquoWordofMouthrdquo(WOM)tacticsbycreatingtownhallsorfreeeventsforresidentstoattendintheirownneighborhoods
5 DeterminethetypeofcollateralUsuallycampaignsthattargetresidentsrequireseveralavenuestoreachthemPeoplearenotalwaysinthesameplaceorspaceHavingakeenunderstandingofyouraudiencewillassistyouwithyourchoicesonwhatneedstobedevelopedSometimesthatmeansimagesforsocialmediaaonepageflierthatgivespeopleinformationonattendinganeventoranin-depthbrochure
Shareyourstoryandsuccess
53
6 ImplementyourstrategyKeepinmindstrategiescanbefluidSometimessomethingmaynotturnoutasexpectedevenifduringplanningsessionsitwasldquosupposedtordquoItisokYoucananalyzewhathappenedduringadebriefsessionandonceyoursurveysandothermeasurementshavebeenfinalized
Visualizations
Withtherightvisualizationyoucanreviewandshareinformationinsightsanalysisperformanceandtrendsallofwhichisvaluabletoacross-sectionofthepublicButwhichvisualizationisrightforyouTochoosetherighttypeofchartgraphdeterminethetypeofquestionyouwanttoanswerSomecommoncharttypesarebelowandmoreinformationcanbefoundhere
Exploreadditionalanalyticaltechniques
Duringthegoal-settingphaseyouestablishedoutputoutcomeandimplementationmeasuresBasedontheresultsofyourpilotyoumayneedtoadjustyourmeasuresWiththepilotdatainhandyoumayhavesomeideasofhowfutureanalysescouldbedifferentandpotentiallyimprovethequalityofyourprogramTakesometimetobrainstormotherrelevantdatasetsthatyoudidnothavethetimetopursueearlierincludingmessydepartmentaldatathatneedsmoreattentionorsourcesofpublicdatafromthestateandfederalgovernmentnonprofitsandevenprivatecompaniesthatcouldsupplementyourunderstandingofanissueUsingmachinelearningtechniquesyoumightbeabletobetter
Shareyourstoryandsuccess
54
predictdemandforhomelessservicesusingNOAAweatherforecastspreemptwatermainbreaksusingsoilstratigraphyorinvestineconomicdevelopmentbymonitoringrealestatelistingdata
Advancedanalyticaltechniquescanalsohelpyourimplementationrunmoreeffectivelybyintegratingdata-informedtargetinganddecisiontoolsintotheprocessByautomatingroutineworkflowsandsurfacingsubtletrendsthesetoolscanhelpyouiterateinamoreprecisemannerandprovidekeystrategicguidancetofront-lineemployeesthatwouldotherwisebeunavailableThesesortsofdata-science-enhancedprocessescanbeclassifiedasoneofsixprojecttypesasexploredfurtherbytheNewOrleansLANOLAlyticsteamFormoreguidanceonscopingasimilarprojecttheGovExadvancedanalyticsscopingguidegivesyouastep-by-stepoverviewofhowtobesuccessful
Usethespacebelowtolistyourideasforpossibleadvancedanalyticsprojectsbytype
Typeofproject Ideas
Findingtheneedleinthehaystack
Prioritizingforimpact
Earlywarningtools
Betterquickerdecisions
Optimizingresourceallocation
Experimentingforwhatworks
Keepthefeedbackloopgoing
GatherinformationonpublicsentimentaboutpolicychangethroughfocusgroupssurveysandpublicmeetingsledbyadvisorygroupmembersNotewhethernewinformationisavailablethatcanhelpanalyzeorcontextualizeyourprogress(egfederalorstatedatareports)Sharetheresultsofyourpilotandthepolicychangeprocesswithothercitiesandresearchers
Shareyourstoryandsuccess
55
WorkingwitharangeofevidencePresentlythereisalotofattentionandinvestmentinevidence-basedpolicymakingincludingthefederalbipartisanCommissiononEvidence-BasedPolicymakingthePewCharitableTrustsreportsandtoolkitsonHowStatesEngageinEvidence-BasedPolicymakingandtheLauraandJohnArnoldFoundationsupportedEvidence-BasedPolicymakingCollaborativeManyoftheseinitiativesfocusonrigorousevaluation--meaningrandomizedcontrolledtrials(RCT)orotherexperimentalmodels--asthebestkindofevidencetouseHoweverthisbodyofevidenceisofteninsufficientforthespecifickindsofquestionsthatgovernmentscanhavewhenmakingadecisionaboutpolicyInthisguideweproposeanapproachtosupportgovernmentsinincorporatingadditionaltypesofevidenceintotheirdecisionmaking
OnewaytounderstandarangeofevidenceistouseNestarsquosStandardsofEvidencetoframeanddefineevidenceTheBritishnonprofitNestaworkstoinjectpositivechangeintothewayUKgovernmentpoliciesandprogramsfunctionOneoftheirkeyareasofworkhasbeendevelopingWhatWorksCentresorganizationsthatcollectcommissionsynthesizeanddisseminateevidenceaboutinterventioneffectivenessacrossarangeofsocialimpactareasNestadevelopedthefollowingStandardsofEvidencetoprovideausefulwaytodocumenthowdifferentkindsofinformationdemonstratedifferentlevelsofevidence
Workingwitharangeofevidence
56
Theimportanceofpotentiallybiasedinformation
Onechallengepresentedbyanevidenceclassificationthatacknowledgesthevalueofnon-scientificformsofevidenceistofigureouthowtobalanceallofthepiecestogetherwhenyouusethemTounderstandhowtodothiswehavetothinkabouthowtopositionpotentiallybiasedinformationindecisionmaking
WhenwelookatinformationwhichhasnotbeenrigorouslyevaluatedweacceptthatthisinformationmightbebiasedThewholereasonweconductrigorousevaluationistoremovebiasfromourobservationsWeconductstructuredtrialstoproduceresultsthatareconsistentandgeneralizableIfafindingisunbiasedthisaccuracymdashthislackofbiasmdashcanbeprovedbecauseanyindependenttestercanreproduceitwhentheyfollowanidenticalprocess
YoucanlookatpotentiallybiasedinformationasjustinformationthatisearlieroninitspathtorefinementreplicationandverifiedaccuracyApieceofinformationbasedonanyoneindividualrsquosobservationsislikelytobebiasedbecauseitiscomingfromjustonesourceBiasisreducedbygatheringtogetherlotsofindependentsourcesofinformationandfiguringoutitsmostcommonattributes
MaybethebiasedinformationmakesaclaimthatistoolargemdashitsayssomethingistrueunderallconditionswhenitisreallyonlytrueundercertainconditionsorthataneffectofaninterventionisbiggerthanitstrueaverageorthataneffectlastsovertimewhenitdoesnotPerhapschangingtheclaimtobemorespecificmdashcriticallyreviewingtheinformationtoseeifspecificconditionsmustbemettoachievetheoutcomesmdashwouldmaketheclaimmoreaccurate
IfyoutestedyourclaimfounditdidnotholdupinsomecaseschangedittesteditagainuntilyourclaimrepeatedlyheldupthenyoucoulddemonstratethattheclaimwasnolongerbiasedbutaccurateThisprocessoftestingimprovingtestingandvalidatingiswhatwegetfromstructuredscientificstudy
Withindividualpiecesofinformation--crediblehypothesescasestudiesexpertopinion--youcannotassumethatkindofaccuracyAtthesametimetheseearlyobservationsarefoundationaltoanyreplicablescientificobservationsWewillnevergetanywhereifwedonotstartwiththesepiecesofinformationeveniftheymightbebiased
Combininglevelsofevidencetogetthemostdetailedinformationforyourcity
ItisespeciallyimportanttoacceptdifferentformsofevidencewhenyoufacequestionsthathavenotyetbeentestedbyindependentstudiesSomeofthemostimportantquestionsrelatetothingsthatareveryspecifictoyoursituationYoufaceaparticularcombinationof
Workingwitharangeofevidence
57
issuesactorstrendsandresourcesthatisultimatelydifferentfromeveryotherplaceintheworld
ThisisimportantbecauseaspolicypractitionersandnotscientistsyoucaremoreaboutyourspecificcasethanahypotheticalaveragegeneralcaseYoucarelessaboutthetotallikelyrangeofimpactsfromapolicythanhowthissingleimplementationwillworkYouwantinformationaboutpoliciesthatwillprovideyouwiththebestpossiblepredictionabouthowtheywillworkinyourcontext
ByputtingdifferentkindsofevidenceintothesameframeworkNestarsquosStandardsofEvidenceimplythatwhenyoucannotgetrigorouslytestedexperimentallyverifiedinformationitisappropriatetoworkwithwhatwedohaveThisisrelevantforthemunicipalimplementationcontextbecausenostudypulledfromaliteraturereviewwillaccountforallofthevariablesyoufaceinyourowncontext
Intheabsenceofhavingastudythatpredictshowaninterventionworksinyourspecificcontextyouwillneednon-scientificinformationtotrytofillinthosegapsYouwillneedtogatherstoriesfromresidentsstaffmembersandotherstakeholdersthatsitonLevels1or2ofthestandardsofevidence
Someofthekindsofinformationyouwillprobablygetmoreeasilyfrompeoplethanstudiesinclude
howpeopleincitieslikeyoursreactedtoaninterventioninpublicforumshowpeoplewhoimplementedaninterventioninacitylikeyoursfiguredouthowtogetitpassedhowthecitycommunicatedpubliclyabouttheapparentbenefitsanddrawbacksoftheinterventionthepoliticalacceptabilityoftheintervention
Youcanremainsimultaneouslyawarethattheinformationyouaregatheringthroughindividualstoriesisbiased--notgeneralizableinformedbyspecificperspectives--whilealsorecognizingthathavingsomeinformationaboutsomethingimportantisbetterthannoinformationEvidencefromthefirsttwoevidencelevelsareessentialbuildingblocksofknowledgeandperfectlyusefulwhenyoucannotgetdetailedunbiasedknowledge
Youhavetostartsomewhereandyoushouldalwaysstartwithevidenceyouhaveaboutyourveryspecificconditionsthatyouknowareimportantforsuccess
Workingwitharangeofevidence
58
Thankyou
Interviewswithcities
GovExheldsemi-structuredinterviewswithsixstaffmembersinfourcitiestoestablishtheinitialstepsoftheRoadmapWeaskedrespondentstotalkaboutaninitiativeorpolicythattheyhavebeenresponsibleforimplementingintheircityRespondentsfromthreecitiesspokeabouttheireffortstoaddresshomelessnessandrespondentsfromonecityspokeaboutblightAfterrespondentsidentifiedtheproblemtheyspokeabouthowthecityprioritizedthisissuethevariousstepsinvolvedandbarrierspresent
GovExconductedtextanalysisoftheseinterviewswhichrevealedthatpeoplewasthemostcommonlyusedtermduringtheseconversationswhendiscussingthestepstochangeaprogramorpolicyThisisathemethroughoutthisRoadmapThefirstsectionbeginswithidentifyingtherelevantpeopleTheconversationsaroundevidencearefocusedonhumanbehaviorratherthanmethodologyorstrengthofevidence
Thankyou
59
OncewehadadraftoftheRoadmapwefollowedupwithcitiesonanindividualbasistolearnmoreaboutwhatwouldbehelpfulWearegratefultothefollowingcitystafffortheinterviewsandindividualconversations
MichaelBaskinChiefPolicyOfficerChattanoogaTN
MelissaBridgesPerformanceandInnovationCoordinatorLittleRockAR
VictoriaCarreoacutenAdministrativeOfficerLasVegasNV
TyroneGrandstrandCommunityDevelopmentPlannerFargoND
TibLaughlinDirectorofGeneralServicesKansasCityKS
TimMorelandDirectorofPerformanceManagementandOpenDataChattanoogaTN
CharlesRosendahlManagementAnalystDentonTX
Thankyou
60
MelissaSiebenAssistantCountyAdministratorKansasCityKS
GinaThompsonDirectorofStrategicServicesLewisvilleTX
JHUFacultycollaboration
GovExalsoconvenedthecitycontactsandfacultypartnersinaseriesofAdvisoryGroupmeetingsThesemeetingsservedtoconnectcitiesandfacultytounderstandoneanothersperspective
StefanBaralBloombergSchoolofPublicHealth
AshleyBernerJHUSchoolofEducation
MelissaDavey-RothwellBloombergSchoolofPublicHealth
JonLinksBloombergSchoolofPublicHealth
TamarMendelsonBloombergSchoolofPublicHealth
KristinMmariBloombergSchoolofPublicHealth
KeshiaPollackPorterBloombergSchoolofPublicHealth
JoshSharfsteinBloombergSchoolofPublicHealth
LizStuartBloombergSchoolofPublicHealth
LindsayThompsonCareyBusinessSchool
DanielWebsterBloombergSchoolofPublicHealth
Partnersindataandevidence-basedpolicy
Wealsospokewithpartnerswhoareexpertsinthefieldofevidence-basedpolicywhosharedtheirknowledgeandresourcesofcurrentandfuturethinkinginevidence-informedthinking
WendyBarrenoAssociateDirectorWilsonSheehanLabforEconomicOpportunitiesUniversityofNotreDame
TiffanyDavenportDirectorofDataPartnershipsBaltimoresPromise
TomiHiersDirectorBaltimoreCivicSiteAnnieECaseyFoundation
ChrisKingsleySeniorAssociateAnnieECaseyFoundation
KathrynOliverAssociateProfessorLondonSchoolofHygieneandTropicalMedicine
Thankyou
61
KristenWatkinsSeniorPolicyAssociateJ-PAL
GovExParticipants
GovExAnalystMiriamMcKinneyanalyzedtheinterviewtranscriptsandconductedtextanalysisSeniorImplementationAdvisorEmilyShawwillintegrateRoadmapactivitieswithadditionalservicedeliveryactivitiesatGovEx
Thankyou
62
ReferencesAbrahamKGampHaskinsR(2017)Thepromiseofevidence-basedpolicymakingReportofthecommissiononevidence-basedpolicymakingWashingtonDCCommissiononEvidence-basedPolicymakingBigmove2Extendingthegreen(nd)Retrievedfromhttpswwwcityoflewisvillecomdoing-businesslewisville-2025big-move-2-extending-the-greenBatesMAampGlennersterR(2017)ThegeneralizabilitypuzzleRetrievedfromhttpsssirorgarticlesentrythe_generalizability_puzzleBuildwithnotfor(2018January)RetrievedfromhttpsbuildwithnotforwordpresscomCairneyPampOliverK(2017)Evidence-basedpolicymakingisnotlikeevidence-basedmedicinesohowfarshouldyougotobridgethedividebetweenevidenceandpolicyHealthResearchPolicyandSystemsdoi101186s12961-017-0192-xCairneyPOliverKampWellsteadA(2016)TobridgethedividebetweenevidenceandpolicyReduceambiguityasmuchasuncertaintyPublicAdministrationReview76(3)399-402Retrievedfromhttpdxdoiorg101111puar12555Centreforpublicimpact(nd)RetrievedfromhttpswwwcentreforpublicimpactorgCernaL(2013)ThenatureofpolicychangeandimplementationAreviewofdifferenttheoreticalapproachesOrganisationforEconomicCo-OperationandDevelopmentRetrievedfromhttpswwwoecdorgeduceriTheNatureofPolicyChangeandImplementationpdfColoradoSprings(nd)Retrievedfromhttpscoloradospringsgovcommunicationspagewhat-expect-when-you-speakupContinuumofcare(coc)program(nd)RetrievedfromhttpswwwhudexchangeinfoprogramscocCouncilpresidentkicksoff$12millionyouthfund(nd)Retrievedfromhttpwwwbaltimorecitycouncilcomcontentcouncil-president-kicks-12-million-youth-fundDurlakJAampDuPreEP(2008)ImplementationmattersAreviewofresearchontheinfluenceofimplementationonprogramoutcomesandthefactorsaffectingimplementationAmericanJournalofCommunityPsychology41327-350Retrievedfromhttpswwwncbinlmnihgovpubmed18322790Fishbonediagram(ishikawa)(nd)Retrievedfromhttpasqorglearn-about-qualitycause-analysis-toolsoverviewfishbonehtmlFivewhysGettingtotherootofaproblemquickly(nd)RetrievedfromhttpswwwmindtoolscompagesarticlenewTMC_5Whtm
References
63
FligsteinNampMcAdamD(2011)TowardageneraltheoryofstrategicactionfieldsSociologicalTheory29(1)1-26RetrievedfromhttpsociologyberkeleyedusitesdefaultfilesfacultyfligsteinFligstein20McAdam20Soc20Theory20PaperpdHeinkelP(2018January12)CitymullspermitsforbikesharingcompaniesRetrievedfromhttpwwwdentonrccomnewsdenton20180112city-mulls-permits-bike-sharing-companiesHowisareamedianincomecalculated(2012October)Retrievedfromhttpswwwhudexchangeinfofaqs843how-is-area-median-income-calculatedLampkinLMampHatryHP(2003)KeystepsinoutcomemanagementUrbanInstituteRetrievedfromhttpswwwurbanorgresearchpublicationkey-steps-outcome-managementMarshDampSmithM(2000)UnderstandingpolicynetworksTowardsadialecticalapproachPoliticalStudies484-21Retrievedfromhttpjournalssagepubcomdoiabs1011111467-924800247McCannL(nd)RetrievedfromhttplaurenellencomMoultonSampSandfortJR(2016)ThestrategicactionfieldframeworkforpolicyimplementationresearchPolicyStudiesJournal45(1)Retrievedfromhttponlinelibrarywileycomstore101111psj12147assetpsj12147pdfv=1ampt=jdkhhcp4amps=e184c625e1855a6b1615ccb99ee27b1c0b3212d8PaynterB(2018January1)What100citiesarelearningfromeachotherbysharingtheirdataRetrievedfromhttpswwwfastcompanycom40518004what-100-cities-are-learning-from-each-other-by-sharing-their-dataPeakacademy(nd)Retrievedfromhttpswwwdenvergovorgcontentdenvergovenmayors-officeprograms-initiativespeak-performancepeak-academyhtmlPinnockHBarwickMCarpenterCREldridgeSGrandesGGriffithsCJRycroft-MaloneJMeissnerPMurrayEPatelASheikhAampTaylorSJC(2017)Standardsforreportingimplementationstudies(stari)ExplanationandelaborationdocumentBMJOpenRetrievedfromhttpbmjopenbmjcomcontentbmjopen74e013318fullpdfProctorESilmereHRaghavanRHovmandPAaronsGBungerAGriffeyRampHensleyM(2011)OutcomesforimplementationresearchConceptualdistinctionsmeasurementchallengesandresearchagendaAdministrationandPolicyinMentalHealth38(2)65-76doi101007s10488-010-0319-7TandonSD(2018)ProgressincommunityhealthpartnershipsResearcheducationandactionRetrievedfromhttpswwwpressjhuedujournalsprogress-community-health-partnerships-research-education-and-actionProvanKGampKenisP(2007)ModesofnetworkgovernanceStructuremanagementandeffectivenessJournalofPublicAdministrationResearchandTheory
References
64
18(2)229-252Retrievedfromhttpsacademicoupcomjpartarticle182229935895SabatierPA(1988)Anadvocacycoalitionframeworkofpolicychangeandtheroleofpolicy-orientedlearningthereinPolicySciences21(3)129-168Retrievedfromhttpwwwjstororgstable4532139SandstromAampCarlssonL(2008)TheperformanceofpolicynetworksTherelationbetweennetworkstructureandnetworkperformancePolicyStudiesJournal36497-524Retrievedfromhttponlinelibrarywileycomdoi101111j1541-0072200800281xabstractSchwartzMampWilsonE(nd)WhocanaffordtoliveinahomeAlookatdatafromthe2006americancommunitysurveyRetrievedfromhttpswwwcensusgovhousingcensuspublicationswho-can-affordpdfShearerJCAbelsonAKouyateBLavisJNampWaltG(2016)WhydopolicieschangeInstitutionsinterestsideasandnetworksinthreecasesofpolicyreformHealthPolicyandPlanning31(9)1200-1211Retrievedfromhttpsdoiorg101093heapolczw052ThelessonsofadministrativedataHigh-profilepolicy-relevantresearchpoweredbyadministrativedata(nd)Retrievedfromhttpswwwpovertyactionlaborgsitesdefaultfilesresources20161114-The-Lessons-of-Administrative-DatapdfToolsandotherneedsforqualitativeandquantitativemethods(2015February)AgencyforHealthcareResearchandQualityRetrievedfromhttpwwwahrqgovprofessionalsquality-patient-safetytalkingqualityassesstoolshtml
WeaverRCampBagchi-SenS(2013)SpatialanalysisofurbandeclineThegeographyofblightAppliedGeography4061-70
References
65
UniversityPartners
Name Title Affiliation Area(s)ofexpertise
Contactinformation
Examplesofworkingwithnon-academicdecisionmakers
WendyPBarreno
AssociateDirector
WilsonSheehanLabforEconomicOpportunitiesUniversityofNotreDame
LEOsfocusareasincludehealtheducationhousingchildwelfareeconomicopportunitycriminaljustice
wbarrenonduduorleondedu
MelissaDavey-RothwellPhD
AssociateScientist
JohnsHopkinsBloombergSchoolofPublicHealth
Health mdavey1jhuedu410-614-5854
BaltimoreCityHealthDepartmentLeadersMarylandDepartmentofHealthAddictionRecoveryServicesinDCExecutiveDirectorsofcommunityorganizationsacrosstheUSthatfocusonHIVSTIsubstanceabuseandimplementationscience
ElizabethStuart
AssociateDeanforEducationandProfessorDepartmentsofMentalHealthBiostatisticsandHealthPolicyand
JohnsHopkinsBloombergSchoolofPublicHealth
Educationsocialpolicystatisticalmethods
estuartjhuedu
Resources
66
Management
Design-ThinkingResources
EngagementGuideforCivicTech
Journeymap
PerformanceManagementResources
DenverPeakAcademy
Proxymeasuresguide
Talkingaboutperformancewithoutcomplicatingit(video)
SettingPerformanceTargetsguide
Datavisualizationguide
Performancemanagementgettingstartedguide
Exercises
Fishboneexercise
5Whys
AdministrativeData
TheLessonsofAdministrativeData(JPAL)
EvaluationResources
Quantitativeandqualitativemethods(USDepartmentofHealthandHumanServicesAgencyforHealthcareResearchandQuality)
KeyStepsinOutcomeManagement(UrbanInstitute)
RCT-YES
Sageresearchmethods
BestPracticesforMixedMethodsResearchintheHealthSciences
ACCORDChecklist
Pilotandimplementationplanning
Benchmarkingexamplesandproxymeasures
Seattleopendataplaybook
Resources
67
Datasciencescopingguide
Resources
68
AboutGovExGovExhelpsgovernmentsusedataandevidencetoimproveresidentsrsquolivesWesupportgovernmentsinbuildingtheirinternalcapacitytooperatesustainableanalyticspracticeswhichimprovecommunityoutcomesinfieldsfrompublicsafetytoeconomicdevelopmentGovExhasaproventrackrecordofworkinginconcertwithpartnerstoprovidetechnicalassistanceandtrainingatscale
SinceourlaunchinApril2015GovExhas
Trainedover2000governmentemployeesinpersonandonlineintopicsincludingdatamanagementperformancemanagementopendatacommunityengagementandanalytics
Providedtechnicalassistancetomorethan100localgovernmentsacrosstheglobeimprovingtheiruseofdatatoachieveefficienciesinservicedeliveryandbetteroutcomesforresidents
Assessedthecapacityofmorethan130governmentalorganizationstodeterminehowleadersstaffandexternalstakeholdersmanageandusedata
Localgovernmentsareincreasinglyonthefrontlinesofaddressinglargescaleproblemswithglobalimplicationssuchasclimatechangepublicsafetyandeconomicinequality
Whileevidence-basedinterventionstohelpsolvethoseproblemsareavailablelocalgovernmentstaffhavelimitedtimeandcapacitytofindandimplementresearch-basedsolutionsAsaresultcitiescontinuetoworkthewaytheyalwayshave--makingincrementalimprovementstoprogramsbutnotalwayscreatingchangeatasystem-level
ThroughresourcedevelopmentandcollaborationwithJHUfacultyGovExissupportingcityleadersinusingdataandevidencetomakedecisionsandcreatinglastingrelationshipswithresearchers
TheRoadmapwasdevelopedbyGovExaspartofitspartnershipwithBloombergPhilanthropiesWhatWorksCitiesinitiative
AboutGovEx
69
- Title page
- Introduction and purpose of the Roadmap
- Using the Roadmap
-
- Gather information about the problem
- Strategically engage key partners
- Review the evidence and collect details
- Decide on a policy to implement based on local factors
- Establish a framework to monitor progress
- Assess data accessibility quality and security
- Design implementation and evaluation
- Share your story and success
-
- Working with a range of evidence
- Thank you
- References
- Resources
- About GovEx
-
thoseitems
Takeaction
DiscussionsaboutwhetheryourindicatorsaremovingintherightdirectionandwhatmightbedrivingtrendsmayleadtoadiscoverythattherearenotenoughresourcesallocatedtothispolicyorissueTheremaybelimitedfundingforstaffprogrammingortechnologythatisdrivingthedatainthewrongdirectionIfCityleadershipisintheroomthisisagreattimetomakethecaseforadditionalfundingduringthebudgetseasonBycollectinganalyzinganddiscussingdataonthispolicyroutinelyyouarebuildingevidencetomakethecasefortherightleveloffundingtoreachtheestablishedtargets
Establishaframeworktomonitorprogress
39
Assessdataaccessibilityqualityandsecurity
Determineaccessibilityandqualityofdata
OnceyouhavecompiledyourwishlistofindicatorsitistimetodeterminetheaccessibilityandqualityofthedataTopreventcomplicationyoumaywishtoprioritizerelevantindicatorsforwhichyouarealreadycollectingdataAvoidthetemptationofadoptingatechnologysolutiontocollectnewdataatthispointsuchasaphotosurveytoolorotherappFornowyouaretryingtodeterminewhichdatasetsarerelevantaccessibleandhighqualityBesuretoworkwithyourkeyinternalpartnerswhileyouareidentifyingyourmeasuresDepartmentstaffwhoworkonprogramsarefamiliarwiththeindicatorsthatwillaccuratelysupporttheoutcomesanddatabaseadministratorswillbevaluablewhendiscussingaccessibilityandqualityofthedata
Abrieflistofconsiderationsfortheaccessibilityofyourdataisbelow
ArewecurrentlycollectingthisdatasetorwillwebecollectingitforthefirsttimeHowfarbackdoestherecordgo(timehorizon)WouldweneednewtechnologytocollectoranalyzethedataCanthedatabecollectedmanuallyorcanitbeautomatedHowfrequentlyisthedataproduced(continuouslyhourlydailyweeklymonthlyquarterlyyearlyetc)IsthedataavailablefortheentireCitypopulationorasubsetofthepopulationHowgranularisthedata(individuallevelCensusblockneighborhoodcitywideetc)DoesthedatacontainsensitiveinformationIsthereaproxymeasurethatwecanusefornowIsthedatamachinereadable
AsnotedbytheWilsonSheehanLabforEconomicOpportunities(LEO)attheUniversityofNotreDameleveragingadministrativedatacansignificantlyreducethecostsandtimelinesofrigorousevaluationCitiescanestablishaninternaldatasharingagreementacrossdepartmentsincludinglanguagethatenablesadministrativedatatobeusedforresearchpurposesCitiescanalsoidentifyexternaladministrativedata-holderssuchasacountyschooldistrictpolicedepartmenthospitaloranotherstatethatmayhaveoutcomedataimportantformeasuringprogramimpactCitiescanleverageexistingrelationshipsandorworkwithresearchinstitutionstodevelopstreamlineddatasharingprocessesandagreementswiththeseexternalentities
Assessdataaccessibilityqualityandsecurity
40
Evaluatethequalityofyourdata
QualityisasimportantasaccessibilityToevaluatedataqualityyoumaywishtocreatearubricliketheonebelow
Value ValueDescription
3 ExcellentDataQualityThedataiscompleteaccurateandupdatedregularly
2 AdequateDataQualityThedataissomewhatcompleteaccurateandupdatedfairlyregularly
1 PoorDataQualityThedataisnotcompleteaccurateorupdatedregularly
ThisisagreattimetobuildonthedatainventorythatyoustartedinthegatherinformationpageSuggestedadditionalfieldstoaddtoyourdatainventorytemplatearebelow
Program DatasetTitle
DatasetDescription DataSource
Departmentorexternalpartner
DoestheCityalreadycollectthisdata
Whatisthenameoftheprogram
Human-readablenameofthedataset
Shouldbeunderstandablebynon-technicalusers
Whatistheoriginalsystemapplicationfilethathousesit
Departmentorexternalpartnerthatmaintainsthedata
Isthisdataalreadycollected
Updatefrequency-ThefileformatforthedatasetTypicallycsvxlsxshapefiletxtetc
Format-Howoftenthedatasetisupdated
Timerange-Theperiodoftimecoveredinthisparticulardataset
Datause-Howisthisdatasetcommonlyused
Datausers-Whotypicallyusesthisdataset
Dataownerstitle-Titleofthedataowner
Dataownersposition-DataOwnersjobtitle
Dataownersemailaddress-DataOwnersemailaddressWhomanagesthedataandorisresponsibleforgrantingpermissiontoaccessthedataWhounderstandswhatthedatasetincludesandcananswerquestionsaboutit
Dataqualityconcerns-Describetheconcernsyoumayhaveaboutthequalityofthedataset
Assessdataaccessibilityqualityandsecurity
41
Division-DivisionwherethedataismaintainedIfmorethanonedivisionisresponsiblelisttheprimaryownerofthedataset
Sensitivedatacomments-DescribetheissuesthatwouldpreventthecityfromsharingthedatapubliclyCommonsensitivityconcernsincludeprivacyviolationssecurityissuesorhighcostorstaffdemandstopostthedata
Publishingstatus-Thistheplacetoputanycommentsregardingthecurrentstatusofthedataset
Publishedlink-Thisisthelinktothepublisheddatasetontheopendataportalorcitywebsite
Reviewdatasecurityandprivacyprotectionprotocols
OftenapolicychangealsoinvolvesachangeinthedatathatisbeingcollectedhowthatdataisbeinghandledwhohasaccesstoitandahostofotheradjustmentsAsyoutakestockofthedataassetsthatareavailableforyourinitiativeitisvitaltoalsothinkthroughwhatwillbedonetoprotectthatdataandtheprivacyofthepeopleitrepresentsAbigpartofgooddatasecurityinvolvesstrongdatagovernancepracticesthatdefineandidentifysensitivedataandconsciouslyworktobalancethevalueandriskofstoringsuchsensitivedataOnewayofevaluatingyourdatacomesfromcaselawrelatedtotheUSFreedomofInformationActwhichwehavetranslatedtothechecklistbelow
1 Woulddisclosureresultinasubstantialinvasionofprivacy2 Haveweconsideredtheextentorvalueofthepublicinterestandthepurposeorobject
oftheindividualsseekingdisclosure3 Istheinformationavailablefromothersources4 Wastheinformationgivenwithanexpectationofconfidentiality5 Isitpossibletomouldreliefsoastolimittheinvasionofindividualprivacy
TheseriskscouldbemitigatedusingavarietyofstrategiesfromaggregatingdatatoobfuscatingindividualstoredactingfieldsthatmayincludePersonallyIdentifiableInformation(PII)orre-evaluatingyourdatacollectionmethodstofindalternativewaysofgettingtheinformationyouneedThesestepsshouldbecodifiedandmanagedbyadatagovernancecommitteeifpossibleForthesensitivedatathatisdeemedessentialcreateaplanaroundpotentialdatabreachesinordertoquicklyandconfidentlyrespondtoemergencysituations
ProactivelypreparingforthesesituationsisoftenthebeststrategyfordatasecuritybutdonotforgetthetechnicalsideInvolveyourITadministratorsinconversationsaboutdatastorageandaccessensuringthattheywillbeprotectedbyenterprise-gradecybersecuritysystems
Assessdataaccessibilityqualityandsecurity
42
Assessdataaccessibilityqualityandsecurity
43
Designimplementationandevaluation
Designtheimplementation
NowthatyouhaveyouraccountabilityplanandkeyindicatorsitistimetogetspecificaboutyourplanforimplementationThetemplatebelowwillhelpyouthinkthroughthespecificmechanismsofimplementingpolicychange
UnderstandingorganizationsandsystemsiscriticaltounderstandingandmakingdecisionsaboutwhereandhowinterventionsshouldbeappliedApolicyimplementedinasinglesettingmayshowpromisingresultsbutimplementingmultiplepoliciesandprogramsatthesametimewithdifferentactorsresponsibleforeachaspectcanhelpensuresustainablebehaviorchange
InSomervilleMAShapeUpSomerville(SUS)isacitywidepublichealthprogramthatisimplementedacrossmultiplelevelsofinfluenceandsystemsthroughinterventionsbeforeduringandafterschooltoincludechangestobuiltenvironmentsuchasbikelanesregionalmasstransportationandphysicaleducationequipmentinschoolsandgymstoincreasephysicalactivitywellnesspoliciesandschoolgardensnutritioneducationduringandafterschoolandhealthyrestaurantprogramstosupporthealthyfoodandhealthyeatinghabits
Usethetemplatebelowtoplanyourimplementation
Policyimplementationplanningtemplate
ListthespecificdesiredoutcomesInthepreviousstepyouworkedwithdepartmentstafftoestablishspecificmeasurableoutcomesandinterimindicatorsPrioritizetheindicatorsthatwilldemonstrateprogressUsethespacebelowtocapturethedesiredoutcomesofthepolicychange
ExampleShapeupSomervillebeganwiththegoalofpreventingobesityin1stthrough3rdgraders
Yourturn
____________________________________________________________________________
____________________________________________________________________________
Designimplementationandevaluation
44
SpecifythetargetpopulationAreyoudesigninganinterventionthattargetsonlycertainsitesAreyouplanningonimplementingaprogramorinitiativeforaspecifictargetaudienceClearlyspecifythepopulationyouplantotarget
IfyouplantoconductarigorousimpactevaluationasnotedbytheWilsonSheehanLabforEconomicOpportunitiesitisimportanttoestablishconcreteobjectiveeligibilitycriteriaandensurethatthesecriteriaareappliedconsistentlybyallstaffduringscreeningandintakeItisalsovitaltoidentifytheindividualswhoareeligibleforaparticularprogramorserviceevenifeveryoneeligiblecannotbeservedbyexistingresourcesThishelpscitiesidentifyanyunmetneedsandguidefutureresourceallocationwhilealsocreatingthepossibilityofarigorousimpactevaluationbyidentifyingapotentialcomparisongroupFinallywhenconductingscreeningandintakeitisimperativetocollectandmaintainbasicinformation(egnamedateofbirthgender)aboutallprogram-eligibleindividualsnotjustthosewhoparticipateintheprogram
ExampleIntheShapeUpSomervilleexampletheprogrambeganbytargeting1stthrough3rdgraders
Yourturn
____________________________________________________________________________
____________________________________________________________________________
DefinethelocationWherewillthepolicychangetakeplaceWillimplementationtakeplaceatonesiteormultiplesites
ExampleShapeUpSomervilleinterventionsweredeliveredinschools
Yourturn
____________________________________________________________________________
____________________________________________________________________________
ClarifyimplementationpartnersandarticulatetheirrolesWhatarethekeyorganizationsinstitutionsorpeopleresponsibleforimplementationTheprojectteamshouldbeasclearandspecificaspossibleinprovidingadescriptionoftheconcreteactivitiesthatimplementationpartnersaremeanttocarryoutPracticeprofilesmaybeahelpfultooltohelpdefinewhatisrequiredoftheactorswhoareresponsibleforimplementingtheintervention
Designimplementationandevaluation
45
ExampleInSomervillemayoralleadershipandengagementofmultiplecityagencies(publichealthprimarycareschoolsCambridgeHealthAllianceetc)streamlinedandconnectedthevariousShapeUpSomervilleinitiativesandgrantswhichhelpedtokeepthetalentandenergyfocusedonhealthyeatingandactivelivingforallsectorsofSomervilleSUSpartnersultimatelyincludedschoolsnonprofitsCityandstateagenciesandnationalfoundations
Yourturn
____________________________________________________________________________
____________________________________________________________________________
DescribetheinterventionWhatarethestepsactivitiesorprocessesRememberthatthemechanismthespecificstepsorprocessthatleadtobehaviorchangeiskeytounderstandingtheeffectivenessofthepolicychangeandsharingtheresultswithothers
ExampleGoalsofShapeUpSomervilleweretoinfluenceeverypartofanearlyelementaryschoolchildrsquosdayndashbeforeduringandafter-schooltopromotehealthyeatingphysicalactivityandtheirweightstatusInterventionincludedfoodserviceenhancementstoimprovebreakfastandlunchandstaffprofessionaldevelopmentwalktoschoolactivitiesSUSclassroomcurriculumandprofessionaldevelopmentschoolwellnesspolicydevelopmentSUSafter-schoolcurriculumandprofessionaldevelopmentoutreachandeducationtohomethroughmaterialsforumseventsldquoSUSapprovedrdquorestaurantscommunityoutreachandcapacitybuildingthroughpolicydevelopmenttrainingsmediaplacements
Yourturn
____________________________________________________________________________
____________________________________________________________________________
ClarifytimelineandmilestonesfordeliveryandevaluationHowfrequentlywilltheservicesinterventionbedeliveredHowlongwilltheprogramrunWhatisthetimetoexpectedimpacts(eg12monthsafterprogramcompletion)
Designimplementationandevaluation
46
ExampleShapeUpSomervilleledbyTuftsUniversityFriedmanSchoolofNutritionScienceandPolicyconductedaquasi-experimental(nonrandomized)interventionstudyamong1stthrough3rdgradestudentsinSomerville(intervention)andtwocomparisoncommunitiesfor9-monthsduringoneschoolyear(fall2002ndashspring2003)ThestudyshowedpositiveassociationoftheinterventiononchangeinBMISUSreleasedreportssummarizingevaluationsprogresstodateplansforadditionalprojectsandimpactstenandfifteenyearsintotheinitiative
Yourturn
____________________________________________________________________________
____________________________________________________________________________
PlanyouranalysisTheprevioussectionaddressedestablishinganaccountabilityframeworktomonitorprogressandimplementationDeterminewhichindicatorswillbediscussthroughthatprocesswhowillsharethedataandanalysisandhowfeedbackfromthoseconversationswillbeincorporatedbytheprojectteam
ExampleTheMayorrsquosofficeimplementedadata-drivendecisionmakingapproachcalledSomerStatwhichwassynergisticwiththeapproachestakenbytheresearchandevaluationteamsdeployedforthespecificgrants
Yourturn
____________________________________________________________________________
____________________________________________________________________________
PlanforsustainabilityIfthepolicyandprogramchangeissuccessfulhowwilltheprojectteamsustainandexpandtoadditionalsitesorpopulations
ExampleAlthoughmostofthepartnershipsforgedintheearlyyearsweremaintainednewgrantsandcollaborationsexpandedattentiontoolderpublicschoolstudentsactivetransportationandtheGreenLinesubwayextensionthebuiltenvironmentfarmerrsquosmarketeffortssuchasldquoGrowninSomervillerdquowithUnionMainStreeteconomicdevelopmentandhealthdisparitiesaswellasdatacollectionmonitoringandsurveillance
Yourturn
Designimplementationandevaluation
47
____________________________________________________________________________
____________________________________________________________________________
Pilotplanningtemplate
IfyouplantoconductapilotfirstdeterminethepurposeofthepilotDoyouwanttoobservewhetherthepolicychangeresultsinthedesiredbehaviorchangeAreyouconcernedaboutfeasibilityscalabilityorcostDoyouwanttotestwhetheryourdesiredresultscanbeachievedthroughthispolicyapproachClarifythegoalofthepilotandthequestionsyouhopetoanswerThiswillhelpyouidentifythekeyindicatorsthatyouwillusetomeasurewhetherthepilotissuccessfulBeforebeginningestablishabaselineofcurrentperformanceandsetyourtargetsaccordinglybasedonwhatyouhopetoaccomplishthroughthepilot
NextestablishthepilotplanforyourprojectteamincludingmilestonesforteamcheckinstodeterminewhethertomoveaheadorchangecourseThisisseparatefromthemeetingsthatwilloccuraspartoftheaccountabilityframeworkUsethespacebelowtocreateyourtimelineanddevelopyourmeetingschedule
Task Personresponsible Startdate Checkindate Deliverable
Planyourevaluation
WhileyouaredesigningtheimplementationitisalsotimetobegintodevelopyourplanforanalysisAsnotedbytheWilsonSheehanLabforEconomicOpportunitiesdecisionsaboutprogramsetup-includingoutreachscreeningintakedatatrackingandenrollment-candeterminewhetherornotarigorousimpactevaluationwillbepossibleAsdiscussedcityleadersmayfindithelpfultoincludeevaluationpartnersinearlyconversationsaboutprogramsetupDuringpreviousstagesyoumayhaveworkedwitharesearchertoplanyourproposedpolicychangeYoushouldbeclearaboutthepurposeoftheevaluationanddevelopanevaluationplanBelowareadditionalconsiderations
Causationvstrendanalysis
IntheEstablishaframeworktomonitorprogressstagetherewasguidanceontrackingtrendsovertimeItisimportanttonoteherethatmonitoringdataisdifferentfromanalyzingacausaleffectTrendanalysiswillillustratethatchangesareoccurringandmayleadtoquestionsthatcouldbeansweredbyyourevaluationbuttrendanalysisalonedoesnotprovideevidenceaboutwhatisdrivingthosechangesYoumayseechangesinyour
Designimplementationandevaluation
48
outcomeindicatorsbasedonvariablesoutsideoftheCityscontrolForinstanceachangeinfederaltradepolicycanhaveanimpactemploymentratesinyourCityoutsideofyourCitysnewjobtrainingandplacementprogram
ToaddresscausalityyoumustcomparetheoutcomesofindividualswhoreceivedtheinterventionwiththosewhodidnotWithoutacomparisongroupofsimilarindividualswhowerenotexposedtotheinterventionitisimpossibletoknowwhichfactorsareresponsiblefortheoutcomesRandomizedControlTrials(RCTs)inwhichindividualsarerandomlyassignedtotreatmentandcontrolgroupsarethegoldstandardinprovingcausalityIfyoucandemonstratethatoutcomesimprovedforthosewhoreceivedtheinterventionanddidnotimproveatallorimprovedlessforacomparisongroupyoucanbesomewhatcertainthattheinterventionwasthecause
Examplesofothernon-causalimpactevaluationsthatarerigorousandreliableincludecertainquasi-experimentalstudiesusingcomparisongroupdesignssmartinterruptedtimeseriesanalysesthatmodeltrendsovertimeinwaysthatallowestimationofinterventioneffectswelldesignedpre-posttestanalysesneedsassessmentsimplementationstudiesetcFormoreinformationondifferenttypesofevaluationsseetheSageResearchMethodswebsite
Lookfordifferencesingroups
BuildintimetoalsoexaminethepilotdatatoseeifgroupsareexperiencingdifferentresultsReviewingperformancebasedonoutcomemeasuresforapopulationinaggregatemayhidethesubtlewaysthatsomepeoplemaybeaffecteddifferentlysoitisnecessarytocarefullyanalyzethedatafordisparateoutcomesForinstancetheMovingtoOpportunity(MTO)studythatexaminedtheimpactofhousingvouchersonfamilieswholivedinhigh-povertyneighborhoodsfoundthatchildrenwhomovedbeforetheageof13hadbettereducationandhigherincomesasadultsHoweveradultsdidnotseeimprovededucationandemploymentoutcomes
Disaggregatingdataintomorespecificcategoriesofraceandethnicitycanhelpunderstandtrendsanddisparitiesinamoregranularwaywhichcanhelpdecisionmakerstargetpoliciesandinterventionsinamoreequitableway
EarlierintheRoadmapweworkedtoidentifystakeholdersandwhomightbepositivelyandnegativelyimpactedbyachangeinpolicyUsetheresultsofthatexercisetoinvolvethecommunityininterpretingdataanalysisandtogatherqualitativefeedbackfromthecommunitytocallattentiontoissuesForinstanceanincreaseinyouthdetentionmaybeexplainedbyinefficienciesorlackofaccesstotransportation
Designimplementationandevaluation
49
Designimplementationandevaluation
50
Shareyourstoryandsuccess
Sharetheresults
Storytelling
AnintegralstepinyourstrategyissharingtheresultsofyourpilotorpolicychangewithyourcommunityThisincludesdevelopingmessagingaroundtheworkofyourorganizationandcreatingaplantodisseminateyourstoriesWhytellstoriesaboutyourwork
Informandorunderstandasubjectorsituation
Giveneededcontextofdataorbitsofinformation
Relatablestoriesarememorableforreadersviewers
ReadersorviewerswillseetheinformationasmorecompellingwhenthereisademonstratedandrelatablechangeorimpactonthemselvesortheirfriendsneighborsandcolleaguesStoriesareaboutpeopleandgovernmentisfundamentallyabouthelpingpeopleSotellthestoriesofthepeopleyouandyourcitycolleagueshelped
Keepthesetipsinmindasyoudevelopyourstorywithyourteam
WhatwereyourfindingsandwhyshouldanyonecareTellyouraudiencehowtheproblemwassolvedWhathastheimpactbeenonresidentsandcitystafftodateWhatwasthesuccessandweretheremultiplesuccessesalongthewayIfyouhavepreliminaryresultsordatashowingimprovementtosharethisisthetimetodoit
WhowasinvolvedandimpactedReviewthelistofstakeholdersHowwereeachofthemimpactedbytheprogramHumanizeyourstoryandimpactbyaddingadditionalvoicesThesevoicescanbeoneofthefollowing
Apersonorfamilywhowaspositivelyaffectedbythechangesuccess
Aninvolvedthird-partydescribingboththeproblemandthesolution(ieanorganizationnonprofitprovidersbusinesstradeassociationsportsteametc)whoispartofthecommunitythatthegovernmentworkedwithonthechangesuccess
Someonefromthegovernmentitselfwhocanspeakaboutthechangesuccess
TellmoreaboutyourcityDescribeyourcityincludingpopulationdemographicsindustriesetcWhatareitschallengesWhatisitknownforWhatmakesituniqueHowisitsimilarordifferentfromothercitiesintheUS
Shareyourstoryandsuccess
51
DescribeyourcityrsquoschallengeWriteoutthechallengeyourcityisfacing(asdefinedinyourproblemstatement)HowdoesitaffectresidentsHowdoesthisproblemaffecttheworkofcitystaffmembers
AddthedetailsProvidedetailsonyourappliedresearchstrategyTalkaboutwhatyourcityhasdonenthepasttoresolvetheissueDiscusshowyourstrategyhaschangedwhenyoubegantoadoptthisapproachWasmoneysavedorinvested
Usethisstorymapexercisetoorganizeyourthoughtsandguaranteethatyouincludealloftheelementsofatraditionalstory
AfteryoudevelopyourmessagingcreateacommunicationsmarketingandoutreachstrategydesignedtospreadyourmessageYourstrategyshouldwalkyouthroughthefollowingkeyactivities
1 IdentifyandprofileyouraudienceWhencreatinganddisseminatingyourmessageitisalwaysgoodtothinkaboutyouraudienceWhoexactlywouldbeinterestedinwhatyouhavetosayThegeneralpopulationofthecityPeoplewhoworkinnonprofitsBusinessownersItmattersbecausenoteveryonewholivesinacitymaybethetargetoftheinformationBusinessownersmaybemoreinterestedindatashowingeffectsonthebusinessdistrictthanothersandsoonWewillcoverthislaterbutthisstepwillalsohelpyoutothinkofthemosteffectivewaytoreachyouraudience
2 Defineyourcalltoaction(ifyouhaveone)SpendsometimeconsideringwhyyouareinterestedintellingyourstoryDoyousimplywanttoupdatethepubliconyourprogressorcelebrateyoursuccessesAreyouinterestedinyouraudiencechangingitsbehaviororsupportingaspecificinitiativebeinglaunchedbyyourorganizationIfyouwanttohaveyouraudienceactorbehaveinacertainwaybespecificandincorporateitinyourmessaging
Shareyourstoryandsuccess
52
3 SelectperformancemeasuresThinkabouthowyouwoulddetermineifyourcampaignissuccessfulandselectappropriateperformancemeasuresYouwillrevisitthesemeasuresattheendofyourcampaigncommunicationsplanimplementationtodeterminewhetheryouneedtomodifyyourcommunicationsandoutreachstrategyforthefuture
4 MethodstocommunicateyourstoryTherearemanywaystocommunicatesuccessofyourprojectHerearesomesuggestionsonreachingaudiencesItisimportanttodeterminewhichmethodsarethemosteffectiveforyourdesiredaudiencetoreceiveinformationDoyourresearch(PewResearchCenterhasexcellentstudiesonthereachofmediadigitalmediaandmarketing)ForexampleifseniorsareyourtargetpopulationandtheyarelessinclinedtouseInstagram(thatistrue)youmaywanttoconsiderFacebookorthenightlylocalnewsbothofwhichareusedmoreoftenbyseniorsBelowisalistofsomemethodsandtoolsyoucanuse
i Pressreleasestolocalnewspapersandtvstations
ii ShortsmartphonevideosAgreatwaytoengageanaudienceisthroughvideoAskonequestionofsomeonewhoispartofthestoryMaybethereisasocialworkerwhocandiscusshungerinyourcityThesevideoscanbereusedonmultipleplatforms
iii Long-formwriting
iv TellingindividualstoriesthroughaseriesofphotosandshortnarrativesforpublicationonFacebookorInstagram
v Op-edsopinionpiecesplacedinthelocalpaper
vi Emailupdatestoyourlist
vii Connectingwithappropriateorganizationsthatareinvestedinthetopicandworkwiththecommunityandaskthemtodisseminateinformationtotheirmembers
viii EmployingldquoWordofMouthrdquo(WOM)tacticsbycreatingtownhallsorfreeeventsforresidentstoattendintheirownneighborhoods
5 DeterminethetypeofcollateralUsuallycampaignsthattargetresidentsrequireseveralavenuestoreachthemPeoplearenotalwaysinthesameplaceorspaceHavingakeenunderstandingofyouraudiencewillassistyouwithyourchoicesonwhatneedstobedevelopedSometimesthatmeansimagesforsocialmediaaonepageflierthatgivespeopleinformationonattendinganeventoranin-depthbrochure
Shareyourstoryandsuccess
53
6 ImplementyourstrategyKeepinmindstrategiescanbefluidSometimessomethingmaynotturnoutasexpectedevenifduringplanningsessionsitwasldquosupposedtordquoItisokYoucananalyzewhathappenedduringadebriefsessionandonceyoursurveysandothermeasurementshavebeenfinalized
Visualizations
Withtherightvisualizationyoucanreviewandshareinformationinsightsanalysisperformanceandtrendsallofwhichisvaluabletoacross-sectionofthepublicButwhichvisualizationisrightforyouTochoosetherighttypeofchartgraphdeterminethetypeofquestionyouwanttoanswerSomecommoncharttypesarebelowandmoreinformationcanbefoundhere
Exploreadditionalanalyticaltechniques
Duringthegoal-settingphaseyouestablishedoutputoutcomeandimplementationmeasuresBasedontheresultsofyourpilotyoumayneedtoadjustyourmeasuresWiththepilotdatainhandyoumayhavesomeideasofhowfutureanalysescouldbedifferentandpotentiallyimprovethequalityofyourprogramTakesometimetobrainstormotherrelevantdatasetsthatyoudidnothavethetimetopursueearlierincludingmessydepartmentaldatathatneedsmoreattentionorsourcesofpublicdatafromthestateandfederalgovernmentnonprofitsandevenprivatecompaniesthatcouldsupplementyourunderstandingofanissueUsingmachinelearningtechniquesyoumightbeabletobetter
Shareyourstoryandsuccess
54
predictdemandforhomelessservicesusingNOAAweatherforecastspreemptwatermainbreaksusingsoilstratigraphyorinvestineconomicdevelopmentbymonitoringrealestatelistingdata
Advancedanalyticaltechniquescanalsohelpyourimplementationrunmoreeffectivelybyintegratingdata-informedtargetinganddecisiontoolsintotheprocessByautomatingroutineworkflowsandsurfacingsubtletrendsthesetoolscanhelpyouiterateinamoreprecisemannerandprovidekeystrategicguidancetofront-lineemployeesthatwouldotherwisebeunavailableThesesortsofdata-science-enhancedprocessescanbeclassifiedasoneofsixprojecttypesasexploredfurtherbytheNewOrleansLANOLAlyticsteamFormoreguidanceonscopingasimilarprojecttheGovExadvancedanalyticsscopingguidegivesyouastep-by-stepoverviewofhowtobesuccessful
Usethespacebelowtolistyourideasforpossibleadvancedanalyticsprojectsbytype
Typeofproject Ideas
Findingtheneedleinthehaystack
Prioritizingforimpact
Earlywarningtools
Betterquickerdecisions
Optimizingresourceallocation
Experimentingforwhatworks
Keepthefeedbackloopgoing
GatherinformationonpublicsentimentaboutpolicychangethroughfocusgroupssurveysandpublicmeetingsledbyadvisorygroupmembersNotewhethernewinformationisavailablethatcanhelpanalyzeorcontextualizeyourprogress(egfederalorstatedatareports)Sharetheresultsofyourpilotandthepolicychangeprocesswithothercitiesandresearchers
Shareyourstoryandsuccess
55
WorkingwitharangeofevidencePresentlythereisalotofattentionandinvestmentinevidence-basedpolicymakingincludingthefederalbipartisanCommissiononEvidence-BasedPolicymakingthePewCharitableTrustsreportsandtoolkitsonHowStatesEngageinEvidence-BasedPolicymakingandtheLauraandJohnArnoldFoundationsupportedEvidence-BasedPolicymakingCollaborativeManyoftheseinitiativesfocusonrigorousevaluation--meaningrandomizedcontrolledtrials(RCT)orotherexperimentalmodels--asthebestkindofevidencetouseHoweverthisbodyofevidenceisofteninsufficientforthespecifickindsofquestionsthatgovernmentscanhavewhenmakingadecisionaboutpolicyInthisguideweproposeanapproachtosupportgovernmentsinincorporatingadditionaltypesofevidenceintotheirdecisionmaking
OnewaytounderstandarangeofevidenceistouseNestarsquosStandardsofEvidencetoframeanddefineevidenceTheBritishnonprofitNestaworkstoinjectpositivechangeintothewayUKgovernmentpoliciesandprogramsfunctionOneoftheirkeyareasofworkhasbeendevelopingWhatWorksCentresorganizationsthatcollectcommissionsynthesizeanddisseminateevidenceaboutinterventioneffectivenessacrossarangeofsocialimpactareasNestadevelopedthefollowingStandardsofEvidencetoprovideausefulwaytodocumenthowdifferentkindsofinformationdemonstratedifferentlevelsofevidence
Workingwitharangeofevidence
56
Theimportanceofpotentiallybiasedinformation
Onechallengepresentedbyanevidenceclassificationthatacknowledgesthevalueofnon-scientificformsofevidenceistofigureouthowtobalanceallofthepiecestogetherwhenyouusethemTounderstandhowtodothiswehavetothinkabouthowtopositionpotentiallybiasedinformationindecisionmaking
WhenwelookatinformationwhichhasnotbeenrigorouslyevaluatedweacceptthatthisinformationmightbebiasedThewholereasonweconductrigorousevaluationistoremovebiasfromourobservationsWeconductstructuredtrialstoproduceresultsthatareconsistentandgeneralizableIfafindingisunbiasedthisaccuracymdashthislackofbiasmdashcanbeprovedbecauseanyindependenttestercanreproduceitwhentheyfollowanidenticalprocess
YoucanlookatpotentiallybiasedinformationasjustinformationthatisearlieroninitspathtorefinementreplicationandverifiedaccuracyApieceofinformationbasedonanyoneindividualrsquosobservationsislikelytobebiasedbecauseitiscomingfromjustonesourceBiasisreducedbygatheringtogetherlotsofindependentsourcesofinformationandfiguringoutitsmostcommonattributes
MaybethebiasedinformationmakesaclaimthatistoolargemdashitsayssomethingistrueunderallconditionswhenitisreallyonlytrueundercertainconditionsorthataneffectofaninterventionisbiggerthanitstrueaverageorthataneffectlastsovertimewhenitdoesnotPerhapschangingtheclaimtobemorespecificmdashcriticallyreviewingtheinformationtoseeifspecificconditionsmustbemettoachievetheoutcomesmdashwouldmaketheclaimmoreaccurate
IfyoutestedyourclaimfounditdidnotholdupinsomecaseschangedittesteditagainuntilyourclaimrepeatedlyheldupthenyoucoulddemonstratethattheclaimwasnolongerbiasedbutaccurateThisprocessoftestingimprovingtestingandvalidatingiswhatwegetfromstructuredscientificstudy
Withindividualpiecesofinformation--crediblehypothesescasestudiesexpertopinion--youcannotassumethatkindofaccuracyAtthesametimetheseearlyobservationsarefoundationaltoanyreplicablescientificobservationsWewillnevergetanywhereifwedonotstartwiththesepiecesofinformationeveniftheymightbebiased
Combininglevelsofevidencetogetthemostdetailedinformationforyourcity
ItisespeciallyimportanttoacceptdifferentformsofevidencewhenyoufacequestionsthathavenotyetbeentestedbyindependentstudiesSomeofthemostimportantquestionsrelatetothingsthatareveryspecifictoyoursituationYoufaceaparticularcombinationof
Workingwitharangeofevidence
57
issuesactorstrendsandresourcesthatisultimatelydifferentfromeveryotherplaceintheworld
ThisisimportantbecauseaspolicypractitionersandnotscientistsyoucaremoreaboutyourspecificcasethanahypotheticalaveragegeneralcaseYoucarelessaboutthetotallikelyrangeofimpactsfromapolicythanhowthissingleimplementationwillworkYouwantinformationaboutpoliciesthatwillprovideyouwiththebestpossiblepredictionabouthowtheywillworkinyourcontext
ByputtingdifferentkindsofevidenceintothesameframeworkNestarsquosStandardsofEvidenceimplythatwhenyoucannotgetrigorouslytestedexperimentallyverifiedinformationitisappropriatetoworkwithwhatwedohaveThisisrelevantforthemunicipalimplementationcontextbecausenostudypulledfromaliteraturereviewwillaccountforallofthevariablesyoufaceinyourowncontext
Intheabsenceofhavingastudythatpredictshowaninterventionworksinyourspecificcontextyouwillneednon-scientificinformationtotrytofillinthosegapsYouwillneedtogatherstoriesfromresidentsstaffmembersandotherstakeholdersthatsitonLevels1or2ofthestandardsofevidence
Someofthekindsofinformationyouwillprobablygetmoreeasilyfrompeoplethanstudiesinclude
howpeopleincitieslikeyoursreactedtoaninterventioninpublicforumshowpeoplewhoimplementedaninterventioninacitylikeyoursfiguredouthowtogetitpassedhowthecitycommunicatedpubliclyabouttheapparentbenefitsanddrawbacksoftheinterventionthepoliticalacceptabilityoftheintervention
Youcanremainsimultaneouslyawarethattheinformationyouaregatheringthroughindividualstoriesisbiased--notgeneralizableinformedbyspecificperspectives--whilealsorecognizingthathavingsomeinformationaboutsomethingimportantisbetterthannoinformationEvidencefromthefirsttwoevidencelevelsareessentialbuildingblocksofknowledgeandperfectlyusefulwhenyoucannotgetdetailedunbiasedknowledge
Youhavetostartsomewhereandyoushouldalwaysstartwithevidenceyouhaveaboutyourveryspecificconditionsthatyouknowareimportantforsuccess
Workingwitharangeofevidence
58
Thankyou
Interviewswithcities
GovExheldsemi-structuredinterviewswithsixstaffmembersinfourcitiestoestablishtheinitialstepsoftheRoadmapWeaskedrespondentstotalkaboutaninitiativeorpolicythattheyhavebeenresponsibleforimplementingintheircityRespondentsfromthreecitiesspokeabouttheireffortstoaddresshomelessnessandrespondentsfromonecityspokeaboutblightAfterrespondentsidentifiedtheproblemtheyspokeabouthowthecityprioritizedthisissuethevariousstepsinvolvedandbarrierspresent
GovExconductedtextanalysisoftheseinterviewswhichrevealedthatpeoplewasthemostcommonlyusedtermduringtheseconversationswhendiscussingthestepstochangeaprogramorpolicyThisisathemethroughoutthisRoadmapThefirstsectionbeginswithidentifyingtherelevantpeopleTheconversationsaroundevidencearefocusedonhumanbehaviorratherthanmethodologyorstrengthofevidence
Thankyou
59
OncewehadadraftoftheRoadmapwefollowedupwithcitiesonanindividualbasistolearnmoreaboutwhatwouldbehelpfulWearegratefultothefollowingcitystafffortheinterviewsandindividualconversations
MichaelBaskinChiefPolicyOfficerChattanoogaTN
MelissaBridgesPerformanceandInnovationCoordinatorLittleRockAR
VictoriaCarreoacutenAdministrativeOfficerLasVegasNV
TyroneGrandstrandCommunityDevelopmentPlannerFargoND
TibLaughlinDirectorofGeneralServicesKansasCityKS
TimMorelandDirectorofPerformanceManagementandOpenDataChattanoogaTN
CharlesRosendahlManagementAnalystDentonTX
Thankyou
60
MelissaSiebenAssistantCountyAdministratorKansasCityKS
GinaThompsonDirectorofStrategicServicesLewisvilleTX
JHUFacultycollaboration
GovExalsoconvenedthecitycontactsandfacultypartnersinaseriesofAdvisoryGroupmeetingsThesemeetingsservedtoconnectcitiesandfacultytounderstandoneanothersperspective
StefanBaralBloombergSchoolofPublicHealth
AshleyBernerJHUSchoolofEducation
MelissaDavey-RothwellBloombergSchoolofPublicHealth
JonLinksBloombergSchoolofPublicHealth
TamarMendelsonBloombergSchoolofPublicHealth
KristinMmariBloombergSchoolofPublicHealth
KeshiaPollackPorterBloombergSchoolofPublicHealth
JoshSharfsteinBloombergSchoolofPublicHealth
LizStuartBloombergSchoolofPublicHealth
LindsayThompsonCareyBusinessSchool
DanielWebsterBloombergSchoolofPublicHealth
Partnersindataandevidence-basedpolicy
Wealsospokewithpartnerswhoareexpertsinthefieldofevidence-basedpolicywhosharedtheirknowledgeandresourcesofcurrentandfuturethinkinginevidence-informedthinking
WendyBarrenoAssociateDirectorWilsonSheehanLabforEconomicOpportunitiesUniversityofNotreDame
TiffanyDavenportDirectorofDataPartnershipsBaltimoresPromise
TomiHiersDirectorBaltimoreCivicSiteAnnieECaseyFoundation
ChrisKingsleySeniorAssociateAnnieECaseyFoundation
KathrynOliverAssociateProfessorLondonSchoolofHygieneandTropicalMedicine
Thankyou
61
KristenWatkinsSeniorPolicyAssociateJ-PAL
GovExParticipants
GovExAnalystMiriamMcKinneyanalyzedtheinterviewtranscriptsandconductedtextanalysisSeniorImplementationAdvisorEmilyShawwillintegrateRoadmapactivitieswithadditionalservicedeliveryactivitiesatGovEx
Thankyou
62
ReferencesAbrahamKGampHaskinsR(2017)Thepromiseofevidence-basedpolicymakingReportofthecommissiononevidence-basedpolicymakingWashingtonDCCommissiononEvidence-basedPolicymakingBigmove2Extendingthegreen(nd)Retrievedfromhttpswwwcityoflewisvillecomdoing-businesslewisville-2025big-move-2-extending-the-greenBatesMAampGlennersterR(2017)ThegeneralizabilitypuzzleRetrievedfromhttpsssirorgarticlesentrythe_generalizability_puzzleBuildwithnotfor(2018January)RetrievedfromhttpsbuildwithnotforwordpresscomCairneyPampOliverK(2017)Evidence-basedpolicymakingisnotlikeevidence-basedmedicinesohowfarshouldyougotobridgethedividebetweenevidenceandpolicyHealthResearchPolicyandSystemsdoi101186s12961-017-0192-xCairneyPOliverKampWellsteadA(2016)TobridgethedividebetweenevidenceandpolicyReduceambiguityasmuchasuncertaintyPublicAdministrationReview76(3)399-402Retrievedfromhttpdxdoiorg101111puar12555Centreforpublicimpact(nd)RetrievedfromhttpswwwcentreforpublicimpactorgCernaL(2013)ThenatureofpolicychangeandimplementationAreviewofdifferenttheoreticalapproachesOrganisationforEconomicCo-OperationandDevelopmentRetrievedfromhttpswwwoecdorgeduceriTheNatureofPolicyChangeandImplementationpdfColoradoSprings(nd)Retrievedfromhttpscoloradospringsgovcommunicationspagewhat-expect-when-you-speakupContinuumofcare(coc)program(nd)RetrievedfromhttpswwwhudexchangeinfoprogramscocCouncilpresidentkicksoff$12millionyouthfund(nd)Retrievedfromhttpwwwbaltimorecitycouncilcomcontentcouncil-president-kicks-12-million-youth-fundDurlakJAampDuPreEP(2008)ImplementationmattersAreviewofresearchontheinfluenceofimplementationonprogramoutcomesandthefactorsaffectingimplementationAmericanJournalofCommunityPsychology41327-350Retrievedfromhttpswwwncbinlmnihgovpubmed18322790Fishbonediagram(ishikawa)(nd)Retrievedfromhttpasqorglearn-about-qualitycause-analysis-toolsoverviewfishbonehtmlFivewhysGettingtotherootofaproblemquickly(nd)RetrievedfromhttpswwwmindtoolscompagesarticlenewTMC_5Whtm
References
63
FligsteinNampMcAdamD(2011)TowardageneraltheoryofstrategicactionfieldsSociologicalTheory29(1)1-26RetrievedfromhttpsociologyberkeleyedusitesdefaultfilesfacultyfligsteinFligstein20McAdam20Soc20Theory20PaperpdHeinkelP(2018January12)CitymullspermitsforbikesharingcompaniesRetrievedfromhttpwwwdentonrccomnewsdenton20180112city-mulls-permits-bike-sharing-companiesHowisareamedianincomecalculated(2012October)Retrievedfromhttpswwwhudexchangeinfofaqs843how-is-area-median-income-calculatedLampkinLMampHatryHP(2003)KeystepsinoutcomemanagementUrbanInstituteRetrievedfromhttpswwwurbanorgresearchpublicationkey-steps-outcome-managementMarshDampSmithM(2000)UnderstandingpolicynetworksTowardsadialecticalapproachPoliticalStudies484-21Retrievedfromhttpjournalssagepubcomdoiabs1011111467-924800247McCannL(nd)RetrievedfromhttplaurenellencomMoultonSampSandfortJR(2016)ThestrategicactionfieldframeworkforpolicyimplementationresearchPolicyStudiesJournal45(1)Retrievedfromhttponlinelibrarywileycomstore101111psj12147assetpsj12147pdfv=1ampt=jdkhhcp4amps=e184c625e1855a6b1615ccb99ee27b1c0b3212d8PaynterB(2018January1)What100citiesarelearningfromeachotherbysharingtheirdataRetrievedfromhttpswwwfastcompanycom40518004what-100-cities-are-learning-from-each-other-by-sharing-their-dataPeakacademy(nd)Retrievedfromhttpswwwdenvergovorgcontentdenvergovenmayors-officeprograms-initiativespeak-performancepeak-academyhtmlPinnockHBarwickMCarpenterCREldridgeSGrandesGGriffithsCJRycroft-MaloneJMeissnerPMurrayEPatelASheikhAampTaylorSJC(2017)Standardsforreportingimplementationstudies(stari)ExplanationandelaborationdocumentBMJOpenRetrievedfromhttpbmjopenbmjcomcontentbmjopen74e013318fullpdfProctorESilmereHRaghavanRHovmandPAaronsGBungerAGriffeyRampHensleyM(2011)OutcomesforimplementationresearchConceptualdistinctionsmeasurementchallengesandresearchagendaAdministrationandPolicyinMentalHealth38(2)65-76doi101007s10488-010-0319-7TandonSD(2018)ProgressincommunityhealthpartnershipsResearcheducationandactionRetrievedfromhttpswwwpressjhuedujournalsprogress-community-health-partnerships-research-education-and-actionProvanKGampKenisP(2007)ModesofnetworkgovernanceStructuremanagementandeffectivenessJournalofPublicAdministrationResearchandTheory
References
64
18(2)229-252Retrievedfromhttpsacademicoupcomjpartarticle182229935895SabatierPA(1988)Anadvocacycoalitionframeworkofpolicychangeandtheroleofpolicy-orientedlearningthereinPolicySciences21(3)129-168Retrievedfromhttpwwwjstororgstable4532139SandstromAampCarlssonL(2008)TheperformanceofpolicynetworksTherelationbetweennetworkstructureandnetworkperformancePolicyStudiesJournal36497-524Retrievedfromhttponlinelibrarywileycomdoi101111j1541-0072200800281xabstractSchwartzMampWilsonE(nd)WhocanaffordtoliveinahomeAlookatdatafromthe2006americancommunitysurveyRetrievedfromhttpswwwcensusgovhousingcensuspublicationswho-can-affordpdfShearerJCAbelsonAKouyateBLavisJNampWaltG(2016)WhydopolicieschangeInstitutionsinterestsideasandnetworksinthreecasesofpolicyreformHealthPolicyandPlanning31(9)1200-1211Retrievedfromhttpsdoiorg101093heapolczw052ThelessonsofadministrativedataHigh-profilepolicy-relevantresearchpoweredbyadministrativedata(nd)Retrievedfromhttpswwwpovertyactionlaborgsitesdefaultfilesresources20161114-The-Lessons-of-Administrative-DatapdfToolsandotherneedsforqualitativeandquantitativemethods(2015February)AgencyforHealthcareResearchandQualityRetrievedfromhttpwwwahrqgovprofessionalsquality-patient-safetytalkingqualityassesstoolshtml
WeaverRCampBagchi-SenS(2013)SpatialanalysisofurbandeclineThegeographyofblightAppliedGeography4061-70
References
65
UniversityPartners
Name Title Affiliation Area(s)ofexpertise
Contactinformation
Examplesofworkingwithnon-academicdecisionmakers
WendyPBarreno
AssociateDirector
WilsonSheehanLabforEconomicOpportunitiesUniversityofNotreDame
LEOsfocusareasincludehealtheducationhousingchildwelfareeconomicopportunitycriminaljustice
wbarrenonduduorleondedu
MelissaDavey-RothwellPhD
AssociateScientist
JohnsHopkinsBloombergSchoolofPublicHealth
Health mdavey1jhuedu410-614-5854
BaltimoreCityHealthDepartmentLeadersMarylandDepartmentofHealthAddictionRecoveryServicesinDCExecutiveDirectorsofcommunityorganizationsacrosstheUSthatfocusonHIVSTIsubstanceabuseandimplementationscience
ElizabethStuart
AssociateDeanforEducationandProfessorDepartmentsofMentalHealthBiostatisticsandHealthPolicyand
JohnsHopkinsBloombergSchoolofPublicHealth
Educationsocialpolicystatisticalmethods
estuartjhuedu
Resources
66
Management
Design-ThinkingResources
EngagementGuideforCivicTech
Journeymap
PerformanceManagementResources
DenverPeakAcademy
Proxymeasuresguide
Talkingaboutperformancewithoutcomplicatingit(video)
SettingPerformanceTargetsguide
Datavisualizationguide
Performancemanagementgettingstartedguide
Exercises
Fishboneexercise
5Whys
AdministrativeData
TheLessonsofAdministrativeData(JPAL)
EvaluationResources
Quantitativeandqualitativemethods(USDepartmentofHealthandHumanServicesAgencyforHealthcareResearchandQuality)
KeyStepsinOutcomeManagement(UrbanInstitute)
RCT-YES
Sageresearchmethods
BestPracticesforMixedMethodsResearchintheHealthSciences
ACCORDChecklist
Pilotandimplementationplanning
Benchmarkingexamplesandproxymeasures
Seattleopendataplaybook
Resources
67
Datasciencescopingguide
Resources
68
AboutGovExGovExhelpsgovernmentsusedataandevidencetoimproveresidentsrsquolivesWesupportgovernmentsinbuildingtheirinternalcapacitytooperatesustainableanalyticspracticeswhichimprovecommunityoutcomesinfieldsfrompublicsafetytoeconomicdevelopmentGovExhasaproventrackrecordofworkinginconcertwithpartnerstoprovidetechnicalassistanceandtrainingatscale
SinceourlaunchinApril2015GovExhas
Trainedover2000governmentemployeesinpersonandonlineintopicsincludingdatamanagementperformancemanagementopendatacommunityengagementandanalytics
Providedtechnicalassistancetomorethan100localgovernmentsacrosstheglobeimprovingtheiruseofdatatoachieveefficienciesinservicedeliveryandbetteroutcomesforresidents
Assessedthecapacityofmorethan130governmentalorganizationstodeterminehowleadersstaffandexternalstakeholdersmanageandusedata
Localgovernmentsareincreasinglyonthefrontlinesofaddressinglargescaleproblemswithglobalimplicationssuchasclimatechangepublicsafetyandeconomicinequality
Whileevidence-basedinterventionstohelpsolvethoseproblemsareavailablelocalgovernmentstaffhavelimitedtimeandcapacitytofindandimplementresearch-basedsolutionsAsaresultcitiescontinuetoworkthewaytheyalwayshave--makingincrementalimprovementstoprogramsbutnotalwayscreatingchangeatasystem-level
ThroughresourcedevelopmentandcollaborationwithJHUfacultyGovExissupportingcityleadersinusingdataandevidencetomakedecisionsandcreatinglastingrelationshipswithresearchers
TheRoadmapwasdevelopedbyGovExaspartofitspartnershipwithBloombergPhilanthropiesWhatWorksCitiesinitiative
AboutGovEx
69
- Title page
- Introduction and purpose of the Roadmap
- Using the Roadmap
-
- Gather information about the problem
- Strategically engage key partners
- Review the evidence and collect details
- Decide on a policy to implement based on local factors
- Establish a framework to monitor progress
- Assess data accessibility quality and security
- Design implementation and evaluation
- Share your story and success
-
- Working with a range of evidence
- Thank you
- References
- Resources
- About GovEx
-
Assessdataaccessibilityqualityandsecurity
Determineaccessibilityandqualityofdata
OnceyouhavecompiledyourwishlistofindicatorsitistimetodeterminetheaccessibilityandqualityofthedataTopreventcomplicationyoumaywishtoprioritizerelevantindicatorsforwhichyouarealreadycollectingdataAvoidthetemptationofadoptingatechnologysolutiontocollectnewdataatthispointsuchasaphotosurveytoolorotherappFornowyouaretryingtodeterminewhichdatasetsarerelevantaccessibleandhighqualityBesuretoworkwithyourkeyinternalpartnerswhileyouareidentifyingyourmeasuresDepartmentstaffwhoworkonprogramsarefamiliarwiththeindicatorsthatwillaccuratelysupporttheoutcomesanddatabaseadministratorswillbevaluablewhendiscussingaccessibilityandqualityofthedata
Abrieflistofconsiderationsfortheaccessibilityofyourdataisbelow
ArewecurrentlycollectingthisdatasetorwillwebecollectingitforthefirsttimeHowfarbackdoestherecordgo(timehorizon)WouldweneednewtechnologytocollectoranalyzethedataCanthedatabecollectedmanuallyorcanitbeautomatedHowfrequentlyisthedataproduced(continuouslyhourlydailyweeklymonthlyquarterlyyearlyetc)IsthedataavailablefortheentireCitypopulationorasubsetofthepopulationHowgranularisthedata(individuallevelCensusblockneighborhoodcitywideetc)DoesthedatacontainsensitiveinformationIsthereaproxymeasurethatwecanusefornowIsthedatamachinereadable
AsnotedbytheWilsonSheehanLabforEconomicOpportunities(LEO)attheUniversityofNotreDameleveragingadministrativedatacansignificantlyreducethecostsandtimelinesofrigorousevaluationCitiescanestablishaninternaldatasharingagreementacrossdepartmentsincludinglanguagethatenablesadministrativedatatobeusedforresearchpurposesCitiescanalsoidentifyexternaladministrativedata-holderssuchasacountyschooldistrictpolicedepartmenthospitaloranotherstatethatmayhaveoutcomedataimportantformeasuringprogramimpactCitiescanleverageexistingrelationshipsandorworkwithresearchinstitutionstodevelopstreamlineddatasharingprocessesandagreementswiththeseexternalentities
Assessdataaccessibilityqualityandsecurity
40
Evaluatethequalityofyourdata
QualityisasimportantasaccessibilityToevaluatedataqualityyoumaywishtocreatearubricliketheonebelow
Value ValueDescription
3 ExcellentDataQualityThedataiscompleteaccurateandupdatedregularly
2 AdequateDataQualityThedataissomewhatcompleteaccurateandupdatedfairlyregularly
1 PoorDataQualityThedataisnotcompleteaccurateorupdatedregularly
ThisisagreattimetobuildonthedatainventorythatyoustartedinthegatherinformationpageSuggestedadditionalfieldstoaddtoyourdatainventorytemplatearebelow
Program DatasetTitle
DatasetDescription DataSource
Departmentorexternalpartner
DoestheCityalreadycollectthisdata
Whatisthenameoftheprogram
Human-readablenameofthedataset
Shouldbeunderstandablebynon-technicalusers
Whatistheoriginalsystemapplicationfilethathousesit
Departmentorexternalpartnerthatmaintainsthedata
Isthisdataalreadycollected
Updatefrequency-ThefileformatforthedatasetTypicallycsvxlsxshapefiletxtetc
Format-Howoftenthedatasetisupdated
Timerange-Theperiodoftimecoveredinthisparticulardataset
Datause-Howisthisdatasetcommonlyused
Datausers-Whotypicallyusesthisdataset
Dataownerstitle-Titleofthedataowner
Dataownersposition-DataOwnersjobtitle
Dataownersemailaddress-DataOwnersemailaddressWhomanagesthedataandorisresponsibleforgrantingpermissiontoaccessthedataWhounderstandswhatthedatasetincludesandcananswerquestionsaboutit
Dataqualityconcerns-Describetheconcernsyoumayhaveaboutthequalityofthedataset
Assessdataaccessibilityqualityandsecurity
41
Division-DivisionwherethedataismaintainedIfmorethanonedivisionisresponsiblelisttheprimaryownerofthedataset
Sensitivedatacomments-DescribetheissuesthatwouldpreventthecityfromsharingthedatapubliclyCommonsensitivityconcernsincludeprivacyviolationssecurityissuesorhighcostorstaffdemandstopostthedata
Publishingstatus-Thistheplacetoputanycommentsregardingthecurrentstatusofthedataset
Publishedlink-Thisisthelinktothepublisheddatasetontheopendataportalorcitywebsite
Reviewdatasecurityandprivacyprotectionprotocols
OftenapolicychangealsoinvolvesachangeinthedatathatisbeingcollectedhowthatdataisbeinghandledwhohasaccesstoitandahostofotheradjustmentsAsyoutakestockofthedataassetsthatareavailableforyourinitiativeitisvitaltoalsothinkthroughwhatwillbedonetoprotectthatdataandtheprivacyofthepeopleitrepresentsAbigpartofgooddatasecurityinvolvesstrongdatagovernancepracticesthatdefineandidentifysensitivedataandconsciouslyworktobalancethevalueandriskofstoringsuchsensitivedataOnewayofevaluatingyourdatacomesfromcaselawrelatedtotheUSFreedomofInformationActwhichwehavetranslatedtothechecklistbelow
1 Woulddisclosureresultinasubstantialinvasionofprivacy2 Haveweconsideredtheextentorvalueofthepublicinterestandthepurposeorobject
oftheindividualsseekingdisclosure3 Istheinformationavailablefromothersources4 Wastheinformationgivenwithanexpectationofconfidentiality5 Isitpossibletomouldreliefsoastolimittheinvasionofindividualprivacy
TheseriskscouldbemitigatedusingavarietyofstrategiesfromaggregatingdatatoobfuscatingindividualstoredactingfieldsthatmayincludePersonallyIdentifiableInformation(PII)orre-evaluatingyourdatacollectionmethodstofindalternativewaysofgettingtheinformationyouneedThesestepsshouldbecodifiedandmanagedbyadatagovernancecommitteeifpossibleForthesensitivedatathatisdeemedessentialcreateaplanaroundpotentialdatabreachesinordertoquicklyandconfidentlyrespondtoemergencysituations
ProactivelypreparingforthesesituationsisoftenthebeststrategyfordatasecuritybutdonotforgetthetechnicalsideInvolveyourITadministratorsinconversationsaboutdatastorageandaccessensuringthattheywillbeprotectedbyenterprise-gradecybersecuritysystems
Assessdataaccessibilityqualityandsecurity
42
Assessdataaccessibilityqualityandsecurity
43
Designimplementationandevaluation
Designtheimplementation
NowthatyouhaveyouraccountabilityplanandkeyindicatorsitistimetogetspecificaboutyourplanforimplementationThetemplatebelowwillhelpyouthinkthroughthespecificmechanismsofimplementingpolicychange
UnderstandingorganizationsandsystemsiscriticaltounderstandingandmakingdecisionsaboutwhereandhowinterventionsshouldbeappliedApolicyimplementedinasinglesettingmayshowpromisingresultsbutimplementingmultiplepoliciesandprogramsatthesametimewithdifferentactorsresponsibleforeachaspectcanhelpensuresustainablebehaviorchange
InSomervilleMAShapeUpSomerville(SUS)isacitywidepublichealthprogramthatisimplementedacrossmultiplelevelsofinfluenceandsystemsthroughinterventionsbeforeduringandafterschooltoincludechangestobuiltenvironmentsuchasbikelanesregionalmasstransportationandphysicaleducationequipmentinschoolsandgymstoincreasephysicalactivitywellnesspoliciesandschoolgardensnutritioneducationduringandafterschoolandhealthyrestaurantprogramstosupporthealthyfoodandhealthyeatinghabits
Usethetemplatebelowtoplanyourimplementation
Policyimplementationplanningtemplate
ListthespecificdesiredoutcomesInthepreviousstepyouworkedwithdepartmentstafftoestablishspecificmeasurableoutcomesandinterimindicatorsPrioritizetheindicatorsthatwilldemonstrateprogressUsethespacebelowtocapturethedesiredoutcomesofthepolicychange
ExampleShapeupSomervillebeganwiththegoalofpreventingobesityin1stthrough3rdgraders
Yourturn
____________________________________________________________________________
____________________________________________________________________________
Designimplementationandevaluation
44
SpecifythetargetpopulationAreyoudesigninganinterventionthattargetsonlycertainsitesAreyouplanningonimplementingaprogramorinitiativeforaspecifictargetaudienceClearlyspecifythepopulationyouplantotarget
IfyouplantoconductarigorousimpactevaluationasnotedbytheWilsonSheehanLabforEconomicOpportunitiesitisimportanttoestablishconcreteobjectiveeligibilitycriteriaandensurethatthesecriteriaareappliedconsistentlybyallstaffduringscreeningandintakeItisalsovitaltoidentifytheindividualswhoareeligibleforaparticularprogramorserviceevenifeveryoneeligiblecannotbeservedbyexistingresourcesThishelpscitiesidentifyanyunmetneedsandguidefutureresourceallocationwhilealsocreatingthepossibilityofarigorousimpactevaluationbyidentifyingapotentialcomparisongroupFinallywhenconductingscreeningandintakeitisimperativetocollectandmaintainbasicinformation(egnamedateofbirthgender)aboutallprogram-eligibleindividualsnotjustthosewhoparticipateintheprogram
ExampleIntheShapeUpSomervilleexampletheprogrambeganbytargeting1stthrough3rdgraders
Yourturn
____________________________________________________________________________
____________________________________________________________________________
DefinethelocationWherewillthepolicychangetakeplaceWillimplementationtakeplaceatonesiteormultiplesites
ExampleShapeUpSomervilleinterventionsweredeliveredinschools
Yourturn
____________________________________________________________________________
____________________________________________________________________________
ClarifyimplementationpartnersandarticulatetheirrolesWhatarethekeyorganizationsinstitutionsorpeopleresponsibleforimplementationTheprojectteamshouldbeasclearandspecificaspossibleinprovidingadescriptionoftheconcreteactivitiesthatimplementationpartnersaremeanttocarryoutPracticeprofilesmaybeahelpfultooltohelpdefinewhatisrequiredoftheactorswhoareresponsibleforimplementingtheintervention
Designimplementationandevaluation
45
ExampleInSomervillemayoralleadershipandengagementofmultiplecityagencies(publichealthprimarycareschoolsCambridgeHealthAllianceetc)streamlinedandconnectedthevariousShapeUpSomervilleinitiativesandgrantswhichhelpedtokeepthetalentandenergyfocusedonhealthyeatingandactivelivingforallsectorsofSomervilleSUSpartnersultimatelyincludedschoolsnonprofitsCityandstateagenciesandnationalfoundations
Yourturn
____________________________________________________________________________
____________________________________________________________________________
DescribetheinterventionWhatarethestepsactivitiesorprocessesRememberthatthemechanismthespecificstepsorprocessthatleadtobehaviorchangeiskeytounderstandingtheeffectivenessofthepolicychangeandsharingtheresultswithothers
ExampleGoalsofShapeUpSomervilleweretoinfluenceeverypartofanearlyelementaryschoolchildrsquosdayndashbeforeduringandafter-schooltopromotehealthyeatingphysicalactivityandtheirweightstatusInterventionincludedfoodserviceenhancementstoimprovebreakfastandlunchandstaffprofessionaldevelopmentwalktoschoolactivitiesSUSclassroomcurriculumandprofessionaldevelopmentschoolwellnesspolicydevelopmentSUSafter-schoolcurriculumandprofessionaldevelopmentoutreachandeducationtohomethroughmaterialsforumseventsldquoSUSapprovedrdquorestaurantscommunityoutreachandcapacitybuildingthroughpolicydevelopmenttrainingsmediaplacements
Yourturn
____________________________________________________________________________
____________________________________________________________________________
ClarifytimelineandmilestonesfordeliveryandevaluationHowfrequentlywilltheservicesinterventionbedeliveredHowlongwilltheprogramrunWhatisthetimetoexpectedimpacts(eg12monthsafterprogramcompletion)
Designimplementationandevaluation
46
ExampleShapeUpSomervilleledbyTuftsUniversityFriedmanSchoolofNutritionScienceandPolicyconductedaquasi-experimental(nonrandomized)interventionstudyamong1stthrough3rdgradestudentsinSomerville(intervention)andtwocomparisoncommunitiesfor9-monthsduringoneschoolyear(fall2002ndashspring2003)ThestudyshowedpositiveassociationoftheinterventiononchangeinBMISUSreleasedreportssummarizingevaluationsprogresstodateplansforadditionalprojectsandimpactstenandfifteenyearsintotheinitiative
Yourturn
____________________________________________________________________________
____________________________________________________________________________
PlanyouranalysisTheprevioussectionaddressedestablishinganaccountabilityframeworktomonitorprogressandimplementationDeterminewhichindicatorswillbediscussthroughthatprocesswhowillsharethedataandanalysisandhowfeedbackfromthoseconversationswillbeincorporatedbytheprojectteam
ExampleTheMayorrsquosofficeimplementedadata-drivendecisionmakingapproachcalledSomerStatwhichwassynergisticwiththeapproachestakenbytheresearchandevaluationteamsdeployedforthespecificgrants
Yourturn
____________________________________________________________________________
____________________________________________________________________________
PlanforsustainabilityIfthepolicyandprogramchangeissuccessfulhowwilltheprojectteamsustainandexpandtoadditionalsitesorpopulations
ExampleAlthoughmostofthepartnershipsforgedintheearlyyearsweremaintainednewgrantsandcollaborationsexpandedattentiontoolderpublicschoolstudentsactivetransportationandtheGreenLinesubwayextensionthebuiltenvironmentfarmerrsquosmarketeffortssuchasldquoGrowninSomervillerdquowithUnionMainStreeteconomicdevelopmentandhealthdisparitiesaswellasdatacollectionmonitoringandsurveillance
Yourturn
Designimplementationandevaluation
47
____________________________________________________________________________
____________________________________________________________________________
Pilotplanningtemplate
IfyouplantoconductapilotfirstdeterminethepurposeofthepilotDoyouwanttoobservewhetherthepolicychangeresultsinthedesiredbehaviorchangeAreyouconcernedaboutfeasibilityscalabilityorcostDoyouwanttotestwhetheryourdesiredresultscanbeachievedthroughthispolicyapproachClarifythegoalofthepilotandthequestionsyouhopetoanswerThiswillhelpyouidentifythekeyindicatorsthatyouwillusetomeasurewhetherthepilotissuccessfulBeforebeginningestablishabaselineofcurrentperformanceandsetyourtargetsaccordinglybasedonwhatyouhopetoaccomplishthroughthepilot
NextestablishthepilotplanforyourprojectteamincludingmilestonesforteamcheckinstodeterminewhethertomoveaheadorchangecourseThisisseparatefromthemeetingsthatwilloccuraspartoftheaccountabilityframeworkUsethespacebelowtocreateyourtimelineanddevelopyourmeetingschedule
Task Personresponsible Startdate Checkindate Deliverable
Planyourevaluation
WhileyouaredesigningtheimplementationitisalsotimetobegintodevelopyourplanforanalysisAsnotedbytheWilsonSheehanLabforEconomicOpportunitiesdecisionsaboutprogramsetup-includingoutreachscreeningintakedatatrackingandenrollment-candeterminewhetherornotarigorousimpactevaluationwillbepossibleAsdiscussedcityleadersmayfindithelpfultoincludeevaluationpartnersinearlyconversationsaboutprogramsetupDuringpreviousstagesyoumayhaveworkedwitharesearchertoplanyourproposedpolicychangeYoushouldbeclearaboutthepurposeoftheevaluationanddevelopanevaluationplanBelowareadditionalconsiderations
Causationvstrendanalysis
IntheEstablishaframeworktomonitorprogressstagetherewasguidanceontrackingtrendsovertimeItisimportanttonoteherethatmonitoringdataisdifferentfromanalyzingacausaleffectTrendanalysiswillillustratethatchangesareoccurringandmayleadtoquestionsthatcouldbeansweredbyyourevaluationbuttrendanalysisalonedoesnotprovideevidenceaboutwhatisdrivingthosechangesYoumayseechangesinyour
Designimplementationandevaluation
48
outcomeindicatorsbasedonvariablesoutsideoftheCityscontrolForinstanceachangeinfederaltradepolicycanhaveanimpactemploymentratesinyourCityoutsideofyourCitysnewjobtrainingandplacementprogram
ToaddresscausalityyoumustcomparetheoutcomesofindividualswhoreceivedtheinterventionwiththosewhodidnotWithoutacomparisongroupofsimilarindividualswhowerenotexposedtotheinterventionitisimpossibletoknowwhichfactorsareresponsiblefortheoutcomesRandomizedControlTrials(RCTs)inwhichindividualsarerandomlyassignedtotreatmentandcontrolgroupsarethegoldstandardinprovingcausalityIfyoucandemonstratethatoutcomesimprovedforthosewhoreceivedtheinterventionanddidnotimproveatallorimprovedlessforacomparisongroupyoucanbesomewhatcertainthattheinterventionwasthecause
Examplesofothernon-causalimpactevaluationsthatarerigorousandreliableincludecertainquasi-experimentalstudiesusingcomparisongroupdesignssmartinterruptedtimeseriesanalysesthatmodeltrendsovertimeinwaysthatallowestimationofinterventioneffectswelldesignedpre-posttestanalysesneedsassessmentsimplementationstudiesetcFormoreinformationondifferenttypesofevaluationsseetheSageResearchMethodswebsite
Lookfordifferencesingroups
BuildintimetoalsoexaminethepilotdatatoseeifgroupsareexperiencingdifferentresultsReviewingperformancebasedonoutcomemeasuresforapopulationinaggregatemayhidethesubtlewaysthatsomepeoplemaybeaffecteddifferentlysoitisnecessarytocarefullyanalyzethedatafordisparateoutcomesForinstancetheMovingtoOpportunity(MTO)studythatexaminedtheimpactofhousingvouchersonfamilieswholivedinhigh-povertyneighborhoodsfoundthatchildrenwhomovedbeforetheageof13hadbettereducationandhigherincomesasadultsHoweveradultsdidnotseeimprovededucationandemploymentoutcomes
Disaggregatingdataintomorespecificcategoriesofraceandethnicitycanhelpunderstandtrendsanddisparitiesinamoregranularwaywhichcanhelpdecisionmakerstargetpoliciesandinterventionsinamoreequitableway
EarlierintheRoadmapweworkedtoidentifystakeholdersandwhomightbepositivelyandnegativelyimpactedbyachangeinpolicyUsetheresultsofthatexercisetoinvolvethecommunityininterpretingdataanalysisandtogatherqualitativefeedbackfromthecommunitytocallattentiontoissuesForinstanceanincreaseinyouthdetentionmaybeexplainedbyinefficienciesorlackofaccesstotransportation
Designimplementationandevaluation
49
Designimplementationandevaluation
50
Shareyourstoryandsuccess
Sharetheresults
Storytelling
AnintegralstepinyourstrategyissharingtheresultsofyourpilotorpolicychangewithyourcommunityThisincludesdevelopingmessagingaroundtheworkofyourorganizationandcreatingaplantodisseminateyourstoriesWhytellstoriesaboutyourwork
Informandorunderstandasubjectorsituation
Giveneededcontextofdataorbitsofinformation
Relatablestoriesarememorableforreadersviewers
ReadersorviewerswillseetheinformationasmorecompellingwhenthereisademonstratedandrelatablechangeorimpactonthemselvesortheirfriendsneighborsandcolleaguesStoriesareaboutpeopleandgovernmentisfundamentallyabouthelpingpeopleSotellthestoriesofthepeopleyouandyourcitycolleagueshelped
Keepthesetipsinmindasyoudevelopyourstorywithyourteam
WhatwereyourfindingsandwhyshouldanyonecareTellyouraudiencehowtheproblemwassolvedWhathastheimpactbeenonresidentsandcitystafftodateWhatwasthesuccessandweretheremultiplesuccessesalongthewayIfyouhavepreliminaryresultsordatashowingimprovementtosharethisisthetimetodoit
WhowasinvolvedandimpactedReviewthelistofstakeholdersHowwereeachofthemimpactedbytheprogramHumanizeyourstoryandimpactbyaddingadditionalvoicesThesevoicescanbeoneofthefollowing
Apersonorfamilywhowaspositivelyaffectedbythechangesuccess
Aninvolvedthird-partydescribingboththeproblemandthesolution(ieanorganizationnonprofitprovidersbusinesstradeassociationsportsteametc)whoispartofthecommunitythatthegovernmentworkedwithonthechangesuccess
Someonefromthegovernmentitselfwhocanspeakaboutthechangesuccess
TellmoreaboutyourcityDescribeyourcityincludingpopulationdemographicsindustriesetcWhatareitschallengesWhatisitknownforWhatmakesituniqueHowisitsimilarordifferentfromothercitiesintheUS
Shareyourstoryandsuccess
51
DescribeyourcityrsquoschallengeWriteoutthechallengeyourcityisfacing(asdefinedinyourproblemstatement)HowdoesitaffectresidentsHowdoesthisproblemaffecttheworkofcitystaffmembers
AddthedetailsProvidedetailsonyourappliedresearchstrategyTalkaboutwhatyourcityhasdonenthepasttoresolvetheissueDiscusshowyourstrategyhaschangedwhenyoubegantoadoptthisapproachWasmoneysavedorinvested
Usethisstorymapexercisetoorganizeyourthoughtsandguaranteethatyouincludealloftheelementsofatraditionalstory
AfteryoudevelopyourmessagingcreateacommunicationsmarketingandoutreachstrategydesignedtospreadyourmessageYourstrategyshouldwalkyouthroughthefollowingkeyactivities
1 IdentifyandprofileyouraudienceWhencreatinganddisseminatingyourmessageitisalwaysgoodtothinkaboutyouraudienceWhoexactlywouldbeinterestedinwhatyouhavetosayThegeneralpopulationofthecityPeoplewhoworkinnonprofitsBusinessownersItmattersbecausenoteveryonewholivesinacitymaybethetargetoftheinformationBusinessownersmaybemoreinterestedindatashowingeffectsonthebusinessdistrictthanothersandsoonWewillcoverthislaterbutthisstepwillalsohelpyoutothinkofthemosteffectivewaytoreachyouraudience
2 Defineyourcalltoaction(ifyouhaveone)SpendsometimeconsideringwhyyouareinterestedintellingyourstoryDoyousimplywanttoupdatethepubliconyourprogressorcelebrateyoursuccessesAreyouinterestedinyouraudiencechangingitsbehaviororsupportingaspecificinitiativebeinglaunchedbyyourorganizationIfyouwanttohaveyouraudienceactorbehaveinacertainwaybespecificandincorporateitinyourmessaging
Shareyourstoryandsuccess
52
3 SelectperformancemeasuresThinkabouthowyouwoulddetermineifyourcampaignissuccessfulandselectappropriateperformancemeasuresYouwillrevisitthesemeasuresattheendofyourcampaigncommunicationsplanimplementationtodeterminewhetheryouneedtomodifyyourcommunicationsandoutreachstrategyforthefuture
4 MethodstocommunicateyourstoryTherearemanywaystocommunicatesuccessofyourprojectHerearesomesuggestionsonreachingaudiencesItisimportanttodeterminewhichmethodsarethemosteffectiveforyourdesiredaudiencetoreceiveinformationDoyourresearch(PewResearchCenterhasexcellentstudiesonthereachofmediadigitalmediaandmarketing)ForexampleifseniorsareyourtargetpopulationandtheyarelessinclinedtouseInstagram(thatistrue)youmaywanttoconsiderFacebookorthenightlylocalnewsbothofwhichareusedmoreoftenbyseniorsBelowisalistofsomemethodsandtoolsyoucanuse
i Pressreleasestolocalnewspapersandtvstations
ii ShortsmartphonevideosAgreatwaytoengageanaudienceisthroughvideoAskonequestionofsomeonewhoispartofthestoryMaybethereisasocialworkerwhocandiscusshungerinyourcityThesevideoscanbereusedonmultipleplatforms
iii Long-formwriting
iv TellingindividualstoriesthroughaseriesofphotosandshortnarrativesforpublicationonFacebookorInstagram
v Op-edsopinionpiecesplacedinthelocalpaper
vi Emailupdatestoyourlist
vii Connectingwithappropriateorganizationsthatareinvestedinthetopicandworkwiththecommunityandaskthemtodisseminateinformationtotheirmembers
viii EmployingldquoWordofMouthrdquo(WOM)tacticsbycreatingtownhallsorfreeeventsforresidentstoattendintheirownneighborhoods
5 DeterminethetypeofcollateralUsuallycampaignsthattargetresidentsrequireseveralavenuestoreachthemPeoplearenotalwaysinthesameplaceorspaceHavingakeenunderstandingofyouraudiencewillassistyouwithyourchoicesonwhatneedstobedevelopedSometimesthatmeansimagesforsocialmediaaonepageflierthatgivespeopleinformationonattendinganeventoranin-depthbrochure
Shareyourstoryandsuccess
53
6 ImplementyourstrategyKeepinmindstrategiescanbefluidSometimessomethingmaynotturnoutasexpectedevenifduringplanningsessionsitwasldquosupposedtordquoItisokYoucananalyzewhathappenedduringadebriefsessionandonceyoursurveysandothermeasurementshavebeenfinalized
Visualizations
Withtherightvisualizationyoucanreviewandshareinformationinsightsanalysisperformanceandtrendsallofwhichisvaluabletoacross-sectionofthepublicButwhichvisualizationisrightforyouTochoosetherighttypeofchartgraphdeterminethetypeofquestionyouwanttoanswerSomecommoncharttypesarebelowandmoreinformationcanbefoundhere
Exploreadditionalanalyticaltechniques
Duringthegoal-settingphaseyouestablishedoutputoutcomeandimplementationmeasuresBasedontheresultsofyourpilotyoumayneedtoadjustyourmeasuresWiththepilotdatainhandyoumayhavesomeideasofhowfutureanalysescouldbedifferentandpotentiallyimprovethequalityofyourprogramTakesometimetobrainstormotherrelevantdatasetsthatyoudidnothavethetimetopursueearlierincludingmessydepartmentaldatathatneedsmoreattentionorsourcesofpublicdatafromthestateandfederalgovernmentnonprofitsandevenprivatecompaniesthatcouldsupplementyourunderstandingofanissueUsingmachinelearningtechniquesyoumightbeabletobetter
Shareyourstoryandsuccess
54
predictdemandforhomelessservicesusingNOAAweatherforecastspreemptwatermainbreaksusingsoilstratigraphyorinvestineconomicdevelopmentbymonitoringrealestatelistingdata
Advancedanalyticaltechniquescanalsohelpyourimplementationrunmoreeffectivelybyintegratingdata-informedtargetinganddecisiontoolsintotheprocessByautomatingroutineworkflowsandsurfacingsubtletrendsthesetoolscanhelpyouiterateinamoreprecisemannerandprovidekeystrategicguidancetofront-lineemployeesthatwouldotherwisebeunavailableThesesortsofdata-science-enhancedprocessescanbeclassifiedasoneofsixprojecttypesasexploredfurtherbytheNewOrleansLANOLAlyticsteamFormoreguidanceonscopingasimilarprojecttheGovExadvancedanalyticsscopingguidegivesyouastep-by-stepoverviewofhowtobesuccessful
Usethespacebelowtolistyourideasforpossibleadvancedanalyticsprojectsbytype
Typeofproject Ideas
Findingtheneedleinthehaystack
Prioritizingforimpact
Earlywarningtools
Betterquickerdecisions
Optimizingresourceallocation
Experimentingforwhatworks
Keepthefeedbackloopgoing
GatherinformationonpublicsentimentaboutpolicychangethroughfocusgroupssurveysandpublicmeetingsledbyadvisorygroupmembersNotewhethernewinformationisavailablethatcanhelpanalyzeorcontextualizeyourprogress(egfederalorstatedatareports)Sharetheresultsofyourpilotandthepolicychangeprocesswithothercitiesandresearchers
Shareyourstoryandsuccess
55
WorkingwitharangeofevidencePresentlythereisalotofattentionandinvestmentinevidence-basedpolicymakingincludingthefederalbipartisanCommissiononEvidence-BasedPolicymakingthePewCharitableTrustsreportsandtoolkitsonHowStatesEngageinEvidence-BasedPolicymakingandtheLauraandJohnArnoldFoundationsupportedEvidence-BasedPolicymakingCollaborativeManyoftheseinitiativesfocusonrigorousevaluation--meaningrandomizedcontrolledtrials(RCT)orotherexperimentalmodels--asthebestkindofevidencetouseHoweverthisbodyofevidenceisofteninsufficientforthespecifickindsofquestionsthatgovernmentscanhavewhenmakingadecisionaboutpolicyInthisguideweproposeanapproachtosupportgovernmentsinincorporatingadditionaltypesofevidenceintotheirdecisionmaking
OnewaytounderstandarangeofevidenceistouseNestarsquosStandardsofEvidencetoframeanddefineevidenceTheBritishnonprofitNestaworkstoinjectpositivechangeintothewayUKgovernmentpoliciesandprogramsfunctionOneoftheirkeyareasofworkhasbeendevelopingWhatWorksCentresorganizationsthatcollectcommissionsynthesizeanddisseminateevidenceaboutinterventioneffectivenessacrossarangeofsocialimpactareasNestadevelopedthefollowingStandardsofEvidencetoprovideausefulwaytodocumenthowdifferentkindsofinformationdemonstratedifferentlevelsofevidence
Workingwitharangeofevidence
56
Theimportanceofpotentiallybiasedinformation
Onechallengepresentedbyanevidenceclassificationthatacknowledgesthevalueofnon-scientificformsofevidenceistofigureouthowtobalanceallofthepiecestogetherwhenyouusethemTounderstandhowtodothiswehavetothinkabouthowtopositionpotentiallybiasedinformationindecisionmaking
WhenwelookatinformationwhichhasnotbeenrigorouslyevaluatedweacceptthatthisinformationmightbebiasedThewholereasonweconductrigorousevaluationistoremovebiasfromourobservationsWeconductstructuredtrialstoproduceresultsthatareconsistentandgeneralizableIfafindingisunbiasedthisaccuracymdashthislackofbiasmdashcanbeprovedbecauseanyindependenttestercanreproduceitwhentheyfollowanidenticalprocess
YoucanlookatpotentiallybiasedinformationasjustinformationthatisearlieroninitspathtorefinementreplicationandverifiedaccuracyApieceofinformationbasedonanyoneindividualrsquosobservationsislikelytobebiasedbecauseitiscomingfromjustonesourceBiasisreducedbygatheringtogetherlotsofindependentsourcesofinformationandfiguringoutitsmostcommonattributes
MaybethebiasedinformationmakesaclaimthatistoolargemdashitsayssomethingistrueunderallconditionswhenitisreallyonlytrueundercertainconditionsorthataneffectofaninterventionisbiggerthanitstrueaverageorthataneffectlastsovertimewhenitdoesnotPerhapschangingtheclaimtobemorespecificmdashcriticallyreviewingtheinformationtoseeifspecificconditionsmustbemettoachievetheoutcomesmdashwouldmaketheclaimmoreaccurate
IfyoutestedyourclaimfounditdidnotholdupinsomecaseschangedittesteditagainuntilyourclaimrepeatedlyheldupthenyoucoulddemonstratethattheclaimwasnolongerbiasedbutaccurateThisprocessoftestingimprovingtestingandvalidatingiswhatwegetfromstructuredscientificstudy
Withindividualpiecesofinformation--crediblehypothesescasestudiesexpertopinion--youcannotassumethatkindofaccuracyAtthesametimetheseearlyobservationsarefoundationaltoanyreplicablescientificobservationsWewillnevergetanywhereifwedonotstartwiththesepiecesofinformationeveniftheymightbebiased
Combininglevelsofevidencetogetthemostdetailedinformationforyourcity
ItisespeciallyimportanttoacceptdifferentformsofevidencewhenyoufacequestionsthathavenotyetbeentestedbyindependentstudiesSomeofthemostimportantquestionsrelatetothingsthatareveryspecifictoyoursituationYoufaceaparticularcombinationof
Workingwitharangeofevidence
57
issuesactorstrendsandresourcesthatisultimatelydifferentfromeveryotherplaceintheworld
ThisisimportantbecauseaspolicypractitionersandnotscientistsyoucaremoreaboutyourspecificcasethanahypotheticalaveragegeneralcaseYoucarelessaboutthetotallikelyrangeofimpactsfromapolicythanhowthissingleimplementationwillworkYouwantinformationaboutpoliciesthatwillprovideyouwiththebestpossiblepredictionabouthowtheywillworkinyourcontext
ByputtingdifferentkindsofevidenceintothesameframeworkNestarsquosStandardsofEvidenceimplythatwhenyoucannotgetrigorouslytestedexperimentallyverifiedinformationitisappropriatetoworkwithwhatwedohaveThisisrelevantforthemunicipalimplementationcontextbecausenostudypulledfromaliteraturereviewwillaccountforallofthevariablesyoufaceinyourowncontext
Intheabsenceofhavingastudythatpredictshowaninterventionworksinyourspecificcontextyouwillneednon-scientificinformationtotrytofillinthosegapsYouwillneedtogatherstoriesfromresidentsstaffmembersandotherstakeholdersthatsitonLevels1or2ofthestandardsofevidence
Someofthekindsofinformationyouwillprobablygetmoreeasilyfrompeoplethanstudiesinclude
howpeopleincitieslikeyoursreactedtoaninterventioninpublicforumshowpeoplewhoimplementedaninterventioninacitylikeyoursfiguredouthowtogetitpassedhowthecitycommunicatedpubliclyabouttheapparentbenefitsanddrawbacksoftheinterventionthepoliticalacceptabilityoftheintervention
Youcanremainsimultaneouslyawarethattheinformationyouaregatheringthroughindividualstoriesisbiased--notgeneralizableinformedbyspecificperspectives--whilealsorecognizingthathavingsomeinformationaboutsomethingimportantisbetterthannoinformationEvidencefromthefirsttwoevidencelevelsareessentialbuildingblocksofknowledgeandperfectlyusefulwhenyoucannotgetdetailedunbiasedknowledge
Youhavetostartsomewhereandyoushouldalwaysstartwithevidenceyouhaveaboutyourveryspecificconditionsthatyouknowareimportantforsuccess
Workingwitharangeofevidence
58
Thankyou
Interviewswithcities
GovExheldsemi-structuredinterviewswithsixstaffmembersinfourcitiestoestablishtheinitialstepsoftheRoadmapWeaskedrespondentstotalkaboutaninitiativeorpolicythattheyhavebeenresponsibleforimplementingintheircityRespondentsfromthreecitiesspokeabouttheireffortstoaddresshomelessnessandrespondentsfromonecityspokeaboutblightAfterrespondentsidentifiedtheproblemtheyspokeabouthowthecityprioritizedthisissuethevariousstepsinvolvedandbarrierspresent
GovExconductedtextanalysisoftheseinterviewswhichrevealedthatpeoplewasthemostcommonlyusedtermduringtheseconversationswhendiscussingthestepstochangeaprogramorpolicyThisisathemethroughoutthisRoadmapThefirstsectionbeginswithidentifyingtherelevantpeopleTheconversationsaroundevidencearefocusedonhumanbehaviorratherthanmethodologyorstrengthofevidence
Thankyou
59
OncewehadadraftoftheRoadmapwefollowedupwithcitiesonanindividualbasistolearnmoreaboutwhatwouldbehelpfulWearegratefultothefollowingcitystafffortheinterviewsandindividualconversations
MichaelBaskinChiefPolicyOfficerChattanoogaTN
MelissaBridgesPerformanceandInnovationCoordinatorLittleRockAR
VictoriaCarreoacutenAdministrativeOfficerLasVegasNV
TyroneGrandstrandCommunityDevelopmentPlannerFargoND
TibLaughlinDirectorofGeneralServicesKansasCityKS
TimMorelandDirectorofPerformanceManagementandOpenDataChattanoogaTN
CharlesRosendahlManagementAnalystDentonTX
Thankyou
60
MelissaSiebenAssistantCountyAdministratorKansasCityKS
GinaThompsonDirectorofStrategicServicesLewisvilleTX
JHUFacultycollaboration
GovExalsoconvenedthecitycontactsandfacultypartnersinaseriesofAdvisoryGroupmeetingsThesemeetingsservedtoconnectcitiesandfacultytounderstandoneanothersperspective
StefanBaralBloombergSchoolofPublicHealth
AshleyBernerJHUSchoolofEducation
MelissaDavey-RothwellBloombergSchoolofPublicHealth
JonLinksBloombergSchoolofPublicHealth
TamarMendelsonBloombergSchoolofPublicHealth
KristinMmariBloombergSchoolofPublicHealth
KeshiaPollackPorterBloombergSchoolofPublicHealth
JoshSharfsteinBloombergSchoolofPublicHealth
LizStuartBloombergSchoolofPublicHealth
LindsayThompsonCareyBusinessSchool
DanielWebsterBloombergSchoolofPublicHealth
Partnersindataandevidence-basedpolicy
Wealsospokewithpartnerswhoareexpertsinthefieldofevidence-basedpolicywhosharedtheirknowledgeandresourcesofcurrentandfuturethinkinginevidence-informedthinking
WendyBarrenoAssociateDirectorWilsonSheehanLabforEconomicOpportunitiesUniversityofNotreDame
TiffanyDavenportDirectorofDataPartnershipsBaltimoresPromise
TomiHiersDirectorBaltimoreCivicSiteAnnieECaseyFoundation
ChrisKingsleySeniorAssociateAnnieECaseyFoundation
KathrynOliverAssociateProfessorLondonSchoolofHygieneandTropicalMedicine
Thankyou
61
KristenWatkinsSeniorPolicyAssociateJ-PAL
GovExParticipants
GovExAnalystMiriamMcKinneyanalyzedtheinterviewtranscriptsandconductedtextanalysisSeniorImplementationAdvisorEmilyShawwillintegrateRoadmapactivitieswithadditionalservicedeliveryactivitiesatGovEx
Thankyou
62
ReferencesAbrahamKGampHaskinsR(2017)Thepromiseofevidence-basedpolicymakingReportofthecommissiononevidence-basedpolicymakingWashingtonDCCommissiononEvidence-basedPolicymakingBigmove2Extendingthegreen(nd)Retrievedfromhttpswwwcityoflewisvillecomdoing-businesslewisville-2025big-move-2-extending-the-greenBatesMAampGlennersterR(2017)ThegeneralizabilitypuzzleRetrievedfromhttpsssirorgarticlesentrythe_generalizability_puzzleBuildwithnotfor(2018January)RetrievedfromhttpsbuildwithnotforwordpresscomCairneyPampOliverK(2017)Evidence-basedpolicymakingisnotlikeevidence-basedmedicinesohowfarshouldyougotobridgethedividebetweenevidenceandpolicyHealthResearchPolicyandSystemsdoi101186s12961-017-0192-xCairneyPOliverKampWellsteadA(2016)TobridgethedividebetweenevidenceandpolicyReduceambiguityasmuchasuncertaintyPublicAdministrationReview76(3)399-402Retrievedfromhttpdxdoiorg101111puar12555Centreforpublicimpact(nd)RetrievedfromhttpswwwcentreforpublicimpactorgCernaL(2013)ThenatureofpolicychangeandimplementationAreviewofdifferenttheoreticalapproachesOrganisationforEconomicCo-OperationandDevelopmentRetrievedfromhttpswwwoecdorgeduceriTheNatureofPolicyChangeandImplementationpdfColoradoSprings(nd)Retrievedfromhttpscoloradospringsgovcommunicationspagewhat-expect-when-you-speakupContinuumofcare(coc)program(nd)RetrievedfromhttpswwwhudexchangeinfoprogramscocCouncilpresidentkicksoff$12millionyouthfund(nd)Retrievedfromhttpwwwbaltimorecitycouncilcomcontentcouncil-president-kicks-12-million-youth-fundDurlakJAampDuPreEP(2008)ImplementationmattersAreviewofresearchontheinfluenceofimplementationonprogramoutcomesandthefactorsaffectingimplementationAmericanJournalofCommunityPsychology41327-350Retrievedfromhttpswwwncbinlmnihgovpubmed18322790Fishbonediagram(ishikawa)(nd)Retrievedfromhttpasqorglearn-about-qualitycause-analysis-toolsoverviewfishbonehtmlFivewhysGettingtotherootofaproblemquickly(nd)RetrievedfromhttpswwwmindtoolscompagesarticlenewTMC_5Whtm
References
63
FligsteinNampMcAdamD(2011)TowardageneraltheoryofstrategicactionfieldsSociologicalTheory29(1)1-26RetrievedfromhttpsociologyberkeleyedusitesdefaultfilesfacultyfligsteinFligstein20McAdam20Soc20Theory20PaperpdHeinkelP(2018January12)CitymullspermitsforbikesharingcompaniesRetrievedfromhttpwwwdentonrccomnewsdenton20180112city-mulls-permits-bike-sharing-companiesHowisareamedianincomecalculated(2012October)Retrievedfromhttpswwwhudexchangeinfofaqs843how-is-area-median-income-calculatedLampkinLMampHatryHP(2003)KeystepsinoutcomemanagementUrbanInstituteRetrievedfromhttpswwwurbanorgresearchpublicationkey-steps-outcome-managementMarshDampSmithM(2000)UnderstandingpolicynetworksTowardsadialecticalapproachPoliticalStudies484-21Retrievedfromhttpjournalssagepubcomdoiabs1011111467-924800247McCannL(nd)RetrievedfromhttplaurenellencomMoultonSampSandfortJR(2016)ThestrategicactionfieldframeworkforpolicyimplementationresearchPolicyStudiesJournal45(1)Retrievedfromhttponlinelibrarywileycomstore101111psj12147assetpsj12147pdfv=1ampt=jdkhhcp4amps=e184c625e1855a6b1615ccb99ee27b1c0b3212d8PaynterB(2018January1)What100citiesarelearningfromeachotherbysharingtheirdataRetrievedfromhttpswwwfastcompanycom40518004what-100-cities-are-learning-from-each-other-by-sharing-their-dataPeakacademy(nd)Retrievedfromhttpswwwdenvergovorgcontentdenvergovenmayors-officeprograms-initiativespeak-performancepeak-academyhtmlPinnockHBarwickMCarpenterCREldridgeSGrandesGGriffithsCJRycroft-MaloneJMeissnerPMurrayEPatelASheikhAampTaylorSJC(2017)Standardsforreportingimplementationstudies(stari)ExplanationandelaborationdocumentBMJOpenRetrievedfromhttpbmjopenbmjcomcontentbmjopen74e013318fullpdfProctorESilmereHRaghavanRHovmandPAaronsGBungerAGriffeyRampHensleyM(2011)OutcomesforimplementationresearchConceptualdistinctionsmeasurementchallengesandresearchagendaAdministrationandPolicyinMentalHealth38(2)65-76doi101007s10488-010-0319-7TandonSD(2018)ProgressincommunityhealthpartnershipsResearcheducationandactionRetrievedfromhttpswwwpressjhuedujournalsprogress-community-health-partnerships-research-education-and-actionProvanKGampKenisP(2007)ModesofnetworkgovernanceStructuremanagementandeffectivenessJournalofPublicAdministrationResearchandTheory
References
64
18(2)229-252Retrievedfromhttpsacademicoupcomjpartarticle182229935895SabatierPA(1988)Anadvocacycoalitionframeworkofpolicychangeandtheroleofpolicy-orientedlearningthereinPolicySciences21(3)129-168Retrievedfromhttpwwwjstororgstable4532139SandstromAampCarlssonL(2008)TheperformanceofpolicynetworksTherelationbetweennetworkstructureandnetworkperformancePolicyStudiesJournal36497-524Retrievedfromhttponlinelibrarywileycomdoi101111j1541-0072200800281xabstractSchwartzMampWilsonE(nd)WhocanaffordtoliveinahomeAlookatdatafromthe2006americancommunitysurveyRetrievedfromhttpswwwcensusgovhousingcensuspublicationswho-can-affordpdfShearerJCAbelsonAKouyateBLavisJNampWaltG(2016)WhydopolicieschangeInstitutionsinterestsideasandnetworksinthreecasesofpolicyreformHealthPolicyandPlanning31(9)1200-1211Retrievedfromhttpsdoiorg101093heapolczw052ThelessonsofadministrativedataHigh-profilepolicy-relevantresearchpoweredbyadministrativedata(nd)Retrievedfromhttpswwwpovertyactionlaborgsitesdefaultfilesresources20161114-The-Lessons-of-Administrative-DatapdfToolsandotherneedsforqualitativeandquantitativemethods(2015February)AgencyforHealthcareResearchandQualityRetrievedfromhttpwwwahrqgovprofessionalsquality-patient-safetytalkingqualityassesstoolshtml
WeaverRCampBagchi-SenS(2013)SpatialanalysisofurbandeclineThegeographyofblightAppliedGeography4061-70
References
65
UniversityPartners
Name Title Affiliation Area(s)ofexpertise
Contactinformation
Examplesofworkingwithnon-academicdecisionmakers
WendyPBarreno
AssociateDirector
WilsonSheehanLabforEconomicOpportunitiesUniversityofNotreDame
LEOsfocusareasincludehealtheducationhousingchildwelfareeconomicopportunitycriminaljustice
wbarrenonduduorleondedu
MelissaDavey-RothwellPhD
AssociateScientist
JohnsHopkinsBloombergSchoolofPublicHealth
Health mdavey1jhuedu410-614-5854
BaltimoreCityHealthDepartmentLeadersMarylandDepartmentofHealthAddictionRecoveryServicesinDCExecutiveDirectorsofcommunityorganizationsacrosstheUSthatfocusonHIVSTIsubstanceabuseandimplementationscience
ElizabethStuart
AssociateDeanforEducationandProfessorDepartmentsofMentalHealthBiostatisticsandHealthPolicyand
JohnsHopkinsBloombergSchoolofPublicHealth
Educationsocialpolicystatisticalmethods
estuartjhuedu
Resources
66
Management
Design-ThinkingResources
EngagementGuideforCivicTech
Journeymap
PerformanceManagementResources
DenverPeakAcademy
Proxymeasuresguide
Talkingaboutperformancewithoutcomplicatingit(video)
SettingPerformanceTargetsguide
Datavisualizationguide
Performancemanagementgettingstartedguide
Exercises
Fishboneexercise
5Whys
AdministrativeData
TheLessonsofAdministrativeData(JPAL)
EvaluationResources
Quantitativeandqualitativemethods(USDepartmentofHealthandHumanServicesAgencyforHealthcareResearchandQuality)
KeyStepsinOutcomeManagement(UrbanInstitute)
RCT-YES
Sageresearchmethods
BestPracticesforMixedMethodsResearchintheHealthSciences
ACCORDChecklist
Pilotandimplementationplanning
Benchmarkingexamplesandproxymeasures
Seattleopendataplaybook
Resources
67
Datasciencescopingguide
Resources
68
AboutGovExGovExhelpsgovernmentsusedataandevidencetoimproveresidentsrsquolivesWesupportgovernmentsinbuildingtheirinternalcapacitytooperatesustainableanalyticspracticeswhichimprovecommunityoutcomesinfieldsfrompublicsafetytoeconomicdevelopmentGovExhasaproventrackrecordofworkinginconcertwithpartnerstoprovidetechnicalassistanceandtrainingatscale
SinceourlaunchinApril2015GovExhas
Trainedover2000governmentemployeesinpersonandonlineintopicsincludingdatamanagementperformancemanagementopendatacommunityengagementandanalytics
Providedtechnicalassistancetomorethan100localgovernmentsacrosstheglobeimprovingtheiruseofdatatoachieveefficienciesinservicedeliveryandbetteroutcomesforresidents
Assessedthecapacityofmorethan130governmentalorganizationstodeterminehowleadersstaffandexternalstakeholdersmanageandusedata
Localgovernmentsareincreasinglyonthefrontlinesofaddressinglargescaleproblemswithglobalimplicationssuchasclimatechangepublicsafetyandeconomicinequality
Whileevidence-basedinterventionstohelpsolvethoseproblemsareavailablelocalgovernmentstaffhavelimitedtimeandcapacitytofindandimplementresearch-basedsolutionsAsaresultcitiescontinuetoworkthewaytheyalwayshave--makingincrementalimprovementstoprogramsbutnotalwayscreatingchangeatasystem-level
ThroughresourcedevelopmentandcollaborationwithJHUfacultyGovExissupportingcityleadersinusingdataandevidencetomakedecisionsandcreatinglastingrelationshipswithresearchers
TheRoadmapwasdevelopedbyGovExaspartofitspartnershipwithBloombergPhilanthropiesWhatWorksCitiesinitiative
AboutGovEx
69
- Title page
- Introduction and purpose of the Roadmap
- Using the Roadmap
-
- Gather information about the problem
- Strategically engage key partners
- Review the evidence and collect details
- Decide on a policy to implement based on local factors
- Establish a framework to monitor progress
- Assess data accessibility quality and security
- Design implementation and evaluation
- Share your story and success
-
- Working with a range of evidence
- Thank you
- References
- Resources
- About GovEx
-
Evaluatethequalityofyourdata
QualityisasimportantasaccessibilityToevaluatedataqualityyoumaywishtocreatearubricliketheonebelow
Value ValueDescription
3 ExcellentDataQualityThedataiscompleteaccurateandupdatedregularly
2 AdequateDataQualityThedataissomewhatcompleteaccurateandupdatedfairlyregularly
1 PoorDataQualityThedataisnotcompleteaccurateorupdatedregularly
ThisisagreattimetobuildonthedatainventorythatyoustartedinthegatherinformationpageSuggestedadditionalfieldstoaddtoyourdatainventorytemplatearebelow
Program DatasetTitle
DatasetDescription DataSource
Departmentorexternalpartner
DoestheCityalreadycollectthisdata
Whatisthenameoftheprogram
Human-readablenameofthedataset
Shouldbeunderstandablebynon-technicalusers
Whatistheoriginalsystemapplicationfilethathousesit
Departmentorexternalpartnerthatmaintainsthedata
Isthisdataalreadycollected
Updatefrequency-ThefileformatforthedatasetTypicallycsvxlsxshapefiletxtetc
Format-Howoftenthedatasetisupdated
Timerange-Theperiodoftimecoveredinthisparticulardataset
Datause-Howisthisdatasetcommonlyused
Datausers-Whotypicallyusesthisdataset
Dataownerstitle-Titleofthedataowner
Dataownersposition-DataOwnersjobtitle
Dataownersemailaddress-DataOwnersemailaddressWhomanagesthedataandorisresponsibleforgrantingpermissiontoaccessthedataWhounderstandswhatthedatasetincludesandcananswerquestionsaboutit
Dataqualityconcerns-Describetheconcernsyoumayhaveaboutthequalityofthedataset
Assessdataaccessibilityqualityandsecurity
41
Division-DivisionwherethedataismaintainedIfmorethanonedivisionisresponsiblelisttheprimaryownerofthedataset
Sensitivedatacomments-DescribetheissuesthatwouldpreventthecityfromsharingthedatapubliclyCommonsensitivityconcernsincludeprivacyviolationssecurityissuesorhighcostorstaffdemandstopostthedata
Publishingstatus-Thistheplacetoputanycommentsregardingthecurrentstatusofthedataset
Publishedlink-Thisisthelinktothepublisheddatasetontheopendataportalorcitywebsite
Reviewdatasecurityandprivacyprotectionprotocols
OftenapolicychangealsoinvolvesachangeinthedatathatisbeingcollectedhowthatdataisbeinghandledwhohasaccesstoitandahostofotheradjustmentsAsyoutakestockofthedataassetsthatareavailableforyourinitiativeitisvitaltoalsothinkthroughwhatwillbedonetoprotectthatdataandtheprivacyofthepeopleitrepresentsAbigpartofgooddatasecurityinvolvesstrongdatagovernancepracticesthatdefineandidentifysensitivedataandconsciouslyworktobalancethevalueandriskofstoringsuchsensitivedataOnewayofevaluatingyourdatacomesfromcaselawrelatedtotheUSFreedomofInformationActwhichwehavetranslatedtothechecklistbelow
1 Woulddisclosureresultinasubstantialinvasionofprivacy2 Haveweconsideredtheextentorvalueofthepublicinterestandthepurposeorobject
oftheindividualsseekingdisclosure3 Istheinformationavailablefromothersources4 Wastheinformationgivenwithanexpectationofconfidentiality5 Isitpossibletomouldreliefsoastolimittheinvasionofindividualprivacy
TheseriskscouldbemitigatedusingavarietyofstrategiesfromaggregatingdatatoobfuscatingindividualstoredactingfieldsthatmayincludePersonallyIdentifiableInformation(PII)orre-evaluatingyourdatacollectionmethodstofindalternativewaysofgettingtheinformationyouneedThesestepsshouldbecodifiedandmanagedbyadatagovernancecommitteeifpossibleForthesensitivedatathatisdeemedessentialcreateaplanaroundpotentialdatabreachesinordertoquicklyandconfidentlyrespondtoemergencysituations
ProactivelypreparingforthesesituationsisoftenthebeststrategyfordatasecuritybutdonotforgetthetechnicalsideInvolveyourITadministratorsinconversationsaboutdatastorageandaccessensuringthattheywillbeprotectedbyenterprise-gradecybersecuritysystems
Assessdataaccessibilityqualityandsecurity
42
Assessdataaccessibilityqualityandsecurity
43
Designimplementationandevaluation
Designtheimplementation
NowthatyouhaveyouraccountabilityplanandkeyindicatorsitistimetogetspecificaboutyourplanforimplementationThetemplatebelowwillhelpyouthinkthroughthespecificmechanismsofimplementingpolicychange
UnderstandingorganizationsandsystemsiscriticaltounderstandingandmakingdecisionsaboutwhereandhowinterventionsshouldbeappliedApolicyimplementedinasinglesettingmayshowpromisingresultsbutimplementingmultiplepoliciesandprogramsatthesametimewithdifferentactorsresponsibleforeachaspectcanhelpensuresustainablebehaviorchange
InSomervilleMAShapeUpSomerville(SUS)isacitywidepublichealthprogramthatisimplementedacrossmultiplelevelsofinfluenceandsystemsthroughinterventionsbeforeduringandafterschooltoincludechangestobuiltenvironmentsuchasbikelanesregionalmasstransportationandphysicaleducationequipmentinschoolsandgymstoincreasephysicalactivitywellnesspoliciesandschoolgardensnutritioneducationduringandafterschoolandhealthyrestaurantprogramstosupporthealthyfoodandhealthyeatinghabits
Usethetemplatebelowtoplanyourimplementation
Policyimplementationplanningtemplate
ListthespecificdesiredoutcomesInthepreviousstepyouworkedwithdepartmentstafftoestablishspecificmeasurableoutcomesandinterimindicatorsPrioritizetheindicatorsthatwilldemonstrateprogressUsethespacebelowtocapturethedesiredoutcomesofthepolicychange
ExampleShapeupSomervillebeganwiththegoalofpreventingobesityin1stthrough3rdgraders
Yourturn
____________________________________________________________________________
____________________________________________________________________________
Designimplementationandevaluation
44
SpecifythetargetpopulationAreyoudesigninganinterventionthattargetsonlycertainsitesAreyouplanningonimplementingaprogramorinitiativeforaspecifictargetaudienceClearlyspecifythepopulationyouplantotarget
IfyouplantoconductarigorousimpactevaluationasnotedbytheWilsonSheehanLabforEconomicOpportunitiesitisimportanttoestablishconcreteobjectiveeligibilitycriteriaandensurethatthesecriteriaareappliedconsistentlybyallstaffduringscreeningandintakeItisalsovitaltoidentifytheindividualswhoareeligibleforaparticularprogramorserviceevenifeveryoneeligiblecannotbeservedbyexistingresourcesThishelpscitiesidentifyanyunmetneedsandguidefutureresourceallocationwhilealsocreatingthepossibilityofarigorousimpactevaluationbyidentifyingapotentialcomparisongroupFinallywhenconductingscreeningandintakeitisimperativetocollectandmaintainbasicinformation(egnamedateofbirthgender)aboutallprogram-eligibleindividualsnotjustthosewhoparticipateintheprogram
ExampleIntheShapeUpSomervilleexampletheprogrambeganbytargeting1stthrough3rdgraders
Yourturn
____________________________________________________________________________
____________________________________________________________________________
DefinethelocationWherewillthepolicychangetakeplaceWillimplementationtakeplaceatonesiteormultiplesites
ExampleShapeUpSomervilleinterventionsweredeliveredinschools
Yourturn
____________________________________________________________________________
____________________________________________________________________________
ClarifyimplementationpartnersandarticulatetheirrolesWhatarethekeyorganizationsinstitutionsorpeopleresponsibleforimplementationTheprojectteamshouldbeasclearandspecificaspossibleinprovidingadescriptionoftheconcreteactivitiesthatimplementationpartnersaremeanttocarryoutPracticeprofilesmaybeahelpfultooltohelpdefinewhatisrequiredoftheactorswhoareresponsibleforimplementingtheintervention
Designimplementationandevaluation
45
ExampleInSomervillemayoralleadershipandengagementofmultiplecityagencies(publichealthprimarycareschoolsCambridgeHealthAllianceetc)streamlinedandconnectedthevariousShapeUpSomervilleinitiativesandgrantswhichhelpedtokeepthetalentandenergyfocusedonhealthyeatingandactivelivingforallsectorsofSomervilleSUSpartnersultimatelyincludedschoolsnonprofitsCityandstateagenciesandnationalfoundations
Yourturn
____________________________________________________________________________
____________________________________________________________________________
DescribetheinterventionWhatarethestepsactivitiesorprocessesRememberthatthemechanismthespecificstepsorprocessthatleadtobehaviorchangeiskeytounderstandingtheeffectivenessofthepolicychangeandsharingtheresultswithothers
ExampleGoalsofShapeUpSomervilleweretoinfluenceeverypartofanearlyelementaryschoolchildrsquosdayndashbeforeduringandafter-schooltopromotehealthyeatingphysicalactivityandtheirweightstatusInterventionincludedfoodserviceenhancementstoimprovebreakfastandlunchandstaffprofessionaldevelopmentwalktoschoolactivitiesSUSclassroomcurriculumandprofessionaldevelopmentschoolwellnesspolicydevelopmentSUSafter-schoolcurriculumandprofessionaldevelopmentoutreachandeducationtohomethroughmaterialsforumseventsldquoSUSapprovedrdquorestaurantscommunityoutreachandcapacitybuildingthroughpolicydevelopmenttrainingsmediaplacements
Yourturn
____________________________________________________________________________
____________________________________________________________________________
ClarifytimelineandmilestonesfordeliveryandevaluationHowfrequentlywilltheservicesinterventionbedeliveredHowlongwilltheprogramrunWhatisthetimetoexpectedimpacts(eg12monthsafterprogramcompletion)
Designimplementationandevaluation
46
ExampleShapeUpSomervilleledbyTuftsUniversityFriedmanSchoolofNutritionScienceandPolicyconductedaquasi-experimental(nonrandomized)interventionstudyamong1stthrough3rdgradestudentsinSomerville(intervention)andtwocomparisoncommunitiesfor9-monthsduringoneschoolyear(fall2002ndashspring2003)ThestudyshowedpositiveassociationoftheinterventiononchangeinBMISUSreleasedreportssummarizingevaluationsprogresstodateplansforadditionalprojectsandimpactstenandfifteenyearsintotheinitiative
Yourturn
____________________________________________________________________________
____________________________________________________________________________
PlanyouranalysisTheprevioussectionaddressedestablishinganaccountabilityframeworktomonitorprogressandimplementationDeterminewhichindicatorswillbediscussthroughthatprocesswhowillsharethedataandanalysisandhowfeedbackfromthoseconversationswillbeincorporatedbytheprojectteam
ExampleTheMayorrsquosofficeimplementedadata-drivendecisionmakingapproachcalledSomerStatwhichwassynergisticwiththeapproachestakenbytheresearchandevaluationteamsdeployedforthespecificgrants
Yourturn
____________________________________________________________________________
____________________________________________________________________________
PlanforsustainabilityIfthepolicyandprogramchangeissuccessfulhowwilltheprojectteamsustainandexpandtoadditionalsitesorpopulations
ExampleAlthoughmostofthepartnershipsforgedintheearlyyearsweremaintainednewgrantsandcollaborationsexpandedattentiontoolderpublicschoolstudentsactivetransportationandtheGreenLinesubwayextensionthebuiltenvironmentfarmerrsquosmarketeffortssuchasldquoGrowninSomervillerdquowithUnionMainStreeteconomicdevelopmentandhealthdisparitiesaswellasdatacollectionmonitoringandsurveillance
Yourturn
Designimplementationandevaluation
47
____________________________________________________________________________
____________________________________________________________________________
Pilotplanningtemplate
IfyouplantoconductapilotfirstdeterminethepurposeofthepilotDoyouwanttoobservewhetherthepolicychangeresultsinthedesiredbehaviorchangeAreyouconcernedaboutfeasibilityscalabilityorcostDoyouwanttotestwhetheryourdesiredresultscanbeachievedthroughthispolicyapproachClarifythegoalofthepilotandthequestionsyouhopetoanswerThiswillhelpyouidentifythekeyindicatorsthatyouwillusetomeasurewhetherthepilotissuccessfulBeforebeginningestablishabaselineofcurrentperformanceandsetyourtargetsaccordinglybasedonwhatyouhopetoaccomplishthroughthepilot
NextestablishthepilotplanforyourprojectteamincludingmilestonesforteamcheckinstodeterminewhethertomoveaheadorchangecourseThisisseparatefromthemeetingsthatwilloccuraspartoftheaccountabilityframeworkUsethespacebelowtocreateyourtimelineanddevelopyourmeetingschedule
Task Personresponsible Startdate Checkindate Deliverable
Planyourevaluation
WhileyouaredesigningtheimplementationitisalsotimetobegintodevelopyourplanforanalysisAsnotedbytheWilsonSheehanLabforEconomicOpportunitiesdecisionsaboutprogramsetup-includingoutreachscreeningintakedatatrackingandenrollment-candeterminewhetherornotarigorousimpactevaluationwillbepossibleAsdiscussedcityleadersmayfindithelpfultoincludeevaluationpartnersinearlyconversationsaboutprogramsetupDuringpreviousstagesyoumayhaveworkedwitharesearchertoplanyourproposedpolicychangeYoushouldbeclearaboutthepurposeoftheevaluationanddevelopanevaluationplanBelowareadditionalconsiderations
Causationvstrendanalysis
IntheEstablishaframeworktomonitorprogressstagetherewasguidanceontrackingtrendsovertimeItisimportanttonoteherethatmonitoringdataisdifferentfromanalyzingacausaleffectTrendanalysiswillillustratethatchangesareoccurringandmayleadtoquestionsthatcouldbeansweredbyyourevaluationbuttrendanalysisalonedoesnotprovideevidenceaboutwhatisdrivingthosechangesYoumayseechangesinyour
Designimplementationandevaluation
48
outcomeindicatorsbasedonvariablesoutsideoftheCityscontrolForinstanceachangeinfederaltradepolicycanhaveanimpactemploymentratesinyourCityoutsideofyourCitysnewjobtrainingandplacementprogram
ToaddresscausalityyoumustcomparetheoutcomesofindividualswhoreceivedtheinterventionwiththosewhodidnotWithoutacomparisongroupofsimilarindividualswhowerenotexposedtotheinterventionitisimpossibletoknowwhichfactorsareresponsiblefortheoutcomesRandomizedControlTrials(RCTs)inwhichindividualsarerandomlyassignedtotreatmentandcontrolgroupsarethegoldstandardinprovingcausalityIfyoucandemonstratethatoutcomesimprovedforthosewhoreceivedtheinterventionanddidnotimproveatallorimprovedlessforacomparisongroupyoucanbesomewhatcertainthattheinterventionwasthecause
Examplesofothernon-causalimpactevaluationsthatarerigorousandreliableincludecertainquasi-experimentalstudiesusingcomparisongroupdesignssmartinterruptedtimeseriesanalysesthatmodeltrendsovertimeinwaysthatallowestimationofinterventioneffectswelldesignedpre-posttestanalysesneedsassessmentsimplementationstudiesetcFormoreinformationondifferenttypesofevaluationsseetheSageResearchMethodswebsite
Lookfordifferencesingroups
BuildintimetoalsoexaminethepilotdatatoseeifgroupsareexperiencingdifferentresultsReviewingperformancebasedonoutcomemeasuresforapopulationinaggregatemayhidethesubtlewaysthatsomepeoplemaybeaffecteddifferentlysoitisnecessarytocarefullyanalyzethedatafordisparateoutcomesForinstancetheMovingtoOpportunity(MTO)studythatexaminedtheimpactofhousingvouchersonfamilieswholivedinhigh-povertyneighborhoodsfoundthatchildrenwhomovedbeforetheageof13hadbettereducationandhigherincomesasadultsHoweveradultsdidnotseeimprovededucationandemploymentoutcomes
Disaggregatingdataintomorespecificcategoriesofraceandethnicitycanhelpunderstandtrendsanddisparitiesinamoregranularwaywhichcanhelpdecisionmakerstargetpoliciesandinterventionsinamoreequitableway
EarlierintheRoadmapweworkedtoidentifystakeholdersandwhomightbepositivelyandnegativelyimpactedbyachangeinpolicyUsetheresultsofthatexercisetoinvolvethecommunityininterpretingdataanalysisandtogatherqualitativefeedbackfromthecommunitytocallattentiontoissuesForinstanceanincreaseinyouthdetentionmaybeexplainedbyinefficienciesorlackofaccesstotransportation
Designimplementationandevaluation
49
Designimplementationandevaluation
50
Shareyourstoryandsuccess
Sharetheresults
Storytelling
AnintegralstepinyourstrategyissharingtheresultsofyourpilotorpolicychangewithyourcommunityThisincludesdevelopingmessagingaroundtheworkofyourorganizationandcreatingaplantodisseminateyourstoriesWhytellstoriesaboutyourwork
Informandorunderstandasubjectorsituation
Giveneededcontextofdataorbitsofinformation
Relatablestoriesarememorableforreadersviewers
ReadersorviewerswillseetheinformationasmorecompellingwhenthereisademonstratedandrelatablechangeorimpactonthemselvesortheirfriendsneighborsandcolleaguesStoriesareaboutpeopleandgovernmentisfundamentallyabouthelpingpeopleSotellthestoriesofthepeopleyouandyourcitycolleagueshelped
Keepthesetipsinmindasyoudevelopyourstorywithyourteam
WhatwereyourfindingsandwhyshouldanyonecareTellyouraudiencehowtheproblemwassolvedWhathastheimpactbeenonresidentsandcitystafftodateWhatwasthesuccessandweretheremultiplesuccessesalongthewayIfyouhavepreliminaryresultsordatashowingimprovementtosharethisisthetimetodoit
WhowasinvolvedandimpactedReviewthelistofstakeholdersHowwereeachofthemimpactedbytheprogramHumanizeyourstoryandimpactbyaddingadditionalvoicesThesevoicescanbeoneofthefollowing
Apersonorfamilywhowaspositivelyaffectedbythechangesuccess
Aninvolvedthird-partydescribingboththeproblemandthesolution(ieanorganizationnonprofitprovidersbusinesstradeassociationsportsteametc)whoispartofthecommunitythatthegovernmentworkedwithonthechangesuccess
Someonefromthegovernmentitselfwhocanspeakaboutthechangesuccess
TellmoreaboutyourcityDescribeyourcityincludingpopulationdemographicsindustriesetcWhatareitschallengesWhatisitknownforWhatmakesituniqueHowisitsimilarordifferentfromothercitiesintheUS
Shareyourstoryandsuccess
51
DescribeyourcityrsquoschallengeWriteoutthechallengeyourcityisfacing(asdefinedinyourproblemstatement)HowdoesitaffectresidentsHowdoesthisproblemaffecttheworkofcitystaffmembers
AddthedetailsProvidedetailsonyourappliedresearchstrategyTalkaboutwhatyourcityhasdonenthepasttoresolvetheissueDiscusshowyourstrategyhaschangedwhenyoubegantoadoptthisapproachWasmoneysavedorinvested
Usethisstorymapexercisetoorganizeyourthoughtsandguaranteethatyouincludealloftheelementsofatraditionalstory
AfteryoudevelopyourmessagingcreateacommunicationsmarketingandoutreachstrategydesignedtospreadyourmessageYourstrategyshouldwalkyouthroughthefollowingkeyactivities
1 IdentifyandprofileyouraudienceWhencreatinganddisseminatingyourmessageitisalwaysgoodtothinkaboutyouraudienceWhoexactlywouldbeinterestedinwhatyouhavetosayThegeneralpopulationofthecityPeoplewhoworkinnonprofitsBusinessownersItmattersbecausenoteveryonewholivesinacitymaybethetargetoftheinformationBusinessownersmaybemoreinterestedindatashowingeffectsonthebusinessdistrictthanothersandsoonWewillcoverthislaterbutthisstepwillalsohelpyoutothinkofthemosteffectivewaytoreachyouraudience
2 Defineyourcalltoaction(ifyouhaveone)SpendsometimeconsideringwhyyouareinterestedintellingyourstoryDoyousimplywanttoupdatethepubliconyourprogressorcelebrateyoursuccessesAreyouinterestedinyouraudiencechangingitsbehaviororsupportingaspecificinitiativebeinglaunchedbyyourorganizationIfyouwanttohaveyouraudienceactorbehaveinacertainwaybespecificandincorporateitinyourmessaging
Shareyourstoryandsuccess
52
3 SelectperformancemeasuresThinkabouthowyouwoulddetermineifyourcampaignissuccessfulandselectappropriateperformancemeasuresYouwillrevisitthesemeasuresattheendofyourcampaigncommunicationsplanimplementationtodeterminewhetheryouneedtomodifyyourcommunicationsandoutreachstrategyforthefuture
4 MethodstocommunicateyourstoryTherearemanywaystocommunicatesuccessofyourprojectHerearesomesuggestionsonreachingaudiencesItisimportanttodeterminewhichmethodsarethemosteffectiveforyourdesiredaudiencetoreceiveinformationDoyourresearch(PewResearchCenterhasexcellentstudiesonthereachofmediadigitalmediaandmarketing)ForexampleifseniorsareyourtargetpopulationandtheyarelessinclinedtouseInstagram(thatistrue)youmaywanttoconsiderFacebookorthenightlylocalnewsbothofwhichareusedmoreoftenbyseniorsBelowisalistofsomemethodsandtoolsyoucanuse
i Pressreleasestolocalnewspapersandtvstations
ii ShortsmartphonevideosAgreatwaytoengageanaudienceisthroughvideoAskonequestionofsomeonewhoispartofthestoryMaybethereisasocialworkerwhocandiscusshungerinyourcityThesevideoscanbereusedonmultipleplatforms
iii Long-formwriting
iv TellingindividualstoriesthroughaseriesofphotosandshortnarrativesforpublicationonFacebookorInstagram
v Op-edsopinionpiecesplacedinthelocalpaper
vi Emailupdatestoyourlist
vii Connectingwithappropriateorganizationsthatareinvestedinthetopicandworkwiththecommunityandaskthemtodisseminateinformationtotheirmembers
viii EmployingldquoWordofMouthrdquo(WOM)tacticsbycreatingtownhallsorfreeeventsforresidentstoattendintheirownneighborhoods
5 DeterminethetypeofcollateralUsuallycampaignsthattargetresidentsrequireseveralavenuestoreachthemPeoplearenotalwaysinthesameplaceorspaceHavingakeenunderstandingofyouraudiencewillassistyouwithyourchoicesonwhatneedstobedevelopedSometimesthatmeansimagesforsocialmediaaonepageflierthatgivespeopleinformationonattendinganeventoranin-depthbrochure
Shareyourstoryandsuccess
53
6 ImplementyourstrategyKeepinmindstrategiescanbefluidSometimessomethingmaynotturnoutasexpectedevenifduringplanningsessionsitwasldquosupposedtordquoItisokYoucananalyzewhathappenedduringadebriefsessionandonceyoursurveysandothermeasurementshavebeenfinalized
Visualizations
Withtherightvisualizationyoucanreviewandshareinformationinsightsanalysisperformanceandtrendsallofwhichisvaluabletoacross-sectionofthepublicButwhichvisualizationisrightforyouTochoosetherighttypeofchartgraphdeterminethetypeofquestionyouwanttoanswerSomecommoncharttypesarebelowandmoreinformationcanbefoundhere
Exploreadditionalanalyticaltechniques
Duringthegoal-settingphaseyouestablishedoutputoutcomeandimplementationmeasuresBasedontheresultsofyourpilotyoumayneedtoadjustyourmeasuresWiththepilotdatainhandyoumayhavesomeideasofhowfutureanalysescouldbedifferentandpotentiallyimprovethequalityofyourprogramTakesometimetobrainstormotherrelevantdatasetsthatyoudidnothavethetimetopursueearlierincludingmessydepartmentaldatathatneedsmoreattentionorsourcesofpublicdatafromthestateandfederalgovernmentnonprofitsandevenprivatecompaniesthatcouldsupplementyourunderstandingofanissueUsingmachinelearningtechniquesyoumightbeabletobetter
Shareyourstoryandsuccess
54
predictdemandforhomelessservicesusingNOAAweatherforecastspreemptwatermainbreaksusingsoilstratigraphyorinvestineconomicdevelopmentbymonitoringrealestatelistingdata
Advancedanalyticaltechniquescanalsohelpyourimplementationrunmoreeffectivelybyintegratingdata-informedtargetinganddecisiontoolsintotheprocessByautomatingroutineworkflowsandsurfacingsubtletrendsthesetoolscanhelpyouiterateinamoreprecisemannerandprovidekeystrategicguidancetofront-lineemployeesthatwouldotherwisebeunavailableThesesortsofdata-science-enhancedprocessescanbeclassifiedasoneofsixprojecttypesasexploredfurtherbytheNewOrleansLANOLAlyticsteamFormoreguidanceonscopingasimilarprojecttheGovExadvancedanalyticsscopingguidegivesyouastep-by-stepoverviewofhowtobesuccessful
Usethespacebelowtolistyourideasforpossibleadvancedanalyticsprojectsbytype
Typeofproject Ideas
Findingtheneedleinthehaystack
Prioritizingforimpact
Earlywarningtools
Betterquickerdecisions
Optimizingresourceallocation
Experimentingforwhatworks
Keepthefeedbackloopgoing
GatherinformationonpublicsentimentaboutpolicychangethroughfocusgroupssurveysandpublicmeetingsledbyadvisorygroupmembersNotewhethernewinformationisavailablethatcanhelpanalyzeorcontextualizeyourprogress(egfederalorstatedatareports)Sharetheresultsofyourpilotandthepolicychangeprocesswithothercitiesandresearchers
Shareyourstoryandsuccess
55
WorkingwitharangeofevidencePresentlythereisalotofattentionandinvestmentinevidence-basedpolicymakingincludingthefederalbipartisanCommissiononEvidence-BasedPolicymakingthePewCharitableTrustsreportsandtoolkitsonHowStatesEngageinEvidence-BasedPolicymakingandtheLauraandJohnArnoldFoundationsupportedEvidence-BasedPolicymakingCollaborativeManyoftheseinitiativesfocusonrigorousevaluation--meaningrandomizedcontrolledtrials(RCT)orotherexperimentalmodels--asthebestkindofevidencetouseHoweverthisbodyofevidenceisofteninsufficientforthespecifickindsofquestionsthatgovernmentscanhavewhenmakingadecisionaboutpolicyInthisguideweproposeanapproachtosupportgovernmentsinincorporatingadditionaltypesofevidenceintotheirdecisionmaking
OnewaytounderstandarangeofevidenceistouseNestarsquosStandardsofEvidencetoframeanddefineevidenceTheBritishnonprofitNestaworkstoinjectpositivechangeintothewayUKgovernmentpoliciesandprogramsfunctionOneoftheirkeyareasofworkhasbeendevelopingWhatWorksCentresorganizationsthatcollectcommissionsynthesizeanddisseminateevidenceaboutinterventioneffectivenessacrossarangeofsocialimpactareasNestadevelopedthefollowingStandardsofEvidencetoprovideausefulwaytodocumenthowdifferentkindsofinformationdemonstratedifferentlevelsofevidence
Workingwitharangeofevidence
56
Theimportanceofpotentiallybiasedinformation
Onechallengepresentedbyanevidenceclassificationthatacknowledgesthevalueofnon-scientificformsofevidenceistofigureouthowtobalanceallofthepiecestogetherwhenyouusethemTounderstandhowtodothiswehavetothinkabouthowtopositionpotentiallybiasedinformationindecisionmaking
WhenwelookatinformationwhichhasnotbeenrigorouslyevaluatedweacceptthatthisinformationmightbebiasedThewholereasonweconductrigorousevaluationistoremovebiasfromourobservationsWeconductstructuredtrialstoproduceresultsthatareconsistentandgeneralizableIfafindingisunbiasedthisaccuracymdashthislackofbiasmdashcanbeprovedbecauseanyindependenttestercanreproduceitwhentheyfollowanidenticalprocess
YoucanlookatpotentiallybiasedinformationasjustinformationthatisearlieroninitspathtorefinementreplicationandverifiedaccuracyApieceofinformationbasedonanyoneindividualrsquosobservationsislikelytobebiasedbecauseitiscomingfromjustonesourceBiasisreducedbygatheringtogetherlotsofindependentsourcesofinformationandfiguringoutitsmostcommonattributes
MaybethebiasedinformationmakesaclaimthatistoolargemdashitsayssomethingistrueunderallconditionswhenitisreallyonlytrueundercertainconditionsorthataneffectofaninterventionisbiggerthanitstrueaverageorthataneffectlastsovertimewhenitdoesnotPerhapschangingtheclaimtobemorespecificmdashcriticallyreviewingtheinformationtoseeifspecificconditionsmustbemettoachievetheoutcomesmdashwouldmaketheclaimmoreaccurate
IfyoutestedyourclaimfounditdidnotholdupinsomecaseschangedittesteditagainuntilyourclaimrepeatedlyheldupthenyoucoulddemonstratethattheclaimwasnolongerbiasedbutaccurateThisprocessoftestingimprovingtestingandvalidatingiswhatwegetfromstructuredscientificstudy
Withindividualpiecesofinformation--crediblehypothesescasestudiesexpertopinion--youcannotassumethatkindofaccuracyAtthesametimetheseearlyobservationsarefoundationaltoanyreplicablescientificobservationsWewillnevergetanywhereifwedonotstartwiththesepiecesofinformationeveniftheymightbebiased
Combininglevelsofevidencetogetthemostdetailedinformationforyourcity
ItisespeciallyimportanttoacceptdifferentformsofevidencewhenyoufacequestionsthathavenotyetbeentestedbyindependentstudiesSomeofthemostimportantquestionsrelatetothingsthatareveryspecifictoyoursituationYoufaceaparticularcombinationof
Workingwitharangeofevidence
57
issuesactorstrendsandresourcesthatisultimatelydifferentfromeveryotherplaceintheworld
ThisisimportantbecauseaspolicypractitionersandnotscientistsyoucaremoreaboutyourspecificcasethanahypotheticalaveragegeneralcaseYoucarelessaboutthetotallikelyrangeofimpactsfromapolicythanhowthissingleimplementationwillworkYouwantinformationaboutpoliciesthatwillprovideyouwiththebestpossiblepredictionabouthowtheywillworkinyourcontext
ByputtingdifferentkindsofevidenceintothesameframeworkNestarsquosStandardsofEvidenceimplythatwhenyoucannotgetrigorouslytestedexperimentallyverifiedinformationitisappropriatetoworkwithwhatwedohaveThisisrelevantforthemunicipalimplementationcontextbecausenostudypulledfromaliteraturereviewwillaccountforallofthevariablesyoufaceinyourowncontext
Intheabsenceofhavingastudythatpredictshowaninterventionworksinyourspecificcontextyouwillneednon-scientificinformationtotrytofillinthosegapsYouwillneedtogatherstoriesfromresidentsstaffmembersandotherstakeholdersthatsitonLevels1or2ofthestandardsofevidence
Someofthekindsofinformationyouwillprobablygetmoreeasilyfrompeoplethanstudiesinclude
howpeopleincitieslikeyoursreactedtoaninterventioninpublicforumshowpeoplewhoimplementedaninterventioninacitylikeyoursfiguredouthowtogetitpassedhowthecitycommunicatedpubliclyabouttheapparentbenefitsanddrawbacksoftheinterventionthepoliticalacceptabilityoftheintervention
Youcanremainsimultaneouslyawarethattheinformationyouaregatheringthroughindividualstoriesisbiased--notgeneralizableinformedbyspecificperspectives--whilealsorecognizingthathavingsomeinformationaboutsomethingimportantisbetterthannoinformationEvidencefromthefirsttwoevidencelevelsareessentialbuildingblocksofknowledgeandperfectlyusefulwhenyoucannotgetdetailedunbiasedknowledge
Youhavetostartsomewhereandyoushouldalwaysstartwithevidenceyouhaveaboutyourveryspecificconditionsthatyouknowareimportantforsuccess
Workingwitharangeofevidence
58
Thankyou
Interviewswithcities
GovExheldsemi-structuredinterviewswithsixstaffmembersinfourcitiestoestablishtheinitialstepsoftheRoadmapWeaskedrespondentstotalkaboutaninitiativeorpolicythattheyhavebeenresponsibleforimplementingintheircityRespondentsfromthreecitiesspokeabouttheireffortstoaddresshomelessnessandrespondentsfromonecityspokeaboutblightAfterrespondentsidentifiedtheproblemtheyspokeabouthowthecityprioritizedthisissuethevariousstepsinvolvedandbarrierspresent
GovExconductedtextanalysisoftheseinterviewswhichrevealedthatpeoplewasthemostcommonlyusedtermduringtheseconversationswhendiscussingthestepstochangeaprogramorpolicyThisisathemethroughoutthisRoadmapThefirstsectionbeginswithidentifyingtherelevantpeopleTheconversationsaroundevidencearefocusedonhumanbehaviorratherthanmethodologyorstrengthofevidence
Thankyou
59
OncewehadadraftoftheRoadmapwefollowedupwithcitiesonanindividualbasistolearnmoreaboutwhatwouldbehelpfulWearegratefultothefollowingcitystafffortheinterviewsandindividualconversations
MichaelBaskinChiefPolicyOfficerChattanoogaTN
MelissaBridgesPerformanceandInnovationCoordinatorLittleRockAR
VictoriaCarreoacutenAdministrativeOfficerLasVegasNV
TyroneGrandstrandCommunityDevelopmentPlannerFargoND
TibLaughlinDirectorofGeneralServicesKansasCityKS
TimMorelandDirectorofPerformanceManagementandOpenDataChattanoogaTN
CharlesRosendahlManagementAnalystDentonTX
Thankyou
60
MelissaSiebenAssistantCountyAdministratorKansasCityKS
GinaThompsonDirectorofStrategicServicesLewisvilleTX
JHUFacultycollaboration
GovExalsoconvenedthecitycontactsandfacultypartnersinaseriesofAdvisoryGroupmeetingsThesemeetingsservedtoconnectcitiesandfacultytounderstandoneanothersperspective
StefanBaralBloombergSchoolofPublicHealth
AshleyBernerJHUSchoolofEducation
MelissaDavey-RothwellBloombergSchoolofPublicHealth
JonLinksBloombergSchoolofPublicHealth
TamarMendelsonBloombergSchoolofPublicHealth
KristinMmariBloombergSchoolofPublicHealth
KeshiaPollackPorterBloombergSchoolofPublicHealth
JoshSharfsteinBloombergSchoolofPublicHealth
LizStuartBloombergSchoolofPublicHealth
LindsayThompsonCareyBusinessSchool
DanielWebsterBloombergSchoolofPublicHealth
Partnersindataandevidence-basedpolicy
Wealsospokewithpartnerswhoareexpertsinthefieldofevidence-basedpolicywhosharedtheirknowledgeandresourcesofcurrentandfuturethinkinginevidence-informedthinking
WendyBarrenoAssociateDirectorWilsonSheehanLabforEconomicOpportunitiesUniversityofNotreDame
TiffanyDavenportDirectorofDataPartnershipsBaltimoresPromise
TomiHiersDirectorBaltimoreCivicSiteAnnieECaseyFoundation
ChrisKingsleySeniorAssociateAnnieECaseyFoundation
KathrynOliverAssociateProfessorLondonSchoolofHygieneandTropicalMedicine
Thankyou
61
KristenWatkinsSeniorPolicyAssociateJ-PAL
GovExParticipants
GovExAnalystMiriamMcKinneyanalyzedtheinterviewtranscriptsandconductedtextanalysisSeniorImplementationAdvisorEmilyShawwillintegrateRoadmapactivitieswithadditionalservicedeliveryactivitiesatGovEx
Thankyou
62
ReferencesAbrahamKGampHaskinsR(2017)Thepromiseofevidence-basedpolicymakingReportofthecommissiononevidence-basedpolicymakingWashingtonDCCommissiononEvidence-basedPolicymakingBigmove2Extendingthegreen(nd)Retrievedfromhttpswwwcityoflewisvillecomdoing-businesslewisville-2025big-move-2-extending-the-greenBatesMAampGlennersterR(2017)ThegeneralizabilitypuzzleRetrievedfromhttpsssirorgarticlesentrythe_generalizability_puzzleBuildwithnotfor(2018January)RetrievedfromhttpsbuildwithnotforwordpresscomCairneyPampOliverK(2017)Evidence-basedpolicymakingisnotlikeevidence-basedmedicinesohowfarshouldyougotobridgethedividebetweenevidenceandpolicyHealthResearchPolicyandSystemsdoi101186s12961-017-0192-xCairneyPOliverKampWellsteadA(2016)TobridgethedividebetweenevidenceandpolicyReduceambiguityasmuchasuncertaintyPublicAdministrationReview76(3)399-402Retrievedfromhttpdxdoiorg101111puar12555Centreforpublicimpact(nd)RetrievedfromhttpswwwcentreforpublicimpactorgCernaL(2013)ThenatureofpolicychangeandimplementationAreviewofdifferenttheoreticalapproachesOrganisationforEconomicCo-OperationandDevelopmentRetrievedfromhttpswwwoecdorgeduceriTheNatureofPolicyChangeandImplementationpdfColoradoSprings(nd)Retrievedfromhttpscoloradospringsgovcommunicationspagewhat-expect-when-you-speakupContinuumofcare(coc)program(nd)RetrievedfromhttpswwwhudexchangeinfoprogramscocCouncilpresidentkicksoff$12millionyouthfund(nd)Retrievedfromhttpwwwbaltimorecitycouncilcomcontentcouncil-president-kicks-12-million-youth-fundDurlakJAampDuPreEP(2008)ImplementationmattersAreviewofresearchontheinfluenceofimplementationonprogramoutcomesandthefactorsaffectingimplementationAmericanJournalofCommunityPsychology41327-350Retrievedfromhttpswwwncbinlmnihgovpubmed18322790Fishbonediagram(ishikawa)(nd)Retrievedfromhttpasqorglearn-about-qualitycause-analysis-toolsoverviewfishbonehtmlFivewhysGettingtotherootofaproblemquickly(nd)RetrievedfromhttpswwwmindtoolscompagesarticlenewTMC_5Whtm
References
63
FligsteinNampMcAdamD(2011)TowardageneraltheoryofstrategicactionfieldsSociologicalTheory29(1)1-26RetrievedfromhttpsociologyberkeleyedusitesdefaultfilesfacultyfligsteinFligstein20McAdam20Soc20Theory20PaperpdHeinkelP(2018January12)CitymullspermitsforbikesharingcompaniesRetrievedfromhttpwwwdentonrccomnewsdenton20180112city-mulls-permits-bike-sharing-companiesHowisareamedianincomecalculated(2012October)Retrievedfromhttpswwwhudexchangeinfofaqs843how-is-area-median-income-calculatedLampkinLMampHatryHP(2003)KeystepsinoutcomemanagementUrbanInstituteRetrievedfromhttpswwwurbanorgresearchpublicationkey-steps-outcome-managementMarshDampSmithM(2000)UnderstandingpolicynetworksTowardsadialecticalapproachPoliticalStudies484-21Retrievedfromhttpjournalssagepubcomdoiabs1011111467-924800247McCannL(nd)RetrievedfromhttplaurenellencomMoultonSampSandfortJR(2016)ThestrategicactionfieldframeworkforpolicyimplementationresearchPolicyStudiesJournal45(1)Retrievedfromhttponlinelibrarywileycomstore101111psj12147assetpsj12147pdfv=1ampt=jdkhhcp4amps=e184c625e1855a6b1615ccb99ee27b1c0b3212d8PaynterB(2018January1)What100citiesarelearningfromeachotherbysharingtheirdataRetrievedfromhttpswwwfastcompanycom40518004what-100-cities-are-learning-from-each-other-by-sharing-their-dataPeakacademy(nd)Retrievedfromhttpswwwdenvergovorgcontentdenvergovenmayors-officeprograms-initiativespeak-performancepeak-academyhtmlPinnockHBarwickMCarpenterCREldridgeSGrandesGGriffithsCJRycroft-MaloneJMeissnerPMurrayEPatelASheikhAampTaylorSJC(2017)Standardsforreportingimplementationstudies(stari)ExplanationandelaborationdocumentBMJOpenRetrievedfromhttpbmjopenbmjcomcontentbmjopen74e013318fullpdfProctorESilmereHRaghavanRHovmandPAaronsGBungerAGriffeyRampHensleyM(2011)OutcomesforimplementationresearchConceptualdistinctionsmeasurementchallengesandresearchagendaAdministrationandPolicyinMentalHealth38(2)65-76doi101007s10488-010-0319-7TandonSD(2018)ProgressincommunityhealthpartnershipsResearcheducationandactionRetrievedfromhttpswwwpressjhuedujournalsprogress-community-health-partnerships-research-education-and-actionProvanKGampKenisP(2007)ModesofnetworkgovernanceStructuremanagementandeffectivenessJournalofPublicAdministrationResearchandTheory
References
64
18(2)229-252Retrievedfromhttpsacademicoupcomjpartarticle182229935895SabatierPA(1988)Anadvocacycoalitionframeworkofpolicychangeandtheroleofpolicy-orientedlearningthereinPolicySciences21(3)129-168Retrievedfromhttpwwwjstororgstable4532139SandstromAampCarlssonL(2008)TheperformanceofpolicynetworksTherelationbetweennetworkstructureandnetworkperformancePolicyStudiesJournal36497-524Retrievedfromhttponlinelibrarywileycomdoi101111j1541-0072200800281xabstractSchwartzMampWilsonE(nd)WhocanaffordtoliveinahomeAlookatdatafromthe2006americancommunitysurveyRetrievedfromhttpswwwcensusgovhousingcensuspublicationswho-can-affordpdfShearerJCAbelsonAKouyateBLavisJNampWaltG(2016)WhydopolicieschangeInstitutionsinterestsideasandnetworksinthreecasesofpolicyreformHealthPolicyandPlanning31(9)1200-1211Retrievedfromhttpsdoiorg101093heapolczw052ThelessonsofadministrativedataHigh-profilepolicy-relevantresearchpoweredbyadministrativedata(nd)Retrievedfromhttpswwwpovertyactionlaborgsitesdefaultfilesresources20161114-The-Lessons-of-Administrative-DatapdfToolsandotherneedsforqualitativeandquantitativemethods(2015February)AgencyforHealthcareResearchandQualityRetrievedfromhttpwwwahrqgovprofessionalsquality-patient-safetytalkingqualityassesstoolshtml
WeaverRCampBagchi-SenS(2013)SpatialanalysisofurbandeclineThegeographyofblightAppliedGeography4061-70
References
65
UniversityPartners
Name Title Affiliation Area(s)ofexpertise
Contactinformation
Examplesofworkingwithnon-academicdecisionmakers
WendyPBarreno
AssociateDirector
WilsonSheehanLabforEconomicOpportunitiesUniversityofNotreDame
LEOsfocusareasincludehealtheducationhousingchildwelfareeconomicopportunitycriminaljustice
wbarrenonduduorleondedu
MelissaDavey-RothwellPhD
AssociateScientist
JohnsHopkinsBloombergSchoolofPublicHealth
Health mdavey1jhuedu410-614-5854
BaltimoreCityHealthDepartmentLeadersMarylandDepartmentofHealthAddictionRecoveryServicesinDCExecutiveDirectorsofcommunityorganizationsacrosstheUSthatfocusonHIVSTIsubstanceabuseandimplementationscience
ElizabethStuart
AssociateDeanforEducationandProfessorDepartmentsofMentalHealthBiostatisticsandHealthPolicyand
JohnsHopkinsBloombergSchoolofPublicHealth
Educationsocialpolicystatisticalmethods
estuartjhuedu
Resources
66
Management
Design-ThinkingResources
EngagementGuideforCivicTech
Journeymap
PerformanceManagementResources
DenverPeakAcademy
Proxymeasuresguide
Talkingaboutperformancewithoutcomplicatingit(video)
SettingPerformanceTargetsguide
Datavisualizationguide
Performancemanagementgettingstartedguide
Exercises
Fishboneexercise
5Whys
AdministrativeData
TheLessonsofAdministrativeData(JPAL)
EvaluationResources
Quantitativeandqualitativemethods(USDepartmentofHealthandHumanServicesAgencyforHealthcareResearchandQuality)
KeyStepsinOutcomeManagement(UrbanInstitute)
RCT-YES
Sageresearchmethods
BestPracticesforMixedMethodsResearchintheHealthSciences
ACCORDChecklist
Pilotandimplementationplanning
Benchmarkingexamplesandproxymeasures
Seattleopendataplaybook
Resources
67
Datasciencescopingguide
Resources
68
AboutGovExGovExhelpsgovernmentsusedataandevidencetoimproveresidentsrsquolivesWesupportgovernmentsinbuildingtheirinternalcapacitytooperatesustainableanalyticspracticeswhichimprovecommunityoutcomesinfieldsfrompublicsafetytoeconomicdevelopmentGovExhasaproventrackrecordofworkinginconcertwithpartnerstoprovidetechnicalassistanceandtrainingatscale
SinceourlaunchinApril2015GovExhas
Trainedover2000governmentemployeesinpersonandonlineintopicsincludingdatamanagementperformancemanagementopendatacommunityengagementandanalytics
Providedtechnicalassistancetomorethan100localgovernmentsacrosstheglobeimprovingtheiruseofdatatoachieveefficienciesinservicedeliveryandbetteroutcomesforresidents
Assessedthecapacityofmorethan130governmentalorganizationstodeterminehowleadersstaffandexternalstakeholdersmanageandusedata
Localgovernmentsareincreasinglyonthefrontlinesofaddressinglargescaleproblemswithglobalimplicationssuchasclimatechangepublicsafetyandeconomicinequality
Whileevidence-basedinterventionstohelpsolvethoseproblemsareavailablelocalgovernmentstaffhavelimitedtimeandcapacitytofindandimplementresearch-basedsolutionsAsaresultcitiescontinuetoworkthewaytheyalwayshave--makingincrementalimprovementstoprogramsbutnotalwayscreatingchangeatasystem-level
ThroughresourcedevelopmentandcollaborationwithJHUfacultyGovExissupportingcityleadersinusingdataandevidencetomakedecisionsandcreatinglastingrelationshipswithresearchers
TheRoadmapwasdevelopedbyGovExaspartofitspartnershipwithBloombergPhilanthropiesWhatWorksCitiesinitiative
AboutGovEx
69
- Title page
- Introduction and purpose of the Roadmap
- Using the Roadmap
-
- Gather information about the problem
- Strategically engage key partners
- Review the evidence and collect details
- Decide on a policy to implement based on local factors
- Establish a framework to monitor progress
- Assess data accessibility quality and security
- Design implementation and evaluation
- Share your story and success
-
- Working with a range of evidence
- Thank you
- References
- Resources
- About GovEx
-
Division-DivisionwherethedataismaintainedIfmorethanonedivisionisresponsiblelisttheprimaryownerofthedataset
Sensitivedatacomments-DescribetheissuesthatwouldpreventthecityfromsharingthedatapubliclyCommonsensitivityconcernsincludeprivacyviolationssecurityissuesorhighcostorstaffdemandstopostthedata
Publishingstatus-Thistheplacetoputanycommentsregardingthecurrentstatusofthedataset
Publishedlink-Thisisthelinktothepublisheddatasetontheopendataportalorcitywebsite
Reviewdatasecurityandprivacyprotectionprotocols
OftenapolicychangealsoinvolvesachangeinthedatathatisbeingcollectedhowthatdataisbeinghandledwhohasaccesstoitandahostofotheradjustmentsAsyoutakestockofthedataassetsthatareavailableforyourinitiativeitisvitaltoalsothinkthroughwhatwillbedonetoprotectthatdataandtheprivacyofthepeopleitrepresentsAbigpartofgooddatasecurityinvolvesstrongdatagovernancepracticesthatdefineandidentifysensitivedataandconsciouslyworktobalancethevalueandriskofstoringsuchsensitivedataOnewayofevaluatingyourdatacomesfromcaselawrelatedtotheUSFreedomofInformationActwhichwehavetranslatedtothechecklistbelow
1 Woulddisclosureresultinasubstantialinvasionofprivacy2 Haveweconsideredtheextentorvalueofthepublicinterestandthepurposeorobject
oftheindividualsseekingdisclosure3 Istheinformationavailablefromothersources4 Wastheinformationgivenwithanexpectationofconfidentiality5 Isitpossibletomouldreliefsoastolimittheinvasionofindividualprivacy
TheseriskscouldbemitigatedusingavarietyofstrategiesfromaggregatingdatatoobfuscatingindividualstoredactingfieldsthatmayincludePersonallyIdentifiableInformation(PII)orre-evaluatingyourdatacollectionmethodstofindalternativewaysofgettingtheinformationyouneedThesestepsshouldbecodifiedandmanagedbyadatagovernancecommitteeifpossibleForthesensitivedatathatisdeemedessentialcreateaplanaroundpotentialdatabreachesinordertoquicklyandconfidentlyrespondtoemergencysituations
ProactivelypreparingforthesesituationsisoftenthebeststrategyfordatasecuritybutdonotforgetthetechnicalsideInvolveyourITadministratorsinconversationsaboutdatastorageandaccessensuringthattheywillbeprotectedbyenterprise-gradecybersecuritysystems
Assessdataaccessibilityqualityandsecurity
42
Assessdataaccessibilityqualityandsecurity
43
Designimplementationandevaluation
Designtheimplementation
NowthatyouhaveyouraccountabilityplanandkeyindicatorsitistimetogetspecificaboutyourplanforimplementationThetemplatebelowwillhelpyouthinkthroughthespecificmechanismsofimplementingpolicychange
UnderstandingorganizationsandsystemsiscriticaltounderstandingandmakingdecisionsaboutwhereandhowinterventionsshouldbeappliedApolicyimplementedinasinglesettingmayshowpromisingresultsbutimplementingmultiplepoliciesandprogramsatthesametimewithdifferentactorsresponsibleforeachaspectcanhelpensuresustainablebehaviorchange
InSomervilleMAShapeUpSomerville(SUS)isacitywidepublichealthprogramthatisimplementedacrossmultiplelevelsofinfluenceandsystemsthroughinterventionsbeforeduringandafterschooltoincludechangestobuiltenvironmentsuchasbikelanesregionalmasstransportationandphysicaleducationequipmentinschoolsandgymstoincreasephysicalactivitywellnesspoliciesandschoolgardensnutritioneducationduringandafterschoolandhealthyrestaurantprogramstosupporthealthyfoodandhealthyeatinghabits
Usethetemplatebelowtoplanyourimplementation
Policyimplementationplanningtemplate
ListthespecificdesiredoutcomesInthepreviousstepyouworkedwithdepartmentstafftoestablishspecificmeasurableoutcomesandinterimindicatorsPrioritizetheindicatorsthatwilldemonstrateprogressUsethespacebelowtocapturethedesiredoutcomesofthepolicychange
ExampleShapeupSomervillebeganwiththegoalofpreventingobesityin1stthrough3rdgraders
Yourturn
____________________________________________________________________________
____________________________________________________________________________
Designimplementationandevaluation
44
SpecifythetargetpopulationAreyoudesigninganinterventionthattargetsonlycertainsitesAreyouplanningonimplementingaprogramorinitiativeforaspecifictargetaudienceClearlyspecifythepopulationyouplantotarget
IfyouplantoconductarigorousimpactevaluationasnotedbytheWilsonSheehanLabforEconomicOpportunitiesitisimportanttoestablishconcreteobjectiveeligibilitycriteriaandensurethatthesecriteriaareappliedconsistentlybyallstaffduringscreeningandintakeItisalsovitaltoidentifytheindividualswhoareeligibleforaparticularprogramorserviceevenifeveryoneeligiblecannotbeservedbyexistingresourcesThishelpscitiesidentifyanyunmetneedsandguidefutureresourceallocationwhilealsocreatingthepossibilityofarigorousimpactevaluationbyidentifyingapotentialcomparisongroupFinallywhenconductingscreeningandintakeitisimperativetocollectandmaintainbasicinformation(egnamedateofbirthgender)aboutallprogram-eligibleindividualsnotjustthosewhoparticipateintheprogram
ExampleIntheShapeUpSomervilleexampletheprogrambeganbytargeting1stthrough3rdgraders
Yourturn
____________________________________________________________________________
____________________________________________________________________________
DefinethelocationWherewillthepolicychangetakeplaceWillimplementationtakeplaceatonesiteormultiplesites
ExampleShapeUpSomervilleinterventionsweredeliveredinschools
Yourturn
____________________________________________________________________________
____________________________________________________________________________
ClarifyimplementationpartnersandarticulatetheirrolesWhatarethekeyorganizationsinstitutionsorpeopleresponsibleforimplementationTheprojectteamshouldbeasclearandspecificaspossibleinprovidingadescriptionoftheconcreteactivitiesthatimplementationpartnersaremeanttocarryoutPracticeprofilesmaybeahelpfultooltohelpdefinewhatisrequiredoftheactorswhoareresponsibleforimplementingtheintervention
Designimplementationandevaluation
45
ExampleInSomervillemayoralleadershipandengagementofmultiplecityagencies(publichealthprimarycareschoolsCambridgeHealthAllianceetc)streamlinedandconnectedthevariousShapeUpSomervilleinitiativesandgrantswhichhelpedtokeepthetalentandenergyfocusedonhealthyeatingandactivelivingforallsectorsofSomervilleSUSpartnersultimatelyincludedschoolsnonprofitsCityandstateagenciesandnationalfoundations
Yourturn
____________________________________________________________________________
____________________________________________________________________________
DescribetheinterventionWhatarethestepsactivitiesorprocessesRememberthatthemechanismthespecificstepsorprocessthatleadtobehaviorchangeiskeytounderstandingtheeffectivenessofthepolicychangeandsharingtheresultswithothers
ExampleGoalsofShapeUpSomervilleweretoinfluenceeverypartofanearlyelementaryschoolchildrsquosdayndashbeforeduringandafter-schooltopromotehealthyeatingphysicalactivityandtheirweightstatusInterventionincludedfoodserviceenhancementstoimprovebreakfastandlunchandstaffprofessionaldevelopmentwalktoschoolactivitiesSUSclassroomcurriculumandprofessionaldevelopmentschoolwellnesspolicydevelopmentSUSafter-schoolcurriculumandprofessionaldevelopmentoutreachandeducationtohomethroughmaterialsforumseventsldquoSUSapprovedrdquorestaurantscommunityoutreachandcapacitybuildingthroughpolicydevelopmenttrainingsmediaplacements
Yourturn
____________________________________________________________________________
____________________________________________________________________________
ClarifytimelineandmilestonesfordeliveryandevaluationHowfrequentlywilltheservicesinterventionbedeliveredHowlongwilltheprogramrunWhatisthetimetoexpectedimpacts(eg12monthsafterprogramcompletion)
Designimplementationandevaluation
46
ExampleShapeUpSomervilleledbyTuftsUniversityFriedmanSchoolofNutritionScienceandPolicyconductedaquasi-experimental(nonrandomized)interventionstudyamong1stthrough3rdgradestudentsinSomerville(intervention)andtwocomparisoncommunitiesfor9-monthsduringoneschoolyear(fall2002ndashspring2003)ThestudyshowedpositiveassociationoftheinterventiononchangeinBMISUSreleasedreportssummarizingevaluationsprogresstodateplansforadditionalprojectsandimpactstenandfifteenyearsintotheinitiative
Yourturn
____________________________________________________________________________
____________________________________________________________________________
PlanyouranalysisTheprevioussectionaddressedestablishinganaccountabilityframeworktomonitorprogressandimplementationDeterminewhichindicatorswillbediscussthroughthatprocesswhowillsharethedataandanalysisandhowfeedbackfromthoseconversationswillbeincorporatedbytheprojectteam
ExampleTheMayorrsquosofficeimplementedadata-drivendecisionmakingapproachcalledSomerStatwhichwassynergisticwiththeapproachestakenbytheresearchandevaluationteamsdeployedforthespecificgrants
Yourturn
____________________________________________________________________________
____________________________________________________________________________
PlanforsustainabilityIfthepolicyandprogramchangeissuccessfulhowwilltheprojectteamsustainandexpandtoadditionalsitesorpopulations
ExampleAlthoughmostofthepartnershipsforgedintheearlyyearsweremaintainednewgrantsandcollaborationsexpandedattentiontoolderpublicschoolstudentsactivetransportationandtheGreenLinesubwayextensionthebuiltenvironmentfarmerrsquosmarketeffortssuchasldquoGrowninSomervillerdquowithUnionMainStreeteconomicdevelopmentandhealthdisparitiesaswellasdatacollectionmonitoringandsurveillance
Yourturn
Designimplementationandevaluation
47
____________________________________________________________________________
____________________________________________________________________________
Pilotplanningtemplate
IfyouplantoconductapilotfirstdeterminethepurposeofthepilotDoyouwanttoobservewhetherthepolicychangeresultsinthedesiredbehaviorchangeAreyouconcernedaboutfeasibilityscalabilityorcostDoyouwanttotestwhetheryourdesiredresultscanbeachievedthroughthispolicyapproachClarifythegoalofthepilotandthequestionsyouhopetoanswerThiswillhelpyouidentifythekeyindicatorsthatyouwillusetomeasurewhetherthepilotissuccessfulBeforebeginningestablishabaselineofcurrentperformanceandsetyourtargetsaccordinglybasedonwhatyouhopetoaccomplishthroughthepilot
NextestablishthepilotplanforyourprojectteamincludingmilestonesforteamcheckinstodeterminewhethertomoveaheadorchangecourseThisisseparatefromthemeetingsthatwilloccuraspartoftheaccountabilityframeworkUsethespacebelowtocreateyourtimelineanddevelopyourmeetingschedule
Task Personresponsible Startdate Checkindate Deliverable
Planyourevaluation
WhileyouaredesigningtheimplementationitisalsotimetobegintodevelopyourplanforanalysisAsnotedbytheWilsonSheehanLabforEconomicOpportunitiesdecisionsaboutprogramsetup-includingoutreachscreeningintakedatatrackingandenrollment-candeterminewhetherornotarigorousimpactevaluationwillbepossibleAsdiscussedcityleadersmayfindithelpfultoincludeevaluationpartnersinearlyconversationsaboutprogramsetupDuringpreviousstagesyoumayhaveworkedwitharesearchertoplanyourproposedpolicychangeYoushouldbeclearaboutthepurposeoftheevaluationanddevelopanevaluationplanBelowareadditionalconsiderations
Causationvstrendanalysis
IntheEstablishaframeworktomonitorprogressstagetherewasguidanceontrackingtrendsovertimeItisimportanttonoteherethatmonitoringdataisdifferentfromanalyzingacausaleffectTrendanalysiswillillustratethatchangesareoccurringandmayleadtoquestionsthatcouldbeansweredbyyourevaluationbuttrendanalysisalonedoesnotprovideevidenceaboutwhatisdrivingthosechangesYoumayseechangesinyour
Designimplementationandevaluation
48
outcomeindicatorsbasedonvariablesoutsideoftheCityscontrolForinstanceachangeinfederaltradepolicycanhaveanimpactemploymentratesinyourCityoutsideofyourCitysnewjobtrainingandplacementprogram
ToaddresscausalityyoumustcomparetheoutcomesofindividualswhoreceivedtheinterventionwiththosewhodidnotWithoutacomparisongroupofsimilarindividualswhowerenotexposedtotheinterventionitisimpossibletoknowwhichfactorsareresponsiblefortheoutcomesRandomizedControlTrials(RCTs)inwhichindividualsarerandomlyassignedtotreatmentandcontrolgroupsarethegoldstandardinprovingcausalityIfyoucandemonstratethatoutcomesimprovedforthosewhoreceivedtheinterventionanddidnotimproveatallorimprovedlessforacomparisongroupyoucanbesomewhatcertainthattheinterventionwasthecause
Examplesofothernon-causalimpactevaluationsthatarerigorousandreliableincludecertainquasi-experimentalstudiesusingcomparisongroupdesignssmartinterruptedtimeseriesanalysesthatmodeltrendsovertimeinwaysthatallowestimationofinterventioneffectswelldesignedpre-posttestanalysesneedsassessmentsimplementationstudiesetcFormoreinformationondifferenttypesofevaluationsseetheSageResearchMethodswebsite
Lookfordifferencesingroups
BuildintimetoalsoexaminethepilotdatatoseeifgroupsareexperiencingdifferentresultsReviewingperformancebasedonoutcomemeasuresforapopulationinaggregatemayhidethesubtlewaysthatsomepeoplemaybeaffecteddifferentlysoitisnecessarytocarefullyanalyzethedatafordisparateoutcomesForinstancetheMovingtoOpportunity(MTO)studythatexaminedtheimpactofhousingvouchersonfamilieswholivedinhigh-povertyneighborhoodsfoundthatchildrenwhomovedbeforetheageof13hadbettereducationandhigherincomesasadultsHoweveradultsdidnotseeimprovededucationandemploymentoutcomes
Disaggregatingdataintomorespecificcategoriesofraceandethnicitycanhelpunderstandtrendsanddisparitiesinamoregranularwaywhichcanhelpdecisionmakerstargetpoliciesandinterventionsinamoreequitableway
EarlierintheRoadmapweworkedtoidentifystakeholdersandwhomightbepositivelyandnegativelyimpactedbyachangeinpolicyUsetheresultsofthatexercisetoinvolvethecommunityininterpretingdataanalysisandtogatherqualitativefeedbackfromthecommunitytocallattentiontoissuesForinstanceanincreaseinyouthdetentionmaybeexplainedbyinefficienciesorlackofaccesstotransportation
Designimplementationandevaluation
49
Designimplementationandevaluation
50
Shareyourstoryandsuccess
Sharetheresults
Storytelling
AnintegralstepinyourstrategyissharingtheresultsofyourpilotorpolicychangewithyourcommunityThisincludesdevelopingmessagingaroundtheworkofyourorganizationandcreatingaplantodisseminateyourstoriesWhytellstoriesaboutyourwork
Informandorunderstandasubjectorsituation
Giveneededcontextofdataorbitsofinformation
Relatablestoriesarememorableforreadersviewers
ReadersorviewerswillseetheinformationasmorecompellingwhenthereisademonstratedandrelatablechangeorimpactonthemselvesortheirfriendsneighborsandcolleaguesStoriesareaboutpeopleandgovernmentisfundamentallyabouthelpingpeopleSotellthestoriesofthepeopleyouandyourcitycolleagueshelped
Keepthesetipsinmindasyoudevelopyourstorywithyourteam
WhatwereyourfindingsandwhyshouldanyonecareTellyouraudiencehowtheproblemwassolvedWhathastheimpactbeenonresidentsandcitystafftodateWhatwasthesuccessandweretheremultiplesuccessesalongthewayIfyouhavepreliminaryresultsordatashowingimprovementtosharethisisthetimetodoit
WhowasinvolvedandimpactedReviewthelistofstakeholdersHowwereeachofthemimpactedbytheprogramHumanizeyourstoryandimpactbyaddingadditionalvoicesThesevoicescanbeoneofthefollowing
Apersonorfamilywhowaspositivelyaffectedbythechangesuccess
Aninvolvedthird-partydescribingboththeproblemandthesolution(ieanorganizationnonprofitprovidersbusinesstradeassociationsportsteametc)whoispartofthecommunitythatthegovernmentworkedwithonthechangesuccess
Someonefromthegovernmentitselfwhocanspeakaboutthechangesuccess
TellmoreaboutyourcityDescribeyourcityincludingpopulationdemographicsindustriesetcWhatareitschallengesWhatisitknownforWhatmakesituniqueHowisitsimilarordifferentfromothercitiesintheUS
Shareyourstoryandsuccess
51
DescribeyourcityrsquoschallengeWriteoutthechallengeyourcityisfacing(asdefinedinyourproblemstatement)HowdoesitaffectresidentsHowdoesthisproblemaffecttheworkofcitystaffmembers
AddthedetailsProvidedetailsonyourappliedresearchstrategyTalkaboutwhatyourcityhasdonenthepasttoresolvetheissueDiscusshowyourstrategyhaschangedwhenyoubegantoadoptthisapproachWasmoneysavedorinvested
Usethisstorymapexercisetoorganizeyourthoughtsandguaranteethatyouincludealloftheelementsofatraditionalstory
AfteryoudevelopyourmessagingcreateacommunicationsmarketingandoutreachstrategydesignedtospreadyourmessageYourstrategyshouldwalkyouthroughthefollowingkeyactivities
1 IdentifyandprofileyouraudienceWhencreatinganddisseminatingyourmessageitisalwaysgoodtothinkaboutyouraudienceWhoexactlywouldbeinterestedinwhatyouhavetosayThegeneralpopulationofthecityPeoplewhoworkinnonprofitsBusinessownersItmattersbecausenoteveryonewholivesinacitymaybethetargetoftheinformationBusinessownersmaybemoreinterestedindatashowingeffectsonthebusinessdistrictthanothersandsoonWewillcoverthislaterbutthisstepwillalsohelpyoutothinkofthemosteffectivewaytoreachyouraudience
2 Defineyourcalltoaction(ifyouhaveone)SpendsometimeconsideringwhyyouareinterestedintellingyourstoryDoyousimplywanttoupdatethepubliconyourprogressorcelebrateyoursuccessesAreyouinterestedinyouraudiencechangingitsbehaviororsupportingaspecificinitiativebeinglaunchedbyyourorganizationIfyouwanttohaveyouraudienceactorbehaveinacertainwaybespecificandincorporateitinyourmessaging
Shareyourstoryandsuccess
52
3 SelectperformancemeasuresThinkabouthowyouwoulddetermineifyourcampaignissuccessfulandselectappropriateperformancemeasuresYouwillrevisitthesemeasuresattheendofyourcampaigncommunicationsplanimplementationtodeterminewhetheryouneedtomodifyyourcommunicationsandoutreachstrategyforthefuture
4 MethodstocommunicateyourstoryTherearemanywaystocommunicatesuccessofyourprojectHerearesomesuggestionsonreachingaudiencesItisimportanttodeterminewhichmethodsarethemosteffectiveforyourdesiredaudiencetoreceiveinformationDoyourresearch(PewResearchCenterhasexcellentstudiesonthereachofmediadigitalmediaandmarketing)ForexampleifseniorsareyourtargetpopulationandtheyarelessinclinedtouseInstagram(thatistrue)youmaywanttoconsiderFacebookorthenightlylocalnewsbothofwhichareusedmoreoftenbyseniorsBelowisalistofsomemethodsandtoolsyoucanuse
i Pressreleasestolocalnewspapersandtvstations
ii ShortsmartphonevideosAgreatwaytoengageanaudienceisthroughvideoAskonequestionofsomeonewhoispartofthestoryMaybethereisasocialworkerwhocandiscusshungerinyourcityThesevideoscanbereusedonmultipleplatforms
iii Long-formwriting
iv TellingindividualstoriesthroughaseriesofphotosandshortnarrativesforpublicationonFacebookorInstagram
v Op-edsopinionpiecesplacedinthelocalpaper
vi Emailupdatestoyourlist
vii Connectingwithappropriateorganizationsthatareinvestedinthetopicandworkwiththecommunityandaskthemtodisseminateinformationtotheirmembers
viii EmployingldquoWordofMouthrdquo(WOM)tacticsbycreatingtownhallsorfreeeventsforresidentstoattendintheirownneighborhoods
5 DeterminethetypeofcollateralUsuallycampaignsthattargetresidentsrequireseveralavenuestoreachthemPeoplearenotalwaysinthesameplaceorspaceHavingakeenunderstandingofyouraudiencewillassistyouwithyourchoicesonwhatneedstobedevelopedSometimesthatmeansimagesforsocialmediaaonepageflierthatgivespeopleinformationonattendinganeventoranin-depthbrochure
Shareyourstoryandsuccess
53
6 ImplementyourstrategyKeepinmindstrategiescanbefluidSometimessomethingmaynotturnoutasexpectedevenifduringplanningsessionsitwasldquosupposedtordquoItisokYoucananalyzewhathappenedduringadebriefsessionandonceyoursurveysandothermeasurementshavebeenfinalized
Visualizations
Withtherightvisualizationyoucanreviewandshareinformationinsightsanalysisperformanceandtrendsallofwhichisvaluabletoacross-sectionofthepublicButwhichvisualizationisrightforyouTochoosetherighttypeofchartgraphdeterminethetypeofquestionyouwanttoanswerSomecommoncharttypesarebelowandmoreinformationcanbefoundhere
Exploreadditionalanalyticaltechniques
Duringthegoal-settingphaseyouestablishedoutputoutcomeandimplementationmeasuresBasedontheresultsofyourpilotyoumayneedtoadjustyourmeasuresWiththepilotdatainhandyoumayhavesomeideasofhowfutureanalysescouldbedifferentandpotentiallyimprovethequalityofyourprogramTakesometimetobrainstormotherrelevantdatasetsthatyoudidnothavethetimetopursueearlierincludingmessydepartmentaldatathatneedsmoreattentionorsourcesofpublicdatafromthestateandfederalgovernmentnonprofitsandevenprivatecompaniesthatcouldsupplementyourunderstandingofanissueUsingmachinelearningtechniquesyoumightbeabletobetter
Shareyourstoryandsuccess
54
predictdemandforhomelessservicesusingNOAAweatherforecastspreemptwatermainbreaksusingsoilstratigraphyorinvestineconomicdevelopmentbymonitoringrealestatelistingdata
Advancedanalyticaltechniquescanalsohelpyourimplementationrunmoreeffectivelybyintegratingdata-informedtargetinganddecisiontoolsintotheprocessByautomatingroutineworkflowsandsurfacingsubtletrendsthesetoolscanhelpyouiterateinamoreprecisemannerandprovidekeystrategicguidancetofront-lineemployeesthatwouldotherwisebeunavailableThesesortsofdata-science-enhancedprocessescanbeclassifiedasoneofsixprojecttypesasexploredfurtherbytheNewOrleansLANOLAlyticsteamFormoreguidanceonscopingasimilarprojecttheGovExadvancedanalyticsscopingguidegivesyouastep-by-stepoverviewofhowtobesuccessful
Usethespacebelowtolistyourideasforpossibleadvancedanalyticsprojectsbytype
Typeofproject Ideas
Findingtheneedleinthehaystack
Prioritizingforimpact
Earlywarningtools
Betterquickerdecisions
Optimizingresourceallocation
Experimentingforwhatworks
Keepthefeedbackloopgoing
GatherinformationonpublicsentimentaboutpolicychangethroughfocusgroupssurveysandpublicmeetingsledbyadvisorygroupmembersNotewhethernewinformationisavailablethatcanhelpanalyzeorcontextualizeyourprogress(egfederalorstatedatareports)Sharetheresultsofyourpilotandthepolicychangeprocesswithothercitiesandresearchers
Shareyourstoryandsuccess
55
WorkingwitharangeofevidencePresentlythereisalotofattentionandinvestmentinevidence-basedpolicymakingincludingthefederalbipartisanCommissiononEvidence-BasedPolicymakingthePewCharitableTrustsreportsandtoolkitsonHowStatesEngageinEvidence-BasedPolicymakingandtheLauraandJohnArnoldFoundationsupportedEvidence-BasedPolicymakingCollaborativeManyoftheseinitiativesfocusonrigorousevaluation--meaningrandomizedcontrolledtrials(RCT)orotherexperimentalmodels--asthebestkindofevidencetouseHoweverthisbodyofevidenceisofteninsufficientforthespecifickindsofquestionsthatgovernmentscanhavewhenmakingadecisionaboutpolicyInthisguideweproposeanapproachtosupportgovernmentsinincorporatingadditionaltypesofevidenceintotheirdecisionmaking
OnewaytounderstandarangeofevidenceistouseNestarsquosStandardsofEvidencetoframeanddefineevidenceTheBritishnonprofitNestaworkstoinjectpositivechangeintothewayUKgovernmentpoliciesandprogramsfunctionOneoftheirkeyareasofworkhasbeendevelopingWhatWorksCentresorganizationsthatcollectcommissionsynthesizeanddisseminateevidenceaboutinterventioneffectivenessacrossarangeofsocialimpactareasNestadevelopedthefollowingStandardsofEvidencetoprovideausefulwaytodocumenthowdifferentkindsofinformationdemonstratedifferentlevelsofevidence
Workingwitharangeofevidence
56
Theimportanceofpotentiallybiasedinformation
Onechallengepresentedbyanevidenceclassificationthatacknowledgesthevalueofnon-scientificformsofevidenceistofigureouthowtobalanceallofthepiecestogetherwhenyouusethemTounderstandhowtodothiswehavetothinkabouthowtopositionpotentiallybiasedinformationindecisionmaking
WhenwelookatinformationwhichhasnotbeenrigorouslyevaluatedweacceptthatthisinformationmightbebiasedThewholereasonweconductrigorousevaluationistoremovebiasfromourobservationsWeconductstructuredtrialstoproduceresultsthatareconsistentandgeneralizableIfafindingisunbiasedthisaccuracymdashthislackofbiasmdashcanbeprovedbecauseanyindependenttestercanreproduceitwhentheyfollowanidenticalprocess
YoucanlookatpotentiallybiasedinformationasjustinformationthatisearlieroninitspathtorefinementreplicationandverifiedaccuracyApieceofinformationbasedonanyoneindividualrsquosobservationsislikelytobebiasedbecauseitiscomingfromjustonesourceBiasisreducedbygatheringtogetherlotsofindependentsourcesofinformationandfiguringoutitsmostcommonattributes
MaybethebiasedinformationmakesaclaimthatistoolargemdashitsayssomethingistrueunderallconditionswhenitisreallyonlytrueundercertainconditionsorthataneffectofaninterventionisbiggerthanitstrueaverageorthataneffectlastsovertimewhenitdoesnotPerhapschangingtheclaimtobemorespecificmdashcriticallyreviewingtheinformationtoseeifspecificconditionsmustbemettoachievetheoutcomesmdashwouldmaketheclaimmoreaccurate
IfyoutestedyourclaimfounditdidnotholdupinsomecaseschangedittesteditagainuntilyourclaimrepeatedlyheldupthenyoucoulddemonstratethattheclaimwasnolongerbiasedbutaccurateThisprocessoftestingimprovingtestingandvalidatingiswhatwegetfromstructuredscientificstudy
Withindividualpiecesofinformation--crediblehypothesescasestudiesexpertopinion--youcannotassumethatkindofaccuracyAtthesametimetheseearlyobservationsarefoundationaltoanyreplicablescientificobservationsWewillnevergetanywhereifwedonotstartwiththesepiecesofinformationeveniftheymightbebiased
Combininglevelsofevidencetogetthemostdetailedinformationforyourcity
ItisespeciallyimportanttoacceptdifferentformsofevidencewhenyoufacequestionsthathavenotyetbeentestedbyindependentstudiesSomeofthemostimportantquestionsrelatetothingsthatareveryspecifictoyoursituationYoufaceaparticularcombinationof
Workingwitharangeofevidence
57
issuesactorstrendsandresourcesthatisultimatelydifferentfromeveryotherplaceintheworld
ThisisimportantbecauseaspolicypractitionersandnotscientistsyoucaremoreaboutyourspecificcasethanahypotheticalaveragegeneralcaseYoucarelessaboutthetotallikelyrangeofimpactsfromapolicythanhowthissingleimplementationwillworkYouwantinformationaboutpoliciesthatwillprovideyouwiththebestpossiblepredictionabouthowtheywillworkinyourcontext
ByputtingdifferentkindsofevidenceintothesameframeworkNestarsquosStandardsofEvidenceimplythatwhenyoucannotgetrigorouslytestedexperimentallyverifiedinformationitisappropriatetoworkwithwhatwedohaveThisisrelevantforthemunicipalimplementationcontextbecausenostudypulledfromaliteraturereviewwillaccountforallofthevariablesyoufaceinyourowncontext
Intheabsenceofhavingastudythatpredictshowaninterventionworksinyourspecificcontextyouwillneednon-scientificinformationtotrytofillinthosegapsYouwillneedtogatherstoriesfromresidentsstaffmembersandotherstakeholdersthatsitonLevels1or2ofthestandardsofevidence
Someofthekindsofinformationyouwillprobablygetmoreeasilyfrompeoplethanstudiesinclude
howpeopleincitieslikeyoursreactedtoaninterventioninpublicforumshowpeoplewhoimplementedaninterventioninacitylikeyoursfiguredouthowtogetitpassedhowthecitycommunicatedpubliclyabouttheapparentbenefitsanddrawbacksoftheinterventionthepoliticalacceptabilityoftheintervention
Youcanremainsimultaneouslyawarethattheinformationyouaregatheringthroughindividualstoriesisbiased--notgeneralizableinformedbyspecificperspectives--whilealsorecognizingthathavingsomeinformationaboutsomethingimportantisbetterthannoinformationEvidencefromthefirsttwoevidencelevelsareessentialbuildingblocksofknowledgeandperfectlyusefulwhenyoucannotgetdetailedunbiasedknowledge
Youhavetostartsomewhereandyoushouldalwaysstartwithevidenceyouhaveaboutyourveryspecificconditionsthatyouknowareimportantforsuccess
Workingwitharangeofevidence
58
Thankyou
Interviewswithcities
GovExheldsemi-structuredinterviewswithsixstaffmembersinfourcitiestoestablishtheinitialstepsoftheRoadmapWeaskedrespondentstotalkaboutaninitiativeorpolicythattheyhavebeenresponsibleforimplementingintheircityRespondentsfromthreecitiesspokeabouttheireffortstoaddresshomelessnessandrespondentsfromonecityspokeaboutblightAfterrespondentsidentifiedtheproblemtheyspokeabouthowthecityprioritizedthisissuethevariousstepsinvolvedandbarrierspresent
GovExconductedtextanalysisoftheseinterviewswhichrevealedthatpeoplewasthemostcommonlyusedtermduringtheseconversationswhendiscussingthestepstochangeaprogramorpolicyThisisathemethroughoutthisRoadmapThefirstsectionbeginswithidentifyingtherelevantpeopleTheconversationsaroundevidencearefocusedonhumanbehaviorratherthanmethodologyorstrengthofevidence
Thankyou
59
OncewehadadraftoftheRoadmapwefollowedupwithcitiesonanindividualbasistolearnmoreaboutwhatwouldbehelpfulWearegratefultothefollowingcitystafffortheinterviewsandindividualconversations
MichaelBaskinChiefPolicyOfficerChattanoogaTN
MelissaBridgesPerformanceandInnovationCoordinatorLittleRockAR
VictoriaCarreoacutenAdministrativeOfficerLasVegasNV
TyroneGrandstrandCommunityDevelopmentPlannerFargoND
TibLaughlinDirectorofGeneralServicesKansasCityKS
TimMorelandDirectorofPerformanceManagementandOpenDataChattanoogaTN
CharlesRosendahlManagementAnalystDentonTX
Thankyou
60
MelissaSiebenAssistantCountyAdministratorKansasCityKS
GinaThompsonDirectorofStrategicServicesLewisvilleTX
JHUFacultycollaboration
GovExalsoconvenedthecitycontactsandfacultypartnersinaseriesofAdvisoryGroupmeetingsThesemeetingsservedtoconnectcitiesandfacultytounderstandoneanothersperspective
StefanBaralBloombergSchoolofPublicHealth
AshleyBernerJHUSchoolofEducation
MelissaDavey-RothwellBloombergSchoolofPublicHealth
JonLinksBloombergSchoolofPublicHealth
TamarMendelsonBloombergSchoolofPublicHealth
KristinMmariBloombergSchoolofPublicHealth
KeshiaPollackPorterBloombergSchoolofPublicHealth
JoshSharfsteinBloombergSchoolofPublicHealth
LizStuartBloombergSchoolofPublicHealth
LindsayThompsonCareyBusinessSchool
DanielWebsterBloombergSchoolofPublicHealth
Partnersindataandevidence-basedpolicy
Wealsospokewithpartnerswhoareexpertsinthefieldofevidence-basedpolicywhosharedtheirknowledgeandresourcesofcurrentandfuturethinkinginevidence-informedthinking
WendyBarrenoAssociateDirectorWilsonSheehanLabforEconomicOpportunitiesUniversityofNotreDame
TiffanyDavenportDirectorofDataPartnershipsBaltimoresPromise
TomiHiersDirectorBaltimoreCivicSiteAnnieECaseyFoundation
ChrisKingsleySeniorAssociateAnnieECaseyFoundation
KathrynOliverAssociateProfessorLondonSchoolofHygieneandTropicalMedicine
Thankyou
61
KristenWatkinsSeniorPolicyAssociateJ-PAL
GovExParticipants
GovExAnalystMiriamMcKinneyanalyzedtheinterviewtranscriptsandconductedtextanalysisSeniorImplementationAdvisorEmilyShawwillintegrateRoadmapactivitieswithadditionalservicedeliveryactivitiesatGovEx
Thankyou
62
ReferencesAbrahamKGampHaskinsR(2017)Thepromiseofevidence-basedpolicymakingReportofthecommissiononevidence-basedpolicymakingWashingtonDCCommissiononEvidence-basedPolicymakingBigmove2Extendingthegreen(nd)Retrievedfromhttpswwwcityoflewisvillecomdoing-businesslewisville-2025big-move-2-extending-the-greenBatesMAampGlennersterR(2017)ThegeneralizabilitypuzzleRetrievedfromhttpsssirorgarticlesentrythe_generalizability_puzzleBuildwithnotfor(2018January)RetrievedfromhttpsbuildwithnotforwordpresscomCairneyPampOliverK(2017)Evidence-basedpolicymakingisnotlikeevidence-basedmedicinesohowfarshouldyougotobridgethedividebetweenevidenceandpolicyHealthResearchPolicyandSystemsdoi101186s12961-017-0192-xCairneyPOliverKampWellsteadA(2016)TobridgethedividebetweenevidenceandpolicyReduceambiguityasmuchasuncertaintyPublicAdministrationReview76(3)399-402Retrievedfromhttpdxdoiorg101111puar12555Centreforpublicimpact(nd)RetrievedfromhttpswwwcentreforpublicimpactorgCernaL(2013)ThenatureofpolicychangeandimplementationAreviewofdifferenttheoreticalapproachesOrganisationforEconomicCo-OperationandDevelopmentRetrievedfromhttpswwwoecdorgeduceriTheNatureofPolicyChangeandImplementationpdfColoradoSprings(nd)Retrievedfromhttpscoloradospringsgovcommunicationspagewhat-expect-when-you-speakupContinuumofcare(coc)program(nd)RetrievedfromhttpswwwhudexchangeinfoprogramscocCouncilpresidentkicksoff$12millionyouthfund(nd)Retrievedfromhttpwwwbaltimorecitycouncilcomcontentcouncil-president-kicks-12-million-youth-fundDurlakJAampDuPreEP(2008)ImplementationmattersAreviewofresearchontheinfluenceofimplementationonprogramoutcomesandthefactorsaffectingimplementationAmericanJournalofCommunityPsychology41327-350Retrievedfromhttpswwwncbinlmnihgovpubmed18322790Fishbonediagram(ishikawa)(nd)Retrievedfromhttpasqorglearn-about-qualitycause-analysis-toolsoverviewfishbonehtmlFivewhysGettingtotherootofaproblemquickly(nd)RetrievedfromhttpswwwmindtoolscompagesarticlenewTMC_5Whtm
References
63
FligsteinNampMcAdamD(2011)TowardageneraltheoryofstrategicactionfieldsSociologicalTheory29(1)1-26RetrievedfromhttpsociologyberkeleyedusitesdefaultfilesfacultyfligsteinFligstein20McAdam20Soc20Theory20PaperpdHeinkelP(2018January12)CitymullspermitsforbikesharingcompaniesRetrievedfromhttpwwwdentonrccomnewsdenton20180112city-mulls-permits-bike-sharing-companiesHowisareamedianincomecalculated(2012October)Retrievedfromhttpswwwhudexchangeinfofaqs843how-is-area-median-income-calculatedLampkinLMampHatryHP(2003)KeystepsinoutcomemanagementUrbanInstituteRetrievedfromhttpswwwurbanorgresearchpublicationkey-steps-outcome-managementMarshDampSmithM(2000)UnderstandingpolicynetworksTowardsadialecticalapproachPoliticalStudies484-21Retrievedfromhttpjournalssagepubcomdoiabs1011111467-924800247McCannL(nd)RetrievedfromhttplaurenellencomMoultonSampSandfortJR(2016)ThestrategicactionfieldframeworkforpolicyimplementationresearchPolicyStudiesJournal45(1)Retrievedfromhttponlinelibrarywileycomstore101111psj12147assetpsj12147pdfv=1ampt=jdkhhcp4amps=e184c625e1855a6b1615ccb99ee27b1c0b3212d8PaynterB(2018January1)What100citiesarelearningfromeachotherbysharingtheirdataRetrievedfromhttpswwwfastcompanycom40518004what-100-cities-are-learning-from-each-other-by-sharing-their-dataPeakacademy(nd)Retrievedfromhttpswwwdenvergovorgcontentdenvergovenmayors-officeprograms-initiativespeak-performancepeak-academyhtmlPinnockHBarwickMCarpenterCREldridgeSGrandesGGriffithsCJRycroft-MaloneJMeissnerPMurrayEPatelASheikhAampTaylorSJC(2017)Standardsforreportingimplementationstudies(stari)ExplanationandelaborationdocumentBMJOpenRetrievedfromhttpbmjopenbmjcomcontentbmjopen74e013318fullpdfProctorESilmereHRaghavanRHovmandPAaronsGBungerAGriffeyRampHensleyM(2011)OutcomesforimplementationresearchConceptualdistinctionsmeasurementchallengesandresearchagendaAdministrationandPolicyinMentalHealth38(2)65-76doi101007s10488-010-0319-7TandonSD(2018)ProgressincommunityhealthpartnershipsResearcheducationandactionRetrievedfromhttpswwwpressjhuedujournalsprogress-community-health-partnerships-research-education-and-actionProvanKGampKenisP(2007)ModesofnetworkgovernanceStructuremanagementandeffectivenessJournalofPublicAdministrationResearchandTheory
References
64
18(2)229-252Retrievedfromhttpsacademicoupcomjpartarticle182229935895SabatierPA(1988)Anadvocacycoalitionframeworkofpolicychangeandtheroleofpolicy-orientedlearningthereinPolicySciences21(3)129-168Retrievedfromhttpwwwjstororgstable4532139SandstromAampCarlssonL(2008)TheperformanceofpolicynetworksTherelationbetweennetworkstructureandnetworkperformancePolicyStudiesJournal36497-524Retrievedfromhttponlinelibrarywileycomdoi101111j1541-0072200800281xabstractSchwartzMampWilsonE(nd)WhocanaffordtoliveinahomeAlookatdatafromthe2006americancommunitysurveyRetrievedfromhttpswwwcensusgovhousingcensuspublicationswho-can-affordpdfShearerJCAbelsonAKouyateBLavisJNampWaltG(2016)WhydopolicieschangeInstitutionsinterestsideasandnetworksinthreecasesofpolicyreformHealthPolicyandPlanning31(9)1200-1211Retrievedfromhttpsdoiorg101093heapolczw052ThelessonsofadministrativedataHigh-profilepolicy-relevantresearchpoweredbyadministrativedata(nd)Retrievedfromhttpswwwpovertyactionlaborgsitesdefaultfilesresources20161114-The-Lessons-of-Administrative-DatapdfToolsandotherneedsforqualitativeandquantitativemethods(2015February)AgencyforHealthcareResearchandQualityRetrievedfromhttpwwwahrqgovprofessionalsquality-patient-safetytalkingqualityassesstoolshtml
WeaverRCampBagchi-SenS(2013)SpatialanalysisofurbandeclineThegeographyofblightAppliedGeography4061-70
References
65
UniversityPartners
Name Title Affiliation Area(s)ofexpertise
Contactinformation
Examplesofworkingwithnon-academicdecisionmakers
WendyPBarreno
AssociateDirector
WilsonSheehanLabforEconomicOpportunitiesUniversityofNotreDame
LEOsfocusareasincludehealtheducationhousingchildwelfareeconomicopportunitycriminaljustice
wbarrenonduduorleondedu
MelissaDavey-RothwellPhD
AssociateScientist
JohnsHopkinsBloombergSchoolofPublicHealth
Health mdavey1jhuedu410-614-5854
BaltimoreCityHealthDepartmentLeadersMarylandDepartmentofHealthAddictionRecoveryServicesinDCExecutiveDirectorsofcommunityorganizationsacrosstheUSthatfocusonHIVSTIsubstanceabuseandimplementationscience
ElizabethStuart
AssociateDeanforEducationandProfessorDepartmentsofMentalHealthBiostatisticsandHealthPolicyand
JohnsHopkinsBloombergSchoolofPublicHealth
Educationsocialpolicystatisticalmethods
estuartjhuedu
Resources
66
Management
Design-ThinkingResources
EngagementGuideforCivicTech
Journeymap
PerformanceManagementResources
DenverPeakAcademy
Proxymeasuresguide
Talkingaboutperformancewithoutcomplicatingit(video)
SettingPerformanceTargetsguide
Datavisualizationguide
Performancemanagementgettingstartedguide
Exercises
Fishboneexercise
5Whys
AdministrativeData
TheLessonsofAdministrativeData(JPAL)
EvaluationResources
Quantitativeandqualitativemethods(USDepartmentofHealthandHumanServicesAgencyforHealthcareResearchandQuality)
KeyStepsinOutcomeManagement(UrbanInstitute)
RCT-YES
Sageresearchmethods
BestPracticesforMixedMethodsResearchintheHealthSciences
ACCORDChecklist
Pilotandimplementationplanning
Benchmarkingexamplesandproxymeasures
Seattleopendataplaybook
Resources
67
Datasciencescopingguide
Resources
68
AboutGovExGovExhelpsgovernmentsusedataandevidencetoimproveresidentsrsquolivesWesupportgovernmentsinbuildingtheirinternalcapacitytooperatesustainableanalyticspracticeswhichimprovecommunityoutcomesinfieldsfrompublicsafetytoeconomicdevelopmentGovExhasaproventrackrecordofworkinginconcertwithpartnerstoprovidetechnicalassistanceandtrainingatscale
SinceourlaunchinApril2015GovExhas
Trainedover2000governmentemployeesinpersonandonlineintopicsincludingdatamanagementperformancemanagementopendatacommunityengagementandanalytics
Providedtechnicalassistancetomorethan100localgovernmentsacrosstheglobeimprovingtheiruseofdatatoachieveefficienciesinservicedeliveryandbetteroutcomesforresidents
Assessedthecapacityofmorethan130governmentalorganizationstodeterminehowleadersstaffandexternalstakeholdersmanageandusedata
Localgovernmentsareincreasinglyonthefrontlinesofaddressinglargescaleproblemswithglobalimplicationssuchasclimatechangepublicsafetyandeconomicinequality
Whileevidence-basedinterventionstohelpsolvethoseproblemsareavailablelocalgovernmentstaffhavelimitedtimeandcapacitytofindandimplementresearch-basedsolutionsAsaresultcitiescontinuetoworkthewaytheyalwayshave--makingincrementalimprovementstoprogramsbutnotalwayscreatingchangeatasystem-level
ThroughresourcedevelopmentandcollaborationwithJHUfacultyGovExissupportingcityleadersinusingdataandevidencetomakedecisionsandcreatinglastingrelationshipswithresearchers
TheRoadmapwasdevelopedbyGovExaspartofitspartnershipwithBloombergPhilanthropiesWhatWorksCitiesinitiative
AboutGovEx
69
- Title page
- Introduction and purpose of the Roadmap
- Using the Roadmap
-
- Gather information about the problem
- Strategically engage key partners
- Review the evidence and collect details
- Decide on a policy to implement based on local factors
- Establish a framework to monitor progress
- Assess data accessibility quality and security
- Design implementation and evaluation
- Share your story and success
-
- Working with a range of evidence
- Thank you
- References
- Resources
- About GovEx
-
Assessdataaccessibilityqualityandsecurity
43
Designimplementationandevaluation
Designtheimplementation
NowthatyouhaveyouraccountabilityplanandkeyindicatorsitistimetogetspecificaboutyourplanforimplementationThetemplatebelowwillhelpyouthinkthroughthespecificmechanismsofimplementingpolicychange
UnderstandingorganizationsandsystemsiscriticaltounderstandingandmakingdecisionsaboutwhereandhowinterventionsshouldbeappliedApolicyimplementedinasinglesettingmayshowpromisingresultsbutimplementingmultiplepoliciesandprogramsatthesametimewithdifferentactorsresponsibleforeachaspectcanhelpensuresustainablebehaviorchange
InSomervilleMAShapeUpSomerville(SUS)isacitywidepublichealthprogramthatisimplementedacrossmultiplelevelsofinfluenceandsystemsthroughinterventionsbeforeduringandafterschooltoincludechangestobuiltenvironmentsuchasbikelanesregionalmasstransportationandphysicaleducationequipmentinschoolsandgymstoincreasephysicalactivitywellnesspoliciesandschoolgardensnutritioneducationduringandafterschoolandhealthyrestaurantprogramstosupporthealthyfoodandhealthyeatinghabits
Usethetemplatebelowtoplanyourimplementation
Policyimplementationplanningtemplate
ListthespecificdesiredoutcomesInthepreviousstepyouworkedwithdepartmentstafftoestablishspecificmeasurableoutcomesandinterimindicatorsPrioritizetheindicatorsthatwilldemonstrateprogressUsethespacebelowtocapturethedesiredoutcomesofthepolicychange
ExampleShapeupSomervillebeganwiththegoalofpreventingobesityin1stthrough3rdgraders
Yourturn
____________________________________________________________________________
____________________________________________________________________________
Designimplementationandevaluation
44
SpecifythetargetpopulationAreyoudesigninganinterventionthattargetsonlycertainsitesAreyouplanningonimplementingaprogramorinitiativeforaspecifictargetaudienceClearlyspecifythepopulationyouplantotarget
IfyouplantoconductarigorousimpactevaluationasnotedbytheWilsonSheehanLabforEconomicOpportunitiesitisimportanttoestablishconcreteobjectiveeligibilitycriteriaandensurethatthesecriteriaareappliedconsistentlybyallstaffduringscreeningandintakeItisalsovitaltoidentifytheindividualswhoareeligibleforaparticularprogramorserviceevenifeveryoneeligiblecannotbeservedbyexistingresourcesThishelpscitiesidentifyanyunmetneedsandguidefutureresourceallocationwhilealsocreatingthepossibilityofarigorousimpactevaluationbyidentifyingapotentialcomparisongroupFinallywhenconductingscreeningandintakeitisimperativetocollectandmaintainbasicinformation(egnamedateofbirthgender)aboutallprogram-eligibleindividualsnotjustthosewhoparticipateintheprogram
ExampleIntheShapeUpSomervilleexampletheprogrambeganbytargeting1stthrough3rdgraders
Yourturn
____________________________________________________________________________
____________________________________________________________________________
DefinethelocationWherewillthepolicychangetakeplaceWillimplementationtakeplaceatonesiteormultiplesites
ExampleShapeUpSomervilleinterventionsweredeliveredinschools
Yourturn
____________________________________________________________________________
____________________________________________________________________________
ClarifyimplementationpartnersandarticulatetheirrolesWhatarethekeyorganizationsinstitutionsorpeopleresponsibleforimplementationTheprojectteamshouldbeasclearandspecificaspossibleinprovidingadescriptionoftheconcreteactivitiesthatimplementationpartnersaremeanttocarryoutPracticeprofilesmaybeahelpfultooltohelpdefinewhatisrequiredoftheactorswhoareresponsibleforimplementingtheintervention
Designimplementationandevaluation
45
ExampleInSomervillemayoralleadershipandengagementofmultiplecityagencies(publichealthprimarycareschoolsCambridgeHealthAllianceetc)streamlinedandconnectedthevariousShapeUpSomervilleinitiativesandgrantswhichhelpedtokeepthetalentandenergyfocusedonhealthyeatingandactivelivingforallsectorsofSomervilleSUSpartnersultimatelyincludedschoolsnonprofitsCityandstateagenciesandnationalfoundations
Yourturn
____________________________________________________________________________
____________________________________________________________________________
DescribetheinterventionWhatarethestepsactivitiesorprocessesRememberthatthemechanismthespecificstepsorprocessthatleadtobehaviorchangeiskeytounderstandingtheeffectivenessofthepolicychangeandsharingtheresultswithothers
ExampleGoalsofShapeUpSomervilleweretoinfluenceeverypartofanearlyelementaryschoolchildrsquosdayndashbeforeduringandafter-schooltopromotehealthyeatingphysicalactivityandtheirweightstatusInterventionincludedfoodserviceenhancementstoimprovebreakfastandlunchandstaffprofessionaldevelopmentwalktoschoolactivitiesSUSclassroomcurriculumandprofessionaldevelopmentschoolwellnesspolicydevelopmentSUSafter-schoolcurriculumandprofessionaldevelopmentoutreachandeducationtohomethroughmaterialsforumseventsldquoSUSapprovedrdquorestaurantscommunityoutreachandcapacitybuildingthroughpolicydevelopmenttrainingsmediaplacements
Yourturn
____________________________________________________________________________
____________________________________________________________________________
ClarifytimelineandmilestonesfordeliveryandevaluationHowfrequentlywilltheservicesinterventionbedeliveredHowlongwilltheprogramrunWhatisthetimetoexpectedimpacts(eg12monthsafterprogramcompletion)
Designimplementationandevaluation
46
ExampleShapeUpSomervilleledbyTuftsUniversityFriedmanSchoolofNutritionScienceandPolicyconductedaquasi-experimental(nonrandomized)interventionstudyamong1stthrough3rdgradestudentsinSomerville(intervention)andtwocomparisoncommunitiesfor9-monthsduringoneschoolyear(fall2002ndashspring2003)ThestudyshowedpositiveassociationoftheinterventiononchangeinBMISUSreleasedreportssummarizingevaluationsprogresstodateplansforadditionalprojectsandimpactstenandfifteenyearsintotheinitiative
Yourturn
____________________________________________________________________________
____________________________________________________________________________
PlanyouranalysisTheprevioussectionaddressedestablishinganaccountabilityframeworktomonitorprogressandimplementationDeterminewhichindicatorswillbediscussthroughthatprocesswhowillsharethedataandanalysisandhowfeedbackfromthoseconversationswillbeincorporatedbytheprojectteam
ExampleTheMayorrsquosofficeimplementedadata-drivendecisionmakingapproachcalledSomerStatwhichwassynergisticwiththeapproachestakenbytheresearchandevaluationteamsdeployedforthespecificgrants
Yourturn
____________________________________________________________________________
____________________________________________________________________________
PlanforsustainabilityIfthepolicyandprogramchangeissuccessfulhowwilltheprojectteamsustainandexpandtoadditionalsitesorpopulations
ExampleAlthoughmostofthepartnershipsforgedintheearlyyearsweremaintainednewgrantsandcollaborationsexpandedattentiontoolderpublicschoolstudentsactivetransportationandtheGreenLinesubwayextensionthebuiltenvironmentfarmerrsquosmarketeffortssuchasldquoGrowninSomervillerdquowithUnionMainStreeteconomicdevelopmentandhealthdisparitiesaswellasdatacollectionmonitoringandsurveillance
Yourturn
Designimplementationandevaluation
47
____________________________________________________________________________
____________________________________________________________________________
Pilotplanningtemplate
IfyouplantoconductapilotfirstdeterminethepurposeofthepilotDoyouwanttoobservewhetherthepolicychangeresultsinthedesiredbehaviorchangeAreyouconcernedaboutfeasibilityscalabilityorcostDoyouwanttotestwhetheryourdesiredresultscanbeachievedthroughthispolicyapproachClarifythegoalofthepilotandthequestionsyouhopetoanswerThiswillhelpyouidentifythekeyindicatorsthatyouwillusetomeasurewhetherthepilotissuccessfulBeforebeginningestablishabaselineofcurrentperformanceandsetyourtargetsaccordinglybasedonwhatyouhopetoaccomplishthroughthepilot
NextestablishthepilotplanforyourprojectteamincludingmilestonesforteamcheckinstodeterminewhethertomoveaheadorchangecourseThisisseparatefromthemeetingsthatwilloccuraspartoftheaccountabilityframeworkUsethespacebelowtocreateyourtimelineanddevelopyourmeetingschedule
Task Personresponsible Startdate Checkindate Deliverable
Planyourevaluation
WhileyouaredesigningtheimplementationitisalsotimetobegintodevelopyourplanforanalysisAsnotedbytheWilsonSheehanLabforEconomicOpportunitiesdecisionsaboutprogramsetup-includingoutreachscreeningintakedatatrackingandenrollment-candeterminewhetherornotarigorousimpactevaluationwillbepossibleAsdiscussedcityleadersmayfindithelpfultoincludeevaluationpartnersinearlyconversationsaboutprogramsetupDuringpreviousstagesyoumayhaveworkedwitharesearchertoplanyourproposedpolicychangeYoushouldbeclearaboutthepurposeoftheevaluationanddevelopanevaluationplanBelowareadditionalconsiderations
Causationvstrendanalysis
IntheEstablishaframeworktomonitorprogressstagetherewasguidanceontrackingtrendsovertimeItisimportanttonoteherethatmonitoringdataisdifferentfromanalyzingacausaleffectTrendanalysiswillillustratethatchangesareoccurringandmayleadtoquestionsthatcouldbeansweredbyyourevaluationbuttrendanalysisalonedoesnotprovideevidenceaboutwhatisdrivingthosechangesYoumayseechangesinyour
Designimplementationandevaluation
48
outcomeindicatorsbasedonvariablesoutsideoftheCityscontrolForinstanceachangeinfederaltradepolicycanhaveanimpactemploymentratesinyourCityoutsideofyourCitysnewjobtrainingandplacementprogram
ToaddresscausalityyoumustcomparetheoutcomesofindividualswhoreceivedtheinterventionwiththosewhodidnotWithoutacomparisongroupofsimilarindividualswhowerenotexposedtotheinterventionitisimpossibletoknowwhichfactorsareresponsiblefortheoutcomesRandomizedControlTrials(RCTs)inwhichindividualsarerandomlyassignedtotreatmentandcontrolgroupsarethegoldstandardinprovingcausalityIfyoucandemonstratethatoutcomesimprovedforthosewhoreceivedtheinterventionanddidnotimproveatallorimprovedlessforacomparisongroupyoucanbesomewhatcertainthattheinterventionwasthecause
Examplesofothernon-causalimpactevaluationsthatarerigorousandreliableincludecertainquasi-experimentalstudiesusingcomparisongroupdesignssmartinterruptedtimeseriesanalysesthatmodeltrendsovertimeinwaysthatallowestimationofinterventioneffectswelldesignedpre-posttestanalysesneedsassessmentsimplementationstudiesetcFormoreinformationondifferenttypesofevaluationsseetheSageResearchMethodswebsite
Lookfordifferencesingroups
BuildintimetoalsoexaminethepilotdatatoseeifgroupsareexperiencingdifferentresultsReviewingperformancebasedonoutcomemeasuresforapopulationinaggregatemayhidethesubtlewaysthatsomepeoplemaybeaffecteddifferentlysoitisnecessarytocarefullyanalyzethedatafordisparateoutcomesForinstancetheMovingtoOpportunity(MTO)studythatexaminedtheimpactofhousingvouchersonfamilieswholivedinhigh-povertyneighborhoodsfoundthatchildrenwhomovedbeforetheageof13hadbettereducationandhigherincomesasadultsHoweveradultsdidnotseeimprovededucationandemploymentoutcomes
Disaggregatingdataintomorespecificcategoriesofraceandethnicitycanhelpunderstandtrendsanddisparitiesinamoregranularwaywhichcanhelpdecisionmakerstargetpoliciesandinterventionsinamoreequitableway
EarlierintheRoadmapweworkedtoidentifystakeholdersandwhomightbepositivelyandnegativelyimpactedbyachangeinpolicyUsetheresultsofthatexercisetoinvolvethecommunityininterpretingdataanalysisandtogatherqualitativefeedbackfromthecommunitytocallattentiontoissuesForinstanceanincreaseinyouthdetentionmaybeexplainedbyinefficienciesorlackofaccesstotransportation
Designimplementationandevaluation
49
Designimplementationandevaluation
50
Shareyourstoryandsuccess
Sharetheresults
Storytelling
AnintegralstepinyourstrategyissharingtheresultsofyourpilotorpolicychangewithyourcommunityThisincludesdevelopingmessagingaroundtheworkofyourorganizationandcreatingaplantodisseminateyourstoriesWhytellstoriesaboutyourwork
Informandorunderstandasubjectorsituation
Giveneededcontextofdataorbitsofinformation
Relatablestoriesarememorableforreadersviewers
ReadersorviewerswillseetheinformationasmorecompellingwhenthereisademonstratedandrelatablechangeorimpactonthemselvesortheirfriendsneighborsandcolleaguesStoriesareaboutpeopleandgovernmentisfundamentallyabouthelpingpeopleSotellthestoriesofthepeopleyouandyourcitycolleagueshelped
Keepthesetipsinmindasyoudevelopyourstorywithyourteam
WhatwereyourfindingsandwhyshouldanyonecareTellyouraudiencehowtheproblemwassolvedWhathastheimpactbeenonresidentsandcitystafftodateWhatwasthesuccessandweretheremultiplesuccessesalongthewayIfyouhavepreliminaryresultsordatashowingimprovementtosharethisisthetimetodoit
WhowasinvolvedandimpactedReviewthelistofstakeholdersHowwereeachofthemimpactedbytheprogramHumanizeyourstoryandimpactbyaddingadditionalvoicesThesevoicescanbeoneofthefollowing
Apersonorfamilywhowaspositivelyaffectedbythechangesuccess
Aninvolvedthird-partydescribingboththeproblemandthesolution(ieanorganizationnonprofitprovidersbusinesstradeassociationsportsteametc)whoispartofthecommunitythatthegovernmentworkedwithonthechangesuccess
Someonefromthegovernmentitselfwhocanspeakaboutthechangesuccess
TellmoreaboutyourcityDescribeyourcityincludingpopulationdemographicsindustriesetcWhatareitschallengesWhatisitknownforWhatmakesituniqueHowisitsimilarordifferentfromothercitiesintheUS
Shareyourstoryandsuccess
51
DescribeyourcityrsquoschallengeWriteoutthechallengeyourcityisfacing(asdefinedinyourproblemstatement)HowdoesitaffectresidentsHowdoesthisproblemaffecttheworkofcitystaffmembers
AddthedetailsProvidedetailsonyourappliedresearchstrategyTalkaboutwhatyourcityhasdonenthepasttoresolvetheissueDiscusshowyourstrategyhaschangedwhenyoubegantoadoptthisapproachWasmoneysavedorinvested
Usethisstorymapexercisetoorganizeyourthoughtsandguaranteethatyouincludealloftheelementsofatraditionalstory
AfteryoudevelopyourmessagingcreateacommunicationsmarketingandoutreachstrategydesignedtospreadyourmessageYourstrategyshouldwalkyouthroughthefollowingkeyactivities
1 IdentifyandprofileyouraudienceWhencreatinganddisseminatingyourmessageitisalwaysgoodtothinkaboutyouraudienceWhoexactlywouldbeinterestedinwhatyouhavetosayThegeneralpopulationofthecityPeoplewhoworkinnonprofitsBusinessownersItmattersbecausenoteveryonewholivesinacitymaybethetargetoftheinformationBusinessownersmaybemoreinterestedindatashowingeffectsonthebusinessdistrictthanothersandsoonWewillcoverthislaterbutthisstepwillalsohelpyoutothinkofthemosteffectivewaytoreachyouraudience
2 Defineyourcalltoaction(ifyouhaveone)SpendsometimeconsideringwhyyouareinterestedintellingyourstoryDoyousimplywanttoupdatethepubliconyourprogressorcelebrateyoursuccessesAreyouinterestedinyouraudiencechangingitsbehaviororsupportingaspecificinitiativebeinglaunchedbyyourorganizationIfyouwanttohaveyouraudienceactorbehaveinacertainwaybespecificandincorporateitinyourmessaging
Shareyourstoryandsuccess
52
3 SelectperformancemeasuresThinkabouthowyouwoulddetermineifyourcampaignissuccessfulandselectappropriateperformancemeasuresYouwillrevisitthesemeasuresattheendofyourcampaigncommunicationsplanimplementationtodeterminewhetheryouneedtomodifyyourcommunicationsandoutreachstrategyforthefuture
4 MethodstocommunicateyourstoryTherearemanywaystocommunicatesuccessofyourprojectHerearesomesuggestionsonreachingaudiencesItisimportanttodeterminewhichmethodsarethemosteffectiveforyourdesiredaudiencetoreceiveinformationDoyourresearch(PewResearchCenterhasexcellentstudiesonthereachofmediadigitalmediaandmarketing)ForexampleifseniorsareyourtargetpopulationandtheyarelessinclinedtouseInstagram(thatistrue)youmaywanttoconsiderFacebookorthenightlylocalnewsbothofwhichareusedmoreoftenbyseniorsBelowisalistofsomemethodsandtoolsyoucanuse
i Pressreleasestolocalnewspapersandtvstations
ii ShortsmartphonevideosAgreatwaytoengageanaudienceisthroughvideoAskonequestionofsomeonewhoispartofthestoryMaybethereisasocialworkerwhocandiscusshungerinyourcityThesevideoscanbereusedonmultipleplatforms
iii Long-formwriting
iv TellingindividualstoriesthroughaseriesofphotosandshortnarrativesforpublicationonFacebookorInstagram
v Op-edsopinionpiecesplacedinthelocalpaper
vi Emailupdatestoyourlist
vii Connectingwithappropriateorganizationsthatareinvestedinthetopicandworkwiththecommunityandaskthemtodisseminateinformationtotheirmembers
viii EmployingldquoWordofMouthrdquo(WOM)tacticsbycreatingtownhallsorfreeeventsforresidentstoattendintheirownneighborhoods
5 DeterminethetypeofcollateralUsuallycampaignsthattargetresidentsrequireseveralavenuestoreachthemPeoplearenotalwaysinthesameplaceorspaceHavingakeenunderstandingofyouraudiencewillassistyouwithyourchoicesonwhatneedstobedevelopedSometimesthatmeansimagesforsocialmediaaonepageflierthatgivespeopleinformationonattendinganeventoranin-depthbrochure
Shareyourstoryandsuccess
53
6 ImplementyourstrategyKeepinmindstrategiescanbefluidSometimessomethingmaynotturnoutasexpectedevenifduringplanningsessionsitwasldquosupposedtordquoItisokYoucananalyzewhathappenedduringadebriefsessionandonceyoursurveysandothermeasurementshavebeenfinalized
Visualizations
Withtherightvisualizationyoucanreviewandshareinformationinsightsanalysisperformanceandtrendsallofwhichisvaluabletoacross-sectionofthepublicButwhichvisualizationisrightforyouTochoosetherighttypeofchartgraphdeterminethetypeofquestionyouwanttoanswerSomecommoncharttypesarebelowandmoreinformationcanbefoundhere
Exploreadditionalanalyticaltechniques
Duringthegoal-settingphaseyouestablishedoutputoutcomeandimplementationmeasuresBasedontheresultsofyourpilotyoumayneedtoadjustyourmeasuresWiththepilotdatainhandyoumayhavesomeideasofhowfutureanalysescouldbedifferentandpotentiallyimprovethequalityofyourprogramTakesometimetobrainstormotherrelevantdatasetsthatyoudidnothavethetimetopursueearlierincludingmessydepartmentaldatathatneedsmoreattentionorsourcesofpublicdatafromthestateandfederalgovernmentnonprofitsandevenprivatecompaniesthatcouldsupplementyourunderstandingofanissueUsingmachinelearningtechniquesyoumightbeabletobetter
Shareyourstoryandsuccess
54
predictdemandforhomelessservicesusingNOAAweatherforecastspreemptwatermainbreaksusingsoilstratigraphyorinvestineconomicdevelopmentbymonitoringrealestatelistingdata
Advancedanalyticaltechniquescanalsohelpyourimplementationrunmoreeffectivelybyintegratingdata-informedtargetinganddecisiontoolsintotheprocessByautomatingroutineworkflowsandsurfacingsubtletrendsthesetoolscanhelpyouiterateinamoreprecisemannerandprovidekeystrategicguidancetofront-lineemployeesthatwouldotherwisebeunavailableThesesortsofdata-science-enhancedprocessescanbeclassifiedasoneofsixprojecttypesasexploredfurtherbytheNewOrleansLANOLAlyticsteamFormoreguidanceonscopingasimilarprojecttheGovExadvancedanalyticsscopingguidegivesyouastep-by-stepoverviewofhowtobesuccessful
Usethespacebelowtolistyourideasforpossibleadvancedanalyticsprojectsbytype
Typeofproject Ideas
Findingtheneedleinthehaystack
Prioritizingforimpact
Earlywarningtools
Betterquickerdecisions
Optimizingresourceallocation
Experimentingforwhatworks
Keepthefeedbackloopgoing
GatherinformationonpublicsentimentaboutpolicychangethroughfocusgroupssurveysandpublicmeetingsledbyadvisorygroupmembersNotewhethernewinformationisavailablethatcanhelpanalyzeorcontextualizeyourprogress(egfederalorstatedatareports)Sharetheresultsofyourpilotandthepolicychangeprocesswithothercitiesandresearchers
Shareyourstoryandsuccess
55
WorkingwitharangeofevidencePresentlythereisalotofattentionandinvestmentinevidence-basedpolicymakingincludingthefederalbipartisanCommissiononEvidence-BasedPolicymakingthePewCharitableTrustsreportsandtoolkitsonHowStatesEngageinEvidence-BasedPolicymakingandtheLauraandJohnArnoldFoundationsupportedEvidence-BasedPolicymakingCollaborativeManyoftheseinitiativesfocusonrigorousevaluation--meaningrandomizedcontrolledtrials(RCT)orotherexperimentalmodels--asthebestkindofevidencetouseHoweverthisbodyofevidenceisofteninsufficientforthespecifickindsofquestionsthatgovernmentscanhavewhenmakingadecisionaboutpolicyInthisguideweproposeanapproachtosupportgovernmentsinincorporatingadditionaltypesofevidenceintotheirdecisionmaking
OnewaytounderstandarangeofevidenceistouseNestarsquosStandardsofEvidencetoframeanddefineevidenceTheBritishnonprofitNestaworkstoinjectpositivechangeintothewayUKgovernmentpoliciesandprogramsfunctionOneoftheirkeyareasofworkhasbeendevelopingWhatWorksCentresorganizationsthatcollectcommissionsynthesizeanddisseminateevidenceaboutinterventioneffectivenessacrossarangeofsocialimpactareasNestadevelopedthefollowingStandardsofEvidencetoprovideausefulwaytodocumenthowdifferentkindsofinformationdemonstratedifferentlevelsofevidence
Workingwitharangeofevidence
56
Theimportanceofpotentiallybiasedinformation
Onechallengepresentedbyanevidenceclassificationthatacknowledgesthevalueofnon-scientificformsofevidenceistofigureouthowtobalanceallofthepiecestogetherwhenyouusethemTounderstandhowtodothiswehavetothinkabouthowtopositionpotentiallybiasedinformationindecisionmaking
WhenwelookatinformationwhichhasnotbeenrigorouslyevaluatedweacceptthatthisinformationmightbebiasedThewholereasonweconductrigorousevaluationistoremovebiasfromourobservationsWeconductstructuredtrialstoproduceresultsthatareconsistentandgeneralizableIfafindingisunbiasedthisaccuracymdashthislackofbiasmdashcanbeprovedbecauseanyindependenttestercanreproduceitwhentheyfollowanidenticalprocess
YoucanlookatpotentiallybiasedinformationasjustinformationthatisearlieroninitspathtorefinementreplicationandverifiedaccuracyApieceofinformationbasedonanyoneindividualrsquosobservationsislikelytobebiasedbecauseitiscomingfromjustonesourceBiasisreducedbygatheringtogetherlotsofindependentsourcesofinformationandfiguringoutitsmostcommonattributes
MaybethebiasedinformationmakesaclaimthatistoolargemdashitsayssomethingistrueunderallconditionswhenitisreallyonlytrueundercertainconditionsorthataneffectofaninterventionisbiggerthanitstrueaverageorthataneffectlastsovertimewhenitdoesnotPerhapschangingtheclaimtobemorespecificmdashcriticallyreviewingtheinformationtoseeifspecificconditionsmustbemettoachievetheoutcomesmdashwouldmaketheclaimmoreaccurate
IfyoutestedyourclaimfounditdidnotholdupinsomecaseschangedittesteditagainuntilyourclaimrepeatedlyheldupthenyoucoulddemonstratethattheclaimwasnolongerbiasedbutaccurateThisprocessoftestingimprovingtestingandvalidatingiswhatwegetfromstructuredscientificstudy
Withindividualpiecesofinformation--crediblehypothesescasestudiesexpertopinion--youcannotassumethatkindofaccuracyAtthesametimetheseearlyobservationsarefoundationaltoanyreplicablescientificobservationsWewillnevergetanywhereifwedonotstartwiththesepiecesofinformationeveniftheymightbebiased
Combininglevelsofevidencetogetthemostdetailedinformationforyourcity
ItisespeciallyimportanttoacceptdifferentformsofevidencewhenyoufacequestionsthathavenotyetbeentestedbyindependentstudiesSomeofthemostimportantquestionsrelatetothingsthatareveryspecifictoyoursituationYoufaceaparticularcombinationof
Workingwitharangeofevidence
57
issuesactorstrendsandresourcesthatisultimatelydifferentfromeveryotherplaceintheworld
ThisisimportantbecauseaspolicypractitionersandnotscientistsyoucaremoreaboutyourspecificcasethanahypotheticalaveragegeneralcaseYoucarelessaboutthetotallikelyrangeofimpactsfromapolicythanhowthissingleimplementationwillworkYouwantinformationaboutpoliciesthatwillprovideyouwiththebestpossiblepredictionabouthowtheywillworkinyourcontext
ByputtingdifferentkindsofevidenceintothesameframeworkNestarsquosStandardsofEvidenceimplythatwhenyoucannotgetrigorouslytestedexperimentallyverifiedinformationitisappropriatetoworkwithwhatwedohaveThisisrelevantforthemunicipalimplementationcontextbecausenostudypulledfromaliteraturereviewwillaccountforallofthevariablesyoufaceinyourowncontext
Intheabsenceofhavingastudythatpredictshowaninterventionworksinyourspecificcontextyouwillneednon-scientificinformationtotrytofillinthosegapsYouwillneedtogatherstoriesfromresidentsstaffmembersandotherstakeholdersthatsitonLevels1or2ofthestandardsofevidence
Someofthekindsofinformationyouwillprobablygetmoreeasilyfrompeoplethanstudiesinclude
howpeopleincitieslikeyoursreactedtoaninterventioninpublicforumshowpeoplewhoimplementedaninterventioninacitylikeyoursfiguredouthowtogetitpassedhowthecitycommunicatedpubliclyabouttheapparentbenefitsanddrawbacksoftheinterventionthepoliticalacceptabilityoftheintervention
Youcanremainsimultaneouslyawarethattheinformationyouaregatheringthroughindividualstoriesisbiased--notgeneralizableinformedbyspecificperspectives--whilealsorecognizingthathavingsomeinformationaboutsomethingimportantisbetterthannoinformationEvidencefromthefirsttwoevidencelevelsareessentialbuildingblocksofknowledgeandperfectlyusefulwhenyoucannotgetdetailedunbiasedknowledge
Youhavetostartsomewhereandyoushouldalwaysstartwithevidenceyouhaveaboutyourveryspecificconditionsthatyouknowareimportantforsuccess
Workingwitharangeofevidence
58
Thankyou
Interviewswithcities
GovExheldsemi-structuredinterviewswithsixstaffmembersinfourcitiestoestablishtheinitialstepsoftheRoadmapWeaskedrespondentstotalkaboutaninitiativeorpolicythattheyhavebeenresponsibleforimplementingintheircityRespondentsfromthreecitiesspokeabouttheireffortstoaddresshomelessnessandrespondentsfromonecityspokeaboutblightAfterrespondentsidentifiedtheproblemtheyspokeabouthowthecityprioritizedthisissuethevariousstepsinvolvedandbarrierspresent
GovExconductedtextanalysisoftheseinterviewswhichrevealedthatpeoplewasthemostcommonlyusedtermduringtheseconversationswhendiscussingthestepstochangeaprogramorpolicyThisisathemethroughoutthisRoadmapThefirstsectionbeginswithidentifyingtherelevantpeopleTheconversationsaroundevidencearefocusedonhumanbehaviorratherthanmethodologyorstrengthofevidence
Thankyou
59
OncewehadadraftoftheRoadmapwefollowedupwithcitiesonanindividualbasistolearnmoreaboutwhatwouldbehelpfulWearegratefultothefollowingcitystafffortheinterviewsandindividualconversations
MichaelBaskinChiefPolicyOfficerChattanoogaTN
MelissaBridgesPerformanceandInnovationCoordinatorLittleRockAR
VictoriaCarreoacutenAdministrativeOfficerLasVegasNV
TyroneGrandstrandCommunityDevelopmentPlannerFargoND
TibLaughlinDirectorofGeneralServicesKansasCityKS
TimMorelandDirectorofPerformanceManagementandOpenDataChattanoogaTN
CharlesRosendahlManagementAnalystDentonTX
Thankyou
60
MelissaSiebenAssistantCountyAdministratorKansasCityKS
GinaThompsonDirectorofStrategicServicesLewisvilleTX
JHUFacultycollaboration
GovExalsoconvenedthecitycontactsandfacultypartnersinaseriesofAdvisoryGroupmeetingsThesemeetingsservedtoconnectcitiesandfacultytounderstandoneanothersperspective
StefanBaralBloombergSchoolofPublicHealth
AshleyBernerJHUSchoolofEducation
MelissaDavey-RothwellBloombergSchoolofPublicHealth
JonLinksBloombergSchoolofPublicHealth
TamarMendelsonBloombergSchoolofPublicHealth
KristinMmariBloombergSchoolofPublicHealth
KeshiaPollackPorterBloombergSchoolofPublicHealth
JoshSharfsteinBloombergSchoolofPublicHealth
LizStuartBloombergSchoolofPublicHealth
LindsayThompsonCareyBusinessSchool
DanielWebsterBloombergSchoolofPublicHealth
Partnersindataandevidence-basedpolicy
Wealsospokewithpartnerswhoareexpertsinthefieldofevidence-basedpolicywhosharedtheirknowledgeandresourcesofcurrentandfuturethinkinginevidence-informedthinking
WendyBarrenoAssociateDirectorWilsonSheehanLabforEconomicOpportunitiesUniversityofNotreDame
TiffanyDavenportDirectorofDataPartnershipsBaltimoresPromise
TomiHiersDirectorBaltimoreCivicSiteAnnieECaseyFoundation
ChrisKingsleySeniorAssociateAnnieECaseyFoundation
KathrynOliverAssociateProfessorLondonSchoolofHygieneandTropicalMedicine
Thankyou
61
KristenWatkinsSeniorPolicyAssociateJ-PAL
GovExParticipants
GovExAnalystMiriamMcKinneyanalyzedtheinterviewtranscriptsandconductedtextanalysisSeniorImplementationAdvisorEmilyShawwillintegrateRoadmapactivitieswithadditionalservicedeliveryactivitiesatGovEx
Thankyou
62
ReferencesAbrahamKGampHaskinsR(2017)Thepromiseofevidence-basedpolicymakingReportofthecommissiononevidence-basedpolicymakingWashingtonDCCommissiononEvidence-basedPolicymakingBigmove2Extendingthegreen(nd)Retrievedfromhttpswwwcityoflewisvillecomdoing-businesslewisville-2025big-move-2-extending-the-greenBatesMAampGlennersterR(2017)ThegeneralizabilitypuzzleRetrievedfromhttpsssirorgarticlesentrythe_generalizability_puzzleBuildwithnotfor(2018January)RetrievedfromhttpsbuildwithnotforwordpresscomCairneyPampOliverK(2017)Evidence-basedpolicymakingisnotlikeevidence-basedmedicinesohowfarshouldyougotobridgethedividebetweenevidenceandpolicyHealthResearchPolicyandSystemsdoi101186s12961-017-0192-xCairneyPOliverKampWellsteadA(2016)TobridgethedividebetweenevidenceandpolicyReduceambiguityasmuchasuncertaintyPublicAdministrationReview76(3)399-402Retrievedfromhttpdxdoiorg101111puar12555Centreforpublicimpact(nd)RetrievedfromhttpswwwcentreforpublicimpactorgCernaL(2013)ThenatureofpolicychangeandimplementationAreviewofdifferenttheoreticalapproachesOrganisationforEconomicCo-OperationandDevelopmentRetrievedfromhttpswwwoecdorgeduceriTheNatureofPolicyChangeandImplementationpdfColoradoSprings(nd)Retrievedfromhttpscoloradospringsgovcommunicationspagewhat-expect-when-you-speakupContinuumofcare(coc)program(nd)RetrievedfromhttpswwwhudexchangeinfoprogramscocCouncilpresidentkicksoff$12millionyouthfund(nd)Retrievedfromhttpwwwbaltimorecitycouncilcomcontentcouncil-president-kicks-12-million-youth-fundDurlakJAampDuPreEP(2008)ImplementationmattersAreviewofresearchontheinfluenceofimplementationonprogramoutcomesandthefactorsaffectingimplementationAmericanJournalofCommunityPsychology41327-350Retrievedfromhttpswwwncbinlmnihgovpubmed18322790Fishbonediagram(ishikawa)(nd)Retrievedfromhttpasqorglearn-about-qualitycause-analysis-toolsoverviewfishbonehtmlFivewhysGettingtotherootofaproblemquickly(nd)RetrievedfromhttpswwwmindtoolscompagesarticlenewTMC_5Whtm
References
63
FligsteinNampMcAdamD(2011)TowardageneraltheoryofstrategicactionfieldsSociologicalTheory29(1)1-26RetrievedfromhttpsociologyberkeleyedusitesdefaultfilesfacultyfligsteinFligstein20McAdam20Soc20Theory20PaperpdHeinkelP(2018January12)CitymullspermitsforbikesharingcompaniesRetrievedfromhttpwwwdentonrccomnewsdenton20180112city-mulls-permits-bike-sharing-companiesHowisareamedianincomecalculated(2012October)Retrievedfromhttpswwwhudexchangeinfofaqs843how-is-area-median-income-calculatedLampkinLMampHatryHP(2003)KeystepsinoutcomemanagementUrbanInstituteRetrievedfromhttpswwwurbanorgresearchpublicationkey-steps-outcome-managementMarshDampSmithM(2000)UnderstandingpolicynetworksTowardsadialecticalapproachPoliticalStudies484-21Retrievedfromhttpjournalssagepubcomdoiabs1011111467-924800247McCannL(nd)RetrievedfromhttplaurenellencomMoultonSampSandfortJR(2016)ThestrategicactionfieldframeworkforpolicyimplementationresearchPolicyStudiesJournal45(1)Retrievedfromhttponlinelibrarywileycomstore101111psj12147assetpsj12147pdfv=1ampt=jdkhhcp4amps=e184c625e1855a6b1615ccb99ee27b1c0b3212d8PaynterB(2018January1)What100citiesarelearningfromeachotherbysharingtheirdataRetrievedfromhttpswwwfastcompanycom40518004what-100-cities-are-learning-from-each-other-by-sharing-their-dataPeakacademy(nd)Retrievedfromhttpswwwdenvergovorgcontentdenvergovenmayors-officeprograms-initiativespeak-performancepeak-academyhtmlPinnockHBarwickMCarpenterCREldridgeSGrandesGGriffithsCJRycroft-MaloneJMeissnerPMurrayEPatelASheikhAampTaylorSJC(2017)Standardsforreportingimplementationstudies(stari)ExplanationandelaborationdocumentBMJOpenRetrievedfromhttpbmjopenbmjcomcontentbmjopen74e013318fullpdfProctorESilmereHRaghavanRHovmandPAaronsGBungerAGriffeyRampHensleyM(2011)OutcomesforimplementationresearchConceptualdistinctionsmeasurementchallengesandresearchagendaAdministrationandPolicyinMentalHealth38(2)65-76doi101007s10488-010-0319-7TandonSD(2018)ProgressincommunityhealthpartnershipsResearcheducationandactionRetrievedfromhttpswwwpressjhuedujournalsprogress-community-health-partnerships-research-education-and-actionProvanKGampKenisP(2007)ModesofnetworkgovernanceStructuremanagementandeffectivenessJournalofPublicAdministrationResearchandTheory
References
64
18(2)229-252Retrievedfromhttpsacademicoupcomjpartarticle182229935895SabatierPA(1988)Anadvocacycoalitionframeworkofpolicychangeandtheroleofpolicy-orientedlearningthereinPolicySciences21(3)129-168Retrievedfromhttpwwwjstororgstable4532139SandstromAampCarlssonL(2008)TheperformanceofpolicynetworksTherelationbetweennetworkstructureandnetworkperformancePolicyStudiesJournal36497-524Retrievedfromhttponlinelibrarywileycomdoi101111j1541-0072200800281xabstractSchwartzMampWilsonE(nd)WhocanaffordtoliveinahomeAlookatdatafromthe2006americancommunitysurveyRetrievedfromhttpswwwcensusgovhousingcensuspublicationswho-can-affordpdfShearerJCAbelsonAKouyateBLavisJNampWaltG(2016)WhydopolicieschangeInstitutionsinterestsideasandnetworksinthreecasesofpolicyreformHealthPolicyandPlanning31(9)1200-1211Retrievedfromhttpsdoiorg101093heapolczw052ThelessonsofadministrativedataHigh-profilepolicy-relevantresearchpoweredbyadministrativedata(nd)Retrievedfromhttpswwwpovertyactionlaborgsitesdefaultfilesresources20161114-The-Lessons-of-Administrative-DatapdfToolsandotherneedsforqualitativeandquantitativemethods(2015February)AgencyforHealthcareResearchandQualityRetrievedfromhttpwwwahrqgovprofessionalsquality-patient-safetytalkingqualityassesstoolshtml
WeaverRCampBagchi-SenS(2013)SpatialanalysisofurbandeclineThegeographyofblightAppliedGeography4061-70
References
65
UniversityPartners
Name Title Affiliation Area(s)ofexpertise
Contactinformation
Examplesofworkingwithnon-academicdecisionmakers
WendyPBarreno
AssociateDirector
WilsonSheehanLabforEconomicOpportunitiesUniversityofNotreDame
LEOsfocusareasincludehealtheducationhousingchildwelfareeconomicopportunitycriminaljustice
wbarrenonduduorleondedu
MelissaDavey-RothwellPhD
AssociateScientist
JohnsHopkinsBloombergSchoolofPublicHealth
Health mdavey1jhuedu410-614-5854
BaltimoreCityHealthDepartmentLeadersMarylandDepartmentofHealthAddictionRecoveryServicesinDCExecutiveDirectorsofcommunityorganizationsacrosstheUSthatfocusonHIVSTIsubstanceabuseandimplementationscience
ElizabethStuart
AssociateDeanforEducationandProfessorDepartmentsofMentalHealthBiostatisticsandHealthPolicyand
JohnsHopkinsBloombergSchoolofPublicHealth
Educationsocialpolicystatisticalmethods
estuartjhuedu
Resources
66
Management
Design-ThinkingResources
EngagementGuideforCivicTech
Journeymap
PerformanceManagementResources
DenverPeakAcademy
Proxymeasuresguide
Talkingaboutperformancewithoutcomplicatingit(video)
SettingPerformanceTargetsguide
Datavisualizationguide
Performancemanagementgettingstartedguide
Exercises
Fishboneexercise
5Whys
AdministrativeData
TheLessonsofAdministrativeData(JPAL)
EvaluationResources
Quantitativeandqualitativemethods(USDepartmentofHealthandHumanServicesAgencyforHealthcareResearchandQuality)
KeyStepsinOutcomeManagement(UrbanInstitute)
RCT-YES
Sageresearchmethods
BestPracticesforMixedMethodsResearchintheHealthSciences
ACCORDChecklist
Pilotandimplementationplanning
Benchmarkingexamplesandproxymeasures
Seattleopendataplaybook
Resources
67
Datasciencescopingguide
Resources
68
AboutGovExGovExhelpsgovernmentsusedataandevidencetoimproveresidentsrsquolivesWesupportgovernmentsinbuildingtheirinternalcapacitytooperatesustainableanalyticspracticeswhichimprovecommunityoutcomesinfieldsfrompublicsafetytoeconomicdevelopmentGovExhasaproventrackrecordofworkinginconcertwithpartnerstoprovidetechnicalassistanceandtrainingatscale
SinceourlaunchinApril2015GovExhas
Trainedover2000governmentemployeesinpersonandonlineintopicsincludingdatamanagementperformancemanagementopendatacommunityengagementandanalytics
Providedtechnicalassistancetomorethan100localgovernmentsacrosstheglobeimprovingtheiruseofdatatoachieveefficienciesinservicedeliveryandbetteroutcomesforresidents
Assessedthecapacityofmorethan130governmentalorganizationstodeterminehowleadersstaffandexternalstakeholdersmanageandusedata
Localgovernmentsareincreasinglyonthefrontlinesofaddressinglargescaleproblemswithglobalimplicationssuchasclimatechangepublicsafetyandeconomicinequality
Whileevidence-basedinterventionstohelpsolvethoseproblemsareavailablelocalgovernmentstaffhavelimitedtimeandcapacitytofindandimplementresearch-basedsolutionsAsaresultcitiescontinuetoworkthewaytheyalwayshave--makingincrementalimprovementstoprogramsbutnotalwayscreatingchangeatasystem-level
ThroughresourcedevelopmentandcollaborationwithJHUfacultyGovExissupportingcityleadersinusingdataandevidencetomakedecisionsandcreatinglastingrelationshipswithresearchers
TheRoadmapwasdevelopedbyGovExaspartofitspartnershipwithBloombergPhilanthropiesWhatWorksCitiesinitiative
AboutGovEx
69
- Title page
- Introduction and purpose of the Roadmap
- Using the Roadmap
-
- Gather information about the problem
- Strategically engage key partners
- Review the evidence and collect details
- Decide on a policy to implement based on local factors
- Establish a framework to monitor progress
- Assess data accessibility quality and security
- Design implementation and evaluation
- Share your story and success
-
- Working with a range of evidence
- Thank you
- References
- Resources
- About GovEx
-
Designimplementationandevaluation
Designtheimplementation
NowthatyouhaveyouraccountabilityplanandkeyindicatorsitistimetogetspecificaboutyourplanforimplementationThetemplatebelowwillhelpyouthinkthroughthespecificmechanismsofimplementingpolicychange
UnderstandingorganizationsandsystemsiscriticaltounderstandingandmakingdecisionsaboutwhereandhowinterventionsshouldbeappliedApolicyimplementedinasinglesettingmayshowpromisingresultsbutimplementingmultiplepoliciesandprogramsatthesametimewithdifferentactorsresponsibleforeachaspectcanhelpensuresustainablebehaviorchange
InSomervilleMAShapeUpSomerville(SUS)isacitywidepublichealthprogramthatisimplementedacrossmultiplelevelsofinfluenceandsystemsthroughinterventionsbeforeduringandafterschooltoincludechangestobuiltenvironmentsuchasbikelanesregionalmasstransportationandphysicaleducationequipmentinschoolsandgymstoincreasephysicalactivitywellnesspoliciesandschoolgardensnutritioneducationduringandafterschoolandhealthyrestaurantprogramstosupporthealthyfoodandhealthyeatinghabits
Usethetemplatebelowtoplanyourimplementation
Policyimplementationplanningtemplate
ListthespecificdesiredoutcomesInthepreviousstepyouworkedwithdepartmentstafftoestablishspecificmeasurableoutcomesandinterimindicatorsPrioritizetheindicatorsthatwilldemonstrateprogressUsethespacebelowtocapturethedesiredoutcomesofthepolicychange
ExampleShapeupSomervillebeganwiththegoalofpreventingobesityin1stthrough3rdgraders
Yourturn
____________________________________________________________________________
____________________________________________________________________________
Designimplementationandevaluation
44
SpecifythetargetpopulationAreyoudesigninganinterventionthattargetsonlycertainsitesAreyouplanningonimplementingaprogramorinitiativeforaspecifictargetaudienceClearlyspecifythepopulationyouplantotarget
IfyouplantoconductarigorousimpactevaluationasnotedbytheWilsonSheehanLabforEconomicOpportunitiesitisimportanttoestablishconcreteobjectiveeligibilitycriteriaandensurethatthesecriteriaareappliedconsistentlybyallstaffduringscreeningandintakeItisalsovitaltoidentifytheindividualswhoareeligibleforaparticularprogramorserviceevenifeveryoneeligiblecannotbeservedbyexistingresourcesThishelpscitiesidentifyanyunmetneedsandguidefutureresourceallocationwhilealsocreatingthepossibilityofarigorousimpactevaluationbyidentifyingapotentialcomparisongroupFinallywhenconductingscreeningandintakeitisimperativetocollectandmaintainbasicinformation(egnamedateofbirthgender)aboutallprogram-eligibleindividualsnotjustthosewhoparticipateintheprogram
ExampleIntheShapeUpSomervilleexampletheprogrambeganbytargeting1stthrough3rdgraders
Yourturn
____________________________________________________________________________
____________________________________________________________________________
DefinethelocationWherewillthepolicychangetakeplaceWillimplementationtakeplaceatonesiteormultiplesites
ExampleShapeUpSomervilleinterventionsweredeliveredinschools
Yourturn
____________________________________________________________________________
____________________________________________________________________________
ClarifyimplementationpartnersandarticulatetheirrolesWhatarethekeyorganizationsinstitutionsorpeopleresponsibleforimplementationTheprojectteamshouldbeasclearandspecificaspossibleinprovidingadescriptionoftheconcreteactivitiesthatimplementationpartnersaremeanttocarryoutPracticeprofilesmaybeahelpfultooltohelpdefinewhatisrequiredoftheactorswhoareresponsibleforimplementingtheintervention
Designimplementationandevaluation
45
ExampleInSomervillemayoralleadershipandengagementofmultiplecityagencies(publichealthprimarycareschoolsCambridgeHealthAllianceetc)streamlinedandconnectedthevariousShapeUpSomervilleinitiativesandgrantswhichhelpedtokeepthetalentandenergyfocusedonhealthyeatingandactivelivingforallsectorsofSomervilleSUSpartnersultimatelyincludedschoolsnonprofitsCityandstateagenciesandnationalfoundations
Yourturn
____________________________________________________________________________
____________________________________________________________________________
DescribetheinterventionWhatarethestepsactivitiesorprocessesRememberthatthemechanismthespecificstepsorprocessthatleadtobehaviorchangeiskeytounderstandingtheeffectivenessofthepolicychangeandsharingtheresultswithothers
ExampleGoalsofShapeUpSomervilleweretoinfluenceeverypartofanearlyelementaryschoolchildrsquosdayndashbeforeduringandafter-schooltopromotehealthyeatingphysicalactivityandtheirweightstatusInterventionincludedfoodserviceenhancementstoimprovebreakfastandlunchandstaffprofessionaldevelopmentwalktoschoolactivitiesSUSclassroomcurriculumandprofessionaldevelopmentschoolwellnesspolicydevelopmentSUSafter-schoolcurriculumandprofessionaldevelopmentoutreachandeducationtohomethroughmaterialsforumseventsldquoSUSapprovedrdquorestaurantscommunityoutreachandcapacitybuildingthroughpolicydevelopmenttrainingsmediaplacements
Yourturn
____________________________________________________________________________
____________________________________________________________________________
ClarifytimelineandmilestonesfordeliveryandevaluationHowfrequentlywilltheservicesinterventionbedeliveredHowlongwilltheprogramrunWhatisthetimetoexpectedimpacts(eg12monthsafterprogramcompletion)
Designimplementationandevaluation
46
ExampleShapeUpSomervilleledbyTuftsUniversityFriedmanSchoolofNutritionScienceandPolicyconductedaquasi-experimental(nonrandomized)interventionstudyamong1stthrough3rdgradestudentsinSomerville(intervention)andtwocomparisoncommunitiesfor9-monthsduringoneschoolyear(fall2002ndashspring2003)ThestudyshowedpositiveassociationoftheinterventiononchangeinBMISUSreleasedreportssummarizingevaluationsprogresstodateplansforadditionalprojectsandimpactstenandfifteenyearsintotheinitiative
Yourturn
____________________________________________________________________________
____________________________________________________________________________
PlanyouranalysisTheprevioussectionaddressedestablishinganaccountabilityframeworktomonitorprogressandimplementationDeterminewhichindicatorswillbediscussthroughthatprocesswhowillsharethedataandanalysisandhowfeedbackfromthoseconversationswillbeincorporatedbytheprojectteam
ExampleTheMayorrsquosofficeimplementedadata-drivendecisionmakingapproachcalledSomerStatwhichwassynergisticwiththeapproachestakenbytheresearchandevaluationteamsdeployedforthespecificgrants
Yourturn
____________________________________________________________________________
____________________________________________________________________________
PlanforsustainabilityIfthepolicyandprogramchangeissuccessfulhowwilltheprojectteamsustainandexpandtoadditionalsitesorpopulations
ExampleAlthoughmostofthepartnershipsforgedintheearlyyearsweremaintainednewgrantsandcollaborationsexpandedattentiontoolderpublicschoolstudentsactivetransportationandtheGreenLinesubwayextensionthebuiltenvironmentfarmerrsquosmarketeffortssuchasldquoGrowninSomervillerdquowithUnionMainStreeteconomicdevelopmentandhealthdisparitiesaswellasdatacollectionmonitoringandsurveillance
Yourturn
Designimplementationandevaluation
47
____________________________________________________________________________
____________________________________________________________________________
Pilotplanningtemplate
IfyouplantoconductapilotfirstdeterminethepurposeofthepilotDoyouwanttoobservewhetherthepolicychangeresultsinthedesiredbehaviorchangeAreyouconcernedaboutfeasibilityscalabilityorcostDoyouwanttotestwhetheryourdesiredresultscanbeachievedthroughthispolicyapproachClarifythegoalofthepilotandthequestionsyouhopetoanswerThiswillhelpyouidentifythekeyindicatorsthatyouwillusetomeasurewhetherthepilotissuccessfulBeforebeginningestablishabaselineofcurrentperformanceandsetyourtargetsaccordinglybasedonwhatyouhopetoaccomplishthroughthepilot
NextestablishthepilotplanforyourprojectteamincludingmilestonesforteamcheckinstodeterminewhethertomoveaheadorchangecourseThisisseparatefromthemeetingsthatwilloccuraspartoftheaccountabilityframeworkUsethespacebelowtocreateyourtimelineanddevelopyourmeetingschedule
Task Personresponsible Startdate Checkindate Deliverable
Planyourevaluation
WhileyouaredesigningtheimplementationitisalsotimetobegintodevelopyourplanforanalysisAsnotedbytheWilsonSheehanLabforEconomicOpportunitiesdecisionsaboutprogramsetup-includingoutreachscreeningintakedatatrackingandenrollment-candeterminewhetherornotarigorousimpactevaluationwillbepossibleAsdiscussedcityleadersmayfindithelpfultoincludeevaluationpartnersinearlyconversationsaboutprogramsetupDuringpreviousstagesyoumayhaveworkedwitharesearchertoplanyourproposedpolicychangeYoushouldbeclearaboutthepurposeoftheevaluationanddevelopanevaluationplanBelowareadditionalconsiderations
Causationvstrendanalysis
IntheEstablishaframeworktomonitorprogressstagetherewasguidanceontrackingtrendsovertimeItisimportanttonoteherethatmonitoringdataisdifferentfromanalyzingacausaleffectTrendanalysiswillillustratethatchangesareoccurringandmayleadtoquestionsthatcouldbeansweredbyyourevaluationbuttrendanalysisalonedoesnotprovideevidenceaboutwhatisdrivingthosechangesYoumayseechangesinyour
Designimplementationandevaluation
48
outcomeindicatorsbasedonvariablesoutsideoftheCityscontrolForinstanceachangeinfederaltradepolicycanhaveanimpactemploymentratesinyourCityoutsideofyourCitysnewjobtrainingandplacementprogram
ToaddresscausalityyoumustcomparetheoutcomesofindividualswhoreceivedtheinterventionwiththosewhodidnotWithoutacomparisongroupofsimilarindividualswhowerenotexposedtotheinterventionitisimpossibletoknowwhichfactorsareresponsiblefortheoutcomesRandomizedControlTrials(RCTs)inwhichindividualsarerandomlyassignedtotreatmentandcontrolgroupsarethegoldstandardinprovingcausalityIfyoucandemonstratethatoutcomesimprovedforthosewhoreceivedtheinterventionanddidnotimproveatallorimprovedlessforacomparisongroupyoucanbesomewhatcertainthattheinterventionwasthecause
Examplesofothernon-causalimpactevaluationsthatarerigorousandreliableincludecertainquasi-experimentalstudiesusingcomparisongroupdesignssmartinterruptedtimeseriesanalysesthatmodeltrendsovertimeinwaysthatallowestimationofinterventioneffectswelldesignedpre-posttestanalysesneedsassessmentsimplementationstudiesetcFormoreinformationondifferenttypesofevaluationsseetheSageResearchMethodswebsite
Lookfordifferencesingroups
BuildintimetoalsoexaminethepilotdatatoseeifgroupsareexperiencingdifferentresultsReviewingperformancebasedonoutcomemeasuresforapopulationinaggregatemayhidethesubtlewaysthatsomepeoplemaybeaffecteddifferentlysoitisnecessarytocarefullyanalyzethedatafordisparateoutcomesForinstancetheMovingtoOpportunity(MTO)studythatexaminedtheimpactofhousingvouchersonfamilieswholivedinhigh-povertyneighborhoodsfoundthatchildrenwhomovedbeforetheageof13hadbettereducationandhigherincomesasadultsHoweveradultsdidnotseeimprovededucationandemploymentoutcomes
Disaggregatingdataintomorespecificcategoriesofraceandethnicitycanhelpunderstandtrendsanddisparitiesinamoregranularwaywhichcanhelpdecisionmakerstargetpoliciesandinterventionsinamoreequitableway
EarlierintheRoadmapweworkedtoidentifystakeholdersandwhomightbepositivelyandnegativelyimpactedbyachangeinpolicyUsetheresultsofthatexercisetoinvolvethecommunityininterpretingdataanalysisandtogatherqualitativefeedbackfromthecommunitytocallattentiontoissuesForinstanceanincreaseinyouthdetentionmaybeexplainedbyinefficienciesorlackofaccesstotransportation
Designimplementationandevaluation
49
Designimplementationandevaluation
50
Shareyourstoryandsuccess
Sharetheresults
Storytelling
AnintegralstepinyourstrategyissharingtheresultsofyourpilotorpolicychangewithyourcommunityThisincludesdevelopingmessagingaroundtheworkofyourorganizationandcreatingaplantodisseminateyourstoriesWhytellstoriesaboutyourwork
Informandorunderstandasubjectorsituation
Giveneededcontextofdataorbitsofinformation
Relatablestoriesarememorableforreadersviewers
ReadersorviewerswillseetheinformationasmorecompellingwhenthereisademonstratedandrelatablechangeorimpactonthemselvesortheirfriendsneighborsandcolleaguesStoriesareaboutpeopleandgovernmentisfundamentallyabouthelpingpeopleSotellthestoriesofthepeopleyouandyourcitycolleagueshelped
Keepthesetipsinmindasyoudevelopyourstorywithyourteam
WhatwereyourfindingsandwhyshouldanyonecareTellyouraudiencehowtheproblemwassolvedWhathastheimpactbeenonresidentsandcitystafftodateWhatwasthesuccessandweretheremultiplesuccessesalongthewayIfyouhavepreliminaryresultsordatashowingimprovementtosharethisisthetimetodoit
WhowasinvolvedandimpactedReviewthelistofstakeholdersHowwereeachofthemimpactedbytheprogramHumanizeyourstoryandimpactbyaddingadditionalvoicesThesevoicescanbeoneofthefollowing
Apersonorfamilywhowaspositivelyaffectedbythechangesuccess
Aninvolvedthird-partydescribingboththeproblemandthesolution(ieanorganizationnonprofitprovidersbusinesstradeassociationsportsteametc)whoispartofthecommunitythatthegovernmentworkedwithonthechangesuccess
Someonefromthegovernmentitselfwhocanspeakaboutthechangesuccess
TellmoreaboutyourcityDescribeyourcityincludingpopulationdemographicsindustriesetcWhatareitschallengesWhatisitknownforWhatmakesituniqueHowisitsimilarordifferentfromothercitiesintheUS
Shareyourstoryandsuccess
51
DescribeyourcityrsquoschallengeWriteoutthechallengeyourcityisfacing(asdefinedinyourproblemstatement)HowdoesitaffectresidentsHowdoesthisproblemaffecttheworkofcitystaffmembers
AddthedetailsProvidedetailsonyourappliedresearchstrategyTalkaboutwhatyourcityhasdonenthepasttoresolvetheissueDiscusshowyourstrategyhaschangedwhenyoubegantoadoptthisapproachWasmoneysavedorinvested
Usethisstorymapexercisetoorganizeyourthoughtsandguaranteethatyouincludealloftheelementsofatraditionalstory
AfteryoudevelopyourmessagingcreateacommunicationsmarketingandoutreachstrategydesignedtospreadyourmessageYourstrategyshouldwalkyouthroughthefollowingkeyactivities
1 IdentifyandprofileyouraudienceWhencreatinganddisseminatingyourmessageitisalwaysgoodtothinkaboutyouraudienceWhoexactlywouldbeinterestedinwhatyouhavetosayThegeneralpopulationofthecityPeoplewhoworkinnonprofitsBusinessownersItmattersbecausenoteveryonewholivesinacitymaybethetargetoftheinformationBusinessownersmaybemoreinterestedindatashowingeffectsonthebusinessdistrictthanothersandsoonWewillcoverthislaterbutthisstepwillalsohelpyoutothinkofthemosteffectivewaytoreachyouraudience
2 Defineyourcalltoaction(ifyouhaveone)SpendsometimeconsideringwhyyouareinterestedintellingyourstoryDoyousimplywanttoupdatethepubliconyourprogressorcelebrateyoursuccessesAreyouinterestedinyouraudiencechangingitsbehaviororsupportingaspecificinitiativebeinglaunchedbyyourorganizationIfyouwanttohaveyouraudienceactorbehaveinacertainwaybespecificandincorporateitinyourmessaging
Shareyourstoryandsuccess
52
3 SelectperformancemeasuresThinkabouthowyouwoulddetermineifyourcampaignissuccessfulandselectappropriateperformancemeasuresYouwillrevisitthesemeasuresattheendofyourcampaigncommunicationsplanimplementationtodeterminewhetheryouneedtomodifyyourcommunicationsandoutreachstrategyforthefuture
4 MethodstocommunicateyourstoryTherearemanywaystocommunicatesuccessofyourprojectHerearesomesuggestionsonreachingaudiencesItisimportanttodeterminewhichmethodsarethemosteffectiveforyourdesiredaudiencetoreceiveinformationDoyourresearch(PewResearchCenterhasexcellentstudiesonthereachofmediadigitalmediaandmarketing)ForexampleifseniorsareyourtargetpopulationandtheyarelessinclinedtouseInstagram(thatistrue)youmaywanttoconsiderFacebookorthenightlylocalnewsbothofwhichareusedmoreoftenbyseniorsBelowisalistofsomemethodsandtoolsyoucanuse
i Pressreleasestolocalnewspapersandtvstations
ii ShortsmartphonevideosAgreatwaytoengageanaudienceisthroughvideoAskonequestionofsomeonewhoispartofthestoryMaybethereisasocialworkerwhocandiscusshungerinyourcityThesevideoscanbereusedonmultipleplatforms
iii Long-formwriting
iv TellingindividualstoriesthroughaseriesofphotosandshortnarrativesforpublicationonFacebookorInstagram
v Op-edsopinionpiecesplacedinthelocalpaper
vi Emailupdatestoyourlist
vii Connectingwithappropriateorganizationsthatareinvestedinthetopicandworkwiththecommunityandaskthemtodisseminateinformationtotheirmembers
viii EmployingldquoWordofMouthrdquo(WOM)tacticsbycreatingtownhallsorfreeeventsforresidentstoattendintheirownneighborhoods
5 DeterminethetypeofcollateralUsuallycampaignsthattargetresidentsrequireseveralavenuestoreachthemPeoplearenotalwaysinthesameplaceorspaceHavingakeenunderstandingofyouraudiencewillassistyouwithyourchoicesonwhatneedstobedevelopedSometimesthatmeansimagesforsocialmediaaonepageflierthatgivespeopleinformationonattendinganeventoranin-depthbrochure
Shareyourstoryandsuccess
53
6 ImplementyourstrategyKeepinmindstrategiescanbefluidSometimessomethingmaynotturnoutasexpectedevenifduringplanningsessionsitwasldquosupposedtordquoItisokYoucananalyzewhathappenedduringadebriefsessionandonceyoursurveysandothermeasurementshavebeenfinalized
Visualizations
Withtherightvisualizationyoucanreviewandshareinformationinsightsanalysisperformanceandtrendsallofwhichisvaluabletoacross-sectionofthepublicButwhichvisualizationisrightforyouTochoosetherighttypeofchartgraphdeterminethetypeofquestionyouwanttoanswerSomecommoncharttypesarebelowandmoreinformationcanbefoundhere
Exploreadditionalanalyticaltechniques
Duringthegoal-settingphaseyouestablishedoutputoutcomeandimplementationmeasuresBasedontheresultsofyourpilotyoumayneedtoadjustyourmeasuresWiththepilotdatainhandyoumayhavesomeideasofhowfutureanalysescouldbedifferentandpotentiallyimprovethequalityofyourprogramTakesometimetobrainstormotherrelevantdatasetsthatyoudidnothavethetimetopursueearlierincludingmessydepartmentaldatathatneedsmoreattentionorsourcesofpublicdatafromthestateandfederalgovernmentnonprofitsandevenprivatecompaniesthatcouldsupplementyourunderstandingofanissueUsingmachinelearningtechniquesyoumightbeabletobetter
Shareyourstoryandsuccess
54
predictdemandforhomelessservicesusingNOAAweatherforecastspreemptwatermainbreaksusingsoilstratigraphyorinvestineconomicdevelopmentbymonitoringrealestatelistingdata
Advancedanalyticaltechniquescanalsohelpyourimplementationrunmoreeffectivelybyintegratingdata-informedtargetinganddecisiontoolsintotheprocessByautomatingroutineworkflowsandsurfacingsubtletrendsthesetoolscanhelpyouiterateinamoreprecisemannerandprovidekeystrategicguidancetofront-lineemployeesthatwouldotherwisebeunavailableThesesortsofdata-science-enhancedprocessescanbeclassifiedasoneofsixprojecttypesasexploredfurtherbytheNewOrleansLANOLAlyticsteamFormoreguidanceonscopingasimilarprojecttheGovExadvancedanalyticsscopingguidegivesyouastep-by-stepoverviewofhowtobesuccessful
Usethespacebelowtolistyourideasforpossibleadvancedanalyticsprojectsbytype
Typeofproject Ideas
Findingtheneedleinthehaystack
Prioritizingforimpact
Earlywarningtools
Betterquickerdecisions
Optimizingresourceallocation
Experimentingforwhatworks
Keepthefeedbackloopgoing
GatherinformationonpublicsentimentaboutpolicychangethroughfocusgroupssurveysandpublicmeetingsledbyadvisorygroupmembersNotewhethernewinformationisavailablethatcanhelpanalyzeorcontextualizeyourprogress(egfederalorstatedatareports)Sharetheresultsofyourpilotandthepolicychangeprocesswithothercitiesandresearchers
Shareyourstoryandsuccess
55
WorkingwitharangeofevidencePresentlythereisalotofattentionandinvestmentinevidence-basedpolicymakingincludingthefederalbipartisanCommissiononEvidence-BasedPolicymakingthePewCharitableTrustsreportsandtoolkitsonHowStatesEngageinEvidence-BasedPolicymakingandtheLauraandJohnArnoldFoundationsupportedEvidence-BasedPolicymakingCollaborativeManyoftheseinitiativesfocusonrigorousevaluation--meaningrandomizedcontrolledtrials(RCT)orotherexperimentalmodels--asthebestkindofevidencetouseHoweverthisbodyofevidenceisofteninsufficientforthespecifickindsofquestionsthatgovernmentscanhavewhenmakingadecisionaboutpolicyInthisguideweproposeanapproachtosupportgovernmentsinincorporatingadditionaltypesofevidenceintotheirdecisionmaking
OnewaytounderstandarangeofevidenceistouseNestarsquosStandardsofEvidencetoframeanddefineevidenceTheBritishnonprofitNestaworkstoinjectpositivechangeintothewayUKgovernmentpoliciesandprogramsfunctionOneoftheirkeyareasofworkhasbeendevelopingWhatWorksCentresorganizationsthatcollectcommissionsynthesizeanddisseminateevidenceaboutinterventioneffectivenessacrossarangeofsocialimpactareasNestadevelopedthefollowingStandardsofEvidencetoprovideausefulwaytodocumenthowdifferentkindsofinformationdemonstratedifferentlevelsofevidence
Workingwitharangeofevidence
56
Theimportanceofpotentiallybiasedinformation
Onechallengepresentedbyanevidenceclassificationthatacknowledgesthevalueofnon-scientificformsofevidenceistofigureouthowtobalanceallofthepiecestogetherwhenyouusethemTounderstandhowtodothiswehavetothinkabouthowtopositionpotentiallybiasedinformationindecisionmaking
WhenwelookatinformationwhichhasnotbeenrigorouslyevaluatedweacceptthatthisinformationmightbebiasedThewholereasonweconductrigorousevaluationistoremovebiasfromourobservationsWeconductstructuredtrialstoproduceresultsthatareconsistentandgeneralizableIfafindingisunbiasedthisaccuracymdashthislackofbiasmdashcanbeprovedbecauseanyindependenttestercanreproduceitwhentheyfollowanidenticalprocess
YoucanlookatpotentiallybiasedinformationasjustinformationthatisearlieroninitspathtorefinementreplicationandverifiedaccuracyApieceofinformationbasedonanyoneindividualrsquosobservationsislikelytobebiasedbecauseitiscomingfromjustonesourceBiasisreducedbygatheringtogetherlotsofindependentsourcesofinformationandfiguringoutitsmostcommonattributes
MaybethebiasedinformationmakesaclaimthatistoolargemdashitsayssomethingistrueunderallconditionswhenitisreallyonlytrueundercertainconditionsorthataneffectofaninterventionisbiggerthanitstrueaverageorthataneffectlastsovertimewhenitdoesnotPerhapschangingtheclaimtobemorespecificmdashcriticallyreviewingtheinformationtoseeifspecificconditionsmustbemettoachievetheoutcomesmdashwouldmaketheclaimmoreaccurate
IfyoutestedyourclaimfounditdidnotholdupinsomecaseschangedittesteditagainuntilyourclaimrepeatedlyheldupthenyoucoulddemonstratethattheclaimwasnolongerbiasedbutaccurateThisprocessoftestingimprovingtestingandvalidatingiswhatwegetfromstructuredscientificstudy
Withindividualpiecesofinformation--crediblehypothesescasestudiesexpertopinion--youcannotassumethatkindofaccuracyAtthesametimetheseearlyobservationsarefoundationaltoanyreplicablescientificobservationsWewillnevergetanywhereifwedonotstartwiththesepiecesofinformationeveniftheymightbebiased
Combininglevelsofevidencetogetthemostdetailedinformationforyourcity
ItisespeciallyimportanttoacceptdifferentformsofevidencewhenyoufacequestionsthathavenotyetbeentestedbyindependentstudiesSomeofthemostimportantquestionsrelatetothingsthatareveryspecifictoyoursituationYoufaceaparticularcombinationof
Workingwitharangeofevidence
57
issuesactorstrendsandresourcesthatisultimatelydifferentfromeveryotherplaceintheworld
ThisisimportantbecauseaspolicypractitionersandnotscientistsyoucaremoreaboutyourspecificcasethanahypotheticalaveragegeneralcaseYoucarelessaboutthetotallikelyrangeofimpactsfromapolicythanhowthissingleimplementationwillworkYouwantinformationaboutpoliciesthatwillprovideyouwiththebestpossiblepredictionabouthowtheywillworkinyourcontext
ByputtingdifferentkindsofevidenceintothesameframeworkNestarsquosStandardsofEvidenceimplythatwhenyoucannotgetrigorouslytestedexperimentallyverifiedinformationitisappropriatetoworkwithwhatwedohaveThisisrelevantforthemunicipalimplementationcontextbecausenostudypulledfromaliteraturereviewwillaccountforallofthevariablesyoufaceinyourowncontext
Intheabsenceofhavingastudythatpredictshowaninterventionworksinyourspecificcontextyouwillneednon-scientificinformationtotrytofillinthosegapsYouwillneedtogatherstoriesfromresidentsstaffmembersandotherstakeholdersthatsitonLevels1or2ofthestandardsofevidence
Someofthekindsofinformationyouwillprobablygetmoreeasilyfrompeoplethanstudiesinclude
howpeopleincitieslikeyoursreactedtoaninterventioninpublicforumshowpeoplewhoimplementedaninterventioninacitylikeyoursfiguredouthowtogetitpassedhowthecitycommunicatedpubliclyabouttheapparentbenefitsanddrawbacksoftheinterventionthepoliticalacceptabilityoftheintervention
Youcanremainsimultaneouslyawarethattheinformationyouaregatheringthroughindividualstoriesisbiased--notgeneralizableinformedbyspecificperspectives--whilealsorecognizingthathavingsomeinformationaboutsomethingimportantisbetterthannoinformationEvidencefromthefirsttwoevidencelevelsareessentialbuildingblocksofknowledgeandperfectlyusefulwhenyoucannotgetdetailedunbiasedknowledge
Youhavetostartsomewhereandyoushouldalwaysstartwithevidenceyouhaveaboutyourveryspecificconditionsthatyouknowareimportantforsuccess
Workingwitharangeofevidence
58
Thankyou
Interviewswithcities
GovExheldsemi-structuredinterviewswithsixstaffmembersinfourcitiestoestablishtheinitialstepsoftheRoadmapWeaskedrespondentstotalkaboutaninitiativeorpolicythattheyhavebeenresponsibleforimplementingintheircityRespondentsfromthreecitiesspokeabouttheireffortstoaddresshomelessnessandrespondentsfromonecityspokeaboutblightAfterrespondentsidentifiedtheproblemtheyspokeabouthowthecityprioritizedthisissuethevariousstepsinvolvedandbarrierspresent
GovExconductedtextanalysisoftheseinterviewswhichrevealedthatpeoplewasthemostcommonlyusedtermduringtheseconversationswhendiscussingthestepstochangeaprogramorpolicyThisisathemethroughoutthisRoadmapThefirstsectionbeginswithidentifyingtherelevantpeopleTheconversationsaroundevidencearefocusedonhumanbehaviorratherthanmethodologyorstrengthofevidence
Thankyou
59
OncewehadadraftoftheRoadmapwefollowedupwithcitiesonanindividualbasistolearnmoreaboutwhatwouldbehelpfulWearegratefultothefollowingcitystafffortheinterviewsandindividualconversations
MichaelBaskinChiefPolicyOfficerChattanoogaTN
MelissaBridgesPerformanceandInnovationCoordinatorLittleRockAR
VictoriaCarreoacutenAdministrativeOfficerLasVegasNV
TyroneGrandstrandCommunityDevelopmentPlannerFargoND
TibLaughlinDirectorofGeneralServicesKansasCityKS
TimMorelandDirectorofPerformanceManagementandOpenDataChattanoogaTN
CharlesRosendahlManagementAnalystDentonTX
Thankyou
60
MelissaSiebenAssistantCountyAdministratorKansasCityKS
GinaThompsonDirectorofStrategicServicesLewisvilleTX
JHUFacultycollaboration
GovExalsoconvenedthecitycontactsandfacultypartnersinaseriesofAdvisoryGroupmeetingsThesemeetingsservedtoconnectcitiesandfacultytounderstandoneanothersperspective
StefanBaralBloombergSchoolofPublicHealth
AshleyBernerJHUSchoolofEducation
MelissaDavey-RothwellBloombergSchoolofPublicHealth
JonLinksBloombergSchoolofPublicHealth
TamarMendelsonBloombergSchoolofPublicHealth
KristinMmariBloombergSchoolofPublicHealth
KeshiaPollackPorterBloombergSchoolofPublicHealth
JoshSharfsteinBloombergSchoolofPublicHealth
LizStuartBloombergSchoolofPublicHealth
LindsayThompsonCareyBusinessSchool
DanielWebsterBloombergSchoolofPublicHealth
Partnersindataandevidence-basedpolicy
Wealsospokewithpartnerswhoareexpertsinthefieldofevidence-basedpolicywhosharedtheirknowledgeandresourcesofcurrentandfuturethinkinginevidence-informedthinking
WendyBarrenoAssociateDirectorWilsonSheehanLabforEconomicOpportunitiesUniversityofNotreDame
TiffanyDavenportDirectorofDataPartnershipsBaltimoresPromise
TomiHiersDirectorBaltimoreCivicSiteAnnieECaseyFoundation
ChrisKingsleySeniorAssociateAnnieECaseyFoundation
KathrynOliverAssociateProfessorLondonSchoolofHygieneandTropicalMedicine
Thankyou
61
KristenWatkinsSeniorPolicyAssociateJ-PAL
GovExParticipants
GovExAnalystMiriamMcKinneyanalyzedtheinterviewtranscriptsandconductedtextanalysisSeniorImplementationAdvisorEmilyShawwillintegrateRoadmapactivitieswithadditionalservicedeliveryactivitiesatGovEx
Thankyou
62
ReferencesAbrahamKGampHaskinsR(2017)Thepromiseofevidence-basedpolicymakingReportofthecommissiononevidence-basedpolicymakingWashingtonDCCommissiononEvidence-basedPolicymakingBigmove2Extendingthegreen(nd)Retrievedfromhttpswwwcityoflewisvillecomdoing-businesslewisville-2025big-move-2-extending-the-greenBatesMAampGlennersterR(2017)ThegeneralizabilitypuzzleRetrievedfromhttpsssirorgarticlesentrythe_generalizability_puzzleBuildwithnotfor(2018January)RetrievedfromhttpsbuildwithnotforwordpresscomCairneyPampOliverK(2017)Evidence-basedpolicymakingisnotlikeevidence-basedmedicinesohowfarshouldyougotobridgethedividebetweenevidenceandpolicyHealthResearchPolicyandSystemsdoi101186s12961-017-0192-xCairneyPOliverKampWellsteadA(2016)TobridgethedividebetweenevidenceandpolicyReduceambiguityasmuchasuncertaintyPublicAdministrationReview76(3)399-402Retrievedfromhttpdxdoiorg101111puar12555Centreforpublicimpact(nd)RetrievedfromhttpswwwcentreforpublicimpactorgCernaL(2013)ThenatureofpolicychangeandimplementationAreviewofdifferenttheoreticalapproachesOrganisationforEconomicCo-OperationandDevelopmentRetrievedfromhttpswwwoecdorgeduceriTheNatureofPolicyChangeandImplementationpdfColoradoSprings(nd)Retrievedfromhttpscoloradospringsgovcommunicationspagewhat-expect-when-you-speakupContinuumofcare(coc)program(nd)RetrievedfromhttpswwwhudexchangeinfoprogramscocCouncilpresidentkicksoff$12millionyouthfund(nd)Retrievedfromhttpwwwbaltimorecitycouncilcomcontentcouncil-president-kicks-12-million-youth-fundDurlakJAampDuPreEP(2008)ImplementationmattersAreviewofresearchontheinfluenceofimplementationonprogramoutcomesandthefactorsaffectingimplementationAmericanJournalofCommunityPsychology41327-350Retrievedfromhttpswwwncbinlmnihgovpubmed18322790Fishbonediagram(ishikawa)(nd)Retrievedfromhttpasqorglearn-about-qualitycause-analysis-toolsoverviewfishbonehtmlFivewhysGettingtotherootofaproblemquickly(nd)RetrievedfromhttpswwwmindtoolscompagesarticlenewTMC_5Whtm
References
63
FligsteinNampMcAdamD(2011)TowardageneraltheoryofstrategicactionfieldsSociologicalTheory29(1)1-26RetrievedfromhttpsociologyberkeleyedusitesdefaultfilesfacultyfligsteinFligstein20McAdam20Soc20Theory20PaperpdHeinkelP(2018January12)CitymullspermitsforbikesharingcompaniesRetrievedfromhttpwwwdentonrccomnewsdenton20180112city-mulls-permits-bike-sharing-companiesHowisareamedianincomecalculated(2012October)Retrievedfromhttpswwwhudexchangeinfofaqs843how-is-area-median-income-calculatedLampkinLMampHatryHP(2003)KeystepsinoutcomemanagementUrbanInstituteRetrievedfromhttpswwwurbanorgresearchpublicationkey-steps-outcome-managementMarshDampSmithM(2000)UnderstandingpolicynetworksTowardsadialecticalapproachPoliticalStudies484-21Retrievedfromhttpjournalssagepubcomdoiabs1011111467-924800247McCannL(nd)RetrievedfromhttplaurenellencomMoultonSampSandfortJR(2016)ThestrategicactionfieldframeworkforpolicyimplementationresearchPolicyStudiesJournal45(1)Retrievedfromhttponlinelibrarywileycomstore101111psj12147assetpsj12147pdfv=1ampt=jdkhhcp4amps=e184c625e1855a6b1615ccb99ee27b1c0b3212d8PaynterB(2018January1)What100citiesarelearningfromeachotherbysharingtheirdataRetrievedfromhttpswwwfastcompanycom40518004what-100-cities-are-learning-from-each-other-by-sharing-their-dataPeakacademy(nd)Retrievedfromhttpswwwdenvergovorgcontentdenvergovenmayors-officeprograms-initiativespeak-performancepeak-academyhtmlPinnockHBarwickMCarpenterCREldridgeSGrandesGGriffithsCJRycroft-MaloneJMeissnerPMurrayEPatelASheikhAampTaylorSJC(2017)Standardsforreportingimplementationstudies(stari)ExplanationandelaborationdocumentBMJOpenRetrievedfromhttpbmjopenbmjcomcontentbmjopen74e013318fullpdfProctorESilmereHRaghavanRHovmandPAaronsGBungerAGriffeyRampHensleyM(2011)OutcomesforimplementationresearchConceptualdistinctionsmeasurementchallengesandresearchagendaAdministrationandPolicyinMentalHealth38(2)65-76doi101007s10488-010-0319-7TandonSD(2018)ProgressincommunityhealthpartnershipsResearcheducationandactionRetrievedfromhttpswwwpressjhuedujournalsprogress-community-health-partnerships-research-education-and-actionProvanKGampKenisP(2007)ModesofnetworkgovernanceStructuremanagementandeffectivenessJournalofPublicAdministrationResearchandTheory
References
64
18(2)229-252Retrievedfromhttpsacademicoupcomjpartarticle182229935895SabatierPA(1988)Anadvocacycoalitionframeworkofpolicychangeandtheroleofpolicy-orientedlearningthereinPolicySciences21(3)129-168Retrievedfromhttpwwwjstororgstable4532139SandstromAampCarlssonL(2008)TheperformanceofpolicynetworksTherelationbetweennetworkstructureandnetworkperformancePolicyStudiesJournal36497-524Retrievedfromhttponlinelibrarywileycomdoi101111j1541-0072200800281xabstractSchwartzMampWilsonE(nd)WhocanaffordtoliveinahomeAlookatdatafromthe2006americancommunitysurveyRetrievedfromhttpswwwcensusgovhousingcensuspublicationswho-can-affordpdfShearerJCAbelsonAKouyateBLavisJNampWaltG(2016)WhydopolicieschangeInstitutionsinterestsideasandnetworksinthreecasesofpolicyreformHealthPolicyandPlanning31(9)1200-1211Retrievedfromhttpsdoiorg101093heapolczw052ThelessonsofadministrativedataHigh-profilepolicy-relevantresearchpoweredbyadministrativedata(nd)Retrievedfromhttpswwwpovertyactionlaborgsitesdefaultfilesresources20161114-The-Lessons-of-Administrative-DatapdfToolsandotherneedsforqualitativeandquantitativemethods(2015February)AgencyforHealthcareResearchandQualityRetrievedfromhttpwwwahrqgovprofessionalsquality-patient-safetytalkingqualityassesstoolshtml
WeaverRCampBagchi-SenS(2013)SpatialanalysisofurbandeclineThegeographyofblightAppliedGeography4061-70
References
65
UniversityPartners
Name Title Affiliation Area(s)ofexpertise
Contactinformation
Examplesofworkingwithnon-academicdecisionmakers
WendyPBarreno
AssociateDirector
WilsonSheehanLabforEconomicOpportunitiesUniversityofNotreDame
LEOsfocusareasincludehealtheducationhousingchildwelfareeconomicopportunitycriminaljustice
wbarrenonduduorleondedu
MelissaDavey-RothwellPhD
AssociateScientist
JohnsHopkinsBloombergSchoolofPublicHealth
Health mdavey1jhuedu410-614-5854
BaltimoreCityHealthDepartmentLeadersMarylandDepartmentofHealthAddictionRecoveryServicesinDCExecutiveDirectorsofcommunityorganizationsacrosstheUSthatfocusonHIVSTIsubstanceabuseandimplementationscience
ElizabethStuart
AssociateDeanforEducationandProfessorDepartmentsofMentalHealthBiostatisticsandHealthPolicyand
JohnsHopkinsBloombergSchoolofPublicHealth
Educationsocialpolicystatisticalmethods
estuartjhuedu
Resources
66
Management
Design-ThinkingResources
EngagementGuideforCivicTech
Journeymap
PerformanceManagementResources
DenverPeakAcademy
Proxymeasuresguide
Talkingaboutperformancewithoutcomplicatingit(video)
SettingPerformanceTargetsguide
Datavisualizationguide
Performancemanagementgettingstartedguide
Exercises
Fishboneexercise
5Whys
AdministrativeData
TheLessonsofAdministrativeData(JPAL)
EvaluationResources
Quantitativeandqualitativemethods(USDepartmentofHealthandHumanServicesAgencyforHealthcareResearchandQuality)
KeyStepsinOutcomeManagement(UrbanInstitute)
RCT-YES
Sageresearchmethods
BestPracticesforMixedMethodsResearchintheHealthSciences
ACCORDChecklist
Pilotandimplementationplanning
Benchmarkingexamplesandproxymeasures
Seattleopendataplaybook
Resources
67
Datasciencescopingguide
Resources
68
AboutGovExGovExhelpsgovernmentsusedataandevidencetoimproveresidentsrsquolivesWesupportgovernmentsinbuildingtheirinternalcapacitytooperatesustainableanalyticspracticeswhichimprovecommunityoutcomesinfieldsfrompublicsafetytoeconomicdevelopmentGovExhasaproventrackrecordofworkinginconcertwithpartnerstoprovidetechnicalassistanceandtrainingatscale
SinceourlaunchinApril2015GovExhas
Trainedover2000governmentemployeesinpersonandonlineintopicsincludingdatamanagementperformancemanagementopendatacommunityengagementandanalytics
Providedtechnicalassistancetomorethan100localgovernmentsacrosstheglobeimprovingtheiruseofdatatoachieveefficienciesinservicedeliveryandbetteroutcomesforresidents
Assessedthecapacityofmorethan130governmentalorganizationstodeterminehowleadersstaffandexternalstakeholdersmanageandusedata
Localgovernmentsareincreasinglyonthefrontlinesofaddressinglargescaleproblemswithglobalimplicationssuchasclimatechangepublicsafetyandeconomicinequality
Whileevidence-basedinterventionstohelpsolvethoseproblemsareavailablelocalgovernmentstaffhavelimitedtimeandcapacitytofindandimplementresearch-basedsolutionsAsaresultcitiescontinuetoworkthewaytheyalwayshave--makingincrementalimprovementstoprogramsbutnotalwayscreatingchangeatasystem-level
ThroughresourcedevelopmentandcollaborationwithJHUfacultyGovExissupportingcityleadersinusingdataandevidencetomakedecisionsandcreatinglastingrelationshipswithresearchers
TheRoadmapwasdevelopedbyGovExaspartofitspartnershipwithBloombergPhilanthropiesWhatWorksCitiesinitiative
AboutGovEx
69
- Title page
- Introduction and purpose of the Roadmap
- Using the Roadmap
-
- Gather information about the problem
- Strategically engage key partners
- Review the evidence and collect details
- Decide on a policy to implement based on local factors
- Establish a framework to monitor progress
- Assess data accessibility quality and security
- Design implementation and evaluation
- Share your story and success
-
- Working with a range of evidence
- Thank you
- References
- Resources
- About GovEx
-
SpecifythetargetpopulationAreyoudesigninganinterventionthattargetsonlycertainsitesAreyouplanningonimplementingaprogramorinitiativeforaspecifictargetaudienceClearlyspecifythepopulationyouplantotarget
IfyouplantoconductarigorousimpactevaluationasnotedbytheWilsonSheehanLabforEconomicOpportunitiesitisimportanttoestablishconcreteobjectiveeligibilitycriteriaandensurethatthesecriteriaareappliedconsistentlybyallstaffduringscreeningandintakeItisalsovitaltoidentifytheindividualswhoareeligibleforaparticularprogramorserviceevenifeveryoneeligiblecannotbeservedbyexistingresourcesThishelpscitiesidentifyanyunmetneedsandguidefutureresourceallocationwhilealsocreatingthepossibilityofarigorousimpactevaluationbyidentifyingapotentialcomparisongroupFinallywhenconductingscreeningandintakeitisimperativetocollectandmaintainbasicinformation(egnamedateofbirthgender)aboutallprogram-eligibleindividualsnotjustthosewhoparticipateintheprogram
ExampleIntheShapeUpSomervilleexampletheprogrambeganbytargeting1stthrough3rdgraders
Yourturn
____________________________________________________________________________
____________________________________________________________________________
DefinethelocationWherewillthepolicychangetakeplaceWillimplementationtakeplaceatonesiteormultiplesites
ExampleShapeUpSomervilleinterventionsweredeliveredinschools
Yourturn
____________________________________________________________________________
____________________________________________________________________________
ClarifyimplementationpartnersandarticulatetheirrolesWhatarethekeyorganizationsinstitutionsorpeopleresponsibleforimplementationTheprojectteamshouldbeasclearandspecificaspossibleinprovidingadescriptionoftheconcreteactivitiesthatimplementationpartnersaremeanttocarryoutPracticeprofilesmaybeahelpfultooltohelpdefinewhatisrequiredoftheactorswhoareresponsibleforimplementingtheintervention
Designimplementationandevaluation
45
ExampleInSomervillemayoralleadershipandengagementofmultiplecityagencies(publichealthprimarycareschoolsCambridgeHealthAllianceetc)streamlinedandconnectedthevariousShapeUpSomervilleinitiativesandgrantswhichhelpedtokeepthetalentandenergyfocusedonhealthyeatingandactivelivingforallsectorsofSomervilleSUSpartnersultimatelyincludedschoolsnonprofitsCityandstateagenciesandnationalfoundations
Yourturn
____________________________________________________________________________
____________________________________________________________________________
DescribetheinterventionWhatarethestepsactivitiesorprocessesRememberthatthemechanismthespecificstepsorprocessthatleadtobehaviorchangeiskeytounderstandingtheeffectivenessofthepolicychangeandsharingtheresultswithothers
ExampleGoalsofShapeUpSomervilleweretoinfluenceeverypartofanearlyelementaryschoolchildrsquosdayndashbeforeduringandafter-schooltopromotehealthyeatingphysicalactivityandtheirweightstatusInterventionincludedfoodserviceenhancementstoimprovebreakfastandlunchandstaffprofessionaldevelopmentwalktoschoolactivitiesSUSclassroomcurriculumandprofessionaldevelopmentschoolwellnesspolicydevelopmentSUSafter-schoolcurriculumandprofessionaldevelopmentoutreachandeducationtohomethroughmaterialsforumseventsldquoSUSapprovedrdquorestaurantscommunityoutreachandcapacitybuildingthroughpolicydevelopmenttrainingsmediaplacements
Yourturn
____________________________________________________________________________
____________________________________________________________________________
ClarifytimelineandmilestonesfordeliveryandevaluationHowfrequentlywilltheservicesinterventionbedeliveredHowlongwilltheprogramrunWhatisthetimetoexpectedimpacts(eg12monthsafterprogramcompletion)
Designimplementationandevaluation
46
ExampleShapeUpSomervilleledbyTuftsUniversityFriedmanSchoolofNutritionScienceandPolicyconductedaquasi-experimental(nonrandomized)interventionstudyamong1stthrough3rdgradestudentsinSomerville(intervention)andtwocomparisoncommunitiesfor9-monthsduringoneschoolyear(fall2002ndashspring2003)ThestudyshowedpositiveassociationoftheinterventiononchangeinBMISUSreleasedreportssummarizingevaluationsprogresstodateplansforadditionalprojectsandimpactstenandfifteenyearsintotheinitiative
Yourturn
____________________________________________________________________________
____________________________________________________________________________
PlanyouranalysisTheprevioussectionaddressedestablishinganaccountabilityframeworktomonitorprogressandimplementationDeterminewhichindicatorswillbediscussthroughthatprocesswhowillsharethedataandanalysisandhowfeedbackfromthoseconversationswillbeincorporatedbytheprojectteam
ExampleTheMayorrsquosofficeimplementedadata-drivendecisionmakingapproachcalledSomerStatwhichwassynergisticwiththeapproachestakenbytheresearchandevaluationteamsdeployedforthespecificgrants
Yourturn
____________________________________________________________________________
____________________________________________________________________________
PlanforsustainabilityIfthepolicyandprogramchangeissuccessfulhowwilltheprojectteamsustainandexpandtoadditionalsitesorpopulations
ExampleAlthoughmostofthepartnershipsforgedintheearlyyearsweremaintainednewgrantsandcollaborationsexpandedattentiontoolderpublicschoolstudentsactivetransportationandtheGreenLinesubwayextensionthebuiltenvironmentfarmerrsquosmarketeffortssuchasldquoGrowninSomervillerdquowithUnionMainStreeteconomicdevelopmentandhealthdisparitiesaswellasdatacollectionmonitoringandsurveillance
Yourturn
Designimplementationandevaluation
47
____________________________________________________________________________
____________________________________________________________________________
Pilotplanningtemplate
IfyouplantoconductapilotfirstdeterminethepurposeofthepilotDoyouwanttoobservewhetherthepolicychangeresultsinthedesiredbehaviorchangeAreyouconcernedaboutfeasibilityscalabilityorcostDoyouwanttotestwhetheryourdesiredresultscanbeachievedthroughthispolicyapproachClarifythegoalofthepilotandthequestionsyouhopetoanswerThiswillhelpyouidentifythekeyindicatorsthatyouwillusetomeasurewhetherthepilotissuccessfulBeforebeginningestablishabaselineofcurrentperformanceandsetyourtargetsaccordinglybasedonwhatyouhopetoaccomplishthroughthepilot
NextestablishthepilotplanforyourprojectteamincludingmilestonesforteamcheckinstodeterminewhethertomoveaheadorchangecourseThisisseparatefromthemeetingsthatwilloccuraspartoftheaccountabilityframeworkUsethespacebelowtocreateyourtimelineanddevelopyourmeetingschedule
Task Personresponsible Startdate Checkindate Deliverable
Planyourevaluation
WhileyouaredesigningtheimplementationitisalsotimetobegintodevelopyourplanforanalysisAsnotedbytheWilsonSheehanLabforEconomicOpportunitiesdecisionsaboutprogramsetup-includingoutreachscreeningintakedatatrackingandenrollment-candeterminewhetherornotarigorousimpactevaluationwillbepossibleAsdiscussedcityleadersmayfindithelpfultoincludeevaluationpartnersinearlyconversationsaboutprogramsetupDuringpreviousstagesyoumayhaveworkedwitharesearchertoplanyourproposedpolicychangeYoushouldbeclearaboutthepurposeoftheevaluationanddevelopanevaluationplanBelowareadditionalconsiderations
Causationvstrendanalysis
IntheEstablishaframeworktomonitorprogressstagetherewasguidanceontrackingtrendsovertimeItisimportanttonoteherethatmonitoringdataisdifferentfromanalyzingacausaleffectTrendanalysiswillillustratethatchangesareoccurringandmayleadtoquestionsthatcouldbeansweredbyyourevaluationbuttrendanalysisalonedoesnotprovideevidenceaboutwhatisdrivingthosechangesYoumayseechangesinyour
Designimplementationandevaluation
48
outcomeindicatorsbasedonvariablesoutsideoftheCityscontrolForinstanceachangeinfederaltradepolicycanhaveanimpactemploymentratesinyourCityoutsideofyourCitysnewjobtrainingandplacementprogram
ToaddresscausalityyoumustcomparetheoutcomesofindividualswhoreceivedtheinterventionwiththosewhodidnotWithoutacomparisongroupofsimilarindividualswhowerenotexposedtotheinterventionitisimpossibletoknowwhichfactorsareresponsiblefortheoutcomesRandomizedControlTrials(RCTs)inwhichindividualsarerandomlyassignedtotreatmentandcontrolgroupsarethegoldstandardinprovingcausalityIfyoucandemonstratethatoutcomesimprovedforthosewhoreceivedtheinterventionanddidnotimproveatallorimprovedlessforacomparisongroupyoucanbesomewhatcertainthattheinterventionwasthecause
Examplesofothernon-causalimpactevaluationsthatarerigorousandreliableincludecertainquasi-experimentalstudiesusingcomparisongroupdesignssmartinterruptedtimeseriesanalysesthatmodeltrendsovertimeinwaysthatallowestimationofinterventioneffectswelldesignedpre-posttestanalysesneedsassessmentsimplementationstudiesetcFormoreinformationondifferenttypesofevaluationsseetheSageResearchMethodswebsite
Lookfordifferencesingroups
BuildintimetoalsoexaminethepilotdatatoseeifgroupsareexperiencingdifferentresultsReviewingperformancebasedonoutcomemeasuresforapopulationinaggregatemayhidethesubtlewaysthatsomepeoplemaybeaffecteddifferentlysoitisnecessarytocarefullyanalyzethedatafordisparateoutcomesForinstancetheMovingtoOpportunity(MTO)studythatexaminedtheimpactofhousingvouchersonfamilieswholivedinhigh-povertyneighborhoodsfoundthatchildrenwhomovedbeforetheageof13hadbettereducationandhigherincomesasadultsHoweveradultsdidnotseeimprovededucationandemploymentoutcomes
Disaggregatingdataintomorespecificcategoriesofraceandethnicitycanhelpunderstandtrendsanddisparitiesinamoregranularwaywhichcanhelpdecisionmakerstargetpoliciesandinterventionsinamoreequitableway
EarlierintheRoadmapweworkedtoidentifystakeholdersandwhomightbepositivelyandnegativelyimpactedbyachangeinpolicyUsetheresultsofthatexercisetoinvolvethecommunityininterpretingdataanalysisandtogatherqualitativefeedbackfromthecommunitytocallattentiontoissuesForinstanceanincreaseinyouthdetentionmaybeexplainedbyinefficienciesorlackofaccesstotransportation
Designimplementationandevaluation
49
Designimplementationandevaluation
50
Shareyourstoryandsuccess
Sharetheresults
Storytelling
AnintegralstepinyourstrategyissharingtheresultsofyourpilotorpolicychangewithyourcommunityThisincludesdevelopingmessagingaroundtheworkofyourorganizationandcreatingaplantodisseminateyourstoriesWhytellstoriesaboutyourwork
Informandorunderstandasubjectorsituation
Giveneededcontextofdataorbitsofinformation
Relatablestoriesarememorableforreadersviewers
ReadersorviewerswillseetheinformationasmorecompellingwhenthereisademonstratedandrelatablechangeorimpactonthemselvesortheirfriendsneighborsandcolleaguesStoriesareaboutpeopleandgovernmentisfundamentallyabouthelpingpeopleSotellthestoriesofthepeopleyouandyourcitycolleagueshelped
Keepthesetipsinmindasyoudevelopyourstorywithyourteam
WhatwereyourfindingsandwhyshouldanyonecareTellyouraudiencehowtheproblemwassolvedWhathastheimpactbeenonresidentsandcitystafftodateWhatwasthesuccessandweretheremultiplesuccessesalongthewayIfyouhavepreliminaryresultsordatashowingimprovementtosharethisisthetimetodoit
WhowasinvolvedandimpactedReviewthelistofstakeholdersHowwereeachofthemimpactedbytheprogramHumanizeyourstoryandimpactbyaddingadditionalvoicesThesevoicescanbeoneofthefollowing
Apersonorfamilywhowaspositivelyaffectedbythechangesuccess
Aninvolvedthird-partydescribingboththeproblemandthesolution(ieanorganizationnonprofitprovidersbusinesstradeassociationsportsteametc)whoispartofthecommunitythatthegovernmentworkedwithonthechangesuccess
Someonefromthegovernmentitselfwhocanspeakaboutthechangesuccess
TellmoreaboutyourcityDescribeyourcityincludingpopulationdemographicsindustriesetcWhatareitschallengesWhatisitknownforWhatmakesituniqueHowisitsimilarordifferentfromothercitiesintheUS
Shareyourstoryandsuccess
51
DescribeyourcityrsquoschallengeWriteoutthechallengeyourcityisfacing(asdefinedinyourproblemstatement)HowdoesitaffectresidentsHowdoesthisproblemaffecttheworkofcitystaffmembers
AddthedetailsProvidedetailsonyourappliedresearchstrategyTalkaboutwhatyourcityhasdonenthepasttoresolvetheissueDiscusshowyourstrategyhaschangedwhenyoubegantoadoptthisapproachWasmoneysavedorinvested
Usethisstorymapexercisetoorganizeyourthoughtsandguaranteethatyouincludealloftheelementsofatraditionalstory
AfteryoudevelopyourmessagingcreateacommunicationsmarketingandoutreachstrategydesignedtospreadyourmessageYourstrategyshouldwalkyouthroughthefollowingkeyactivities
1 IdentifyandprofileyouraudienceWhencreatinganddisseminatingyourmessageitisalwaysgoodtothinkaboutyouraudienceWhoexactlywouldbeinterestedinwhatyouhavetosayThegeneralpopulationofthecityPeoplewhoworkinnonprofitsBusinessownersItmattersbecausenoteveryonewholivesinacitymaybethetargetoftheinformationBusinessownersmaybemoreinterestedindatashowingeffectsonthebusinessdistrictthanothersandsoonWewillcoverthislaterbutthisstepwillalsohelpyoutothinkofthemosteffectivewaytoreachyouraudience
2 Defineyourcalltoaction(ifyouhaveone)SpendsometimeconsideringwhyyouareinterestedintellingyourstoryDoyousimplywanttoupdatethepubliconyourprogressorcelebrateyoursuccessesAreyouinterestedinyouraudiencechangingitsbehaviororsupportingaspecificinitiativebeinglaunchedbyyourorganizationIfyouwanttohaveyouraudienceactorbehaveinacertainwaybespecificandincorporateitinyourmessaging
Shareyourstoryandsuccess
52
3 SelectperformancemeasuresThinkabouthowyouwoulddetermineifyourcampaignissuccessfulandselectappropriateperformancemeasuresYouwillrevisitthesemeasuresattheendofyourcampaigncommunicationsplanimplementationtodeterminewhetheryouneedtomodifyyourcommunicationsandoutreachstrategyforthefuture
4 MethodstocommunicateyourstoryTherearemanywaystocommunicatesuccessofyourprojectHerearesomesuggestionsonreachingaudiencesItisimportanttodeterminewhichmethodsarethemosteffectiveforyourdesiredaudiencetoreceiveinformationDoyourresearch(PewResearchCenterhasexcellentstudiesonthereachofmediadigitalmediaandmarketing)ForexampleifseniorsareyourtargetpopulationandtheyarelessinclinedtouseInstagram(thatistrue)youmaywanttoconsiderFacebookorthenightlylocalnewsbothofwhichareusedmoreoftenbyseniorsBelowisalistofsomemethodsandtoolsyoucanuse
i Pressreleasestolocalnewspapersandtvstations
ii ShortsmartphonevideosAgreatwaytoengageanaudienceisthroughvideoAskonequestionofsomeonewhoispartofthestoryMaybethereisasocialworkerwhocandiscusshungerinyourcityThesevideoscanbereusedonmultipleplatforms
iii Long-formwriting
iv TellingindividualstoriesthroughaseriesofphotosandshortnarrativesforpublicationonFacebookorInstagram
v Op-edsopinionpiecesplacedinthelocalpaper
vi Emailupdatestoyourlist
vii Connectingwithappropriateorganizationsthatareinvestedinthetopicandworkwiththecommunityandaskthemtodisseminateinformationtotheirmembers
viii EmployingldquoWordofMouthrdquo(WOM)tacticsbycreatingtownhallsorfreeeventsforresidentstoattendintheirownneighborhoods
5 DeterminethetypeofcollateralUsuallycampaignsthattargetresidentsrequireseveralavenuestoreachthemPeoplearenotalwaysinthesameplaceorspaceHavingakeenunderstandingofyouraudiencewillassistyouwithyourchoicesonwhatneedstobedevelopedSometimesthatmeansimagesforsocialmediaaonepageflierthatgivespeopleinformationonattendinganeventoranin-depthbrochure
Shareyourstoryandsuccess
53
6 ImplementyourstrategyKeepinmindstrategiescanbefluidSometimessomethingmaynotturnoutasexpectedevenifduringplanningsessionsitwasldquosupposedtordquoItisokYoucananalyzewhathappenedduringadebriefsessionandonceyoursurveysandothermeasurementshavebeenfinalized
Visualizations
Withtherightvisualizationyoucanreviewandshareinformationinsightsanalysisperformanceandtrendsallofwhichisvaluabletoacross-sectionofthepublicButwhichvisualizationisrightforyouTochoosetherighttypeofchartgraphdeterminethetypeofquestionyouwanttoanswerSomecommoncharttypesarebelowandmoreinformationcanbefoundhere
Exploreadditionalanalyticaltechniques
Duringthegoal-settingphaseyouestablishedoutputoutcomeandimplementationmeasuresBasedontheresultsofyourpilotyoumayneedtoadjustyourmeasuresWiththepilotdatainhandyoumayhavesomeideasofhowfutureanalysescouldbedifferentandpotentiallyimprovethequalityofyourprogramTakesometimetobrainstormotherrelevantdatasetsthatyoudidnothavethetimetopursueearlierincludingmessydepartmentaldatathatneedsmoreattentionorsourcesofpublicdatafromthestateandfederalgovernmentnonprofitsandevenprivatecompaniesthatcouldsupplementyourunderstandingofanissueUsingmachinelearningtechniquesyoumightbeabletobetter
Shareyourstoryandsuccess
54
predictdemandforhomelessservicesusingNOAAweatherforecastspreemptwatermainbreaksusingsoilstratigraphyorinvestineconomicdevelopmentbymonitoringrealestatelistingdata
Advancedanalyticaltechniquescanalsohelpyourimplementationrunmoreeffectivelybyintegratingdata-informedtargetinganddecisiontoolsintotheprocessByautomatingroutineworkflowsandsurfacingsubtletrendsthesetoolscanhelpyouiterateinamoreprecisemannerandprovidekeystrategicguidancetofront-lineemployeesthatwouldotherwisebeunavailableThesesortsofdata-science-enhancedprocessescanbeclassifiedasoneofsixprojecttypesasexploredfurtherbytheNewOrleansLANOLAlyticsteamFormoreguidanceonscopingasimilarprojecttheGovExadvancedanalyticsscopingguidegivesyouastep-by-stepoverviewofhowtobesuccessful
Usethespacebelowtolistyourideasforpossibleadvancedanalyticsprojectsbytype
Typeofproject Ideas
Findingtheneedleinthehaystack
Prioritizingforimpact
Earlywarningtools
Betterquickerdecisions
Optimizingresourceallocation
Experimentingforwhatworks
Keepthefeedbackloopgoing
GatherinformationonpublicsentimentaboutpolicychangethroughfocusgroupssurveysandpublicmeetingsledbyadvisorygroupmembersNotewhethernewinformationisavailablethatcanhelpanalyzeorcontextualizeyourprogress(egfederalorstatedatareports)Sharetheresultsofyourpilotandthepolicychangeprocesswithothercitiesandresearchers
Shareyourstoryandsuccess
55
WorkingwitharangeofevidencePresentlythereisalotofattentionandinvestmentinevidence-basedpolicymakingincludingthefederalbipartisanCommissiononEvidence-BasedPolicymakingthePewCharitableTrustsreportsandtoolkitsonHowStatesEngageinEvidence-BasedPolicymakingandtheLauraandJohnArnoldFoundationsupportedEvidence-BasedPolicymakingCollaborativeManyoftheseinitiativesfocusonrigorousevaluation--meaningrandomizedcontrolledtrials(RCT)orotherexperimentalmodels--asthebestkindofevidencetouseHoweverthisbodyofevidenceisofteninsufficientforthespecifickindsofquestionsthatgovernmentscanhavewhenmakingadecisionaboutpolicyInthisguideweproposeanapproachtosupportgovernmentsinincorporatingadditionaltypesofevidenceintotheirdecisionmaking
OnewaytounderstandarangeofevidenceistouseNestarsquosStandardsofEvidencetoframeanddefineevidenceTheBritishnonprofitNestaworkstoinjectpositivechangeintothewayUKgovernmentpoliciesandprogramsfunctionOneoftheirkeyareasofworkhasbeendevelopingWhatWorksCentresorganizationsthatcollectcommissionsynthesizeanddisseminateevidenceaboutinterventioneffectivenessacrossarangeofsocialimpactareasNestadevelopedthefollowingStandardsofEvidencetoprovideausefulwaytodocumenthowdifferentkindsofinformationdemonstratedifferentlevelsofevidence
Workingwitharangeofevidence
56
Theimportanceofpotentiallybiasedinformation
Onechallengepresentedbyanevidenceclassificationthatacknowledgesthevalueofnon-scientificformsofevidenceistofigureouthowtobalanceallofthepiecestogetherwhenyouusethemTounderstandhowtodothiswehavetothinkabouthowtopositionpotentiallybiasedinformationindecisionmaking
WhenwelookatinformationwhichhasnotbeenrigorouslyevaluatedweacceptthatthisinformationmightbebiasedThewholereasonweconductrigorousevaluationistoremovebiasfromourobservationsWeconductstructuredtrialstoproduceresultsthatareconsistentandgeneralizableIfafindingisunbiasedthisaccuracymdashthislackofbiasmdashcanbeprovedbecauseanyindependenttestercanreproduceitwhentheyfollowanidenticalprocess
YoucanlookatpotentiallybiasedinformationasjustinformationthatisearlieroninitspathtorefinementreplicationandverifiedaccuracyApieceofinformationbasedonanyoneindividualrsquosobservationsislikelytobebiasedbecauseitiscomingfromjustonesourceBiasisreducedbygatheringtogetherlotsofindependentsourcesofinformationandfiguringoutitsmostcommonattributes
MaybethebiasedinformationmakesaclaimthatistoolargemdashitsayssomethingistrueunderallconditionswhenitisreallyonlytrueundercertainconditionsorthataneffectofaninterventionisbiggerthanitstrueaverageorthataneffectlastsovertimewhenitdoesnotPerhapschangingtheclaimtobemorespecificmdashcriticallyreviewingtheinformationtoseeifspecificconditionsmustbemettoachievetheoutcomesmdashwouldmaketheclaimmoreaccurate
IfyoutestedyourclaimfounditdidnotholdupinsomecaseschangedittesteditagainuntilyourclaimrepeatedlyheldupthenyoucoulddemonstratethattheclaimwasnolongerbiasedbutaccurateThisprocessoftestingimprovingtestingandvalidatingiswhatwegetfromstructuredscientificstudy
Withindividualpiecesofinformation--crediblehypothesescasestudiesexpertopinion--youcannotassumethatkindofaccuracyAtthesametimetheseearlyobservationsarefoundationaltoanyreplicablescientificobservationsWewillnevergetanywhereifwedonotstartwiththesepiecesofinformationeveniftheymightbebiased
Combininglevelsofevidencetogetthemostdetailedinformationforyourcity
ItisespeciallyimportanttoacceptdifferentformsofevidencewhenyoufacequestionsthathavenotyetbeentestedbyindependentstudiesSomeofthemostimportantquestionsrelatetothingsthatareveryspecifictoyoursituationYoufaceaparticularcombinationof
Workingwitharangeofevidence
57
issuesactorstrendsandresourcesthatisultimatelydifferentfromeveryotherplaceintheworld
ThisisimportantbecauseaspolicypractitionersandnotscientistsyoucaremoreaboutyourspecificcasethanahypotheticalaveragegeneralcaseYoucarelessaboutthetotallikelyrangeofimpactsfromapolicythanhowthissingleimplementationwillworkYouwantinformationaboutpoliciesthatwillprovideyouwiththebestpossiblepredictionabouthowtheywillworkinyourcontext
ByputtingdifferentkindsofevidenceintothesameframeworkNestarsquosStandardsofEvidenceimplythatwhenyoucannotgetrigorouslytestedexperimentallyverifiedinformationitisappropriatetoworkwithwhatwedohaveThisisrelevantforthemunicipalimplementationcontextbecausenostudypulledfromaliteraturereviewwillaccountforallofthevariablesyoufaceinyourowncontext
Intheabsenceofhavingastudythatpredictshowaninterventionworksinyourspecificcontextyouwillneednon-scientificinformationtotrytofillinthosegapsYouwillneedtogatherstoriesfromresidentsstaffmembersandotherstakeholdersthatsitonLevels1or2ofthestandardsofevidence
Someofthekindsofinformationyouwillprobablygetmoreeasilyfrompeoplethanstudiesinclude
howpeopleincitieslikeyoursreactedtoaninterventioninpublicforumshowpeoplewhoimplementedaninterventioninacitylikeyoursfiguredouthowtogetitpassedhowthecitycommunicatedpubliclyabouttheapparentbenefitsanddrawbacksoftheinterventionthepoliticalacceptabilityoftheintervention
Youcanremainsimultaneouslyawarethattheinformationyouaregatheringthroughindividualstoriesisbiased--notgeneralizableinformedbyspecificperspectives--whilealsorecognizingthathavingsomeinformationaboutsomethingimportantisbetterthannoinformationEvidencefromthefirsttwoevidencelevelsareessentialbuildingblocksofknowledgeandperfectlyusefulwhenyoucannotgetdetailedunbiasedknowledge
Youhavetostartsomewhereandyoushouldalwaysstartwithevidenceyouhaveaboutyourveryspecificconditionsthatyouknowareimportantforsuccess
Workingwitharangeofevidence
58
Thankyou
Interviewswithcities
GovExheldsemi-structuredinterviewswithsixstaffmembersinfourcitiestoestablishtheinitialstepsoftheRoadmapWeaskedrespondentstotalkaboutaninitiativeorpolicythattheyhavebeenresponsibleforimplementingintheircityRespondentsfromthreecitiesspokeabouttheireffortstoaddresshomelessnessandrespondentsfromonecityspokeaboutblightAfterrespondentsidentifiedtheproblemtheyspokeabouthowthecityprioritizedthisissuethevariousstepsinvolvedandbarrierspresent
GovExconductedtextanalysisoftheseinterviewswhichrevealedthatpeoplewasthemostcommonlyusedtermduringtheseconversationswhendiscussingthestepstochangeaprogramorpolicyThisisathemethroughoutthisRoadmapThefirstsectionbeginswithidentifyingtherelevantpeopleTheconversationsaroundevidencearefocusedonhumanbehaviorratherthanmethodologyorstrengthofevidence
Thankyou
59
OncewehadadraftoftheRoadmapwefollowedupwithcitiesonanindividualbasistolearnmoreaboutwhatwouldbehelpfulWearegratefultothefollowingcitystafffortheinterviewsandindividualconversations
MichaelBaskinChiefPolicyOfficerChattanoogaTN
MelissaBridgesPerformanceandInnovationCoordinatorLittleRockAR
VictoriaCarreoacutenAdministrativeOfficerLasVegasNV
TyroneGrandstrandCommunityDevelopmentPlannerFargoND
TibLaughlinDirectorofGeneralServicesKansasCityKS
TimMorelandDirectorofPerformanceManagementandOpenDataChattanoogaTN
CharlesRosendahlManagementAnalystDentonTX
Thankyou
60
MelissaSiebenAssistantCountyAdministratorKansasCityKS
GinaThompsonDirectorofStrategicServicesLewisvilleTX
JHUFacultycollaboration
GovExalsoconvenedthecitycontactsandfacultypartnersinaseriesofAdvisoryGroupmeetingsThesemeetingsservedtoconnectcitiesandfacultytounderstandoneanothersperspective
StefanBaralBloombergSchoolofPublicHealth
AshleyBernerJHUSchoolofEducation
MelissaDavey-RothwellBloombergSchoolofPublicHealth
JonLinksBloombergSchoolofPublicHealth
TamarMendelsonBloombergSchoolofPublicHealth
KristinMmariBloombergSchoolofPublicHealth
KeshiaPollackPorterBloombergSchoolofPublicHealth
JoshSharfsteinBloombergSchoolofPublicHealth
LizStuartBloombergSchoolofPublicHealth
LindsayThompsonCareyBusinessSchool
DanielWebsterBloombergSchoolofPublicHealth
Partnersindataandevidence-basedpolicy
Wealsospokewithpartnerswhoareexpertsinthefieldofevidence-basedpolicywhosharedtheirknowledgeandresourcesofcurrentandfuturethinkinginevidence-informedthinking
WendyBarrenoAssociateDirectorWilsonSheehanLabforEconomicOpportunitiesUniversityofNotreDame
TiffanyDavenportDirectorofDataPartnershipsBaltimoresPromise
TomiHiersDirectorBaltimoreCivicSiteAnnieECaseyFoundation
ChrisKingsleySeniorAssociateAnnieECaseyFoundation
KathrynOliverAssociateProfessorLondonSchoolofHygieneandTropicalMedicine
Thankyou
61
KristenWatkinsSeniorPolicyAssociateJ-PAL
GovExParticipants
GovExAnalystMiriamMcKinneyanalyzedtheinterviewtranscriptsandconductedtextanalysisSeniorImplementationAdvisorEmilyShawwillintegrateRoadmapactivitieswithadditionalservicedeliveryactivitiesatGovEx
Thankyou
62
ReferencesAbrahamKGampHaskinsR(2017)Thepromiseofevidence-basedpolicymakingReportofthecommissiononevidence-basedpolicymakingWashingtonDCCommissiononEvidence-basedPolicymakingBigmove2Extendingthegreen(nd)Retrievedfromhttpswwwcityoflewisvillecomdoing-businesslewisville-2025big-move-2-extending-the-greenBatesMAampGlennersterR(2017)ThegeneralizabilitypuzzleRetrievedfromhttpsssirorgarticlesentrythe_generalizability_puzzleBuildwithnotfor(2018January)RetrievedfromhttpsbuildwithnotforwordpresscomCairneyPampOliverK(2017)Evidence-basedpolicymakingisnotlikeevidence-basedmedicinesohowfarshouldyougotobridgethedividebetweenevidenceandpolicyHealthResearchPolicyandSystemsdoi101186s12961-017-0192-xCairneyPOliverKampWellsteadA(2016)TobridgethedividebetweenevidenceandpolicyReduceambiguityasmuchasuncertaintyPublicAdministrationReview76(3)399-402Retrievedfromhttpdxdoiorg101111puar12555Centreforpublicimpact(nd)RetrievedfromhttpswwwcentreforpublicimpactorgCernaL(2013)ThenatureofpolicychangeandimplementationAreviewofdifferenttheoreticalapproachesOrganisationforEconomicCo-OperationandDevelopmentRetrievedfromhttpswwwoecdorgeduceriTheNatureofPolicyChangeandImplementationpdfColoradoSprings(nd)Retrievedfromhttpscoloradospringsgovcommunicationspagewhat-expect-when-you-speakupContinuumofcare(coc)program(nd)RetrievedfromhttpswwwhudexchangeinfoprogramscocCouncilpresidentkicksoff$12millionyouthfund(nd)Retrievedfromhttpwwwbaltimorecitycouncilcomcontentcouncil-president-kicks-12-million-youth-fundDurlakJAampDuPreEP(2008)ImplementationmattersAreviewofresearchontheinfluenceofimplementationonprogramoutcomesandthefactorsaffectingimplementationAmericanJournalofCommunityPsychology41327-350Retrievedfromhttpswwwncbinlmnihgovpubmed18322790Fishbonediagram(ishikawa)(nd)Retrievedfromhttpasqorglearn-about-qualitycause-analysis-toolsoverviewfishbonehtmlFivewhysGettingtotherootofaproblemquickly(nd)RetrievedfromhttpswwwmindtoolscompagesarticlenewTMC_5Whtm
References
63
FligsteinNampMcAdamD(2011)TowardageneraltheoryofstrategicactionfieldsSociologicalTheory29(1)1-26RetrievedfromhttpsociologyberkeleyedusitesdefaultfilesfacultyfligsteinFligstein20McAdam20Soc20Theory20PaperpdHeinkelP(2018January12)CitymullspermitsforbikesharingcompaniesRetrievedfromhttpwwwdentonrccomnewsdenton20180112city-mulls-permits-bike-sharing-companiesHowisareamedianincomecalculated(2012October)Retrievedfromhttpswwwhudexchangeinfofaqs843how-is-area-median-income-calculatedLampkinLMampHatryHP(2003)KeystepsinoutcomemanagementUrbanInstituteRetrievedfromhttpswwwurbanorgresearchpublicationkey-steps-outcome-managementMarshDampSmithM(2000)UnderstandingpolicynetworksTowardsadialecticalapproachPoliticalStudies484-21Retrievedfromhttpjournalssagepubcomdoiabs1011111467-924800247McCannL(nd)RetrievedfromhttplaurenellencomMoultonSampSandfortJR(2016)ThestrategicactionfieldframeworkforpolicyimplementationresearchPolicyStudiesJournal45(1)Retrievedfromhttponlinelibrarywileycomstore101111psj12147assetpsj12147pdfv=1ampt=jdkhhcp4amps=e184c625e1855a6b1615ccb99ee27b1c0b3212d8PaynterB(2018January1)What100citiesarelearningfromeachotherbysharingtheirdataRetrievedfromhttpswwwfastcompanycom40518004what-100-cities-are-learning-from-each-other-by-sharing-their-dataPeakacademy(nd)Retrievedfromhttpswwwdenvergovorgcontentdenvergovenmayors-officeprograms-initiativespeak-performancepeak-academyhtmlPinnockHBarwickMCarpenterCREldridgeSGrandesGGriffithsCJRycroft-MaloneJMeissnerPMurrayEPatelASheikhAampTaylorSJC(2017)Standardsforreportingimplementationstudies(stari)ExplanationandelaborationdocumentBMJOpenRetrievedfromhttpbmjopenbmjcomcontentbmjopen74e013318fullpdfProctorESilmereHRaghavanRHovmandPAaronsGBungerAGriffeyRampHensleyM(2011)OutcomesforimplementationresearchConceptualdistinctionsmeasurementchallengesandresearchagendaAdministrationandPolicyinMentalHealth38(2)65-76doi101007s10488-010-0319-7TandonSD(2018)ProgressincommunityhealthpartnershipsResearcheducationandactionRetrievedfromhttpswwwpressjhuedujournalsprogress-community-health-partnerships-research-education-and-actionProvanKGampKenisP(2007)ModesofnetworkgovernanceStructuremanagementandeffectivenessJournalofPublicAdministrationResearchandTheory
References
64
18(2)229-252Retrievedfromhttpsacademicoupcomjpartarticle182229935895SabatierPA(1988)Anadvocacycoalitionframeworkofpolicychangeandtheroleofpolicy-orientedlearningthereinPolicySciences21(3)129-168Retrievedfromhttpwwwjstororgstable4532139SandstromAampCarlssonL(2008)TheperformanceofpolicynetworksTherelationbetweennetworkstructureandnetworkperformancePolicyStudiesJournal36497-524Retrievedfromhttponlinelibrarywileycomdoi101111j1541-0072200800281xabstractSchwartzMampWilsonE(nd)WhocanaffordtoliveinahomeAlookatdatafromthe2006americancommunitysurveyRetrievedfromhttpswwwcensusgovhousingcensuspublicationswho-can-affordpdfShearerJCAbelsonAKouyateBLavisJNampWaltG(2016)WhydopolicieschangeInstitutionsinterestsideasandnetworksinthreecasesofpolicyreformHealthPolicyandPlanning31(9)1200-1211Retrievedfromhttpsdoiorg101093heapolczw052ThelessonsofadministrativedataHigh-profilepolicy-relevantresearchpoweredbyadministrativedata(nd)Retrievedfromhttpswwwpovertyactionlaborgsitesdefaultfilesresources20161114-The-Lessons-of-Administrative-DatapdfToolsandotherneedsforqualitativeandquantitativemethods(2015February)AgencyforHealthcareResearchandQualityRetrievedfromhttpwwwahrqgovprofessionalsquality-patient-safetytalkingqualityassesstoolshtml
WeaverRCampBagchi-SenS(2013)SpatialanalysisofurbandeclineThegeographyofblightAppliedGeography4061-70
References
65
UniversityPartners
Name Title Affiliation Area(s)ofexpertise
Contactinformation
Examplesofworkingwithnon-academicdecisionmakers
WendyPBarreno
AssociateDirector
WilsonSheehanLabforEconomicOpportunitiesUniversityofNotreDame
LEOsfocusareasincludehealtheducationhousingchildwelfareeconomicopportunitycriminaljustice
wbarrenonduduorleondedu
MelissaDavey-RothwellPhD
AssociateScientist
JohnsHopkinsBloombergSchoolofPublicHealth
Health mdavey1jhuedu410-614-5854
BaltimoreCityHealthDepartmentLeadersMarylandDepartmentofHealthAddictionRecoveryServicesinDCExecutiveDirectorsofcommunityorganizationsacrosstheUSthatfocusonHIVSTIsubstanceabuseandimplementationscience
ElizabethStuart
AssociateDeanforEducationandProfessorDepartmentsofMentalHealthBiostatisticsandHealthPolicyand
JohnsHopkinsBloombergSchoolofPublicHealth
Educationsocialpolicystatisticalmethods
estuartjhuedu
Resources
66
Management
Design-ThinkingResources
EngagementGuideforCivicTech
Journeymap
PerformanceManagementResources
DenverPeakAcademy
Proxymeasuresguide
Talkingaboutperformancewithoutcomplicatingit(video)
SettingPerformanceTargetsguide
Datavisualizationguide
Performancemanagementgettingstartedguide
Exercises
Fishboneexercise
5Whys
AdministrativeData
TheLessonsofAdministrativeData(JPAL)
EvaluationResources
Quantitativeandqualitativemethods(USDepartmentofHealthandHumanServicesAgencyforHealthcareResearchandQuality)
KeyStepsinOutcomeManagement(UrbanInstitute)
RCT-YES
Sageresearchmethods
BestPracticesforMixedMethodsResearchintheHealthSciences
ACCORDChecklist
Pilotandimplementationplanning
Benchmarkingexamplesandproxymeasures
Seattleopendataplaybook
Resources
67
Datasciencescopingguide
Resources
68
AboutGovExGovExhelpsgovernmentsusedataandevidencetoimproveresidentsrsquolivesWesupportgovernmentsinbuildingtheirinternalcapacitytooperatesustainableanalyticspracticeswhichimprovecommunityoutcomesinfieldsfrompublicsafetytoeconomicdevelopmentGovExhasaproventrackrecordofworkinginconcertwithpartnerstoprovidetechnicalassistanceandtrainingatscale
SinceourlaunchinApril2015GovExhas
Trainedover2000governmentemployeesinpersonandonlineintopicsincludingdatamanagementperformancemanagementopendatacommunityengagementandanalytics
Providedtechnicalassistancetomorethan100localgovernmentsacrosstheglobeimprovingtheiruseofdatatoachieveefficienciesinservicedeliveryandbetteroutcomesforresidents
Assessedthecapacityofmorethan130governmentalorganizationstodeterminehowleadersstaffandexternalstakeholdersmanageandusedata
Localgovernmentsareincreasinglyonthefrontlinesofaddressinglargescaleproblemswithglobalimplicationssuchasclimatechangepublicsafetyandeconomicinequality
Whileevidence-basedinterventionstohelpsolvethoseproblemsareavailablelocalgovernmentstaffhavelimitedtimeandcapacitytofindandimplementresearch-basedsolutionsAsaresultcitiescontinuetoworkthewaytheyalwayshave--makingincrementalimprovementstoprogramsbutnotalwayscreatingchangeatasystem-level
ThroughresourcedevelopmentandcollaborationwithJHUfacultyGovExissupportingcityleadersinusingdataandevidencetomakedecisionsandcreatinglastingrelationshipswithresearchers
TheRoadmapwasdevelopedbyGovExaspartofitspartnershipwithBloombergPhilanthropiesWhatWorksCitiesinitiative
AboutGovEx
69
- Title page
- Introduction and purpose of the Roadmap
- Using the Roadmap
-
- Gather information about the problem
- Strategically engage key partners
- Review the evidence and collect details
- Decide on a policy to implement based on local factors
- Establish a framework to monitor progress
- Assess data accessibility quality and security
- Design implementation and evaluation
- Share your story and success
-
- Working with a range of evidence
- Thank you
- References
- Resources
- About GovEx
-
ExampleInSomervillemayoralleadershipandengagementofmultiplecityagencies(publichealthprimarycareschoolsCambridgeHealthAllianceetc)streamlinedandconnectedthevariousShapeUpSomervilleinitiativesandgrantswhichhelpedtokeepthetalentandenergyfocusedonhealthyeatingandactivelivingforallsectorsofSomervilleSUSpartnersultimatelyincludedschoolsnonprofitsCityandstateagenciesandnationalfoundations
Yourturn
____________________________________________________________________________
____________________________________________________________________________
DescribetheinterventionWhatarethestepsactivitiesorprocessesRememberthatthemechanismthespecificstepsorprocessthatleadtobehaviorchangeiskeytounderstandingtheeffectivenessofthepolicychangeandsharingtheresultswithothers
ExampleGoalsofShapeUpSomervilleweretoinfluenceeverypartofanearlyelementaryschoolchildrsquosdayndashbeforeduringandafter-schooltopromotehealthyeatingphysicalactivityandtheirweightstatusInterventionincludedfoodserviceenhancementstoimprovebreakfastandlunchandstaffprofessionaldevelopmentwalktoschoolactivitiesSUSclassroomcurriculumandprofessionaldevelopmentschoolwellnesspolicydevelopmentSUSafter-schoolcurriculumandprofessionaldevelopmentoutreachandeducationtohomethroughmaterialsforumseventsldquoSUSapprovedrdquorestaurantscommunityoutreachandcapacitybuildingthroughpolicydevelopmenttrainingsmediaplacements
Yourturn
____________________________________________________________________________
____________________________________________________________________________
ClarifytimelineandmilestonesfordeliveryandevaluationHowfrequentlywilltheservicesinterventionbedeliveredHowlongwilltheprogramrunWhatisthetimetoexpectedimpacts(eg12monthsafterprogramcompletion)
Designimplementationandevaluation
46
ExampleShapeUpSomervilleledbyTuftsUniversityFriedmanSchoolofNutritionScienceandPolicyconductedaquasi-experimental(nonrandomized)interventionstudyamong1stthrough3rdgradestudentsinSomerville(intervention)andtwocomparisoncommunitiesfor9-monthsduringoneschoolyear(fall2002ndashspring2003)ThestudyshowedpositiveassociationoftheinterventiononchangeinBMISUSreleasedreportssummarizingevaluationsprogresstodateplansforadditionalprojectsandimpactstenandfifteenyearsintotheinitiative
Yourturn
____________________________________________________________________________
____________________________________________________________________________
PlanyouranalysisTheprevioussectionaddressedestablishinganaccountabilityframeworktomonitorprogressandimplementationDeterminewhichindicatorswillbediscussthroughthatprocesswhowillsharethedataandanalysisandhowfeedbackfromthoseconversationswillbeincorporatedbytheprojectteam
ExampleTheMayorrsquosofficeimplementedadata-drivendecisionmakingapproachcalledSomerStatwhichwassynergisticwiththeapproachestakenbytheresearchandevaluationteamsdeployedforthespecificgrants
Yourturn
____________________________________________________________________________
____________________________________________________________________________
PlanforsustainabilityIfthepolicyandprogramchangeissuccessfulhowwilltheprojectteamsustainandexpandtoadditionalsitesorpopulations
ExampleAlthoughmostofthepartnershipsforgedintheearlyyearsweremaintainednewgrantsandcollaborationsexpandedattentiontoolderpublicschoolstudentsactivetransportationandtheGreenLinesubwayextensionthebuiltenvironmentfarmerrsquosmarketeffortssuchasldquoGrowninSomervillerdquowithUnionMainStreeteconomicdevelopmentandhealthdisparitiesaswellasdatacollectionmonitoringandsurveillance
Yourturn
Designimplementationandevaluation
47
____________________________________________________________________________
____________________________________________________________________________
Pilotplanningtemplate
IfyouplantoconductapilotfirstdeterminethepurposeofthepilotDoyouwanttoobservewhetherthepolicychangeresultsinthedesiredbehaviorchangeAreyouconcernedaboutfeasibilityscalabilityorcostDoyouwanttotestwhetheryourdesiredresultscanbeachievedthroughthispolicyapproachClarifythegoalofthepilotandthequestionsyouhopetoanswerThiswillhelpyouidentifythekeyindicatorsthatyouwillusetomeasurewhetherthepilotissuccessfulBeforebeginningestablishabaselineofcurrentperformanceandsetyourtargetsaccordinglybasedonwhatyouhopetoaccomplishthroughthepilot
NextestablishthepilotplanforyourprojectteamincludingmilestonesforteamcheckinstodeterminewhethertomoveaheadorchangecourseThisisseparatefromthemeetingsthatwilloccuraspartoftheaccountabilityframeworkUsethespacebelowtocreateyourtimelineanddevelopyourmeetingschedule
Task Personresponsible Startdate Checkindate Deliverable
Planyourevaluation
WhileyouaredesigningtheimplementationitisalsotimetobegintodevelopyourplanforanalysisAsnotedbytheWilsonSheehanLabforEconomicOpportunitiesdecisionsaboutprogramsetup-includingoutreachscreeningintakedatatrackingandenrollment-candeterminewhetherornotarigorousimpactevaluationwillbepossibleAsdiscussedcityleadersmayfindithelpfultoincludeevaluationpartnersinearlyconversationsaboutprogramsetupDuringpreviousstagesyoumayhaveworkedwitharesearchertoplanyourproposedpolicychangeYoushouldbeclearaboutthepurposeoftheevaluationanddevelopanevaluationplanBelowareadditionalconsiderations
Causationvstrendanalysis
IntheEstablishaframeworktomonitorprogressstagetherewasguidanceontrackingtrendsovertimeItisimportanttonoteherethatmonitoringdataisdifferentfromanalyzingacausaleffectTrendanalysiswillillustratethatchangesareoccurringandmayleadtoquestionsthatcouldbeansweredbyyourevaluationbuttrendanalysisalonedoesnotprovideevidenceaboutwhatisdrivingthosechangesYoumayseechangesinyour
Designimplementationandevaluation
48
outcomeindicatorsbasedonvariablesoutsideoftheCityscontrolForinstanceachangeinfederaltradepolicycanhaveanimpactemploymentratesinyourCityoutsideofyourCitysnewjobtrainingandplacementprogram
ToaddresscausalityyoumustcomparetheoutcomesofindividualswhoreceivedtheinterventionwiththosewhodidnotWithoutacomparisongroupofsimilarindividualswhowerenotexposedtotheinterventionitisimpossibletoknowwhichfactorsareresponsiblefortheoutcomesRandomizedControlTrials(RCTs)inwhichindividualsarerandomlyassignedtotreatmentandcontrolgroupsarethegoldstandardinprovingcausalityIfyoucandemonstratethatoutcomesimprovedforthosewhoreceivedtheinterventionanddidnotimproveatallorimprovedlessforacomparisongroupyoucanbesomewhatcertainthattheinterventionwasthecause
Examplesofothernon-causalimpactevaluationsthatarerigorousandreliableincludecertainquasi-experimentalstudiesusingcomparisongroupdesignssmartinterruptedtimeseriesanalysesthatmodeltrendsovertimeinwaysthatallowestimationofinterventioneffectswelldesignedpre-posttestanalysesneedsassessmentsimplementationstudiesetcFormoreinformationondifferenttypesofevaluationsseetheSageResearchMethodswebsite
Lookfordifferencesingroups
BuildintimetoalsoexaminethepilotdatatoseeifgroupsareexperiencingdifferentresultsReviewingperformancebasedonoutcomemeasuresforapopulationinaggregatemayhidethesubtlewaysthatsomepeoplemaybeaffecteddifferentlysoitisnecessarytocarefullyanalyzethedatafordisparateoutcomesForinstancetheMovingtoOpportunity(MTO)studythatexaminedtheimpactofhousingvouchersonfamilieswholivedinhigh-povertyneighborhoodsfoundthatchildrenwhomovedbeforetheageof13hadbettereducationandhigherincomesasadultsHoweveradultsdidnotseeimprovededucationandemploymentoutcomes
Disaggregatingdataintomorespecificcategoriesofraceandethnicitycanhelpunderstandtrendsanddisparitiesinamoregranularwaywhichcanhelpdecisionmakerstargetpoliciesandinterventionsinamoreequitableway
EarlierintheRoadmapweworkedtoidentifystakeholdersandwhomightbepositivelyandnegativelyimpactedbyachangeinpolicyUsetheresultsofthatexercisetoinvolvethecommunityininterpretingdataanalysisandtogatherqualitativefeedbackfromthecommunitytocallattentiontoissuesForinstanceanincreaseinyouthdetentionmaybeexplainedbyinefficienciesorlackofaccesstotransportation
Designimplementationandevaluation
49
Designimplementationandevaluation
50
Shareyourstoryandsuccess
Sharetheresults
Storytelling
AnintegralstepinyourstrategyissharingtheresultsofyourpilotorpolicychangewithyourcommunityThisincludesdevelopingmessagingaroundtheworkofyourorganizationandcreatingaplantodisseminateyourstoriesWhytellstoriesaboutyourwork
Informandorunderstandasubjectorsituation
Giveneededcontextofdataorbitsofinformation
Relatablestoriesarememorableforreadersviewers
ReadersorviewerswillseetheinformationasmorecompellingwhenthereisademonstratedandrelatablechangeorimpactonthemselvesortheirfriendsneighborsandcolleaguesStoriesareaboutpeopleandgovernmentisfundamentallyabouthelpingpeopleSotellthestoriesofthepeopleyouandyourcitycolleagueshelped
Keepthesetipsinmindasyoudevelopyourstorywithyourteam
WhatwereyourfindingsandwhyshouldanyonecareTellyouraudiencehowtheproblemwassolvedWhathastheimpactbeenonresidentsandcitystafftodateWhatwasthesuccessandweretheremultiplesuccessesalongthewayIfyouhavepreliminaryresultsordatashowingimprovementtosharethisisthetimetodoit
WhowasinvolvedandimpactedReviewthelistofstakeholdersHowwereeachofthemimpactedbytheprogramHumanizeyourstoryandimpactbyaddingadditionalvoicesThesevoicescanbeoneofthefollowing
Apersonorfamilywhowaspositivelyaffectedbythechangesuccess
Aninvolvedthird-partydescribingboththeproblemandthesolution(ieanorganizationnonprofitprovidersbusinesstradeassociationsportsteametc)whoispartofthecommunitythatthegovernmentworkedwithonthechangesuccess
Someonefromthegovernmentitselfwhocanspeakaboutthechangesuccess
TellmoreaboutyourcityDescribeyourcityincludingpopulationdemographicsindustriesetcWhatareitschallengesWhatisitknownforWhatmakesituniqueHowisitsimilarordifferentfromothercitiesintheUS
Shareyourstoryandsuccess
51
DescribeyourcityrsquoschallengeWriteoutthechallengeyourcityisfacing(asdefinedinyourproblemstatement)HowdoesitaffectresidentsHowdoesthisproblemaffecttheworkofcitystaffmembers
AddthedetailsProvidedetailsonyourappliedresearchstrategyTalkaboutwhatyourcityhasdonenthepasttoresolvetheissueDiscusshowyourstrategyhaschangedwhenyoubegantoadoptthisapproachWasmoneysavedorinvested
Usethisstorymapexercisetoorganizeyourthoughtsandguaranteethatyouincludealloftheelementsofatraditionalstory
AfteryoudevelopyourmessagingcreateacommunicationsmarketingandoutreachstrategydesignedtospreadyourmessageYourstrategyshouldwalkyouthroughthefollowingkeyactivities
1 IdentifyandprofileyouraudienceWhencreatinganddisseminatingyourmessageitisalwaysgoodtothinkaboutyouraudienceWhoexactlywouldbeinterestedinwhatyouhavetosayThegeneralpopulationofthecityPeoplewhoworkinnonprofitsBusinessownersItmattersbecausenoteveryonewholivesinacitymaybethetargetoftheinformationBusinessownersmaybemoreinterestedindatashowingeffectsonthebusinessdistrictthanothersandsoonWewillcoverthislaterbutthisstepwillalsohelpyoutothinkofthemosteffectivewaytoreachyouraudience
2 Defineyourcalltoaction(ifyouhaveone)SpendsometimeconsideringwhyyouareinterestedintellingyourstoryDoyousimplywanttoupdatethepubliconyourprogressorcelebrateyoursuccessesAreyouinterestedinyouraudiencechangingitsbehaviororsupportingaspecificinitiativebeinglaunchedbyyourorganizationIfyouwanttohaveyouraudienceactorbehaveinacertainwaybespecificandincorporateitinyourmessaging
Shareyourstoryandsuccess
52
3 SelectperformancemeasuresThinkabouthowyouwoulddetermineifyourcampaignissuccessfulandselectappropriateperformancemeasuresYouwillrevisitthesemeasuresattheendofyourcampaigncommunicationsplanimplementationtodeterminewhetheryouneedtomodifyyourcommunicationsandoutreachstrategyforthefuture
4 MethodstocommunicateyourstoryTherearemanywaystocommunicatesuccessofyourprojectHerearesomesuggestionsonreachingaudiencesItisimportanttodeterminewhichmethodsarethemosteffectiveforyourdesiredaudiencetoreceiveinformationDoyourresearch(PewResearchCenterhasexcellentstudiesonthereachofmediadigitalmediaandmarketing)ForexampleifseniorsareyourtargetpopulationandtheyarelessinclinedtouseInstagram(thatistrue)youmaywanttoconsiderFacebookorthenightlylocalnewsbothofwhichareusedmoreoftenbyseniorsBelowisalistofsomemethodsandtoolsyoucanuse
i Pressreleasestolocalnewspapersandtvstations
ii ShortsmartphonevideosAgreatwaytoengageanaudienceisthroughvideoAskonequestionofsomeonewhoispartofthestoryMaybethereisasocialworkerwhocandiscusshungerinyourcityThesevideoscanbereusedonmultipleplatforms
iii Long-formwriting
iv TellingindividualstoriesthroughaseriesofphotosandshortnarrativesforpublicationonFacebookorInstagram
v Op-edsopinionpiecesplacedinthelocalpaper
vi Emailupdatestoyourlist
vii Connectingwithappropriateorganizationsthatareinvestedinthetopicandworkwiththecommunityandaskthemtodisseminateinformationtotheirmembers
viii EmployingldquoWordofMouthrdquo(WOM)tacticsbycreatingtownhallsorfreeeventsforresidentstoattendintheirownneighborhoods
5 DeterminethetypeofcollateralUsuallycampaignsthattargetresidentsrequireseveralavenuestoreachthemPeoplearenotalwaysinthesameplaceorspaceHavingakeenunderstandingofyouraudiencewillassistyouwithyourchoicesonwhatneedstobedevelopedSometimesthatmeansimagesforsocialmediaaonepageflierthatgivespeopleinformationonattendinganeventoranin-depthbrochure
Shareyourstoryandsuccess
53
6 ImplementyourstrategyKeepinmindstrategiescanbefluidSometimessomethingmaynotturnoutasexpectedevenifduringplanningsessionsitwasldquosupposedtordquoItisokYoucananalyzewhathappenedduringadebriefsessionandonceyoursurveysandothermeasurementshavebeenfinalized
Visualizations
Withtherightvisualizationyoucanreviewandshareinformationinsightsanalysisperformanceandtrendsallofwhichisvaluabletoacross-sectionofthepublicButwhichvisualizationisrightforyouTochoosetherighttypeofchartgraphdeterminethetypeofquestionyouwanttoanswerSomecommoncharttypesarebelowandmoreinformationcanbefoundhere
Exploreadditionalanalyticaltechniques
Duringthegoal-settingphaseyouestablishedoutputoutcomeandimplementationmeasuresBasedontheresultsofyourpilotyoumayneedtoadjustyourmeasuresWiththepilotdatainhandyoumayhavesomeideasofhowfutureanalysescouldbedifferentandpotentiallyimprovethequalityofyourprogramTakesometimetobrainstormotherrelevantdatasetsthatyoudidnothavethetimetopursueearlierincludingmessydepartmentaldatathatneedsmoreattentionorsourcesofpublicdatafromthestateandfederalgovernmentnonprofitsandevenprivatecompaniesthatcouldsupplementyourunderstandingofanissueUsingmachinelearningtechniquesyoumightbeabletobetter
Shareyourstoryandsuccess
54
predictdemandforhomelessservicesusingNOAAweatherforecastspreemptwatermainbreaksusingsoilstratigraphyorinvestineconomicdevelopmentbymonitoringrealestatelistingdata
Advancedanalyticaltechniquescanalsohelpyourimplementationrunmoreeffectivelybyintegratingdata-informedtargetinganddecisiontoolsintotheprocessByautomatingroutineworkflowsandsurfacingsubtletrendsthesetoolscanhelpyouiterateinamoreprecisemannerandprovidekeystrategicguidancetofront-lineemployeesthatwouldotherwisebeunavailableThesesortsofdata-science-enhancedprocessescanbeclassifiedasoneofsixprojecttypesasexploredfurtherbytheNewOrleansLANOLAlyticsteamFormoreguidanceonscopingasimilarprojecttheGovExadvancedanalyticsscopingguidegivesyouastep-by-stepoverviewofhowtobesuccessful
Usethespacebelowtolistyourideasforpossibleadvancedanalyticsprojectsbytype
Typeofproject Ideas
Findingtheneedleinthehaystack
Prioritizingforimpact
Earlywarningtools
Betterquickerdecisions
Optimizingresourceallocation
Experimentingforwhatworks
Keepthefeedbackloopgoing
GatherinformationonpublicsentimentaboutpolicychangethroughfocusgroupssurveysandpublicmeetingsledbyadvisorygroupmembersNotewhethernewinformationisavailablethatcanhelpanalyzeorcontextualizeyourprogress(egfederalorstatedatareports)Sharetheresultsofyourpilotandthepolicychangeprocesswithothercitiesandresearchers
Shareyourstoryandsuccess
55
WorkingwitharangeofevidencePresentlythereisalotofattentionandinvestmentinevidence-basedpolicymakingincludingthefederalbipartisanCommissiononEvidence-BasedPolicymakingthePewCharitableTrustsreportsandtoolkitsonHowStatesEngageinEvidence-BasedPolicymakingandtheLauraandJohnArnoldFoundationsupportedEvidence-BasedPolicymakingCollaborativeManyoftheseinitiativesfocusonrigorousevaluation--meaningrandomizedcontrolledtrials(RCT)orotherexperimentalmodels--asthebestkindofevidencetouseHoweverthisbodyofevidenceisofteninsufficientforthespecifickindsofquestionsthatgovernmentscanhavewhenmakingadecisionaboutpolicyInthisguideweproposeanapproachtosupportgovernmentsinincorporatingadditionaltypesofevidenceintotheirdecisionmaking
OnewaytounderstandarangeofevidenceistouseNestarsquosStandardsofEvidencetoframeanddefineevidenceTheBritishnonprofitNestaworkstoinjectpositivechangeintothewayUKgovernmentpoliciesandprogramsfunctionOneoftheirkeyareasofworkhasbeendevelopingWhatWorksCentresorganizationsthatcollectcommissionsynthesizeanddisseminateevidenceaboutinterventioneffectivenessacrossarangeofsocialimpactareasNestadevelopedthefollowingStandardsofEvidencetoprovideausefulwaytodocumenthowdifferentkindsofinformationdemonstratedifferentlevelsofevidence
Workingwitharangeofevidence
56
Theimportanceofpotentiallybiasedinformation
Onechallengepresentedbyanevidenceclassificationthatacknowledgesthevalueofnon-scientificformsofevidenceistofigureouthowtobalanceallofthepiecestogetherwhenyouusethemTounderstandhowtodothiswehavetothinkabouthowtopositionpotentiallybiasedinformationindecisionmaking
WhenwelookatinformationwhichhasnotbeenrigorouslyevaluatedweacceptthatthisinformationmightbebiasedThewholereasonweconductrigorousevaluationistoremovebiasfromourobservationsWeconductstructuredtrialstoproduceresultsthatareconsistentandgeneralizableIfafindingisunbiasedthisaccuracymdashthislackofbiasmdashcanbeprovedbecauseanyindependenttestercanreproduceitwhentheyfollowanidenticalprocess
YoucanlookatpotentiallybiasedinformationasjustinformationthatisearlieroninitspathtorefinementreplicationandverifiedaccuracyApieceofinformationbasedonanyoneindividualrsquosobservationsislikelytobebiasedbecauseitiscomingfromjustonesourceBiasisreducedbygatheringtogetherlotsofindependentsourcesofinformationandfiguringoutitsmostcommonattributes
MaybethebiasedinformationmakesaclaimthatistoolargemdashitsayssomethingistrueunderallconditionswhenitisreallyonlytrueundercertainconditionsorthataneffectofaninterventionisbiggerthanitstrueaverageorthataneffectlastsovertimewhenitdoesnotPerhapschangingtheclaimtobemorespecificmdashcriticallyreviewingtheinformationtoseeifspecificconditionsmustbemettoachievetheoutcomesmdashwouldmaketheclaimmoreaccurate
IfyoutestedyourclaimfounditdidnotholdupinsomecaseschangedittesteditagainuntilyourclaimrepeatedlyheldupthenyoucoulddemonstratethattheclaimwasnolongerbiasedbutaccurateThisprocessoftestingimprovingtestingandvalidatingiswhatwegetfromstructuredscientificstudy
Withindividualpiecesofinformation--crediblehypothesescasestudiesexpertopinion--youcannotassumethatkindofaccuracyAtthesametimetheseearlyobservationsarefoundationaltoanyreplicablescientificobservationsWewillnevergetanywhereifwedonotstartwiththesepiecesofinformationeveniftheymightbebiased
Combininglevelsofevidencetogetthemostdetailedinformationforyourcity
ItisespeciallyimportanttoacceptdifferentformsofevidencewhenyoufacequestionsthathavenotyetbeentestedbyindependentstudiesSomeofthemostimportantquestionsrelatetothingsthatareveryspecifictoyoursituationYoufaceaparticularcombinationof
Workingwitharangeofevidence
57
issuesactorstrendsandresourcesthatisultimatelydifferentfromeveryotherplaceintheworld
ThisisimportantbecauseaspolicypractitionersandnotscientistsyoucaremoreaboutyourspecificcasethanahypotheticalaveragegeneralcaseYoucarelessaboutthetotallikelyrangeofimpactsfromapolicythanhowthissingleimplementationwillworkYouwantinformationaboutpoliciesthatwillprovideyouwiththebestpossiblepredictionabouthowtheywillworkinyourcontext
ByputtingdifferentkindsofevidenceintothesameframeworkNestarsquosStandardsofEvidenceimplythatwhenyoucannotgetrigorouslytestedexperimentallyverifiedinformationitisappropriatetoworkwithwhatwedohaveThisisrelevantforthemunicipalimplementationcontextbecausenostudypulledfromaliteraturereviewwillaccountforallofthevariablesyoufaceinyourowncontext
Intheabsenceofhavingastudythatpredictshowaninterventionworksinyourspecificcontextyouwillneednon-scientificinformationtotrytofillinthosegapsYouwillneedtogatherstoriesfromresidentsstaffmembersandotherstakeholdersthatsitonLevels1or2ofthestandardsofevidence
Someofthekindsofinformationyouwillprobablygetmoreeasilyfrompeoplethanstudiesinclude
howpeopleincitieslikeyoursreactedtoaninterventioninpublicforumshowpeoplewhoimplementedaninterventioninacitylikeyoursfiguredouthowtogetitpassedhowthecitycommunicatedpubliclyabouttheapparentbenefitsanddrawbacksoftheinterventionthepoliticalacceptabilityoftheintervention
Youcanremainsimultaneouslyawarethattheinformationyouaregatheringthroughindividualstoriesisbiased--notgeneralizableinformedbyspecificperspectives--whilealsorecognizingthathavingsomeinformationaboutsomethingimportantisbetterthannoinformationEvidencefromthefirsttwoevidencelevelsareessentialbuildingblocksofknowledgeandperfectlyusefulwhenyoucannotgetdetailedunbiasedknowledge
Youhavetostartsomewhereandyoushouldalwaysstartwithevidenceyouhaveaboutyourveryspecificconditionsthatyouknowareimportantforsuccess
Workingwitharangeofevidence
58
Thankyou
Interviewswithcities
GovExheldsemi-structuredinterviewswithsixstaffmembersinfourcitiestoestablishtheinitialstepsoftheRoadmapWeaskedrespondentstotalkaboutaninitiativeorpolicythattheyhavebeenresponsibleforimplementingintheircityRespondentsfromthreecitiesspokeabouttheireffortstoaddresshomelessnessandrespondentsfromonecityspokeaboutblightAfterrespondentsidentifiedtheproblemtheyspokeabouthowthecityprioritizedthisissuethevariousstepsinvolvedandbarrierspresent
GovExconductedtextanalysisoftheseinterviewswhichrevealedthatpeoplewasthemostcommonlyusedtermduringtheseconversationswhendiscussingthestepstochangeaprogramorpolicyThisisathemethroughoutthisRoadmapThefirstsectionbeginswithidentifyingtherelevantpeopleTheconversationsaroundevidencearefocusedonhumanbehaviorratherthanmethodologyorstrengthofevidence
Thankyou
59
OncewehadadraftoftheRoadmapwefollowedupwithcitiesonanindividualbasistolearnmoreaboutwhatwouldbehelpfulWearegratefultothefollowingcitystafffortheinterviewsandindividualconversations
MichaelBaskinChiefPolicyOfficerChattanoogaTN
MelissaBridgesPerformanceandInnovationCoordinatorLittleRockAR
VictoriaCarreoacutenAdministrativeOfficerLasVegasNV
TyroneGrandstrandCommunityDevelopmentPlannerFargoND
TibLaughlinDirectorofGeneralServicesKansasCityKS
TimMorelandDirectorofPerformanceManagementandOpenDataChattanoogaTN
CharlesRosendahlManagementAnalystDentonTX
Thankyou
60
MelissaSiebenAssistantCountyAdministratorKansasCityKS
GinaThompsonDirectorofStrategicServicesLewisvilleTX
JHUFacultycollaboration
GovExalsoconvenedthecitycontactsandfacultypartnersinaseriesofAdvisoryGroupmeetingsThesemeetingsservedtoconnectcitiesandfacultytounderstandoneanothersperspective
StefanBaralBloombergSchoolofPublicHealth
AshleyBernerJHUSchoolofEducation
MelissaDavey-RothwellBloombergSchoolofPublicHealth
JonLinksBloombergSchoolofPublicHealth
TamarMendelsonBloombergSchoolofPublicHealth
KristinMmariBloombergSchoolofPublicHealth
KeshiaPollackPorterBloombergSchoolofPublicHealth
JoshSharfsteinBloombergSchoolofPublicHealth
LizStuartBloombergSchoolofPublicHealth
LindsayThompsonCareyBusinessSchool
DanielWebsterBloombergSchoolofPublicHealth
Partnersindataandevidence-basedpolicy
Wealsospokewithpartnerswhoareexpertsinthefieldofevidence-basedpolicywhosharedtheirknowledgeandresourcesofcurrentandfuturethinkinginevidence-informedthinking
WendyBarrenoAssociateDirectorWilsonSheehanLabforEconomicOpportunitiesUniversityofNotreDame
TiffanyDavenportDirectorofDataPartnershipsBaltimoresPromise
TomiHiersDirectorBaltimoreCivicSiteAnnieECaseyFoundation
ChrisKingsleySeniorAssociateAnnieECaseyFoundation
KathrynOliverAssociateProfessorLondonSchoolofHygieneandTropicalMedicine
Thankyou
61
KristenWatkinsSeniorPolicyAssociateJ-PAL
GovExParticipants
GovExAnalystMiriamMcKinneyanalyzedtheinterviewtranscriptsandconductedtextanalysisSeniorImplementationAdvisorEmilyShawwillintegrateRoadmapactivitieswithadditionalservicedeliveryactivitiesatGovEx
Thankyou
62
ReferencesAbrahamKGampHaskinsR(2017)Thepromiseofevidence-basedpolicymakingReportofthecommissiononevidence-basedpolicymakingWashingtonDCCommissiononEvidence-basedPolicymakingBigmove2Extendingthegreen(nd)Retrievedfromhttpswwwcityoflewisvillecomdoing-businesslewisville-2025big-move-2-extending-the-greenBatesMAampGlennersterR(2017)ThegeneralizabilitypuzzleRetrievedfromhttpsssirorgarticlesentrythe_generalizability_puzzleBuildwithnotfor(2018January)RetrievedfromhttpsbuildwithnotforwordpresscomCairneyPampOliverK(2017)Evidence-basedpolicymakingisnotlikeevidence-basedmedicinesohowfarshouldyougotobridgethedividebetweenevidenceandpolicyHealthResearchPolicyandSystemsdoi101186s12961-017-0192-xCairneyPOliverKampWellsteadA(2016)TobridgethedividebetweenevidenceandpolicyReduceambiguityasmuchasuncertaintyPublicAdministrationReview76(3)399-402Retrievedfromhttpdxdoiorg101111puar12555Centreforpublicimpact(nd)RetrievedfromhttpswwwcentreforpublicimpactorgCernaL(2013)ThenatureofpolicychangeandimplementationAreviewofdifferenttheoreticalapproachesOrganisationforEconomicCo-OperationandDevelopmentRetrievedfromhttpswwwoecdorgeduceriTheNatureofPolicyChangeandImplementationpdfColoradoSprings(nd)Retrievedfromhttpscoloradospringsgovcommunicationspagewhat-expect-when-you-speakupContinuumofcare(coc)program(nd)RetrievedfromhttpswwwhudexchangeinfoprogramscocCouncilpresidentkicksoff$12millionyouthfund(nd)Retrievedfromhttpwwwbaltimorecitycouncilcomcontentcouncil-president-kicks-12-million-youth-fundDurlakJAampDuPreEP(2008)ImplementationmattersAreviewofresearchontheinfluenceofimplementationonprogramoutcomesandthefactorsaffectingimplementationAmericanJournalofCommunityPsychology41327-350Retrievedfromhttpswwwncbinlmnihgovpubmed18322790Fishbonediagram(ishikawa)(nd)Retrievedfromhttpasqorglearn-about-qualitycause-analysis-toolsoverviewfishbonehtmlFivewhysGettingtotherootofaproblemquickly(nd)RetrievedfromhttpswwwmindtoolscompagesarticlenewTMC_5Whtm
References
63
FligsteinNampMcAdamD(2011)TowardageneraltheoryofstrategicactionfieldsSociologicalTheory29(1)1-26RetrievedfromhttpsociologyberkeleyedusitesdefaultfilesfacultyfligsteinFligstein20McAdam20Soc20Theory20PaperpdHeinkelP(2018January12)CitymullspermitsforbikesharingcompaniesRetrievedfromhttpwwwdentonrccomnewsdenton20180112city-mulls-permits-bike-sharing-companiesHowisareamedianincomecalculated(2012October)Retrievedfromhttpswwwhudexchangeinfofaqs843how-is-area-median-income-calculatedLampkinLMampHatryHP(2003)KeystepsinoutcomemanagementUrbanInstituteRetrievedfromhttpswwwurbanorgresearchpublicationkey-steps-outcome-managementMarshDampSmithM(2000)UnderstandingpolicynetworksTowardsadialecticalapproachPoliticalStudies484-21Retrievedfromhttpjournalssagepubcomdoiabs1011111467-924800247McCannL(nd)RetrievedfromhttplaurenellencomMoultonSampSandfortJR(2016)ThestrategicactionfieldframeworkforpolicyimplementationresearchPolicyStudiesJournal45(1)Retrievedfromhttponlinelibrarywileycomstore101111psj12147assetpsj12147pdfv=1ampt=jdkhhcp4amps=e184c625e1855a6b1615ccb99ee27b1c0b3212d8PaynterB(2018January1)What100citiesarelearningfromeachotherbysharingtheirdataRetrievedfromhttpswwwfastcompanycom40518004what-100-cities-are-learning-from-each-other-by-sharing-their-dataPeakacademy(nd)Retrievedfromhttpswwwdenvergovorgcontentdenvergovenmayors-officeprograms-initiativespeak-performancepeak-academyhtmlPinnockHBarwickMCarpenterCREldridgeSGrandesGGriffithsCJRycroft-MaloneJMeissnerPMurrayEPatelASheikhAampTaylorSJC(2017)Standardsforreportingimplementationstudies(stari)ExplanationandelaborationdocumentBMJOpenRetrievedfromhttpbmjopenbmjcomcontentbmjopen74e013318fullpdfProctorESilmereHRaghavanRHovmandPAaronsGBungerAGriffeyRampHensleyM(2011)OutcomesforimplementationresearchConceptualdistinctionsmeasurementchallengesandresearchagendaAdministrationandPolicyinMentalHealth38(2)65-76doi101007s10488-010-0319-7TandonSD(2018)ProgressincommunityhealthpartnershipsResearcheducationandactionRetrievedfromhttpswwwpressjhuedujournalsprogress-community-health-partnerships-research-education-and-actionProvanKGampKenisP(2007)ModesofnetworkgovernanceStructuremanagementandeffectivenessJournalofPublicAdministrationResearchandTheory
References
64
18(2)229-252Retrievedfromhttpsacademicoupcomjpartarticle182229935895SabatierPA(1988)Anadvocacycoalitionframeworkofpolicychangeandtheroleofpolicy-orientedlearningthereinPolicySciences21(3)129-168Retrievedfromhttpwwwjstororgstable4532139SandstromAampCarlssonL(2008)TheperformanceofpolicynetworksTherelationbetweennetworkstructureandnetworkperformancePolicyStudiesJournal36497-524Retrievedfromhttponlinelibrarywileycomdoi101111j1541-0072200800281xabstractSchwartzMampWilsonE(nd)WhocanaffordtoliveinahomeAlookatdatafromthe2006americancommunitysurveyRetrievedfromhttpswwwcensusgovhousingcensuspublicationswho-can-affordpdfShearerJCAbelsonAKouyateBLavisJNampWaltG(2016)WhydopolicieschangeInstitutionsinterestsideasandnetworksinthreecasesofpolicyreformHealthPolicyandPlanning31(9)1200-1211Retrievedfromhttpsdoiorg101093heapolczw052ThelessonsofadministrativedataHigh-profilepolicy-relevantresearchpoweredbyadministrativedata(nd)Retrievedfromhttpswwwpovertyactionlaborgsitesdefaultfilesresources20161114-The-Lessons-of-Administrative-DatapdfToolsandotherneedsforqualitativeandquantitativemethods(2015February)AgencyforHealthcareResearchandQualityRetrievedfromhttpwwwahrqgovprofessionalsquality-patient-safetytalkingqualityassesstoolshtml
WeaverRCampBagchi-SenS(2013)SpatialanalysisofurbandeclineThegeographyofblightAppliedGeography4061-70
References
65
UniversityPartners
Name Title Affiliation Area(s)ofexpertise
Contactinformation
Examplesofworkingwithnon-academicdecisionmakers
WendyPBarreno
AssociateDirector
WilsonSheehanLabforEconomicOpportunitiesUniversityofNotreDame
LEOsfocusareasincludehealtheducationhousingchildwelfareeconomicopportunitycriminaljustice
wbarrenonduduorleondedu
MelissaDavey-RothwellPhD
AssociateScientist
JohnsHopkinsBloombergSchoolofPublicHealth
Health mdavey1jhuedu410-614-5854
BaltimoreCityHealthDepartmentLeadersMarylandDepartmentofHealthAddictionRecoveryServicesinDCExecutiveDirectorsofcommunityorganizationsacrosstheUSthatfocusonHIVSTIsubstanceabuseandimplementationscience
ElizabethStuart
AssociateDeanforEducationandProfessorDepartmentsofMentalHealthBiostatisticsandHealthPolicyand
JohnsHopkinsBloombergSchoolofPublicHealth
Educationsocialpolicystatisticalmethods
estuartjhuedu
Resources
66
Management
Design-ThinkingResources
EngagementGuideforCivicTech
Journeymap
PerformanceManagementResources
DenverPeakAcademy
Proxymeasuresguide
Talkingaboutperformancewithoutcomplicatingit(video)
SettingPerformanceTargetsguide
Datavisualizationguide
Performancemanagementgettingstartedguide
Exercises
Fishboneexercise
5Whys
AdministrativeData
TheLessonsofAdministrativeData(JPAL)
EvaluationResources
Quantitativeandqualitativemethods(USDepartmentofHealthandHumanServicesAgencyforHealthcareResearchandQuality)
KeyStepsinOutcomeManagement(UrbanInstitute)
RCT-YES
Sageresearchmethods
BestPracticesforMixedMethodsResearchintheHealthSciences
ACCORDChecklist
Pilotandimplementationplanning
Benchmarkingexamplesandproxymeasures
Seattleopendataplaybook
Resources
67
Datasciencescopingguide
Resources
68
AboutGovExGovExhelpsgovernmentsusedataandevidencetoimproveresidentsrsquolivesWesupportgovernmentsinbuildingtheirinternalcapacitytooperatesustainableanalyticspracticeswhichimprovecommunityoutcomesinfieldsfrompublicsafetytoeconomicdevelopmentGovExhasaproventrackrecordofworkinginconcertwithpartnerstoprovidetechnicalassistanceandtrainingatscale
SinceourlaunchinApril2015GovExhas
Trainedover2000governmentemployeesinpersonandonlineintopicsincludingdatamanagementperformancemanagementopendatacommunityengagementandanalytics
Providedtechnicalassistancetomorethan100localgovernmentsacrosstheglobeimprovingtheiruseofdatatoachieveefficienciesinservicedeliveryandbetteroutcomesforresidents
Assessedthecapacityofmorethan130governmentalorganizationstodeterminehowleadersstaffandexternalstakeholdersmanageandusedata
Localgovernmentsareincreasinglyonthefrontlinesofaddressinglargescaleproblemswithglobalimplicationssuchasclimatechangepublicsafetyandeconomicinequality
Whileevidence-basedinterventionstohelpsolvethoseproblemsareavailablelocalgovernmentstaffhavelimitedtimeandcapacitytofindandimplementresearch-basedsolutionsAsaresultcitiescontinuetoworkthewaytheyalwayshave--makingincrementalimprovementstoprogramsbutnotalwayscreatingchangeatasystem-level
ThroughresourcedevelopmentandcollaborationwithJHUfacultyGovExissupportingcityleadersinusingdataandevidencetomakedecisionsandcreatinglastingrelationshipswithresearchers
TheRoadmapwasdevelopedbyGovExaspartofitspartnershipwithBloombergPhilanthropiesWhatWorksCitiesinitiative
AboutGovEx
69
- Title page
- Introduction and purpose of the Roadmap
- Using the Roadmap
-
- Gather information about the problem
- Strategically engage key partners
- Review the evidence and collect details
- Decide on a policy to implement based on local factors
- Establish a framework to monitor progress
- Assess data accessibility quality and security
- Design implementation and evaluation
- Share your story and success
-
- Working with a range of evidence
- Thank you
- References
- Resources
- About GovEx
-
ExampleShapeUpSomervilleledbyTuftsUniversityFriedmanSchoolofNutritionScienceandPolicyconductedaquasi-experimental(nonrandomized)interventionstudyamong1stthrough3rdgradestudentsinSomerville(intervention)andtwocomparisoncommunitiesfor9-monthsduringoneschoolyear(fall2002ndashspring2003)ThestudyshowedpositiveassociationoftheinterventiononchangeinBMISUSreleasedreportssummarizingevaluationsprogresstodateplansforadditionalprojectsandimpactstenandfifteenyearsintotheinitiative
Yourturn
____________________________________________________________________________
____________________________________________________________________________
PlanyouranalysisTheprevioussectionaddressedestablishinganaccountabilityframeworktomonitorprogressandimplementationDeterminewhichindicatorswillbediscussthroughthatprocesswhowillsharethedataandanalysisandhowfeedbackfromthoseconversationswillbeincorporatedbytheprojectteam
ExampleTheMayorrsquosofficeimplementedadata-drivendecisionmakingapproachcalledSomerStatwhichwassynergisticwiththeapproachestakenbytheresearchandevaluationteamsdeployedforthespecificgrants
Yourturn
____________________________________________________________________________
____________________________________________________________________________
PlanforsustainabilityIfthepolicyandprogramchangeissuccessfulhowwilltheprojectteamsustainandexpandtoadditionalsitesorpopulations
ExampleAlthoughmostofthepartnershipsforgedintheearlyyearsweremaintainednewgrantsandcollaborationsexpandedattentiontoolderpublicschoolstudentsactivetransportationandtheGreenLinesubwayextensionthebuiltenvironmentfarmerrsquosmarketeffortssuchasldquoGrowninSomervillerdquowithUnionMainStreeteconomicdevelopmentandhealthdisparitiesaswellasdatacollectionmonitoringandsurveillance
Yourturn
Designimplementationandevaluation
47
____________________________________________________________________________
____________________________________________________________________________
Pilotplanningtemplate
IfyouplantoconductapilotfirstdeterminethepurposeofthepilotDoyouwanttoobservewhetherthepolicychangeresultsinthedesiredbehaviorchangeAreyouconcernedaboutfeasibilityscalabilityorcostDoyouwanttotestwhetheryourdesiredresultscanbeachievedthroughthispolicyapproachClarifythegoalofthepilotandthequestionsyouhopetoanswerThiswillhelpyouidentifythekeyindicatorsthatyouwillusetomeasurewhetherthepilotissuccessfulBeforebeginningestablishabaselineofcurrentperformanceandsetyourtargetsaccordinglybasedonwhatyouhopetoaccomplishthroughthepilot
NextestablishthepilotplanforyourprojectteamincludingmilestonesforteamcheckinstodeterminewhethertomoveaheadorchangecourseThisisseparatefromthemeetingsthatwilloccuraspartoftheaccountabilityframeworkUsethespacebelowtocreateyourtimelineanddevelopyourmeetingschedule
Task Personresponsible Startdate Checkindate Deliverable
Planyourevaluation
WhileyouaredesigningtheimplementationitisalsotimetobegintodevelopyourplanforanalysisAsnotedbytheWilsonSheehanLabforEconomicOpportunitiesdecisionsaboutprogramsetup-includingoutreachscreeningintakedatatrackingandenrollment-candeterminewhetherornotarigorousimpactevaluationwillbepossibleAsdiscussedcityleadersmayfindithelpfultoincludeevaluationpartnersinearlyconversationsaboutprogramsetupDuringpreviousstagesyoumayhaveworkedwitharesearchertoplanyourproposedpolicychangeYoushouldbeclearaboutthepurposeoftheevaluationanddevelopanevaluationplanBelowareadditionalconsiderations
Causationvstrendanalysis
IntheEstablishaframeworktomonitorprogressstagetherewasguidanceontrackingtrendsovertimeItisimportanttonoteherethatmonitoringdataisdifferentfromanalyzingacausaleffectTrendanalysiswillillustratethatchangesareoccurringandmayleadtoquestionsthatcouldbeansweredbyyourevaluationbuttrendanalysisalonedoesnotprovideevidenceaboutwhatisdrivingthosechangesYoumayseechangesinyour
Designimplementationandevaluation
48
outcomeindicatorsbasedonvariablesoutsideoftheCityscontrolForinstanceachangeinfederaltradepolicycanhaveanimpactemploymentratesinyourCityoutsideofyourCitysnewjobtrainingandplacementprogram
ToaddresscausalityyoumustcomparetheoutcomesofindividualswhoreceivedtheinterventionwiththosewhodidnotWithoutacomparisongroupofsimilarindividualswhowerenotexposedtotheinterventionitisimpossibletoknowwhichfactorsareresponsiblefortheoutcomesRandomizedControlTrials(RCTs)inwhichindividualsarerandomlyassignedtotreatmentandcontrolgroupsarethegoldstandardinprovingcausalityIfyoucandemonstratethatoutcomesimprovedforthosewhoreceivedtheinterventionanddidnotimproveatallorimprovedlessforacomparisongroupyoucanbesomewhatcertainthattheinterventionwasthecause
Examplesofothernon-causalimpactevaluationsthatarerigorousandreliableincludecertainquasi-experimentalstudiesusingcomparisongroupdesignssmartinterruptedtimeseriesanalysesthatmodeltrendsovertimeinwaysthatallowestimationofinterventioneffectswelldesignedpre-posttestanalysesneedsassessmentsimplementationstudiesetcFormoreinformationondifferenttypesofevaluationsseetheSageResearchMethodswebsite
Lookfordifferencesingroups
BuildintimetoalsoexaminethepilotdatatoseeifgroupsareexperiencingdifferentresultsReviewingperformancebasedonoutcomemeasuresforapopulationinaggregatemayhidethesubtlewaysthatsomepeoplemaybeaffecteddifferentlysoitisnecessarytocarefullyanalyzethedatafordisparateoutcomesForinstancetheMovingtoOpportunity(MTO)studythatexaminedtheimpactofhousingvouchersonfamilieswholivedinhigh-povertyneighborhoodsfoundthatchildrenwhomovedbeforetheageof13hadbettereducationandhigherincomesasadultsHoweveradultsdidnotseeimprovededucationandemploymentoutcomes
Disaggregatingdataintomorespecificcategoriesofraceandethnicitycanhelpunderstandtrendsanddisparitiesinamoregranularwaywhichcanhelpdecisionmakerstargetpoliciesandinterventionsinamoreequitableway
EarlierintheRoadmapweworkedtoidentifystakeholdersandwhomightbepositivelyandnegativelyimpactedbyachangeinpolicyUsetheresultsofthatexercisetoinvolvethecommunityininterpretingdataanalysisandtogatherqualitativefeedbackfromthecommunitytocallattentiontoissuesForinstanceanincreaseinyouthdetentionmaybeexplainedbyinefficienciesorlackofaccesstotransportation
Designimplementationandevaluation
49
Designimplementationandevaluation
50
Shareyourstoryandsuccess
Sharetheresults
Storytelling
AnintegralstepinyourstrategyissharingtheresultsofyourpilotorpolicychangewithyourcommunityThisincludesdevelopingmessagingaroundtheworkofyourorganizationandcreatingaplantodisseminateyourstoriesWhytellstoriesaboutyourwork
Informandorunderstandasubjectorsituation
Giveneededcontextofdataorbitsofinformation
Relatablestoriesarememorableforreadersviewers
ReadersorviewerswillseetheinformationasmorecompellingwhenthereisademonstratedandrelatablechangeorimpactonthemselvesortheirfriendsneighborsandcolleaguesStoriesareaboutpeopleandgovernmentisfundamentallyabouthelpingpeopleSotellthestoriesofthepeopleyouandyourcitycolleagueshelped
Keepthesetipsinmindasyoudevelopyourstorywithyourteam
WhatwereyourfindingsandwhyshouldanyonecareTellyouraudiencehowtheproblemwassolvedWhathastheimpactbeenonresidentsandcitystafftodateWhatwasthesuccessandweretheremultiplesuccessesalongthewayIfyouhavepreliminaryresultsordatashowingimprovementtosharethisisthetimetodoit
WhowasinvolvedandimpactedReviewthelistofstakeholdersHowwereeachofthemimpactedbytheprogramHumanizeyourstoryandimpactbyaddingadditionalvoicesThesevoicescanbeoneofthefollowing
Apersonorfamilywhowaspositivelyaffectedbythechangesuccess
Aninvolvedthird-partydescribingboththeproblemandthesolution(ieanorganizationnonprofitprovidersbusinesstradeassociationsportsteametc)whoispartofthecommunitythatthegovernmentworkedwithonthechangesuccess
Someonefromthegovernmentitselfwhocanspeakaboutthechangesuccess
TellmoreaboutyourcityDescribeyourcityincludingpopulationdemographicsindustriesetcWhatareitschallengesWhatisitknownforWhatmakesituniqueHowisitsimilarordifferentfromothercitiesintheUS
Shareyourstoryandsuccess
51
DescribeyourcityrsquoschallengeWriteoutthechallengeyourcityisfacing(asdefinedinyourproblemstatement)HowdoesitaffectresidentsHowdoesthisproblemaffecttheworkofcitystaffmembers
AddthedetailsProvidedetailsonyourappliedresearchstrategyTalkaboutwhatyourcityhasdonenthepasttoresolvetheissueDiscusshowyourstrategyhaschangedwhenyoubegantoadoptthisapproachWasmoneysavedorinvested
Usethisstorymapexercisetoorganizeyourthoughtsandguaranteethatyouincludealloftheelementsofatraditionalstory
AfteryoudevelopyourmessagingcreateacommunicationsmarketingandoutreachstrategydesignedtospreadyourmessageYourstrategyshouldwalkyouthroughthefollowingkeyactivities
1 IdentifyandprofileyouraudienceWhencreatinganddisseminatingyourmessageitisalwaysgoodtothinkaboutyouraudienceWhoexactlywouldbeinterestedinwhatyouhavetosayThegeneralpopulationofthecityPeoplewhoworkinnonprofitsBusinessownersItmattersbecausenoteveryonewholivesinacitymaybethetargetoftheinformationBusinessownersmaybemoreinterestedindatashowingeffectsonthebusinessdistrictthanothersandsoonWewillcoverthislaterbutthisstepwillalsohelpyoutothinkofthemosteffectivewaytoreachyouraudience
2 Defineyourcalltoaction(ifyouhaveone)SpendsometimeconsideringwhyyouareinterestedintellingyourstoryDoyousimplywanttoupdatethepubliconyourprogressorcelebrateyoursuccessesAreyouinterestedinyouraudiencechangingitsbehaviororsupportingaspecificinitiativebeinglaunchedbyyourorganizationIfyouwanttohaveyouraudienceactorbehaveinacertainwaybespecificandincorporateitinyourmessaging
Shareyourstoryandsuccess
52
3 SelectperformancemeasuresThinkabouthowyouwoulddetermineifyourcampaignissuccessfulandselectappropriateperformancemeasuresYouwillrevisitthesemeasuresattheendofyourcampaigncommunicationsplanimplementationtodeterminewhetheryouneedtomodifyyourcommunicationsandoutreachstrategyforthefuture
4 MethodstocommunicateyourstoryTherearemanywaystocommunicatesuccessofyourprojectHerearesomesuggestionsonreachingaudiencesItisimportanttodeterminewhichmethodsarethemosteffectiveforyourdesiredaudiencetoreceiveinformationDoyourresearch(PewResearchCenterhasexcellentstudiesonthereachofmediadigitalmediaandmarketing)ForexampleifseniorsareyourtargetpopulationandtheyarelessinclinedtouseInstagram(thatistrue)youmaywanttoconsiderFacebookorthenightlylocalnewsbothofwhichareusedmoreoftenbyseniorsBelowisalistofsomemethodsandtoolsyoucanuse
i Pressreleasestolocalnewspapersandtvstations
ii ShortsmartphonevideosAgreatwaytoengageanaudienceisthroughvideoAskonequestionofsomeonewhoispartofthestoryMaybethereisasocialworkerwhocandiscusshungerinyourcityThesevideoscanbereusedonmultipleplatforms
iii Long-formwriting
iv TellingindividualstoriesthroughaseriesofphotosandshortnarrativesforpublicationonFacebookorInstagram
v Op-edsopinionpiecesplacedinthelocalpaper
vi Emailupdatestoyourlist
vii Connectingwithappropriateorganizationsthatareinvestedinthetopicandworkwiththecommunityandaskthemtodisseminateinformationtotheirmembers
viii EmployingldquoWordofMouthrdquo(WOM)tacticsbycreatingtownhallsorfreeeventsforresidentstoattendintheirownneighborhoods
5 DeterminethetypeofcollateralUsuallycampaignsthattargetresidentsrequireseveralavenuestoreachthemPeoplearenotalwaysinthesameplaceorspaceHavingakeenunderstandingofyouraudiencewillassistyouwithyourchoicesonwhatneedstobedevelopedSometimesthatmeansimagesforsocialmediaaonepageflierthatgivespeopleinformationonattendinganeventoranin-depthbrochure
Shareyourstoryandsuccess
53
6 ImplementyourstrategyKeepinmindstrategiescanbefluidSometimessomethingmaynotturnoutasexpectedevenifduringplanningsessionsitwasldquosupposedtordquoItisokYoucananalyzewhathappenedduringadebriefsessionandonceyoursurveysandothermeasurementshavebeenfinalized
Visualizations
Withtherightvisualizationyoucanreviewandshareinformationinsightsanalysisperformanceandtrendsallofwhichisvaluabletoacross-sectionofthepublicButwhichvisualizationisrightforyouTochoosetherighttypeofchartgraphdeterminethetypeofquestionyouwanttoanswerSomecommoncharttypesarebelowandmoreinformationcanbefoundhere
Exploreadditionalanalyticaltechniques
Duringthegoal-settingphaseyouestablishedoutputoutcomeandimplementationmeasuresBasedontheresultsofyourpilotyoumayneedtoadjustyourmeasuresWiththepilotdatainhandyoumayhavesomeideasofhowfutureanalysescouldbedifferentandpotentiallyimprovethequalityofyourprogramTakesometimetobrainstormotherrelevantdatasetsthatyoudidnothavethetimetopursueearlierincludingmessydepartmentaldatathatneedsmoreattentionorsourcesofpublicdatafromthestateandfederalgovernmentnonprofitsandevenprivatecompaniesthatcouldsupplementyourunderstandingofanissueUsingmachinelearningtechniquesyoumightbeabletobetter
Shareyourstoryandsuccess
54
predictdemandforhomelessservicesusingNOAAweatherforecastspreemptwatermainbreaksusingsoilstratigraphyorinvestineconomicdevelopmentbymonitoringrealestatelistingdata
Advancedanalyticaltechniquescanalsohelpyourimplementationrunmoreeffectivelybyintegratingdata-informedtargetinganddecisiontoolsintotheprocessByautomatingroutineworkflowsandsurfacingsubtletrendsthesetoolscanhelpyouiterateinamoreprecisemannerandprovidekeystrategicguidancetofront-lineemployeesthatwouldotherwisebeunavailableThesesortsofdata-science-enhancedprocessescanbeclassifiedasoneofsixprojecttypesasexploredfurtherbytheNewOrleansLANOLAlyticsteamFormoreguidanceonscopingasimilarprojecttheGovExadvancedanalyticsscopingguidegivesyouastep-by-stepoverviewofhowtobesuccessful
Usethespacebelowtolistyourideasforpossibleadvancedanalyticsprojectsbytype
Typeofproject Ideas
Findingtheneedleinthehaystack
Prioritizingforimpact
Earlywarningtools
Betterquickerdecisions
Optimizingresourceallocation
Experimentingforwhatworks
Keepthefeedbackloopgoing
GatherinformationonpublicsentimentaboutpolicychangethroughfocusgroupssurveysandpublicmeetingsledbyadvisorygroupmembersNotewhethernewinformationisavailablethatcanhelpanalyzeorcontextualizeyourprogress(egfederalorstatedatareports)Sharetheresultsofyourpilotandthepolicychangeprocesswithothercitiesandresearchers
Shareyourstoryandsuccess
55
WorkingwitharangeofevidencePresentlythereisalotofattentionandinvestmentinevidence-basedpolicymakingincludingthefederalbipartisanCommissiononEvidence-BasedPolicymakingthePewCharitableTrustsreportsandtoolkitsonHowStatesEngageinEvidence-BasedPolicymakingandtheLauraandJohnArnoldFoundationsupportedEvidence-BasedPolicymakingCollaborativeManyoftheseinitiativesfocusonrigorousevaluation--meaningrandomizedcontrolledtrials(RCT)orotherexperimentalmodels--asthebestkindofevidencetouseHoweverthisbodyofevidenceisofteninsufficientforthespecifickindsofquestionsthatgovernmentscanhavewhenmakingadecisionaboutpolicyInthisguideweproposeanapproachtosupportgovernmentsinincorporatingadditionaltypesofevidenceintotheirdecisionmaking
OnewaytounderstandarangeofevidenceistouseNestarsquosStandardsofEvidencetoframeanddefineevidenceTheBritishnonprofitNestaworkstoinjectpositivechangeintothewayUKgovernmentpoliciesandprogramsfunctionOneoftheirkeyareasofworkhasbeendevelopingWhatWorksCentresorganizationsthatcollectcommissionsynthesizeanddisseminateevidenceaboutinterventioneffectivenessacrossarangeofsocialimpactareasNestadevelopedthefollowingStandardsofEvidencetoprovideausefulwaytodocumenthowdifferentkindsofinformationdemonstratedifferentlevelsofevidence
Workingwitharangeofevidence
56
Theimportanceofpotentiallybiasedinformation
Onechallengepresentedbyanevidenceclassificationthatacknowledgesthevalueofnon-scientificformsofevidenceistofigureouthowtobalanceallofthepiecestogetherwhenyouusethemTounderstandhowtodothiswehavetothinkabouthowtopositionpotentiallybiasedinformationindecisionmaking
WhenwelookatinformationwhichhasnotbeenrigorouslyevaluatedweacceptthatthisinformationmightbebiasedThewholereasonweconductrigorousevaluationistoremovebiasfromourobservationsWeconductstructuredtrialstoproduceresultsthatareconsistentandgeneralizableIfafindingisunbiasedthisaccuracymdashthislackofbiasmdashcanbeprovedbecauseanyindependenttestercanreproduceitwhentheyfollowanidenticalprocess
YoucanlookatpotentiallybiasedinformationasjustinformationthatisearlieroninitspathtorefinementreplicationandverifiedaccuracyApieceofinformationbasedonanyoneindividualrsquosobservationsislikelytobebiasedbecauseitiscomingfromjustonesourceBiasisreducedbygatheringtogetherlotsofindependentsourcesofinformationandfiguringoutitsmostcommonattributes
MaybethebiasedinformationmakesaclaimthatistoolargemdashitsayssomethingistrueunderallconditionswhenitisreallyonlytrueundercertainconditionsorthataneffectofaninterventionisbiggerthanitstrueaverageorthataneffectlastsovertimewhenitdoesnotPerhapschangingtheclaimtobemorespecificmdashcriticallyreviewingtheinformationtoseeifspecificconditionsmustbemettoachievetheoutcomesmdashwouldmaketheclaimmoreaccurate
IfyoutestedyourclaimfounditdidnotholdupinsomecaseschangedittesteditagainuntilyourclaimrepeatedlyheldupthenyoucoulddemonstratethattheclaimwasnolongerbiasedbutaccurateThisprocessoftestingimprovingtestingandvalidatingiswhatwegetfromstructuredscientificstudy
Withindividualpiecesofinformation--crediblehypothesescasestudiesexpertopinion--youcannotassumethatkindofaccuracyAtthesametimetheseearlyobservationsarefoundationaltoanyreplicablescientificobservationsWewillnevergetanywhereifwedonotstartwiththesepiecesofinformationeveniftheymightbebiased
Combininglevelsofevidencetogetthemostdetailedinformationforyourcity
ItisespeciallyimportanttoacceptdifferentformsofevidencewhenyoufacequestionsthathavenotyetbeentestedbyindependentstudiesSomeofthemostimportantquestionsrelatetothingsthatareveryspecifictoyoursituationYoufaceaparticularcombinationof
Workingwitharangeofevidence
57
issuesactorstrendsandresourcesthatisultimatelydifferentfromeveryotherplaceintheworld
ThisisimportantbecauseaspolicypractitionersandnotscientistsyoucaremoreaboutyourspecificcasethanahypotheticalaveragegeneralcaseYoucarelessaboutthetotallikelyrangeofimpactsfromapolicythanhowthissingleimplementationwillworkYouwantinformationaboutpoliciesthatwillprovideyouwiththebestpossiblepredictionabouthowtheywillworkinyourcontext
ByputtingdifferentkindsofevidenceintothesameframeworkNestarsquosStandardsofEvidenceimplythatwhenyoucannotgetrigorouslytestedexperimentallyverifiedinformationitisappropriatetoworkwithwhatwedohaveThisisrelevantforthemunicipalimplementationcontextbecausenostudypulledfromaliteraturereviewwillaccountforallofthevariablesyoufaceinyourowncontext
Intheabsenceofhavingastudythatpredictshowaninterventionworksinyourspecificcontextyouwillneednon-scientificinformationtotrytofillinthosegapsYouwillneedtogatherstoriesfromresidentsstaffmembersandotherstakeholdersthatsitonLevels1or2ofthestandardsofevidence
Someofthekindsofinformationyouwillprobablygetmoreeasilyfrompeoplethanstudiesinclude
howpeopleincitieslikeyoursreactedtoaninterventioninpublicforumshowpeoplewhoimplementedaninterventioninacitylikeyoursfiguredouthowtogetitpassedhowthecitycommunicatedpubliclyabouttheapparentbenefitsanddrawbacksoftheinterventionthepoliticalacceptabilityoftheintervention
Youcanremainsimultaneouslyawarethattheinformationyouaregatheringthroughindividualstoriesisbiased--notgeneralizableinformedbyspecificperspectives--whilealsorecognizingthathavingsomeinformationaboutsomethingimportantisbetterthannoinformationEvidencefromthefirsttwoevidencelevelsareessentialbuildingblocksofknowledgeandperfectlyusefulwhenyoucannotgetdetailedunbiasedknowledge
Youhavetostartsomewhereandyoushouldalwaysstartwithevidenceyouhaveaboutyourveryspecificconditionsthatyouknowareimportantforsuccess
Workingwitharangeofevidence
58
Thankyou
Interviewswithcities
GovExheldsemi-structuredinterviewswithsixstaffmembersinfourcitiestoestablishtheinitialstepsoftheRoadmapWeaskedrespondentstotalkaboutaninitiativeorpolicythattheyhavebeenresponsibleforimplementingintheircityRespondentsfromthreecitiesspokeabouttheireffortstoaddresshomelessnessandrespondentsfromonecityspokeaboutblightAfterrespondentsidentifiedtheproblemtheyspokeabouthowthecityprioritizedthisissuethevariousstepsinvolvedandbarrierspresent
GovExconductedtextanalysisoftheseinterviewswhichrevealedthatpeoplewasthemostcommonlyusedtermduringtheseconversationswhendiscussingthestepstochangeaprogramorpolicyThisisathemethroughoutthisRoadmapThefirstsectionbeginswithidentifyingtherelevantpeopleTheconversationsaroundevidencearefocusedonhumanbehaviorratherthanmethodologyorstrengthofevidence
Thankyou
59
OncewehadadraftoftheRoadmapwefollowedupwithcitiesonanindividualbasistolearnmoreaboutwhatwouldbehelpfulWearegratefultothefollowingcitystafffortheinterviewsandindividualconversations
MichaelBaskinChiefPolicyOfficerChattanoogaTN
MelissaBridgesPerformanceandInnovationCoordinatorLittleRockAR
VictoriaCarreoacutenAdministrativeOfficerLasVegasNV
TyroneGrandstrandCommunityDevelopmentPlannerFargoND
TibLaughlinDirectorofGeneralServicesKansasCityKS
TimMorelandDirectorofPerformanceManagementandOpenDataChattanoogaTN
CharlesRosendahlManagementAnalystDentonTX
Thankyou
60
MelissaSiebenAssistantCountyAdministratorKansasCityKS
GinaThompsonDirectorofStrategicServicesLewisvilleTX
JHUFacultycollaboration
GovExalsoconvenedthecitycontactsandfacultypartnersinaseriesofAdvisoryGroupmeetingsThesemeetingsservedtoconnectcitiesandfacultytounderstandoneanothersperspective
StefanBaralBloombergSchoolofPublicHealth
AshleyBernerJHUSchoolofEducation
MelissaDavey-RothwellBloombergSchoolofPublicHealth
JonLinksBloombergSchoolofPublicHealth
TamarMendelsonBloombergSchoolofPublicHealth
KristinMmariBloombergSchoolofPublicHealth
KeshiaPollackPorterBloombergSchoolofPublicHealth
JoshSharfsteinBloombergSchoolofPublicHealth
LizStuartBloombergSchoolofPublicHealth
LindsayThompsonCareyBusinessSchool
DanielWebsterBloombergSchoolofPublicHealth
Partnersindataandevidence-basedpolicy
Wealsospokewithpartnerswhoareexpertsinthefieldofevidence-basedpolicywhosharedtheirknowledgeandresourcesofcurrentandfuturethinkinginevidence-informedthinking
WendyBarrenoAssociateDirectorWilsonSheehanLabforEconomicOpportunitiesUniversityofNotreDame
TiffanyDavenportDirectorofDataPartnershipsBaltimoresPromise
TomiHiersDirectorBaltimoreCivicSiteAnnieECaseyFoundation
ChrisKingsleySeniorAssociateAnnieECaseyFoundation
KathrynOliverAssociateProfessorLondonSchoolofHygieneandTropicalMedicine
Thankyou
61
KristenWatkinsSeniorPolicyAssociateJ-PAL
GovExParticipants
GovExAnalystMiriamMcKinneyanalyzedtheinterviewtranscriptsandconductedtextanalysisSeniorImplementationAdvisorEmilyShawwillintegrateRoadmapactivitieswithadditionalservicedeliveryactivitiesatGovEx
Thankyou
62
ReferencesAbrahamKGampHaskinsR(2017)Thepromiseofevidence-basedpolicymakingReportofthecommissiononevidence-basedpolicymakingWashingtonDCCommissiononEvidence-basedPolicymakingBigmove2Extendingthegreen(nd)Retrievedfromhttpswwwcityoflewisvillecomdoing-businesslewisville-2025big-move-2-extending-the-greenBatesMAampGlennersterR(2017)ThegeneralizabilitypuzzleRetrievedfromhttpsssirorgarticlesentrythe_generalizability_puzzleBuildwithnotfor(2018January)RetrievedfromhttpsbuildwithnotforwordpresscomCairneyPampOliverK(2017)Evidence-basedpolicymakingisnotlikeevidence-basedmedicinesohowfarshouldyougotobridgethedividebetweenevidenceandpolicyHealthResearchPolicyandSystemsdoi101186s12961-017-0192-xCairneyPOliverKampWellsteadA(2016)TobridgethedividebetweenevidenceandpolicyReduceambiguityasmuchasuncertaintyPublicAdministrationReview76(3)399-402Retrievedfromhttpdxdoiorg101111puar12555Centreforpublicimpact(nd)RetrievedfromhttpswwwcentreforpublicimpactorgCernaL(2013)ThenatureofpolicychangeandimplementationAreviewofdifferenttheoreticalapproachesOrganisationforEconomicCo-OperationandDevelopmentRetrievedfromhttpswwwoecdorgeduceriTheNatureofPolicyChangeandImplementationpdfColoradoSprings(nd)Retrievedfromhttpscoloradospringsgovcommunicationspagewhat-expect-when-you-speakupContinuumofcare(coc)program(nd)RetrievedfromhttpswwwhudexchangeinfoprogramscocCouncilpresidentkicksoff$12millionyouthfund(nd)Retrievedfromhttpwwwbaltimorecitycouncilcomcontentcouncil-president-kicks-12-million-youth-fundDurlakJAampDuPreEP(2008)ImplementationmattersAreviewofresearchontheinfluenceofimplementationonprogramoutcomesandthefactorsaffectingimplementationAmericanJournalofCommunityPsychology41327-350Retrievedfromhttpswwwncbinlmnihgovpubmed18322790Fishbonediagram(ishikawa)(nd)Retrievedfromhttpasqorglearn-about-qualitycause-analysis-toolsoverviewfishbonehtmlFivewhysGettingtotherootofaproblemquickly(nd)RetrievedfromhttpswwwmindtoolscompagesarticlenewTMC_5Whtm
References
63
FligsteinNampMcAdamD(2011)TowardageneraltheoryofstrategicactionfieldsSociologicalTheory29(1)1-26RetrievedfromhttpsociologyberkeleyedusitesdefaultfilesfacultyfligsteinFligstein20McAdam20Soc20Theory20PaperpdHeinkelP(2018January12)CitymullspermitsforbikesharingcompaniesRetrievedfromhttpwwwdentonrccomnewsdenton20180112city-mulls-permits-bike-sharing-companiesHowisareamedianincomecalculated(2012October)Retrievedfromhttpswwwhudexchangeinfofaqs843how-is-area-median-income-calculatedLampkinLMampHatryHP(2003)KeystepsinoutcomemanagementUrbanInstituteRetrievedfromhttpswwwurbanorgresearchpublicationkey-steps-outcome-managementMarshDampSmithM(2000)UnderstandingpolicynetworksTowardsadialecticalapproachPoliticalStudies484-21Retrievedfromhttpjournalssagepubcomdoiabs1011111467-924800247McCannL(nd)RetrievedfromhttplaurenellencomMoultonSampSandfortJR(2016)ThestrategicactionfieldframeworkforpolicyimplementationresearchPolicyStudiesJournal45(1)Retrievedfromhttponlinelibrarywileycomstore101111psj12147assetpsj12147pdfv=1ampt=jdkhhcp4amps=e184c625e1855a6b1615ccb99ee27b1c0b3212d8PaynterB(2018January1)What100citiesarelearningfromeachotherbysharingtheirdataRetrievedfromhttpswwwfastcompanycom40518004what-100-cities-are-learning-from-each-other-by-sharing-their-dataPeakacademy(nd)Retrievedfromhttpswwwdenvergovorgcontentdenvergovenmayors-officeprograms-initiativespeak-performancepeak-academyhtmlPinnockHBarwickMCarpenterCREldridgeSGrandesGGriffithsCJRycroft-MaloneJMeissnerPMurrayEPatelASheikhAampTaylorSJC(2017)Standardsforreportingimplementationstudies(stari)ExplanationandelaborationdocumentBMJOpenRetrievedfromhttpbmjopenbmjcomcontentbmjopen74e013318fullpdfProctorESilmereHRaghavanRHovmandPAaronsGBungerAGriffeyRampHensleyM(2011)OutcomesforimplementationresearchConceptualdistinctionsmeasurementchallengesandresearchagendaAdministrationandPolicyinMentalHealth38(2)65-76doi101007s10488-010-0319-7TandonSD(2018)ProgressincommunityhealthpartnershipsResearcheducationandactionRetrievedfromhttpswwwpressjhuedujournalsprogress-community-health-partnerships-research-education-and-actionProvanKGampKenisP(2007)ModesofnetworkgovernanceStructuremanagementandeffectivenessJournalofPublicAdministrationResearchandTheory
References
64
18(2)229-252Retrievedfromhttpsacademicoupcomjpartarticle182229935895SabatierPA(1988)Anadvocacycoalitionframeworkofpolicychangeandtheroleofpolicy-orientedlearningthereinPolicySciences21(3)129-168Retrievedfromhttpwwwjstororgstable4532139SandstromAampCarlssonL(2008)TheperformanceofpolicynetworksTherelationbetweennetworkstructureandnetworkperformancePolicyStudiesJournal36497-524Retrievedfromhttponlinelibrarywileycomdoi101111j1541-0072200800281xabstractSchwartzMampWilsonE(nd)WhocanaffordtoliveinahomeAlookatdatafromthe2006americancommunitysurveyRetrievedfromhttpswwwcensusgovhousingcensuspublicationswho-can-affordpdfShearerJCAbelsonAKouyateBLavisJNampWaltG(2016)WhydopolicieschangeInstitutionsinterestsideasandnetworksinthreecasesofpolicyreformHealthPolicyandPlanning31(9)1200-1211Retrievedfromhttpsdoiorg101093heapolczw052ThelessonsofadministrativedataHigh-profilepolicy-relevantresearchpoweredbyadministrativedata(nd)Retrievedfromhttpswwwpovertyactionlaborgsitesdefaultfilesresources20161114-The-Lessons-of-Administrative-DatapdfToolsandotherneedsforqualitativeandquantitativemethods(2015February)AgencyforHealthcareResearchandQualityRetrievedfromhttpwwwahrqgovprofessionalsquality-patient-safetytalkingqualityassesstoolshtml
WeaverRCampBagchi-SenS(2013)SpatialanalysisofurbandeclineThegeographyofblightAppliedGeography4061-70
References
65
UniversityPartners
Name Title Affiliation Area(s)ofexpertise
Contactinformation
Examplesofworkingwithnon-academicdecisionmakers
WendyPBarreno
AssociateDirector
WilsonSheehanLabforEconomicOpportunitiesUniversityofNotreDame
LEOsfocusareasincludehealtheducationhousingchildwelfareeconomicopportunitycriminaljustice
wbarrenonduduorleondedu
MelissaDavey-RothwellPhD
AssociateScientist
JohnsHopkinsBloombergSchoolofPublicHealth
Health mdavey1jhuedu410-614-5854
BaltimoreCityHealthDepartmentLeadersMarylandDepartmentofHealthAddictionRecoveryServicesinDCExecutiveDirectorsofcommunityorganizationsacrosstheUSthatfocusonHIVSTIsubstanceabuseandimplementationscience
ElizabethStuart
AssociateDeanforEducationandProfessorDepartmentsofMentalHealthBiostatisticsandHealthPolicyand
JohnsHopkinsBloombergSchoolofPublicHealth
Educationsocialpolicystatisticalmethods
estuartjhuedu
Resources
66
Management
Design-ThinkingResources
EngagementGuideforCivicTech
Journeymap
PerformanceManagementResources
DenverPeakAcademy
Proxymeasuresguide
Talkingaboutperformancewithoutcomplicatingit(video)
SettingPerformanceTargetsguide
Datavisualizationguide
Performancemanagementgettingstartedguide
Exercises
Fishboneexercise
5Whys
AdministrativeData
TheLessonsofAdministrativeData(JPAL)
EvaluationResources
Quantitativeandqualitativemethods(USDepartmentofHealthandHumanServicesAgencyforHealthcareResearchandQuality)
KeyStepsinOutcomeManagement(UrbanInstitute)
RCT-YES
Sageresearchmethods
BestPracticesforMixedMethodsResearchintheHealthSciences
ACCORDChecklist
Pilotandimplementationplanning
Benchmarkingexamplesandproxymeasures
Seattleopendataplaybook
Resources
67
Datasciencescopingguide
Resources
68
AboutGovExGovExhelpsgovernmentsusedataandevidencetoimproveresidentsrsquolivesWesupportgovernmentsinbuildingtheirinternalcapacitytooperatesustainableanalyticspracticeswhichimprovecommunityoutcomesinfieldsfrompublicsafetytoeconomicdevelopmentGovExhasaproventrackrecordofworkinginconcertwithpartnerstoprovidetechnicalassistanceandtrainingatscale
SinceourlaunchinApril2015GovExhas
Trainedover2000governmentemployeesinpersonandonlineintopicsincludingdatamanagementperformancemanagementopendatacommunityengagementandanalytics
Providedtechnicalassistancetomorethan100localgovernmentsacrosstheglobeimprovingtheiruseofdatatoachieveefficienciesinservicedeliveryandbetteroutcomesforresidents
Assessedthecapacityofmorethan130governmentalorganizationstodeterminehowleadersstaffandexternalstakeholdersmanageandusedata
Localgovernmentsareincreasinglyonthefrontlinesofaddressinglargescaleproblemswithglobalimplicationssuchasclimatechangepublicsafetyandeconomicinequality
Whileevidence-basedinterventionstohelpsolvethoseproblemsareavailablelocalgovernmentstaffhavelimitedtimeandcapacitytofindandimplementresearch-basedsolutionsAsaresultcitiescontinuetoworkthewaytheyalwayshave--makingincrementalimprovementstoprogramsbutnotalwayscreatingchangeatasystem-level
ThroughresourcedevelopmentandcollaborationwithJHUfacultyGovExissupportingcityleadersinusingdataandevidencetomakedecisionsandcreatinglastingrelationshipswithresearchers
TheRoadmapwasdevelopedbyGovExaspartofitspartnershipwithBloombergPhilanthropiesWhatWorksCitiesinitiative
AboutGovEx
69
- Title page
- Introduction and purpose of the Roadmap
- Using the Roadmap
-
- Gather information about the problem
- Strategically engage key partners
- Review the evidence and collect details
- Decide on a policy to implement based on local factors
- Establish a framework to monitor progress
- Assess data accessibility quality and security
- Design implementation and evaluation
- Share your story and success
-
- Working with a range of evidence
- Thank you
- References
- Resources
- About GovEx
-
____________________________________________________________________________
____________________________________________________________________________
Pilotplanningtemplate
IfyouplantoconductapilotfirstdeterminethepurposeofthepilotDoyouwanttoobservewhetherthepolicychangeresultsinthedesiredbehaviorchangeAreyouconcernedaboutfeasibilityscalabilityorcostDoyouwanttotestwhetheryourdesiredresultscanbeachievedthroughthispolicyapproachClarifythegoalofthepilotandthequestionsyouhopetoanswerThiswillhelpyouidentifythekeyindicatorsthatyouwillusetomeasurewhetherthepilotissuccessfulBeforebeginningestablishabaselineofcurrentperformanceandsetyourtargetsaccordinglybasedonwhatyouhopetoaccomplishthroughthepilot
NextestablishthepilotplanforyourprojectteamincludingmilestonesforteamcheckinstodeterminewhethertomoveaheadorchangecourseThisisseparatefromthemeetingsthatwilloccuraspartoftheaccountabilityframeworkUsethespacebelowtocreateyourtimelineanddevelopyourmeetingschedule
Task Personresponsible Startdate Checkindate Deliverable
Planyourevaluation
WhileyouaredesigningtheimplementationitisalsotimetobegintodevelopyourplanforanalysisAsnotedbytheWilsonSheehanLabforEconomicOpportunitiesdecisionsaboutprogramsetup-includingoutreachscreeningintakedatatrackingandenrollment-candeterminewhetherornotarigorousimpactevaluationwillbepossibleAsdiscussedcityleadersmayfindithelpfultoincludeevaluationpartnersinearlyconversationsaboutprogramsetupDuringpreviousstagesyoumayhaveworkedwitharesearchertoplanyourproposedpolicychangeYoushouldbeclearaboutthepurposeoftheevaluationanddevelopanevaluationplanBelowareadditionalconsiderations
Causationvstrendanalysis
IntheEstablishaframeworktomonitorprogressstagetherewasguidanceontrackingtrendsovertimeItisimportanttonoteherethatmonitoringdataisdifferentfromanalyzingacausaleffectTrendanalysiswillillustratethatchangesareoccurringandmayleadtoquestionsthatcouldbeansweredbyyourevaluationbuttrendanalysisalonedoesnotprovideevidenceaboutwhatisdrivingthosechangesYoumayseechangesinyour
Designimplementationandevaluation
48
outcomeindicatorsbasedonvariablesoutsideoftheCityscontrolForinstanceachangeinfederaltradepolicycanhaveanimpactemploymentratesinyourCityoutsideofyourCitysnewjobtrainingandplacementprogram
ToaddresscausalityyoumustcomparetheoutcomesofindividualswhoreceivedtheinterventionwiththosewhodidnotWithoutacomparisongroupofsimilarindividualswhowerenotexposedtotheinterventionitisimpossibletoknowwhichfactorsareresponsiblefortheoutcomesRandomizedControlTrials(RCTs)inwhichindividualsarerandomlyassignedtotreatmentandcontrolgroupsarethegoldstandardinprovingcausalityIfyoucandemonstratethatoutcomesimprovedforthosewhoreceivedtheinterventionanddidnotimproveatallorimprovedlessforacomparisongroupyoucanbesomewhatcertainthattheinterventionwasthecause
Examplesofothernon-causalimpactevaluationsthatarerigorousandreliableincludecertainquasi-experimentalstudiesusingcomparisongroupdesignssmartinterruptedtimeseriesanalysesthatmodeltrendsovertimeinwaysthatallowestimationofinterventioneffectswelldesignedpre-posttestanalysesneedsassessmentsimplementationstudiesetcFormoreinformationondifferenttypesofevaluationsseetheSageResearchMethodswebsite
Lookfordifferencesingroups
BuildintimetoalsoexaminethepilotdatatoseeifgroupsareexperiencingdifferentresultsReviewingperformancebasedonoutcomemeasuresforapopulationinaggregatemayhidethesubtlewaysthatsomepeoplemaybeaffecteddifferentlysoitisnecessarytocarefullyanalyzethedatafordisparateoutcomesForinstancetheMovingtoOpportunity(MTO)studythatexaminedtheimpactofhousingvouchersonfamilieswholivedinhigh-povertyneighborhoodsfoundthatchildrenwhomovedbeforetheageof13hadbettereducationandhigherincomesasadultsHoweveradultsdidnotseeimprovededucationandemploymentoutcomes
Disaggregatingdataintomorespecificcategoriesofraceandethnicitycanhelpunderstandtrendsanddisparitiesinamoregranularwaywhichcanhelpdecisionmakerstargetpoliciesandinterventionsinamoreequitableway
EarlierintheRoadmapweworkedtoidentifystakeholdersandwhomightbepositivelyandnegativelyimpactedbyachangeinpolicyUsetheresultsofthatexercisetoinvolvethecommunityininterpretingdataanalysisandtogatherqualitativefeedbackfromthecommunitytocallattentiontoissuesForinstanceanincreaseinyouthdetentionmaybeexplainedbyinefficienciesorlackofaccesstotransportation
Designimplementationandevaluation
49
Designimplementationandevaluation
50
Shareyourstoryandsuccess
Sharetheresults
Storytelling
AnintegralstepinyourstrategyissharingtheresultsofyourpilotorpolicychangewithyourcommunityThisincludesdevelopingmessagingaroundtheworkofyourorganizationandcreatingaplantodisseminateyourstoriesWhytellstoriesaboutyourwork
Informandorunderstandasubjectorsituation
Giveneededcontextofdataorbitsofinformation
Relatablestoriesarememorableforreadersviewers
ReadersorviewerswillseetheinformationasmorecompellingwhenthereisademonstratedandrelatablechangeorimpactonthemselvesortheirfriendsneighborsandcolleaguesStoriesareaboutpeopleandgovernmentisfundamentallyabouthelpingpeopleSotellthestoriesofthepeopleyouandyourcitycolleagueshelped
Keepthesetipsinmindasyoudevelopyourstorywithyourteam
WhatwereyourfindingsandwhyshouldanyonecareTellyouraudiencehowtheproblemwassolvedWhathastheimpactbeenonresidentsandcitystafftodateWhatwasthesuccessandweretheremultiplesuccessesalongthewayIfyouhavepreliminaryresultsordatashowingimprovementtosharethisisthetimetodoit
WhowasinvolvedandimpactedReviewthelistofstakeholdersHowwereeachofthemimpactedbytheprogramHumanizeyourstoryandimpactbyaddingadditionalvoicesThesevoicescanbeoneofthefollowing
Apersonorfamilywhowaspositivelyaffectedbythechangesuccess
Aninvolvedthird-partydescribingboththeproblemandthesolution(ieanorganizationnonprofitprovidersbusinesstradeassociationsportsteametc)whoispartofthecommunitythatthegovernmentworkedwithonthechangesuccess
Someonefromthegovernmentitselfwhocanspeakaboutthechangesuccess
TellmoreaboutyourcityDescribeyourcityincludingpopulationdemographicsindustriesetcWhatareitschallengesWhatisitknownforWhatmakesituniqueHowisitsimilarordifferentfromothercitiesintheUS
Shareyourstoryandsuccess
51
DescribeyourcityrsquoschallengeWriteoutthechallengeyourcityisfacing(asdefinedinyourproblemstatement)HowdoesitaffectresidentsHowdoesthisproblemaffecttheworkofcitystaffmembers
AddthedetailsProvidedetailsonyourappliedresearchstrategyTalkaboutwhatyourcityhasdonenthepasttoresolvetheissueDiscusshowyourstrategyhaschangedwhenyoubegantoadoptthisapproachWasmoneysavedorinvested
Usethisstorymapexercisetoorganizeyourthoughtsandguaranteethatyouincludealloftheelementsofatraditionalstory
AfteryoudevelopyourmessagingcreateacommunicationsmarketingandoutreachstrategydesignedtospreadyourmessageYourstrategyshouldwalkyouthroughthefollowingkeyactivities
1 IdentifyandprofileyouraudienceWhencreatinganddisseminatingyourmessageitisalwaysgoodtothinkaboutyouraudienceWhoexactlywouldbeinterestedinwhatyouhavetosayThegeneralpopulationofthecityPeoplewhoworkinnonprofitsBusinessownersItmattersbecausenoteveryonewholivesinacitymaybethetargetoftheinformationBusinessownersmaybemoreinterestedindatashowingeffectsonthebusinessdistrictthanothersandsoonWewillcoverthislaterbutthisstepwillalsohelpyoutothinkofthemosteffectivewaytoreachyouraudience
2 Defineyourcalltoaction(ifyouhaveone)SpendsometimeconsideringwhyyouareinterestedintellingyourstoryDoyousimplywanttoupdatethepubliconyourprogressorcelebrateyoursuccessesAreyouinterestedinyouraudiencechangingitsbehaviororsupportingaspecificinitiativebeinglaunchedbyyourorganizationIfyouwanttohaveyouraudienceactorbehaveinacertainwaybespecificandincorporateitinyourmessaging
Shareyourstoryandsuccess
52
3 SelectperformancemeasuresThinkabouthowyouwoulddetermineifyourcampaignissuccessfulandselectappropriateperformancemeasuresYouwillrevisitthesemeasuresattheendofyourcampaigncommunicationsplanimplementationtodeterminewhetheryouneedtomodifyyourcommunicationsandoutreachstrategyforthefuture
4 MethodstocommunicateyourstoryTherearemanywaystocommunicatesuccessofyourprojectHerearesomesuggestionsonreachingaudiencesItisimportanttodeterminewhichmethodsarethemosteffectiveforyourdesiredaudiencetoreceiveinformationDoyourresearch(PewResearchCenterhasexcellentstudiesonthereachofmediadigitalmediaandmarketing)ForexampleifseniorsareyourtargetpopulationandtheyarelessinclinedtouseInstagram(thatistrue)youmaywanttoconsiderFacebookorthenightlylocalnewsbothofwhichareusedmoreoftenbyseniorsBelowisalistofsomemethodsandtoolsyoucanuse
i Pressreleasestolocalnewspapersandtvstations
ii ShortsmartphonevideosAgreatwaytoengageanaudienceisthroughvideoAskonequestionofsomeonewhoispartofthestoryMaybethereisasocialworkerwhocandiscusshungerinyourcityThesevideoscanbereusedonmultipleplatforms
iii Long-formwriting
iv TellingindividualstoriesthroughaseriesofphotosandshortnarrativesforpublicationonFacebookorInstagram
v Op-edsopinionpiecesplacedinthelocalpaper
vi Emailupdatestoyourlist
vii Connectingwithappropriateorganizationsthatareinvestedinthetopicandworkwiththecommunityandaskthemtodisseminateinformationtotheirmembers
viii EmployingldquoWordofMouthrdquo(WOM)tacticsbycreatingtownhallsorfreeeventsforresidentstoattendintheirownneighborhoods
5 DeterminethetypeofcollateralUsuallycampaignsthattargetresidentsrequireseveralavenuestoreachthemPeoplearenotalwaysinthesameplaceorspaceHavingakeenunderstandingofyouraudiencewillassistyouwithyourchoicesonwhatneedstobedevelopedSometimesthatmeansimagesforsocialmediaaonepageflierthatgivespeopleinformationonattendinganeventoranin-depthbrochure
Shareyourstoryandsuccess
53
6 ImplementyourstrategyKeepinmindstrategiescanbefluidSometimessomethingmaynotturnoutasexpectedevenifduringplanningsessionsitwasldquosupposedtordquoItisokYoucananalyzewhathappenedduringadebriefsessionandonceyoursurveysandothermeasurementshavebeenfinalized
Visualizations
Withtherightvisualizationyoucanreviewandshareinformationinsightsanalysisperformanceandtrendsallofwhichisvaluabletoacross-sectionofthepublicButwhichvisualizationisrightforyouTochoosetherighttypeofchartgraphdeterminethetypeofquestionyouwanttoanswerSomecommoncharttypesarebelowandmoreinformationcanbefoundhere
Exploreadditionalanalyticaltechniques
Duringthegoal-settingphaseyouestablishedoutputoutcomeandimplementationmeasuresBasedontheresultsofyourpilotyoumayneedtoadjustyourmeasuresWiththepilotdatainhandyoumayhavesomeideasofhowfutureanalysescouldbedifferentandpotentiallyimprovethequalityofyourprogramTakesometimetobrainstormotherrelevantdatasetsthatyoudidnothavethetimetopursueearlierincludingmessydepartmentaldatathatneedsmoreattentionorsourcesofpublicdatafromthestateandfederalgovernmentnonprofitsandevenprivatecompaniesthatcouldsupplementyourunderstandingofanissueUsingmachinelearningtechniquesyoumightbeabletobetter
Shareyourstoryandsuccess
54
predictdemandforhomelessservicesusingNOAAweatherforecastspreemptwatermainbreaksusingsoilstratigraphyorinvestineconomicdevelopmentbymonitoringrealestatelistingdata
Advancedanalyticaltechniquescanalsohelpyourimplementationrunmoreeffectivelybyintegratingdata-informedtargetinganddecisiontoolsintotheprocessByautomatingroutineworkflowsandsurfacingsubtletrendsthesetoolscanhelpyouiterateinamoreprecisemannerandprovidekeystrategicguidancetofront-lineemployeesthatwouldotherwisebeunavailableThesesortsofdata-science-enhancedprocessescanbeclassifiedasoneofsixprojecttypesasexploredfurtherbytheNewOrleansLANOLAlyticsteamFormoreguidanceonscopingasimilarprojecttheGovExadvancedanalyticsscopingguidegivesyouastep-by-stepoverviewofhowtobesuccessful
Usethespacebelowtolistyourideasforpossibleadvancedanalyticsprojectsbytype
Typeofproject Ideas
Findingtheneedleinthehaystack
Prioritizingforimpact
Earlywarningtools
Betterquickerdecisions
Optimizingresourceallocation
Experimentingforwhatworks
Keepthefeedbackloopgoing
GatherinformationonpublicsentimentaboutpolicychangethroughfocusgroupssurveysandpublicmeetingsledbyadvisorygroupmembersNotewhethernewinformationisavailablethatcanhelpanalyzeorcontextualizeyourprogress(egfederalorstatedatareports)Sharetheresultsofyourpilotandthepolicychangeprocesswithothercitiesandresearchers
Shareyourstoryandsuccess
55
WorkingwitharangeofevidencePresentlythereisalotofattentionandinvestmentinevidence-basedpolicymakingincludingthefederalbipartisanCommissiononEvidence-BasedPolicymakingthePewCharitableTrustsreportsandtoolkitsonHowStatesEngageinEvidence-BasedPolicymakingandtheLauraandJohnArnoldFoundationsupportedEvidence-BasedPolicymakingCollaborativeManyoftheseinitiativesfocusonrigorousevaluation--meaningrandomizedcontrolledtrials(RCT)orotherexperimentalmodels--asthebestkindofevidencetouseHoweverthisbodyofevidenceisofteninsufficientforthespecifickindsofquestionsthatgovernmentscanhavewhenmakingadecisionaboutpolicyInthisguideweproposeanapproachtosupportgovernmentsinincorporatingadditionaltypesofevidenceintotheirdecisionmaking
OnewaytounderstandarangeofevidenceistouseNestarsquosStandardsofEvidencetoframeanddefineevidenceTheBritishnonprofitNestaworkstoinjectpositivechangeintothewayUKgovernmentpoliciesandprogramsfunctionOneoftheirkeyareasofworkhasbeendevelopingWhatWorksCentresorganizationsthatcollectcommissionsynthesizeanddisseminateevidenceaboutinterventioneffectivenessacrossarangeofsocialimpactareasNestadevelopedthefollowingStandardsofEvidencetoprovideausefulwaytodocumenthowdifferentkindsofinformationdemonstratedifferentlevelsofevidence
Workingwitharangeofevidence
56
Theimportanceofpotentiallybiasedinformation
Onechallengepresentedbyanevidenceclassificationthatacknowledgesthevalueofnon-scientificformsofevidenceistofigureouthowtobalanceallofthepiecestogetherwhenyouusethemTounderstandhowtodothiswehavetothinkabouthowtopositionpotentiallybiasedinformationindecisionmaking
WhenwelookatinformationwhichhasnotbeenrigorouslyevaluatedweacceptthatthisinformationmightbebiasedThewholereasonweconductrigorousevaluationistoremovebiasfromourobservationsWeconductstructuredtrialstoproduceresultsthatareconsistentandgeneralizableIfafindingisunbiasedthisaccuracymdashthislackofbiasmdashcanbeprovedbecauseanyindependenttestercanreproduceitwhentheyfollowanidenticalprocess
YoucanlookatpotentiallybiasedinformationasjustinformationthatisearlieroninitspathtorefinementreplicationandverifiedaccuracyApieceofinformationbasedonanyoneindividualrsquosobservationsislikelytobebiasedbecauseitiscomingfromjustonesourceBiasisreducedbygatheringtogetherlotsofindependentsourcesofinformationandfiguringoutitsmostcommonattributes
MaybethebiasedinformationmakesaclaimthatistoolargemdashitsayssomethingistrueunderallconditionswhenitisreallyonlytrueundercertainconditionsorthataneffectofaninterventionisbiggerthanitstrueaverageorthataneffectlastsovertimewhenitdoesnotPerhapschangingtheclaimtobemorespecificmdashcriticallyreviewingtheinformationtoseeifspecificconditionsmustbemettoachievetheoutcomesmdashwouldmaketheclaimmoreaccurate
IfyoutestedyourclaimfounditdidnotholdupinsomecaseschangedittesteditagainuntilyourclaimrepeatedlyheldupthenyoucoulddemonstratethattheclaimwasnolongerbiasedbutaccurateThisprocessoftestingimprovingtestingandvalidatingiswhatwegetfromstructuredscientificstudy
Withindividualpiecesofinformation--crediblehypothesescasestudiesexpertopinion--youcannotassumethatkindofaccuracyAtthesametimetheseearlyobservationsarefoundationaltoanyreplicablescientificobservationsWewillnevergetanywhereifwedonotstartwiththesepiecesofinformationeveniftheymightbebiased
Combininglevelsofevidencetogetthemostdetailedinformationforyourcity
ItisespeciallyimportanttoacceptdifferentformsofevidencewhenyoufacequestionsthathavenotyetbeentestedbyindependentstudiesSomeofthemostimportantquestionsrelatetothingsthatareveryspecifictoyoursituationYoufaceaparticularcombinationof
Workingwitharangeofevidence
57
issuesactorstrendsandresourcesthatisultimatelydifferentfromeveryotherplaceintheworld
ThisisimportantbecauseaspolicypractitionersandnotscientistsyoucaremoreaboutyourspecificcasethanahypotheticalaveragegeneralcaseYoucarelessaboutthetotallikelyrangeofimpactsfromapolicythanhowthissingleimplementationwillworkYouwantinformationaboutpoliciesthatwillprovideyouwiththebestpossiblepredictionabouthowtheywillworkinyourcontext
ByputtingdifferentkindsofevidenceintothesameframeworkNestarsquosStandardsofEvidenceimplythatwhenyoucannotgetrigorouslytestedexperimentallyverifiedinformationitisappropriatetoworkwithwhatwedohaveThisisrelevantforthemunicipalimplementationcontextbecausenostudypulledfromaliteraturereviewwillaccountforallofthevariablesyoufaceinyourowncontext
Intheabsenceofhavingastudythatpredictshowaninterventionworksinyourspecificcontextyouwillneednon-scientificinformationtotrytofillinthosegapsYouwillneedtogatherstoriesfromresidentsstaffmembersandotherstakeholdersthatsitonLevels1or2ofthestandardsofevidence
Someofthekindsofinformationyouwillprobablygetmoreeasilyfrompeoplethanstudiesinclude
howpeopleincitieslikeyoursreactedtoaninterventioninpublicforumshowpeoplewhoimplementedaninterventioninacitylikeyoursfiguredouthowtogetitpassedhowthecitycommunicatedpubliclyabouttheapparentbenefitsanddrawbacksoftheinterventionthepoliticalacceptabilityoftheintervention
Youcanremainsimultaneouslyawarethattheinformationyouaregatheringthroughindividualstoriesisbiased--notgeneralizableinformedbyspecificperspectives--whilealsorecognizingthathavingsomeinformationaboutsomethingimportantisbetterthannoinformationEvidencefromthefirsttwoevidencelevelsareessentialbuildingblocksofknowledgeandperfectlyusefulwhenyoucannotgetdetailedunbiasedknowledge
Youhavetostartsomewhereandyoushouldalwaysstartwithevidenceyouhaveaboutyourveryspecificconditionsthatyouknowareimportantforsuccess
Workingwitharangeofevidence
58
Thankyou
Interviewswithcities
GovExheldsemi-structuredinterviewswithsixstaffmembersinfourcitiestoestablishtheinitialstepsoftheRoadmapWeaskedrespondentstotalkaboutaninitiativeorpolicythattheyhavebeenresponsibleforimplementingintheircityRespondentsfromthreecitiesspokeabouttheireffortstoaddresshomelessnessandrespondentsfromonecityspokeaboutblightAfterrespondentsidentifiedtheproblemtheyspokeabouthowthecityprioritizedthisissuethevariousstepsinvolvedandbarrierspresent
GovExconductedtextanalysisoftheseinterviewswhichrevealedthatpeoplewasthemostcommonlyusedtermduringtheseconversationswhendiscussingthestepstochangeaprogramorpolicyThisisathemethroughoutthisRoadmapThefirstsectionbeginswithidentifyingtherelevantpeopleTheconversationsaroundevidencearefocusedonhumanbehaviorratherthanmethodologyorstrengthofevidence
Thankyou
59
OncewehadadraftoftheRoadmapwefollowedupwithcitiesonanindividualbasistolearnmoreaboutwhatwouldbehelpfulWearegratefultothefollowingcitystafffortheinterviewsandindividualconversations
MichaelBaskinChiefPolicyOfficerChattanoogaTN
MelissaBridgesPerformanceandInnovationCoordinatorLittleRockAR
VictoriaCarreoacutenAdministrativeOfficerLasVegasNV
TyroneGrandstrandCommunityDevelopmentPlannerFargoND
TibLaughlinDirectorofGeneralServicesKansasCityKS
TimMorelandDirectorofPerformanceManagementandOpenDataChattanoogaTN
CharlesRosendahlManagementAnalystDentonTX
Thankyou
60
MelissaSiebenAssistantCountyAdministratorKansasCityKS
GinaThompsonDirectorofStrategicServicesLewisvilleTX
JHUFacultycollaboration
GovExalsoconvenedthecitycontactsandfacultypartnersinaseriesofAdvisoryGroupmeetingsThesemeetingsservedtoconnectcitiesandfacultytounderstandoneanothersperspective
StefanBaralBloombergSchoolofPublicHealth
AshleyBernerJHUSchoolofEducation
MelissaDavey-RothwellBloombergSchoolofPublicHealth
JonLinksBloombergSchoolofPublicHealth
TamarMendelsonBloombergSchoolofPublicHealth
KristinMmariBloombergSchoolofPublicHealth
KeshiaPollackPorterBloombergSchoolofPublicHealth
JoshSharfsteinBloombergSchoolofPublicHealth
LizStuartBloombergSchoolofPublicHealth
LindsayThompsonCareyBusinessSchool
DanielWebsterBloombergSchoolofPublicHealth
Partnersindataandevidence-basedpolicy
Wealsospokewithpartnerswhoareexpertsinthefieldofevidence-basedpolicywhosharedtheirknowledgeandresourcesofcurrentandfuturethinkinginevidence-informedthinking
WendyBarrenoAssociateDirectorWilsonSheehanLabforEconomicOpportunitiesUniversityofNotreDame
TiffanyDavenportDirectorofDataPartnershipsBaltimoresPromise
TomiHiersDirectorBaltimoreCivicSiteAnnieECaseyFoundation
ChrisKingsleySeniorAssociateAnnieECaseyFoundation
KathrynOliverAssociateProfessorLondonSchoolofHygieneandTropicalMedicine
Thankyou
61
KristenWatkinsSeniorPolicyAssociateJ-PAL
GovExParticipants
GovExAnalystMiriamMcKinneyanalyzedtheinterviewtranscriptsandconductedtextanalysisSeniorImplementationAdvisorEmilyShawwillintegrateRoadmapactivitieswithadditionalservicedeliveryactivitiesatGovEx
Thankyou
62
ReferencesAbrahamKGampHaskinsR(2017)Thepromiseofevidence-basedpolicymakingReportofthecommissiononevidence-basedpolicymakingWashingtonDCCommissiononEvidence-basedPolicymakingBigmove2Extendingthegreen(nd)Retrievedfromhttpswwwcityoflewisvillecomdoing-businesslewisville-2025big-move-2-extending-the-greenBatesMAampGlennersterR(2017)ThegeneralizabilitypuzzleRetrievedfromhttpsssirorgarticlesentrythe_generalizability_puzzleBuildwithnotfor(2018January)RetrievedfromhttpsbuildwithnotforwordpresscomCairneyPampOliverK(2017)Evidence-basedpolicymakingisnotlikeevidence-basedmedicinesohowfarshouldyougotobridgethedividebetweenevidenceandpolicyHealthResearchPolicyandSystemsdoi101186s12961-017-0192-xCairneyPOliverKampWellsteadA(2016)TobridgethedividebetweenevidenceandpolicyReduceambiguityasmuchasuncertaintyPublicAdministrationReview76(3)399-402Retrievedfromhttpdxdoiorg101111puar12555Centreforpublicimpact(nd)RetrievedfromhttpswwwcentreforpublicimpactorgCernaL(2013)ThenatureofpolicychangeandimplementationAreviewofdifferenttheoreticalapproachesOrganisationforEconomicCo-OperationandDevelopmentRetrievedfromhttpswwwoecdorgeduceriTheNatureofPolicyChangeandImplementationpdfColoradoSprings(nd)Retrievedfromhttpscoloradospringsgovcommunicationspagewhat-expect-when-you-speakupContinuumofcare(coc)program(nd)RetrievedfromhttpswwwhudexchangeinfoprogramscocCouncilpresidentkicksoff$12millionyouthfund(nd)Retrievedfromhttpwwwbaltimorecitycouncilcomcontentcouncil-president-kicks-12-million-youth-fundDurlakJAampDuPreEP(2008)ImplementationmattersAreviewofresearchontheinfluenceofimplementationonprogramoutcomesandthefactorsaffectingimplementationAmericanJournalofCommunityPsychology41327-350Retrievedfromhttpswwwncbinlmnihgovpubmed18322790Fishbonediagram(ishikawa)(nd)Retrievedfromhttpasqorglearn-about-qualitycause-analysis-toolsoverviewfishbonehtmlFivewhysGettingtotherootofaproblemquickly(nd)RetrievedfromhttpswwwmindtoolscompagesarticlenewTMC_5Whtm
References
63
FligsteinNampMcAdamD(2011)TowardageneraltheoryofstrategicactionfieldsSociologicalTheory29(1)1-26RetrievedfromhttpsociologyberkeleyedusitesdefaultfilesfacultyfligsteinFligstein20McAdam20Soc20Theory20PaperpdHeinkelP(2018January12)CitymullspermitsforbikesharingcompaniesRetrievedfromhttpwwwdentonrccomnewsdenton20180112city-mulls-permits-bike-sharing-companiesHowisareamedianincomecalculated(2012October)Retrievedfromhttpswwwhudexchangeinfofaqs843how-is-area-median-income-calculatedLampkinLMampHatryHP(2003)KeystepsinoutcomemanagementUrbanInstituteRetrievedfromhttpswwwurbanorgresearchpublicationkey-steps-outcome-managementMarshDampSmithM(2000)UnderstandingpolicynetworksTowardsadialecticalapproachPoliticalStudies484-21Retrievedfromhttpjournalssagepubcomdoiabs1011111467-924800247McCannL(nd)RetrievedfromhttplaurenellencomMoultonSampSandfortJR(2016)ThestrategicactionfieldframeworkforpolicyimplementationresearchPolicyStudiesJournal45(1)Retrievedfromhttponlinelibrarywileycomstore101111psj12147assetpsj12147pdfv=1ampt=jdkhhcp4amps=e184c625e1855a6b1615ccb99ee27b1c0b3212d8PaynterB(2018January1)What100citiesarelearningfromeachotherbysharingtheirdataRetrievedfromhttpswwwfastcompanycom40518004what-100-cities-are-learning-from-each-other-by-sharing-their-dataPeakacademy(nd)Retrievedfromhttpswwwdenvergovorgcontentdenvergovenmayors-officeprograms-initiativespeak-performancepeak-academyhtmlPinnockHBarwickMCarpenterCREldridgeSGrandesGGriffithsCJRycroft-MaloneJMeissnerPMurrayEPatelASheikhAampTaylorSJC(2017)Standardsforreportingimplementationstudies(stari)ExplanationandelaborationdocumentBMJOpenRetrievedfromhttpbmjopenbmjcomcontentbmjopen74e013318fullpdfProctorESilmereHRaghavanRHovmandPAaronsGBungerAGriffeyRampHensleyM(2011)OutcomesforimplementationresearchConceptualdistinctionsmeasurementchallengesandresearchagendaAdministrationandPolicyinMentalHealth38(2)65-76doi101007s10488-010-0319-7TandonSD(2018)ProgressincommunityhealthpartnershipsResearcheducationandactionRetrievedfromhttpswwwpressjhuedujournalsprogress-community-health-partnerships-research-education-and-actionProvanKGampKenisP(2007)ModesofnetworkgovernanceStructuremanagementandeffectivenessJournalofPublicAdministrationResearchandTheory
References
64
18(2)229-252Retrievedfromhttpsacademicoupcomjpartarticle182229935895SabatierPA(1988)Anadvocacycoalitionframeworkofpolicychangeandtheroleofpolicy-orientedlearningthereinPolicySciences21(3)129-168Retrievedfromhttpwwwjstororgstable4532139SandstromAampCarlssonL(2008)TheperformanceofpolicynetworksTherelationbetweennetworkstructureandnetworkperformancePolicyStudiesJournal36497-524Retrievedfromhttponlinelibrarywileycomdoi101111j1541-0072200800281xabstractSchwartzMampWilsonE(nd)WhocanaffordtoliveinahomeAlookatdatafromthe2006americancommunitysurveyRetrievedfromhttpswwwcensusgovhousingcensuspublicationswho-can-affordpdfShearerJCAbelsonAKouyateBLavisJNampWaltG(2016)WhydopolicieschangeInstitutionsinterestsideasandnetworksinthreecasesofpolicyreformHealthPolicyandPlanning31(9)1200-1211Retrievedfromhttpsdoiorg101093heapolczw052ThelessonsofadministrativedataHigh-profilepolicy-relevantresearchpoweredbyadministrativedata(nd)Retrievedfromhttpswwwpovertyactionlaborgsitesdefaultfilesresources20161114-The-Lessons-of-Administrative-DatapdfToolsandotherneedsforqualitativeandquantitativemethods(2015February)AgencyforHealthcareResearchandQualityRetrievedfromhttpwwwahrqgovprofessionalsquality-patient-safetytalkingqualityassesstoolshtml
WeaverRCampBagchi-SenS(2013)SpatialanalysisofurbandeclineThegeographyofblightAppliedGeography4061-70
References
65
UniversityPartners
Name Title Affiliation Area(s)ofexpertise
Contactinformation
Examplesofworkingwithnon-academicdecisionmakers
WendyPBarreno
AssociateDirector
WilsonSheehanLabforEconomicOpportunitiesUniversityofNotreDame
LEOsfocusareasincludehealtheducationhousingchildwelfareeconomicopportunitycriminaljustice
wbarrenonduduorleondedu
MelissaDavey-RothwellPhD
AssociateScientist
JohnsHopkinsBloombergSchoolofPublicHealth
Health mdavey1jhuedu410-614-5854
BaltimoreCityHealthDepartmentLeadersMarylandDepartmentofHealthAddictionRecoveryServicesinDCExecutiveDirectorsofcommunityorganizationsacrosstheUSthatfocusonHIVSTIsubstanceabuseandimplementationscience
ElizabethStuart
AssociateDeanforEducationandProfessorDepartmentsofMentalHealthBiostatisticsandHealthPolicyand
JohnsHopkinsBloombergSchoolofPublicHealth
Educationsocialpolicystatisticalmethods
estuartjhuedu
Resources
66
Management
Design-ThinkingResources
EngagementGuideforCivicTech
Journeymap
PerformanceManagementResources
DenverPeakAcademy
Proxymeasuresguide
Talkingaboutperformancewithoutcomplicatingit(video)
SettingPerformanceTargetsguide
Datavisualizationguide
Performancemanagementgettingstartedguide
Exercises
Fishboneexercise
5Whys
AdministrativeData
TheLessonsofAdministrativeData(JPAL)
EvaluationResources
Quantitativeandqualitativemethods(USDepartmentofHealthandHumanServicesAgencyforHealthcareResearchandQuality)
KeyStepsinOutcomeManagement(UrbanInstitute)
RCT-YES
Sageresearchmethods
BestPracticesforMixedMethodsResearchintheHealthSciences
ACCORDChecklist
Pilotandimplementationplanning
Benchmarkingexamplesandproxymeasures
Seattleopendataplaybook
Resources
67
Datasciencescopingguide
Resources
68
AboutGovExGovExhelpsgovernmentsusedataandevidencetoimproveresidentsrsquolivesWesupportgovernmentsinbuildingtheirinternalcapacitytooperatesustainableanalyticspracticeswhichimprovecommunityoutcomesinfieldsfrompublicsafetytoeconomicdevelopmentGovExhasaproventrackrecordofworkinginconcertwithpartnerstoprovidetechnicalassistanceandtrainingatscale
SinceourlaunchinApril2015GovExhas
Trainedover2000governmentemployeesinpersonandonlineintopicsincludingdatamanagementperformancemanagementopendatacommunityengagementandanalytics
Providedtechnicalassistancetomorethan100localgovernmentsacrosstheglobeimprovingtheiruseofdatatoachieveefficienciesinservicedeliveryandbetteroutcomesforresidents
Assessedthecapacityofmorethan130governmentalorganizationstodeterminehowleadersstaffandexternalstakeholdersmanageandusedata
Localgovernmentsareincreasinglyonthefrontlinesofaddressinglargescaleproblemswithglobalimplicationssuchasclimatechangepublicsafetyandeconomicinequality
Whileevidence-basedinterventionstohelpsolvethoseproblemsareavailablelocalgovernmentstaffhavelimitedtimeandcapacitytofindandimplementresearch-basedsolutionsAsaresultcitiescontinuetoworkthewaytheyalwayshave--makingincrementalimprovementstoprogramsbutnotalwayscreatingchangeatasystem-level
ThroughresourcedevelopmentandcollaborationwithJHUfacultyGovExissupportingcityleadersinusingdataandevidencetomakedecisionsandcreatinglastingrelationshipswithresearchers
TheRoadmapwasdevelopedbyGovExaspartofitspartnershipwithBloombergPhilanthropiesWhatWorksCitiesinitiative
AboutGovEx
69
- Title page
- Introduction and purpose of the Roadmap
- Using the Roadmap
-
- Gather information about the problem
- Strategically engage key partners
- Review the evidence and collect details
- Decide on a policy to implement based on local factors
- Establish a framework to monitor progress
- Assess data accessibility quality and security
- Design implementation and evaluation
- Share your story and success
-
- Working with a range of evidence
- Thank you
- References
- Resources
- About GovEx
-
outcomeindicatorsbasedonvariablesoutsideoftheCityscontrolForinstanceachangeinfederaltradepolicycanhaveanimpactemploymentratesinyourCityoutsideofyourCitysnewjobtrainingandplacementprogram
ToaddresscausalityyoumustcomparetheoutcomesofindividualswhoreceivedtheinterventionwiththosewhodidnotWithoutacomparisongroupofsimilarindividualswhowerenotexposedtotheinterventionitisimpossibletoknowwhichfactorsareresponsiblefortheoutcomesRandomizedControlTrials(RCTs)inwhichindividualsarerandomlyassignedtotreatmentandcontrolgroupsarethegoldstandardinprovingcausalityIfyoucandemonstratethatoutcomesimprovedforthosewhoreceivedtheinterventionanddidnotimproveatallorimprovedlessforacomparisongroupyoucanbesomewhatcertainthattheinterventionwasthecause
Examplesofothernon-causalimpactevaluationsthatarerigorousandreliableincludecertainquasi-experimentalstudiesusingcomparisongroupdesignssmartinterruptedtimeseriesanalysesthatmodeltrendsovertimeinwaysthatallowestimationofinterventioneffectswelldesignedpre-posttestanalysesneedsassessmentsimplementationstudiesetcFormoreinformationondifferenttypesofevaluationsseetheSageResearchMethodswebsite
Lookfordifferencesingroups
BuildintimetoalsoexaminethepilotdatatoseeifgroupsareexperiencingdifferentresultsReviewingperformancebasedonoutcomemeasuresforapopulationinaggregatemayhidethesubtlewaysthatsomepeoplemaybeaffecteddifferentlysoitisnecessarytocarefullyanalyzethedatafordisparateoutcomesForinstancetheMovingtoOpportunity(MTO)studythatexaminedtheimpactofhousingvouchersonfamilieswholivedinhigh-povertyneighborhoodsfoundthatchildrenwhomovedbeforetheageof13hadbettereducationandhigherincomesasadultsHoweveradultsdidnotseeimprovededucationandemploymentoutcomes
Disaggregatingdataintomorespecificcategoriesofraceandethnicitycanhelpunderstandtrendsanddisparitiesinamoregranularwaywhichcanhelpdecisionmakerstargetpoliciesandinterventionsinamoreequitableway
EarlierintheRoadmapweworkedtoidentifystakeholdersandwhomightbepositivelyandnegativelyimpactedbyachangeinpolicyUsetheresultsofthatexercisetoinvolvethecommunityininterpretingdataanalysisandtogatherqualitativefeedbackfromthecommunitytocallattentiontoissuesForinstanceanincreaseinyouthdetentionmaybeexplainedbyinefficienciesorlackofaccesstotransportation
Designimplementationandevaluation
49
Designimplementationandevaluation
50
Shareyourstoryandsuccess
Sharetheresults
Storytelling
AnintegralstepinyourstrategyissharingtheresultsofyourpilotorpolicychangewithyourcommunityThisincludesdevelopingmessagingaroundtheworkofyourorganizationandcreatingaplantodisseminateyourstoriesWhytellstoriesaboutyourwork
Informandorunderstandasubjectorsituation
Giveneededcontextofdataorbitsofinformation
Relatablestoriesarememorableforreadersviewers
ReadersorviewerswillseetheinformationasmorecompellingwhenthereisademonstratedandrelatablechangeorimpactonthemselvesortheirfriendsneighborsandcolleaguesStoriesareaboutpeopleandgovernmentisfundamentallyabouthelpingpeopleSotellthestoriesofthepeopleyouandyourcitycolleagueshelped
Keepthesetipsinmindasyoudevelopyourstorywithyourteam
WhatwereyourfindingsandwhyshouldanyonecareTellyouraudiencehowtheproblemwassolvedWhathastheimpactbeenonresidentsandcitystafftodateWhatwasthesuccessandweretheremultiplesuccessesalongthewayIfyouhavepreliminaryresultsordatashowingimprovementtosharethisisthetimetodoit
WhowasinvolvedandimpactedReviewthelistofstakeholdersHowwereeachofthemimpactedbytheprogramHumanizeyourstoryandimpactbyaddingadditionalvoicesThesevoicescanbeoneofthefollowing
Apersonorfamilywhowaspositivelyaffectedbythechangesuccess
Aninvolvedthird-partydescribingboththeproblemandthesolution(ieanorganizationnonprofitprovidersbusinesstradeassociationsportsteametc)whoispartofthecommunitythatthegovernmentworkedwithonthechangesuccess
Someonefromthegovernmentitselfwhocanspeakaboutthechangesuccess
TellmoreaboutyourcityDescribeyourcityincludingpopulationdemographicsindustriesetcWhatareitschallengesWhatisitknownforWhatmakesituniqueHowisitsimilarordifferentfromothercitiesintheUS
Shareyourstoryandsuccess
51
DescribeyourcityrsquoschallengeWriteoutthechallengeyourcityisfacing(asdefinedinyourproblemstatement)HowdoesitaffectresidentsHowdoesthisproblemaffecttheworkofcitystaffmembers
AddthedetailsProvidedetailsonyourappliedresearchstrategyTalkaboutwhatyourcityhasdonenthepasttoresolvetheissueDiscusshowyourstrategyhaschangedwhenyoubegantoadoptthisapproachWasmoneysavedorinvested
Usethisstorymapexercisetoorganizeyourthoughtsandguaranteethatyouincludealloftheelementsofatraditionalstory
AfteryoudevelopyourmessagingcreateacommunicationsmarketingandoutreachstrategydesignedtospreadyourmessageYourstrategyshouldwalkyouthroughthefollowingkeyactivities
1 IdentifyandprofileyouraudienceWhencreatinganddisseminatingyourmessageitisalwaysgoodtothinkaboutyouraudienceWhoexactlywouldbeinterestedinwhatyouhavetosayThegeneralpopulationofthecityPeoplewhoworkinnonprofitsBusinessownersItmattersbecausenoteveryonewholivesinacitymaybethetargetoftheinformationBusinessownersmaybemoreinterestedindatashowingeffectsonthebusinessdistrictthanothersandsoonWewillcoverthislaterbutthisstepwillalsohelpyoutothinkofthemosteffectivewaytoreachyouraudience
2 Defineyourcalltoaction(ifyouhaveone)SpendsometimeconsideringwhyyouareinterestedintellingyourstoryDoyousimplywanttoupdatethepubliconyourprogressorcelebrateyoursuccessesAreyouinterestedinyouraudiencechangingitsbehaviororsupportingaspecificinitiativebeinglaunchedbyyourorganizationIfyouwanttohaveyouraudienceactorbehaveinacertainwaybespecificandincorporateitinyourmessaging
Shareyourstoryandsuccess
52
3 SelectperformancemeasuresThinkabouthowyouwoulddetermineifyourcampaignissuccessfulandselectappropriateperformancemeasuresYouwillrevisitthesemeasuresattheendofyourcampaigncommunicationsplanimplementationtodeterminewhetheryouneedtomodifyyourcommunicationsandoutreachstrategyforthefuture
4 MethodstocommunicateyourstoryTherearemanywaystocommunicatesuccessofyourprojectHerearesomesuggestionsonreachingaudiencesItisimportanttodeterminewhichmethodsarethemosteffectiveforyourdesiredaudiencetoreceiveinformationDoyourresearch(PewResearchCenterhasexcellentstudiesonthereachofmediadigitalmediaandmarketing)ForexampleifseniorsareyourtargetpopulationandtheyarelessinclinedtouseInstagram(thatistrue)youmaywanttoconsiderFacebookorthenightlylocalnewsbothofwhichareusedmoreoftenbyseniorsBelowisalistofsomemethodsandtoolsyoucanuse
i Pressreleasestolocalnewspapersandtvstations
ii ShortsmartphonevideosAgreatwaytoengageanaudienceisthroughvideoAskonequestionofsomeonewhoispartofthestoryMaybethereisasocialworkerwhocandiscusshungerinyourcityThesevideoscanbereusedonmultipleplatforms
iii Long-formwriting
iv TellingindividualstoriesthroughaseriesofphotosandshortnarrativesforpublicationonFacebookorInstagram
v Op-edsopinionpiecesplacedinthelocalpaper
vi Emailupdatestoyourlist
vii Connectingwithappropriateorganizationsthatareinvestedinthetopicandworkwiththecommunityandaskthemtodisseminateinformationtotheirmembers
viii EmployingldquoWordofMouthrdquo(WOM)tacticsbycreatingtownhallsorfreeeventsforresidentstoattendintheirownneighborhoods
5 DeterminethetypeofcollateralUsuallycampaignsthattargetresidentsrequireseveralavenuestoreachthemPeoplearenotalwaysinthesameplaceorspaceHavingakeenunderstandingofyouraudiencewillassistyouwithyourchoicesonwhatneedstobedevelopedSometimesthatmeansimagesforsocialmediaaonepageflierthatgivespeopleinformationonattendinganeventoranin-depthbrochure
Shareyourstoryandsuccess
53
6 ImplementyourstrategyKeepinmindstrategiescanbefluidSometimessomethingmaynotturnoutasexpectedevenifduringplanningsessionsitwasldquosupposedtordquoItisokYoucananalyzewhathappenedduringadebriefsessionandonceyoursurveysandothermeasurementshavebeenfinalized
Visualizations
Withtherightvisualizationyoucanreviewandshareinformationinsightsanalysisperformanceandtrendsallofwhichisvaluabletoacross-sectionofthepublicButwhichvisualizationisrightforyouTochoosetherighttypeofchartgraphdeterminethetypeofquestionyouwanttoanswerSomecommoncharttypesarebelowandmoreinformationcanbefoundhere
Exploreadditionalanalyticaltechniques
Duringthegoal-settingphaseyouestablishedoutputoutcomeandimplementationmeasuresBasedontheresultsofyourpilotyoumayneedtoadjustyourmeasuresWiththepilotdatainhandyoumayhavesomeideasofhowfutureanalysescouldbedifferentandpotentiallyimprovethequalityofyourprogramTakesometimetobrainstormotherrelevantdatasetsthatyoudidnothavethetimetopursueearlierincludingmessydepartmentaldatathatneedsmoreattentionorsourcesofpublicdatafromthestateandfederalgovernmentnonprofitsandevenprivatecompaniesthatcouldsupplementyourunderstandingofanissueUsingmachinelearningtechniquesyoumightbeabletobetter
Shareyourstoryandsuccess
54
predictdemandforhomelessservicesusingNOAAweatherforecastspreemptwatermainbreaksusingsoilstratigraphyorinvestineconomicdevelopmentbymonitoringrealestatelistingdata
Advancedanalyticaltechniquescanalsohelpyourimplementationrunmoreeffectivelybyintegratingdata-informedtargetinganddecisiontoolsintotheprocessByautomatingroutineworkflowsandsurfacingsubtletrendsthesetoolscanhelpyouiterateinamoreprecisemannerandprovidekeystrategicguidancetofront-lineemployeesthatwouldotherwisebeunavailableThesesortsofdata-science-enhancedprocessescanbeclassifiedasoneofsixprojecttypesasexploredfurtherbytheNewOrleansLANOLAlyticsteamFormoreguidanceonscopingasimilarprojecttheGovExadvancedanalyticsscopingguidegivesyouastep-by-stepoverviewofhowtobesuccessful
Usethespacebelowtolistyourideasforpossibleadvancedanalyticsprojectsbytype
Typeofproject Ideas
Findingtheneedleinthehaystack
Prioritizingforimpact
Earlywarningtools
Betterquickerdecisions
Optimizingresourceallocation
Experimentingforwhatworks
Keepthefeedbackloopgoing
GatherinformationonpublicsentimentaboutpolicychangethroughfocusgroupssurveysandpublicmeetingsledbyadvisorygroupmembersNotewhethernewinformationisavailablethatcanhelpanalyzeorcontextualizeyourprogress(egfederalorstatedatareports)Sharetheresultsofyourpilotandthepolicychangeprocesswithothercitiesandresearchers
Shareyourstoryandsuccess
55
WorkingwitharangeofevidencePresentlythereisalotofattentionandinvestmentinevidence-basedpolicymakingincludingthefederalbipartisanCommissiononEvidence-BasedPolicymakingthePewCharitableTrustsreportsandtoolkitsonHowStatesEngageinEvidence-BasedPolicymakingandtheLauraandJohnArnoldFoundationsupportedEvidence-BasedPolicymakingCollaborativeManyoftheseinitiativesfocusonrigorousevaluation--meaningrandomizedcontrolledtrials(RCT)orotherexperimentalmodels--asthebestkindofevidencetouseHoweverthisbodyofevidenceisofteninsufficientforthespecifickindsofquestionsthatgovernmentscanhavewhenmakingadecisionaboutpolicyInthisguideweproposeanapproachtosupportgovernmentsinincorporatingadditionaltypesofevidenceintotheirdecisionmaking
OnewaytounderstandarangeofevidenceistouseNestarsquosStandardsofEvidencetoframeanddefineevidenceTheBritishnonprofitNestaworkstoinjectpositivechangeintothewayUKgovernmentpoliciesandprogramsfunctionOneoftheirkeyareasofworkhasbeendevelopingWhatWorksCentresorganizationsthatcollectcommissionsynthesizeanddisseminateevidenceaboutinterventioneffectivenessacrossarangeofsocialimpactareasNestadevelopedthefollowingStandardsofEvidencetoprovideausefulwaytodocumenthowdifferentkindsofinformationdemonstratedifferentlevelsofevidence
Workingwitharangeofevidence
56
Theimportanceofpotentiallybiasedinformation
Onechallengepresentedbyanevidenceclassificationthatacknowledgesthevalueofnon-scientificformsofevidenceistofigureouthowtobalanceallofthepiecestogetherwhenyouusethemTounderstandhowtodothiswehavetothinkabouthowtopositionpotentiallybiasedinformationindecisionmaking
WhenwelookatinformationwhichhasnotbeenrigorouslyevaluatedweacceptthatthisinformationmightbebiasedThewholereasonweconductrigorousevaluationistoremovebiasfromourobservationsWeconductstructuredtrialstoproduceresultsthatareconsistentandgeneralizableIfafindingisunbiasedthisaccuracymdashthislackofbiasmdashcanbeprovedbecauseanyindependenttestercanreproduceitwhentheyfollowanidenticalprocess
YoucanlookatpotentiallybiasedinformationasjustinformationthatisearlieroninitspathtorefinementreplicationandverifiedaccuracyApieceofinformationbasedonanyoneindividualrsquosobservationsislikelytobebiasedbecauseitiscomingfromjustonesourceBiasisreducedbygatheringtogetherlotsofindependentsourcesofinformationandfiguringoutitsmostcommonattributes
MaybethebiasedinformationmakesaclaimthatistoolargemdashitsayssomethingistrueunderallconditionswhenitisreallyonlytrueundercertainconditionsorthataneffectofaninterventionisbiggerthanitstrueaverageorthataneffectlastsovertimewhenitdoesnotPerhapschangingtheclaimtobemorespecificmdashcriticallyreviewingtheinformationtoseeifspecificconditionsmustbemettoachievetheoutcomesmdashwouldmaketheclaimmoreaccurate
IfyoutestedyourclaimfounditdidnotholdupinsomecaseschangedittesteditagainuntilyourclaimrepeatedlyheldupthenyoucoulddemonstratethattheclaimwasnolongerbiasedbutaccurateThisprocessoftestingimprovingtestingandvalidatingiswhatwegetfromstructuredscientificstudy
Withindividualpiecesofinformation--crediblehypothesescasestudiesexpertopinion--youcannotassumethatkindofaccuracyAtthesametimetheseearlyobservationsarefoundationaltoanyreplicablescientificobservationsWewillnevergetanywhereifwedonotstartwiththesepiecesofinformationeveniftheymightbebiased
Combininglevelsofevidencetogetthemostdetailedinformationforyourcity
ItisespeciallyimportanttoacceptdifferentformsofevidencewhenyoufacequestionsthathavenotyetbeentestedbyindependentstudiesSomeofthemostimportantquestionsrelatetothingsthatareveryspecifictoyoursituationYoufaceaparticularcombinationof
Workingwitharangeofevidence
57
issuesactorstrendsandresourcesthatisultimatelydifferentfromeveryotherplaceintheworld
ThisisimportantbecauseaspolicypractitionersandnotscientistsyoucaremoreaboutyourspecificcasethanahypotheticalaveragegeneralcaseYoucarelessaboutthetotallikelyrangeofimpactsfromapolicythanhowthissingleimplementationwillworkYouwantinformationaboutpoliciesthatwillprovideyouwiththebestpossiblepredictionabouthowtheywillworkinyourcontext
ByputtingdifferentkindsofevidenceintothesameframeworkNestarsquosStandardsofEvidenceimplythatwhenyoucannotgetrigorouslytestedexperimentallyverifiedinformationitisappropriatetoworkwithwhatwedohaveThisisrelevantforthemunicipalimplementationcontextbecausenostudypulledfromaliteraturereviewwillaccountforallofthevariablesyoufaceinyourowncontext
Intheabsenceofhavingastudythatpredictshowaninterventionworksinyourspecificcontextyouwillneednon-scientificinformationtotrytofillinthosegapsYouwillneedtogatherstoriesfromresidentsstaffmembersandotherstakeholdersthatsitonLevels1or2ofthestandardsofevidence
Someofthekindsofinformationyouwillprobablygetmoreeasilyfrompeoplethanstudiesinclude
howpeopleincitieslikeyoursreactedtoaninterventioninpublicforumshowpeoplewhoimplementedaninterventioninacitylikeyoursfiguredouthowtogetitpassedhowthecitycommunicatedpubliclyabouttheapparentbenefitsanddrawbacksoftheinterventionthepoliticalacceptabilityoftheintervention
Youcanremainsimultaneouslyawarethattheinformationyouaregatheringthroughindividualstoriesisbiased--notgeneralizableinformedbyspecificperspectives--whilealsorecognizingthathavingsomeinformationaboutsomethingimportantisbetterthannoinformationEvidencefromthefirsttwoevidencelevelsareessentialbuildingblocksofknowledgeandperfectlyusefulwhenyoucannotgetdetailedunbiasedknowledge
Youhavetostartsomewhereandyoushouldalwaysstartwithevidenceyouhaveaboutyourveryspecificconditionsthatyouknowareimportantforsuccess
Workingwitharangeofevidence
58
Thankyou
Interviewswithcities
GovExheldsemi-structuredinterviewswithsixstaffmembersinfourcitiestoestablishtheinitialstepsoftheRoadmapWeaskedrespondentstotalkaboutaninitiativeorpolicythattheyhavebeenresponsibleforimplementingintheircityRespondentsfromthreecitiesspokeabouttheireffortstoaddresshomelessnessandrespondentsfromonecityspokeaboutblightAfterrespondentsidentifiedtheproblemtheyspokeabouthowthecityprioritizedthisissuethevariousstepsinvolvedandbarrierspresent
GovExconductedtextanalysisoftheseinterviewswhichrevealedthatpeoplewasthemostcommonlyusedtermduringtheseconversationswhendiscussingthestepstochangeaprogramorpolicyThisisathemethroughoutthisRoadmapThefirstsectionbeginswithidentifyingtherelevantpeopleTheconversationsaroundevidencearefocusedonhumanbehaviorratherthanmethodologyorstrengthofevidence
Thankyou
59
OncewehadadraftoftheRoadmapwefollowedupwithcitiesonanindividualbasistolearnmoreaboutwhatwouldbehelpfulWearegratefultothefollowingcitystafffortheinterviewsandindividualconversations
MichaelBaskinChiefPolicyOfficerChattanoogaTN
MelissaBridgesPerformanceandInnovationCoordinatorLittleRockAR
VictoriaCarreoacutenAdministrativeOfficerLasVegasNV
TyroneGrandstrandCommunityDevelopmentPlannerFargoND
TibLaughlinDirectorofGeneralServicesKansasCityKS
TimMorelandDirectorofPerformanceManagementandOpenDataChattanoogaTN
CharlesRosendahlManagementAnalystDentonTX
Thankyou
60
MelissaSiebenAssistantCountyAdministratorKansasCityKS
GinaThompsonDirectorofStrategicServicesLewisvilleTX
JHUFacultycollaboration
GovExalsoconvenedthecitycontactsandfacultypartnersinaseriesofAdvisoryGroupmeetingsThesemeetingsservedtoconnectcitiesandfacultytounderstandoneanothersperspective
StefanBaralBloombergSchoolofPublicHealth
AshleyBernerJHUSchoolofEducation
MelissaDavey-RothwellBloombergSchoolofPublicHealth
JonLinksBloombergSchoolofPublicHealth
TamarMendelsonBloombergSchoolofPublicHealth
KristinMmariBloombergSchoolofPublicHealth
KeshiaPollackPorterBloombergSchoolofPublicHealth
JoshSharfsteinBloombergSchoolofPublicHealth
LizStuartBloombergSchoolofPublicHealth
LindsayThompsonCareyBusinessSchool
DanielWebsterBloombergSchoolofPublicHealth
Partnersindataandevidence-basedpolicy
Wealsospokewithpartnerswhoareexpertsinthefieldofevidence-basedpolicywhosharedtheirknowledgeandresourcesofcurrentandfuturethinkinginevidence-informedthinking
WendyBarrenoAssociateDirectorWilsonSheehanLabforEconomicOpportunitiesUniversityofNotreDame
TiffanyDavenportDirectorofDataPartnershipsBaltimoresPromise
TomiHiersDirectorBaltimoreCivicSiteAnnieECaseyFoundation
ChrisKingsleySeniorAssociateAnnieECaseyFoundation
KathrynOliverAssociateProfessorLondonSchoolofHygieneandTropicalMedicine
Thankyou
61
KristenWatkinsSeniorPolicyAssociateJ-PAL
GovExParticipants
GovExAnalystMiriamMcKinneyanalyzedtheinterviewtranscriptsandconductedtextanalysisSeniorImplementationAdvisorEmilyShawwillintegrateRoadmapactivitieswithadditionalservicedeliveryactivitiesatGovEx
Thankyou
62
ReferencesAbrahamKGampHaskinsR(2017)Thepromiseofevidence-basedpolicymakingReportofthecommissiononevidence-basedpolicymakingWashingtonDCCommissiononEvidence-basedPolicymakingBigmove2Extendingthegreen(nd)Retrievedfromhttpswwwcityoflewisvillecomdoing-businesslewisville-2025big-move-2-extending-the-greenBatesMAampGlennersterR(2017)ThegeneralizabilitypuzzleRetrievedfromhttpsssirorgarticlesentrythe_generalizability_puzzleBuildwithnotfor(2018January)RetrievedfromhttpsbuildwithnotforwordpresscomCairneyPampOliverK(2017)Evidence-basedpolicymakingisnotlikeevidence-basedmedicinesohowfarshouldyougotobridgethedividebetweenevidenceandpolicyHealthResearchPolicyandSystemsdoi101186s12961-017-0192-xCairneyPOliverKampWellsteadA(2016)TobridgethedividebetweenevidenceandpolicyReduceambiguityasmuchasuncertaintyPublicAdministrationReview76(3)399-402Retrievedfromhttpdxdoiorg101111puar12555Centreforpublicimpact(nd)RetrievedfromhttpswwwcentreforpublicimpactorgCernaL(2013)ThenatureofpolicychangeandimplementationAreviewofdifferenttheoreticalapproachesOrganisationforEconomicCo-OperationandDevelopmentRetrievedfromhttpswwwoecdorgeduceriTheNatureofPolicyChangeandImplementationpdfColoradoSprings(nd)Retrievedfromhttpscoloradospringsgovcommunicationspagewhat-expect-when-you-speakupContinuumofcare(coc)program(nd)RetrievedfromhttpswwwhudexchangeinfoprogramscocCouncilpresidentkicksoff$12millionyouthfund(nd)Retrievedfromhttpwwwbaltimorecitycouncilcomcontentcouncil-president-kicks-12-million-youth-fundDurlakJAampDuPreEP(2008)ImplementationmattersAreviewofresearchontheinfluenceofimplementationonprogramoutcomesandthefactorsaffectingimplementationAmericanJournalofCommunityPsychology41327-350Retrievedfromhttpswwwncbinlmnihgovpubmed18322790Fishbonediagram(ishikawa)(nd)Retrievedfromhttpasqorglearn-about-qualitycause-analysis-toolsoverviewfishbonehtmlFivewhysGettingtotherootofaproblemquickly(nd)RetrievedfromhttpswwwmindtoolscompagesarticlenewTMC_5Whtm
References
63
FligsteinNampMcAdamD(2011)TowardageneraltheoryofstrategicactionfieldsSociologicalTheory29(1)1-26RetrievedfromhttpsociologyberkeleyedusitesdefaultfilesfacultyfligsteinFligstein20McAdam20Soc20Theory20PaperpdHeinkelP(2018January12)CitymullspermitsforbikesharingcompaniesRetrievedfromhttpwwwdentonrccomnewsdenton20180112city-mulls-permits-bike-sharing-companiesHowisareamedianincomecalculated(2012October)Retrievedfromhttpswwwhudexchangeinfofaqs843how-is-area-median-income-calculatedLampkinLMampHatryHP(2003)KeystepsinoutcomemanagementUrbanInstituteRetrievedfromhttpswwwurbanorgresearchpublicationkey-steps-outcome-managementMarshDampSmithM(2000)UnderstandingpolicynetworksTowardsadialecticalapproachPoliticalStudies484-21Retrievedfromhttpjournalssagepubcomdoiabs1011111467-924800247McCannL(nd)RetrievedfromhttplaurenellencomMoultonSampSandfortJR(2016)ThestrategicactionfieldframeworkforpolicyimplementationresearchPolicyStudiesJournal45(1)Retrievedfromhttponlinelibrarywileycomstore101111psj12147assetpsj12147pdfv=1ampt=jdkhhcp4amps=e184c625e1855a6b1615ccb99ee27b1c0b3212d8PaynterB(2018January1)What100citiesarelearningfromeachotherbysharingtheirdataRetrievedfromhttpswwwfastcompanycom40518004what-100-cities-are-learning-from-each-other-by-sharing-their-dataPeakacademy(nd)Retrievedfromhttpswwwdenvergovorgcontentdenvergovenmayors-officeprograms-initiativespeak-performancepeak-academyhtmlPinnockHBarwickMCarpenterCREldridgeSGrandesGGriffithsCJRycroft-MaloneJMeissnerPMurrayEPatelASheikhAampTaylorSJC(2017)Standardsforreportingimplementationstudies(stari)ExplanationandelaborationdocumentBMJOpenRetrievedfromhttpbmjopenbmjcomcontentbmjopen74e013318fullpdfProctorESilmereHRaghavanRHovmandPAaronsGBungerAGriffeyRampHensleyM(2011)OutcomesforimplementationresearchConceptualdistinctionsmeasurementchallengesandresearchagendaAdministrationandPolicyinMentalHealth38(2)65-76doi101007s10488-010-0319-7TandonSD(2018)ProgressincommunityhealthpartnershipsResearcheducationandactionRetrievedfromhttpswwwpressjhuedujournalsprogress-community-health-partnerships-research-education-and-actionProvanKGampKenisP(2007)ModesofnetworkgovernanceStructuremanagementandeffectivenessJournalofPublicAdministrationResearchandTheory
References
64
18(2)229-252Retrievedfromhttpsacademicoupcomjpartarticle182229935895SabatierPA(1988)Anadvocacycoalitionframeworkofpolicychangeandtheroleofpolicy-orientedlearningthereinPolicySciences21(3)129-168Retrievedfromhttpwwwjstororgstable4532139SandstromAampCarlssonL(2008)TheperformanceofpolicynetworksTherelationbetweennetworkstructureandnetworkperformancePolicyStudiesJournal36497-524Retrievedfromhttponlinelibrarywileycomdoi101111j1541-0072200800281xabstractSchwartzMampWilsonE(nd)WhocanaffordtoliveinahomeAlookatdatafromthe2006americancommunitysurveyRetrievedfromhttpswwwcensusgovhousingcensuspublicationswho-can-affordpdfShearerJCAbelsonAKouyateBLavisJNampWaltG(2016)WhydopolicieschangeInstitutionsinterestsideasandnetworksinthreecasesofpolicyreformHealthPolicyandPlanning31(9)1200-1211Retrievedfromhttpsdoiorg101093heapolczw052ThelessonsofadministrativedataHigh-profilepolicy-relevantresearchpoweredbyadministrativedata(nd)Retrievedfromhttpswwwpovertyactionlaborgsitesdefaultfilesresources20161114-The-Lessons-of-Administrative-DatapdfToolsandotherneedsforqualitativeandquantitativemethods(2015February)AgencyforHealthcareResearchandQualityRetrievedfromhttpwwwahrqgovprofessionalsquality-patient-safetytalkingqualityassesstoolshtml
WeaverRCampBagchi-SenS(2013)SpatialanalysisofurbandeclineThegeographyofblightAppliedGeography4061-70
References
65
UniversityPartners
Name Title Affiliation Area(s)ofexpertise
Contactinformation
Examplesofworkingwithnon-academicdecisionmakers
WendyPBarreno
AssociateDirector
WilsonSheehanLabforEconomicOpportunitiesUniversityofNotreDame
LEOsfocusareasincludehealtheducationhousingchildwelfareeconomicopportunitycriminaljustice
wbarrenonduduorleondedu
MelissaDavey-RothwellPhD
AssociateScientist
JohnsHopkinsBloombergSchoolofPublicHealth
Health mdavey1jhuedu410-614-5854
BaltimoreCityHealthDepartmentLeadersMarylandDepartmentofHealthAddictionRecoveryServicesinDCExecutiveDirectorsofcommunityorganizationsacrosstheUSthatfocusonHIVSTIsubstanceabuseandimplementationscience
ElizabethStuart
AssociateDeanforEducationandProfessorDepartmentsofMentalHealthBiostatisticsandHealthPolicyand
JohnsHopkinsBloombergSchoolofPublicHealth
Educationsocialpolicystatisticalmethods
estuartjhuedu
Resources
66
Management
Design-ThinkingResources
EngagementGuideforCivicTech
Journeymap
PerformanceManagementResources
DenverPeakAcademy
Proxymeasuresguide
Talkingaboutperformancewithoutcomplicatingit(video)
SettingPerformanceTargetsguide
Datavisualizationguide
Performancemanagementgettingstartedguide
Exercises
Fishboneexercise
5Whys
AdministrativeData
TheLessonsofAdministrativeData(JPAL)
EvaluationResources
Quantitativeandqualitativemethods(USDepartmentofHealthandHumanServicesAgencyforHealthcareResearchandQuality)
KeyStepsinOutcomeManagement(UrbanInstitute)
RCT-YES
Sageresearchmethods
BestPracticesforMixedMethodsResearchintheHealthSciences
ACCORDChecklist
Pilotandimplementationplanning
Benchmarkingexamplesandproxymeasures
Seattleopendataplaybook
Resources
67
Datasciencescopingguide
Resources
68
AboutGovExGovExhelpsgovernmentsusedataandevidencetoimproveresidentsrsquolivesWesupportgovernmentsinbuildingtheirinternalcapacitytooperatesustainableanalyticspracticeswhichimprovecommunityoutcomesinfieldsfrompublicsafetytoeconomicdevelopmentGovExhasaproventrackrecordofworkinginconcertwithpartnerstoprovidetechnicalassistanceandtrainingatscale
SinceourlaunchinApril2015GovExhas
Trainedover2000governmentemployeesinpersonandonlineintopicsincludingdatamanagementperformancemanagementopendatacommunityengagementandanalytics
Providedtechnicalassistancetomorethan100localgovernmentsacrosstheglobeimprovingtheiruseofdatatoachieveefficienciesinservicedeliveryandbetteroutcomesforresidents
Assessedthecapacityofmorethan130governmentalorganizationstodeterminehowleadersstaffandexternalstakeholdersmanageandusedata
Localgovernmentsareincreasinglyonthefrontlinesofaddressinglargescaleproblemswithglobalimplicationssuchasclimatechangepublicsafetyandeconomicinequality
Whileevidence-basedinterventionstohelpsolvethoseproblemsareavailablelocalgovernmentstaffhavelimitedtimeandcapacitytofindandimplementresearch-basedsolutionsAsaresultcitiescontinuetoworkthewaytheyalwayshave--makingincrementalimprovementstoprogramsbutnotalwayscreatingchangeatasystem-level
ThroughresourcedevelopmentandcollaborationwithJHUfacultyGovExissupportingcityleadersinusingdataandevidencetomakedecisionsandcreatinglastingrelationshipswithresearchers
TheRoadmapwasdevelopedbyGovExaspartofitspartnershipwithBloombergPhilanthropiesWhatWorksCitiesinitiative
AboutGovEx
69
- Title page
- Introduction and purpose of the Roadmap
- Using the Roadmap
-
- Gather information about the problem
- Strategically engage key partners
- Review the evidence and collect details
- Decide on a policy to implement based on local factors
- Establish a framework to monitor progress
- Assess data accessibility quality and security
- Design implementation and evaluation
- Share your story and success
-
- Working with a range of evidence
- Thank you
- References
- Resources
- About GovEx
-
Designimplementationandevaluation
50
Shareyourstoryandsuccess
Sharetheresults
Storytelling
AnintegralstepinyourstrategyissharingtheresultsofyourpilotorpolicychangewithyourcommunityThisincludesdevelopingmessagingaroundtheworkofyourorganizationandcreatingaplantodisseminateyourstoriesWhytellstoriesaboutyourwork
Informandorunderstandasubjectorsituation
Giveneededcontextofdataorbitsofinformation
Relatablestoriesarememorableforreadersviewers
ReadersorviewerswillseetheinformationasmorecompellingwhenthereisademonstratedandrelatablechangeorimpactonthemselvesortheirfriendsneighborsandcolleaguesStoriesareaboutpeopleandgovernmentisfundamentallyabouthelpingpeopleSotellthestoriesofthepeopleyouandyourcitycolleagueshelped
Keepthesetipsinmindasyoudevelopyourstorywithyourteam
WhatwereyourfindingsandwhyshouldanyonecareTellyouraudiencehowtheproblemwassolvedWhathastheimpactbeenonresidentsandcitystafftodateWhatwasthesuccessandweretheremultiplesuccessesalongthewayIfyouhavepreliminaryresultsordatashowingimprovementtosharethisisthetimetodoit
WhowasinvolvedandimpactedReviewthelistofstakeholdersHowwereeachofthemimpactedbytheprogramHumanizeyourstoryandimpactbyaddingadditionalvoicesThesevoicescanbeoneofthefollowing
Apersonorfamilywhowaspositivelyaffectedbythechangesuccess
Aninvolvedthird-partydescribingboththeproblemandthesolution(ieanorganizationnonprofitprovidersbusinesstradeassociationsportsteametc)whoispartofthecommunitythatthegovernmentworkedwithonthechangesuccess
Someonefromthegovernmentitselfwhocanspeakaboutthechangesuccess
TellmoreaboutyourcityDescribeyourcityincludingpopulationdemographicsindustriesetcWhatareitschallengesWhatisitknownforWhatmakesituniqueHowisitsimilarordifferentfromothercitiesintheUS
Shareyourstoryandsuccess
51
DescribeyourcityrsquoschallengeWriteoutthechallengeyourcityisfacing(asdefinedinyourproblemstatement)HowdoesitaffectresidentsHowdoesthisproblemaffecttheworkofcitystaffmembers
AddthedetailsProvidedetailsonyourappliedresearchstrategyTalkaboutwhatyourcityhasdonenthepasttoresolvetheissueDiscusshowyourstrategyhaschangedwhenyoubegantoadoptthisapproachWasmoneysavedorinvested
Usethisstorymapexercisetoorganizeyourthoughtsandguaranteethatyouincludealloftheelementsofatraditionalstory
AfteryoudevelopyourmessagingcreateacommunicationsmarketingandoutreachstrategydesignedtospreadyourmessageYourstrategyshouldwalkyouthroughthefollowingkeyactivities
1 IdentifyandprofileyouraudienceWhencreatinganddisseminatingyourmessageitisalwaysgoodtothinkaboutyouraudienceWhoexactlywouldbeinterestedinwhatyouhavetosayThegeneralpopulationofthecityPeoplewhoworkinnonprofitsBusinessownersItmattersbecausenoteveryonewholivesinacitymaybethetargetoftheinformationBusinessownersmaybemoreinterestedindatashowingeffectsonthebusinessdistrictthanothersandsoonWewillcoverthislaterbutthisstepwillalsohelpyoutothinkofthemosteffectivewaytoreachyouraudience
2 Defineyourcalltoaction(ifyouhaveone)SpendsometimeconsideringwhyyouareinterestedintellingyourstoryDoyousimplywanttoupdatethepubliconyourprogressorcelebrateyoursuccessesAreyouinterestedinyouraudiencechangingitsbehaviororsupportingaspecificinitiativebeinglaunchedbyyourorganizationIfyouwanttohaveyouraudienceactorbehaveinacertainwaybespecificandincorporateitinyourmessaging
Shareyourstoryandsuccess
52
3 SelectperformancemeasuresThinkabouthowyouwoulddetermineifyourcampaignissuccessfulandselectappropriateperformancemeasuresYouwillrevisitthesemeasuresattheendofyourcampaigncommunicationsplanimplementationtodeterminewhetheryouneedtomodifyyourcommunicationsandoutreachstrategyforthefuture
4 MethodstocommunicateyourstoryTherearemanywaystocommunicatesuccessofyourprojectHerearesomesuggestionsonreachingaudiencesItisimportanttodeterminewhichmethodsarethemosteffectiveforyourdesiredaudiencetoreceiveinformationDoyourresearch(PewResearchCenterhasexcellentstudiesonthereachofmediadigitalmediaandmarketing)ForexampleifseniorsareyourtargetpopulationandtheyarelessinclinedtouseInstagram(thatistrue)youmaywanttoconsiderFacebookorthenightlylocalnewsbothofwhichareusedmoreoftenbyseniorsBelowisalistofsomemethodsandtoolsyoucanuse
i Pressreleasestolocalnewspapersandtvstations
ii ShortsmartphonevideosAgreatwaytoengageanaudienceisthroughvideoAskonequestionofsomeonewhoispartofthestoryMaybethereisasocialworkerwhocandiscusshungerinyourcityThesevideoscanbereusedonmultipleplatforms
iii Long-formwriting
iv TellingindividualstoriesthroughaseriesofphotosandshortnarrativesforpublicationonFacebookorInstagram
v Op-edsopinionpiecesplacedinthelocalpaper
vi Emailupdatestoyourlist
vii Connectingwithappropriateorganizationsthatareinvestedinthetopicandworkwiththecommunityandaskthemtodisseminateinformationtotheirmembers
viii EmployingldquoWordofMouthrdquo(WOM)tacticsbycreatingtownhallsorfreeeventsforresidentstoattendintheirownneighborhoods
5 DeterminethetypeofcollateralUsuallycampaignsthattargetresidentsrequireseveralavenuestoreachthemPeoplearenotalwaysinthesameplaceorspaceHavingakeenunderstandingofyouraudiencewillassistyouwithyourchoicesonwhatneedstobedevelopedSometimesthatmeansimagesforsocialmediaaonepageflierthatgivespeopleinformationonattendinganeventoranin-depthbrochure
Shareyourstoryandsuccess
53
6 ImplementyourstrategyKeepinmindstrategiescanbefluidSometimessomethingmaynotturnoutasexpectedevenifduringplanningsessionsitwasldquosupposedtordquoItisokYoucananalyzewhathappenedduringadebriefsessionandonceyoursurveysandothermeasurementshavebeenfinalized
Visualizations
Withtherightvisualizationyoucanreviewandshareinformationinsightsanalysisperformanceandtrendsallofwhichisvaluabletoacross-sectionofthepublicButwhichvisualizationisrightforyouTochoosetherighttypeofchartgraphdeterminethetypeofquestionyouwanttoanswerSomecommoncharttypesarebelowandmoreinformationcanbefoundhere
Exploreadditionalanalyticaltechniques
Duringthegoal-settingphaseyouestablishedoutputoutcomeandimplementationmeasuresBasedontheresultsofyourpilotyoumayneedtoadjustyourmeasuresWiththepilotdatainhandyoumayhavesomeideasofhowfutureanalysescouldbedifferentandpotentiallyimprovethequalityofyourprogramTakesometimetobrainstormotherrelevantdatasetsthatyoudidnothavethetimetopursueearlierincludingmessydepartmentaldatathatneedsmoreattentionorsourcesofpublicdatafromthestateandfederalgovernmentnonprofitsandevenprivatecompaniesthatcouldsupplementyourunderstandingofanissueUsingmachinelearningtechniquesyoumightbeabletobetter
Shareyourstoryandsuccess
54
predictdemandforhomelessservicesusingNOAAweatherforecastspreemptwatermainbreaksusingsoilstratigraphyorinvestineconomicdevelopmentbymonitoringrealestatelistingdata
Advancedanalyticaltechniquescanalsohelpyourimplementationrunmoreeffectivelybyintegratingdata-informedtargetinganddecisiontoolsintotheprocessByautomatingroutineworkflowsandsurfacingsubtletrendsthesetoolscanhelpyouiterateinamoreprecisemannerandprovidekeystrategicguidancetofront-lineemployeesthatwouldotherwisebeunavailableThesesortsofdata-science-enhancedprocessescanbeclassifiedasoneofsixprojecttypesasexploredfurtherbytheNewOrleansLANOLAlyticsteamFormoreguidanceonscopingasimilarprojecttheGovExadvancedanalyticsscopingguidegivesyouastep-by-stepoverviewofhowtobesuccessful
Usethespacebelowtolistyourideasforpossibleadvancedanalyticsprojectsbytype
Typeofproject Ideas
Findingtheneedleinthehaystack
Prioritizingforimpact
Earlywarningtools
Betterquickerdecisions
Optimizingresourceallocation
Experimentingforwhatworks
Keepthefeedbackloopgoing
GatherinformationonpublicsentimentaboutpolicychangethroughfocusgroupssurveysandpublicmeetingsledbyadvisorygroupmembersNotewhethernewinformationisavailablethatcanhelpanalyzeorcontextualizeyourprogress(egfederalorstatedatareports)Sharetheresultsofyourpilotandthepolicychangeprocesswithothercitiesandresearchers
Shareyourstoryandsuccess
55
WorkingwitharangeofevidencePresentlythereisalotofattentionandinvestmentinevidence-basedpolicymakingincludingthefederalbipartisanCommissiononEvidence-BasedPolicymakingthePewCharitableTrustsreportsandtoolkitsonHowStatesEngageinEvidence-BasedPolicymakingandtheLauraandJohnArnoldFoundationsupportedEvidence-BasedPolicymakingCollaborativeManyoftheseinitiativesfocusonrigorousevaluation--meaningrandomizedcontrolledtrials(RCT)orotherexperimentalmodels--asthebestkindofevidencetouseHoweverthisbodyofevidenceisofteninsufficientforthespecifickindsofquestionsthatgovernmentscanhavewhenmakingadecisionaboutpolicyInthisguideweproposeanapproachtosupportgovernmentsinincorporatingadditionaltypesofevidenceintotheirdecisionmaking
OnewaytounderstandarangeofevidenceistouseNestarsquosStandardsofEvidencetoframeanddefineevidenceTheBritishnonprofitNestaworkstoinjectpositivechangeintothewayUKgovernmentpoliciesandprogramsfunctionOneoftheirkeyareasofworkhasbeendevelopingWhatWorksCentresorganizationsthatcollectcommissionsynthesizeanddisseminateevidenceaboutinterventioneffectivenessacrossarangeofsocialimpactareasNestadevelopedthefollowingStandardsofEvidencetoprovideausefulwaytodocumenthowdifferentkindsofinformationdemonstratedifferentlevelsofevidence
Workingwitharangeofevidence
56
Theimportanceofpotentiallybiasedinformation
Onechallengepresentedbyanevidenceclassificationthatacknowledgesthevalueofnon-scientificformsofevidenceistofigureouthowtobalanceallofthepiecestogetherwhenyouusethemTounderstandhowtodothiswehavetothinkabouthowtopositionpotentiallybiasedinformationindecisionmaking
WhenwelookatinformationwhichhasnotbeenrigorouslyevaluatedweacceptthatthisinformationmightbebiasedThewholereasonweconductrigorousevaluationistoremovebiasfromourobservationsWeconductstructuredtrialstoproduceresultsthatareconsistentandgeneralizableIfafindingisunbiasedthisaccuracymdashthislackofbiasmdashcanbeprovedbecauseanyindependenttestercanreproduceitwhentheyfollowanidenticalprocess
YoucanlookatpotentiallybiasedinformationasjustinformationthatisearlieroninitspathtorefinementreplicationandverifiedaccuracyApieceofinformationbasedonanyoneindividualrsquosobservationsislikelytobebiasedbecauseitiscomingfromjustonesourceBiasisreducedbygatheringtogetherlotsofindependentsourcesofinformationandfiguringoutitsmostcommonattributes
MaybethebiasedinformationmakesaclaimthatistoolargemdashitsayssomethingistrueunderallconditionswhenitisreallyonlytrueundercertainconditionsorthataneffectofaninterventionisbiggerthanitstrueaverageorthataneffectlastsovertimewhenitdoesnotPerhapschangingtheclaimtobemorespecificmdashcriticallyreviewingtheinformationtoseeifspecificconditionsmustbemettoachievetheoutcomesmdashwouldmaketheclaimmoreaccurate
IfyoutestedyourclaimfounditdidnotholdupinsomecaseschangedittesteditagainuntilyourclaimrepeatedlyheldupthenyoucoulddemonstratethattheclaimwasnolongerbiasedbutaccurateThisprocessoftestingimprovingtestingandvalidatingiswhatwegetfromstructuredscientificstudy
Withindividualpiecesofinformation--crediblehypothesescasestudiesexpertopinion--youcannotassumethatkindofaccuracyAtthesametimetheseearlyobservationsarefoundationaltoanyreplicablescientificobservationsWewillnevergetanywhereifwedonotstartwiththesepiecesofinformationeveniftheymightbebiased
Combininglevelsofevidencetogetthemostdetailedinformationforyourcity
ItisespeciallyimportanttoacceptdifferentformsofevidencewhenyoufacequestionsthathavenotyetbeentestedbyindependentstudiesSomeofthemostimportantquestionsrelatetothingsthatareveryspecifictoyoursituationYoufaceaparticularcombinationof
Workingwitharangeofevidence
57
issuesactorstrendsandresourcesthatisultimatelydifferentfromeveryotherplaceintheworld
ThisisimportantbecauseaspolicypractitionersandnotscientistsyoucaremoreaboutyourspecificcasethanahypotheticalaveragegeneralcaseYoucarelessaboutthetotallikelyrangeofimpactsfromapolicythanhowthissingleimplementationwillworkYouwantinformationaboutpoliciesthatwillprovideyouwiththebestpossiblepredictionabouthowtheywillworkinyourcontext
ByputtingdifferentkindsofevidenceintothesameframeworkNestarsquosStandardsofEvidenceimplythatwhenyoucannotgetrigorouslytestedexperimentallyverifiedinformationitisappropriatetoworkwithwhatwedohaveThisisrelevantforthemunicipalimplementationcontextbecausenostudypulledfromaliteraturereviewwillaccountforallofthevariablesyoufaceinyourowncontext
Intheabsenceofhavingastudythatpredictshowaninterventionworksinyourspecificcontextyouwillneednon-scientificinformationtotrytofillinthosegapsYouwillneedtogatherstoriesfromresidentsstaffmembersandotherstakeholdersthatsitonLevels1or2ofthestandardsofevidence
Someofthekindsofinformationyouwillprobablygetmoreeasilyfrompeoplethanstudiesinclude
howpeopleincitieslikeyoursreactedtoaninterventioninpublicforumshowpeoplewhoimplementedaninterventioninacitylikeyoursfiguredouthowtogetitpassedhowthecitycommunicatedpubliclyabouttheapparentbenefitsanddrawbacksoftheinterventionthepoliticalacceptabilityoftheintervention
Youcanremainsimultaneouslyawarethattheinformationyouaregatheringthroughindividualstoriesisbiased--notgeneralizableinformedbyspecificperspectives--whilealsorecognizingthathavingsomeinformationaboutsomethingimportantisbetterthannoinformationEvidencefromthefirsttwoevidencelevelsareessentialbuildingblocksofknowledgeandperfectlyusefulwhenyoucannotgetdetailedunbiasedknowledge
Youhavetostartsomewhereandyoushouldalwaysstartwithevidenceyouhaveaboutyourveryspecificconditionsthatyouknowareimportantforsuccess
Workingwitharangeofevidence
58
Thankyou
Interviewswithcities
GovExheldsemi-structuredinterviewswithsixstaffmembersinfourcitiestoestablishtheinitialstepsoftheRoadmapWeaskedrespondentstotalkaboutaninitiativeorpolicythattheyhavebeenresponsibleforimplementingintheircityRespondentsfromthreecitiesspokeabouttheireffortstoaddresshomelessnessandrespondentsfromonecityspokeaboutblightAfterrespondentsidentifiedtheproblemtheyspokeabouthowthecityprioritizedthisissuethevariousstepsinvolvedandbarrierspresent
GovExconductedtextanalysisoftheseinterviewswhichrevealedthatpeoplewasthemostcommonlyusedtermduringtheseconversationswhendiscussingthestepstochangeaprogramorpolicyThisisathemethroughoutthisRoadmapThefirstsectionbeginswithidentifyingtherelevantpeopleTheconversationsaroundevidencearefocusedonhumanbehaviorratherthanmethodologyorstrengthofevidence
Thankyou
59
OncewehadadraftoftheRoadmapwefollowedupwithcitiesonanindividualbasistolearnmoreaboutwhatwouldbehelpfulWearegratefultothefollowingcitystafffortheinterviewsandindividualconversations
MichaelBaskinChiefPolicyOfficerChattanoogaTN
MelissaBridgesPerformanceandInnovationCoordinatorLittleRockAR
VictoriaCarreoacutenAdministrativeOfficerLasVegasNV
TyroneGrandstrandCommunityDevelopmentPlannerFargoND
TibLaughlinDirectorofGeneralServicesKansasCityKS
TimMorelandDirectorofPerformanceManagementandOpenDataChattanoogaTN
CharlesRosendahlManagementAnalystDentonTX
Thankyou
60
MelissaSiebenAssistantCountyAdministratorKansasCityKS
GinaThompsonDirectorofStrategicServicesLewisvilleTX
JHUFacultycollaboration
GovExalsoconvenedthecitycontactsandfacultypartnersinaseriesofAdvisoryGroupmeetingsThesemeetingsservedtoconnectcitiesandfacultytounderstandoneanothersperspective
StefanBaralBloombergSchoolofPublicHealth
AshleyBernerJHUSchoolofEducation
MelissaDavey-RothwellBloombergSchoolofPublicHealth
JonLinksBloombergSchoolofPublicHealth
TamarMendelsonBloombergSchoolofPublicHealth
KristinMmariBloombergSchoolofPublicHealth
KeshiaPollackPorterBloombergSchoolofPublicHealth
JoshSharfsteinBloombergSchoolofPublicHealth
LizStuartBloombergSchoolofPublicHealth
LindsayThompsonCareyBusinessSchool
DanielWebsterBloombergSchoolofPublicHealth
Partnersindataandevidence-basedpolicy
Wealsospokewithpartnerswhoareexpertsinthefieldofevidence-basedpolicywhosharedtheirknowledgeandresourcesofcurrentandfuturethinkinginevidence-informedthinking
WendyBarrenoAssociateDirectorWilsonSheehanLabforEconomicOpportunitiesUniversityofNotreDame
TiffanyDavenportDirectorofDataPartnershipsBaltimoresPromise
TomiHiersDirectorBaltimoreCivicSiteAnnieECaseyFoundation
ChrisKingsleySeniorAssociateAnnieECaseyFoundation
KathrynOliverAssociateProfessorLondonSchoolofHygieneandTropicalMedicine
Thankyou
61
KristenWatkinsSeniorPolicyAssociateJ-PAL
GovExParticipants
GovExAnalystMiriamMcKinneyanalyzedtheinterviewtranscriptsandconductedtextanalysisSeniorImplementationAdvisorEmilyShawwillintegrateRoadmapactivitieswithadditionalservicedeliveryactivitiesatGovEx
Thankyou
62
ReferencesAbrahamKGampHaskinsR(2017)Thepromiseofevidence-basedpolicymakingReportofthecommissiononevidence-basedpolicymakingWashingtonDCCommissiononEvidence-basedPolicymakingBigmove2Extendingthegreen(nd)Retrievedfromhttpswwwcityoflewisvillecomdoing-businesslewisville-2025big-move-2-extending-the-greenBatesMAampGlennersterR(2017)ThegeneralizabilitypuzzleRetrievedfromhttpsssirorgarticlesentrythe_generalizability_puzzleBuildwithnotfor(2018January)RetrievedfromhttpsbuildwithnotforwordpresscomCairneyPampOliverK(2017)Evidence-basedpolicymakingisnotlikeevidence-basedmedicinesohowfarshouldyougotobridgethedividebetweenevidenceandpolicyHealthResearchPolicyandSystemsdoi101186s12961-017-0192-xCairneyPOliverKampWellsteadA(2016)TobridgethedividebetweenevidenceandpolicyReduceambiguityasmuchasuncertaintyPublicAdministrationReview76(3)399-402Retrievedfromhttpdxdoiorg101111puar12555Centreforpublicimpact(nd)RetrievedfromhttpswwwcentreforpublicimpactorgCernaL(2013)ThenatureofpolicychangeandimplementationAreviewofdifferenttheoreticalapproachesOrganisationforEconomicCo-OperationandDevelopmentRetrievedfromhttpswwwoecdorgeduceriTheNatureofPolicyChangeandImplementationpdfColoradoSprings(nd)Retrievedfromhttpscoloradospringsgovcommunicationspagewhat-expect-when-you-speakupContinuumofcare(coc)program(nd)RetrievedfromhttpswwwhudexchangeinfoprogramscocCouncilpresidentkicksoff$12millionyouthfund(nd)Retrievedfromhttpwwwbaltimorecitycouncilcomcontentcouncil-president-kicks-12-million-youth-fundDurlakJAampDuPreEP(2008)ImplementationmattersAreviewofresearchontheinfluenceofimplementationonprogramoutcomesandthefactorsaffectingimplementationAmericanJournalofCommunityPsychology41327-350Retrievedfromhttpswwwncbinlmnihgovpubmed18322790Fishbonediagram(ishikawa)(nd)Retrievedfromhttpasqorglearn-about-qualitycause-analysis-toolsoverviewfishbonehtmlFivewhysGettingtotherootofaproblemquickly(nd)RetrievedfromhttpswwwmindtoolscompagesarticlenewTMC_5Whtm
References
63
FligsteinNampMcAdamD(2011)TowardageneraltheoryofstrategicactionfieldsSociologicalTheory29(1)1-26RetrievedfromhttpsociologyberkeleyedusitesdefaultfilesfacultyfligsteinFligstein20McAdam20Soc20Theory20PaperpdHeinkelP(2018January12)CitymullspermitsforbikesharingcompaniesRetrievedfromhttpwwwdentonrccomnewsdenton20180112city-mulls-permits-bike-sharing-companiesHowisareamedianincomecalculated(2012October)Retrievedfromhttpswwwhudexchangeinfofaqs843how-is-area-median-income-calculatedLampkinLMampHatryHP(2003)KeystepsinoutcomemanagementUrbanInstituteRetrievedfromhttpswwwurbanorgresearchpublicationkey-steps-outcome-managementMarshDampSmithM(2000)UnderstandingpolicynetworksTowardsadialecticalapproachPoliticalStudies484-21Retrievedfromhttpjournalssagepubcomdoiabs1011111467-924800247McCannL(nd)RetrievedfromhttplaurenellencomMoultonSampSandfortJR(2016)ThestrategicactionfieldframeworkforpolicyimplementationresearchPolicyStudiesJournal45(1)Retrievedfromhttponlinelibrarywileycomstore101111psj12147assetpsj12147pdfv=1ampt=jdkhhcp4amps=e184c625e1855a6b1615ccb99ee27b1c0b3212d8PaynterB(2018January1)What100citiesarelearningfromeachotherbysharingtheirdataRetrievedfromhttpswwwfastcompanycom40518004what-100-cities-are-learning-from-each-other-by-sharing-their-dataPeakacademy(nd)Retrievedfromhttpswwwdenvergovorgcontentdenvergovenmayors-officeprograms-initiativespeak-performancepeak-academyhtmlPinnockHBarwickMCarpenterCREldridgeSGrandesGGriffithsCJRycroft-MaloneJMeissnerPMurrayEPatelASheikhAampTaylorSJC(2017)Standardsforreportingimplementationstudies(stari)ExplanationandelaborationdocumentBMJOpenRetrievedfromhttpbmjopenbmjcomcontentbmjopen74e013318fullpdfProctorESilmereHRaghavanRHovmandPAaronsGBungerAGriffeyRampHensleyM(2011)OutcomesforimplementationresearchConceptualdistinctionsmeasurementchallengesandresearchagendaAdministrationandPolicyinMentalHealth38(2)65-76doi101007s10488-010-0319-7TandonSD(2018)ProgressincommunityhealthpartnershipsResearcheducationandactionRetrievedfromhttpswwwpressjhuedujournalsprogress-community-health-partnerships-research-education-and-actionProvanKGampKenisP(2007)ModesofnetworkgovernanceStructuremanagementandeffectivenessJournalofPublicAdministrationResearchandTheory
References
64
18(2)229-252Retrievedfromhttpsacademicoupcomjpartarticle182229935895SabatierPA(1988)Anadvocacycoalitionframeworkofpolicychangeandtheroleofpolicy-orientedlearningthereinPolicySciences21(3)129-168Retrievedfromhttpwwwjstororgstable4532139SandstromAampCarlssonL(2008)TheperformanceofpolicynetworksTherelationbetweennetworkstructureandnetworkperformancePolicyStudiesJournal36497-524Retrievedfromhttponlinelibrarywileycomdoi101111j1541-0072200800281xabstractSchwartzMampWilsonE(nd)WhocanaffordtoliveinahomeAlookatdatafromthe2006americancommunitysurveyRetrievedfromhttpswwwcensusgovhousingcensuspublicationswho-can-affordpdfShearerJCAbelsonAKouyateBLavisJNampWaltG(2016)WhydopolicieschangeInstitutionsinterestsideasandnetworksinthreecasesofpolicyreformHealthPolicyandPlanning31(9)1200-1211Retrievedfromhttpsdoiorg101093heapolczw052ThelessonsofadministrativedataHigh-profilepolicy-relevantresearchpoweredbyadministrativedata(nd)Retrievedfromhttpswwwpovertyactionlaborgsitesdefaultfilesresources20161114-The-Lessons-of-Administrative-DatapdfToolsandotherneedsforqualitativeandquantitativemethods(2015February)AgencyforHealthcareResearchandQualityRetrievedfromhttpwwwahrqgovprofessionalsquality-patient-safetytalkingqualityassesstoolshtml
WeaverRCampBagchi-SenS(2013)SpatialanalysisofurbandeclineThegeographyofblightAppliedGeography4061-70
References
65
UniversityPartners
Name Title Affiliation Area(s)ofexpertise
Contactinformation
Examplesofworkingwithnon-academicdecisionmakers
WendyPBarreno
AssociateDirector
WilsonSheehanLabforEconomicOpportunitiesUniversityofNotreDame
LEOsfocusareasincludehealtheducationhousingchildwelfareeconomicopportunitycriminaljustice
wbarrenonduduorleondedu
MelissaDavey-RothwellPhD
AssociateScientist
JohnsHopkinsBloombergSchoolofPublicHealth
Health mdavey1jhuedu410-614-5854
BaltimoreCityHealthDepartmentLeadersMarylandDepartmentofHealthAddictionRecoveryServicesinDCExecutiveDirectorsofcommunityorganizationsacrosstheUSthatfocusonHIVSTIsubstanceabuseandimplementationscience
ElizabethStuart
AssociateDeanforEducationandProfessorDepartmentsofMentalHealthBiostatisticsandHealthPolicyand
JohnsHopkinsBloombergSchoolofPublicHealth
Educationsocialpolicystatisticalmethods
estuartjhuedu
Resources
66
Management
Design-ThinkingResources
EngagementGuideforCivicTech
Journeymap
PerformanceManagementResources
DenverPeakAcademy
Proxymeasuresguide
Talkingaboutperformancewithoutcomplicatingit(video)
SettingPerformanceTargetsguide
Datavisualizationguide
Performancemanagementgettingstartedguide
Exercises
Fishboneexercise
5Whys
AdministrativeData
TheLessonsofAdministrativeData(JPAL)
EvaluationResources
Quantitativeandqualitativemethods(USDepartmentofHealthandHumanServicesAgencyforHealthcareResearchandQuality)
KeyStepsinOutcomeManagement(UrbanInstitute)
RCT-YES
Sageresearchmethods
BestPracticesforMixedMethodsResearchintheHealthSciences
ACCORDChecklist
Pilotandimplementationplanning
Benchmarkingexamplesandproxymeasures
Seattleopendataplaybook
Resources
67
Datasciencescopingguide
Resources
68
AboutGovExGovExhelpsgovernmentsusedataandevidencetoimproveresidentsrsquolivesWesupportgovernmentsinbuildingtheirinternalcapacitytooperatesustainableanalyticspracticeswhichimprovecommunityoutcomesinfieldsfrompublicsafetytoeconomicdevelopmentGovExhasaproventrackrecordofworkinginconcertwithpartnerstoprovidetechnicalassistanceandtrainingatscale
SinceourlaunchinApril2015GovExhas
Trainedover2000governmentemployeesinpersonandonlineintopicsincludingdatamanagementperformancemanagementopendatacommunityengagementandanalytics
Providedtechnicalassistancetomorethan100localgovernmentsacrosstheglobeimprovingtheiruseofdatatoachieveefficienciesinservicedeliveryandbetteroutcomesforresidents
Assessedthecapacityofmorethan130governmentalorganizationstodeterminehowleadersstaffandexternalstakeholdersmanageandusedata
Localgovernmentsareincreasinglyonthefrontlinesofaddressinglargescaleproblemswithglobalimplicationssuchasclimatechangepublicsafetyandeconomicinequality
Whileevidence-basedinterventionstohelpsolvethoseproblemsareavailablelocalgovernmentstaffhavelimitedtimeandcapacitytofindandimplementresearch-basedsolutionsAsaresultcitiescontinuetoworkthewaytheyalwayshave--makingincrementalimprovementstoprogramsbutnotalwayscreatingchangeatasystem-level
ThroughresourcedevelopmentandcollaborationwithJHUfacultyGovExissupportingcityleadersinusingdataandevidencetomakedecisionsandcreatinglastingrelationshipswithresearchers
TheRoadmapwasdevelopedbyGovExaspartofitspartnershipwithBloombergPhilanthropiesWhatWorksCitiesinitiative
AboutGovEx
69
- Title page
- Introduction and purpose of the Roadmap
- Using the Roadmap
-
- Gather information about the problem
- Strategically engage key partners
- Review the evidence and collect details
- Decide on a policy to implement based on local factors
- Establish a framework to monitor progress
- Assess data accessibility quality and security
- Design implementation and evaluation
- Share your story and success
-
- Working with a range of evidence
- Thank you
- References
- Resources
- About GovEx
-
Shareyourstoryandsuccess
Sharetheresults
Storytelling
AnintegralstepinyourstrategyissharingtheresultsofyourpilotorpolicychangewithyourcommunityThisincludesdevelopingmessagingaroundtheworkofyourorganizationandcreatingaplantodisseminateyourstoriesWhytellstoriesaboutyourwork
Informandorunderstandasubjectorsituation
Giveneededcontextofdataorbitsofinformation
Relatablestoriesarememorableforreadersviewers
ReadersorviewerswillseetheinformationasmorecompellingwhenthereisademonstratedandrelatablechangeorimpactonthemselvesortheirfriendsneighborsandcolleaguesStoriesareaboutpeopleandgovernmentisfundamentallyabouthelpingpeopleSotellthestoriesofthepeopleyouandyourcitycolleagueshelped
Keepthesetipsinmindasyoudevelopyourstorywithyourteam
WhatwereyourfindingsandwhyshouldanyonecareTellyouraudiencehowtheproblemwassolvedWhathastheimpactbeenonresidentsandcitystafftodateWhatwasthesuccessandweretheremultiplesuccessesalongthewayIfyouhavepreliminaryresultsordatashowingimprovementtosharethisisthetimetodoit
WhowasinvolvedandimpactedReviewthelistofstakeholdersHowwereeachofthemimpactedbytheprogramHumanizeyourstoryandimpactbyaddingadditionalvoicesThesevoicescanbeoneofthefollowing
Apersonorfamilywhowaspositivelyaffectedbythechangesuccess
Aninvolvedthird-partydescribingboththeproblemandthesolution(ieanorganizationnonprofitprovidersbusinesstradeassociationsportsteametc)whoispartofthecommunitythatthegovernmentworkedwithonthechangesuccess
Someonefromthegovernmentitselfwhocanspeakaboutthechangesuccess
TellmoreaboutyourcityDescribeyourcityincludingpopulationdemographicsindustriesetcWhatareitschallengesWhatisitknownforWhatmakesituniqueHowisitsimilarordifferentfromothercitiesintheUS
Shareyourstoryandsuccess
51
DescribeyourcityrsquoschallengeWriteoutthechallengeyourcityisfacing(asdefinedinyourproblemstatement)HowdoesitaffectresidentsHowdoesthisproblemaffecttheworkofcitystaffmembers
AddthedetailsProvidedetailsonyourappliedresearchstrategyTalkaboutwhatyourcityhasdonenthepasttoresolvetheissueDiscusshowyourstrategyhaschangedwhenyoubegantoadoptthisapproachWasmoneysavedorinvested
Usethisstorymapexercisetoorganizeyourthoughtsandguaranteethatyouincludealloftheelementsofatraditionalstory
AfteryoudevelopyourmessagingcreateacommunicationsmarketingandoutreachstrategydesignedtospreadyourmessageYourstrategyshouldwalkyouthroughthefollowingkeyactivities
1 IdentifyandprofileyouraudienceWhencreatinganddisseminatingyourmessageitisalwaysgoodtothinkaboutyouraudienceWhoexactlywouldbeinterestedinwhatyouhavetosayThegeneralpopulationofthecityPeoplewhoworkinnonprofitsBusinessownersItmattersbecausenoteveryonewholivesinacitymaybethetargetoftheinformationBusinessownersmaybemoreinterestedindatashowingeffectsonthebusinessdistrictthanothersandsoonWewillcoverthislaterbutthisstepwillalsohelpyoutothinkofthemosteffectivewaytoreachyouraudience
2 Defineyourcalltoaction(ifyouhaveone)SpendsometimeconsideringwhyyouareinterestedintellingyourstoryDoyousimplywanttoupdatethepubliconyourprogressorcelebrateyoursuccessesAreyouinterestedinyouraudiencechangingitsbehaviororsupportingaspecificinitiativebeinglaunchedbyyourorganizationIfyouwanttohaveyouraudienceactorbehaveinacertainwaybespecificandincorporateitinyourmessaging
Shareyourstoryandsuccess
52
3 SelectperformancemeasuresThinkabouthowyouwoulddetermineifyourcampaignissuccessfulandselectappropriateperformancemeasuresYouwillrevisitthesemeasuresattheendofyourcampaigncommunicationsplanimplementationtodeterminewhetheryouneedtomodifyyourcommunicationsandoutreachstrategyforthefuture
4 MethodstocommunicateyourstoryTherearemanywaystocommunicatesuccessofyourprojectHerearesomesuggestionsonreachingaudiencesItisimportanttodeterminewhichmethodsarethemosteffectiveforyourdesiredaudiencetoreceiveinformationDoyourresearch(PewResearchCenterhasexcellentstudiesonthereachofmediadigitalmediaandmarketing)ForexampleifseniorsareyourtargetpopulationandtheyarelessinclinedtouseInstagram(thatistrue)youmaywanttoconsiderFacebookorthenightlylocalnewsbothofwhichareusedmoreoftenbyseniorsBelowisalistofsomemethodsandtoolsyoucanuse
i Pressreleasestolocalnewspapersandtvstations
ii ShortsmartphonevideosAgreatwaytoengageanaudienceisthroughvideoAskonequestionofsomeonewhoispartofthestoryMaybethereisasocialworkerwhocandiscusshungerinyourcityThesevideoscanbereusedonmultipleplatforms
iii Long-formwriting
iv TellingindividualstoriesthroughaseriesofphotosandshortnarrativesforpublicationonFacebookorInstagram
v Op-edsopinionpiecesplacedinthelocalpaper
vi Emailupdatestoyourlist
vii Connectingwithappropriateorganizationsthatareinvestedinthetopicandworkwiththecommunityandaskthemtodisseminateinformationtotheirmembers
viii EmployingldquoWordofMouthrdquo(WOM)tacticsbycreatingtownhallsorfreeeventsforresidentstoattendintheirownneighborhoods
5 DeterminethetypeofcollateralUsuallycampaignsthattargetresidentsrequireseveralavenuestoreachthemPeoplearenotalwaysinthesameplaceorspaceHavingakeenunderstandingofyouraudiencewillassistyouwithyourchoicesonwhatneedstobedevelopedSometimesthatmeansimagesforsocialmediaaonepageflierthatgivespeopleinformationonattendinganeventoranin-depthbrochure
Shareyourstoryandsuccess
53
6 ImplementyourstrategyKeepinmindstrategiescanbefluidSometimessomethingmaynotturnoutasexpectedevenifduringplanningsessionsitwasldquosupposedtordquoItisokYoucananalyzewhathappenedduringadebriefsessionandonceyoursurveysandothermeasurementshavebeenfinalized
Visualizations
Withtherightvisualizationyoucanreviewandshareinformationinsightsanalysisperformanceandtrendsallofwhichisvaluabletoacross-sectionofthepublicButwhichvisualizationisrightforyouTochoosetherighttypeofchartgraphdeterminethetypeofquestionyouwanttoanswerSomecommoncharttypesarebelowandmoreinformationcanbefoundhere
Exploreadditionalanalyticaltechniques
Duringthegoal-settingphaseyouestablishedoutputoutcomeandimplementationmeasuresBasedontheresultsofyourpilotyoumayneedtoadjustyourmeasuresWiththepilotdatainhandyoumayhavesomeideasofhowfutureanalysescouldbedifferentandpotentiallyimprovethequalityofyourprogramTakesometimetobrainstormotherrelevantdatasetsthatyoudidnothavethetimetopursueearlierincludingmessydepartmentaldatathatneedsmoreattentionorsourcesofpublicdatafromthestateandfederalgovernmentnonprofitsandevenprivatecompaniesthatcouldsupplementyourunderstandingofanissueUsingmachinelearningtechniquesyoumightbeabletobetter
Shareyourstoryandsuccess
54
predictdemandforhomelessservicesusingNOAAweatherforecastspreemptwatermainbreaksusingsoilstratigraphyorinvestineconomicdevelopmentbymonitoringrealestatelistingdata
Advancedanalyticaltechniquescanalsohelpyourimplementationrunmoreeffectivelybyintegratingdata-informedtargetinganddecisiontoolsintotheprocessByautomatingroutineworkflowsandsurfacingsubtletrendsthesetoolscanhelpyouiterateinamoreprecisemannerandprovidekeystrategicguidancetofront-lineemployeesthatwouldotherwisebeunavailableThesesortsofdata-science-enhancedprocessescanbeclassifiedasoneofsixprojecttypesasexploredfurtherbytheNewOrleansLANOLAlyticsteamFormoreguidanceonscopingasimilarprojecttheGovExadvancedanalyticsscopingguidegivesyouastep-by-stepoverviewofhowtobesuccessful
Usethespacebelowtolistyourideasforpossibleadvancedanalyticsprojectsbytype
Typeofproject Ideas
Findingtheneedleinthehaystack
Prioritizingforimpact
Earlywarningtools
Betterquickerdecisions
Optimizingresourceallocation
Experimentingforwhatworks
Keepthefeedbackloopgoing
GatherinformationonpublicsentimentaboutpolicychangethroughfocusgroupssurveysandpublicmeetingsledbyadvisorygroupmembersNotewhethernewinformationisavailablethatcanhelpanalyzeorcontextualizeyourprogress(egfederalorstatedatareports)Sharetheresultsofyourpilotandthepolicychangeprocesswithothercitiesandresearchers
Shareyourstoryandsuccess
55
WorkingwitharangeofevidencePresentlythereisalotofattentionandinvestmentinevidence-basedpolicymakingincludingthefederalbipartisanCommissiononEvidence-BasedPolicymakingthePewCharitableTrustsreportsandtoolkitsonHowStatesEngageinEvidence-BasedPolicymakingandtheLauraandJohnArnoldFoundationsupportedEvidence-BasedPolicymakingCollaborativeManyoftheseinitiativesfocusonrigorousevaluation--meaningrandomizedcontrolledtrials(RCT)orotherexperimentalmodels--asthebestkindofevidencetouseHoweverthisbodyofevidenceisofteninsufficientforthespecifickindsofquestionsthatgovernmentscanhavewhenmakingadecisionaboutpolicyInthisguideweproposeanapproachtosupportgovernmentsinincorporatingadditionaltypesofevidenceintotheirdecisionmaking
OnewaytounderstandarangeofevidenceistouseNestarsquosStandardsofEvidencetoframeanddefineevidenceTheBritishnonprofitNestaworkstoinjectpositivechangeintothewayUKgovernmentpoliciesandprogramsfunctionOneoftheirkeyareasofworkhasbeendevelopingWhatWorksCentresorganizationsthatcollectcommissionsynthesizeanddisseminateevidenceaboutinterventioneffectivenessacrossarangeofsocialimpactareasNestadevelopedthefollowingStandardsofEvidencetoprovideausefulwaytodocumenthowdifferentkindsofinformationdemonstratedifferentlevelsofevidence
Workingwitharangeofevidence
56
Theimportanceofpotentiallybiasedinformation
Onechallengepresentedbyanevidenceclassificationthatacknowledgesthevalueofnon-scientificformsofevidenceistofigureouthowtobalanceallofthepiecestogetherwhenyouusethemTounderstandhowtodothiswehavetothinkabouthowtopositionpotentiallybiasedinformationindecisionmaking
WhenwelookatinformationwhichhasnotbeenrigorouslyevaluatedweacceptthatthisinformationmightbebiasedThewholereasonweconductrigorousevaluationistoremovebiasfromourobservationsWeconductstructuredtrialstoproduceresultsthatareconsistentandgeneralizableIfafindingisunbiasedthisaccuracymdashthislackofbiasmdashcanbeprovedbecauseanyindependenttestercanreproduceitwhentheyfollowanidenticalprocess
YoucanlookatpotentiallybiasedinformationasjustinformationthatisearlieroninitspathtorefinementreplicationandverifiedaccuracyApieceofinformationbasedonanyoneindividualrsquosobservationsislikelytobebiasedbecauseitiscomingfromjustonesourceBiasisreducedbygatheringtogetherlotsofindependentsourcesofinformationandfiguringoutitsmostcommonattributes
MaybethebiasedinformationmakesaclaimthatistoolargemdashitsayssomethingistrueunderallconditionswhenitisreallyonlytrueundercertainconditionsorthataneffectofaninterventionisbiggerthanitstrueaverageorthataneffectlastsovertimewhenitdoesnotPerhapschangingtheclaimtobemorespecificmdashcriticallyreviewingtheinformationtoseeifspecificconditionsmustbemettoachievetheoutcomesmdashwouldmaketheclaimmoreaccurate
IfyoutestedyourclaimfounditdidnotholdupinsomecaseschangedittesteditagainuntilyourclaimrepeatedlyheldupthenyoucoulddemonstratethattheclaimwasnolongerbiasedbutaccurateThisprocessoftestingimprovingtestingandvalidatingiswhatwegetfromstructuredscientificstudy
Withindividualpiecesofinformation--crediblehypothesescasestudiesexpertopinion--youcannotassumethatkindofaccuracyAtthesametimetheseearlyobservationsarefoundationaltoanyreplicablescientificobservationsWewillnevergetanywhereifwedonotstartwiththesepiecesofinformationeveniftheymightbebiased
Combininglevelsofevidencetogetthemostdetailedinformationforyourcity
ItisespeciallyimportanttoacceptdifferentformsofevidencewhenyoufacequestionsthathavenotyetbeentestedbyindependentstudiesSomeofthemostimportantquestionsrelatetothingsthatareveryspecifictoyoursituationYoufaceaparticularcombinationof
Workingwitharangeofevidence
57
issuesactorstrendsandresourcesthatisultimatelydifferentfromeveryotherplaceintheworld
ThisisimportantbecauseaspolicypractitionersandnotscientistsyoucaremoreaboutyourspecificcasethanahypotheticalaveragegeneralcaseYoucarelessaboutthetotallikelyrangeofimpactsfromapolicythanhowthissingleimplementationwillworkYouwantinformationaboutpoliciesthatwillprovideyouwiththebestpossiblepredictionabouthowtheywillworkinyourcontext
ByputtingdifferentkindsofevidenceintothesameframeworkNestarsquosStandardsofEvidenceimplythatwhenyoucannotgetrigorouslytestedexperimentallyverifiedinformationitisappropriatetoworkwithwhatwedohaveThisisrelevantforthemunicipalimplementationcontextbecausenostudypulledfromaliteraturereviewwillaccountforallofthevariablesyoufaceinyourowncontext
Intheabsenceofhavingastudythatpredictshowaninterventionworksinyourspecificcontextyouwillneednon-scientificinformationtotrytofillinthosegapsYouwillneedtogatherstoriesfromresidentsstaffmembersandotherstakeholdersthatsitonLevels1or2ofthestandardsofevidence
Someofthekindsofinformationyouwillprobablygetmoreeasilyfrompeoplethanstudiesinclude
howpeopleincitieslikeyoursreactedtoaninterventioninpublicforumshowpeoplewhoimplementedaninterventioninacitylikeyoursfiguredouthowtogetitpassedhowthecitycommunicatedpubliclyabouttheapparentbenefitsanddrawbacksoftheinterventionthepoliticalacceptabilityoftheintervention
Youcanremainsimultaneouslyawarethattheinformationyouaregatheringthroughindividualstoriesisbiased--notgeneralizableinformedbyspecificperspectives--whilealsorecognizingthathavingsomeinformationaboutsomethingimportantisbetterthannoinformationEvidencefromthefirsttwoevidencelevelsareessentialbuildingblocksofknowledgeandperfectlyusefulwhenyoucannotgetdetailedunbiasedknowledge
Youhavetostartsomewhereandyoushouldalwaysstartwithevidenceyouhaveaboutyourveryspecificconditionsthatyouknowareimportantforsuccess
Workingwitharangeofevidence
58
Thankyou
Interviewswithcities
GovExheldsemi-structuredinterviewswithsixstaffmembersinfourcitiestoestablishtheinitialstepsoftheRoadmapWeaskedrespondentstotalkaboutaninitiativeorpolicythattheyhavebeenresponsibleforimplementingintheircityRespondentsfromthreecitiesspokeabouttheireffortstoaddresshomelessnessandrespondentsfromonecityspokeaboutblightAfterrespondentsidentifiedtheproblemtheyspokeabouthowthecityprioritizedthisissuethevariousstepsinvolvedandbarrierspresent
GovExconductedtextanalysisoftheseinterviewswhichrevealedthatpeoplewasthemostcommonlyusedtermduringtheseconversationswhendiscussingthestepstochangeaprogramorpolicyThisisathemethroughoutthisRoadmapThefirstsectionbeginswithidentifyingtherelevantpeopleTheconversationsaroundevidencearefocusedonhumanbehaviorratherthanmethodologyorstrengthofevidence
Thankyou
59
OncewehadadraftoftheRoadmapwefollowedupwithcitiesonanindividualbasistolearnmoreaboutwhatwouldbehelpfulWearegratefultothefollowingcitystafffortheinterviewsandindividualconversations
MichaelBaskinChiefPolicyOfficerChattanoogaTN
MelissaBridgesPerformanceandInnovationCoordinatorLittleRockAR
VictoriaCarreoacutenAdministrativeOfficerLasVegasNV
TyroneGrandstrandCommunityDevelopmentPlannerFargoND
TibLaughlinDirectorofGeneralServicesKansasCityKS
TimMorelandDirectorofPerformanceManagementandOpenDataChattanoogaTN
CharlesRosendahlManagementAnalystDentonTX
Thankyou
60
MelissaSiebenAssistantCountyAdministratorKansasCityKS
GinaThompsonDirectorofStrategicServicesLewisvilleTX
JHUFacultycollaboration
GovExalsoconvenedthecitycontactsandfacultypartnersinaseriesofAdvisoryGroupmeetingsThesemeetingsservedtoconnectcitiesandfacultytounderstandoneanothersperspective
StefanBaralBloombergSchoolofPublicHealth
AshleyBernerJHUSchoolofEducation
MelissaDavey-RothwellBloombergSchoolofPublicHealth
JonLinksBloombergSchoolofPublicHealth
TamarMendelsonBloombergSchoolofPublicHealth
KristinMmariBloombergSchoolofPublicHealth
KeshiaPollackPorterBloombergSchoolofPublicHealth
JoshSharfsteinBloombergSchoolofPublicHealth
LizStuartBloombergSchoolofPublicHealth
LindsayThompsonCareyBusinessSchool
DanielWebsterBloombergSchoolofPublicHealth
Partnersindataandevidence-basedpolicy
Wealsospokewithpartnerswhoareexpertsinthefieldofevidence-basedpolicywhosharedtheirknowledgeandresourcesofcurrentandfuturethinkinginevidence-informedthinking
WendyBarrenoAssociateDirectorWilsonSheehanLabforEconomicOpportunitiesUniversityofNotreDame
TiffanyDavenportDirectorofDataPartnershipsBaltimoresPromise
TomiHiersDirectorBaltimoreCivicSiteAnnieECaseyFoundation
ChrisKingsleySeniorAssociateAnnieECaseyFoundation
KathrynOliverAssociateProfessorLondonSchoolofHygieneandTropicalMedicine
Thankyou
61
KristenWatkinsSeniorPolicyAssociateJ-PAL
GovExParticipants
GovExAnalystMiriamMcKinneyanalyzedtheinterviewtranscriptsandconductedtextanalysisSeniorImplementationAdvisorEmilyShawwillintegrateRoadmapactivitieswithadditionalservicedeliveryactivitiesatGovEx
Thankyou
62
ReferencesAbrahamKGampHaskinsR(2017)Thepromiseofevidence-basedpolicymakingReportofthecommissiononevidence-basedpolicymakingWashingtonDCCommissiononEvidence-basedPolicymakingBigmove2Extendingthegreen(nd)Retrievedfromhttpswwwcityoflewisvillecomdoing-businesslewisville-2025big-move-2-extending-the-greenBatesMAampGlennersterR(2017)ThegeneralizabilitypuzzleRetrievedfromhttpsssirorgarticlesentrythe_generalizability_puzzleBuildwithnotfor(2018January)RetrievedfromhttpsbuildwithnotforwordpresscomCairneyPampOliverK(2017)Evidence-basedpolicymakingisnotlikeevidence-basedmedicinesohowfarshouldyougotobridgethedividebetweenevidenceandpolicyHealthResearchPolicyandSystemsdoi101186s12961-017-0192-xCairneyPOliverKampWellsteadA(2016)TobridgethedividebetweenevidenceandpolicyReduceambiguityasmuchasuncertaintyPublicAdministrationReview76(3)399-402Retrievedfromhttpdxdoiorg101111puar12555Centreforpublicimpact(nd)RetrievedfromhttpswwwcentreforpublicimpactorgCernaL(2013)ThenatureofpolicychangeandimplementationAreviewofdifferenttheoreticalapproachesOrganisationforEconomicCo-OperationandDevelopmentRetrievedfromhttpswwwoecdorgeduceriTheNatureofPolicyChangeandImplementationpdfColoradoSprings(nd)Retrievedfromhttpscoloradospringsgovcommunicationspagewhat-expect-when-you-speakupContinuumofcare(coc)program(nd)RetrievedfromhttpswwwhudexchangeinfoprogramscocCouncilpresidentkicksoff$12millionyouthfund(nd)Retrievedfromhttpwwwbaltimorecitycouncilcomcontentcouncil-president-kicks-12-million-youth-fundDurlakJAampDuPreEP(2008)ImplementationmattersAreviewofresearchontheinfluenceofimplementationonprogramoutcomesandthefactorsaffectingimplementationAmericanJournalofCommunityPsychology41327-350Retrievedfromhttpswwwncbinlmnihgovpubmed18322790Fishbonediagram(ishikawa)(nd)Retrievedfromhttpasqorglearn-about-qualitycause-analysis-toolsoverviewfishbonehtmlFivewhysGettingtotherootofaproblemquickly(nd)RetrievedfromhttpswwwmindtoolscompagesarticlenewTMC_5Whtm
References
63
FligsteinNampMcAdamD(2011)TowardageneraltheoryofstrategicactionfieldsSociologicalTheory29(1)1-26RetrievedfromhttpsociologyberkeleyedusitesdefaultfilesfacultyfligsteinFligstein20McAdam20Soc20Theory20PaperpdHeinkelP(2018January12)CitymullspermitsforbikesharingcompaniesRetrievedfromhttpwwwdentonrccomnewsdenton20180112city-mulls-permits-bike-sharing-companiesHowisareamedianincomecalculated(2012October)Retrievedfromhttpswwwhudexchangeinfofaqs843how-is-area-median-income-calculatedLampkinLMampHatryHP(2003)KeystepsinoutcomemanagementUrbanInstituteRetrievedfromhttpswwwurbanorgresearchpublicationkey-steps-outcome-managementMarshDampSmithM(2000)UnderstandingpolicynetworksTowardsadialecticalapproachPoliticalStudies484-21Retrievedfromhttpjournalssagepubcomdoiabs1011111467-924800247McCannL(nd)RetrievedfromhttplaurenellencomMoultonSampSandfortJR(2016)ThestrategicactionfieldframeworkforpolicyimplementationresearchPolicyStudiesJournal45(1)Retrievedfromhttponlinelibrarywileycomstore101111psj12147assetpsj12147pdfv=1ampt=jdkhhcp4amps=e184c625e1855a6b1615ccb99ee27b1c0b3212d8PaynterB(2018January1)What100citiesarelearningfromeachotherbysharingtheirdataRetrievedfromhttpswwwfastcompanycom40518004what-100-cities-are-learning-from-each-other-by-sharing-their-dataPeakacademy(nd)Retrievedfromhttpswwwdenvergovorgcontentdenvergovenmayors-officeprograms-initiativespeak-performancepeak-academyhtmlPinnockHBarwickMCarpenterCREldridgeSGrandesGGriffithsCJRycroft-MaloneJMeissnerPMurrayEPatelASheikhAampTaylorSJC(2017)Standardsforreportingimplementationstudies(stari)ExplanationandelaborationdocumentBMJOpenRetrievedfromhttpbmjopenbmjcomcontentbmjopen74e013318fullpdfProctorESilmereHRaghavanRHovmandPAaronsGBungerAGriffeyRampHensleyM(2011)OutcomesforimplementationresearchConceptualdistinctionsmeasurementchallengesandresearchagendaAdministrationandPolicyinMentalHealth38(2)65-76doi101007s10488-010-0319-7TandonSD(2018)ProgressincommunityhealthpartnershipsResearcheducationandactionRetrievedfromhttpswwwpressjhuedujournalsprogress-community-health-partnerships-research-education-and-actionProvanKGampKenisP(2007)ModesofnetworkgovernanceStructuremanagementandeffectivenessJournalofPublicAdministrationResearchandTheory
References
64
18(2)229-252Retrievedfromhttpsacademicoupcomjpartarticle182229935895SabatierPA(1988)Anadvocacycoalitionframeworkofpolicychangeandtheroleofpolicy-orientedlearningthereinPolicySciences21(3)129-168Retrievedfromhttpwwwjstororgstable4532139SandstromAampCarlssonL(2008)TheperformanceofpolicynetworksTherelationbetweennetworkstructureandnetworkperformancePolicyStudiesJournal36497-524Retrievedfromhttponlinelibrarywileycomdoi101111j1541-0072200800281xabstractSchwartzMampWilsonE(nd)WhocanaffordtoliveinahomeAlookatdatafromthe2006americancommunitysurveyRetrievedfromhttpswwwcensusgovhousingcensuspublicationswho-can-affordpdfShearerJCAbelsonAKouyateBLavisJNampWaltG(2016)WhydopolicieschangeInstitutionsinterestsideasandnetworksinthreecasesofpolicyreformHealthPolicyandPlanning31(9)1200-1211Retrievedfromhttpsdoiorg101093heapolczw052ThelessonsofadministrativedataHigh-profilepolicy-relevantresearchpoweredbyadministrativedata(nd)Retrievedfromhttpswwwpovertyactionlaborgsitesdefaultfilesresources20161114-The-Lessons-of-Administrative-DatapdfToolsandotherneedsforqualitativeandquantitativemethods(2015February)AgencyforHealthcareResearchandQualityRetrievedfromhttpwwwahrqgovprofessionalsquality-patient-safetytalkingqualityassesstoolshtml
WeaverRCampBagchi-SenS(2013)SpatialanalysisofurbandeclineThegeographyofblightAppliedGeography4061-70
References
65
UniversityPartners
Name Title Affiliation Area(s)ofexpertise
Contactinformation
Examplesofworkingwithnon-academicdecisionmakers
WendyPBarreno
AssociateDirector
WilsonSheehanLabforEconomicOpportunitiesUniversityofNotreDame
LEOsfocusareasincludehealtheducationhousingchildwelfareeconomicopportunitycriminaljustice
wbarrenonduduorleondedu
MelissaDavey-RothwellPhD
AssociateScientist
JohnsHopkinsBloombergSchoolofPublicHealth
Health mdavey1jhuedu410-614-5854
BaltimoreCityHealthDepartmentLeadersMarylandDepartmentofHealthAddictionRecoveryServicesinDCExecutiveDirectorsofcommunityorganizationsacrosstheUSthatfocusonHIVSTIsubstanceabuseandimplementationscience
ElizabethStuart
AssociateDeanforEducationandProfessorDepartmentsofMentalHealthBiostatisticsandHealthPolicyand
JohnsHopkinsBloombergSchoolofPublicHealth
Educationsocialpolicystatisticalmethods
estuartjhuedu
Resources
66
Management
Design-ThinkingResources
EngagementGuideforCivicTech
Journeymap
PerformanceManagementResources
DenverPeakAcademy
Proxymeasuresguide
Talkingaboutperformancewithoutcomplicatingit(video)
SettingPerformanceTargetsguide
Datavisualizationguide
Performancemanagementgettingstartedguide
Exercises
Fishboneexercise
5Whys
AdministrativeData
TheLessonsofAdministrativeData(JPAL)
EvaluationResources
Quantitativeandqualitativemethods(USDepartmentofHealthandHumanServicesAgencyforHealthcareResearchandQuality)
KeyStepsinOutcomeManagement(UrbanInstitute)
RCT-YES
Sageresearchmethods
BestPracticesforMixedMethodsResearchintheHealthSciences
ACCORDChecklist
Pilotandimplementationplanning
Benchmarkingexamplesandproxymeasures
Seattleopendataplaybook
Resources
67
Datasciencescopingguide
Resources
68
AboutGovExGovExhelpsgovernmentsusedataandevidencetoimproveresidentsrsquolivesWesupportgovernmentsinbuildingtheirinternalcapacitytooperatesustainableanalyticspracticeswhichimprovecommunityoutcomesinfieldsfrompublicsafetytoeconomicdevelopmentGovExhasaproventrackrecordofworkinginconcertwithpartnerstoprovidetechnicalassistanceandtrainingatscale
SinceourlaunchinApril2015GovExhas
Trainedover2000governmentemployeesinpersonandonlineintopicsincludingdatamanagementperformancemanagementopendatacommunityengagementandanalytics
Providedtechnicalassistancetomorethan100localgovernmentsacrosstheglobeimprovingtheiruseofdatatoachieveefficienciesinservicedeliveryandbetteroutcomesforresidents
Assessedthecapacityofmorethan130governmentalorganizationstodeterminehowleadersstaffandexternalstakeholdersmanageandusedata
Localgovernmentsareincreasinglyonthefrontlinesofaddressinglargescaleproblemswithglobalimplicationssuchasclimatechangepublicsafetyandeconomicinequality
Whileevidence-basedinterventionstohelpsolvethoseproblemsareavailablelocalgovernmentstaffhavelimitedtimeandcapacitytofindandimplementresearch-basedsolutionsAsaresultcitiescontinuetoworkthewaytheyalwayshave--makingincrementalimprovementstoprogramsbutnotalwayscreatingchangeatasystem-level
ThroughresourcedevelopmentandcollaborationwithJHUfacultyGovExissupportingcityleadersinusingdataandevidencetomakedecisionsandcreatinglastingrelationshipswithresearchers
TheRoadmapwasdevelopedbyGovExaspartofitspartnershipwithBloombergPhilanthropiesWhatWorksCitiesinitiative
AboutGovEx
69
- Title page
- Introduction and purpose of the Roadmap
- Using the Roadmap
-
- Gather information about the problem
- Strategically engage key partners
- Review the evidence and collect details
- Decide on a policy to implement based on local factors
- Establish a framework to monitor progress
- Assess data accessibility quality and security
- Design implementation and evaluation
- Share your story and success
-
- Working with a range of evidence
- Thank you
- References
- Resources
- About GovEx
-
DescribeyourcityrsquoschallengeWriteoutthechallengeyourcityisfacing(asdefinedinyourproblemstatement)HowdoesitaffectresidentsHowdoesthisproblemaffecttheworkofcitystaffmembers
AddthedetailsProvidedetailsonyourappliedresearchstrategyTalkaboutwhatyourcityhasdonenthepasttoresolvetheissueDiscusshowyourstrategyhaschangedwhenyoubegantoadoptthisapproachWasmoneysavedorinvested
Usethisstorymapexercisetoorganizeyourthoughtsandguaranteethatyouincludealloftheelementsofatraditionalstory
AfteryoudevelopyourmessagingcreateacommunicationsmarketingandoutreachstrategydesignedtospreadyourmessageYourstrategyshouldwalkyouthroughthefollowingkeyactivities
1 IdentifyandprofileyouraudienceWhencreatinganddisseminatingyourmessageitisalwaysgoodtothinkaboutyouraudienceWhoexactlywouldbeinterestedinwhatyouhavetosayThegeneralpopulationofthecityPeoplewhoworkinnonprofitsBusinessownersItmattersbecausenoteveryonewholivesinacitymaybethetargetoftheinformationBusinessownersmaybemoreinterestedindatashowingeffectsonthebusinessdistrictthanothersandsoonWewillcoverthislaterbutthisstepwillalsohelpyoutothinkofthemosteffectivewaytoreachyouraudience
2 Defineyourcalltoaction(ifyouhaveone)SpendsometimeconsideringwhyyouareinterestedintellingyourstoryDoyousimplywanttoupdatethepubliconyourprogressorcelebrateyoursuccessesAreyouinterestedinyouraudiencechangingitsbehaviororsupportingaspecificinitiativebeinglaunchedbyyourorganizationIfyouwanttohaveyouraudienceactorbehaveinacertainwaybespecificandincorporateitinyourmessaging
Shareyourstoryandsuccess
52
3 SelectperformancemeasuresThinkabouthowyouwoulddetermineifyourcampaignissuccessfulandselectappropriateperformancemeasuresYouwillrevisitthesemeasuresattheendofyourcampaigncommunicationsplanimplementationtodeterminewhetheryouneedtomodifyyourcommunicationsandoutreachstrategyforthefuture
4 MethodstocommunicateyourstoryTherearemanywaystocommunicatesuccessofyourprojectHerearesomesuggestionsonreachingaudiencesItisimportanttodeterminewhichmethodsarethemosteffectiveforyourdesiredaudiencetoreceiveinformationDoyourresearch(PewResearchCenterhasexcellentstudiesonthereachofmediadigitalmediaandmarketing)ForexampleifseniorsareyourtargetpopulationandtheyarelessinclinedtouseInstagram(thatistrue)youmaywanttoconsiderFacebookorthenightlylocalnewsbothofwhichareusedmoreoftenbyseniorsBelowisalistofsomemethodsandtoolsyoucanuse
i Pressreleasestolocalnewspapersandtvstations
ii ShortsmartphonevideosAgreatwaytoengageanaudienceisthroughvideoAskonequestionofsomeonewhoispartofthestoryMaybethereisasocialworkerwhocandiscusshungerinyourcityThesevideoscanbereusedonmultipleplatforms
iii Long-formwriting
iv TellingindividualstoriesthroughaseriesofphotosandshortnarrativesforpublicationonFacebookorInstagram
v Op-edsopinionpiecesplacedinthelocalpaper
vi Emailupdatestoyourlist
vii Connectingwithappropriateorganizationsthatareinvestedinthetopicandworkwiththecommunityandaskthemtodisseminateinformationtotheirmembers
viii EmployingldquoWordofMouthrdquo(WOM)tacticsbycreatingtownhallsorfreeeventsforresidentstoattendintheirownneighborhoods
5 DeterminethetypeofcollateralUsuallycampaignsthattargetresidentsrequireseveralavenuestoreachthemPeoplearenotalwaysinthesameplaceorspaceHavingakeenunderstandingofyouraudiencewillassistyouwithyourchoicesonwhatneedstobedevelopedSometimesthatmeansimagesforsocialmediaaonepageflierthatgivespeopleinformationonattendinganeventoranin-depthbrochure
Shareyourstoryandsuccess
53
6 ImplementyourstrategyKeepinmindstrategiescanbefluidSometimessomethingmaynotturnoutasexpectedevenifduringplanningsessionsitwasldquosupposedtordquoItisokYoucananalyzewhathappenedduringadebriefsessionandonceyoursurveysandothermeasurementshavebeenfinalized
Visualizations
Withtherightvisualizationyoucanreviewandshareinformationinsightsanalysisperformanceandtrendsallofwhichisvaluabletoacross-sectionofthepublicButwhichvisualizationisrightforyouTochoosetherighttypeofchartgraphdeterminethetypeofquestionyouwanttoanswerSomecommoncharttypesarebelowandmoreinformationcanbefoundhere
Exploreadditionalanalyticaltechniques
Duringthegoal-settingphaseyouestablishedoutputoutcomeandimplementationmeasuresBasedontheresultsofyourpilotyoumayneedtoadjustyourmeasuresWiththepilotdatainhandyoumayhavesomeideasofhowfutureanalysescouldbedifferentandpotentiallyimprovethequalityofyourprogramTakesometimetobrainstormotherrelevantdatasetsthatyoudidnothavethetimetopursueearlierincludingmessydepartmentaldatathatneedsmoreattentionorsourcesofpublicdatafromthestateandfederalgovernmentnonprofitsandevenprivatecompaniesthatcouldsupplementyourunderstandingofanissueUsingmachinelearningtechniquesyoumightbeabletobetter
Shareyourstoryandsuccess
54
predictdemandforhomelessservicesusingNOAAweatherforecastspreemptwatermainbreaksusingsoilstratigraphyorinvestineconomicdevelopmentbymonitoringrealestatelistingdata
Advancedanalyticaltechniquescanalsohelpyourimplementationrunmoreeffectivelybyintegratingdata-informedtargetinganddecisiontoolsintotheprocessByautomatingroutineworkflowsandsurfacingsubtletrendsthesetoolscanhelpyouiterateinamoreprecisemannerandprovidekeystrategicguidancetofront-lineemployeesthatwouldotherwisebeunavailableThesesortsofdata-science-enhancedprocessescanbeclassifiedasoneofsixprojecttypesasexploredfurtherbytheNewOrleansLANOLAlyticsteamFormoreguidanceonscopingasimilarprojecttheGovExadvancedanalyticsscopingguidegivesyouastep-by-stepoverviewofhowtobesuccessful
Usethespacebelowtolistyourideasforpossibleadvancedanalyticsprojectsbytype
Typeofproject Ideas
Findingtheneedleinthehaystack
Prioritizingforimpact
Earlywarningtools
Betterquickerdecisions
Optimizingresourceallocation
Experimentingforwhatworks
Keepthefeedbackloopgoing
GatherinformationonpublicsentimentaboutpolicychangethroughfocusgroupssurveysandpublicmeetingsledbyadvisorygroupmembersNotewhethernewinformationisavailablethatcanhelpanalyzeorcontextualizeyourprogress(egfederalorstatedatareports)Sharetheresultsofyourpilotandthepolicychangeprocesswithothercitiesandresearchers
Shareyourstoryandsuccess
55
WorkingwitharangeofevidencePresentlythereisalotofattentionandinvestmentinevidence-basedpolicymakingincludingthefederalbipartisanCommissiononEvidence-BasedPolicymakingthePewCharitableTrustsreportsandtoolkitsonHowStatesEngageinEvidence-BasedPolicymakingandtheLauraandJohnArnoldFoundationsupportedEvidence-BasedPolicymakingCollaborativeManyoftheseinitiativesfocusonrigorousevaluation--meaningrandomizedcontrolledtrials(RCT)orotherexperimentalmodels--asthebestkindofevidencetouseHoweverthisbodyofevidenceisofteninsufficientforthespecifickindsofquestionsthatgovernmentscanhavewhenmakingadecisionaboutpolicyInthisguideweproposeanapproachtosupportgovernmentsinincorporatingadditionaltypesofevidenceintotheirdecisionmaking
OnewaytounderstandarangeofevidenceistouseNestarsquosStandardsofEvidencetoframeanddefineevidenceTheBritishnonprofitNestaworkstoinjectpositivechangeintothewayUKgovernmentpoliciesandprogramsfunctionOneoftheirkeyareasofworkhasbeendevelopingWhatWorksCentresorganizationsthatcollectcommissionsynthesizeanddisseminateevidenceaboutinterventioneffectivenessacrossarangeofsocialimpactareasNestadevelopedthefollowingStandardsofEvidencetoprovideausefulwaytodocumenthowdifferentkindsofinformationdemonstratedifferentlevelsofevidence
Workingwitharangeofevidence
56
Theimportanceofpotentiallybiasedinformation
Onechallengepresentedbyanevidenceclassificationthatacknowledgesthevalueofnon-scientificformsofevidenceistofigureouthowtobalanceallofthepiecestogetherwhenyouusethemTounderstandhowtodothiswehavetothinkabouthowtopositionpotentiallybiasedinformationindecisionmaking
WhenwelookatinformationwhichhasnotbeenrigorouslyevaluatedweacceptthatthisinformationmightbebiasedThewholereasonweconductrigorousevaluationistoremovebiasfromourobservationsWeconductstructuredtrialstoproduceresultsthatareconsistentandgeneralizableIfafindingisunbiasedthisaccuracymdashthislackofbiasmdashcanbeprovedbecauseanyindependenttestercanreproduceitwhentheyfollowanidenticalprocess
YoucanlookatpotentiallybiasedinformationasjustinformationthatisearlieroninitspathtorefinementreplicationandverifiedaccuracyApieceofinformationbasedonanyoneindividualrsquosobservationsislikelytobebiasedbecauseitiscomingfromjustonesourceBiasisreducedbygatheringtogetherlotsofindependentsourcesofinformationandfiguringoutitsmostcommonattributes
MaybethebiasedinformationmakesaclaimthatistoolargemdashitsayssomethingistrueunderallconditionswhenitisreallyonlytrueundercertainconditionsorthataneffectofaninterventionisbiggerthanitstrueaverageorthataneffectlastsovertimewhenitdoesnotPerhapschangingtheclaimtobemorespecificmdashcriticallyreviewingtheinformationtoseeifspecificconditionsmustbemettoachievetheoutcomesmdashwouldmaketheclaimmoreaccurate
IfyoutestedyourclaimfounditdidnotholdupinsomecaseschangedittesteditagainuntilyourclaimrepeatedlyheldupthenyoucoulddemonstratethattheclaimwasnolongerbiasedbutaccurateThisprocessoftestingimprovingtestingandvalidatingiswhatwegetfromstructuredscientificstudy
Withindividualpiecesofinformation--crediblehypothesescasestudiesexpertopinion--youcannotassumethatkindofaccuracyAtthesametimetheseearlyobservationsarefoundationaltoanyreplicablescientificobservationsWewillnevergetanywhereifwedonotstartwiththesepiecesofinformationeveniftheymightbebiased
Combininglevelsofevidencetogetthemostdetailedinformationforyourcity
ItisespeciallyimportanttoacceptdifferentformsofevidencewhenyoufacequestionsthathavenotyetbeentestedbyindependentstudiesSomeofthemostimportantquestionsrelatetothingsthatareveryspecifictoyoursituationYoufaceaparticularcombinationof
Workingwitharangeofevidence
57
issuesactorstrendsandresourcesthatisultimatelydifferentfromeveryotherplaceintheworld
ThisisimportantbecauseaspolicypractitionersandnotscientistsyoucaremoreaboutyourspecificcasethanahypotheticalaveragegeneralcaseYoucarelessaboutthetotallikelyrangeofimpactsfromapolicythanhowthissingleimplementationwillworkYouwantinformationaboutpoliciesthatwillprovideyouwiththebestpossiblepredictionabouthowtheywillworkinyourcontext
ByputtingdifferentkindsofevidenceintothesameframeworkNestarsquosStandardsofEvidenceimplythatwhenyoucannotgetrigorouslytestedexperimentallyverifiedinformationitisappropriatetoworkwithwhatwedohaveThisisrelevantforthemunicipalimplementationcontextbecausenostudypulledfromaliteraturereviewwillaccountforallofthevariablesyoufaceinyourowncontext
Intheabsenceofhavingastudythatpredictshowaninterventionworksinyourspecificcontextyouwillneednon-scientificinformationtotrytofillinthosegapsYouwillneedtogatherstoriesfromresidentsstaffmembersandotherstakeholdersthatsitonLevels1or2ofthestandardsofevidence
Someofthekindsofinformationyouwillprobablygetmoreeasilyfrompeoplethanstudiesinclude
howpeopleincitieslikeyoursreactedtoaninterventioninpublicforumshowpeoplewhoimplementedaninterventioninacitylikeyoursfiguredouthowtogetitpassedhowthecitycommunicatedpubliclyabouttheapparentbenefitsanddrawbacksoftheinterventionthepoliticalacceptabilityoftheintervention
Youcanremainsimultaneouslyawarethattheinformationyouaregatheringthroughindividualstoriesisbiased--notgeneralizableinformedbyspecificperspectives--whilealsorecognizingthathavingsomeinformationaboutsomethingimportantisbetterthannoinformationEvidencefromthefirsttwoevidencelevelsareessentialbuildingblocksofknowledgeandperfectlyusefulwhenyoucannotgetdetailedunbiasedknowledge
Youhavetostartsomewhereandyoushouldalwaysstartwithevidenceyouhaveaboutyourveryspecificconditionsthatyouknowareimportantforsuccess
Workingwitharangeofevidence
58
Thankyou
Interviewswithcities
GovExheldsemi-structuredinterviewswithsixstaffmembersinfourcitiestoestablishtheinitialstepsoftheRoadmapWeaskedrespondentstotalkaboutaninitiativeorpolicythattheyhavebeenresponsibleforimplementingintheircityRespondentsfromthreecitiesspokeabouttheireffortstoaddresshomelessnessandrespondentsfromonecityspokeaboutblightAfterrespondentsidentifiedtheproblemtheyspokeabouthowthecityprioritizedthisissuethevariousstepsinvolvedandbarrierspresent
GovExconductedtextanalysisoftheseinterviewswhichrevealedthatpeoplewasthemostcommonlyusedtermduringtheseconversationswhendiscussingthestepstochangeaprogramorpolicyThisisathemethroughoutthisRoadmapThefirstsectionbeginswithidentifyingtherelevantpeopleTheconversationsaroundevidencearefocusedonhumanbehaviorratherthanmethodologyorstrengthofevidence
Thankyou
59
OncewehadadraftoftheRoadmapwefollowedupwithcitiesonanindividualbasistolearnmoreaboutwhatwouldbehelpfulWearegratefultothefollowingcitystafffortheinterviewsandindividualconversations
MichaelBaskinChiefPolicyOfficerChattanoogaTN
MelissaBridgesPerformanceandInnovationCoordinatorLittleRockAR
VictoriaCarreoacutenAdministrativeOfficerLasVegasNV
TyroneGrandstrandCommunityDevelopmentPlannerFargoND
TibLaughlinDirectorofGeneralServicesKansasCityKS
TimMorelandDirectorofPerformanceManagementandOpenDataChattanoogaTN
CharlesRosendahlManagementAnalystDentonTX
Thankyou
60
MelissaSiebenAssistantCountyAdministratorKansasCityKS
GinaThompsonDirectorofStrategicServicesLewisvilleTX
JHUFacultycollaboration
GovExalsoconvenedthecitycontactsandfacultypartnersinaseriesofAdvisoryGroupmeetingsThesemeetingsservedtoconnectcitiesandfacultytounderstandoneanothersperspective
StefanBaralBloombergSchoolofPublicHealth
AshleyBernerJHUSchoolofEducation
MelissaDavey-RothwellBloombergSchoolofPublicHealth
JonLinksBloombergSchoolofPublicHealth
TamarMendelsonBloombergSchoolofPublicHealth
KristinMmariBloombergSchoolofPublicHealth
KeshiaPollackPorterBloombergSchoolofPublicHealth
JoshSharfsteinBloombergSchoolofPublicHealth
LizStuartBloombergSchoolofPublicHealth
LindsayThompsonCareyBusinessSchool
DanielWebsterBloombergSchoolofPublicHealth
Partnersindataandevidence-basedpolicy
Wealsospokewithpartnerswhoareexpertsinthefieldofevidence-basedpolicywhosharedtheirknowledgeandresourcesofcurrentandfuturethinkinginevidence-informedthinking
WendyBarrenoAssociateDirectorWilsonSheehanLabforEconomicOpportunitiesUniversityofNotreDame
TiffanyDavenportDirectorofDataPartnershipsBaltimoresPromise
TomiHiersDirectorBaltimoreCivicSiteAnnieECaseyFoundation
ChrisKingsleySeniorAssociateAnnieECaseyFoundation
KathrynOliverAssociateProfessorLondonSchoolofHygieneandTropicalMedicine
Thankyou
61
KristenWatkinsSeniorPolicyAssociateJ-PAL
GovExParticipants
GovExAnalystMiriamMcKinneyanalyzedtheinterviewtranscriptsandconductedtextanalysisSeniorImplementationAdvisorEmilyShawwillintegrateRoadmapactivitieswithadditionalservicedeliveryactivitiesatGovEx
Thankyou
62
ReferencesAbrahamKGampHaskinsR(2017)Thepromiseofevidence-basedpolicymakingReportofthecommissiononevidence-basedpolicymakingWashingtonDCCommissiononEvidence-basedPolicymakingBigmove2Extendingthegreen(nd)Retrievedfromhttpswwwcityoflewisvillecomdoing-businesslewisville-2025big-move-2-extending-the-greenBatesMAampGlennersterR(2017)ThegeneralizabilitypuzzleRetrievedfromhttpsssirorgarticlesentrythe_generalizability_puzzleBuildwithnotfor(2018January)RetrievedfromhttpsbuildwithnotforwordpresscomCairneyPampOliverK(2017)Evidence-basedpolicymakingisnotlikeevidence-basedmedicinesohowfarshouldyougotobridgethedividebetweenevidenceandpolicyHealthResearchPolicyandSystemsdoi101186s12961-017-0192-xCairneyPOliverKampWellsteadA(2016)TobridgethedividebetweenevidenceandpolicyReduceambiguityasmuchasuncertaintyPublicAdministrationReview76(3)399-402Retrievedfromhttpdxdoiorg101111puar12555Centreforpublicimpact(nd)RetrievedfromhttpswwwcentreforpublicimpactorgCernaL(2013)ThenatureofpolicychangeandimplementationAreviewofdifferenttheoreticalapproachesOrganisationforEconomicCo-OperationandDevelopmentRetrievedfromhttpswwwoecdorgeduceriTheNatureofPolicyChangeandImplementationpdfColoradoSprings(nd)Retrievedfromhttpscoloradospringsgovcommunicationspagewhat-expect-when-you-speakupContinuumofcare(coc)program(nd)RetrievedfromhttpswwwhudexchangeinfoprogramscocCouncilpresidentkicksoff$12millionyouthfund(nd)Retrievedfromhttpwwwbaltimorecitycouncilcomcontentcouncil-president-kicks-12-million-youth-fundDurlakJAampDuPreEP(2008)ImplementationmattersAreviewofresearchontheinfluenceofimplementationonprogramoutcomesandthefactorsaffectingimplementationAmericanJournalofCommunityPsychology41327-350Retrievedfromhttpswwwncbinlmnihgovpubmed18322790Fishbonediagram(ishikawa)(nd)Retrievedfromhttpasqorglearn-about-qualitycause-analysis-toolsoverviewfishbonehtmlFivewhysGettingtotherootofaproblemquickly(nd)RetrievedfromhttpswwwmindtoolscompagesarticlenewTMC_5Whtm
References
63
FligsteinNampMcAdamD(2011)TowardageneraltheoryofstrategicactionfieldsSociologicalTheory29(1)1-26RetrievedfromhttpsociologyberkeleyedusitesdefaultfilesfacultyfligsteinFligstein20McAdam20Soc20Theory20PaperpdHeinkelP(2018January12)CitymullspermitsforbikesharingcompaniesRetrievedfromhttpwwwdentonrccomnewsdenton20180112city-mulls-permits-bike-sharing-companiesHowisareamedianincomecalculated(2012October)Retrievedfromhttpswwwhudexchangeinfofaqs843how-is-area-median-income-calculatedLampkinLMampHatryHP(2003)KeystepsinoutcomemanagementUrbanInstituteRetrievedfromhttpswwwurbanorgresearchpublicationkey-steps-outcome-managementMarshDampSmithM(2000)UnderstandingpolicynetworksTowardsadialecticalapproachPoliticalStudies484-21Retrievedfromhttpjournalssagepubcomdoiabs1011111467-924800247McCannL(nd)RetrievedfromhttplaurenellencomMoultonSampSandfortJR(2016)ThestrategicactionfieldframeworkforpolicyimplementationresearchPolicyStudiesJournal45(1)Retrievedfromhttponlinelibrarywileycomstore101111psj12147assetpsj12147pdfv=1ampt=jdkhhcp4amps=e184c625e1855a6b1615ccb99ee27b1c0b3212d8PaynterB(2018January1)What100citiesarelearningfromeachotherbysharingtheirdataRetrievedfromhttpswwwfastcompanycom40518004what-100-cities-are-learning-from-each-other-by-sharing-their-dataPeakacademy(nd)Retrievedfromhttpswwwdenvergovorgcontentdenvergovenmayors-officeprograms-initiativespeak-performancepeak-academyhtmlPinnockHBarwickMCarpenterCREldridgeSGrandesGGriffithsCJRycroft-MaloneJMeissnerPMurrayEPatelASheikhAampTaylorSJC(2017)Standardsforreportingimplementationstudies(stari)ExplanationandelaborationdocumentBMJOpenRetrievedfromhttpbmjopenbmjcomcontentbmjopen74e013318fullpdfProctorESilmereHRaghavanRHovmandPAaronsGBungerAGriffeyRampHensleyM(2011)OutcomesforimplementationresearchConceptualdistinctionsmeasurementchallengesandresearchagendaAdministrationandPolicyinMentalHealth38(2)65-76doi101007s10488-010-0319-7TandonSD(2018)ProgressincommunityhealthpartnershipsResearcheducationandactionRetrievedfromhttpswwwpressjhuedujournalsprogress-community-health-partnerships-research-education-and-actionProvanKGampKenisP(2007)ModesofnetworkgovernanceStructuremanagementandeffectivenessJournalofPublicAdministrationResearchandTheory
References
64
18(2)229-252Retrievedfromhttpsacademicoupcomjpartarticle182229935895SabatierPA(1988)Anadvocacycoalitionframeworkofpolicychangeandtheroleofpolicy-orientedlearningthereinPolicySciences21(3)129-168Retrievedfromhttpwwwjstororgstable4532139SandstromAampCarlssonL(2008)TheperformanceofpolicynetworksTherelationbetweennetworkstructureandnetworkperformancePolicyStudiesJournal36497-524Retrievedfromhttponlinelibrarywileycomdoi101111j1541-0072200800281xabstractSchwartzMampWilsonE(nd)WhocanaffordtoliveinahomeAlookatdatafromthe2006americancommunitysurveyRetrievedfromhttpswwwcensusgovhousingcensuspublicationswho-can-affordpdfShearerJCAbelsonAKouyateBLavisJNampWaltG(2016)WhydopolicieschangeInstitutionsinterestsideasandnetworksinthreecasesofpolicyreformHealthPolicyandPlanning31(9)1200-1211Retrievedfromhttpsdoiorg101093heapolczw052ThelessonsofadministrativedataHigh-profilepolicy-relevantresearchpoweredbyadministrativedata(nd)Retrievedfromhttpswwwpovertyactionlaborgsitesdefaultfilesresources20161114-The-Lessons-of-Administrative-DatapdfToolsandotherneedsforqualitativeandquantitativemethods(2015February)AgencyforHealthcareResearchandQualityRetrievedfromhttpwwwahrqgovprofessionalsquality-patient-safetytalkingqualityassesstoolshtml
WeaverRCampBagchi-SenS(2013)SpatialanalysisofurbandeclineThegeographyofblightAppliedGeography4061-70
References
65
UniversityPartners
Name Title Affiliation Area(s)ofexpertise
Contactinformation
Examplesofworkingwithnon-academicdecisionmakers
WendyPBarreno
AssociateDirector
WilsonSheehanLabforEconomicOpportunitiesUniversityofNotreDame
LEOsfocusareasincludehealtheducationhousingchildwelfareeconomicopportunitycriminaljustice
wbarrenonduduorleondedu
MelissaDavey-RothwellPhD
AssociateScientist
JohnsHopkinsBloombergSchoolofPublicHealth
Health mdavey1jhuedu410-614-5854
BaltimoreCityHealthDepartmentLeadersMarylandDepartmentofHealthAddictionRecoveryServicesinDCExecutiveDirectorsofcommunityorganizationsacrosstheUSthatfocusonHIVSTIsubstanceabuseandimplementationscience
ElizabethStuart
AssociateDeanforEducationandProfessorDepartmentsofMentalHealthBiostatisticsandHealthPolicyand
JohnsHopkinsBloombergSchoolofPublicHealth
Educationsocialpolicystatisticalmethods
estuartjhuedu
Resources
66
Management
Design-ThinkingResources
EngagementGuideforCivicTech
Journeymap
PerformanceManagementResources
DenverPeakAcademy
Proxymeasuresguide
Talkingaboutperformancewithoutcomplicatingit(video)
SettingPerformanceTargetsguide
Datavisualizationguide
Performancemanagementgettingstartedguide
Exercises
Fishboneexercise
5Whys
AdministrativeData
TheLessonsofAdministrativeData(JPAL)
EvaluationResources
Quantitativeandqualitativemethods(USDepartmentofHealthandHumanServicesAgencyforHealthcareResearchandQuality)
KeyStepsinOutcomeManagement(UrbanInstitute)
RCT-YES
Sageresearchmethods
BestPracticesforMixedMethodsResearchintheHealthSciences
ACCORDChecklist
Pilotandimplementationplanning
Benchmarkingexamplesandproxymeasures
Seattleopendataplaybook
Resources
67
Datasciencescopingguide
Resources
68
AboutGovExGovExhelpsgovernmentsusedataandevidencetoimproveresidentsrsquolivesWesupportgovernmentsinbuildingtheirinternalcapacitytooperatesustainableanalyticspracticeswhichimprovecommunityoutcomesinfieldsfrompublicsafetytoeconomicdevelopmentGovExhasaproventrackrecordofworkinginconcertwithpartnerstoprovidetechnicalassistanceandtrainingatscale
SinceourlaunchinApril2015GovExhas
Trainedover2000governmentemployeesinpersonandonlineintopicsincludingdatamanagementperformancemanagementopendatacommunityengagementandanalytics
Providedtechnicalassistancetomorethan100localgovernmentsacrosstheglobeimprovingtheiruseofdatatoachieveefficienciesinservicedeliveryandbetteroutcomesforresidents
Assessedthecapacityofmorethan130governmentalorganizationstodeterminehowleadersstaffandexternalstakeholdersmanageandusedata
Localgovernmentsareincreasinglyonthefrontlinesofaddressinglargescaleproblemswithglobalimplicationssuchasclimatechangepublicsafetyandeconomicinequality
Whileevidence-basedinterventionstohelpsolvethoseproblemsareavailablelocalgovernmentstaffhavelimitedtimeandcapacitytofindandimplementresearch-basedsolutionsAsaresultcitiescontinuetoworkthewaytheyalwayshave--makingincrementalimprovementstoprogramsbutnotalwayscreatingchangeatasystem-level
ThroughresourcedevelopmentandcollaborationwithJHUfacultyGovExissupportingcityleadersinusingdataandevidencetomakedecisionsandcreatinglastingrelationshipswithresearchers
TheRoadmapwasdevelopedbyGovExaspartofitspartnershipwithBloombergPhilanthropiesWhatWorksCitiesinitiative
AboutGovEx
69
- Title page
- Introduction and purpose of the Roadmap
- Using the Roadmap
-
- Gather information about the problem
- Strategically engage key partners
- Review the evidence and collect details
- Decide on a policy to implement based on local factors
- Establish a framework to monitor progress
- Assess data accessibility quality and security
- Design implementation and evaluation
- Share your story and success
-
- Working with a range of evidence
- Thank you
- References
- Resources
- About GovEx
-
3 SelectperformancemeasuresThinkabouthowyouwoulddetermineifyourcampaignissuccessfulandselectappropriateperformancemeasuresYouwillrevisitthesemeasuresattheendofyourcampaigncommunicationsplanimplementationtodeterminewhetheryouneedtomodifyyourcommunicationsandoutreachstrategyforthefuture
4 MethodstocommunicateyourstoryTherearemanywaystocommunicatesuccessofyourprojectHerearesomesuggestionsonreachingaudiencesItisimportanttodeterminewhichmethodsarethemosteffectiveforyourdesiredaudiencetoreceiveinformationDoyourresearch(PewResearchCenterhasexcellentstudiesonthereachofmediadigitalmediaandmarketing)ForexampleifseniorsareyourtargetpopulationandtheyarelessinclinedtouseInstagram(thatistrue)youmaywanttoconsiderFacebookorthenightlylocalnewsbothofwhichareusedmoreoftenbyseniorsBelowisalistofsomemethodsandtoolsyoucanuse
i Pressreleasestolocalnewspapersandtvstations
ii ShortsmartphonevideosAgreatwaytoengageanaudienceisthroughvideoAskonequestionofsomeonewhoispartofthestoryMaybethereisasocialworkerwhocandiscusshungerinyourcityThesevideoscanbereusedonmultipleplatforms
iii Long-formwriting
iv TellingindividualstoriesthroughaseriesofphotosandshortnarrativesforpublicationonFacebookorInstagram
v Op-edsopinionpiecesplacedinthelocalpaper
vi Emailupdatestoyourlist
vii Connectingwithappropriateorganizationsthatareinvestedinthetopicandworkwiththecommunityandaskthemtodisseminateinformationtotheirmembers
viii EmployingldquoWordofMouthrdquo(WOM)tacticsbycreatingtownhallsorfreeeventsforresidentstoattendintheirownneighborhoods
5 DeterminethetypeofcollateralUsuallycampaignsthattargetresidentsrequireseveralavenuestoreachthemPeoplearenotalwaysinthesameplaceorspaceHavingakeenunderstandingofyouraudiencewillassistyouwithyourchoicesonwhatneedstobedevelopedSometimesthatmeansimagesforsocialmediaaonepageflierthatgivespeopleinformationonattendinganeventoranin-depthbrochure
Shareyourstoryandsuccess
53
6 ImplementyourstrategyKeepinmindstrategiescanbefluidSometimessomethingmaynotturnoutasexpectedevenifduringplanningsessionsitwasldquosupposedtordquoItisokYoucananalyzewhathappenedduringadebriefsessionandonceyoursurveysandothermeasurementshavebeenfinalized
Visualizations
Withtherightvisualizationyoucanreviewandshareinformationinsightsanalysisperformanceandtrendsallofwhichisvaluabletoacross-sectionofthepublicButwhichvisualizationisrightforyouTochoosetherighttypeofchartgraphdeterminethetypeofquestionyouwanttoanswerSomecommoncharttypesarebelowandmoreinformationcanbefoundhere
Exploreadditionalanalyticaltechniques
Duringthegoal-settingphaseyouestablishedoutputoutcomeandimplementationmeasuresBasedontheresultsofyourpilotyoumayneedtoadjustyourmeasuresWiththepilotdatainhandyoumayhavesomeideasofhowfutureanalysescouldbedifferentandpotentiallyimprovethequalityofyourprogramTakesometimetobrainstormotherrelevantdatasetsthatyoudidnothavethetimetopursueearlierincludingmessydepartmentaldatathatneedsmoreattentionorsourcesofpublicdatafromthestateandfederalgovernmentnonprofitsandevenprivatecompaniesthatcouldsupplementyourunderstandingofanissueUsingmachinelearningtechniquesyoumightbeabletobetter
Shareyourstoryandsuccess
54
predictdemandforhomelessservicesusingNOAAweatherforecastspreemptwatermainbreaksusingsoilstratigraphyorinvestineconomicdevelopmentbymonitoringrealestatelistingdata
Advancedanalyticaltechniquescanalsohelpyourimplementationrunmoreeffectivelybyintegratingdata-informedtargetinganddecisiontoolsintotheprocessByautomatingroutineworkflowsandsurfacingsubtletrendsthesetoolscanhelpyouiterateinamoreprecisemannerandprovidekeystrategicguidancetofront-lineemployeesthatwouldotherwisebeunavailableThesesortsofdata-science-enhancedprocessescanbeclassifiedasoneofsixprojecttypesasexploredfurtherbytheNewOrleansLANOLAlyticsteamFormoreguidanceonscopingasimilarprojecttheGovExadvancedanalyticsscopingguidegivesyouastep-by-stepoverviewofhowtobesuccessful
Usethespacebelowtolistyourideasforpossibleadvancedanalyticsprojectsbytype
Typeofproject Ideas
Findingtheneedleinthehaystack
Prioritizingforimpact
Earlywarningtools
Betterquickerdecisions
Optimizingresourceallocation
Experimentingforwhatworks
Keepthefeedbackloopgoing
GatherinformationonpublicsentimentaboutpolicychangethroughfocusgroupssurveysandpublicmeetingsledbyadvisorygroupmembersNotewhethernewinformationisavailablethatcanhelpanalyzeorcontextualizeyourprogress(egfederalorstatedatareports)Sharetheresultsofyourpilotandthepolicychangeprocesswithothercitiesandresearchers
Shareyourstoryandsuccess
55
WorkingwitharangeofevidencePresentlythereisalotofattentionandinvestmentinevidence-basedpolicymakingincludingthefederalbipartisanCommissiononEvidence-BasedPolicymakingthePewCharitableTrustsreportsandtoolkitsonHowStatesEngageinEvidence-BasedPolicymakingandtheLauraandJohnArnoldFoundationsupportedEvidence-BasedPolicymakingCollaborativeManyoftheseinitiativesfocusonrigorousevaluation--meaningrandomizedcontrolledtrials(RCT)orotherexperimentalmodels--asthebestkindofevidencetouseHoweverthisbodyofevidenceisofteninsufficientforthespecifickindsofquestionsthatgovernmentscanhavewhenmakingadecisionaboutpolicyInthisguideweproposeanapproachtosupportgovernmentsinincorporatingadditionaltypesofevidenceintotheirdecisionmaking
OnewaytounderstandarangeofevidenceistouseNestarsquosStandardsofEvidencetoframeanddefineevidenceTheBritishnonprofitNestaworkstoinjectpositivechangeintothewayUKgovernmentpoliciesandprogramsfunctionOneoftheirkeyareasofworkhasbeendevelopingWhatWorksCentresorganizationsthatcollectcommissionsynthesizeanddisseminateevidenceaboutinterventioneffectivenessacrossarangeofsocialimpactareasNestadevelopedthefollowingStandardsofEvidencetoprovideausefulwaytodocumenthowdifferentkindsofinformationdemonstratedifferentlevelsofevidence
Workingwitharangeofevidence
56
Theimportanceofpotentiallybiasedinformation
Onechallengepresentedbyanevidenceclassificationthatacknowledgesthevalueofnon-scientificformsofevidenceistofigureouthowtobalanceallofthepiecestogetherwhenyouusethemTounderstandhowtodothiswehavetothinkabouthowtopositionpotentiallybiasedinformationindecisionmaking
WhenwelookatinformationwhichhasnotbeenrigorouslyevaluatedweacceptthatthisinformationmightbebiasedThewholereasonweconductrigorousevaluationistoremovebiasfromourobservationsWeconductstructuredtrialstoproduceresultsthatareconsistentandgeneralizableIfafindingisunbiasedthisaccuracymdashthislackofbiasmdashcanbeprovedbecauseanyindependenttestercanreproduceitwhentheyfollowanidenticalprocess
YoucanlookatpotentiallybiasedinformationasjustinformationthatisearlieroninitspathtorefinementreplicationandverifiedaccuracyApieceofinformationbasedonanyoneindividualrsquosobservationsislikelytobebiasedbecauseitiscomingfromjustonesourceBiasisreducedbygatheringtogetherlotsofindependentsourcesofinformationandfiguringoutitsmostcommonattributes
MaybethebiasedinformationmakesaclaimthatistoolargemdashitsayssomethingistrueunderallconditionswhenitisreallyonlytrueundercertainconditionsorthataneffectofaninterventionisbiggerthanitstrueaverageorthataneffectlastsovertimewhenitdoesnotPerhapschangingtheclaimtobemorespecificmdashcriticallyreviewingtheinformationtoseeifspecificconditionsmustbemettoachievetheoutcomesmdashwouldmaketheclaimmoreaccurate
IfyoutestedyourclaimfounditdidnotholdupinsomecaseschangedittesteditagainuntilyourclaimrepeatedlyheldupthenyoucoulddemonstratethattheclaimwasnolongerbiasedbutaccurateThisprocessoftestingimprovingtestingandvalidatingiswhatwegetfromstructuredscientificstudy
Withindividualpiecesofinformation--crediblehypothesescasestudiesexpertopinion--youcannotassumethatkindofaccuracyAtthesametimetheseearlyobservationsarefoundationaltoanyreplicablescientificobservationsWewillnevergetanywhereifwedonotstartwiththesepiecesofinformationeveniftheymightbebiased
Combininglevelsofevidencetogetthemostdetailedinformationforyourcity
ItisespeciallyimportanttoacceptdifferentformsofevidencewhenyoufacequestionsthathavenotyetbeentestedbyindependentstudiesSomeofthemostimportantquestionsrelatetothingsthatareveryspecifictoyoursituationYoufaceaparticularcombinationof
Workingwitharangeofevidence
57
issuesactorstrendsandresourcesthatisultimatelydifferentfromeveryotherplaceintheworld
ThisisimportantbecauseaspolicypractitionersandnotscientistsyoucaremoreaboutyourspecificcasethanahypotheticalaveragegeneralcaseYoucarelessaboutthetotallikelyrangeofimpactsfromapolicythanhowthissingleimplementationwillworkYouwantinformationaboutpoliciesthatwillprovideyouwiththebestpossiblepredictionabouthowtheywillworkinyourcontext
ByputtingdifferentkindsofevidenceintothesameframeworkNestarsquosStandardsofEvidenceimplythatwhenyoucannotgetrigorouslytestedexperimentallyverifiedinformationitisappropriatetoworkwithwhatwedohaveThisisrelevantforthemunicipalimplementationcontextbecausenostudypulledfromaliteraturereviewwillaccountforallofthevariablesyoufaceinyourowncontext
Intheabsenceofhavingastudythatpredictshowaninterventionworksinyourspecificcontextyouwillneednon-scientificinformationtotrytofillinthosegapsYouwillneedtogatherstoriesfromresidentsstaffmembersandotherstakeholdersthatsitonLevels1or2ofthestandardsofevidence
Someofthekindsofinformationyouwillprobablygetmoreeasilyfrompeoplethanstudiesinclude
howpeopleincitieslikeyoursreactedtoaninterventioninpublicforumshowpeoplewhoimplementedaninterventioninacitylikeyoursfiguredouthowtogetitpassedhowthecitycommunicatedpubliclyabouttheapparentbenefitsanddrawbacksoftheinterventionthepoliticalacceptabilityoftheintervention
Youcanremainsimultaneouslyawarethattheinformationyouaregatheringthroughindividualstoriesisbiased--notgeneralizableinformedbyspecificperspectives--whilealsorecognizingthathavingsomeinformationaboutsomethingimportantisbetterthannoinformationEvidencefromthefirsttwoevidencelevelsareessentialbuildingblocksofknowledgeandperfectlyusefulwhenyoucannotgetdetailedunbiasedknowledge
Youhavetostartsomewhereandyoushouldalwaysstartwithevidenceyouhaveaboutyourveryspecificconditionsthatyouknowareimportantforsuccess
Workingwitharangeofevidence
58
Thankyou
Interviewswithcities
GovExheldsemi-structuredinterviewswithsixstaffmembersinfourcitiestoestablishtheinitialstepsoftheRoadmapWeaskedrespondentstotalkaboutaninitiativeorpolicythattheyhavebeenresponsibleforimplementingintheircityRespondentsfromthreecitiesspokeabouttheireffortstoaddresshomelessnessandrespondentsfromonecityspokeaboutblightAfterrespondentsidentifiedtheproblemtheyspokeabouthowthecityprioritizedthisissuethevariousstepsinvolvedandbarrierspresent
GovExconductedtextanalysisoftheseinterviewswhichrevealedthatpeoplewasthemostcommonlyusedtermduringtheseconversationswhendiscussingthestepstochangeaprogramorpolicyThisisathemethroughoutthisRoadmapThefirstsectionbeginswithidentifyingtherelevantpeopleTheconversationsaroundevidencearefocusedonhumanbehaviorratherthanmethodologyorstrengthofevidence
Thankyou
59
OncewehadadraftoftheRoadmapwefollowedupwithcitiesonanindividualbasistolearnmoreaboutwhatwouldbehelpfulWearegratefultothefollowingcitystafffortheinterviewsandindividualconversations
MichaelBaskinChiefPolicyOfficerChattanoogaTN
MelissaBridgesPerformanceandInnovationCoordinatorLittleRockAR
VictoriaCarreoacutenAdministrativeOfficerLasVegasNV
TyroneGrandstrandCommunityDevelopmentPlannerFargoND
TibLaughlinDirectorofGeneralServicesKansasCityKS
TimMorelandDirectorofPerformanceManagementandOpenDataChattanoogaTN
CharlesRosendahlManagementAnalystDentonTX
Thankyou
60
MelissaSiebenAssistantCountyAdministratorKansasCityKS
GinaThompsonDirectorofStrategicServicesLewisvilleTX
JHUFacultycollaboration
GovExalsoconvenedthecitycontactsandfacultypartnersinaseriesofAdvisoryGroupmeetingsThesemeetingsservedtoconnectcitiesandfacultytounderstandoneanothersperspective
StefanBaralBloombergSchoolofPublicHealth
AshleyBernerJHUSchoolofEducation
MelissaDavey-RothwellBloombergSchoolofPublicHealth
JonLinksBloombergSchoolofPublicHealth
TamarMendelsonBloombergSchoolofPublicHealth
KristinMmariBloombergSchoolofPublicHealth
KeshiaPollackPorterBloombergSchoolofPublicHealth
JoshSharfsteinBloombergSchoolofPublicHealth
LizStuartBloombergSchoolofPublicHealth
LindsayThompsonCareyBusinessSchool
DanielWebsterBloombergSchoolofPublicHealth
Partnersindataandevidence-basedpolicy
Wealsospokewithpartnerswhoareexpertsinthefieldofevidence-basedpolicywhosharedtheirknowledgeandresourcesofcurrentandfuturethinkinginevidence-informedthinking
WendyBarrenoAssociateDirectorWilsonSheehanLabforEconomicOpportunitiesUniversityofNotreDame
TiffanyDavenportDirectorofDataPartnershipsBaltimoresPromise
TomiHiersDirectorBaltimoreCivicSiteAnnieECaseyFoundation
ChrisKingsleySeniorAssociateAnnieECaseyFoundation
KathrynOliverAssociateProfessorLondonSchoolofHygieneandTropicalMedicine
Thankyou
61
KristenWatkinsSeniorPolicyAssociateJ-PAL
GovExParticipants
GovExAnalystMiriamMcKinneyanalyzedtheinterviewtranscriptsandconductedtextanalysisSeniorImplementationAdvisorEmilyShawwillintegrateRoadmapactivitieswithadditionalservicedeliveryactivitiesatGovEx
Thankyou
62
ReferencesAbrahamKGampHaskinsR(2017)Thepromiseofevidence-basedpolicymakingReportofthecommissiononevidence-basedpolicymakingWashingtonDCCommissiononEvidence-basedPolicymakingBigmove2Extendingthegreen(nd)Retrievedfromhttpswwwcityoflewisvillecomdoing-businesslewisville-2025big-move-2-extending-the-greenBatesMAampGlennersterR(2017)ThegeneralizabilitypuzzleRetrievedfromhttpsssirorgarticlesentrythe_generalizability_puzzleBuildwithnotfor(2018January)RetrievedfromhttpsbuildwithnotforwordpresscomCairneyPampOliverK(2017)Evidence-basedpolicymakingisnotlikeevidence-basedmedicinesohowfarshouldyougotobridgethedividebetweenevidenceandpolicyHealthResearchPolicyandSystemsdoi101186s12961-017-0192-xCairneyPOliverKampWellsteadA(2016)TobridgethedividebetweenevidenceandpolicyReduceambiguityasmuchasuncertaintyPublicAdministrationReview76(3)399-402Retrievedfromhttpdxdoiorg101111puar12555Centreforpublicimpact(nd)RetrievedfromhttpswwwcentreforpublicimpactorgCernaL(2013)ThenatureofpolicychangeandimplementationAreviewofdifferenttheoreticalapproachesOrganisationforEconomicCo-OperationandDevelopmentRetrievedfromhttpswwwoecdorgeduceriTheNatureofPolicyChangeandImplementationpdfColoradoSprings(nd)Retrievedfromhttpscoloradospringsgovcommunicationspagewhat-expect-when-you-speakupContinuumofcare(coc)program(nd)RetrievedfromhttpswwwhudexchangeinfoprogramscocCouncilpresidentkicksoff$12millionyouthfund(nd)Retrievedfromhttpwwwbaltimorecitycouncilcomcontentcouncil-president-kicks-12-million-youth-fundDurlakJAampDuPreEP(2008)ImplementationmattersAreviewofresearchontheinfluenceofimplementationonprogramoutcomesandthefactorsaffectingimplementationAmericanJournalofCommunityPsychology41327-350Retrievedfromhttpswwwncbinlmnihgovpubmed18322790Fishbonediagram(ishikawa)(nd)Retrievedfromhttpasqorglearn-about-qualitycause-analysis-toolsoverviewfishbonehtmlFivewhysGettingtotherootofaproblemquickly(nd)RetrievedfromhttpswwwmindtoolscompagesarticlenewTMC_5Whtm
References
63
FligsteinNampMcAdamD(2011)TowardageneraltheoryofstrategicactionfieldsSociologicalTheory29(1)1-26RetrievedfromhttpsociologyberkeleyedusitesdefaultfilesfacultyfligsteinFligstein20McAdam20Soc20Theory20PaperpdHeinkelP(2018January12)CitymullspermitsforbikesharingcompaniesRetrievedfromhttpwwwdentonrccomnewsdenton20180112city-mulls-permits-bike-sharing-companiesHowisareamedianincomecalculated(2012October)Retrievedfromhttpswwwhudexchangeinfofaqs843how-is-area-median-income-calculatedLampkinLMampHatryHP(2003)KeystepsinoutcomemanagementUrbanInstituteRetrievedfromhttpswwwurbanorgresearchpublicationkey-steps-outcome-managementMarshDampSmithM(2000)UnderstandingpolicynetworksTowardsadialecticalapproachPoliticalStudies484-21Retrievedfromhttpjournalssagepubcomdoiabs1011111467-924800247McCannL(nd)RetrievedfromhttplaurenellencomMoultonSampSandfortJR(2016)ThestrategicactionfieldframeworkforpolicyimplementationresearchPolicyStudiesJournal45(1)Retrievedfromhttponlinelibrarywileycomstore101111psj12147assetpsj12147pdfv=1ampt=jdkhhcp4amps=e184c625e1855a6b1615ccb99ee27b1c0b3212d8PaynterB(2018January1)What100citiesarelearningfromeachotherbysharingtheirdataRetrievedfromhttpswwwfastcompanycom40518004what-100-cities-are-learning-from-each-other-by-sharing-their-dataPeakacademy(nd)Retrievedfromhttpswwwdenvergovorgcontentdenvergovenmayors-officeprograms-initiativespeak-performancepeak-academyhtmlPinnockHBarwickMCarpenterCREldridgeSGrandesGGriffithsCJRycroft-MaloneJMeissnerPMurrayEPatelASheikhAampTaylorSJC(2017)Standardsforreportingimplementationstudies(stari)ExplanationandelaborationdocumentBMJOpenRetrievedfromhttpbmjopenbmjcomcontentbmjopen74e013318fullpdfProctorESilmereHRaghavanRHovmandPAaronsGBungerAGriffeyRampHensleyM(2011)OutcomesforimplementationresearchConceptualdistinctionsmeasurementchallengesandresearchagendaAdministrationandPolicyinMentalHealth38(2)65-76doi101007s10488-010-0319-7TandonSD(2018)ProgressincommunityhealthpartnershipsResearcheducationandactionRetrievedfromhttpswwwpressjhuedujournalsprogress-community-health-partnerships-research-education-and-actionProvanKGampKenisP(2007)ModesofnetworkgovernanceStructuremanagementandeffectivenessJournalofPublicAdministrationResearchandTheory
References
64
18(2)229-252Retrievedfromhttpsacademicoupcomjpartarticle182229935895SabatierPA(1988)Anadvocacycoalitionframeworkofpolicychangeandtheroleofpolicy-orientedlearningthereinPolicySciences21(3)129-168Retrievedfromhttpwwwjstororgstable4532139SandstromAampCarlssonL(2008)TheperformanceofpolicynetworksTherelationbetweennetworkstructureandnetworkperformancePolicyStudiesJournal36497-524Retrievedfromhttponlinelibrarywileycomdoi101111j1541-0072200800281xabstractSchwartzMampWilsonE(nd)WhocanaffordtoliveinahomeAlookatdatafromthe2006americancommunitysurveyRetrievedfromhttpswwwcensusgovhousingcensuspublicationswho-can-affordpdfShearerJCAbelsonAKouyateBLavisJNampWaltG(2016)WhydopolicieschangeInstitutionsinterestsideasandnetworksinthreecasesofpolicyreformHealthPolicyandPlanning31(9)1200-1211Retrievedfromhttpsdoiorg101093heapolczw052ThelessonsofadministrativedataHigh-profilepolicy-relevantresearchpoweredbyadministrativedata(nd)Retrievedfromhttpswwwpovertyactionlaborgsitesdefaultfilesresources20161114-The-Lessons-of-Administrative-DatapdfToolsandotherneedsforqualitativeandquantitativemethods(2015February)AgencyforHealthcareResearchandQualityRetrievedfromhttpwwwahrqgovprofessionalsquality-patient-safetytalkingqualityassesstoolshtml
WeaverRCampBagchi-SenS(2013)SpatialanalysisofurbandeclineThegeographyofblightAppliedGeography4061-70
References
65
UniversityPartners
Name Title Affiliation Area(s)ofexpertise
Contactinformation
Examplesofworkingwithnon-academicdecisionmakers
WendyPBarreno
AssociateDirector
WilsonSheehanLabforEconomicOpportunitiesUniversityofNotreDame
LEOsfocusareasincludehealtheducationhousingchildwelfareeconomicopportunitycriminaljustice
wbarrenonduduorleondedu
MelissaDavey-RothwellPhD
AssociateScientist
JohnsHopkinsBloombergSchoolofPublicHealth
Health mdavey1jhuedu410-614-5854
BaltimoreCityHealthDepartmentLeadersMarylandDepartmentofHealthAddictionRecoveryServicesinDCExecutiveDirectorsofcommunityorganizationsacrosstheUSthatfocusonHIVSTIsubstanceabuseandimplementationscience
ElizabethStuart
AssociateDeanforEducationandProfessorDepartmentsofMentalHealthBiostatisticsandHealthPolicyand
JohnsHopkinsBloombergSchoolofPublicHealth
Educationsocialpolicystatisticalmethods
estuartjhuedu
Resources
66
Management
Design-ThinkingResources
EngagementGuideforCivicTech
Journeymap
PerformanceManagementResources
DenverPeakAcademy
Proxymeasuresguide
Talkingaboutperformancewithoutcomplicatingit(video)
SettingPerformanceTargetsguide
Datavisualizationguide
Performancemanagementgettingstartedguide
Exercises
Fishboneexercise
5Whys
AdministrativeData
TheLessonsofAdministrativeData(JPAL)
EvaluationResources
Quantitativeandqualitativemethods(USDepartmentofHealthandHumanServicesAgencyforHealthcareResearchandQuality)
KeyStepsinOutcomeManagement(UrbanInstitute)
RCT-YES
Sageresearchmethods
BestPracticesforMixedMethodsResearchintheHealthSciences
ACCORDChecklist
Pilotandimplementationplanning
Benchmarkingexamplesandproxymeasures
Seattleopendataplaybook
Resources
67
Datasciencescopingguide
Resources
68
AboutGovExGovExhelpsgovernmentsusedataandevidencetoimproveresidentsrsquolivesWesupportgovernmentsinbuildingtheirinternalcapacitytooperatesustainableanalyticspracticeswhichimprovecommunityoutcomesinfieldsfrompublicsafetytoeconomicdevelopmentGovExhasaproventrackrecordofworkinginconcertwithpartnerstoprovidetechnicalassistanceandtrainingatscale
SinceourlaunchinApril2015GovExhas
Trainedover2000governmentemployeesinpersonandonlineintopicsincludingdatamanagementperformancemanagementopendatacommunityengagementandanalytics
Providedtechnicalassistancetomorethan100localgovernmentsacrosstheglobeimprovingtheiruseofdatatoachieveefficienciesinservicedeliveryandbetteroutcomesforresidents
Assessedthecapacityofmorethan130governmentalorganizationstodeterminehowleadersstaffandexternalstakeholdersmanageandusedata
Localgovernmentsareincreasinglyonthefrontlinesofaddressinglargescaleproblemswithglobalimplicationssuchasclimatechangepublicsafetyandeconomicinequality
Whileevidence-basedinterventionstohelpsolvethoseproblemsareavailablelocalgovernmentstaffhavelimitedtimeandcapacitytofindandimplementresearch-basedsolutionsAsaresultcitiescontinuetoworkthewaytheyalwayshave--makingincrementalimprovementstoprogramsbutnotalwayscreatingchangeatasystem-level
ThroughresourcedevelopmentandcollaborationwithJHUfacultyGovExissupportingcityleadersinusingdataandevidencetomakedecisionsandcreatinglastingrelationshipswithresearchers
TheRoadmapwasdevelopedbyGovExaspartofitspartnershipwithBloombergPhilanthropiesWhatWorksCitiesinitiative
AboutGovEx
69
- Title page
- Introduction and purpose of the Roadmap
- Using the Roadmap
-
- Gather information about the problem
- Strategically engage key partners
- Review the evidence and collect details
- Decide on a policy to implement based on local factors
- Establish a framework to monitor progress
- Assess data accessibility quality and security
- Design implementation and evaluation
- Share your story and success
-
- Working with a range of evidence
- Thank you
- References
- Resources
- About GovEx
-
6 ImplementyourstrategyKeepinmindstrategiescanbefluidSometimessomethingmaynotturnoutasexpectedevenifduringplanningsessionsitwasldquosupposedtordquoItisokYoucananalyzewhathappenedduringadebriefsessionandonceyoursurveysandothermeasurementshavebeenfinalized
Visualizations
Withtherightvisualizationyoucanreviewandshareinformationinsightsanalysisperformanceandtrendsallofwhichisvaluabletoacross-sectionofthepublicButwhichvisualizationisrightforyouTochoosetherighttypeofchartgraphdeterminethetypeofquestionyouwanttoanswerSomecommoncharttypesarebelowandmoreinformationcanbefoundhere
Exploreadditionalanalyticaltechniques
Duringthegoal-settingphaseyouestablishedoutputoutcomeandimplementationmeasuresBasedontheresultsofyourpilotyoumayneedtoadjustyourmeasuresWiththepilotdatainhandyoumayhavesomeideasofhowfutureanalysescouldbedifferentandpotentiallyimprovethequalityofyourprogramTakesometimetobrainstormotherrelevantdatasetsthatyoudidnothavethetimetopursueearlierincludingmessydepartmentaldatathatneedsmoreattentionorsourcesofpublicdatafromthestateandfederalgovernmentnonprofitsandevenprivatecompaniesthatcouldsupplementyourunderstandingofanissueUsingmachinelearningtechniquesyoumightbeabletobetter
Shareyourstoryandsuccess
54
predictdemandforhomelessservicesusingNOAAweatherforecastspreemptwatermainbreaksusingsoilstratigraphyorinvestineconomicdevelopmentbymonitoringrealestatelistingdata
Advancedanalyticaltechniquescanalsohelpyourimplementationrunmoreeffectivelybyintegratingdata-informedtargetinganddecisiontoolsintotheprocessByautomatingroutineworkflowsandsurfacingsubtletrendsthesetoolscanhelpyouiterateinamoreprecisemannerandprovidekeystrategicguidancetofront-lineemployeesthatwouldotherwisebeunavailableThesesortsofdata-science-enhancedprocessescanbeclassifiedasoneofsixprojecttypesasexploredfurtherbytheNewOrleansLANOLAlyticsteamFormoreguidanceonscopingasimilarprojecttheGovExadvancedanalyticsscopingguidegivesyouastep-by-stepoverviewofhowtobesuccessful
Usethespacebelowtolistyourideasforpossibleadvancedanalyticsprojectsbytype
Typeofproject Ideas
Findingtheneedleinthehaystack
Prioritizingforimpact
Earlywarningtools
Betterquickerdecisions
Optimizingresourceallocation
Experimentingforwhatworks
Keepthefeedbackloopgoing
GatherinformationonpublicsentimentaboutpolicychangethroughfocusgroupssurveysandpublicmeetingsledbyadvisorygroupmembersNotewhethernewinformationisavailablethatcanhelpanalyzeorcontextualizeyourprogress(egfederalorstatedatareports)Sharetheresultsofyourpilotandthepolicychangeprocesswithothercitiesandresearchers
Shareyourstoryandsuccess
55
WorkingwitharangeofevidencePresentlythereisalotofattentionandinvestmentinevidence-basedpolicymakingincludingthefederalbipartisanCommissiononEvidence-BasedPolicymakingthePewCharitableTrustsreportsandtoolkitsonHowStatesEngageinEvidence-BasedPolicymakingandtheLauraandJohnArnoldFoundationsupportedEvidence-BasedPolicymakingCollaborativeManyoftheseinitiativesfocusonrigorousevaluation--meaningrandomizedcontrolledtrials(RCT)orotherexperimentalmodels--asthebestkindofevidencetouseHoweverthisbodyofevidenceisofteninsufficientforthespecifickindsofquestionsthatgovernmentscanhavewhenmakingadecisionaboutpolicyInthisguideweproposeanapproachtosupportgovernmentsinincorporatingadditionaltypesofevidenceintotheirdecisionmaking
OnewaytounderstandarangeofevidenceistouseNestarsquosStandardsofEvidencetoframeanddefineevidenceTheBritishnonprofitNestaworkstoinjectpositivechangeintothewayUKgovernmentpoliciesandprogramsfunctionOneoftheirkeyareasofworkhasbeendevelopingWhatWorksCentresorganizationsthatcollectcommissionsynthesizeanddisseminateevidenceaboutinterventioneffectivenessacrossarangeofsocialimpactareasNestadevelopedthefollowingStandardsofEvidencetoprovideausefulwaytodocumenthowdifferentkindsofinformationdemonstratedifferentlevelsofevidence
Workingwitharangeofevidence
56
Theimportanceofpotentiallybiasedinformation
Onechallengepresentedbyanevidenceclassificationthatacknowledgesthevalueofnon-scientificformsofevidenceistofigureouthowtobalanceallofthepiecestogetherwhenyouusethemTounderstandhowtodothiswehavetothinkabouthowtopositionpotentiallybiasedinformationindecisionmaking
WhenwelookatinformationwhichhasnotbeenrigorouslyevaluatedweacceptthatthisinformationmightbebiasedThewholereasonweconductrigorousevaluationistoremovebiasfromourobservationsWeconductstructuredtrialstoproduceresultsthatareconsistentandgeneralizableIfafindingisunbiasedthisaccuracymdashthislackofbiasmdashcanbeprovedbecauseanyindependenttestercanreproduceitwhentheyfollowanidenticalprocess
YoucanlookatpotentiallybiasedinformationasjustinformationthatisearlieroninitspathtorefinementreplicationandverifiedaccuracyApieceofinformationbasedonanyoneindividualrsquosobservationsislikelytobebiasedbecauseitiscomingfromjustonesourceBiasisreducedbygatheringtogetherlotsofindependentsourcesofinformationandfiguringoutitsmostcommonattributes
MaybethebiasedinformationmakesaclaimthatistoolargemdashitsayssomethingistrueunderallconditionswhenitisreallyonlytrueundercertainconditionsorthataneffectofaninterventionisbiggerthanitstrueaverageorthataneffectlastsovertimewhenitdoesnotPerhapschangingtheclaimtobemorespecificmdashcriticallyreviewingtheinformationtoseeifspecificconditionsmustbemettoachievetheoutcomesmdashwouldmaketheclaimmoreaccurate
IfyoutestedyourclaimfounditdidnotholdupinsomecaseschangedittesteditagainuntilyourclaimrepeatedlyheldupthenyoucoulddemonstratethattheclaimwasnolongerbiasedbutaccurateThisprocessoftestingimprovingtestingandvalidatingiswhatwegetfromstructuredscientificstudy
Withindividualpiecesofinformation--crediblehypothesescasestudiesexpertopinion--youcannotassumethatkindofaccuracyAtthesametimetheseearlyobservationsarefoundationaltoanyreplicablescientificobservationsWewillnevergetanywhereifwedonotstartwiththesepiecesofinformationeveniftheymightbebiased
Combininglevelsofevidencetogetthemostdetailedinformationforyourcity
ItisespeciallyimportanttoacceptdifferentformsofevidencewhenyoufacequestionsthathavenotyetbeentestedbyindependentstudiesSomeofthemostimportantquestionsrelatetothingsthatareveryspecifictoyoursituationYoufaceaparticularcombinationof
Workingwitharangeofevidence
57
issuesactorstrendsandresourcesthatisultimatelydifferentfromeveryotherplaceintheworld
ThisisimportantbecauseaspolicypractitionersandnotscientistsyoucaremoreaboutyourspecificcasethanahypotheticalaveragegeneralcaseYoucarelessaboutthetotallikelyrangeofimpactsfromapolicythanhowthissingleimplementationwillworkYouwantinformationaboutpoliciesthatwillprovideyouwiththebestpossiblepredictionabouthowtheywillworkinyourcontext
ByputtingdifferentkindsofevidenceintothesameframeworkNestarsquosStandardsofEvidenceimplythatwhenyoucannotgetrigorouslytestedexperimentallyverifiedinformationitisappropriatetoworkwithwhatwedohaveThisisrelevantforthemunicipalimplementationcontextbecausenostudypulledfromaliteraturereviewwillaccountforallofthevariablesyoufaceinyourowncontext
Intheabsenceofhavingastudythatpredictshowaninterventionworksinyourspecificcontextyouwillneednon-scientificinformationtotrytofillinthosegapsYouwillneedtogatherstoriesfromresidentsstaffmembersandotherstakeholdersthatsitonLevels1or2ofthestandardsofevidence
Someofthekindsofinformationyouwillprobablygetmoreeasilyfrompeoplethanstudiesinclude
howpeopleincitieslikeyoursreactedtoaninterventioninpublicforumshowpeoplewhoimplementedaninterventioninacitylikeyoursfiguredouthowtogetitpassedhowthecitycommunicatedpubliclyabouttheapparentbenefitsanddrawbacksoftheinterventionthepoliticalacceptabilityoftheintervention
Youcanremainsimultaneouslyawarethattheinformationyouaregatheringthroughindividualstoriesisbiased--notgeneralizableinformedbyspecificperspectives--whilealsorecognizingthathavingsomeinformationaboutsomethingimportantisbetterthannoinformationEvidencefromthefirsttwoevidencelevelsareessentialbuildingblocksofknowledgeandperfectlyusefulwhenyoucannotgetdetailedunbiasedknowledge
Youhavetostartsomewhereandyoushouldalwaysstartwithevidenceyouhaveaboutyourveryspecificconditionsthatyouknowareimportantforsuccess
Workingwitharangeofevidence
58
Thankyou
Interviewswithcities
GovExheldsemi-structuredinterviewswithsixstaffmembersinfourcitiestoestablishtheinitialstepsoftheRoadmapWeaskedrespondentstotalkaboutaninitiativeorpolicythattheyhavebeenresponsibleforimplementingintheircityRespondentsfromthreecitiesspokeabouttheireffortstoaddresshomelessnessandrespondentsfromonecityspokeaboutblightAfterrespondentsidentifiedtheproblemtheyspokeabouthowthecityprioritizedthisissuethevariousstepsinvolvedandbarrierspresent
GovExconductedtextanalysisoftheseinterviewswhichrevealedthatpeoplewasthemostcommonlyusedtermduringtheseconversationswhendiscussingthestepstochangeaprogramorpolicyThisisathemethroughoutthisRoadmapThefirstsectionbeginswithidentifyingtherelevantpeopleTheconversationsaroundevidencearefocusedonhumanbehaviorratherthanmethodologyorstrengthofevidence
Thankyou
59
OncewehadadraftoftheRoadmapwefollowedupwithcitiesonanindividualbasistolearnmoreaboutwhatwouldbehelpfulWearegratefultothefollowingcitystafffortheinterviewsandindividualconversations
MichaelBaskinChiefPolicyOfficerChattanoogaTN
MelissaBridgesPerformanceandInnovationCoordinatorLittleRockAR
VictoriaCarreoacutenAdministrativeOfficerLasVegasNV
TyroneGrandstrandCommunityDevelopmentPlannerFargoND
TibLaughlinDirectorofGeneralServicesKansasCityKS
TimMorelandDirectorofPerformanceManagementandOpenDataChattanoogaTN
CharlesRosendahlManagementAnalystDentonTX
Thankyou
60
MelissaSiebenAssistantCountyAdministratorKansasCityKS
GinaThompsonDirectorofStrategicServicesLewisvilleTX
JHUFacultycollaboration
GovExalsoconvenedthecitycontactsandfacultypartnersinaseriesofAdvisoryGroupmeetingsThesemeetingsservedtoconnectcitiesandfacultytounderstandoneanothersperspective
StefanBaralBloombergSchoolofPublicHealth
AshleyBernerJHUSchoolofEducation
MelissaDavey-RothwellBloombergSchoolofPublicHealth
JonLinksBloombergSchoolofPublicHealth
TamarMendelsonBloombergSchoolofPublicHealth
KristinMmariBloombergSchoolofPublicHealth
KeshiaPollackPorterBloombergSchoolofPublicHealth
JoshSharfsteinBloombergSchoolofPublicHealth
LizStuartBloombergSchoolofPublicHealth
LindsayThompsonCareyBusinessSchool
DanielWebsterBloombergSchoolofPublicHealth
Partnersindataandevidence-basedpolicy
Wealsospokewithpartnerswhoareexpertsinthefieldofevidence-basedpolicywhosharedtheirknowledgeandresourcesofcurrentandfuturethinkinginevidence-informedthinking
WendyBarrenoAssociateDirectorWilsonSheehanLabforEconomicOpportunitiesUniversityofNotreDame
TiffanyDavenportDirectorofDataPartnershipsBaltimoresPromise
TomiHiersDirectorBaltimoreCivicSiteAnnieECaseyFoundation
ChrisKingsleySeniorAssociateAnnieECaseyFoundation
KathrynOliverAssociateProfessorLondonSchoolofHygieneandTropicalMedicine
Thankyou
61
KristenWatkinsSeniorPolicyAssociateJ-PAL
GovExParticipants
GovExAnalystMiriamMcKinneyanalyzedtheinterviewtranscriptsandconductedtextanalysisSeniorImplementationAdvisorEmilyShawwillintegrateRoadmapactivitieswithadditionalservicedeliveryactivitiesatGovEx
Thankyou
62
ReferencesAbrahamKGampHaskinsR(2017)Thepromiseofevidence-basedpolicymakingReportofthecommissiononevidence-basedpolicymakingWashingtonDCCommissiononEvidence-basedPolicymakingBigmove2Extendingthegreen(nd)Retrievedfromhttpswwwcityoflewisvillecomdoing-businesslewisville-2025big-move-2-extending-the-greenBatesMAampGlennersterR(2017)ThegeneralizabilitypuzzleRetrievedfromhttpsssirorgarticlesentrythe_generalizability_puzzleBuildwithnotfor(2018January)RetrievedfromhttpsbuildwithnotforwordpresscomCairneyPampOliverK(2017)Evidence-basedpolicymakingisnotlikeevidence-basedmedicinesohowfarshouldyougotobridgethedividebetweenevidenceandpolicyHealthResearchPolicyandSystemsdoi101186s12961-017-0192-xCairneyPOliverKampWellsteadA(2016)TobridgethedividebetweenevidenceandpolicyReduceambiguityasmuchasuncertaintyPublicAdministrationReview76(3)399-402Retrievedfromhttpdxdoiorg101111puar12555Centreforpublicimpact(nd)RetrievedfromhttpswwwcentreforpublicimpactorgCernaL(2013)ThenatureofpolicychangeandimplementationAreviewofdifferenttheoreticalapproachesOrganisationforEconomicCo-OperationandDevelopmentRetrievedfromhttpswwwoecdorgeduceriTheNatureofPolicyChangeandImplementationpdfColoradoSprings(nd)Retrievedfromhttpscoloradospringsgovcommunicationspagewhat-expect-when-you-speakupContinuumofcare(coc)program(nd)RetrievedfromhttpswwwhudexchangeinfoprogramscocCouncilpresidentkicksoff$12millionyouthfund(nd)Retrievedfromhttpwwwbaltimorecitycouncilcomcontentcouncil-president-kicks-12-million-youth-fundDurlakJAampDuPreEP(2008)ImplementationmattersAreviewofresearchontheinfluenceofimplementationonprogramoutcomesandthefactorsaffectingimplementationAmericanJournalofCommunityPsychology41327-350Retrievedfromhttpswwwncbinlmnihgovpubmed18322790Fishbonediagram(ishikawa)(nd)Retrievedfromhttpasqorglearn-about-qualitycause-analysis-toolsoverviewfishbonehtmlFivewhysGettingtotherootofaproblemquickly(nd)RetrievedfromhttpswwwmindtoolscompagesarticlenewTMC_5Whtm
References
63
FligsteinNampMcAdamD(2011)TowardageneraltheoryofstrategicactionfieldsSociologicalTheory29(1)1-26RetrievedfromhttpsociologyberkeleyedusitesdefaultfilesfacultyfligsteinFligstein20McAdam20Soc20Theory20PaperpdHeinkelP(2018January12)CitymullspermitsforbikesharingcompaniesRetrievedfromhttpwwwdentonrccomnewsdenton20180112city-mulls-permits-bike-sharing-companiesHowisareamedianincomecalculated(2012October)Retrievedfromhttpswwwhudexchangeinfofaqs843how-is-area-median-income-calculatedLampkinLMampHatryHP(2003)KeystepsinoutcomemanagementUrbanInstituteRetrievedfromhttpswwwurbanorgresearchpublicationkey-steps-outcome-managementMarshDampSmithM(2000)UnderstandingpolicynetworksTowardsadialecticalapproachPoliticalStudies484-21Retrievedfromhttpjournalssagepubcomdoiabs1011111467-924800247McCannL(nd)RetrievedfromhttplaurenellencomMoultonSampSandfortJR(2016)ThestrategicactionfieldframeworkforpolicyimplementationresearchPolicyStudiesJournal45(1)Retrievedfromhttponlinelibrarywileycomstore101111psj12147assetpsj12147pdfv=1ampt=jdkhhcp4amps=e184c625e1855a6b1615ccb99ee27b1c0b3212d8PaynterB(2018January1)What100citiesarelearningfromeachotherbysharingtheirdataRetrievedfromhttpswwwfastcompanycom40518004what-100-cities-are-learning-from-each-other-by-sharing-their-dataPeakacademy(nd)Retrievedfromhttpswwwdenvergovorgcontentdenvergovenmayors-officeprograms-initiativespeak-performancepeak-academyhtmlPinnockHBarwickMCarpenterCREldridgeSGrandesGGriffithsCJRycroft-MaloneJMeissnerPMurrayEPatelASheikhAampTaylorSJC(2017)Standardsforreportingimplementationstudies(stari)ExplanationandelaborationdocumentBMJOpenRetrievedfromhttpbmjopenbmjcomcontentbmjopen74e013318fullpdfProctorESilmereHRaghavanRHovmandPAaronsGBungerAGriffeyRampHensleyM(2011)OutcomesforimplementationresearchConceptualdistinctionsmeasurementchallengesandresearchagendaAdministrationandPolicyinMentalHealth38(2)65-76doi101007s10488-010-0319-7TandonSD(2018)ProgressincommunityhealthpartnershipsResearcheducationandactionRetrievedfromhttpswwwpressjhuedujournalsprogress-community-health-partnerships-research-education-and-actionProvanKGampKenisP(2007)ModesofnetworkgovernanceStructuremanagementandeffectivenessJournalofPublicAdministrationResearchandTheory
References
64
18(2)229-252Retrievedfromhttpsacademicoupcomjpartarticle182229935895SabatierPA(1988)Anadvocacycoalitionframeworkofpolicychangeandtheroleofpolicy-orientedlearningthereinPolicySciences21(3)129-168Retrievedfromhttpwwwjstororgstable4532139SandstromAampCarlssonL(2008)TheperformanceofpolicynetworksTherelationbetweennetworkstructureandnetworkperformancePolicyStudiesJournal36497-524Retrievedfromhttponlinelibrarywileycomdoi101111j1541-0072200800281xabstractSchwartzMampWilsonE(nd)WhocanaffordtoliveinahomeAlookatdatafromthe2006americancommunitysurveyRetrievedfromhttpswwwcensusgovhousingcensuspublicationswho-can-affordpdfShearerJCAbelsonAKouyateBLavisJNampWaltG(2016)WhydopolicieschangeInstitutionsinterestsideasandnetworksinthreecasesofpolicyreformHealthPolicyandPlanning31(9)1200-1211Retrievedfromhttpsdoiorg101093heapolczw052ThelessonsofadministrativedataHigh-profilepolicy-relevantresearchpoweredbyadministrativedata(nd)Retrievedfromhttpswwwpovertyactionlaborgsitesdefaultfilesresources20161114-The-Lessons-of-Administrative-DatapdfToolsandotherneedsforqualitativeandquantitativemethods(2015February)AgencyforHealthcareResearchandQualityRetrievedfromhttpwwwahrqgovprofessionalsquality-patient-safetytalkingqualityassesstoolshtml
WeaverRCampBagchi-SenS(2013)SpatialanalysisofurbandeclineThegeographyofblightAppliedGeography4061-70
References
65
UniversityPartners
Name Title Affiliation Area(s)ofexpertise
Contactinformation
Examplesofworkingwithnon-academicdecisionmakers
WendyPBarreno
AssociateDirector
WilsonSheehanLabforEconomicOpportunitiesUniversityofNotreDame
LEOsfocusareasincludehealtheducationhousingchildwelfareeconomicopportunitycriminaljustice
wbarrenonduduorleondedu
MelissaDavey-RothwellPhD
AssociateScientist
JohnsHopkinsBloombergSchoolofPublicHealth
Health mdavey1jhuedu410-614-5854
BaltimoreCityHealthDepartmentLeadersMarylandDepartmentofHealthAddictionRecoveryServicesinDCExecutiveDirectorsofcommunityorganizationsacrosstheUSthatfocusonHIVSTIsubstanceabuseandimplementationscience
ElizabethStuart
AssociateDeanforEducationandProfessorDepartmentsofMentalHealthBiostatisticsandHealthPolicyand
JohnsHopkinsBloombergSchoolofPublicHealth
Educationsocialpolicystatisticalmethods
estuartjhuedu
Resources
66
Management
Design-ThinkingResources
EngagementGuideforCivicTech
Journeymap
PerformanceManagementResources
DenverPeakAcademy
Proxymeasuresguide
Talkingaboutperformancewithoutcomplicatingit(video)
SettingPerformanceTargetsguide
Datavisualizationguide
Performancemanagementgettingstartedguide
Exercises
Fishboneexercise
5Whys
AdministrativeData
TheLessonsofAdministrativeData(JPAL)
EvaluationResources
Quantitativeandqualitativemethods(USDepartmentofHealthandHumanServicesAgencyforHealthcareResearchandQuality)
KeyStepsinOutcomeManagement(UrbanInstitute)
RCT-YES
Sageresearchmethods
BestPracticesforMixedMethodsResearchintheHealthSciences
ACCORDChecklist
Pilotandimplementationplanning
Benchmarkingexamplesandproxymeasures
Seattleopendataplaybook
Resources
67
Datasciencescopingguide
Resources
68
AboutGovExGovExhelpsgovernmentsusedataandevidencetoimproveresidentsrsquolivesWesupportgovernmentsinbuildingtheirinternalcapacitytooperatesustainableanalyticspracticeswhichimprovecommunityoutcomesinfieldsfrompublicsafetytoeconomicdevelopmentGovExhasaproventrackrecordofworkinginconcertwithpartnerstoprovidetechnicalassistanceandtrainingatscale
SinceourlaunchinApril2015GovExhas
Trainedover2000governmentemployeesinpersonandonlineintopicsincludingdatamanagementperformancemanagementopendatacommunityengagementandanalytics
Providedtechnicalassistancetomorethan100localgovernmentsacrosstheglobeimprovingtheiruseofdatatoachieveefficienciesinservicedeliveryandbetteroutcomesforresidents
Assessedthecapacityofmorethan130governmentalorganizationstodeterminehowleadersstaffandexternalstakeholdersmanageandusedata
Localgovernmentsareincreasinglyonthefrontlinesofaddressinglargescaleproblemswithglobalimplicationssuchasclimatechangepublicsafetyandeconomicinequality
Whileevidence-basedinterventionstohelpsolvethoseproblemsareavailablelocalgovernmentstaffhavelimitedtimeandcapacitytofindandimplementresearch-basedsolutionsAsaresultcitiescontinuetoworkthewaytheyalwayshave--makingincrementalimprovementstoprogramsbutnotalwayscreatingchangeatasystem-level
ThroughresourcedevelopmentandcollaborationwithJHUfacultyGovExissupportingcityleadersinusingdataandevidencetomakedecisionsandcreatinglastingrelationshipswithresearchers
TheRoadmapwasdevelopedbyGovExaspartofitspartnershipwithBloombergPhilanthropiesWhatWorksCitiesinitiative
AboutGovEx
69
- Title page
- Introduction and purpose of the Roadmap
- Using the Roadmap
-
- Gather information about the problem
- Strategically engage key partners
- Review the evidence and collect details
- Decide on a policy to implement based on local factors
- Establish a framework to monitor progress
- Assess data accessibility quality and security
- Design implementation and evaluation
- Share your story and success
-
- Working with a range of evidence
- Thank you
- References
- Resources
- About GovEx
-
predictdemandforhomelessservicesusingNOAAweatherforecastspreemptwatermainbreaksusingsoilstratigraphyorinvestineconomicdevelopmentbymonitoringrealestatelistingdata
Advancedanalyticaltechniquescanalsohelpyourimplementationrunmoreeffectivelybyintegratingdata-informedtargetinganddecisiontoolsintotheprocessByautomatingroutineworkflowsandsurfacingsubtletrendsthesetoolscanhelpyouiterateinamoreprecisemannerandprovidekeystrategicguidancetofront-lineemployeesthatwouldotherwisebeunavailableThesesortsofdata-science-enhancedprocessescanbeclassifiedasoneofsixprojecttypesasexploredfurtherbytheNewOrleansLANOLAlyticsteamFormoreguidanceonscopingasimilarprojecttheGovExadvancedanalyticsscopingguidegivesyouastep-by-stepoverviewofhowtobesuccessful
Usethespacebelowtolistyourideasforpossibleadvancedanalyticsprojectsbytype
Typeofproject Ideas
Findingtheneedleinthehaystack
Prioritizingforimpact
Earlywarningtools
Betterquickerdecisions
Optimizingresourceallocation
Experimentingforwhatworks
Keepthefeedbackloopgoing
GatherinformationonpublicsentimentaboutpolicychangethroughfocusgroupssurveysandpublicmeetingsledbyadvisorygroupmembersNotewhethernewinformationisavailablethatcanhelpanalyzeorcontextualizeyourprogress(egfederalorstatedatareports)Sharetheresultsofyourpilotandthepolicychangeprocesswithothercitiesandresearchers
Shareyourstoryandsuccess
55
WorkingwitharangeofevidencePresentlythereisalotofattentionandinvestmentinevidence-basedpolicymakingincludingthefederalbipartisanCommissiononEvidence-BasedPolicymakingthePewCharitableTrustsreportsandtoolkitsonHowStatesEngageinEvidence-BasedPolicymakingandtheLauraandJohnArnoldFoundationsupportedEvidence-BasedPolicymakingCollaborativeManyoftheseinitiativesfocusonrigorousevaluation--meaningrandomizedcontrolledtrials(RCT)orotherexperimentalmodels--asthebestkindofevidencetouseHoweverthisbodyofevidenceisofteninsufficientforthespecifickindsofquestionsthatgovernmentscanhavewhenmakingadecisionaboutpolicyInthisguideweproposeanapproachtosupportgovernmentsinincorporatingadditionaltypesofevidenceintotheirdecisionmaking
OnewaytounderstandarangeofevidenceistouseNestarsquosStandardsofEvidencetoframeanddefineevidenceTheBritishnonprofitNestaworkstoinjectpositivechangeintothewayUKgovernmentpoliciesandprogramsfunctionOneoftheirkeyareasofworkhasbeendevelopingWhatWorksCentresorganizationsthatcollectcommissionsynthesizeanddisseminateevidenceaboutinterventioneffectivenessacrossarangeofsocialimpactareasNestadevelopedthefollowingStandardsofEvidencetoprovideausefulwaytodocumenthowdifferentkindsofinformationdemonstratedifferentlevelsofevidence
Workingwitharangeofevidence
56
Theimportanceofpotentiallybiasedinformation
Onechallengepresentedbyanevidenceclassificationthatacknowledgesthevalueofnon-scientificformsofevidenceistofigureouthowtobalanceallofthepiecestogetherwhenyouusethemTounderstandhowtodothiswehavetothinkabouthowtopositionpotentiallybiasedinformationindecisionmaking
WhenwelookatinformationwhichhasnotbeenrigorouslyevaluatedweacceptthatthisinformationmightbebiasedThewholereasonweconductrigorousevaluationistoremovebiasfromourobservationsWeconductstructuredtrialstoproduceresultsthatareconsistentandgeneralizableIfafindingisunbiasedthisaccuracymdashthislackofbiasmdashcanbeprovedbecauseanyindependenttestercanreproduceitwhentheyfollowanidenticalprocess
YoucanlookatpotentiallybiasedinformationasjustinformationthatisearlieroninitspathtorefinementreplicationandverifiedaccuracyApieceofinformationbasedonanyoneindividualrsquosobservationsislikelytobebiasedbecauseitiscomingfromjustonesourceBiasisreducedbygatheringtogetherlotsofindependentsourcesofinformationandfiguringoutitsmostcommonattributes
MaybethebiasedinformationmakesaclaimthatistoolargemdashitsayssomethingistrueunderallconditionswhenitisreallyonlytrueundercertainconditionsorthataneffectofaninterventionisbiggerthanitstrueaverageorthataneffectlastsovertimewhenitdoesnotPerhapschangingtheclaimtobemorespecificmdashcriticallyreviewingtheinformationtoseeifspecificconditionsmustbemettoachievetheoutcomesmdashwouldmaketheclaimmoreaccurate
IfyoutestedyourclaimfounditdidnotholdupinsomecaseschangedittesteditagainuntilyourclaimrepeatedlyheldupthenyoucoulddemonstratethattheclaimwasnolongerbiasedbutaccurateThisprocessoftestingimprovingtestingandvalidatingiswhatwegetfromstructuredscientificstudy
Withindividualpiecesofinformation--crediblehypothesescasestudiesexpertopinion--youcannotassumethatkindofaccuracyAtthesametimetheseearlyobservationsarefoundationaltoanyreplicablescientificobservationsWewillnevergetanywhereifwedonotstartwiththesepiecesofinformationeveniftheymightbebiased
Combininglevelsofevidencetogetthemostdetailedinformationforyourcity
ItisespeciallyimportanttoacceptdifferentformsofevidencewhenyoufacequestionsthathavenotyetbeentestedbyindependentstudiesSomeofthemostimportantquestionsrelatetothingsthatareveryspecifictoyoursituationYoufaceaparticularcombinationof
Workingwitharangeofevidence
57
issuesactorstrendsandresourcesthatisultimatelydifferentfromeveryotherplaceintheworld
ThisisimportantbecauseaspolicypractitionersandnotscientistsyoucaremoreaboutyourspecificcasethanahypotheticalaveragegeneralcaseYoucarelessaboutthetotallikelyrangeofimpactsfromapolicythanhowthissingleimplementationwillworkYouwantinformationaboutpoliciesthatwillprovideyouwiththebestpossiblepredictionabouthowtheywillworkinyourcontext
ByputtingdifferentkindsofevidenceintothesameframeworkNestarsquosStandardsofEvidenceimplythatwhenyoucannotgetrigorouslytestedexperimentallyverifiedinformationitisappropriatetoworkwithwhatwedohaveThisisrelevantforthemunicipalimplementationcontextbecausenostudypulledfromaliteraturereviewwillaccountforallofthevariablesyoufaceinyourowncontext
Intheabsenceofhavingastudythatpredictshowaninterventionworksinyourspecificcontextyouwillneednon-scientificinformationtotrytofillinthosegapsYouwillneedtogatherstoriesfromresidentsstaffmembersandotherstakeholdersthatsitonLevels1or2ofthestandardsofevidence
Someofthekindsofinformationyouwillprobablygetmoreeasilyfrompeoplethanstudiesinclude
howpeopleincitieslikeyoursreactedtoaninterventioninpublicforumshowpeoplewhoimplementedaninterventioninacitylikeyoursfiguredouthowtogetitpassedhowthecitycommunicatedpubliclyabouttheapparentbenefitsanddrawbacksoftheinterventionthepoliticalacceptabilityoftheintervention
Youcanremainsimultaneouslyawarethattheinformationyouaregatheringthroughindividualstoriesisbiased--notgeneralizableinformedbyspecificperspectives--whilealsorecognizingthathavingsomeinformationaboutsomethingimportantisbetterthannoinformationEvidencefromthefirsttwoevidencelevelsareessentialbuildingblocksofknowledgeandperfectlyusefulwhenyoucannotgetdetailedunbiasedknowledge
Youhavetostartsomewhereandyoushouldalwaysstartwithevidenceyouhaveaboutyourveryspecificconditionsthatyouknowareimportantforsuccess
Workingwitharangeofevidence
58
Thankyou
Interviewswithcities
GovExheldsemi-structuredinterviewswithsixstaffmembersinfourcitiestoestablishtheinitialstepsoftheRoadmapWeaskedrespondentstotalkaboutaninitiativeorpolicythattheyhavebeenresponsibleforimplementingintheircityRespondentsfromthreecitiesspokeabouttheireffortstoaddresshomelessnessandrespondentsfromonecityspokeaboutblightAfterrespondentsidentifiedtheproblemtheyspokeabouthowthecityprioritizedthisissuethevariousstepsinvolvedandbarrierspresent
GovExconductedtextanalysisoftheseinterviewswhichrevealedthatpeoplewasthemostcommonlyusedtermduringtheseconversationswhendiscussingthestepstochangeaprogramorpolicyThisisathemethroughoutthisRoadmapThefirstsectionbeginswithidentifyingtherelevantpeopleTheconversationsaroundevidencearefocusedonhumanbehaviorratherthanmethodologyorstrengthofevidence
Thankyou
59
OncewehadadraftoftheRoadmapwefollowedupwithcitiesonanindividualbasistolearnmoreaboutwhatwouldbehelpfulWearegratefultothefollowingcitystafffortheinterviewsandindividualconversations
MichaelBaskinChiefPolicyOfficerChattanoogaTN
MelissaBridgesPerformanceandInnovationCoordinatorLittleRockAR
VictoriaCarreoacutenAdministrativeOfficerLasVegasNV
TyroneGrandstrandCommunityDevelopmentPlannerFargoND
TibLaughlinDirectorofGeneralServicesKansasCityKS
TimMorelandDirectorofPerformanceManagementandOpenDataChattanoogaTN
CharlesRosendahlManagementAnalystDentonTX
Thankyou
60
MelissaSiebenAssistantCountyAdministratorKansasCityKS
GinaThompsonDirectorofStrategicServicesLewisvilleTX
JHUFacultycollaboration
GovExalsoconvenedthecitycontactsandfacultypartnersinaseriesofAdvisoryGroupmeetingsThesemeetingsservedtoconnectcitiesandfacultytounderstandoneanothersperspective
StefanBaralBloombergSchoolofPublicHealth
AshleyBernerJHUSchoolofEducation
MelissaDavey-RothwellBloombergSchoolofPublicHealth
JonLinksBloombergSchoolofPublicHealth
TamarMendelsonBloombergSchoolofPublicHealth
KristinMmariBloombergSchoolofPublicHealth
KeshiaPollackPorterBloombergSchoolofPublicHealth
JoshSharfsteinBloombergSchoolofPublicHealth
LizStuartBloombergSchoolofPublicHealth
LindsayThompsonCareyBusinessSchool
DanielWebsterBloombergSchoolofPublicHealth
Partnersindataandevidence-basedpolicy
Wealsospokewithpartnerswhoareexpertsinthefieldofevidence-basedpolicywhosharedtheirknowledgeandresourcesofcurrentandfuturethinkinginevidence-informedthinking
WendyBarrenoAssociateDirectorWilsonSheehanLabforEconomicOpportunitiesUniversityofNotreDame
TiffanyDavenportDirectorofDataPartnershipsBaltimoresPromise
TomiHiersDirectorBaltimoreCivicSiteAnnieECaseyFoundation
ChrisKingsleySeniorAssociateAnnieECaseyFoundation
KathrynOliverAssociateProfessorLondonSchoolofHygieneandTropicalMedicine
Thankyou
61
KristenWatkinsSeniorPolicyAssociateJ-PAL
GovExParticipants
GovExAnalystMiriamMcKinneyanalyzedtheinterviewtranscriptsandconductedtextanalysisSeniorImplementationAdvisorEmilyShawwillintegrateRoadmapactivitieswithadditionalservicedeliveryactivitiesatGovEx
Thankyou
62
ReferencesAbrahamKGampHaskinsR(2017)Thepromiseofevidence-basedpolicymakingReportofthecommissiononevidence-basedpolicymakingWashingtonDCCommissiononEvidence-basedPolicymakingBigmove2Extendingthegreen(nd)Retrievedfromhttpswwwcityoflewisvillecomdoing-businesslewisville-2025big-move-2-extending-the-greenBatesMAampGlennersterR(2017)ThegeneralizabilitypuzzleRetrievedfromhttpsssirorgarticlesentrythe_generalizability_puzzleBuildwithnotfor(2018January)RetrievedfromhttpsbuildwithnotforwordpresscomCairneyPampOliverK(2017)Evidence-basedpolicymakingisnotlikeevidence-basedmedicinesohowfarshouldyougotobridgethedividebetweenevidenceandpolicyHealthResearchPolicyandSystemsdoi101186s12961-017-0192-xCairneyPOliverKampWellsteadA(2016)TobridgethedividebetweenevidenceandpolicyReduceambiguityasmuchasuncertaintyPublicAdministrationReview76(3)399-402Retrievedfromhttpdxdoiorg101111puar12555Centreforpublicimpact(nd)RetrievedfromhttpswwwcentreforpublicimpactorgCernaL(2013)ThenatureofpolicychangeandimplementationAreviewofdifferenttheoreticalapproachesOrganisationforEconomicCo-OperationandDevelopmentRetrievedfromhttpswwwoecdorgeduceriTheNatureofPolicyChangeandImplementationpdfColoradoSprings(nd)Retrievedfromhttpscoloradospringsgovcommunicationspagewhat-expect-when-you-speakupContinuumofcare(coc)program(nd)RetrievedfromhttpswwwhudexchangeinfoprogramscocCouncilpresidentkicksoff$12millionyouthfund(nd)Retrievedfromhttpwwwbaltimorecitycouncilcomcontentcouncil-president-kicks-12-million-youth-fundDurlakJAampDuPreEP(2008)ImplementationmattersAreviewofresearchontheinfluenceofimplementationonprogramoutcomesandthefactorsaffectingimplementationAmericanJournalofCommunityPsychology41327-350Retrievedfromhttpswwwncbinlmnihgovpubmed18322790Fishbonediagram(ishikawa)(nd)Retrievedfromhttpasqorglearn-about-qualitycause-analysis-toolsoverviewfishbonehtmlFivewhysGettingtotherootofaproblemquickly(nd)RetrievedfromhttpswwwmindtoolscompagesarticlenewTMC_5Whtm
References
63
FligsteinNampMcAdamD(2011)TowardageneraltheoryofstrategicactionfieldsSociologicalTheory29(1)1-26RetrievedfromhttpsociologyberkeleyedusitesdefaultfilesfacultyfligsteinFligstein20McAdam20Soc20Theory20PaperpdHeinkelP(2018January12)CitymullspermitsforbikesharingcompaniesRetrievedfromhttpwwwdentonrccomnewsdenton20180112city-mulls-permits-bike-sharing-companiesHowisareamedianincomecalculated(2012October)Retrievedfromhttpswwwhudexchangeinfofaqs843how-is-area-median-income-calculatedLampkinLMampHatryHP(2003)KeystepsinoutcomemanagementUrbanInstituteRetrievedfromhttpswwwurbanorgresearchpublicationkey-steps-outcome-managementMarshDampSmithM(2000)UnderstandingpolicynetworksTowardsadialecticalapproachPoliticalStudies484-21Retrievedfromhttpjournalssagepubcomdoiabs1011111467-924800247McCannL(nd)RetrievedfromhttplaurenellencomMoultonSampSandfortJR(2016)ThestrategicactionfieldframeworkforpolicyimplementationresearchPolicyStudiesJournal45(1)Retrievedfromhttponlinelibrarywileycomstore101111psj12147assetpsj12147pdfv=1ampt=jdkhhcp4amps=e184c625e1855a6b1615ccb99ee27b1c0b3212d8PaynterB(2018January1)What100citiesarelearningfromeachotherbysharingtheirdataRetrievedfromhttpswwwfastcompanycom40518004what-100-cities-are-learning-from-each-other-by-sharing-their-dataPeakacademy(nd)Retrievedfromhttpswwwdenvergovorgcontentdenvergovenmayors-officeprograms-initiativespeak-performancepeak-academyhtmlPinnockHBarwickMCarpenterCREldridgeSGrandesGGriffithsCJRycroft-MaloneJMeissnerPMurrayEPatelASheikhAampTaylorSJC(2017)Standardsforreportingimplementationstudies(stari)ExplanationandelaborationdocumentBMJOpenRetrievedfromhttpbmjopenbmjcomcontentbmjopen74e013318fullpdfProctorESilmereHRaghavanRHovmandPAaronsGBungerAGriffeyRampHensleyM(2011)OutcomesforimplementationresearchConceptualdistinctionsmeasurementchallengesandresearchagendaAdministrationandPolicyinMentalHealth38(2)65-76doi101007s10488-010-0319-7TandonSD(2018)ProgressincommunityhealthpartnershipsResearcheducationandactionRetrievedfromhttpswwwpressjhuedujournalsprogress-community-health-partnerships-research-education-and-actionProvanKGampKenisP(2007)ModesofnetworkgovernanceStructuremanagementandeffectivenessJournalofPublicAdministrationResearchandTheory
References
64
18(2)229-252Retrievedfromhttpsacademicoupcomjpartarticle182229935895SabatierPA(1988)Anadvocacycoalitionframeworkofpolicychangeandtheroleofpolicy-orientedlearningthereinPolicySciences21(3)129-168Retrievedfromhttpwwwjstororgstable4532139SandstromAampCarlssonL(2008)TheperformanceofpolicynetworksTherelationbetweennetworkstructureandnetworkperformancePolicyStudiesJournal36497-524Retrievedfromhttponlinelibrarywileycomdoi101111j1541-0072200800281xabstractSchwartzMampWilsonE(nd)WhocanaffordtoliveinahomeAlookatdatafromthe2006americancommunitysurveyRetrievedfromhttpswwwcensusgovhousingcensuspublicationswho-can-affordpdfShearerJCAbelsonAKouyateBLavisJNampWaltG(2016)WhydopolicieschangeInstitutionsinterestsideasandnetworksinthreecasesofpolicyreformHealthPolicyandPlanning31(9)1200-1211Retrievedfromhttpsdoiorg101093heapolczw052ThelessonsofadministrativedataHigh-profilepolicy-relevantresearchpoweredbyadministrativedata(nd)Retrievedfromhttpswwwpovertyactionlaborgsitesdefaultfilesresources20161114-The-Lessons-of-Administrative-DatapdfToolsandotherneedsforqualitativeandquantitativemethods(2015February)AgencyforHealthcareResearchandQualityRetrievedfromhttpwwwahrqgovprofessionalsquality-patient-safetytalkingqualityassesstoolshtml
WeaverRCampBagchi-SenS(2013)SpatialanalysisofurbandeclineThegeographyofblightAppliedGeography4061-70
References
65
UniversityPartners
Name Title Affiliation Area(s)ofexpertise
Contactinformation
Examplesofworkingwithnon-academicdecisionmakers
WendyPBarreno
AssociateDirector
WilsonSheehanLabforEconomicOpportunitiesUniversityofNotreDame
LEOsfocusareasincludehealtheducationhousingchildwelfareeconomicopportunitycriminaljustice
wbarrenonduduorleondedu
MelissaDavey-RothwellPhD
AssociateScientist
JohnsHopkinsBloombergSchoolofPublicHealth
Health mdavey1jhuedu410-614-5854
BaltimoreCityHealthDepartmentLeadersMarylandDepartmentofHealthAddictionRecoveryServicesinDCExecutiveDirectorsofcommunityorganizationsacrosstheUSthatfocusonHIVSTIsubstanceabuseandimplementationscience
ElizabethStuart
AssociateDeanforEducationandProfessorDepartmentsofMentalHealthBiostatisticsandHealthPolicyand
JohnsHopkinsBloombergSchoolofPublicHealth
Educationsocialpolicystatisticalmethods
estuartjhuedu
Resources
66
Management
Design-ThinkingResources
EngagementGuideforCivicTech
Journeymap
PerformanceManagementResources
DenverPeakAcademy
Proxymeasuresguide
Talkingaboutperformancewithoutcomplicatingit(video)
SettingPerformanceTargetsguide
Datavisualizationguide
Performancemanagementgettingstartedguide
Exercises
Fishboneexercise
5Whys
AdministrativeData
TheLessonsofAdministrativeData(JPAL)
EvaluationResources
Quantitativeandqualitativemethods(USDepartmentofHealthandHumanServicesAgencyforHealthcareResearchandQuality)
KeyStepsinOutcomeManagement(UrbanInstitute)
RCT-YES
Sageresearchmethods
BestPracticesforMixedMethodsResearchintheHealthSciences
ACCORDChecklist
Pilotandimplementationplanning
Benchmarkingexamplesandproxymeasures
Seattleopendataplaybook
Resources
67
Datasciencescopingguide
Resources
68
AboutGovExGovExhelpsgovernmentsusedataandevidencetoimproveresidentsrsquolivesWesupportgovernmentsinbuildingtheirinternalcapacitytooperatesustainableanalyticspracticeswhichimprovecommunityoutcomesinfieldsfrompublicsafetytoeconomicdevelopmentGovExhasaproventrackrecordofworkinginconcertwithpartnerstoprovidetechnicalassistanceandtrainingatscale
SinceourlaunchinApril2015GovExhas
Trainedover2000governmentemployeesinpersonandonlineintopicsincludingdatamanagementperformancemanagementopendatacommunityengagementandanalytics
Providedtechnicalassistancetomorethan100localgovernmentsacrosstheglobeimprovingtheiruseofdatatoachieveefficienciesinservicedeliveryandbetteroutcomesforresidents
Assessedthecapacityofmorethan130governmentalorganizationstodeterminehowleadersstaffandexternalstakeholdersmanageandusedata
Localgovernmentsareincreasinglyonthefrontlinesofaddressinglargescaleproblemswithglobalimplicationssuchasclimatechangepublicsafetyandeconomicinequality
Whileevidence-basedinterventionstohelpsolvethoseproblemsareavailablelocalgovernmentstaffhavelimitedtimeandcapacitytofindandimplementresearch-basedsolutionsAsaresultcitiescontinuetoworkthewaytheyalwayshave--makingincrementalimprovementstoprogramsbutnotalwayscreatingchangeatasystem-level
ThroughresourcedevelopmentandcollaborationwithJHUfacultyGovExissupportingcityleadersinusingdataandevidencetomakedecisionsandcreatinglastingrelationshipswithresearchers
TheRoadmapwasdevelopedbyGovExaspartofitspartnershipwithBloombergPhilanthropiesWhatWorksCitiesinitiative
AboutGovEx
69
- Title page
- Introduction and purpose of the Roadmap
- Using the Roadmap
-
- Gather information about the problem
- Strategically engage key partners
- Review the evidence and collect details
- Decide on a policy to implement based on local factors
- Establish a framework to monitor progress
- Assess data accessibility quality and security
- Design implementation and evaluation
- Share your story and success
-
- Working with a range of evidence
- Thank you
- References
- Resources
- About GovEx
-
WorkingwitharangeofevidencePresentlythereisalotofattentionandinvestmentinevidence-basedpolicymakingincludingthefederalbipartisanCommissiononEvidence-BasedPolicymakingthePewCharitableTrustsreportsandtoolkitsonHowStatesEngageinEvidence-BasedPolicymakingandtheLauraandJohnArnoldFoundationsupportedEvidence-BasedPolicymakingCollaborativeManyoftheseinitiativesfocusonrigorousevaluation--meaningrandomizedcontrolledtrials(RCT)orotherexperimentalmodels--asthebestkindofevidencetouseHoweverthisbodyofevidenceisofteninsufficientforthespecifickindsofquestionsthatgovernmentscanhavewhenmakingadecisionaboutpolicyInthisguideweproposeanapproachtosupportgovernmentsinincorporatingadditionaltypesofevidenceintotheirdecisionmaking
OnewaytounderstandarangeofevidenceistouseNestarsquosStandardsofEvidencetoframeanddefineevidenceTheBritishnonprofitNestaworkstoinjectpositivechangeintothewayUKgovernmentpoliciesandprogramsfunctionOneoftheirkeyareasofworkhasbeendevelopingWhatWorksCentresorganizationsthatcollectcommissionsynthesizeanddisseminateevidenceaboutinterventioneffectivenessacrossarangeofsocialimpactareasNestadevelopedthefollowingStandardsofEvidencetoprovideausefulwaytodocumenthowdifferentkindsofinformationdemonstratedifferentlevelsofevidence
Workingwitharangeofevidence
56
Theimportanceofpotentiallybiasedinformation
Onechallengepresentedbyanevidenceclassificationthatacknowledgesthevalueofnon-scientificformsofevidenceistofigureouthowtobalanceallofthepiecestogetherwhenyouusethemTounderstandhowtodothiswehavetothinkabouthowtopositionpotentiallybiasedinformationindecisionmaking
WhenwelookatinformationwhichhasnotbeenrigorouslyevaluatedweacceptthatthisinformationmightbebiasedThewholereasonweconductrigorousevaluationistoremovebiasfromourobservationsWeconductstructuredtrialstoproduceresultsthatareconsistentandgeneralizableIfafindingisunbiasedthisaccuracymdashthislackofbiasmdashcanbeprovedbecauseanyindependenttestercanreproduceitwhentheyfollowanidenticalprocess
YoucanlookatpotentiallybiasedinformationasjustinformationthatisearlieroninitspathtorefinementreplicationandverifiedaccuracyApieceofinformationbasedonanyoneindividualrsquosobservationsislikelytobebiasedbecauseitiscomingfromjustonesourceBiasisreducedbygatheringtogetherlotsofindependentsourcesofinformationandfiguringoutitsmostcommonattributes
MaybethebiasedinformationmakesaclaimthatistoolargemdashitsayssomethingistrueunderallconditionswhenitisreallyonlytrueundercertainconditionsorthataneffectofaninterventionisbiggerthanitstrueaverageorthataneffectlastsovertimewhenitdoesnotPerhapschangingtheclaimtobemorespecificmdashcriticallyreviewingtheinformationtoseeifspecificconditionsmustbemettoachievetheoutcomesmdashwouldmaketheclaimmoreaccurate
IfyoutestedyourclaimfounditdidnotholdupinsomecaseschangedittesteditagainuntilyourclaimrepeatedlyheldupthenyoucoulddemonstratethattheclaimwasnolongerbiasedbutaccurateThisprocessoftestingimprovingtestingandvalidatingiswhatwegetfromstructuredscientificstudy
Withindividualpiecesofinformation--crediblehypothesescasestudiesexpertopinion--youcannotassumethatkindofaccuracyAtthesametimetheseearlyobservationsarefoundationaltoanyreplicablescientificobservationsWewillnevergetanywhereifwedonotstartwiththesepiecesofinformationeveniftheymightbebiased
Combininglevelsofevidencetogetthemostdetailedinformationforyourcity
ItisespeciallyimportanttoacceptdifferentformsofevidencewhenyoufacequestionsthathavenotyetbeentestedbyindependentstudiesSomeofthemostimportantquestionsrelatetothingsthatareveryspecifictoyoursituationYoufaceaparticularcombinationof
Workingwitharangeofevidence
57
issuesactorstrendsandresourcesthatisultimatelydifferentfromeveryotherplaceintheworld
ThisisimportantbecauseaspolicypractitionersandnotscientistsyoucaremoreaboutyourspecificcasethanahypotheticalaveragegeneralcaseYoucarelessaboutthetotallikelyrangeofimpactsfromapolicythanhowthissingleimplementationwillworkYouwantinformationaboutpoliciesthatwillprovideyouwiththebestpossiblepredictionabouthowtheywillworkinyourcontext
ByputtingdifferentkindsofevidenceintothesameframeworkNestarsquosStandardsofEvidenceimplythatwhenyoucannotgetrigorouslytestedexperimentallyverifiedinformationitisappropriatetoworkwithwhatwedohaveThisisrelevantforthemunicipalimplementationcontextbecausenostudypulledfromaliteraturereviewwillaccountforallofthevariablesyoufaceinyourowncontext
Intheabsenceofhavingastudythatpredictshowaninterventionworksinyourspecificcontextyouwillneednon-scientificinformationtotrytofillinthosegapsYouwillneedtogatherstoriesfromresidentsstaffmembersandotherstakeholdersthatsitonLevels1or2ofthestandardsofevidence
Someofthekindsofinformationyouwillprobablygetmoreeasilyfrompeoplethanstudiesinclude
howpeopleincitieslikeyoursreactedtoaninterventioninpublicforumshowpeoplewhoimplementedaninterventioninacitylikeyoursfiguredouthowtogetitpassedhowthecitycommunicatedpubliclyabouttheapparentbenefitsanddrawbacksoftheinterventionthepoliticalacceptabilityoftheintervention
Youcanremainsimultaneouslyawarethattheinformationyouaregatheringthroughindividualstoriesisbiased--notgeneralizableinformedbyspecificperspectives--whilealsorecognizingthathavingsomeinformationaboutsomethingimportantisbetterthannoinformationEvidencefromthefirsttwoevidencelevelsareessentialbuildingblocksofknowledgeandperfectlyusefulwhenyoucannotgetdetailedunbiasedknowledge
Youhavetostartsomewhereandyoushouldalwaysstartwithevidenceyouhaveaboutyourveryspecificconditionsthatyouknowareimportantforsuccess
Workingwitharangeofevidence
58
Thankyou
Interviewswithcities
GovExheldsemi-structuredinterviewswithsixstaffmembersinfourcitiestoestablishtheinitialstepsoftheRoadmapWeaskedrespondentstotalkaboutaninitiativeorpolicythattheyhavebeenresponsibleforimplementingintheircityRespondentsfromthreecitiesspokeabouttheireffortstoaddresshomelessnessandrespondentsfromonecityspokeaboutblightAfterrespondentsidentifiedtheproblemtheyspokeabouthowthecityprioritizedthisissuethevariousstepsinvolvedandbarrierspresent
GovExconductedtextanalysisoftheseinterviewswhichrevealedthatpeoplewasthemostcommonlyusedtermduringtheseconversationswhendiscussingthestepstochangeaprogramorpolicyThisisathemethroughoutthisRoadmapThefirstsectionbeginswithidentifyingtherelevantpeopleTheconversationsaroundevidencearefocusedonhumanbehaviorratherthanmethodologyorstrengthofevidence
Thankyou
59
OncewehadadraftoftheRoadmapwefollowedupwithcitiesonanindividualbasistolearnmoreaboutwhatwouldbehelpfulWearegratefultothefollowingcitystafffortheinterviewsandindividualconversations
MichaelBaskinChiefPolicyOfficerChattanoogaTN
MelissaBridgesPerformanceandInnovationCoordinatorLittleRockAR
VictoriaCarreoacutenAdministrativeOfficerLasVegasNV
TyroneGrandstrandCommunityDevelopmentPlannerFargoND
TibLaughlinDirectorofGeneralServicesKansasCityKS
TimMorelandDirectorofPerformanceManagementandOpenDataChattanoogaTN
CharlesRosendahlManagementAnalystDentonTX
Thankyou
60
MelissaSiebenAssistantCountyAdministratorKansasCityKS
GinaThompsonDirectorofStrategicServicesLewisvilleTX
JHUFacultycollaboration
GovExalsoconvenedthecitycontactsandfacultypartnersinaseriesofAdvisoryGroupmeetingsThesemeetingsservedtoconnectcitiesandfacultytounderstandoneanothersperspective
StefanBaralBloombergSchoolofPublicHealth
AshleyBernerJHUSchoolofEducation
MelissaDavey-RothwellBloombergSchoolofPublicHealth
JonLinksBloombergSchoolofPublicHealth
TamarMendelsonBloombergSchoolofPublicHealth
KristinMmariBloombergSchoolofPublicHealth
KeshiaPollackPorterBloombergSchoolofPublicHealth
JoshSharfsteinBloombergSchoolofPublicHealth
LizStuartBloombergSchoolofPublicHealth
LindsayThompsonCareyBusinessSchool
DanielWebsterBloombergSchoolofPublicHealth
Partnersindataandevidence-basedpolicy
Wealsospokewithpartnerswhoareexpertsinthefieldofevidence-basedpolicywhosharedtheirknowledgeandresourcesofcurrentandfuturethinkinginevidence-informedthinking
WendyBarrenoAssociateDirectorWilsonSheehanLabforEconomicOpportunitiesUniversityofNotreDame
TiffanyDavenportDirectorofDataPartnershipsBaltimoresPromise
TomiHiersDirectorBaltimoreCivicSiteAnnieECaseyFoundation
ChrisKingsleySeniorAssociateAnnieECaseyFoundation
KathrynOliverAssociateProfessorLondonSchoolofHygieneandTropicalMedicine
Thankyou
61
KristenWatkinsSeniorPolicyAssociateJ-PAL
GovExParticipants
GovExAnalystMiriamMcKinneyanalyzedtheinterviewtranscriptsandconductedtextanalysisSeniorImplementationAdvisorEmilyShawwillintegrateRoadmapactivitieswithadditionalservicedeliveryactivitiesatGovEx
Thankyou
62
ReferencesAbrahamKGampHaskinsR(2017)Thepromiseofevidence-basedpolicymakingReportofthecommissiononevidence-basedpolicymakingWashingtonDCCommissiononEvidence-basedPolicymakingBigmove2Extendingthegreen(nd)Retrievedfromhttpswwwcityoflewisvillecomdoing-businesslewisville-2025big-move-2-extending-the-greenBatesMAampGlennersterR(2017)ThegeneralizabilitypuzzleRetrievedfromhttpsssirorgarticlesentrythe_generalizability_puzzleBuildwithnotfor(2018January)RetrievedfromhttpsbuildwithnotforwordpresscomCairneyPampOliverK(2017)Evidence-basedpolicymakingisnotlikeevidence-basedmedicinesohowfarshouldyougotobridgethedividebetweenevidenceandpolicyHealthResearchPolicyandSystemsdoi101186s12961-017-0192-xCairneyPOliverKampWellsteadA(2016)TobridgethedividebetweenevidenceandpolicyReduceambiguityasmuchasuncertaintyPublicAdministrationReview76(3)399-402Retrievedfromhttpdxdoiorg101111puar12555Centreforpublicimpact(nd)RetrievedfromhttpswwwcentreforpublicimpactorgCernaL(2013)ThenatureofpolicychangeandimplementationAreviewofdifferenttheoreticalapproachesOrganisationforEconomicCo-OperationandDevelopmentRetrievedfromhttpswwwoecdorgeduceriTheNatureofPolicyChangeandImplementationpdfColoradoSprings(nd)Retrievedfromhttpscoloradospringsgovcommunicationspagewhat-expect-when-you-speakupContinuumofcare(coc)program(nd)RetrievedfromhttpswwwhudexchangeinfoprogramscocCouncilpresidentkicksoff$12millionyouthfund(nd)Retrievedfromhttpwwwbaltimorecitycouncilcomcontentcouncil-president-kicks-12-million-youth-fundDurlakJAampDuPreEP(2008)ImplementationmattersAreviewofresearchontheinfluenceofimplementationonprogramoutcomesandthefactorsaffectingimplementationAmericanJournalofCommunityPsychology41327-350Retrievedfromhttpswwwncbinlmnihgovpubmed18322790Fishbonediagram(ishikawa)(nd)Retrievedfromhttpasqorglearn-about-qualitycause-analysis-toolsoverviewfishbonehtmlFivewhysGettingtotherootofaproblemquickly(nd)RetrievedfromhttpswwwmindtoolscompagesarticlenewTMC_5Whtm
References
63
FligsteinNampMcAdamD(2011)TowardageneraltheoryofstrategicactionfieldsSociologicalTheory29(1)1-26RetrievedfromhttpsociologyberkeleyedusitesdefaultfilesfacultyfligsteinFligstein20McAdam20Soc20Theory20PaperpdHeinkelP(2018January12)CitymullspermitsforbikesharingcompaniesRetrievedfromhttpwwwdentonrccomnewsdenton20180112city-mulls-permits-bike-sharing-companiesHowisareamedianincomecalculated(2012October)Retrievedfromhttpswwwhudexchangeinfofaqs843how-is-area-median-income-calculatedLampkinLMampHatryHP(2003)KeystepsinoutcomemanagementUrbanInstituteRetrievedfromhttpswwwurbanorgresearchpublicationkey-steps-outcome-managementMarshDampSmithM(2000)UnderstandingpolicynetworksTowardsadialecticalapproachPoliticalStudies484-21Retrievedfromhttpjournalssagepubcomdoiabs1011111467-924800247McCannL(nd)RetrievedfromhttplaurenellencomMoultonSampSandfortJR(2016)ThestrategicactionfieldframeworkforpolicyimplementationresearchPolicyStudiesJournal45(1)Retrievedfromhttponlinelibrarywileycomstore101111psj12147assetpsj12147pdfv=1ampt=jdkhhcp4amps=e184c625e1855a6b1615ccb99ee27b1c0b3212d8PaynterB(2018January1)What100citiesarelearningfromeachotherbysharingtheirdataRetrievedfromhttpswwwfastcompanycom40518004what-100-cities-are-learning-from-each-other-by-sharing-their-dataPeakacademy(nd)Retrievedfromhttpswwwdenvergovorgcontentdenvergovenmayors-officeprograms-initiativespeak-performancepeak-academyhtmlPinnockHBarwickMCarpenterCREldridgeSGrandesGGriffithsCJRycroft-MaloneJMeissnerPMurrayEPatelASheikhAampTaylorSJC(2017)Standardsforreportingimplementationstudies(stari)ExplanationandelaborationdocumentBMJOpenRetrievedfromhttpbmjopenbmjcomcontentbmjopen74e013318fullpdfProctorESilmereHRaghavanRHovmandPAaronsGBungerAGriffeyRampHensleyM(2011)OutcomesforimplementationresearchConceptualdistinctionsmeasurementchallengesandresearchagendaAdministrationandPolicyinMentalHealth38(2)65-76doi101007s10488-010-0319-7TandonSD(2018)ProgressincommunityhealthpartnershipsResearcheducationandactionRetrievedfromhttpswwwpressjhuedujournalsprogress-community-health-partnerships-research-education-and-actionProvanKGampKenisP(2007)ModesofnetworkgovernanceStructuremanagementandeffectivenessJournalofPublicAdministrationResearchandTheory
References
64
18(2)229-252Retrievedfromhttpsacademicoupcomjpartarticle182229935895SabatierPA(1988)Anadvocacycoalitionframeworkofpolicychangeandtheroleofpolicy-orientedlearningthereinPolicySciences21(3)129-168Retrievedfromhttpwwwjstororgstable4532139SandstromAampCarlssonL(2008)TheperformanceofpolicynetworksTherelationbetweennetworkstructureandnetworkperformancePolicyStudiesJournal36497-524Retrievedfromhttponlinelibrarywileycomdoi101111j1541-0072200800281xabstractSchwartzMampWilsonE(nd)WhocanaffordtoliveinahomeAlookatdatafromthe2006americancommunitysurveyRetrievedfromhttpswwwcensusgovhousingcensuspublicationswho-can-affordpdfShearerJCAbelsonAKouyateBLavisJNampWaltG(2016)WhydopolicieschangeInstitutionsinterestsideasandnetworksinthreecasesofpolicyreformHealthPolicyandPlanning31(9)1200-1211Retrievedfromhttpsdoiorg101093heapolczw052ThelessonsofadministrativedataHigh-profilepolicy-relevantresearchpoweredbyadministrativedata(nd)Retrievedfromhttpswwwpovertyactionlaborgsitesdefaultfilesresources20161114-The-Lessons-of-Administrative-DatapdfToolsandotherneedsforqualitativeandquantitativemethods(2015February)AgencyforHealthcareResearchandQualityRetrievedfromhttpwwwahrqgovprofessionalsquality-patient-safetytalkingqualityassesstoolshtml
WeaverRCampBagchi-SenS(2013)SpatialanalysisofurbandeclineThegeographyofblightAppliedGeography4061-70
References
65
UniversityPartners
Name Title Affiliation Area(s)ofexpertise
Contactinformation
Examplesofworkingwithnon-academicdecisionmakers
WendyPBarreno
AssociateDirector
WilsonSheehanLabforEconomicOpportunitiesUniversityofNotreDame
LEOsfocusareasincludehealtheducationhousingchildwelfareeconomicopportunitycriminaljustice
wbarrenonduduorleondedu
MelissaDavey-RothwellPhD
AssociateScientist
JohnsHopkinsBloombergSchoolofPublicHealth
Health mdavey1jhuedu410-614-5854
BaltimoreCityHealthDepartmentLeadersMarylandDepartmentofHealthAddictionRecoveryServicesinDCExecutiveDirectorsofcommunityorganizationsacrosstheUSthatfocusonHIVSTIsubstanceabuseandimplementationscience
ElizabethStuart
AssociateDeanforEducationandProfessorDepartmentsofMentalHealthBiostatisticsandHealthPolicyand
JohnsHopkinsBloombergSchoolofPublicHealth
Educationsocialpolicystatisticalmethods
estuartjhuedu
Resources
66
Management
Design-ThinkingResources
EngagementGuideforCivicTech
Journeymap
PerformanceManagementResources
DenverPeakAcademy
Proxymeasuresguide
Talkingaboutperformancewithoutcomplicatingit(video)
SettingPerformanceTargetsguide
Datavisualizationguide
Performancemanagementgettingstartedguide
Exercises
Fishboneexercise
5Whys
AdministrativeData
TheLessonsofAdministrativeData(JPAL)
EvaluationResources
Quantitativeandqualitativemethods(USDepartmentofHealthandHumanServicesAgencyforHealthcareResearchandQuality)
KeyStepsinOutcomeManagement(UrbanInstitute)
RCT-YES
Sageresearchmethods
BestPracticesforMixedMethodsResearchintheHealthSciences
ACCORDChecklist
Pilotandimplementationplanning
Benchmarkingexamplesandproxymeasures
Seattleopendataplaybook
Resources
67
Datasciencescopingguide
Resources
68
AboutGovExGovExhelpsgovernmentsusedataandevidencetoimproveresidentsrsquolivesWesupportgovernmentsinbuildingtheirinternalcapacitytooperatesustainableanalyticspracticeswhichimprovecommunityoutcomesinfieldsfrompublicsafetytoeconomicdevelopmentGovExhasaproventrackrecordofworkinginconcertwithpartnerstoprovidetechnicalassistanceandtrainingatscale
SinceourlaunchinApril2015GovExhas
Trainedover2000governmentemployeesinpersonandonlineintopicsincludingdatamanagementperformancemanagementopendatacommunityengagementandanalytics
Providedtechnicalassistancetomorethan100localgovernmentsacrosstheglobeimprovingtheiruseofdatatoachieveefficienciesinservicedeliveryandbetteroutcomesforresidents
Assessedthecapacityofmorethan130governmentalorganizationstodeterminehowleadersstaffandexternalstakeholdersmanageandusedata
Localgovernmentsareincreasinglyonthefrontlinesofaddressinglargescaleproblemswithglobalimplicationssuchasclimatechangepublicsafetyandeconomicinequality
Whileevidence-basedinterventionstohelpsolvethoseproblemsareavailablelocalgovernmentstaffhavelimitedtimeandcapacitytofindandimplementresearch-basedsolutionsAsaresultcitiescontinuetoworkthewaytheyalwayshave--makingincrementalimprovementstoprogramsbutnotalwayscreatingchangeatasystem-level
ThroughresourcedevelopmentandcollaborationwithJHUfacultyGovExissupportingcityleadersinusingdataandevidencetomakedecisionsandcreatinglastingrelationshipswithresearchers
TheRoadmapwasdevelopedbyGovExaspartofitspartnershipwithBloombergPhilanthropiesWhatWorksCitiesinitiative
AboutGovEx
69
- Title page
- Introduction and purpose of the Roadmap
- Using the Roadmap
-
- Gather information about the problem
- Strategically engage key partners
- Review the evidence and collect details
- Decide on a policy to implement based on local factors
- Establish a framework to monitor progress
- Assess data accessibility quality and security
- Design implementation and evaluation
- Share your story and success
-
- Working with a range of evidence
- Thank you
- References
- Resources
- About GovEx
-
Theimportanceofpotentiallybiasedinformation
Onechallengepresentedbyanevidenceclassificationthatacknowledgesthevalueofnon-scientificformsofevidenceistofigureouthowtobalanceallofthepiecestogetherwhenyouusethemTounderstandhowtodothiswehavetothinkabouthowtopositionpotentiallybiasedinformationindecisionmaking
WhenwelookatinformationwhichhasnotbeenrigorouslyevaluatedweacceptthatthisinformationmightbebiasedThewholereasonweconductrigorousevaluationistoremovebiasfromourobservationsWeconductstructuredtrialstoproduceresultsthatareconsistentandgeneralizableIfafindingisunbiasedthisaccuracymdashthislackofbiasmdashcanbeprovedbecauseanyindependenttestercanreproduceitwhentheyfollowanidenticalprocess
YoucanlookatpotentiallybiasedinformationasjustinformationthatisearlieroninitspathtorefinementreplicationandverifiedaccuracyApieceofinformationbasedonanyoneindividualrsquosobservationsislikelytobebiasedbecauseitiscomingfromjustonesourceBiasisreducedbygatheringtogetherlotsofindependentsourcesofinformationandfiguringoutitsmostcommonattributes
MaybethebiasedinformationmakesaclaimthatistoolargemdashitsayssomethingistrueunderallconditionswhenitisreallyonlytrueundercertainconditionsorthataneffectofaninterventionisbiggerthanitstrueaverageorthataneffectlastsovertimewhenitdoesnotPerhapschangingtheclaimtobemorespecificmdashcriticallyreviewingtheinformationtoseeifspecificconditionsmustbemettoachievetheoutcomesmdashwouldmaketheclaimmoreaccurate
IfyoutestedyourclaimfounditdidnotholdupinsomecaseschangedittesteditagainuntilyourclaimrepeatedlyheldupthenyoucoulddemonstratethattheclaimwasnolongerbiasedbutaccurateThisprocessoftestingimprovingtestingandvalidatingiswhatwegetfromstructuredscientificstudy
Withindividualpiecesofinformation--crediblehypothesescasestudiesexpertopinion--youcannotassumethatkindofaccuracyAtthesametimetheseearlyobservationsarefoundationaltoanyreplicablescientificobservationsWewillnevergetanywhereifwedonotstartwiththesepiecesofinformationeveniftheymightbebiased
Combininglevelsofevidencetogetthemostdetailedinformationforyourcity
ItisespeciallyimportanttoacceptdifferentformsofevidencewhenyoufacequestionsthathavenotyetbeentestedbyindependentstudiesSomeofthemostimportantquestionsrelatetothingsthatareveryspecifictoyoursituationYoufaceaparticularcombinationof
Workingwitharangeofevidence
57
issuesactorstrendsandresourcesthatisultimatelydifferentfromeveryotherplaceintheworld
ThisisimportantbecauseaspolicypractitionersandnotscientistsyoucaremoreaboutyourspecificcasethanahypotheticalaveragegeneralcaseYoucarelessaboutthetotallikelyrangeofimpactsfromapolicythanhowthissingleimplementationwillworkYouwantinformationaboutpoliciesthatwillprovideyouwiththebestpossiblepredictionabouthowtheywillworkinyourcontext
ByputtingdifferentkindsofevidenceintothesameframeworkNestarsquosStandardsofEvidenceimplythatwhenyoucannotgetrigorouslytestedexperimentallyverifiedinformationitisappropriatetoworkwithwhatwedohaveThisisrelevantforthemunicipalimplementationcontextbecausenostudypulledfromaliteraturereviewwillaccountforallofthevariablesyoufaceinyourowncontext
Intheabsenceofhavingastudythatpredictshowaninterventionworksinyourspecificcontextyouwillneednon-scientificinformationtotrytofillinthosegapsYouwillneedtogatherstoriesfromresidentsstaffmembersandotherstakeholdersthatsitonLevels1or2ofthestandardsofevidence
Someofthekindsofinformationyouwillprobablygetmoreeasilyfrompeoplethanstudiesinclude
howpeopleincitieslikeyoursreactedtoaninterventioninpublicforumshowpeoplewhoimplementedaninterventioninacitylikeyoursfiguredouthowtogetitpassedhowthecitycommunicatedpubliclyabouttheapparentbenefitsanddrawbacksoftheinterventionthepoliticalacceptabilityoftheintervention
Youcanremainsimultaneouslyawarethattheinformationyouaregatheringthroughindividualstoriesisbiased--notgeneralizableinformedbyspecificperspectives--whilealsorecognizingthathavingsomeinformationaboutsomethingimportantisbetterthannoinformationEvidencefromthefirsttwoevidencelevelsareessentialbuildingblocksofknowledgeandperfectlyusefulwhenyoucannotgetdetailedunbiasedknowledge
Youhavetostartsomewhereandyoushouldalwaysstartwithevidenceyouhaveaboutyourveryspecificconditionsthatyouknowareimportantforsuccess
Workingwitharangeofevidence
58
Thankyou
Interviewswithcities
GovExheldsemi-structuredinterviewswithsixstaffmembersinfourcitiestoestablishtheinitialstepsoftheRoadmapWeaskedrespondentstotalkaboutaninitiativeorpolicythattheyhavebeenresponsibleforimplementingintheircityRespondentsfromthreecitiesspokeabouttheireffortstoaddresshomelessnessandrespondentsfromonecityspokeaboutblightAfterrespondentsidentifiedtheproblemtheyspokeabouthowthecityprioritizedthisissuethevariousstepsinvolvedandbarrierspresent
GovExconductedtextanalysisoftheseinterviewswhichrevealedthatpeoplewasthemostcommonlyusedtermduringtheseconversationswhendiscussingthestepstochangeaprogramorpolicyThisisathemethroughoutthisRoadmapThefirstsectionbeginswithidentifyingtherelevantpeopleTheconversationsaroundevidencearefocusedonhumanbehaviorratherthanmethodologyorstrengthofevidence
Thankyou
59
OncewehadadraftoftheRoadmapwefollowedupwithcitiesonanindividualbasistolearnmoreaboutwhatwouldbehelpfulWearegratefultothefollowingcitystafffortheinterviewsandindividualconversations
MichaelBaskinChiefPolicyOfficerChattanoogaTN
MelissaBridgesPerformanceandInnovationCoordinatorLittleRockAR
VictoriaCarreoacutenAdministrativeOfficerLasVegasNV
TyroneGrandstrandCommunityDevelopmentPlannerFargoND
TibLaughlinDirectorofGeneralServicesKansasCityKS
TimMorelandDirectorofPerformanceManagementandOpenDataChattanoogaTN
CharlesRosendahlManagementAnalystDentonTX
Thankyou
60
MelissaSiebenAssistantCountyAdministratorKansasCityKS
GinaThompsonDirectorofStrategicServicesLewisvilleTX
JHUFacultycollaboration
GovExalsoconvenedthecitycontactsandfacultypartnersinaseriesofAdvisoryGroupmeetingsThesemeetingsservedtoconnectcitiesandfacultytounderstandoneanothersperspective
StefanBaralBloombergSchoolofPublicHealth
AshleyBernerJHUSchoolofEducation
MelissaDavey-RothwellBloombergSchoolofPublicHealth
JonLinksBloombergSchoolofPublicHealth
TamarMendelsonBloombergSchoolofPublicHealth
KristinMmariBloombergSchoolofPublicHealth
KeshiaPollackPorterBloombergSchoolofPublicHealth
JoshSharfsteinBloombergSchoolofPublicHealth
LizStuartBloombergSchoolofPublicHealth
LindsayThompsonCareyBusinessSchool
DanielWebsterBloombergSchoolofPublicHealth
Partnersindataandevidence-basedpolicy
Wealsospokewithpartnerswhoareexpertsinthefieldofevidence-basedpolicywhosharedtheirknowledgeandresourcesofcurrentandfuturethinkinginevidence-informedthinking
WendyBarrenoAssociateDirectorWilsonSheehanLabforEconomicOpportunitiesUniversityofNotreDame
TiffanyDavenportDirectorofDataPartnershipsBaltimoresPromise
TomiHiersDirectorBaltimoreCivicSiteAnnieECaseyFoundation
ChrisKingsleySeniorAssociateAnnieECaseyFoundation
KathrynOliverAssociateProfessorLondonSchoolofHygieneandTropicalMedicine
Thankyou
61
KristenWatkinsSeniorPolicyAssociateJ-PAL
GovExParticipants
GovExAnalystMiriamMcKinneyanalyzedtheinterviewtranscriptsandconductedtextanalysisSeniorImplementationAdvisorEmilyShawwillintegrateRoadmapactivitieswithadditionalservicedeliveryactivitiesatGovEx
Thankyou
62
ReferencesAbrahamKGampHaskinsR(2017)Thepromiseofevidence-basedpolicymakingReportofthecommissiononevidence-basedpolicymakingWashingtonDCCommissiononEvidence-basedPolicymakingBigmove2Extendingthegreen(nd)Retrievedfromhttpswwwcityoflewisvillecomdoing-businesslewisville-2025big-move-2-extending-the-greenBatesMAampGlennersterR(2017)ThegeneralizabilitypuzzleRetrievedfromhttpsssirorgarticlesentrythe_generalizability_puzzleBuildwithnotfor(2018January)RetrievedfromhttpsbuildwithnotforwordpresscomCairneyPampOliverK(2017)Evidence-basedpolicymakingisnotlikeevidence-basedmedicinesohowfarshouldyougotobridgethedividebetweenevidenceandpolicyHealthResearchPolicyandSystemsdoi101186s12961-017-0192-xCairneyPOliverKampWellsteadA(2016)TobridgethedividebetweenevidenceandpolicyReduceambiguityasmuchasuncertaintyPublicAdministrationReview76(3)399-402Retrievedfromhttpdxdoiorg101111puar12555Centreforpublicimpact(nd)RetrievedfromhttpswwwcentreforpublicimpactorgCernaL(2013)ThenatureofpolicychangeandimplementationAreviewofdifferenttheoreticalapproachesOrganisationforEconomicCo-OperationandDevelopmentRetrievedfromhttpswwwoecdorgeduceriTheNatureofPolicyChangeandImplementationpdfColoradoSprings(nd)Retrievedfromhttpscoloradospringsgovcommunicationspagewhat-expect-when-you-speakupContinuumofcare(coc)program(nd)RetrievedfromhttpswwwhudexchangeinfoprogramscocCouncilpresidentkicksoff$12millionyouthfund(nd)Retrievedfromhttpwwwbaltimorecitycouncilcomcontentcouncil-president-kicks-12-million-youth-fundDurlakJAampDuPreEP(2008)ImplementationmattersAreviewofresearchontheinfluenceofimplementationonprogramoutcomesandthefactorsaffectingimplementationAmericanJournalofCommunityPsychology41327-350Retrievedfromhttpswwwncbinlmnihgovpubmed18322790Fishbonediagram(ishikawa)(nd)Retrievedfromhttpasqorglearn-about-qualitycause-analysis-toolsoverviewfishbonehtmlFivewhysGettingtotherootofaproblemquickly(nd)RetrievedfromhttpswwwmindtoolscompagesarticlenewTMC_5Whtm
References
63
FligsteinNampMcAdamD(2011)TowardageneraltheoryofstrategicactionfieldsSociologicalTheory29(1)1-26RetrievedfromhttpsociologyberkeleyedusitesdefaultfilesfacultyfligsteinFligstein20McAdam20Soc20Theory20PaperpdHeinkelP(2018January12)CitymullspermitsforbikesharingcompaniesRetrievedfromhttpwwwdentonrccomnewsdenton20180112city-mulls-permits-bike-sharing-companiesHowisareamedianincomecalculated(2012October)Retrievedfromhttpswwwhudexchangeinfofaqs843how-is-area-median-income-calculatedLampkinLMampHatryHP(2003)KeystepsinoutcomemanagementUrbanInstituteRetrievedfromhttpswwwurbanorgresearchpublicationkey-steps-outcome-managementMarshDampSmithM(2000)UnderstandingpolicynetworksTowardsadialecticalapproachPoliticalStudies484-21Retrievedfromhttpjournalssagepubcomdoiabs1011111467-924800247McCannL(nd)RetrievedfromhttplaurenellencomMoultonSampSandfortJR(2016)ThestrategicactionfieldframeworkforpolicyimplementationresearchPolicyStudiesJournal45(1)Retrievedfromhttponlinelibrarywileycomstore101111psj12147assetpsj12147pdfv=1ampt=jdkhhcp4amps=e184c625e1855a6b1615ccb99ee27b1c0b3212d8PaynterB(2018January1)What100citiesarelearningfromeachotherbysharingtheirdataRetrievedfromhttpswwwfastcompanycom40518004what-100-cities-are-learning-from-each-other-by-sharing-their-dataPeakacademy(nd)Retrievedfromhttpswwwdenvergovorgcontentdenvergovenmayors-officeprograms-initiativespeak-performancepeak-academyhtmlPinnockHBarwickMCarpenterCREldridgeSGrandesGGriffithsCJRycroft-MaloneJMeissnerPMurrayEPatelASheikhAampTaylorSJC(2017)Standardsforreportingimplementationstudies(stari)ExplanationandelaborationdocumentBMJOpenRetrievedfromhttpbmjopenbmjcomcontentbmjopen74e013318fullpdfProctorESilmereHRaghavanRHovmandPAaronsGBungerAGriffeyRampHensleyM(2011)OutcomesforimplementationresearchConceptualdistinctionsmeasurementchallengesandresearchagendaAdministrationandPolicyinMentalHealth38(2)65-76doi101007s10488-010-0319-7TandonSD(2018)ProgressincommunityhealthpartnershipsResearcheducationandactionRetrievedfromhttpswwwpressjhuedujournalsprogress-community-health-partnerships-research-education-and-actionProvanKGampKenisP(2007)ModesofnetworkgovernanceStructuremanagementandeffectivenessJournalofPublicAdministrationResearchandTheory
References
64
18(2)229-252Retrievedfromhttpsacademicoupcomjpartarticle182229935895SabatierPA(1988)Anadvocacycoalitionframeworkofpolicychangeandtheroleofpolicy-orientedlearningthereinPolicySciences21(3)129-168Retrievedfromhttpwwwjstororgstable4532139SandstromAampCarlssonL(2008)TheperformanceofpolicynetworksTherelationbetweennetworkstructureandnetworkperformancePolicyStudiesJournal36497-524Retrievedfromhttponlinelibrarywileycomdoi101111j1541-0072200800281xabstractSchwartzMampWilsonE(nd)WhocanaffordtoliveinahomeAlookatdatafromthe2006americancommunitysurveyRetrievedfromhttpswwwcensusgovhousingcensuspublicationswho-can-affordpdfShearerJCAbelsonAKouyateBLavisJNampWaltG(2016)WhydopolicieschangeInstitutionsinterestsideasandnetworksinthreecasesofpolicyreformHealthPolicyandPlanning31(9)1200-1211Retrievedfromhttpsdoiorg101093heapolczw052ThelessonsofadministrativedataHigh-profilepolicy-relevantresearchpoweredbyadministrativedata(nd)Retrievedfromhttpswwwpovertyactionlaborgsitesdefaultfilesresources20161114-The-Lessons-of-Administrative-DatapdfToolsandotherneedsforqualitativeandquantitativemethods(2015February)AgencyforHealthcareResearchandQualityRetrievedfromhttpwwwahrqgovprofessionalsquality-patient-safetytalkingqualityassesstoolshtml
WeaverRCampBagchi-SenS(2013)SpatialanalysisofurbandeclineThegeographyofblightAppliedGeography4061-70
References
65
UniversityPartners
Name Title Affiliation Area(s)ofexpertise
Contactinformation
Examplesofworkingwithnon-academicdecisionmakers
WendyPBarreno
AssociateDirector
WilsonSheehanLabforEconomicOpportunitiesUniversityofNotreDame
LEOsfocusareasincludehealtheducationhousingchildwelfareeconomicopportunitycriminaljustice
wbarrenonduduorleondedu
MelissaDavey-RothwellPhD
AssociateScientist
JohnsHopkinsBloombergSchoolofPublicHealth
Health mdavey1jhuedu410-614-5854
BaltimoreCityHealthDepartmentLeadersMarylandDepartmentofHealthAddictionRecoveryServicesinDCExecutiveDirectorsofcommunityorganizationsacrosstheUSthatfocusonHIVSTIsubstanceabuseandimplementationscience
ElizabethStuart
AssociateDeanforEducationandProfessorDepartmentsofMentalHealthBiostatisticsandHealthPolicyand
JohnsHopkinsBloombergSchoolofPublicHealth
Educationsocialpolicystatisticalmethods
estuartjhuedu
Resources
66
Management
Design-ThinkingResources
EngagementGuideforCivicTech
Journeymap
PerformanceManagementResources
DenverPeakAcademy
Proxymeasuresguide
Talkingaboutperformancewithoutcomplicatingit(video)
SettingPerformanceTargetsguide
Datavisualizationguide
Performancemanagementgettingstartedguide
Exercises
Fishboneexercise
5Whys
AdministrativeData
TheLessonsofAdministrativeData(JPAL)
EvaluationResources
Quantitativeandqualitativemethods(USDepartmentofHealthandHumanServicesAgencyforHealthcareResearchandQuality)
KeyStepsinOutcomeManagement(UrbanInstitute)
RCT-YES
Sageresearchmethods
BestPracticesforMixedMethodsResearchintheHealthSciences
ACCORDChecklist
Pilotandimplementationplanning
Benchmarkingexamplesandproxymeasures
Seattleopendataplaybook
Resources
67
Datasciencescopingguide
Resources
68
AboutGovExGovExhelpsgovernmentsusedataandevidencetoimproveresidentsrsquolivesWesupportgovernmentsinbuildingtheirinternalcapacitytooperatesustainableanalyticspracticeswhichimprovecommunityoutcomesinfieldsfrompublicsafetytoeconomicdevelopmentGovExhasaproventrackrecordofworkinginconcertwithpartnerstoprovidetechnicalassistanceandtrainingatscale
SinceourlaunchinApril2015GovExhas
Trainedover2000governmentemployeesinpersonandonlineintopicsincludingdatamanagementperformancemanagementopendatacommunityengagementandanalytics
Providedtechnicalassistancetomorethan100localgovernmentsacrosstheglobeimprovingtheiruseofdatatoachieveefficienciesinservicedeliveryandbetteroutcomesforresidents
Assessedthecapacityofmorethan130governmentalorganizationstodeterminehowleadersstaffandexternalstakeholdersmanageandusedata
Localgovernmentsareincreasinglyonthefrontlinesofaddressinglargescaleproblemswithglobalimplicationssuchasclimatechangepublicsafetyandeconomicinequality
Whileevidence-basedinterventionstohelpsolvethoseproblemsareavailablelocalgovernmentstaffhavelimitedtimeandcapacitytofindandimplementresearch-basedsolutionsAsaresultcitiescontinuetoworkthewaytheyalwayshave--makingincrementalimprovementstoprogramsbutnotalwayscreatingchangeatasystem-level
ThroughresourcedevelopmentandcollaborationwithJHUfacultyGovExissupportingcityleadersinusingdataandevidencetomakedecisionsandcreatinglastingrelationshipswithresearchers
TheRoadmapwasdevelopedbyGovExaspartofitspartnershipwithBloombergPhilanthropiesWhatWorksCitiesinitiative
AboutGovEx
69
- Title page
- Introduction and purpose of the Roadmap
- Using the Roadmap
-
- Gather information about the problem
- Strategically engage key partners
- Review the evidence and collect details
- Decide on a policy to implement based on local factors
- Establish a framework to monitor progress
- Assess data accessibility quality and security
- Design implementation and evaluation
- Share your story and success
-
- Working with a range of evidence
- Thank you
- References
- Resources
- About GovEx
-
issuesactorstrendsandresourcesthatisultimatelydifferentfromeveryotherplaceintheworld
ThisisimportantbecauseaspolicypractitionersandnotscientistsyoucaremoreaboutyourspecificcasethanahypotheticalaveragegeneralcaseYoucarelessaboutthetotallikelyrangeofimpactsfromapolicythanhowthissingleimplementationwillworkYouwantinformationaboutpoliciesthatwillprovideyouwiththebestpossiblepredictionabouthowtheywillworkinyourcontext
ByputtingdifferentkindsofevidenceintothesameframeworkNestarsquosStandardsofEvidenceimplythatwhenyoucannotgetrigorouslytestedexperimentallyverifiedinformationitisappropriatetoworkwithwhatwedohaveThisisrelevantforthemunicipalimplementationcontextbecausenostudypulledfromaliteraturereviewwillaccountforallofthevariablesyoufaceinyourowncontext
Intheabsenceofhavingastudythatpredictshowaninterventionworksinyourspecificcontextyouwillneednon-scientificinformationtotrytofillinthosegapsYouwillneedtogatherstoriesfromresidentsstaffmembersandotherstakeholdersthatsitonLevels1or2ofthestandardsofevidence
Someofthekindsofinformationyouwillprobablygetmoreeasilyfrompeoplethanstudiesinclude
howpeopleincitieslikeyoursreactedtoaninterventioninpublicforumshowpeoplewhoimplementedaninterventioninacitylikeyoursfiguredouthowtogetitpassedhowthecitycommunicatedpubliclyabouttheapparentbenefitsanddrawbacksoftheinterventionthepoliticalacceptabilityoftheintervention
Youcanremainsimultaneouslyawarethattheinformationyouaregatheringthroughindividualstoriesisbiased--notgeneralizableinformedbyspecificperspectives--whilealsorecognizingthathavingsomeinformationaboutsomethingimportantisbetterthannoinformationEvidencefromthefirsttwoevidencelevelsareessentialbuildingblocksofknowledgeandperfectlyusefulwhenyoucannotgetdetailedunbiasedknowledge
Youhavetostartsomewhereandyoushouldalwaysstartwithevidenceyouhaveaboutyourveryspecificconditionsthatyouknowareimportantforsuccess
Workingwitharangeofevidence
58
Thankyou
Interviewswithcities
GovExheldsemi-structuredinterviewswithsixstaffmembersinfourcitiestoestablishtheinitialstepsoftheRoadmapWeaskedrespondentstotalkaboutaninitiativeorpolicythattheyhavebeenresponsibleforimplementingintheircityRespondentsfromthreecitiesspokeabouttheireffortstoaddresshomelessnessandrespondentsfromonecityspokeaboutblightAfterrespondentsidentifiedtheproblemtheyspokeabouthowthecityprioritizedthisissuethevariousstepsinvolvedandbarrierspresent
GovExconductedtextanalysisoftheseinterviewswhichrevealedthatpeoplewasthemostcommonlyusedtermduringtheseconversationswhendiscussingthestepstochangeaprogramorpolicyThisisathemethroughoutthisRoadmapThefirstsectionbeginswithidentifyingtherelevantpeopleTheconversationsaroundevidencearefocusedonhumanbehaviorratherthanmethodologyorstrengthofevidence
Thankyou
59
OncewehadadraftoftheRoadmapwefollowedupwithcitiesonanindividualbasistolearnmoreaboutwhatwouldbehelpfulWearegratefultothefollowingcitystafffortheinterviewsandindividualconversations
MichaelBaskinChiefPolicyOfficerChattanoogaTN
MelissaBridgesPerformanceandInnovationCoordinatorLittleRockAR
VictoriaCarreoacutenAdministrativeOfficerLasVegasNV
TyroneGrandstrandCommunityDevelopmentPlannerFargoND
TibLaughlinDirectorofGeneralServicesKansasCityKS
TimMorelandDirectorofPerformanceManagementandOpenDataChattanoogaTN
CharlesRosendahlManagementAnalystDentonTX
Thankyou
60
MelissaSiebenAssistantCountyAdministratorKansasCityKS
GinaThompsonDirectorofStrategicServicesLewisvilleTX
JHUFacultycollaboration
GovExalsoconvenedthecitycontactsandfacultypartnersinaseriesofAdvisoryGroupmeetingsThesemeetingsservedtoconnectcitiesandfacultytounderstandoneanothersperspective
StefanBaralBloombergSchoolofPublicHealth
AshleyBernerJHUSchoolofEducation
MelissaDavey-RothwellBloombergSchoolofPublicHealth
JonLinksBloombergSchoolofPublicHealth
TamarMendelsonBloombergSchoolofPublicHealth
KristinMmariBloombergSchoolofPublicHealth
KeshiaPollackPorterBloombergSchoolofPublicHealth
JoshSharfsteinBloombergSchoolofPublicHealth
LizStuartBloombergSchoolofPublicHealth
LindsayThompsonCareyBusinessSchool
DanielWebsterBloombergSchoolofPublicHealth
Partnersindataandevidence-basedpolicy
Wealsospokewithpartnerswhoareexpertsinthefieldofevidence-basedpolicywhosharedtheirknowledgeandresourcesofcurrentandfuturethinkinginevidence-informedthinking
WendyBarrenoAssociateDirectorWilsonSheehanLabforEconomicOpportunitiesUniversityofNotreDame
TiffanyDavenportDirectorofDataPartnershipsBaltimoresPromise
TomiHiersDirectorBaltimoreCivicSiteAnnieECaseyFoundation
ChrisKingsleySeniorAssociateAnnieECaseyFoundation
KathrynOliverAssociateProfessorLondonSchoolofHygieneandTropicalMedicine
Thankyou
61
KristenWatkinsSeniorPolicyAssociateJ-PAL
GovExParticipants
GovExAnalystMiriamMcKinneyanalyzedtheinterviewtranscriptsandconductedtextanalysisSeniorImplementationAdvisorEmilyShawwillintegrateRoadmapactivitieswithadditionalservicedeliveryactivitiesatGovEx
Thankyou
62
ReferencesAbrahamKGampHaskinsR(2017)Thepromiseofevidence-basedpolicymakingReportofthecommissiononevidence-basedpolicymakingWashingtonDCCommissiononEvidence-basedPolicymakingBigmove2Extendingthegreen(nd)Retrievedfromhttpswwwcityoflewisvillecomdoing-businesslewisville-2025big-move-2-extending-the-greenBatesMAampGlennersterR(2017)ThegeneralizabilitypuzzleRetrievedfromhttpsssirorgarticlesentrythe_generalizability_puzzleBuildwithnotfor(2018January)RetrievedfromhttpsbuildwithnotforwordpresscomCairneyPampOliverK(2017)Evidence-basedpolicymakingisnotlikeevidence-basedmedicinesohowfarshouldyougotobridgethedividebetweenevidenceandpolicyHealthResearchPolicyandSystemsdoi101186s12961-017-0192-xCairneyPOliverKampWellsteadA(2016)TobridgethedividebetweenevidenceandpolicyReduceambiguityasmuchasuncertaintyPublicAdministrationReview76(3)399-402Retrievedfromhttpdxdoiorg101111puar12555Centreforpublicimpact(nd)RetrievedfromhttpswwwcentreforpublicimpactorgCernaL(2013)ThenatureofpolicychangeandimplementationAreviewofdifferenttheoreticalapproachesOrganisationforEconomicCo-OperationandDevelopmentRetrievedfromhttpswwwoecdorgeduceriTheNatureofPolicyChangeandImplementationpdfColoradoSprings(nd)Retrievedfromhttpscoloradospringsgovcommunicationspagewhat-expect-when-you-speakupContinuumofcare(coc)program(nd)RetrievedfromhttpswwwhudexchangeinfoprogramscocCouncilpresidentkicksoff$12millionyouthfund(nd)Retrievedfromhttpwwwbaltimorecitycouncilcomcontentcouncil-president-kicks-12-million-youth-fundDurlakJAampDuPreEP(2008)ImplementationmattersAreviewofresearchontheinfluenceofimplementationonprogramoutcomesandthefactorsaffectingimplementationAmericanJournalofCommunityPsychology41327-350Retrievedfromhttpswwwncbinlmnihgovpubmed18322790Fishbonediagram(ishikawa)(nd)Retrievedfromhttpasqorglearn-about-qualitycause-analysis-toolsoverviewfishbonehtmlFivewhysGettingtotherootofaproblemquickly(nd)RetrievedfromhttpswwwmindtoolscompagesarticlenewTMC_5Whtm
References
63
FligsteinNampMcAdamD(2011)TowardageneraltheoryofstrategicactionfieldsSociologicalTheory29(1)1-26RetrievedfromhttpsociologyberkeleyedusitesdefaultfilesfacultyfligsteinFligstein20McAdam20Soc20Theory20PaperpdHeinkelP(2018January12)CitymullspermitsforbikesharingcompaniesRetrievedfromhttpwwwdentonrccomnewsdenton20180112city-mulls-permits-bike-sharing-companiesHowisareamedianincomecalculated(2012October)Retrievedfromhttpswwwhudexchangeinfofaqs843how-is-area-median-income-calculatedLampkinLMampHatryHP(2003)KeystepsinoutcomemanagementUrbanInstituteRetrievedfromhttpswwwurbanorgresearchpublicationkey-steps-outcome-managementMarshDampSmithM(2000)UnderstandingpolicynetworksTowardsadialecticalapproachPoliticalStudies484-21Retrievedfromhttpjournalssagepubcomdoiabs1011111467-924800247McCannL(nd)RetrievedfromhttplaurenellencomMoultonSampSandfortJR(2016)ThestrategicactionfieldframeworkforpolicyimplementationresearchPolicyStudiesJournal45(1)Retrievedfromhttponlinelibrarywileycomstore101111psj12147assetpsj12147pdfv=1ampt=jdkhhcp4amps=e184c625e1855a6b1615ccb99ee27b1c0b3212d8PaynterB(2018January1)What100citiesarelearningfromeachotherbysharingtheirdataRetrievedfromhttpswwwfastcompanycom40518004what-100-cities-are-learning-from-each-other-by-sharing-their-dataPeakacademy(nd)Retrievedfromhttpswwwdenvergovorgcontentdenvergovenmayors-officeprograms-initiativespeak-performancepeak-academyhtmlPinnockHBarwickMCarpenterCREldridgeSGrandesGGriffithsCJRycroft-MaloneJMeissnerPMurrayEPatelASheikhAampTaylorSJC(2017)Standardsforreportingimplementationstudies(stari)ExplanationandelaborationdocumentBMJOpenRetrievedfromhttpbmjopenbmjcomcontentbmjopen74e013318fullpdfProctorESilmereHRaghavanRHovmandPAaronsGBungerAGriffeyRampHensleyM(2011)OutcomesforimplementationresearchConceptualdistinctionsmeasurementchallengesandresearchagendaAdministrationandPolicyinMentalHealth38(2)65-76doi101007s10488-010-0319-7TandonSD(2018)ProgressincommunityhealthpartnershipsResearcheducationandactionRetrievedfromhttpswwwpressjhuedujournalsprogress-community-health-partnerships-research-education-and-actionProvanKGampKenisP(2007)ModesofnetworkgovernanceStructuremanagementandeffectivenessJournalofPublicAdministrationResearchandTheory
References
64
18(2)229-252Retrievedfromhttpsacademicoupcomjpartarticle182229935895SabatierPA(1988)Anadvocacycoalitionframeworkofpolicychangeandtheroleofpolicy-orientedlearningthereinPolicySciences21(3)129-168Retrievedfromhttpwwwjstororgstable4532139SandstromAampCarlssonL(2008)TheperformanceofpolicynetworksTherelationbetweennetworkstructureandnetworkperformancePolicyStudiesJournal36497-524Retrievedfromhttponlinelibrarywileycomdoi101111j1541-0072200800281xabstractSchwartzMampWilsonE(nd)WhocanaffordtoliveinahomeAlookatdatafromthe2006americancommunitysurveyRetrievedfromhttpswwwcensusgovhousingcensuspublicationswho-can-affordpdfShearerJCAbelsonAKouyateBLavisJNampWaltG(2016)WhydopolicieschangeInstitutionsinterestsideasandnetworksinthreecasesofpolicyreformHealthPolicyandPlanning31(9)1200-1211Retrievedfromhttpsdoiorg101093heapolczw052ThelessonsofadministrativedataHigh-profilepolicy-relevantresearchpoweredbyadministrativedata(nd)Retrievedfromhttpswwwpovertyactionlaborgsitesdefaultfilesresources20161114-The-Lessons-of-Administrative-DatapdfToolsandotherneedsforqualitativeandquantitativemethods(2015February)AgencyforHealthcareResearchandQualityRetrievedfromhttpwwwahrqgovprofessionalsquality-patient-safetytalkingqualityassesstoolshtml
WeaverRCampBagchi-SenS(2013)SpatialanalysisofurbandeclineThegeographyofblightAppliedGeography4061-70
References
65
UniversityPartners
Name Title Affiliation Area(s)ofexpertise
Contactinformation
Examplesofworkingwithnon-academicdecisionmakers
WendyPBarreno
AssociateDirector
WilsonSheehanLabforEconomicOpportunitiesUniversityofNotreDame
LEOsfocusareasincludehealtheducationhousingchildwelfareeconomicopportunitycriminaljustice
wbarrenonduduorleondedu
MelissaDavey-RothwellPhD
AssociateScientist
JohnsHopkinsBloombergSchoolofPublicHealth
Health mdavey1jhuedu410-614-5854
BaltimoreCityHealthDepartmentLeadersMarylandDepartmentofHealthAddictionRecoveryServicesinDCExecutiveDirectorsofcommunityorganizationsacrosstheUSthatfocusonHIVSTIsubstanceabuseandimplementationscience
ElizabethStuart
AssociateDeanforEducationandProfessorDepartmentsofMentalHealthBiostatisticsandHealthPolicyand
JohnsHopkinsBloombergSchoolofPublicHealth
Educationsocialpolicystatisticalmethods
estuartjhuedu
Resources
66
Management
Design-ThinkingResources
EngagementGuideforCivicTech
Journeymap
PerformanceManagementResources
DenverPeakAcademy
Proxymeasuresguide
Talkingaboutperformancewithoutcomplicatingit(video)
SettingPerformanceTargetsguide
Datavisualizationguide
Performancemanagementgettingstartedguide
Exercises
Fishboneexercise
5Whys
AdministrativeData
TheLessonsofAdministrativeData(JPAL)
EvaluationResources
Quantitativeandqualitativemethods(USDepartmentofHealthandHumanServicesAgencyforHealthcareResearchandQuality)
KeyStepsinOutcomeManagement(UrbanInstitute)
RCT-YES
Sageresearchmethods
BestPracticesforMixedMethodsResearchintheHealthSciences
ACCORDChecklist
Pilotandimplementationplanning
Benchmarkingexamplesandproxymeasures
Seattleopendataplaybook
Resources
67
Datasciencescopingguide
Resources
68
AboutGovExGovExhelpsgovernmentsusedataandevidencetoimproveresidentsrsquolivesWesupportgovernmentsinbuildingtheirinternalcapacitytooperatesustainableanalyticspracticeswhichimprovecommunityoutcomesinfieldsfrompublicsafetytoeconomicdevelopmentGovExhasaproventrackrecordofworkinginconcertwithpartnerstoprovidetechnicalassistanceandtrainingatscale
SinceourlaunchinApril2015GovExhas
Trainedover2000governmentemployeesinpersonandonlineintopicsincludingdatamanagementperformancemanagementopendatacommunityengagementandanalytics
Providedtechnicalassistancetomorethan100localgovernmentsacrosstheglobeimprovingtheiruseofdatatoachieveefficienciesinservicedeliveryandbetteroutcomesforresidents
Assessedthecapacityofmorethan130governmentalorganizationstodeterminehowleadersstaffandexternalstakeholdersmanageandusedata
Localgovernmentsareincreasinglyonthefrontlinesofaddressinglargescaleproblemswithglobalimplicationssuchasclimatechangepublicsafetyandeconomicinequality
Whileevidence-basedinterventionstohelpsolvethoseproblemsareavailablelocalgovernmentstaffhavelimitedtimeandcapacitytofindandimplementresearch-basedsolutionsAsaresultcitiescontinuetoworkthewaytheyalwayshave--makingincrementalimprovementstoprogramsbutnotalwayscreatingchangeatasystem-level
ThroughresourcedevelopmentandcollaborationwithJHUfacultyGovExissupportingcityleadersinusingdataandevidencetomakedecisionsandcreatinglastingrelationshipswithresearchers
TheRoadmapwasdevelopedbyGovExaspartofitspartnershipwithBloombergPhilanthropiesWhatWorksCitiesinitiative
AboutGovEx
69
- Title page
- Introduction and purpose of the Roadmap
- Using the Roadmap
-
- Gather information about the problem
- Strategically engage key partners
- Review the evidence and collect details
- Decide on a policy to implement based on local factors
- Establish a framework to monitor progress
- Assess data accessibility quality and security
- Design implementation and evaluation
- Share your story and success
-
- Working with a range of evidence
- Thank you
- References
- Resources
- About GovEx
-
Thankyou
Interviewswithcities
GovExheldsemi-structuredinterviewswithsixstaffmembersinfourcitiestoestablishtheinitialstepsoftheRoadmapWeaskedrespondentstotalkaboutaninitiativeorpolicythattheyhavebeenresponsibleforimplementingintheircityRespondentsfromthreecitiesspokeabouttheireffortstoaddresshomelessnessandrespondentsfromonecityspokeaboutblightAfterrespondentsidentifiedtheproblemtheyspokeabouthowthecityprioritizedthisissuethevariousstepsinvolvedandbarrierspresent
GovExconductedtextanalysisoftheseinterviewswhichrevealedthatpeoplewasthemostcommonlyusedtermduringtheseconversationswhendiscussingthestepstochangeaprogramorpolicyThisisathemethroughoutthisRoadmapThefirstsectionbeginswithidentifyingtherelevantpeopleTheconversationsaroundevidencearefocusedonhumanbehaviorratherthanmethodologyorstrengthofevidence
Thankyou
59
OncewehadadraftoftheRoadmapwefollowedupwithcitiesonanindividualbasistolearnmoreaboutwhatwouldbehelpfulWearegratefultothefollowingcitystafffortheinterviewsandindividualconversations
MichaelBaskinChiefPolicyOfficerChattanoogaTN
MelissaBridgesPerformanceandInnovationCoordinatorLittleRockAR
VictoriaCarreoacutenAdministrativeOfficerLasVegasNV
TyroneGrandstrandCommunityDevelopmentPlannerFargoND
TibLaughlinDirectorofGeneralServicesKansasCityKS
TimMorelandDirectorofPerformanceManagementandOpenDataChattanoogaTN
CharlesRosendahlManagementAnalystDentonTX
Thankyou
60
MelissaSiebenAssistantCountyAdministratorKansasCityKS
GinaThompsonDirectorofStrategicServicesLewisvilleTX
JHUFacultycollaboration
GovExalsoconvenedthecitycontactsandfacultypartnersinaseriesofAdvisoryGroupmeetingsThesemeetingsservedtoconnectcitiesandfacultytounderstandoneanothersperspective
StefanBaralBloombergSchoolofPublicHealth
AshleyBernerJHUSchoolofEducation
MelissaDavey-RothwellBloombergSchoolofPublicHealth
JonLinksBloombergSchoolofPublicHealth
TamarMendelsonBloombergSchoolofPublicHealth
KristinMmariBloombergSchoolofPublicHealth
KeshiaPollackPorterBloombergSchoolofPublicHealth
JoshSharfsteinBloombergSchoolofPublicHealth
LizStuartBloombergSchoolofPublicHealth
LindsayThompsonCareyBusinessSchool
DanielWebsterBloombergSchoolofPublicHealth
Partnersindataandevidence-basedpolicy
Wealsospokewithpartnerswhoareexpertsinthefieldofevidence-basedpolicywhosharedtheirknowledgeandresourcesofcurrentandfuturethinkinginevidence-informedthinking
WendyBarrenoAssociateDirectorWilsonSheehanLabforEconomicOpportunitiesUniversityofNotreDame
TiffanyDavenportDirectorofDataPartnershipsBaltimoresPromise
TomiHiersDirectorBaltimoreCivicSiteAnnieECaseyFoundation
ChrisKingsleySeniorAssociateAnnieECaseyFoundation
KathrynOliverAssociateProfessorLondonSchoolofHygieneandTropicalMedicine
Thankyou
61
KristenWatkinsSeniorPolicyAssociateJ-PAL
GovExParticipants
GovExAnalystMiriamMcKinneyanalyzedtheinterviewtranscriptsandconductedtextanalysisSeniorImplementationAdvisorEmilyShawwillintegrateRoadmapactivitieswithadditionalservicedeliveryactivitiesatGovEx
Thankyou
62
ReferencesAbrahamKGampHaskinsR(2017)Thepromiseofevidence-basedpolicymakingReportofthecommissiononevidence-basedpolicymakingWashingtonDCCommissiononEvidence-basedPolicymakingBigmove2Extendingthegreen(nd)Retrievedfromhttpswwwcityoflewisvillecomdoing-businesslewisville-2025big-move-2-extending-the-greenBatesMAampGlennersterR(2017)ThegeneralizabilitypuzzleRetrievedfromhttpsssirorgarticlesentrythe_generalizability_puzzleBuildwithnotfor(2018January)RetrievedfromhttpsbuildwithnotforwordpresscomCairneyPampOliverK(2017)Evidence-basedpolicymakingisnotlikeevidence-basedmedicinesohowfarshouldyougotobridgethedividebetweenevidenceandpolicyHealthResearchPolicyandSystemsdoi101186s12961-017-0192-xCairneyPOliverKampWellsteadA(2016)TobridgethedividebetweenevidenceandpolicyReduceambiguityasmuchasuncertaintyPublicAdministrationReview76(3)399-402Retrievedfromhttpdxdoiorg101111puar12555Centreforpublicimpact(nd)RetrievedfromhttpswwwcentreforpublicimpactorgCernaL(2013)ThenatureofpolicychangeandimplementationAreviewofdifferenttheoreticalapproachesOrganisationforEconomicCo-OperationandDevelopmentRetrievedfromhttpswwwoecdorgeduceriTheNatureofPolicyChangeandImplementationpdfColoradoSprings(nd)Retrievedfromhttpscoloradospringsgovcommunicationspagewhat-expect-when-you-speakupContinuumofcare(coc)program(nd)RetrievedfromhttpswwwhudexchangeinfoprogramscocCouncilpresidentkicksoff$12millionyouthfund(nd)Retrievedfromhttpwwwbaltimorecitycouncilcomcontentcouncil-president-kicks-12-million-youth-fundDurlakJAampDuPreEP(2008)ImplementationmattersAreviewofresearchontheinfluenceofimplementationonprogramoutcomesandthefactorsaffectingimplementationAmericanJournalofCommunityPsychology41327-350Retrievedfromhttpswwwncbinlmnihgovpubmed18322790Fishbonediagram(ishikawa)(nd)Retrievedfromhttpasqorglearn-about-qualitycause-analysis-toolsoverviewfishbonehtmlFivewhysGettingtotherootofaproblemquickly(nd)RetrievedfromhttpswwwmindtoolscompagesarticlenewTMC_5Whtm
References
63
FligsteinNampMcAdamD(2011)TowardageneraltheoryofstrategicactionfieldsSociologicalTheory29(1)1-26RetrievedfromhttpsociologyberkeleyedusitesdefaultfilesfacultyfligsteinFligstein20McAdam20Soc20Theory20PaperpdHeinkelP(2018January12)CitymullspermitsforbikesharingcompaniesRetrievedfromhttpwwwdentonrccomnewsdenton20180112city-mulls-permits-bike-sharing-companiesHowisareamedianincomecalculated(2012October)Retrievedfromhttpswwwhudexchangeinfofaqs843how-is-area-median-income-calculatedLampkinLMampHatryHP(2003)KeystepsinoutcomemanagementUrbanInstituteRetrievedfromhttpswwwurbanorgresearchpublicationkey-steps-outcome-managementMarshDampSmithM(2000)UnderstandingpolicynetworksTowardsadialecticalapproachPoliticalStudies484-21Retrievedfromhttpjournalssagepubcomdoiabs1011111467-924800247McCannL(nd)RetrievedfromhttplaurenellencomMoultonSampSandfortJR(2016)ThestrategicactionfieldframeworkforpolicyimplementationresearchPolicyStudiesJournal45(1)Retrievedfromhttponlinelibrarywileycomstore101111psj12147assetpsj12147pdfv=1ampt=jdkhhcp4amps=e184c625e1855a6b1615ccb99ee27b1c0b3212d8PaynterB(2018January1)What100citiesarelearningfromeachotherbysharingtheirdataRetrievedfromhttpswwwfastcompanycom40518004what-100-cities-are-learning-from-each-other-by-sharing-their-dataPeakacademy(nd)Retrievedfromhttpswwwdenvergovorgcontentdenvergovenmayors-officeprograms-initiativespeak-performancepeak-academyhtmlPinnockHBarwickMCarpenterCREldridgeSGrandesGGriffithsCJRycroft-MaloneJMeissnerPMurrayEPatelASheikhAampTaylorSJC(2017)Standardsforreportingimplementationstudies(stari)ExplanationandelaborationdocumentBMJOpenRetrievedfromhttpbmjopenbmjcomcontentbmjopen74e013318fullpdfProctorESilmereHRaghavanRHovmandPAaronsGBungerAGriffeyRampHensleyM(2011)OutcomesforimplementationresearchConceptualdistinctionsmeasurementchallengesandresearchagendaAdministrationandPolicyinMentalHealth38(2)65-76doi101007s10488-010-0319-7TandonSD(2018)ProgressincommunityhealthpartnershipsResearcheducationandactionRetrievedfromhttpswwwpressjhuedujournalsprogress-community-health-partnerships-research-education-and-actionProvanKGampKenisP(2007)ModesofnetworkgovernanceStructuremanagementandeffectivenessJournalofPublicAdministrationResearchandTheory
References
64
18(2)229-252Retrievedfromhttpsacademicoupcomjpartarticle182229935895SabatierPA(1988)Anadvocacycoalitionframeworkofpolicychangeandtheroleofpolicy-orientedlearningthereinPolicySciences21(3)129-168Retrievedfromhttpwwwjstororgstable4532139SandstromAampCarlssonL(2008)TheperformanceofpolicynetworksTherelationbetweennetworkstructureandnetworkperformancePolicyStudiesJournal36497-524Retrievedfromhttponlinelibrarywileycomdoi101111j1541-0072200800281xabstractSchwartzMampWilsonE(nd)WhocanaffordtoliveinahomeAlookatdatafromthe2006americancommunitysurveyRetrievedfromhttpswwwcensusgovhousingcensuspublicationswho-can-affordpdfShearerJCAbelsonAKouyateBLavisJNampWaltG(2016)WhydopolicieschangeInstitutionsinterestsideasandnetworksinthreecasesofpolicyreformHealthPolicyandPlanning31(9)1200-1211Retrievedfromhttpsdoiorg101093heapolczw052ThelessonsofadministrativedataHigh-profilepolicy-relevantresearchpoweredbyadministrativedata(nd)Retrievedfromhttpswwwpovertyactionlaborgsitesdefaultfilesresources20161114-The-Lessons-of-Administrative-DatapdfToolsandotherneedsforqualitativeandquantitativemethods(2015February)AgencyforHealthcareResearchandQualityRetrievedfromhttpwwwahrqgovprofessionalsquality-patient-safetytalkingqualityassesstoolshtml
WeaverRCampBagchi-SenS(2013)SpatialanalysisofurbandeclineThegeographyofblightAppliedGeography4061-70
References
65
UniversityPartners
Name Title Affiliation Area(s)ofexpertise
Contactinformation
Examplesofworkingwithnon-academicdecisionmakers
WendyPBarreno
AssociateDirector
WilsonSheehanLabforEconomicOpportunitiesUniversityofNotreDame
LEOsfocusareasincludehealtheducationhousingchildwelfareeconomicopportunitycriminaljustice
wbarrenonduduorleondedu
MelissaDavey-RothwellPhD
AssociateScientist
JohnsHopkinsBloombergSchoolofPublicHealth
Health mdavey1jhuedu410-614-5854
BaltimoreCityHealthDepartmentLeadersMarylandDepartmentofHealthAddictionRecoveryServicesinDCExecutiveDirectorsofcommunityorganizationsacrosstheUSthatfocusonHIVSTIsubstanceabuseandimplementationscience
ElizabethStuart
AssociateDeanforEducationandProfessorDepartmentsofMentalHealthBiostatisticsandHealthPolicyand
JohnsHopkinsBloombergSchoolofPublicHealth
Educationsocialpolicystatisticalmethods
estuartjhuedu
Resources
66
Management
Design-ThinkingResources
EngagementGuideforCivicTech
Journeymap
PerformanceManagementResources
DenverPeakAcademy
Proxymeasuresguide
Talkingaboutperformancewithoutcomplicatingit(video)
SettingPerformanceTargetsguide
Datavisualizationguide
Performancemanagementgettingstartedguide
Exercises
Fishboneexercise
5Whys
AdministrativeData
TheLessonsofAdministrativeData(JPAL)
EvaluationResources
Quantitativeandqualitativemethods(USDepartmentofHealthandHumanServicesAgencyforHealthcareResearchandQuality)
KeyStepsinOutcomeManagement(UrbanInstitute)
RCT-YES
Sageresearchmethods
BestPracticesforMixedMethodsResearchintheHealthSciences
ACCORDChecklist
Pilotandimplementationplanning
Benchmarkingexamplesandproxymeasures
Seattleopendataplaybook
Resources
67
Datasciencescopingguide
Resources
68
AboutGovExGovExhelpsgovernmentsusedataandevidencetoimproveresidentsrsquolivesWesupportgovernmentsinbuildingtheirinternalcapacitytooperatesustainableanalyticspracticeswhichimprovecommunityoutcomesinfieldsfrompublicsafetytoeconomicdevelopmentGovExhasaproventrackrecordofworkinginconcertwithpartnerstoprovidetechnicalassistanceandtrainingatscale
SinceourlaunchinApril2015GovExhas
Trainedover2000governmentemployeesinpersonandonlineintopicsincludingdatamanagementperformancemanagementopendatacommunityengagementandanalytics
Providedtechnicalassistancetomorethan100localgovernmentsacrosstheglobeimprovingtheiruseofdatatoachieveefficienciesinservicedeliveryandbetteroutcomesforresidents
Assessedthecapacityofmorethan130governmentalorganizationstodeterminehowleadersstaffandexternalstakeholdersmanageandusedata
Localgovernmentsareincreasinglyonthefrontlinesofaddressinglargescaleproblemswithglobalimplicationssuchasclimatechangepublicsafetyandeconomicinequality
Whileevidence-basedinterventionstohelpsolvethoseproblemsareavailablelocalgovernmentstaffhavelimitedtimeandcapacitytofindandimplementresearch-basedsolutionsAsaresultcitiescontinuetoworkthewaytheyalwayshave--makingincrementalimprovementstoprogramsbutnotalwayscreatingchangeatasystem-level
ThroughresourcedevelopmentandcollaborationwithJHUfacultyGovExissupportingcityleadersinusingdataandevidencetomakedecisionsandcreatinglastingrelationshipswithresearchers
TheRoadmapwasdevelopedbyGovExaspartofitspartnershipwithBloombergPhilanthropiesWhatWorksCitiesinitiative
AboutGovEx
69
- Title page
- Introduction and purpose of the Roadmap
- Using the Roadmap
-
- Gather information about the problem
- Strategically engage key partners
- Review the evidence and collect details
- Decide on a policy to implement based on local factors
- Establish a framework to monitor progress
- Assess data accessibility quality and security
- Design implementation and evaluation
- Share your story and success
-
- Working with a range of evidence
- Thank you
- References
- Resources
- About GovEx
-
OncewehadadraftoftheRoadmapwefollowedupwithcitiesonanindividualbasistolearnmoreaboutwhatwouldbehelpfulWearegratefultothefollowingcitystafffortheinterviewsandindividualconversations
MichaelBaskinChiefPolicyOfficerChattanoogaTN
MelissaBridgesPerformanceandInnovationCoordinatorLittleRockAR
VictoriaCarreoacutenAdministrativeOfficerLasVegasNV
TyroneGrandstrandCommunityDevelopmentPlannerFargoND
TibLaughlinDirectorofGeneralServicesKansasCityKS
TimMorelandDirectorofPerformanceManagementandOpenDataChattanoogaTN
CharlesRosendahlManagementAnalystDentonTX
Thankyou
60
MelissaSiebenAssistantCountyAdministratorKansasCityKS
GinaThompsonDirectorofStrategicServicesLewisvilleTX
JHUFacultycollaboration
GovExalsoconvenedthecitycontactsandfacultypartnersinaseriesofAdvisoryGroupmeetingsThesemeetingsservedtoconnectcitiesandfacultytounderstandoneanothersperspective
StefanBaralBloombergSchoolofPublicHealth
AshleyBernerJHUSchoolofEducation
MelissaDavey-RothwellBloombergSchoolofPublicHealth
JonLinksBloombergSchoolofPublicHealth
TamarMendelsonBloombergSchoolofPublicHealth
KristinMmariBloombergSchoolofPublicHealth
KeshiaPollackPorterBloombergSchoolofPublicHealth
JoshSharfsteinBloombergSchoolofPublicHealth
LizStuartBloombergSchoolofPublicHealth
LindsayThompsonCareyBusinessSchool
DanielWebsterBloombergSchoolofPublicHealth
Partnersindataandevidence-basedpolicy
Wealsospokewithpartnerswhoareexpertsinthefieldofevidence-basedpolicywhosharedtheirknowledgeandresourcesofcurrentandfuturethinkinginevidence-informedthinking
WendyBarrenoAssociateDirectorWilsonSheehanLabforEconomicOpportunitiesUniversityofNotreDame
TiffanyDavenportDirectorofDataPartnershipsBaltimoresPromise
TomiHiersDirectorBaltimoreCivicSiteAnnieECaseyFoundation
ChrisKingsleySeniorAssociateAnnieECaseyFoundation
KathrynOliverAssociateProfessorLondonSchoolofHygieneandTropicalMedicine
Thankyou
61
KristenWatkinsSeniorPolicyAssociateJ-PAL
GovExParticipants
GovExAnalystMiriamMcKinneyanalyzedtheinterviewtranscriptsandconductedtextanalysisSeniorImplementationAdvisorEmilyShawwillintegrateRoadmapactivitieswithadditionalservicedeliveryactivitiesatGovEx
Thankyou
62
ReferencesAbrahamKGampHaskinsR(2017)Thepromiseofevidence-basedpolicymakingReportofthecommissiononevidence-basedpolicymakingWashingtonDCCommissiononEvidence-basedPolicymakingBigmove2Extendingthegreen(nd)Retrievedfromhttpswwwcityoflewisvillecomdoing-businesslewisville-2025big-move-2-extending-the-greenBatesMAampGlennersterR(2017)ThegeneralizabilitypuzzleRetrievedfromhttpsssirorgarticlesentrythe_generalizability_puzzleBuildwithnotfor(2018January)RetrievedfromhttpsbuildwithnotforwordpresscomCairneyPampOliverK(2017)Evidence-basedpolicymakingisnotlikeevidence-basedmedicinesohowfarshouldyougotobridgethedividebetweenevidenceandpolicyHealthResearchPolicyandSystemsdoi101186s12961-017-0192-xCairneyPOliverKampWellsteadA(2016)TobridgethedividebetweenevidenceandpolicyReduceambiguityasmuchasuncertaintyPublicAdministrationReview76(3)399-402Retrievedfromhttpdxdoiorg101111puar12555Centreforpublicimpact(nd)RetrievedfromhttpswwwcentreforpublicimpactorgCernaL(2013)ThenatureofpolicychangeandimplementationAreviewofdifferenttheoreticalapproachesOrganisationforEconomicCo-OperationandDevelopmentRetrievedfromhttpswwwoecdorgeduceriTheNatureofPolicyChangeandImplementationpdfColoradoSprings(nd)Retrievedfromhttpscoloradospringsgovcommunicationspagewhat-expect-when-you-speakupContinuumofcare(coc)program(nd)RetrievedfromhttpswwwhudexchangeinfoprogramscocCouncilpresidentkicksoff$12millionyouthfund(nd)Retrievedfromhttpwwwbaltimorecitycouncilcomcontentcouncil-president-kicks-12-million-youth-fundDurlakJAampDuPreEP(2008)ImplementationmattersAreviewofresearchontheinfluenceofimplementationonprogramoutcomesandthefactorsaffectingimplementationAmericanJournalofCommunityPsychology41327-350Retrievedfromhttpswwwncbinlmnihgovpubmed18322790Fishbonediagram(ishikawa)(nd)Retrievedfromhttpasqorglearn-about-qualitycause-analysis-toolsoverviewfishbonehtmlFivewhysGettingtotherootofaproblemquickly(nd)RetrievedfromhttpswwwmindtoolscompagesarticlenewTMC_5Whtm
References
63
FligsteinNampMcAdamD(2011)TowardageneraltheoryofstrategicactionfieldsSociologicalTheory29(1)1-26RetrievedfromhttpsociologyberkeleyedusitesdefaultfilesfacultyfligsteinFligstein20McAdam20Soc20Theory20PaperpdHeinkelP(2018January12)CitymullspermitsforbikesharingcompaniesRetrievedfromhttpwwwdentonrccomnewsdenton20180112city-mulls-permits-bike-sharing-companiesHowisareamedianincomecalculated(2012October)Retrievedfromhttpswwwhudexchangeinfofaqs843how-is-area-median-income-calculatedLampkinLMampHatryHP(2003)KeystepsinoutcomemanagementUrbanInstituteRetrievedfromhttpswwwurbanorgresearchpublicationkey-steps-outcome-managementMarshDampSmithM(2000)UnderstandingpolicynetworksTowardsadialecticalapproachPoliticalStudies484-21Retrievedfromhttpjournalssagepubcomdoiabs1011111467-924800247McCannL(nd)RetrievedfromhttplaurenellencomMoultonSampSandfortJR(2016)ThestrategicactionfieldframeworkforpolicyimplementationresearchPolicyStudiesJournal45(1)Retrievedfromhttponlinelibrarywileycomstore101111psj12147assetpsj12147pdfv=1ampt=jdkhhcp4amps=e184c625e1855a6b1615ccb99ee27b1c0b3212d8PaynterB(2018January1)What100citiesarelearningfromeachotherbysharingtheirdataRetrievedfromhttpswwwfastcompanycom40518004what-100-cities-are-learning-from-each-other-by-sharing-their-dataPeakacademy(nd)Retrievedfromhttpswwwdenvergovorgcontentdenvergovenmayors-officeprograms-initiativespeak-performancepeak-academyhtmlPinnockHBarwickMCarpenterCREldridgeSGrandesGGriffithsCJRycroft-MaloneJMeissnerPMurrayEPatelASheikhAampTaylorSJC(2017)Standardsforreportingimplementationstudies(stari)ExplanationandelaborationdocumentBMJOpenRetrievedfromhttpbmjopenbmjcomcontentbmjopen74e013318fullpdfProctorESilmereHRaghavanRHovmandPAaronsGBungerAGriffeyRampHensleyM(2011)OutcomesforimplementationresearchConceptualdistinctionsmeasurementchallengesandresearchagendaAdministrationandPolicyinMentalHealth38(2)65-76doi101007s10488-010-0319-7TandonSD(2018)ProgressincommunityhealthpartnershipsResearcheducationandactionRetrievedfromhttpswwwpressjhuedujournalsprogress-community-health-partnerships-research-education-and-actionProvanKGampKenisP(2007)ModesofnetworkgovernanceStructuremanagementandeffectivenessJournalofPublicAdministrationResearchandTheory
References
64
18(2)229-252Retrievedfromhttpsacademicoupcomjpartarticle182229935895SabatierPA(1988)Anadvocacycoalitionframeworkofpolicychangeandtheroleofpolicy-orientedlearningthereinPolicySciences21(3)129-168Retrievedfromhttpwwwjstororgstable4532139SandstromAampCarlssonL(2008)TheperformanceofpolicynetworksTherelationbetweennetworkstructureandnetworkperformancePolicyStudiesJournal36497-524Retrievedfromhttponlinelibrarywileycomdoi101111j1541-0072200800281xabstractSchwartzMampWilsonE(nd)WhocanaffordtoliveinahomeAlookatdatafromthe2006americancommunitysurveyRetrievedfromhttpswwwcensusgovhousingcensuspublicationswho-can-affordpdfShearerJCAbelsonAKouyateBLavisJNampWaltG(2016)WhydopolicieschangeInstitutionsinterestsideasandnetworksinthreecasesofpolicyreformHealthPolicyandPlanning31(9)1200-1211Retrievedfromhttpsdoiorg101093heapolczw052ThelessonsofadministrativedataHigh-profilepolicy-relevantresearchpoweredbyadministrativedata(nd)Retrievedfromhttpswwwpovertyactionlaborgsitesdefaultfilesresources20161114-The-Lessons-of-Administrative-DatapdfToolsandotherneedsforqualitativeandquantitativemethods(2015February)AgencyforHealthcareResearchandQualityRetrievedfromhttpwwwahrqgovprofessionalsquality-patient-safetytalkingqualityassesstoolshtml
WeaverRCampBagchi-SenS(2013)SpatialanalysisofurbandeclineThegeographyofblightAppliedGeography4061-70
References
65
UniversityPartners
Name Title Affiliation Area(s)ofexpertise
Contactinformation
Examplesofworkingwithnon-academicdecisionmakers
WendyPBarreno
AssociateDirector
WilsonSheehanLabforEconomicOpportunitiesUniversityofNotreDame
LEOsfocusareasincludehealtheducationhousingchildwelfareeconomicopportunitycriminaljustice
wbarrenonduduorleondedu
MelissaDavey-RothwellPhD
AssociateScientist
JohnsHopkinsBloombergSchoolofPublicHealth
Health mdavey1jhuedu410-614-5854
BaltimoreCityHealthDepartmentLeadersMarylandDepartmentofHealthAddictionRecoveryServicesinDCExecutiveDirectorsofcommunityorganizationsacrosstheUSthatfocusonHIVSTIsubstanceabuseandimplementationscience
ElizabethStuart
AssociateDeanforEducationandProfessorDepartmentsofMentalHealthBiostatisticsandHealthPolicyand
JohnsHopkinsBloombergSchoolofPublicHealth
Educationsocialpolicystatisticalmethods
estuartjhuedu
Resources
66
Management
Design-ThinkingResources
EngagementGuideforCivicTech
Journeymap
PerformanceManagementResources
DenverPeakAcademy
Proxymeasuresguide
Talkingaboutperformancewithoutcomplicatingit(video)
SettingPerformanceTargetsguide
Datavisualizationguide
Performancemanagementgettingstartedguide
Exercises
Fishboneexercise
5Whys
AdministrativeData
TheLessonsofAdministrativeData(JPAL)
EvaluationResources
Quantitativeandqualitativemethods(USDepartmentofHealthandHumanServicesAgencyforHealthcareResearchandQuality)
KeyStepsinOutcomeManagement(UrbanInstitute)
RCT-YES
Sageresearchmethods
BestPracticesforMixedMethodsResearchintheHealthSciences
ACCORDChecklist
Pilotandimplementationplanning
Benchmarkingexamplesandproxymeasures
Seattleopendataplaybook
Resources
67
Datasciencescopingguide
Resources
68
AboutGovExGovExhelpsgovernmentsusedataandevidencetoimproveresidentsrsquolivesWesupportgovernmentsinbuildingtheirinternalcapacitytooperatesustainableanalyticspracticeswhichimprovecommunityoutcomesinfieldsfrompublicsafetytoeconomicdevelopmentGovExhasaproventrackrecordofworkinginconcertwithpartnerstoprovidetechnicalassistanceandtrainingatscale
SinceourlaunchinApril2015GovExhas
Trainedover2000governmentemployeesinpersonandonlineintopicsincludingdatamanagementperformancemanagementopendatacommunityengagementandanalytics
Providedtechnicalassistancetomorethan100localgovernmentsacrosstheglobeimprovingtheiruseofdatatoachieveefficienciesinservicedeliveryandbetteroutcomesforresidents
Assessedthecapacityofmorethan130governmentalorganizationstodeterminehowleadersstaffandexternalstakeholdersmanageandusedata
Localgovernmentsareincreasinglyonthefrontlinesofaddressinglargescaleproblemswithglobalimplicationssuchasclimatechangepublicsafetyandeconomicinequality
Whileevidence-basedinterventionstohelpsolvethoseproblemsareavailablelocalgovernmentstaffhavelimitedtimeandcapacitytofindandimplementresearch-basedsolutionsAsaresultcitiescontinuetoworkthewaytheyalwayshave--makingincrementalimprovementstoprogramsbutnotalwayscreatingchangeatasystem-level
ThroughresourcedevelopmentandcollaborationwithJHUfacultyGovExissupportingcityleadersinusingdataandevidencetomakedecisionsandcreatinglastingrelationshipswithresearchers
TheRoadmapwasdevelopedbyGovExaspartofitspartnershipwithBloombergPhilanthropiesWhatWorksCitiesinitiative
AboutGovEx
69
- Title page
- Introduction and purpose of the Roadmap
- Using the Roadmap
-
- Gather information about the problem
- Strategically engage key partners
- Review the evidence and collect details
- Decide on a policy to implement based on local factors
- Establish a framework to monitor progress
- Assess data accessibility quality and security
- Design implementation and evaluation
- Share your story and success
-
- Working with a range of evidence
- Thank you
- References
- Resources
- About GovEx
-
MelissaSiebenAssistantCountyAdministratorKansasCityKS
GinaThompsonDirectorofStrategicServicesLewisvilleTX
JHUFacultycollaboration
GovExalsoconvenedthecitycontactsandfacultypartnersinaseriesofAdvisoryGroupmeetingsThesemeetingsservedtoconnectcitiesandfacultytounderstandoneanothersperspective
StefanBaralBloombergSchoolofPublicHealth
AshleyBernerJHUSchoolofEducation
MelissaDavey-RothwellBloombergSchoolofPublicHealth
JonLinksBloombergSchoolofPublicHealth
TamarMendelsonBloombergSchoolofPublicHealth
KristinMmariBloombergSchoolofPublicHealth
KeshiaPollackPorterBloombergSchoolofPublicHealth
JoshSharfsteinBloombergSchoolofPublicHealth
LizStuartBloombergSchoolofPublicHealth
LindsayThompsonCareyBusinessSchool
DanielWebsterBloombergSchoolofPublicHealth
Partnersindataandevidence-basedpolicy
Wealsospokewithpartnerswhoareexpertsinthefieldofevidence-basedpolicywhosharedtheirknowledgeandresourcesofcurrentandfuturethinkinginevidence-informedthinking
WendyBarrenoAssociateDirectorWilsonSheehanLabforEconomicOpportunitiesUniversityofNotreDame
TiffanyDavenportDirectorofDataPartnershipsBaltimoresPromise
TomiHiersDirectorBaltimoreCivicSiteAnnieECaseyFoundation
ChrisKingsleySeniorAssociateAnnieECaseyFoundation
KathrynOliverAssociateProfessorLondonSchoolofHygieneandTropicalMedicine
Thankyou
61
KristenWatkinsSeniorPolicyAssociateJ-PAL
GovExParticipants
GovExAnalystMiriamMcKinneyanalyzedtheinterviewtranscriptsandconductedtextanalysisSeniorImplementationAdvisorEmilyShawwillintegrateRoadmapactivitieswithadditionalservicedeliveryactivitiesatGovEx
Thankyou
62
ReferencesAbrahamKGampHaskinsR(2017)Thepromiseofevidence-basedpolicymakingReportofthecommissiononevidence-basedpolicymakingWashingtonDCCommissiononEvidence-basedPolicymakingBigmove2Extendingthegreen(nd)Retrievedfromhttpswwwcityoflewisvillecomdoing-businesslewisville-2025big-move-2-extending-the-greenBatesMAampGlennersterR(2017)ThegeneralizabilitypuzzleRetrievedfromhttpsssirorgarticlesentrythe_generalizability_puzzleBuildwithnotfor(2018January)RetrievedfromhttpsbuildwithnotforwordpresscomCairneyPampOliverK(2017)Evidence-basedpolicymakingisnotlikeevidence-basedmedicinesohowfarshouldyougotobridgethedividebetweenevidenceandpolicyHealthResearchPolicyandSystemsdoi101186s12961-017-0192-xCairneyPOliverKampWellsteadA(2016)TobridgethedividebetweenevidenceandpolicyReduceambiguityasmuchasuncertaintyPublicAdministrationReview76(3)399-402Retrievedfromhttpdxdoiorg101111puar12555Centreforpublicimpact(nd)RetrievedfromhttpswwwcentreforpublicimpactorgCernaL(2013)ThenatureofpolicychangeandimplementationAreviewofdifferenttheoreticalapproachesOrganisationforEconomicCo-OperationandDevelopmentRetrievedfromhttpswwwoecdorgeduceriTheNatureofPolicyChangeandImplementationpdfColoradoSprings(nd)Retrievedfromhttpscoloradospringsgovcommunicationspagewhat-expect-when-you-speakupContinuumofcare(coc)program(nd)RetrievedfromhttpswwwhudexchangeinfoprogramscocCouncilpresidentkicksoff$12millionyouthfund(nd)Retrievedfromhttpwwwbaltimorecitycouncilcomcontentcouncil-president-kicks-12-million-youth-fundDurlakJAampDuPreEP(2008)ImplementationmattersAreviewofresearchontheinfluenceofimplementationonprogramoutcomesandthefactorsaffectingimplementationAmericanJournalofCommunityPsychology41327-350Retrievedfromhttpswwwncbinlmnihgovpubmed18322790Fishbonediagram(ishikawa)(nd)Retrievedfromhttpasqorglearn-about-qualitycause-analysis-toolsoverviewfishbonehtmlFivewhysGettingtotherootofaproblemquickly(nd)RetrievedfromhttpswwwmindtoolscompagesarticlenewTMC_5Whtm
References
63
FligsteinNampMcAdamD(2011)TowardageneraltheoryofstrategicactionfieldsSociologicalTheory29(1)1-26RetrievedfromhttpsociologyberkeleyedusitesdefaultfilesfacultyfligsteinFligstein20McAdam20Soc20Theory20PaperpdHeinkelP(2018January12)CitymullspermitsforbikesharingcompaniesRetrievedfromhttpwwwdentonrccomnewsdenton20180112city-mulls-permits-bike-sharing-companiesHowisareamedianincomecalculated(2012October)Retrievedfromhttpswwwhudexchangeinfofaqs843how-is-area-median-income-calculatedLampkinLMampHatryHP(2003)KeystepsinoutcomemanagementUrbanInstituteRetrievedfromhttpswwwurbanorgresearchpublicationkey-steps-outcome-managementMarshDampSmithM(2000)UnderstandingpolicynetworksTowardsadialecticalapproachPoliticalStudies484-21Retrievedfromhttpjournalssagepubcomdoiabs1011111467-924800247McCannL(nd)RetrievedfromhttplaurenellencomMoultonSampSandfortJR(2016)ThestrategicactionfieldframeworkforpolicyimplementationresearchPolicyStudiesJournal45(1)Retrievedfromhttponlinelibrarywileycomstore101111psj12147assetpsj12147pdfv=1ampt=jdkhhcp4amps=e184c625e1855a6b1615ccb99ee27b1c0b3212d8PaynterB(2018January1)What100citiesarelearningfromeachotherbysharingtheirdataRetrievedfromhttpswwwfastcompanycom40518004what-100-cities-are-learning-from-each-other-by-sharing-their-dataPeakacademy(nd)Retrievedfromhttpswwwdenvergovorgcontentdenvergovenmayors-officeprograms-initiativespeak-performancepeak-academyhtmlPinnockHBarwickMCarpenterCREldridgeSGrandesGGriffithsCJRycroft-MaloneJMeissnerPMurrayEPatelASheikhAampTaylorSJC(2017)Standardsforreportingimplementationstudies(stari)ExplanationandelaborationdocumentBMJOpenRetrievedfromhttpbmjopenbmjcomcontentbmjopen74e013318fullpdfProctorESilmereHRaghavanRHovmandPAaronsGBungerAGriffeyRampHensleyM(2011)OutcomesforimplementationresearchConceptualdistinctionsmeasurementchallengesandresearchagendaAdministrationandPolicyinMentalHealth38(2)65-76doi101007s10488-010-0319-7TandonSD(2018)ProgressincommunityhealthpartnershipsResearcheducationandactionRetrievedfromhttpswwwpressjhuedujournalsprogress-community-health-partnerships-research-education-and-actionProvanKGampKenisP(2007)ModesofnetworkgovernanceStructuremanagementandeffectivenessJournalofPublicAdministrationResearchandTheory
References
64
18(2)229-252Retrievedfromhttpsacademicoupcomjpartarticle182229935895SabatierPA(1988)Anadvocacycoalitionframeworkofpolicychangeandtheroleofpolicy-orientedlearningthereinPolicySciences21(3)129-168Retrievedfromhttpwwwjstororgstable4532139SandstromAampCarlssonL(2008)TheperformanceofpolicynetworksTherelationbetweennetworkstructureandnetworkperformancePolicyStudiesJournal36497-524Retrievedfromhttponlinelibrarywileycomdoi101111j1541-0072200800281xabstractSchwartzMampWilsonE(nd)WhocanaffordtoliveinahomeAlookatdatafromthe2006americancommunitysurveyRetrievedfromhttpswwwcensusgovhousingcensuspublicationswho-can-affordpdfShearerJCAbelsonAKouyateBLavisJNampWaltG(2016)WhydopolicieschangeInstitutionsinterestsideasandnetworksinthreecasesofpolicyreformHealthPolicyandPlanning31(9)1200-1211Retrievedfromhttpsdoiorg101093heapolczw052ThelessonsofadministrativedataHigh-profilepolicy-relevantresearchpoweredbyadministrativedata(nd)Retrievedfromhttpswwwpovertyactionlaborgsitesdefaultfilesresources20161114-The-Lessons-of-Administrative-DatapdfToolsandotherneedsforqualitativeandquantitativemethods(2015February)AgencyforHealthcareResearchandQualityRetrievedfromhttpwwwahrqgovprofessionalsquality-patient-safetytalkingqualityassesstoolshtml
WeaverRCampBagchi-SenS(2013)SpatialanalysisofurbandeclineThegeographyofblightAppliedGeography4061-70
References
65
UniversityPartners
Name Title Affiliation Area(s)ofexpertise
Contactinformation
Examplesofworkingwithnon-academicdecisionmakers
WendyPBarreno
AssociateDirector
WilsonSheehanLabforEconomicOpportunitiesUniversityofNotreDame
LEOsfocusareasincludehealtheducationhousingchildwelfareeconomicopportunitycriminaljustice
wbarrenonduduorleondedu
MelissaDavey-RothwellPhD
AssociateScientist
JohnsHopkinsBloombergSchoolofPublicHealth
Health mdavey1jhuedu410-614-5854
BaltimoreCityHealthDepartmentLeadersMarylandDepartmentofHealthAddictionRecoveryServicesinDCExecutiveDirectorsofcommunityorganizationsacrosstheUSthatfocusonHIVSTIsubstanceabuseandimplementationscience
ElizabethStuart
AssociateDeanforEducationandProfessorDepartmentsofMentalHealthBiostatisticsandHealthPolicyand
JohnsHopkinsBloombergSchoolofPublicHealth
Educationsocialpolicystatisticalmethods
estuartjhuedu
Resources
66
Management
Design-ThinkingResources
EngagementGuideforCivicTech
Journeymap
PerformanceManagementResources
DenverPeakAcademy
Proxymeasuresguide
Talkingaboutperformancewithoutcomplicatingit(video)
SettingPerformanceTargetsguide
Datavisualizationguide
Performancemanagementgettingstartedguide
Exercises
Fishboneexercise
5Whys
AdministrativeData
TheLessonsofAdministrativeData(JPAL)
EvaluationResources
Quantitativeandqualitativemethods(USDepartmentofHealthandHumanServicesAgencyforHealthcareResearchandQuality)
KeyStepsinOutcomeManagement(UrbanInstitute)
RCT-YES
Sageresearchmethods
BestPracticesforMixedMethodsResearchintheHealthSciences
ACCORDChecklist
Pilotandimplementationplanning
Benchmarkingexamplesandproxymeasures
Seattleopendataplaybook
Resources
67
Datasciencescopingguide
Resources
68
AboutGovExGovExhelpsgovernmentsusedataandevidencetoimproveresidentsrsquolivesWesupportgovernmentsinbuildingtheirinternalcapacitytooperatesustainableanalyticspracticeswhichimprovecommunityoutcomesinfieldsfrompublicsafetytoeconomicdevelopmentGovExhasaproventrackrecordofworkinginconcertwithpartnerstoprovidetechnicalassistanceandtrainingatscale
SinceourlaunchinApril2015GovExhas
Trainedover2000governmentemployeesinpersonandonlineintopicsincludingdatamanagementperformancemanagementopendatacommunityengagementandanalytics
Providedtechnicalassistancetomorethan100localgovernmentsacrosstheglobeimprovingtheiruseofdatatoachieveefficienciesinservicedeliveryandbetteroutcomesforresidents
Assessedthecapacityofmorethan130governmentalorganizationstodeterminehowleadersstaffandexternalstakeholdersmanageandusedata
Localgovernmentsareincreasinglyonthefrontlinesofaddressinglargescaleproblemswithglobalimplicationssuchasclimatechangepublicsafetyandeconomicinequality
Whileevidence-basedinterventionstohelpsolvethoseproblemsareavailablelocalgovernmentstaffhavelimitedtimeandcapacitytofindandimplementresearch-basedsolutionsAsaresultcitiescontinuetoworkthewaytheyalwayshave--makingincrementalimprovementstoprogramsbutnotalwayscreatingchangeatasystem-level
ThroughresourcedevelopmentandcollaborationwithJHUfacultyGovExissupportingcityleadersinusingdataandevidencetomakedecisionsandcreatinglastingrelationshipswithresearchers
TheRoadmapwasdevelopedbyGovExaspartofitspartnershipwithBloombergPhilanthropiesWhatWorksCitiesinitiative
AboutGovEx
69
- Title page
- Introduction and purpose of the Roadmap
- Using the Roadmap
-
- Gather information about the problem
- Strategically engage key partners
- Review the evidence and collect details
- Decide on a policy to implement based on local factors
- Establish a framework to monitor progress
- Assess data accessibility quality and security
- Design implementation and evaluation
- Share your story and success
-
- Working with a range of evidence
- Thank you
- References
- Resources
- About GovEx
-
KristenWatkinsSeniorPolicyAssociateJ-PAL
GovExParticipants
GovExAnalystMiriamMcKinneyanalyzedtheinterviewtranscriptsandconductedtextanalysisSeniorImplementationAdvisorEmilyShawwillintegrateRoadmapactivitieswithadditionalservicedeliveryactivitiesatGovEx
Thankyou
62
ReferencesAbrahamKGampHaskinsR(2017)Thepromiseofevidence-basedpolicymakingReportofthecommissiononevidence-basedpolicymakingWashingtonDCCommissiononEvidence-basedPolicymakingBigmove2Extendingthegreen(nd)Retrievedfromhttpswwwcityoflewisvillecomdoing-businesslewisville-2025big-move-2-extending-the-greenBatesMAampGlennersterR(2017)ThegeneralizabilitypuzzleRetrievedfromhttpsssirorgarticlesentrythe_generalizability_puzzleBuildwithnotfor(2018January)RetrievedfromhttpsbuildwithnotforwordpresscomCairneyPampOliverK(2017)Evidence-basedpolicymakingisnotlikeevidence-basedmedicinesohowfarshouldyougotobridgethedividebetweenevidenceandpolicyHealthResearchPolicyandSystemsdoi101186s12961-017-0192-xCairneyPOliverKampWellsteadA(2016)TobridgethedividebetweenevidenceandpolicyReduceambiguityasmuchasuncertaintyPublicAdministrationReview76(3)399-402Retrievedfromhttpdxdoiorg101111puar12555Centreforpublicimpact(nd)RetrievedfromhttpswwwcentreforpublicimpactorgCernaL(2013)ThenatureofpolicychangeandimplementationAreviewofdifferenttheoreticalapproachesOrganisationforEconomicCo-OperationandDevelopmentRetrievedfromhttpswwwoecdorgeduceriTheNatureofPolicyChangeandImplementationpdfColoradoSprings(nd)Retrievedfromhttpscoloradospringsgovcommunicationspagewhat-expect-when-you-speakupContinuumofcare(coc)program(nd)RetrievedfromhttpswwwhudexchangeinfoprogramscocCouncilpresidentkicksoff$12millionyouthfund(nd)Retrievedfromhttpwwwbaltimorecitycouncilcomcontentcouncil-president-kicks-12-million-youth-fundDurlakJAampDuPreEP(2008)ImplementationmattersAreviewofresearchontheinfluenceofimplementationonprogramoutcomesandthefactorsaffectingimplementationAmericanJournalofCommunityPsychology41327-350Retrievedfromhttpswwwncbinlmnihgovpubmed18322790Fishbonediagram(ishikawa)(nd)Retrievedfromhttpasqorglearn-about-qualitycause-analysis-toolsoverviewfishbonehtmlFivewhysGettingtotherootofaproblemquickly(nd)RetrievedfromhttpswwwmindtoolscompagesarticlenewTMC_5Whtm
References
63
FligsteinNampMcAdamD(2011)TowardageneraltheoryofstrategicactionfieldsSociologicalTheory29(1)1-26RetrievedfromhttpsociologyberkeleyedusitesdefaultfilesfacultyfligsteinFligstein20McAdam20Soc20Theory20PaperpdHeinkelP(2018January12)CitymullspermitsforbikesharingcompaniesRetrievedfromhttpwwwdentonrccomnewsdenton20180112city-mulls-permits-bike-sharing-companiesHowisareamedianincomecalculated(2012October)Retrievedfromhttpswwwhudexchangeinfofaqs843how-is-area-median-income-calculatedLampkinLMampHatryHP(2003)KeystepsinoutcomemanagementUrbanInstituteRetrievedfromhttpswwwurbanorgresearchpublicationkey-steps-outcome-managementMarshDampSmithM(2000)UnderstandingpolicynetworksTowardsadialecticalapproachPoliticalStudies484-21Retrievedfromhttpjournalssagepubcomdoiabs1011111467-924800247McCannL(nd)RetrievedfromhttplaurenellencomMoultonSampSandfortJR(2016)ThestrategicactionfieldframeworkforpolicyimplementationresearchPolicyStudiesJournal45(1)Retrievedfromhttponlinelibrarywileycomstore101111psj12147assetpsj12147pdfv=1ampt=jdkhhcp4amps=e184c625e1855a6b1615ccb99ee27b1c0b3212d8PaynterB(2018January1)What100citiesarelearningfromeachotherbysharingtheirdataRetrievedfromhttpswwwfastcompanycom40518004what-100-cities-are-learning-from-each-other-by-sharing-their-dataPeakacademy(nd)Retrievedfromhttpswwwdenvergovorgcontentdenvergovenmayors-officeprograms-initiativespeak-performancepeak-academyhtmlPinnockHBarwickMCarpenterCREldridgeSGrandesGGriffithsCJRycroft-MaloneJMeissnerPMurrayEPatelASheikhAampTaylorSJC(2017)Standardsforreportingimplementationstudies(stari)ExplanationandelaborationdocumentBMJOpenRetrievedfromhttpbmjopenbmjcomcontentbmjopen74e013318fullpdfProctorESilmereHRaghavanRHovmandPAaronsGBungerAGriffeyRampHensleyM(2011)OutcomesforimplementationresearchConceptualdistinctionsmeasurementchallengesandresearchagendaAdministrationandPolicyinMentalHealth38(2)65-76doi101007s10488-010-0319-7TandonSD(2018)ProgressincommunityhealthpartnershipsResearcheducationandactionRetrievedfromhttpswwwpressjhuedujournalsprogress-community-health-partnerships-research-education-and-actionProvanKGampKenisP(2007)ModesofnetworkgovernanceStructuremanagementandeffectivenessJournalofPublicAdministrationResearchandTheory
References
64
18(2)229-252Retrievedfromhttpsacademicoupcomjpartarticle182229935895SabatierPA(1988)Anadvocacycoalitionframeworkofpolicychangeandtheroleofpolicy-orientedlearningthereinPolicySciences21(3)129-168Retrievedfromhttpwwwjstororgstable4532139SandstromAampCarlssonL(2008)TheperformanceofpolicynetworksTherelationbetweennetworkstructureandnetworkperformancePolicyStudiesJournal36497-524Retrievedfromhttponlinelibrarywileycomdoi101111j1541-0072200800281xabstractSchwartzMampWilsonE(nd)WhocanaffordtoliveinahomeAlookatdatafromthe2006americancommunitysurveyRetrievedfromhttpswwwcensusgovhousingcensuspublicationswho-can-affordpdfShearerJCAbelsonAKouyateBLavisJNampWaltG(2016)WhydopolicieschangeInstitutionsinterestsideasandnetworksinthreecasesofpolicyreformHealthPolicyandPlanning31(9)1200-1211Retrievedfromhttpsdoiorg101093heapolczw052ThelessonsofadministrativedataHigh-profilepolicy-relevantresearchpoweredbyadministrativedata(nd)Retrievedfromhttpswwwpovertyactionlaborgsitesdefaultfilesresources20161114-The-Lessons-of-Administrative-DatapdfToolsandotherneedsforqualitativeandquantitativemethods(2015February)AgencyforHealthcareResearchandQualityRetrievedfromhttpwwwahrqgovprofessionalsquality-patient-safetytalkingqualityassesstoolshtml
WeaverRCampBagchi-SenS(2013)SpatialanalysisofurbandeclineThegeographyofblightAppliedGeography4061-70
References
65
UniversityPartners
Name Title Affiliation Area(s)ofexpertise
Contactinformation
Examplesofworkingwithnon-academicdecisionmakers
WendyPBarreno
AssociateDirector
WilsonSheehanLabforEconomicOpportunitiesUniversityofNotreDame
LEOsfocusareasincludehealtheducationhousingchildwelfareeconomicopportunitycriminaljustice
wbarrenonduduorleondedu
MelissaDavey-RothwellPhD
AssociateScientist
JohnsHopkinsBloombergSchoolofPublicHealth
Health mdavey1jhuedu410-614-5854
BaltimoreCityHealthDepartmentLeadersMarylandDepartmentofHealthAddictionRecoveryServicesinDCExecutiveDirectorsofcommunityorganizationsacrosstheUSthatfocusonHIVSTIsubstanceabuseandimplementationscience
ElizabethStuart
AssociateDeanforEducationandProfessorDepartmentsofMentalHealthBiostatisticsandHealthPolicyand
JohnsHopkinsBloombergSchoolofPublicHealth
Educationsocialpolicystatisticalmethods
estuartjhuedu
Resources
66
Management
Design-ThinkingResources
EngagementGuideforCivicTech
Journeymap
PerformanceManagementResources
DenverPeakAcademy
Proxymeasuresguide
Talkingaboutperformancewithoutcomplicatingit(video)
SettingPerformanceTargetsguide
Datavisualizationguide
Performancemanagementgettingstartedguide
Exercises
Fishboneexercise
5Whys
AdministrativeData
TheLessonsofAdministrativeData(JPAL)
EvaluationResources
Quantitativeandqualitativemethods(USDepartmentofHealthandHumanServicesAgencyforHealthcareResearchandQuality)
KeyStepsinOutcomeManagement(UrbanInstitute)
RCT-YES
Sageresearchmethods
BestPracticesforMixedMethodsResearchintheHealthSciences
ACCORDChecklist
Pilotandimplementationplanning
Benchmarkingexamplesandproxymeasures
Seattleopendataplaybook
Resources
67
Datasciencescopingguide
Resources
68
AboutGovExGovExhelpsgovernmentsusedataandevidencetoimproveresidentsrsquolivesWesupportgovernmentsinbuildingtheirinternalcapacitytooperatesustainableanalyticspracticeswhichimprovecommunityoutcomesinfieldsfrompublicsafetytoeconomicdevelopmentGovExhasaproventrackrecordofworkinginconcertwithpartnerstoprovidetechnicalassistanceandtrainingatscale
SinceourlaunchinApril2015GovExhas
Trainedover2000governmentemployeesinpersonandonlineintopicsincludingdatamanagementperformancemanagementopendatacommunityengagementandanalytics
Providedtechnicalassistancetomorethan100localgovernmentsacrosstheglobeimprovingtheiruseofdatatoachieveefficienciesinservicedeliveryandbetteroutcomesforresidents
Assessedthecapacityofmorethan130governmentalorganizationstodeterminehowleadersstaffandexternalstakeholdersmanageandusedata
Localgovernmentsareincreasinglyonthefrontlinesofaddressinglargescaleproblemswithglobalimplicationssuchasclimatechangepublicsafetyandeconomicinequality
Whileevidence-basedinterventionstohelpsolvethoseproblemsareavailablelocalgovernmentstaffhavelimitedtimeandcapacitytofindandimplementresearch-basedsolutionsAsaresultcitiescontinuetoworkthewaytheyalwayshave--makingincrementalimprovementstoprogramsbutnotalwayscreatingchangeatasystem-level
ThroughresourcedevelopmentandcollaborationwithJHUfacultyGovExissupportingcityleadersinusingdataandevidencetomakedecisionsandcreatinglastingrelationshipswithresearchers
TheRoadmapwasdevelopedbyGovExaspartofitspartnershipwithBloombergPhilanthropiesWhatWorksCitiesinitiative
AboutGovEx
69
- Title page
- Introduction and purpose of the Roadmap
- Using the Roadmap
-
- Gather information about the problem
- Strategically engage key partners
- Review the evidence and collect details
- Decide on a policy to implement based on local factors
- Establish a framework to monitor progress
- Assess data accessibility quality and security
- Design implementation and evaluation
- Share your story and success
-
- Working with a range of evidence
- Thank you
- References
- Resources
- About GovEx
-
ReferencesAbrahamKGampHaskinsR(2017)Thepromiseofevidence-basedpolicymakingReportofthecommissiononevidence-basedpolicymakingWashingtonDCCommissiononEvidence-basedPolicymakingBigmove2Extendingthegreen(nd)Retrievedfromhttpswwwcityoflewisvillecomdoing-businesslewisville-2025big-move-2-extending-the-greenBatesMAampGlennersterR(2017)ThegeneralizabilitypuzzleRetrievedfromhttpsssirorgarticlesentrythe_generalizability_puzzleBuildwithnotfor(2018January)RetrievedfromhttpsbuildwithnotforwordpresscomCairneyPampOliverK(2017)Evidence-basedpolicymakingisnotlikeevidence-basedmedicinesohowfarshouldyougotobridgethedividebetweenevidenceandpolicyHealthResearchPolicyandSystemsdoi101186s12961-017-0192-xCairneyPOliverKampWellsteadA(2016)TobridgethedividebetweenevidenceandpolicyReduceambiguityasmuchasuncertaintyPublicAdministrationReview76(3)399-402Retrievedfromhttpdxdoiorg101111puar12555Centreforpublicimpact(nd)RetrievedfromhttpswwwcentreforpublicimpactorgCernaL(2013)ThenatureofpolicychangeandimplementationAreviewofdifferenttheoreticalapproachesOrganisationforEconomicCo-OperationandDevelopmentRetrievedfromhttpswwwoecdorgeduceriTheNatureofPolicyChangeandImplementationpdfColoradoSprings(nd)Retrievedfromhttpscoloradospringsgovcommunicationspagewhat-expect-when-you-speakupContinuumofcare(coc)program(nd)RetrievedfromhttpswwwhudexchangeinfoprogramscocCouncilpresidentkicksoff$12millionyouthfund(nd)Retrievedfromhttpwwwbaltimorecitycouncilcomcontentcouncil-president-kicks-12-million-youth-fundDurlakJAampDuPreEP(2008)ImplementationmattersAreviewofresearchontheinfluenceofimplementationonprogramoutcomesandthefactorsaffectingimplementationAmericanJournalofCommunityPsychology41327-350Retrievedfromhttpswwwncbinlmnihgovpubmed18322790Fishbonediagram(ishikawa)(nd)Retrievedfromhttpasqorglearn-about-qualitycause-analysis-toolsoverviewfishbonehtmlFivewhysGettingtotherootofaproblemquickly(nd)RetrievedfromhttpswwwmindtoolscompagesarticlenewTMC_5Whtm
References
63
FligsteinNampMcAdamD(2011)TowardageneraltheoryofstrategicactionfieldsSociologicalTheory29(1)1-26RetrievedfromhttpsociologyberkeleyedusitesdefaultfilesfacultyfligsteinFligstein20McAdam20Soc20Theory20PaperpdHeinkelP(2018January12)CitymullspermitsforbikesharingcompaniesRetrievedfromhttpwwwdentonrccomnewsdenton20180112city-mulls-permits-bike-sharing-companiesHowisareamedianincomecalculated(2012October)Retrievedfromhttpswwwhudexchangeinfofaqs843how-is-area-median-income-calculatedLampkinLMampHatryHP(2003)KeystepsinoutcomemanagementUrbanInstituteRetrievedfromhttpswwwurbanorgresearchpublicationkey-steps-outcome-managementMarshDampSmithM(2000)UnderstandingpolicynetworksTowardsadialecticalapproachPoliticalStudies484-21Retrievedfromhttpjournalssagepubcomdoiabs1011111467-924800247McCannL(nd)RetrievedfromhttplaurenellencomMoultonSampSandfortJR(2016)ThestrategicactionfieldframeworkforpolicyimplementationresearchPolicyStudiesJournal45(1)Retrievedfromhttponlinelibrarywileycomstore101111psj12147assetpsj12147pdfv=1ampt=jdkhhcp4amps=e184c625e1855a6b1615ccb99ee27b1c0b3212d8PaynterB(2018January1)What100citiesarelearningfromeachotherbysharingtheirdataRetrievedfromhttpswwwfastcompanycom40518004what-100-cities-are-learning-from-each-other-by-sharing-their-dataPeakacademy(nd)Retrievedfromhttpswwwdenvergovorgcontentdenvergovenmayors-officeprograms-initiativespeak-performancepeak-academyhtmlPinnockHBarwickMCarpenterCREldridgeSGrandesGGriffithsCJRycroft-MaloneJMeissnerPMurrayEPatelASheikhAampTaylorSJC(2017)Standardsforreportingimplementationstudies(stari)ExplanationandelaborationdocumentBMJOpenRetrievedfromhttpbmjopenbmjcomcontentbmjopen74e013318fullpdfProctorESilmereHRaghavanRHovmandPAaronsGBungerAGriffeyRampHensleyM(2011)OutcomesforimplementationresearchConceptualdistinctionsmeasurementchallengesandresearchagendaAdministrationandPolicyinMentalHealth38(2)65-76doi101007s10488-010-0319-7TandonSD(2018)ProgressincommunityhealthpartnershipsResearcheducationandactionRetrievedfromhttpswwwpressjhuedujournalsprogress-community-health-partnerships-research-education-and-actionProvanKGampKenisP(2007)ModesofnetworkgovernanceStructuremanagementandeffectivenessJournalofPublicAdministrationResearchandTheory
References
64
18(2)229-252Retrievedfromhttpsacademicoupcomjpartarticle182229935895SabatierPA(1988)Anadvocacycoalitionframeworkofpolicychangeandtheroleofpolicy-orientedlearningthereinPolicySciences21(3)129-168Retrievedfromhttpwwwjstororgstable4532139SandstromAampCarlssonL(2008)TheperformanceofpolicynetworksTherelationbetweennetworkstructureandnetworkperformancePolicyStudiesJournal36497-524Retrievedfromhttponlinelibrarywileycomdoi101111j1541-0072200800281xabstractSchwartzMampWilsonE(nd)WhocanaffordtoliveinahomeAlookatdatafromthe2006americancommunitysurveyRetrievedfromhttpswwwcensusgovhousingcensuspublicationswho-can-affordpdfShearerJCAbelsonAKouyateBLavisJNampWaltG(2016)WhydopolicieschangeInstitutionsinterestsideasandnetworksinthreecasesofpolicyreformHealthPolicyandPlanning31(9)1200-1211Retrievedfromhttpsdoiorg101093heapolczw052ThelessonsofadministrativedataHigh-profilepolicy-relevantresearchpoweredbyadministrativedata(nd)Retrievedfromhttpswwwpovertyactionlaborgsitesdefaultfilesresources20161114-The-Lessons-of-Administrative-DatapdfToolsandotherneedsforqualitativeandquantitativemethods(2015February)AgencyforHealthcareResearchandQualityRetrievedfromhttpwwwahrqgovprofessionalsquality-patient-safetytalkingqualityassesstoolshtml
WeaverRCampBagchi-SenS(2013)SpatialanalysisofurbandeclineThegeographyofblightAppliedGeography4061-70
References
65
UniversityPartners
Name Title Affiliation Area(s)ofexpertise
Contactinformation
Examplesofworkingwithnon-academicdecisionmakers
WendyPBarreno
AssociateDirector
WilsonSheehanLabforEconomicOpportunitiesUniversityofNotreDame
LEOsfocusareasincludehealtheducationhousingchildwelfareeconomicopportunitycriminaljustice
wbarrenonduduorleondedu
MelissaDavey-RothwellPhD
AssociateScientist
JohnsHopkinsBloombergSchoolofPublicHealth
Health mdavey1jhuedu410-614-5854
BaltimoreCityHealthDepartmentLeadersMarylandDepartmentofHealthAddictionRecoveryServicesinDCExecutiveDirectorsofcommunityorganizationsacrosstheUSthatfocusonHIVSTIsubstanceabuseandimplementationscience
ElizabethStuart
AssociateDeanforEducationandProfessorDepartmentsofMentalHealthBiostatisticsandHealthPolicyand
JohnsHopkinsBloombergSchoolofPublicHealth
Educationsocialpolicystatisticalmethods
estuartjhuedu
Resources
66
Management
Design-ThinkingResources
EngagementGuideforCivicTech
Journeymap
PerformanceManagementResources
DenverPeakAcademy
Proxymeasuresguide
Talkingaboutperformancewithoutcomplicatingit(video)
SettingPerformanceTargetsguide
Datavisualizationguide
Performancemanagementgettingstartedguide
Exercises
Fishboneexercise
5Whys
AdministrativeData
TheLessonsofAdministrativeData(JPAL)
EvaluationResources
Quantitativeandqualitativemethods(USDepartmentofHealthandHumanServicesAgencyforHealthcareResearchandQuality)
KeyStepsinOutcomeManagement(UrbanInstitute)
RCT-YES
Sageresearchmethods
BestPracticesforMixedMethodsResearchintheHealthSciences
ACCORDChecklist
Pilotandimplementationplanning
Benchmarkingexamplesandproxymeasures
Seattleopendataplaybook
Resources
67
Datasciencescopingguide
Resources
68
AboutGovExGovExhelpsgovernmentsusedataandevidencetoimproveresidentsrsquolivesWesupportgovernmentsinbuildingtheirinternalcapacitytooperatesustainableanalyticspracticeswhichimprovecommunityoutcomesinfieldsfrompublicsafetytoeconomicdevelopmentGovExhasaproventrackrecordofworkinginconcertwithpartnerstoprovidetechnicalassistanceandtrainingatscale
SinceourlaunchinApril2015GovExhas
Trainedover2000governmentemployeesinpersonandonlineintopicsincludingdatamanagementperformancemanagementopendatacommunityengagementandanalytics
Providedtechnicalassistancetomorethan100localgovernmentsacrosstheglobeimprovingtheiruseofdatatoachieveefficienciesinservicedeliveryandbetteroutcomesforresidents
Assessedthecapacityofmorethan130governmentalorganizationstodeterminehowleadersstaffandexternalstakeholdersmanageandusedata
Localgovernmentsareincreasinglyonthefrontlinesofaddressinglargescaleproblemswithglobalimplicationssuchasclimatechangepublicsafetyandeconomicinequality
Whileevidence-basedinterventionstohelpsolvethoseproblemsareavailablelocalgovernmentstaffhavelimitedtimeandcapacitytofindandimplementresearch-basedsolutionsAsaresultcitiescontinuetoworkthewaytheyalwayshave--makingincrementalimprovementstoprogramsbutnotalwayscreatingchangeatasystem-level
ThroughresourcedevelopmentandcollaborationwithJHUfacultyGovExissupportingcityleadersinusingdataandevidencetomakedecisionsandcreatinglastingrelationshipswithresearchers
TheRoadmapwasdevelopedbyGovExaspartofitspartnershipwithBloombergPhilanthropiesWhatWorksCitiesinitiative
AboutGovEx
69
- Title page
- Introduction and purpose of the Roadmap
- Using the Roadmap
-
- Gather information about the problem
- Strategically engage key partners
- Review the evidence and collect details
- Decide on a policy to implement based on local factors
- Establish a framework to monitor progress
- Assess data accessibility quality and security
- Design implementation and evaluation
- Share your story and success
-
- Working with a range of evidence
- Thank you
- References
- Resources
- About GovEx
-
FligsteinNampMcAdamD(2011)TowardageneraltheoryofstrategicactionfieldsSociologicalTheory29(1)1-26RetrievedfromhttpsociologyberkeleyedusitesdefaultfilesfacultyfligsteinFligstein20McAdam20Soc20Theory20PaperpdHeinkelP(2018January12)CitymullspermitsforbikesharingcompaniesRetrievedfromhttpwwwdentonrccomnewsdenton20180112city-mulls-permits-bike-sharing-companiesHowisareamedianincomecalculated(2012October)Retrievedfromhttpswwwhudexchangeinfofaqs843how-is-area-median-income-calculatedLampkinLMampHatryHP(2003)KeystepsinoutcomemanagementUrbanInstituteRetrievedfromhttpswwwurbanorgresearchpublicationkey-steps-outcome-managementMarshDampSmithM(2000)UnderstandingpolicynetworksTowardsadialecticalapproachPoliticalStudies484-21Retrievedfromhttpjournalssagepubcomdoiabs1011111467-924800247McCannL(nd)RetrievedfromhttplaurenellencomMoultonSampSandfortJR(2016)ThestrategicactionfieldframeworkforpolicyimplementationresearchPolicyStudiesJournal45(1)Retrievedfromhttponlinelibrarywileycomstore101111psj12147assetpsj12147pdfv=1ampt=jdkhhcp4amps=e184c625e1855a6b1615ccb99ee27b1c0b3212d8PaynterB(2018January1)What100citiesarelearningfromeachotherbysharingtheirdataRetrievedfromhttpswwwfastcompanycom40518004what-100-cities-are-learning-from-each-other-by-sharing-their-dataPeakacademy(nd)Retrievedfromhttpswwwdenvergovorgcontentdenvergovenmayors-officeprograms-initiativespeak-performancepeak-academyhtmlPinnockHBarwickMCarpenterCREldridgeSGrandesGGriffithsCJRycroft-MaloneJMeissnerPMurrayEPatelASheikhAampTaylorSJC(2017)Standardsforreportingimplementationstudies(stari)ExplanationandelaborationdocumentBMJOpenRetrievedfromhttpbmjopenbmjcomcontentbmjopen74e013318fullpdfProctorESilmereHRaghavanRHovmandPAaronsGBungerAGriffeyRampHensleyM(2011)OutcomesforimplementationresearchConceptualdistinctionsmeasurementchallengesandresearchagendaAdministrationandPolicyinMentalHealth38(2)65-76doi101007s10488-010-0319-7TandonSD(2018)ProgressincommunityhealthpartnershipsResearcheducationandactionRetrievedfromhttpswwwpressjhuedujournalsprogress-community-health-partnerships-research-education-and-actionProvanKGampKenisP(2007)ModesofnetworkgovernanceStructuremanagementandeffectivenessJournalofPublicAdministrationResearchandTheory
References
64
18(2)229-252Retrievedfromhttpsacademicoupcomjpartarticle182229935895SabatierPA(1988)Anadvocacycoalitionframeworkofpolicychangeandtheroleofpolicy-orientedlearningthereinPolicySciences21(3)129-168Retrievedfromhttpwwwjstororgstable4532139SandstromAampCarlssonL(2008)TheperformanceofpolicynetworksTherelationbetweennetworkstructureandnetworkperformancePolicyStudiesJournal36497-524Retrievedfromhttponlinelibrarywileycomdoi101111j1541-0072200800281xabstractSchwartzMampWilsonE(nd)WhocanaffordtoliveinahomeAlookatdatafromthe2006americancommunitysurveyRetrievedfromhttpswwwcensusgovhousingcensuspublicationswho-can-affordpdfShearerJCAbelsonAKouyateBLavisJNampWaltG(2016)WhydopolicieschangeInstitutionsinterestsideasandnetworksinthreecasesofpolicyreformHealthPolicyandPlanning31(9)1200-1211Retrievedfromhttpsdoiorg101093heapolczw052ThelessonsofadministrativedataHigh-profilepolicy-relevantresearchpoweredbyadministrativedata(nd)Retrievedfromhttpswwwpovertyactionlaborgsitesdefaultfilesresources20161114-The-Lessons-of-Administrative-DatapdfToolsandotherneedsforqualitativeandquantitativemethods(2015February)AgencyforHealthcareResearchandQualityRetrievedfromhttpwwwahrqgovprofessionalsquality-patient-safetytalkingqualityassesstoolshtml
WeaverRCampBagchi-SenS(2013)SpatialanalysisofurbandeclineThegeographyofblightAppliedGeography4061-70
References
65
UniversityPartners
Name Title Affiliation Area(s)ofexpertise
Contactinformation
Examplesofworkingwithnon-academicdecisionmakers
WendyPBarreno
AssociateDirector
WilsonSheehanLabforEconomicOpportunitiesUniversityofNotreDame
LEOsfocusareasincludehealtheducationhousingchildwelfareeconomicopportunitycriminaljustice
wbarrenonduduorleondedu
MelissaDavey-RothwellPhD
AssociateScientist
JohnsHopkinsBloombergSchoolofPublicHealth
Health mdavey1jhuedu410-614-5854
BaltimoreCityHealthDepartmentLeadersMarylandDepartmentofHealthAddictionRecoveryServicesinDCExecutiveDirectorsofcommunityorganizationsacrosstheUSthatfocusonHIVSTIsubstanceabuseandimplementationscience
ElizabethStuart
AssociateDeanforEducationandProfessorDepartmentsofMentalHealthBiostatisticsandHealthPolicyand
JohnsHopkinsBloombergSchoolofPublicHealth
Educationsocialpolicystatisticalmethods
estuartjhuedu
Resources
66
Management
Design-ThinkingResources
EngagementGuideforCivicTech
Journeymap
PerformanceManagementResources
DenverPeakAcademy
Proxymeasuresguide
Talkingaboutperformancewithoutcomplicatingit(video)
SettingPerformanceTargetsguide
Datavisualizationguide
Performancemanagementgettingstartedguide
Exercises
Fishboneexercise
5Whys
AdministrativeData
TheLessonsofAdministrativeData(JPAL)
EvaluationResources
Quantitativeandqualitativemethods(USDepartmentofHealthandHumanServicesAgencyforHealthcareResearchandQuality)
KeyStepsinOutcomeManagement(UrbanInstitute)
RCT-YES
Sageresearchmethods
BestPracticesforMixedMethodsResearchintheHealthSciences
ACCORDChecklist
Pilotandimplementationplanning
Benchmarkingexamplesandproxymeasures
Seattleopendataplaybook
Resources
67
Datasciencescopingguide
Resources
68
AboutGovExGovExhelpsgovernmentsusedataandevidencetoimproveresidentsrsquolivesWesupportgovernmentsinbuildingtheirinternalcapacitytooperatesustainableanalyticspracticeswhichimprovecommunityoutcomesinfieldsfrompublicsafetytoeconomicdevelopmentGovExhasaproventrackrecordofworkinginconcertwithpartnerstoprovidetechnicalassistanceandtrainingatscale
SinceourlaunchinApril2015GovExhas
Trainedover2000governmentemployeesinpersonandonlineintopicsincludingdatamanagementperformancemanagementopendatacommunityengagementandanalytics
Providedtechnicalassistancetomorethan100localgovernmentsacrosstheglobeimprovingtheiruseofdatatoachieveefficienciesinservicedeliveryandbetteroutcomesforresidents
Assessedthecapacityofmorethan130governmentalorganizationstodeterminehowleadersstaffandexternalstakeholdersmanageandusedata
Localgovernmentsareincreasinglyonthefrontlinesofaddressinglargescaleproblemswithglobalimplicationssuchasclimatechangepublicsafetyandeconomicinequality
Whileevidence-basedinterventionstohelpsolvethoseproblemsareavailablelocalgovernmentstaffhavelimitedtimeandcapacitytofindandimplementresearch-basedsolutionsAsaresultcitiescontinuetoworkthewaytheyalwayshave--makingincrementalimprovementstoprogramsbutnotalwayscreatingchangeatasystem-level
ThroughresourcedevelopmentandcollaborationwithJHUfacultyGovExissupportingcityleadersinusingdataandevidencetomakedecisionsandcreatinglastingrelationshipswithresearchers
TheRoadmapwasdevelopedbyGovExaspartofitspartnershipwithBloombergPhilanthropiesWhatWorksCitiesinitiative
AboutGovEx
69
- Title page
- Introduction and purpose of the Roadmap
- Using the Roadmap
-
- Gather information about the problem
- Strategically engage key partners
- Review the evidence and collect details
- Decide on a policy to implement based on local factors
- Establish a framework to monitor progress
- Assess data accessibility quality and security
- Design implementation and evaluation
- Share your story and success
-
- Working with a range of evidence
- Thank you
- References
- Resources
- About GovEx
-
18(2)229-252Retrievedfromhttpsacademicoupcomjpartarticle182229935895SabatierPA(1988)Anadvocacycoalitionframeworkofpolicychangeandtheroleofpolicy-orientedlearningthereinPolicySciences21(3)129-168Retrievedfromhttpwwwjstororgstable4532139SandstromAampCarlssonL(2008)TheperformanceofpolicynetworksTherelationbetweennetworkstructureandnetworkperformancePolicyStudiesJournal36497-524Retrievedfromhttponlinelibrarywileycomdoi101111j1541-0072200800281xabstractSchwartzMampWilsonE(nd)WhocanaffordtoliveinahomeAlookatdatafromthe2006americancommunitysurveyRetrievedfromhttpswwwcensusgovhousingcensuspublicationswho-can-affordpdfShearerJCAbelsonAKouyateBLavisJNampWaltG(2016)WhydopolicieschangeInstitutionsinterestsideasandnetworksinthreecasesofpolicyreformHealthPolicyandPlanning31(9)1200-1211Retrievedfromhttpsdoiorg101093heapolczw052ThelessonsofadministrativedataHigh-profilepolicy-relevantresearchpoweredbyadministrativedata(nd)Retrievedfromhttpswwwpovertyactionlaborgsitesdefaultfilesresources20161114-The-Lessons-of-Administrative-DatapdfToolsandotherneedsforqualitativeandquantitativemethods(2015February)AgencyforHealthcareResearchandQualityRetrievedfromhttpwwwahrqgovprofessionalsquality-patient-safetytalkingqualityassesstoolshtml
WeaverRCampBagchi-SenS(2013)SpatialanalysisofurbandeclineThegeographyofblightAppliedGeography4061-70
References
65
UniversityPartners
Name Title Affiliation Area(s)ofexpertise
Contactinformation
Examplesofworkingwithnon-academicdecisionmakers
WendyPBarreno
AssociateDirector
WilsonSheehanLabforEconomicOpportunitiesUniversityofNotreDame
LEOsfocusareasincludehealtheducationhousingchildwelfareeconomicopportunitycriminaljustice
wbarrenonduduorleondedu
MelissaDavey-RothwellPhD
AssociateScientist
JohnsHopkinsBloombergSchoolofPublicHealth
Health mdavey1jhuedu410-614-5854
BaltimoreCityHealthDepartmentLeadersMarylandDepartmentofHealthAddictionRecoveryServicesinDCExecutiveDirectorsofcommunityorganizationsacrosstheUSthatfocusonHIVSTIsubstanceabuseandimplementationscience
ElizabethStuart
AssociateDeanforEducationandProfessorDepartmentsofMentalHealthBiostatisticsandHealthPolicyand
JohnsHopkinsBloombergSchoolofPublicHealth
Educationsocialpolicystatisticalmethods
estuartjhuedu
Resources
66
Management
Design-ThinkingResources
EngagementGuideforCivicTech
Journeymap
PerformanceManagementResources
DenverPeakAcademy
Proxymeasuresguide
Talkingaboutperformancewithoutcomplicatingit(video)
SettingPerformanceTargetsguide
Datavisualizationguide
Performancemanagementgettingstartedguide
Exercises
Fishboneexercise
5Whys
AdministrativeData
TheLessonsofAdministrativeData(JPAL)
EvaluationResources
Quantitativeandqualitativemethods(USDepartmentofHealthandHumanServicesAgencyforHealthcareResearchandQuality)
KeyStepsinOutcomeManagement(UrbanInstitute)
RCT-YES
Sageresearchmethods
BestPracticesforMixedMethodsResearchintheHealthSciences
ACCORDChecklist
Pilotandimplementationplanning
Benchmarkingexamplesandproxymeasures
Seattleopendataplaybook
Resources
67
Datasciencescopingguide
Resources
68
AboutGovExGovExhelpsgovernmentsusedataandevidencetoimproveresidentsrsquolivesWesupportgovernmentsinbuildingtheirinternalcapacitytooperatesustainableanalyticspracticeswhichimprovecommunityoutcomesinfieldsfrompublicsafetytoeconomicdevelopmentGovExhasaproventrackrecordofworkinginconcertwithpartnerstoprovidetechnicalassistanceandtrainingatscale
SinceourlaunchinApril2015GovExhas
Trainedover2000governmentemployeesinpersonandonlineintopicsincludingdatamanagementperformancemanagementopendatacommunityengagementandanalytics
Providedtechnicalassistancetomorethan100localgovernmentsacrosstheglobeimprovingtheiruseofdatatoachieveefficienciesinservicedeliveryandbetteroutcomesforresidents
Assessedthecapacityofmorethan130governmentalorganizationstodeterminehowleadersstaffandexternalstakeholdersmanageandusedata
Localgovernmentsareincreasinglyonthefrontlinesofaddressinglargescaleproblemswithglobalimplicationssuchasclimatechangepublicsafetyandeconomicinequality
Whileevidence-basedinterventionstohelpsolvethoseproblemsareavailablelocalgovernmentstaffhavelimitedtimeandcapacitytofindandimplementresearch-basedsolutionsAsaresultcitiescontinuetoworkthewaytheyalwayshave--makingincrementalimprovementstoprogramsbutnotalwayscreatingchangeatasystem-level
ThroughresourcedevelopmentandcollaborationwithJHUfacultyGovExissupportingcityleadersinusingdataandevidencetomakedecisionsandcreatinglastingrelationshipswithresearchers
TheRoadmapwasdevelopedbyGovExaspartofitspartnershipwithBloombergPhilanthropiesWhatWorksCitiesinitiative
AboutGovEx
69
- Title page
- Introduction and purpose of the Roadmap
- Using the Roadmap
-
- Gather information about the problem
- Strategically engage key partners
- Review the evidence and collect details
- Decide on a policy to implement based on local factors
- Establish a framework to monitor progress
- Assess data accessibility quality and security
- Design implementation and evaluation
- Share your story and success
-
- Working with a range of evidence
- Thank you
- References
- Resources
- About GovEx
-
UniversityPartners
Name Title Affiliation Area(s)ofexpertise
Contactinformation
Examplesofworkingwithnon-academicdecisionmakers
WendyPBarreno
AssociateDirector
WilsonSheehanLabforEconomicOpportunitiesUniversityofNotreDame
LEOsfocusareasincludehealtheducationhousingchildwelfareeconomicopportunitycriminaljustice
wbarrenonduduorleondedu
MelissaDavey-RothwellPhD
AssociateScientist
JohnsHopkinsBloombergSchoolofPublicHealth
Health mdavey1jhuedu410-614-5854
BaltimoreCityHealthDepartmentLeadersMarylandDepartmentofHealthAddictionRecoveryServicesinDCExecutiveDirectorsofcommunityorganizationsacrosstheUSthatfocusonHIVSTIsubstanceabuseandimplementationscience
ElizabethStuart
AssociateDeanforEducationandProfessorDepartmentsofMentalHealthBiostatisticsandHealthPolicyand
JohnsHopkinsBloombergSchoolofPublicHealth
Educationsocialpolicystatisticalmethods
estuartjhuedu
Resources
66
Management
Design-ThinkingResources
EngagementGuideforCivicTech
Journeymap
PerformanceManagementResources
DenverPeakAcademy
Proxymeasuresguide
Talkingaboutperformancewithoutcomplicatingit(video)
SettingPerformanceTargetsguide
Datavisualizationguide
Performancemanagementgettingstartedguide
Exercises
Fishboneexercise
5Whys
AdministrativeData
TheLessonsofAdministrativeData(JPAL)
EvaluationResources
Quantitativeandqualitativemethods(USDepartmentofHealthandHumanServicesAgencyforHealthcareResearchandQuality)
KeyStepsinOutcomeManagement(UrbanInstitute)
RCT-YES
Sageresearchmethods
BestPracticesforMixedMethodsResearchintheHealthSciences
ACCORDChecklist
Pilotandimplementationplanning
Benchmarkingexamplesandproxymeasures
Seattleopendataplaybook
Resources
67
Datasciencescopingguide
Resources
68
AboutGovExGovExhelpsgovernmentsusedataandevidencetoimproveresidentsrsquolivesWesupportgovernmentsinbuildingtheirinternalcapacitytooperatesustainableanalyticspracticeswhichimprovecommunityoutcomesinfieldsfrompublicsafetytoeconomicdevelopmentGovExhasaproventrackrecordofworkinginconcertwithpartnerstoprovidetechnicalassistanceandtrainingatscale
SinceourlaunchinApril2015GovExhas
Trainedover2000governmentemployeesinpersonandonlineintopicsincludingdatamanagementperformancemanagementopendatacommunityengagementandanalytics
Providedtechnicalassistancetomorethan100localgovernmentsacrosstheglobeimprovingtheiruseofdatatoachieveefficienciesinservicedeliveryandbetteroutcomesforresidents
Assessedthecapacityofmorethan130governmentalorganizationstodeterminehowleadersstaffandexternalstakeholdersmanageandusedata
Localgovernmentsareincreasinglyonthefrontlinesofaddressinglargescaleproblemswithglobalimplicationssuchasclimatechangepublicsafetyandeconomicinequality
Whileevidence-basedinterventionstohelpsolvethoseproblemsareavailablelocalgovernmentstaffhavelimitedtimeandcapacitytofindandimplementresearch-basedsolutionsAsaresultcitiescontinuetoworkthewaytheyalwayshave--makingincrementalimprovementstoprogramsbutnotalwayscreatingchangeatasystem-level
ThroughresourcedevelopmentandcollaborationwithJHUfacultyGovExissupportingcityleadersinusingdataandevidencetomakedecisionsandcreatinglastingrelationshipswithresearchers
TheRoadmapwasdevelopedbyGovExaspartofitspartnershipwithBloombergPhilanthropiesWhatWorksCitiesinitiative
AboutGovEx
69
- Title page
- Introduction and purpose of the Roadmap
- Using the Roadmap
-
- Gather information about the problem
- Strategically engage key partners
- Review the evidence and collect details
- Decide on a policy to implement based on local factors
- Establish a framework to monitor progress
- Assess data accessibility quality and security
- Design implementation and evaluation
- Share your story and success
-
- Working with a range of evidence
- Thank you
- References
- Resources
- About GovEx
-
Management
Design-ThinkingResources
EngagementGuideforCivicTech
Journeymap
PerformanceManagementResources
DenverPeakAcademy
Proxymeasuresguide
Talkingaboutperformancewithoutcomplicatingit(video)
SettingPerformanceTargetsguide
Datavisualizationguide
Performancemanagementgettingstartedguide
Exercises
Fishboneexercise
5Whys
AdministrativeData
TheLessonsofAdministrativeData(JPAL)
EvaluationResources
Quantitativeandqualitativemethods(USDepartmentofHealthandHumanServicesAgencyforHealthcareResearchandQuality)
KeyStepsinOutcomeManagement(UrbanInstitute)
RCT-YES
Sageresearchmethods
BestPracticesforMixedMethodsResearchintheHealthSciences
ACCORDChecklist
Pilotandimplementationplanning
Benchmarkingexamplesandproxymeasures
Seattleopendataplaybook
Resources
67
Datasciencescopingguide
Resources
68
AboutGovExGovExhelpsgovernmentsusedataandevidencetoimproveresidentsrsquolivesWesupportgovernmentsinbuildingtheirinternalcapacitytooperatesustainableanalyticspracticeswhichimprovecommunityoutcomesinfieldsfrompublicsafetytoeconomicdevelopmentGovExhasaproventrackrecordofworkinginconcertwithpartnerstoprovidetechnicalassistanceandtrainingatscale
SinceourlaunchinApril2015GovExhas
Trainedover2000governmentemployeesinpersonandonlineintopicsincludingdatamanagementperformancemanagementopendatacommunityengagementandanalytics
Providedtechnicalassistancetomorethan100localgovernmentsacrosstheglobeimprovingtheiruseofdatatoachieveefficienciesinservicedeliveryandbetteroutcomesforresidents
Assessedthecapacityofmorethan130governmentalorganizationstodeterminehowleadersstaffandexternalstakeholdersmanageandusedata
Localgovernmentsareincreasinglyonthefrontlinesofaddressinglargescaleproblemswithglobalimplicationssuchasclimatechangepublicsafetyandeconomicinequality
Whileevidence-basedinterventionstohelpsolvethoseproblemsareavailablelocalgovernmentstaffhavelimitedtimeandcapacitytofindandimplementresearch-basedsolutionsAsaresultcitiescontinuetoworkthewaytheyalwayshave--makingincrementalimprovementstoprogramsbutnotalwayscreatingchangeatasystem-level
ThroughresourcedevelopmentandcollaborationwithJHUfacultyGovExissupportingcityleadersinusingdataandevidencetomakedecisionsandcreatinglastingrelationshipswithresearchers
TheRoadmapwasdevelopedbyGovExaspartofitspartnershipwithBloombergPhilanthropiesWhatWorksCitiesinitiative
AboutGovEx
69
- Title page
- Introduction and purpose of the Roadmap
- Using the Roadmap
-
- Gather information about the problem
- Strategically engage key partners
- Review the evidence and collect details
- Decide on a policy to implement based on local factors
- Establish a framework to monitor progress
- Assess data accessibility quality and security
- Design implementation and evaluation
- Share your story and success
-
- Working with a range of evidence
- Thank you
- References
- Resources
- About GovEx
-
Datasciencescopingguide
Resources
68
AboutGovExGovExhelpsgovernmentsusedataandevidencetoimproveresidentsrsquolivesWesupportgovernmentsinbuildingtheirinternalcapacitytooperatesustainableanalyticspracticeswhichimprovecommunityoutcomesinfieldsfrompublicsafetytoeconomicdevelopmentGovExhasaproventrackrecordofworkinginconcertwithpartnerstoprovidetechnicalassistanceandtrainingatscale
SinceourlaunchinApril2015GovExhas
Trainedover2000governmentemployeesinpersonandonlineintopicsincludingdatamanagementperformancemanagementopendatacommunityengagementandanalytics
Providedtechnicalassistancetomorethan100localgovernmentsacrosstheglobeimprovingtheiruseofdatatoachieveefficienciesinservicedeliveryandbetteroutcomesforresidents
Assessedthecapacityofmorethan130governmentalorganizationstodeterminehowleadersstaffandexternalstakeholdersmanageandusedata
Localgovernmentsareincreasinglyonthefrontlinesofaddressinglargescaleproblemswithglobalimplicationssuchasclimatechangepublicsafetyandeconomicinequality
Whileevidence-basedinterventionstohelpsolvethoseproblemsareavailablelocalgovernmentstaffhavelimitedtimeandcapacitytofindandimplementresearch-basedsolutionsAsaresultcitiescontinuetoworkthewaytheyalwayshave--makingincrementalimprovementstoprogramsbutnotalwayscreatingchangeatasystem-level
ThroughresourcedevelopmentandcollaborationwithJHUfacultyGovExissupportingcityleadersinusingdataandevidencetomakedecisionsandcreatinglastingrelationshipswithresearchers
TheRoadmapwasdevelopedbyGovExaspartofitspartnershipwithBloombergPhilanthropiesWhatWorksCitiesinitiative
AboutGovEx
69
- Title page
- Introduction and purpose of the Roadmap
- Using the Roadmap
-
- Gather information about the problem
- Strategically engage key partners
- Review the evidence and collect details
- Decide on a policy to implement based on local factors
- Establish a framework to monitor progress
- Assess data accessibility quality and security
- Design implementation and evaluation
- Share your story and success
-
- Working with a range of evidence
- Thank you
- References
- Resources
- About GovEx
-
AboutGovExGovExhelpsgovernmentsusedataandevidencetoimproveresidentsrsquolivesWesupportgovernmentsinbuildingtheirinternalcapacitytooperatesustainableanalyticspracticeswhichimprovecommunityoutcomesinfieldsfrompublicsafetytoeconomicdevelopmentGovExhasaproventrackrecordofworkinginconcertwithpartnerstoprovidetechnicalassistanceandtrainingatscale
SinceourlaunchinApril2015GovExhas
Trainedover2000governmentemployeesinpersonandonlineintopicsincludingdatamanagementperformancemanagementopendatacommunityengagementandanalytics
Providedtechnicalassistancetomorethan100localgovernmentsacrosstheglobeimprovingtheiruseofdatatoachieveefficienciesinservicedeliveryandbetteroutcomesforresidents
Assessedthecapacityofmorethan130governmentalorganizationstodeterminehowleadersstaffandexternalstakeholdersmanageandusedata
Localgovernmentsareincreasinglyonthefrontlinesofaddressinglargescaleproblemswithglobalimplicationssuchasclimatechangepublicsafetyandeconomicinequality
Whileevidence-basedinterventionstohelpsolvethoseproblemsareavailablelocalgovernmentstaffhavelimitedtimeandcapacitytofindandimplementresearch-basedsolutionsAsaresultcitiescontinuetoworkthewaytheyalwayshave--makingincrementalimprovementstoprogramsbutnotalwayscreatingchangeatasystem-level
ThroughresourcedevelopmentandcollaborationwithJHUfacultyGovExissupportingcityleadersinusingdataandevidencetomakedecisionsandcreatinglastingrelationshipswithresearchers
TheRoadmapwasdevelopedbyGovExaspartofitspartnershipwithBloombergPhilanthropiesWhatWorksCitiesinitiative
AboutGovEx
69
- Title page
- Introduction and purpose of the Roadmap
- Using the Roadmap
-
- Gather information about the problem
- Strategically engage key partners
- Review the evidence and collect details
- Decide on a policy to implement based on local factors
- Establish a framework to monitor progress
- Assess data accessibility quality and security
- Design implementation and evaluation
- Share your story and success
-
- Working with a range of evidence
- Thank you
- References
- Resources
- About GovEx
-