karen shakman evelopment center
TRANSCRIPT
LOGICMODELSTOSUPPORTPROGRAMDESIGN,IMPLEMENTATIONANDEVALUATION
AWORKBOOKCREATEDBY
THEREGIONALEDUCATIONALLABORATORY‐NORTHEASTANDISLANDS
KARENSHAKMAN
EDUCATIONDEVELOPMENTCENTER
ISBE21STCCLCPROGRAMEVALUATIONWEBINAR
APRIL15,2014
PRESENTEDBY:KATRINABLEDSOE,JOSHUACOX,LESLIEGOODYEAR
ANDSHEILARODRIGUEZ
EDC
ProgramandPolicyEvaluationWorkshop
2
TableofContents
IntroductiontoLogicModelstoSupportProgramDesign,Implementation,andEvaluation.................................................................................................................................................................................3Whatisthepurposeofthiswork?..........................................................................................................................3Whytheworkshop?......................................................................................................................................................3
LearningaboutLogicModels........................................................................................................................4AnnotatedAgenda........................................................................................................................................................4
Purpose................................................................................................................................................................6
Agenda..................................................................................................................................................................7
IntroducingtheCase........................................................................................................................................8CaseStudyBrainstorm..............................................................................................................................................10
WhatIsaLogicModel?..................................................................................................................................11TypesofLogicModels...............................................................................................................................................12TheoryApproachModels......................................................................................................................................................12ActivitiesApproachModels..................................................................................................................................................12OutcomesApproachModels.................................................................................................................................................13
Inputs‐Strategies/Activities‐Outputs‐Outcomes.............................................................................................13
ElementsofaLogicModel...........................................................................................................................15ProblemStatement.....................................................................................................................................................15Outcomes.......................................................................................................................................................................17OutcomesChecklist..................................................................................................................................................................19
StrategiesandActivities...........................................................................................................................................20AWordAboutOutputs..............................................................................................................................................22Resources(inputs).....................................................................................................................................................22Assumptions.................................................................................................................................................................24Context............................................................................................................................................................................24
TheLogicinaLogicModel...........................................................................................................................27TheTheoryEmbeddedintheModel:If‐ThenStatements...........................................................................27
NextStepsandClosingWords....................................................................................................................29FinalThoughtsonLogicModels............................................................................................................................29
AppendixA:SimpleLogicModelTemplate...........................................................................................31
AppendixB:CollegeReadySampleLogicModel.................................................................................32
AppendixB:ResourcesforAdditionalExamples................................................................................33LogicModelResources..............................................................................................................................................33EvaluationResources................................................................................................................................................33
ProgramandPolicyEvaluationWorkshop
3
IntroductiontoLogicModelstoSupportProgramDesign,Implementation,andEvaluation
Whatisthepurposeofthiswork?Beginningin2012thetenRegionalEducationalLaboratorieshaveundertakenbuildingandsupportingresearchalliancesontopicsofimportancetostatesorlocalschooldistricts.Someexamplesofthesetopicsincludeeducatoreffectivenessandsupportingallstudentstobecomereadyforcollegeandcareers.Eachofthesealliancesestablishedaresearchagendathatincludesquestionsthatwillguidetheirworkoverthenextthreetofiveyears.TheRegionalEducationalLaboratoryNortheastandIslands(REL‐NEI)hasbeeninvolvedinworkwithasetofeightresearchalliancesthatareintendedtobesustainedcollaborationsamongresearchers,administrators,policymakers,andpractitioners.Thesealliancesfocusonaparticularpriorityforthepurposeofincreasingstateandlocalcapacitytousedataandresearchtoinformdecision‐makinginthatpriorityarea.Wedevelopedthisworkshoptobuildstateanddistrictleaders’capacitytodesign,implement,andevaluateprogramsandpoliciesthataddresssomeofthemostpressingeducationalissues.
Whytheworkshop?Thisworkshopwasdevelopedtoassistgroups,likethealliancesasawholeaswellasmembersoftheallianceswithintheirowneducationalcontexts,tolearnaboutandbuildlogicmodelstosupporteffectiveprogramdesignsandevaluations.Basedonfeedbackfromalliancemembers,REL‐NEIlearnedthatmanyofourdistrict‐andstate‐basedmemberswouldliketobuildtheircapacitytodeveloplogicmodelsthatmaybeutilizedtobothevaluatetheirownprogramsaswellastoworkmoreeffectivelywithevaluatorswhomtheyengagetoconductevaluationsontheirbehalf.Thisworkshopisdesignedtoprovideaprimeronlogicmodeling,ausefultoolforprogramdesign,implementation,andevaluation,andutilizelogicmodelsasatoolforprogramevaluation.TheworkshopwaspresentedforthreedifferentREL‐NEIresearchalliancesin2013,intwodifferentformats.ThePuertoRicoDropoutPreventionResearchAllianceparticipatedinaface‐to‐face3hoursworkshopfocusedonsupportingthealliance’sefforttogenerateacommonvisionfordropoutpreventionwork.TheUrbanSchoolImprovementAllianceandtheNortheastEducatorEffectivenessResearchAlliancebothparticipatedinvirtualwebinarsforabroadaudienceofpractitionersinterestedindevelopingskillsandcapacitytodevelopandutilizelogicmodelstoincreaseindividualandgroupcapacitytodesignandevaluateprograms.
ProgramandPolicyEvaluationWorkshop
4
LearningaboutLogicModels
AnnotatedAgenda3Minutes5Minutes
WelcomeandOverviewIntroducingtheCase
Facilitatorintroductions,followedbyoverviewofagendaandpurposesofworkshop.Purposes: Tointroducelogicmodelsasaneffectivetoolfor
programorpolicydesign,implementation,andevaluation
Toreviewtheelementsofalogicmodel Todiscussthevalueoflogicmodelsfortheir
program
Facilitatorreviewsthecasethatwillbeusedasanexamplethroughouttheworkshop.Activity:Facilitatorthenintroducesinitialactivitytobrainstormthegoalsoftheprograminthecaseandthetypesofquestionsthatmightbeappropriatetoaskaboutthecase.Participantsentercommentsinvirtualchat.
15Minutes WhatisaLogicModel?
Facilitatorintroduceslogicmodelsasausefultoolforprogramdesign,implementationandevaluation.Facilitatordiscussesoverallideaofinputs‐outputs‐outcomesthatdriveslogicmodeldevelopment.Activity:Participantsrespondtomultiple‐choicequestionsaboutinputs,activities/strategies,outputs,andoutcomesfortheCollegeReadycase.
30Minutes ElementsofaLogicModel Facilitatorwillwalkthroughallelementsofalogicmodelinthissectionofthewebinar.
(1) ProblemStatement:Includesadiscussiononwhataproblemstatementisandthequestionsthatneedtobeconsideredindevelopinganappropriateproblemstatement.
(2) Outcomes:Includesdiscussionofshort‐tolong‐termoutcomesandimpacts,aswellasreviewofwhatoutputsareandhowtheydifferfromoutcomes.
(3) Strategiesandactivitiesisthenextelementof
ProgramandPolicyEvaluationWorkshop
5
themodel.Anexamplewillbegivenonwhatthestrategiesandactivitiesareforoneofthecases.
(4) Resources.Inthissectionmaterialandintangible
resourceswillbediscussed.
(5) Assumptions:Assumptionswillbebrieflydiscussed.
(6) Context:Howexternalfactorsimpactprograms
willbediscussed
7Minutes NextSteps&Closing Facilitatorclosesworkshopwithdiscussionofsomenextstepsfordevelopingalogicmodelandareminderofhowlogicmodelscansupporteffectiveprogramdesignandevaluation.Therewillbeaninvitationtobeintouchwithfurtherquestions.Activity:Participantsindicateanextstepfortheirworkinthechat.
ProgramandPolicyEvaluationWorkshop
6
Purpose ThepurposeoftheWorkshopistointroducelogicmodelsasatoolforeffectiveprogramorpolicydesign,implementation,andevaluation.Theworkshopwill: Introducelogicmodelsasaneffectivetoolforprogramandpolicydesign,implementation,
andevaluation; Reviewtheelementsofalogicmodel; Discusstheroleandvalueoflogicmodelsforyourprogram.
ProgramandPolicyEvaluationWorkshop
Page 7
Agenda3Minutes WelcomeandOverview
5Minutes IntroducingtheCase
15Minutes WhatisaLogicModel?
30Minutes ElementsofaLogicModel
7Minutes NextStepsandClosing
ProgramandPolicyEvaluationWorkshop
Page 8
IntroducingtheCase Whatfollowshereisonecase,basedonarealprogramorpolicythatweuseasanexamplethroughouttheworkshop.Thiscaseprovidesacommonlanguagefordiscussingallaspectsofthelogicmodel.CaseStudy:CollegeReadinessHighSchoolProgramCollegeReadyisaschool‐basedcollegeaccessprogramfor9th‐12thgradestudents.StudentsareidentifiedfortheprogrambasedonFreeandReducedLunchstatus,recommendationsfromschoolguidancecounselors,and/orrecommendationsfrom8thgradeEnglishandMathteachers.StudentsparticipateinmonthlymeetingsasagroupwiththeCollegeReadystaff,areprovidedwithone‐on‐onecounselingwithCollegeReadystaff,areassignedanadultmentorandapeermentor,andparticipateinaseriesofeveningandsummerworkshops.Inaddition,familiesmakeacommitmenttotheprogramandattendaseriesofworkshopsspecificallydesignedtopreparethewholefamilyforthecollegeapplicationprocess.Thegoaloftheprogramistosignificantlyincreasecollegeattendanceamongthelow‐incomestudents.Thepurposeofthiscaseandtheactivityassociatedwithitistoprovidesomecommonexamplesforuseasyouworkthroughtheexercisesintheworkshop.
ProgramandPolicyEvaluationWorkshop
Page 9
Activity: Discussion of Case
1. Individually
Directions:Workingonyourown,considertheCollegeReadycaseabove.Whatarethegoalsoftheprogram?Whatmightwewanttoknowaboutit?
2. LargeGroup“Discussion”
Directions:Whatareyourideasaboutthegoalsoftheprogramandwhatyoumightwanttoknowaboutit?
ProgramandPolicyEvaluationWorkshop
Page 10
CaseStudyBrainstormWorkindependentlyfortwominutestolistthepotentialgoalsandquestionsthatwemighthaveabouttheprogram.Considerquestionsofimplementation,effectiveness,andimpact.Whatarethegoalsoftheprogram/policy? Whatdowewanttoknow abouttheprogram/policy?
ProgramandPolicyEvaluationWorkshop
Page 11
WhatIsaLogicModel? Inthissectionoftheworkshopweprovideaprimeronlogicmodels,includingdifferenttypesoflogicmodelsandtheirpotentialpurposes.Inmostbasicterms,logicmodelsprovideakindofmapforaprogramorinitiative,helpingtoclarifyaprogramorpolicy’sdestination,thepathwaystowardthedestinationandmarkersalongtheway.Consider: Whereareyougoing? Howwillyougetthere? Whatwilltellyouyou’vearrived?
Logicmodelsprovideasimplifiedpictureoftherelationshipsbetweentheprogramorpolicyinputs(resources,strategies,activities)andthedesiredoutcomesoftheprogram.Logicmodelspresentatheoryofactionorchangethatdrivestheprogramorpolicy,andmakesexplicitanyassumptionsaboutboththeresourcesatthedisposaloftheprogramaswellastherationalebehindtheeffort.Alogicmodelisvaluableinsupporting: ProgramPlanning ProgramImplementation ProgramMonitoring ProgramEvaluation
Whyusealogicmodel?Alogicmodel: Bringsdetailtobroadgoals; Helpsidentifygapsinprogramlogicandclarifyassumptions; Buildsunderstandingandpromotesconsensus; Makesexplicitunderlyingbeliefs; Helpsclarifywhatisappropriatetoevaluateandwhen; Summarizescomplexprogramsforeffectivecommunication.
Alogicmodelisveryusefulindesigningprogramandpolicyevaluation,asalogicmodelhelpstoclarifybothwhattheprogram,initiative,orpolicyisandwhatitisn’t.Thiskindofclarificationisveryhelpfulinbuildinganevaluationdesignthatcancapturetheprogram’sorpolicy’sinfluenceandimpact.Whatarethelimitationsofalogicmodel?
ProgramandPolicyEvaluationWorkshop
Page 12
Alogicmodelisnotafullydevelopedplanfordesigningormanagingaprogramorpolicy; Alogicmodelisnotanevaluationplan.
Whilelogicmodelsareusefultoolsforbuildingprogramplansorevaluationdesigns,additionalworkisnecessarytocreatebothprogrammaticandevaluationplans.
TypesofLogicModelsNotalllogicmodelsarethesame,noraretheydesignedforthesamepurpose.Justaslogicmodelsmayaidinprogramdesign,implementation,andevaluation,thetypeofmodeldevelopedvariessomewhatbasedonitspurpose.Therearethreemaintypesoflogicmodels: TheoryApproachModel ActivitiesApproachModel OutcomesApproachModel
Theory Approach Models Logicmodelsthatdescribetheoveralltheoryofchangeprovidea“bigpicture”oftheprogramandmaybeusefulforprogramdesignandoverallcommunicationoftheprogramtheory.Thesemodelsprovideacleardescriptionofwhythedevelopersbelievetheprogramorpolicywillbeeffectiveinachievingthegoals.Forexample,intheexampleofcollegereadiness,atheoryapproachlogicmodelmighthelpclarifytheassumptionsimplicitinthepushforprogramsforlow‐incomestudentsandfamilies,anddescribetherelationshipbetweeninitiatingthecollegegoingprocessandtheexpectedoutcomesforstudentswhoparticipate.Consider: Whatmightbethelogicinatheoryapproachmodelforyourownprogramorpolicy?(The
“bigpicture”theoryofchangeaboutyourinitiative?)
Activities Approach Models TheActivitiesApproachmodelismostfocusedonlayingoutthespecificstrategiesandactivitiesassociatedwithaprogram.Thesemodelscloselyexaminetherelationshipamongtheactivities,consideringquestionsofsequenceandtimingofimplementation,aswellashowtheseactivitieslinktooutcomes.Thistypeoflogicmodelismostusefulinprogramimplementation,monitoring,andmanagement.IntheCollegeReadyexample,thistypeoflogicmodelwouldconsiderthedifferentelementsoftheprogramandhowtheywouldbeoptimallyorderedandmanaged.Forexample,whatrolewouldthedifferentmentoringcomponentshave?Howwouldtheyrelatetooneanother?Inthistypeofmodel,relationshipsamongvariablesaremadeexplicitwitharrows,concentriccircles,andothergraphicrepresentationsofrelationships.Consider: Whyconsiderthesequenceandrelationshipamongactivitiesinalogicmodel?Howmight
thathelpyou?
ProgramandPolicyEvaluationWorkshop
Page 13
Outcomes Approach Models OutcomesApproachmodelsaremostusefulforprogramevaluation.Theyconsiderthestrategiesandactivitiesastheyrelatetothedesiredresultsofaprogramorpolicy.Inthesemodels,thefocusisonoutcomes,andtheyoftendividetheoutcomesintoshort‐termandlong‐termoutcomes,andimpacts.Atheoryofchangedrivesthesemodelsjustasitdoestheothers.However,inanoutcomesapproachlogicmodel,theemphasisisonexaminingtheoutcomesandmakingthecasethattheprogramorpolicyisresponsibleforthedesiredoutcomes.Consider: Whydivideoutcomesintoshort‐andlong‐term?Whatisthedifferencebetweenoutcomes
andimpacts?
Inputs‐Strategies/Activities‐Outputs‐OutcomesInitssimplestform,alogicmodelisagraphicrepresentationoftherelationshipamongaprogram’sorpolicy’sinputs(what’sinvestedintheprogram);thestrategies/activities;theoutputs(what’sdonewiththeseinvestments);andwhattheoutcomesare(whatresults).Takeaverysimpleexample:youhaveaheadacheandyouwantittogoaway. Whatistheinput?
o Quiettimeo Watero Ahotcompresso Twoaspirin
Whatisthestrategy/activity?(e.g.whatdoyoudowiththeinputs?)
o Sitquietlyfor5minuteso Drinkafullglassofwatero Puthotcompressono Takeaspirin
Whatistheoutput?
o Youarehydratedo Youarestartingtofeelmorerelaxed
What’stheoutcome?
o Youarerelaxedo Yourheadachegoesawayo Youareabletoreturntoyourworko Youarehappy
ProgramandPolicyEvaluationWorkshop
Page 14
Activity:Inputs‐Strategies/Activities‐Outputs‐Outcomes Nowtakethecaseexampleabove.Considertheinputs,strategies/activities,outputs,andoutcomes.Directions:ReviewtheCollegeReadycaseandincludethepossibleinputs,strategies/activities,outputs,andoutcomesbelow.
Case:___________________________________________
Inputs
Strategies/Activities Outputs Outcomes
ProgramandPolicyEvaluationWorkshop
Page 15
ElementsofaLogicModelThepurposeofthissectionistoprovideanoverviewof—andopportunitiestopractice—allelementsofalogicmodel.Theelementsofalogicmodelarelistedbelow.Inthepagesthatfollow,wewilldelineatetheseelementsanduseexamplesfromthecase. ProblemStatement Resources(inputs) StrategiesandActivities Outputs Short‐termOutcomes Long‐termOutcomes Impact Assumptions Context
ProblemStatementTheproblemstatementistheproblemorchallengeyoufacethattheprogramorpolicyisdesignedtoaddress. Consider:
o Whatistheproblem/issue?o Whyisthisaproblem?(Whatcausestheproblem?)o Forwhom(individual,household,group,community,societyingeneral)doesthis
problemexist?o Whohasastakeintheproblem?(Whocareswhetheritisresolvedornot?)o Whatdoweknowabouttheproblem/issue/peoplethatareinvolved?Whatresearch,
experiencedowehave?Whatdoexistingresearchandexperiencesay?
Finally,askyourself:istheproblemtoobig?Isittoosmall?Yourfinalproblemstatementshouldbetargetedandspecific,butitshouldnotbeasimplerestatementoftheprogramasaneed.Forexample,inthecollegereadycase,youwouldnotwanttheproblemtoread:“Lowincomestudentsarenotattendingcollegeincomparisontotheirmiddleclasspeers”Inthisexample,theproblemisreallyastatementofthelackoftheprogram.Rather,youwanttheproblemtoaddresstherealissuesunderlyingtheneedfortheprogram,suchas“studentslackknowledgeonthecollegegoingprocess”.
ProgramandPolicyEvaluationWorkshop
Page 16
Activity: Problem Statement Considertheproblemstatementmostappropriatetothechallengeyoufaceinyourwork,relatedtoaprogramyouhaveinplaceoroneyouwouldliketoinitiate.Brainstormkeyideasassociatedwiththerelevantproblem.Modelyourbrainstormaftertheexamplebelow,intermsofbrevity.Example:CollegeReadyCase
o Lowincomestudents’collegegoingratesareverylowo Lowincomestudentshavemorelimitedopportunitieso Therearehigherratesofunemploymentamonglowincomeworkerso Lowincomeworkersearnlesswagesthroughouttheirlifetimethantheirmiddleclass
counterparts.
YourBrainstorm:____________________
ProgramandPolicyEvaluationWorkshop
Page 17
OutcomesWhileoutcomesarenotthenextitemoneseeswhenonelooksfromlefttorightacrossatraditionallogicmodel,theyarealogicalnextsteptodiscusswhenexaminingtheelementsofalogicmodel.Outcomesask,“Whatdifferencedoesitmake?”Inotherwords,whatisthedifferencethattheresources,andstrategiesandactivities,takentogether,haveonthevariousparticipantsintheseefforts?Outcomesusuallycomeinstagesandfallalongacontinuumfromshort‐tolong‐termoutcomes.Thelanguagetodescribetheseoutcomesvaries,butforthepurposesofthisworkbook,weuseshort‐termoutcomes,long‐termoutcomes,andimpact.Othertermsyoumayencounterinclude:
o Short‐term:initial,immediate,proximalo Long‐term:medium‐term,intermediate,midpointo Impact:long‐term,final,ultimate,distaloutcome
Short‐termoutcomes:Short‐termoutcomesarethemostimmediateandmeasurable
resultsforparticipantsthatcanbeattributedtothestrategiesandactivities.Forexample,aprogramthatpromotesincreasedparentengagementinstudents’collegeplanningmighthaveashort‐termgoalofincreasedparentalparticipationintheprovidedparentsessions.
Long‐termoutcomes:Long‐termoutcomesarethemoredistant,thoughanticipatedresultsofparticipationinthecollectionofstrategiesandactivities.Whenitcomestoshort‐andlong‐termoutcomes,itisgoodtothinkabouttheoveralltimeframefortheprogram.Sometimes,short‐termisconsideredtobeasshortas6monthsoraslongas3years.Long‐termmightbe2yearsoraslongas6years.Theimportantpointhereistoconsidertheprogramandidentifythetimeframe,specifictotheinitiative,forshorter‐andlonger‐termoutcomes.
Impact:Whenweusetheterm“impact”,wemeanthedesiredoutcomesthatoccurasa
resultoflong‐termimplementationofthestrategiesandactivities.Thesemorelong‐rangegoalsaredependentonsomeconditionsthatgobeyondtheprogram’sscopeofstrategies.Thesemaybecalledthe“blueskies”orthebigpicturetypesofobjectivesfortheprogramandonesthataremoredistantfromtheactualstrategiesandactivities,andlesswithinthecontroloftheprogramorpolicytorealize.Oftentheseareconsideredtobe7‐10yearsoutfrominitialimplementation.
ProgramandPolicyEvaluationWorkshop
Page 18
Activity: Focus on Outcomes Beingclearaboutprogramoutcomesisessentialforbothfocusedprogramimplementationandforeffectiveevaluation.Thetablebelowisdesignedtopromoteastep‐by‐stepapproachtooutcomedevelopment.Thecolumnsare: Whoisthetarget?Whoisthegroupyouaretargetingwithyourstrategy?Isitstudents?
Parents?Aschool?Inthisexample,basedonthecollegereadycase,thetargetisparticipatinghighschoolseniorsinthreehighschoolsthatparticipateintheprogram.
Whatisthechangedesired?Heretheimportantideaistouseanactionverbtodemonstrateakindofchangeoranimpact.Forexample:increase,improve,engage…
Inwhat?Whatistheactivity,strategy,orprograminwhichthetargetpopulationisgoingtoenactthisdesiredchange?Whatistheresultingactioninwhichthetargetpopulationwillengagetoachievethegoal?
Bywhen?Hereiswhereyoubegintoclarifythetimelineforoutcomes.Isaparticularoutcomeashort‐termorlong‐termoutcome?
Directions:Foreachofthecolumnheaders(thetarget,changedesired,etc.),enteranexamplefromyourowncontext,relatedtoaprogramorpolicyinitiativeyouhaveinplaceorwouldliketodevelop.TheTarget Changedesired?(action
verb)Inwhat?(results) Bywhen?
Participatinghighschoolseniorsinthreeurbanhighschools
Increase Completedandsubmittedapplicationstopost‐secondaryinstitutions
ByJune2014
ProgramandPolicyEvaluationWorkshop
Page 19
Outcomes Checklist Considerthefollowingcriteriawhenexaminingyouroutcomes: Aretheoutcomesimportant?Aretheendoutcomesimportant?Dotheyrepresent
significantchangeorimprovementsthatarevaluedbyparticipantsandkeystakeholders?Outcomesmaybeachievablebutnotreallyworththeeffort.Applythe"Whocares?"test.
Aretheoutcomesreasonable?Aretheoutcomes(fromshort‐termtolong‐termtoimpact)connectedtooneanotherandlinkedinareasonableorder?Isitlikelythatonewillleadtothenext?Anotherwaytothinkaboutthisistoconsiderthe“if‐then”statements(orlogicstatements)embeddedinachainofoutcomes.Forexample,usingtheCollegeReadyexample,willincreasedparentparticipationinworkshopsoncollegereadinessleadtostudentscompletingmorecollegeapplications?Willmentoringstudentsincreasematriculatingincollege?Issuesofsequenceandtimingofactivitiesandintendedoutcomesisimportanttoconsider.
Aretheoutcomesrealistic?Aretheoutcomesthatyousuggestrealisticgiventhenatureoftheproblem,yourresources,andyourabilities?Willtheprogramleadtoorhelpcontributetotheseoutcomes?(Becarefultoensurethattheoutcomesarerealisticgiventhelevelofeffort.)Inotherwords,ifyoudeliveroneparenteducationclass,isitrealistictoexpectanincreaseinstudentachievement?Askhardquestionsabouttheoutcomesastheyrelatetotheactualprogramorpolicy.
Areyouattendingtounintentionalorpotentiallynegativeoutcomes?Finally,it’simportanttoanticipateandconsidertheunintendedorpotentiallynegativeoutcomesthatmightresultfromthesetofstrategiesandactivities.Whatarepotentialnegativeeffectsoftheprogramorpolicy?Whatelsemighthappenthatisdifferentfromwhatweintend?Or,howelsemightthesequenceofeventsunfold?Forexample,couldthereadinessprogramleadtostudentsnotapplyingtocollege?Consideringtheunintendedconsequencesallowsprogramandpolicydesignerstoconsiderhowtoprepareforthesepossibleoutcomes,andalsohelpsevaluatorstobeattunedtothesepossibleconsequencesintheevaluationdesign.
ProgramandPolicyEvaluationWorkshop
Page 20
StrategiesandActivitiesStrategiesandactivitiesaretheprogramcomponents,orthegameplanfortheprogramorpolicy.Thisisaninventoryofallthestrategiesandactivitiesdesignedtoachievetheoutcomes.However,itismorethanasimplelistingofactivities.Therearetwoquestionstoaskyourselfwhenyouinventorytheactivities,services,products,andeventsthatmakeuptheprogramorpolicy: Whatistheappropriatesequenceororderoftheseactivities?
ConsidertheCollegeReadycase:Itmaybeimportantthatthementoringelementoftheprogramcomepriortothedeliveryoftheparentworkshopseries.Orperhapstheseactivitiesshouldbeconcurrent.Considertheappropriateorderofactivitiesandhowtheyrelatetooneanother. Aretherecertainactivitiesthat,takentogether,adduptoakindofoverallstrategy?Do
certainactivities“bundle”or“cluster”together?
ConsidertheCollegeReadyexample:Perhapsthereareaseriesofworkshopsrelatedtothecollegegoing,suchashowtoappropriatelycompletecollegeapplications,fillingouttheFAFSA,andeducatingparentsontheprocessandwhatisrequiredofthem,thatbundletogetherasanoverarchingstrategy.Perhapsthisisthe“StudentandParenteducation”strategy.Thismaybedifferentfromotherstrategiesassociatedwiththeinitiative,suchasmentorshiporfamilyengagement.Creatingtheseclustersofactivitieshelpstostreamlinethelogicmodelandalsosupportsevaluation;theevaluationwillthenassessasetofstrategies,ratherthanindividualactivities.
ProgramandPolicyEvaluationWorkshop
Page 21
Activity: Strategies and Activities in Sequence Consideraseriesofactivitiesthatareapartofyourownwork.Listsomerelevantactivities,theirsequenceororderinwhichtheyaresupposedtooccur,andconsidertheoverarchingstrategywithinwhichtheseactivitiesfall.Inotherwords,doesyourchosenprogramorinitiativehaveacoresetofstrategiesthatguidetheactivities,events,programs,etc.thatyouprovide?(Note:thewebinarwillnotincludethisactivitybutthisissomethingyouareencouragedtodoonyourown.)Activities Sequence StrategyDevelopworkshopmaterialsforstudentreadinessprogram
1st Studentandparenteducation
Deliverstudentreadinessworkshops
2nd Studentandparenteducation
Recruitcommunitymentors
1st Mentorshipdevelopment
Activities Sequence Strategy
ProgramandPolicyEvaluationWorkshop
Page 22
AWordAboutOutputsSomelogicmodelsincludebothoutputsandoutcomesinthemodel.Outputsdifferfromoutcomesinthattheycapturedataaboutwhatwedoratherthanwhatweexpecttoachieveasaresultofwhatwedo.Outputscanbestbedescribedasactivitydataandareusefulfortrackingprogramimplementation.Outputsoftenprovidedetailaboutthebreadthandreachofthestrategiesandactivities.Outputscapturesizeandscope;theydescribeorcountstrategiesandactivities,suchasthenumberofparentsessionsdelivered,programparticipationrates,thenumberofmaterialsdevelopedordistributed,andsoforth.UsingtheCollegeReadyprogramasanexample,anotherwaytothinkaboutthedifferencebetweenoutputsandoutcomesistoconsiderthequestions:
Istheparenteducationprogrambeingdeliveredasintended?(outputquestion)
vs.Isthecollegeacceptancerateforparticipatingstudentsincreasing?(outcomequestion)
Onefinalword:itisimportantnottoconfuseoutputsforoutcomes.Aprogramthatisgoodatdeliveringactivitiesandservicesmayachieveitsoutputswithoutachievingitsoutcomes.Yet,itistheoutcomesthatmakethedifferenceinresponsetotheproblemidentified.
Resources(inputs)Resourcesincludeboththematerialandtheintangiblecontributionsthatareorcouldreasonablybeexpectedtobeavailabletoaddresstheproblem. Materialresourcesinclude:
o Moneyo Materialsandequipment
Intangibleresourcesinclude:
o Peopleo Timeo Partnershipso Other?
ProgramandPolicyEvaluationWorkshop
Page 23
Activity: Intangible Resources Consider:Whataretheintangibleresourcesatyourdisposal?Brainstormatleast5non‐monetaryresourcesthatareavailabletoyouinaprogramyouoperateormanage.Brainstorm:IntangibleResources(examplefromCollegeReadycase)
o Communitymentorso Localuniversityspaceforparentmeetingso Volunteercollegeadmissionsdirectorsforapplicationworkshopo Studentvolunteersforchildcareatparentmeetings
Brainstorm:IntangibleResources
ProgramandPolicyEvaluationWorkshop
Page 24
AssumptionsAssumptionsarethebeliefsweholdaboutparticipants,staff,andtheprogram,aswellasourassumptionsabouthowthechangeorimprovementwehopetoseemayberealized.Beingexplicitabouttheseassumptionsisoneofthefirstandmostimportantthingsyoucandoasyouconsiderprogramdesign,implementation,andevaluation.ConsidertheCollegeReadycase:theprogramassumesthatstudentswhoparticipateintheprogramwanttogotocollege,andfurtherassumesthatcollegeenrollmentwillleadtoabetterlifefortheirparticipants.Oftentheassumptionsembeddedinaprogramorpolicyiscriticaltothesuccessorfailureoftheoverallinitiative.Assumptionsmaybeinternal(assumptionsaboutparticipants,resources,andhowtheprogramwillfunction)orexternal(beliefsabouthowchangeoccurs,valuesembeddedintheprogram,orfindingsfrompriorresearch).Let’suseasimpleexample.Remembertheheadache?Youhadaheadache,youtriedafewthingstogetridofit(water,aspirin,etc.)andyoufeltbetter.Theoutcomewasthattheheadachewentaway.However,betweentheproblem(theheadache)andtheoutcome(noheadache)wereseveralassumptions.Forexample,youassumednoallergytoaspirin,thattherewasnoloudnoisepersistinginthebackground,andsoforth.Clarifyingandmakingexplicittheassumptionsbehindtheprogram,bothintermsofthespecificelementsrelatedtoimplementationandtheassumptionsembeddedinthetheorydrivingtheinitiative,iscriticaltothedevelopmentofathoughtfullogicmodel.
ContextContextisthesetofcircumstances,environments,policiesorfactsthatsurroundaparticularevent,situation,etc.Theseincludebutarenotlimitedto,history,culture,climate,socialstrata,region,andpoliticalcircumstances.Contextcandeterminehowtheprogramwillbeimplemented,howitwilloperate,howprogramparticipantswillreceiveit,andhowitwillbesustained.Let’sreturntotheCollegeReadycase.Theprogram’smainclientelearethosestudentswhoreceivefreeorreducedlunch,sosocio‐economicstatuswilllikelyeffecthowparentsandstudentsparticipate.Forinstance,iffamiliesarelowincome,theymighthavemultiplejobsthatmightimpactparticipation.Thisfactormightaffectthetimesofdaythattheprogramisoffered,aswellastheexpectationsforfamilyparticipation.Considertheaspirinexample.Itmightbeagoodideatotakeaspirintorelieveaheadache,butwhatisthecultureinyourhouseholdabouttakingtheoverthecountermedication?Isthisanormwithinyourfamily?Ifnot,youmaynotmakeuseofthatstrategytorelieveyourheadache.Whatstrategiesareculturallyappropriate?Clarifyingthecontextoftheprogramcanhelparticulateunderwhatcircumstancestheprogramwilloperateandbesuccessful.
ProgramandPolicyEvaluationWorkshop
Page 25
Activity: Uncovering Internal and External AssumptionsDirections:Consideryourprogramorinitiative.Brainstormtherangeofassumptionsembeddedintheprogramdesignandintheoveralltheoryofactiondrivingtheinitiative.InternalAssumptions ExternalAssumptions
ProgramandPolicyEvaluationWorkshop
Page 26
Activity: Identifying Contextual FactorsDirections:Consideryourprogramorinitiative.Brainstormexternalfactorsthatinfluencehowtheprogramisadministeredandhowitoperates.ContextualFactors HowOurProgramOperatestoAddressThem
ProgramandPolicyEvaluationWorkshop
Page 27
TheLogicinaLogicModelThepurposeofthissectionistounderstandthelogicembeddedinlogicmodelsandrecognizetheneedtoidentifythepossiblerelationshipsandconnectionsamongvariouselementsofthelogicmodel.
TheTheoryEmbeddedintheModel:If‐ThenStatementsUnderstandingtheseif‐thenrelationshipsisessentialtouncoveringthetheoryofactionortheoryofchangedrivingaprogramorpolicy.Ifyouhaveaccesstoandapplyresources,youwillbeabletodevelopprogramsthatwillbedesignedtoreachthetargetparticipants,andwhenyoureachthesepopulationswiththeprogramsorservicesthatyouhavedeveloped,thenyouwillbeabletomeettheunmetneedsandchangecircumstancesthatwillleadtosolvingtheproblemthatinitiatedthiswork.Tomakethismorereal,let’sconsidertheCollegeReadycase:
Ifwedevelopaseriesofstudentworkshopsfocusedonthecollegegoingprocess. Ifwedevelopaseriesofstudentworkshopsfocusedonthecollegegoingprocess,then
studentsmayattend. Ifstudentsattend,thentheywillhavemoreknowledgeonwhatittakestoapplytocollege. Iftheyhavemoreknowledgeonwhatittakestoapplytocollege,theymaybemorewillingto
considercollegeasanoption. Ifconsidercollegeasanoption,collegeapplicationsincrease.
Notethatthethenclauseinonestatementbecomestheifclausetheproceedingstatement.Thisisimportant;whenwechangethelanguagefromthethentotheifstatement,changesintheintentionofthestatementmayoccur.Insomelogicmodels,theseif‐thenstatementsarewrittenrightintothemodeltomakethetheoryofchangeexplicit.
ProgramandPolicyEvaluationWorkshop
Page 28
Activity: If‐Then Statements Directions:ConsidertheCollegeReadycase.Movethestatementsaroundtomakeaseriesoflogicalif‐thenstatementsbelow.Considerthesequencingofevents.Thestatementsbelowincludestrategiesandactivities,traditionaloutputs,andoutcomes.IF______________________________________________THEN/IF____________________________________________THEN/IF_______________________________________THEN/IF___________________________________________
THEN/IF_______________________________________THEN______________________________________________(1)Wedevelopaseriesofcollegereadinessworkshopsforparents.(2)Parentshelptheirstudentswiththeapplicationprocess.(3)Werecruitparentstoparticipateintheworkshops.(4)Parentsbetterunderstandthetimelinesanddemandsofthecollegeapplicationprocess.(5)Studentsmeetfinancialaidandcollegeapplicationdeadlines.(6)Parentsattendtheworkshops.
ProgramandPolicyEvaluationWorkshop
Page 29
NextStepsandClosingWordsInbuildingalogicmodel,itisimportanttoconsiderthefollowingquestions: DoIunderstandthedifferentelementsofthelogicmodelandhowtheydiffer? WhoshouldIconsultinthedevelopmentofthelogicmodel?Whatcolleaguesandstakeholders
shouldbeparticipantsinthedevelopmentofthelogicmodel? Whowillberesponsibleforseeingthisthrough? HowdoIknowIhavecapturedthetheoryofactionguidingtheprogram? Howwillweusethelogicmodelonceit’sdeveloped?
Activity: Discussion Directions:Takeamomenttoconsiderhowyoucanuselogicmodelsinyourwork.
Whoisusinglogicmodelsaroundyou? Howaretheybeingused? Whatmakesthemuseful?Whatdoyouseeintheoverviewoflogicmodelsherethatmightbe
helpfulorrelevanttoyourwork? Whatmoredoyouneedtomakethemusefulandeffectiveinyourwork?
FinalThoughtsonLogicModelsHereareafewquickremindersaboutwhatalogicmodelis,andwhatitisn’t.Alogicmodelis: Agraphicrepresentationofthetheoryofchangedrivingaprogramorpolicy; Aframeworkforplanning,implementation,andevaluation.
Alogicmodelisnot: Astrategicplan; Anevaluationdesign.
Whilealogicmodelisnotastrategicplanoranevaluationdesign,itcanbeusefulindevelopingeitherofthesemoredetailedresources.Alogicmodelislikelytobemuchmoreeffective,useful,andhonestiftheprocessofgeneratingthelogicmodelhasengagedabroadrangeofstakeholdersinthedesignprocess.Includingkeyvoicessuchasstaff,parents,students,funders,andothers,indiscussionsaboutprogramdesignandevaluationthatwillpromotethebuy‐inandongoingsupportoftheseparticipantsaswellasincreasetheauthenticityofthemodel.Logicmodelsshouldbelivingdocumentsthatarereferredtothroughoutthelifeoftheprogramandtheevaluation,andamendedasneeded.Theyarealsohelpfultoguideaprogramasitevolves,andensurethattheworkoftheprogramremainsfocusedonthekeygoalsandoutcomes.
ProgramandPolicyEvaluationWorkshop
Page 30
Logicmodelsareusefulforprogramevaluation,especiallywhenevaluationisconsideredinconcertwithcreatingthelogicmodelattheearlystagesofprogramdevelopment.Itismuchbettertoconsiderevaluationattheoutsetofaprogramorpolicy’sdevelopmentratherthanasanafterthoughtorhalfwaythroughprogramimplementation.Goodluckwiththisworkandpleasecontactuswithquestions!KatrinaBledsoe,EducationDevelopmentCenter,[email protected],EducationDevelopmentCenter,[email protected],EducationDevelopmentCenter,[email protected],EducationDevelopmentCenter,[email protected]
ProgramandPolicyEvaluationWorkshop
Page 31
AppendixA:SimpleLogicModelTemplateProblemStatement:Resources/Inputs Strategiesand
ActivitiesOutputs Short‐term
OutcomeLong‐termOutcomes
Impacts
Whatresourcesareorcouldreasonablybeavailable?
Whatwilltheactivities,events,etc.be?
Whatare theinitialproductsoftheseactivities?
Whatchangesareexpectedinshort‐term?
Whatchangesdoyouwanttoaftertheinitialoutcomes?
Whatarethehopedforchangesoverthelonghaul?
Assumptions:Context:
ProgramandPolicyEvaluationWorkshop
Page 32
AppendixB:CollegeReadySampleLogicModelProblemStatement:Low‐incomehighstudentsinselectedcommunitiesattendcollegeatalowerratethantheirmiddleclasspeers,leadingtomorelimitedopportunities,higherratesofunemployment,andlowerearnings.Resources/Inputs Strategiesand
ActivitiesOutputs Short‐term
OutcomeLong‐termOutcomes
Impacts
Whatresourcesareorcouldreasonablybeavailable?
Whatwilltheactivities,events,etc.be?
Whataretheinitialproductsoftheseactivities?
Whatchangesareexpectedinshort‐term?
Whatchangeswantedafterinitialoutcomes?
Whatarehopedforchangesoverlonghaul?
‐Partnershipwith3publichighschools‐Communitymentors‐Localuniversityspaceforparentmeetings‐Volunteercollegeadmissionsdirectorsforapplicationworkshop‐Studentvolunteersforchildcareatparentmeetings
‐Localcollegementorshipprogram‐Peermentors‐Studentreadinessprogram(workshops)‐Parenteducation(workshops)
‐Recruitadequate#ofmentorsforstudentcohort‐Developanddeliver12workshopsoncollegeapplicationprocess;SAT/ACT;FAFSA;collegelife‐Developanddeliver6workshopsforparents‐Highinterestandattendanceatallworkshopsforparentsandstudents.
‐Participatingstudentsapplytoatleastonecollegeontime‐Parentsreportincreasedunderstandingofthecollegeapplicationprocess‐Studentsreportincreasedreadinessforcollege‐ParticipatingstudentscompleteFAFSAformsontime
‐Participatingstudentsareacceptedtoandattendcollege,remainingenrolledintothe3rdsemesterofcollege‐ParticipatingstudentsGPAsabove3.0atcollege,intothe3rdsemester‐Increasedparentalengagementinparticipatinghighschools’studentseducation
‐Low‐incomestudentsinparticipatingcommunitiesattendcollegeatsamerateasmiddleclasspeers‐Low‐incomestudentsinparticipatingcommunitiesgraduatefromcollegeatsomerateasmiddleclasspeers‐Participatinghighschoolsseeincreaseinparentandstudentengagement‐Participatinghighschoolsstatetestscoresincreasebyx%
Assumptions:Collegeattendanceisdesiredgoalforparticipatingcommunities;highschoolleaderswillremainconsistentandsupportprogram;parentswillshowinterestandparticipateinprogram.Context:Collegepreparationisafamilyaffair;Targetingstudentsoflowsocio‐economicstatus;previouscollegereadinessprograms.
ProgramandPolicyEvaluationWorkshop
Page 33
AppendixB:ResourcesforAdditionalExamplesTherearemanyterrificresourcesavailableonlineforlogicmodelingandprogramandpolicyevaluation.Manyofthesewereusedinthedevelopmentofthisworkshop.Severaloftheresourcesbelowprovidelinkstoadditionalresources,alsoavailableonline.
LogicModelResourcesUniversityofWisconsinExtensionSchool:http://www.uwex.edu/ces/lmcourse/#TheUniversityofWisconsinExtensionSchoolhasseveralusefulresourcesrelatedtologicmodelsandevaluation.W.K.KelloggFoundationLogicModelDevelopmentGuide.Retrieved3/16/2013fromhttp://www.wkkf.org/knowledge‐center/resources/2006/02/wk‐kellogg‐foundation‐logic‐model‐development‐guide.aspxHarvardFamilyResearchProjectandCoffman,J.(1999).LearningfromLogicModels:AnExampleofaFamily‐SchoolPartnershipProgram.Retrieved3/16/2013fromhttp://www.hfrp.org/publications‐resources/browse‐our‐publications/learning‐from‐logic‐models‐an‐example‐of‐a‐family‐school‐partnership‐program InnovationNetwork,Inc.(dateunknown).LogicModelWorkbook.Retrieved3/18/2013fromhttp://www.innonet.org/client_docs/File/logic_model_workbook.pdf
EvaluationResourcesTheAdministrationforChildrenandFamilies,OfficeofPlanning,Research,andEvaluation(2010).TheProgramManager’sGuidetoEvaluation,SecondEdition.Retrieved3/16/2013fromhttp://www.acf.hhs.gov/programs/opre/resource/the‐program‐managers‐guide‐to‐evaluation‐second‐editionBond,S.,Boyd,S.,andRapp,K.(1997).TakingStock:APracticalGuidetoEvaluatingyourownPrograms.HorizonResearch,Inc.Retrieved3/16/2013fromhttp://www.gao.gov/products/GAO‐12‐208Ghttp://www.horizon‐research.com/reports/1997/taking_stock.phpCentersforDiseaseControl(August,2012).ImplementingEvaluation:LearningandGrowingThroughEvaluation.Retrieved3/16/2013fromhttp://www.cdc.gov/asthma/program_eval/guide.htm
ProgramandPolicyEvaluationWorkshop
Page 34
Chinman,M.,Imm,P.,andWandersman,A.(2004).GettingToOutcomes2004:PromotingAccountabilityThroughMethodsandToolsforPlanning,Implementation,andEvaluation.RandCorporation:SantaMonica,CA.Retrieved3/16/2013fromhttp://www.rand.org/pubs/technical_reports/TR101.html
Patton,M.Q.(1997).Utilization‐FocusedEvaluation:TheNewCenturyText.SAGEPublications:ThousandOaks,CA.UnitedStatesGovernmentAccountabilityOffice(January2012).DesigningEvaluations.Retrieved3/16/2013fromhttp://www.gao.gov/products/GAO‐12‐208GW.K.KelloggFoundationEvaluationHandbook.Retrieved3/16/2013fromhttp://www.wkkf.org/knowledge‐center/resources/2010/w‐k‐kellogg‐foundation‐evaluation‐handbook.aspxTheCDChasarangeofevaluationresources,someproducedattheCDCandsomefromothersources:http://www.cdc.gov/eval/resources/index.htmThesechecklists,createdbyTheEvaluationCenterattheWesternMichiganUniversitymaybeusefulinplanningandmonitoringevaluation:http://www.wmich.edu/evalctr/checklists/Theauthorwouldalsoliketoacknowledgethefollowingpeoplefortheirexpertiseandresourcesindevelopingthesematerials:KatrinaBledsoe,EducationDevelopmentCenter,Inc.LeslieGoodyear,EducationDevelopmentCenter,Inc.BrianLord,EducationDevelopmentCenter,Inc.AnneWang,EducationDevelopmentCenter,Inc.SheilaRodriguez,EducationDevelopmentCenter,Inc.