review of the community action fund (caf)€¦ · (caf). goss gilroy inc. (ggi) was hired by the...

PREPARED FOR: PublicHealthAgencyofCanada

(PHAC) PREPARED BY: GossGilroyInc.

ManagementConsultantsSuite900,150MetcalfeStreetOttawa,ONK2P1P1Tel:(613)230‐5577Fax:(613)235‐9592E‐mail:[email protected]

DATE: September14,2018

Review of the Community Action Fund (CAF)

Final Review Report

Table of Contents

ReviewoftheCommunityActionFund(CAF)i

List of Key Acronyms ....................................................................................... ii

Acknowledgements ........................................................................................ ii

1.0 Introduction .......................................................................................... 1 1.1 ObjectivesandScopeoftheReview.................................................................................1

1.2 Methodology...............................................................................................................................1

2.0 Description of the Community Action Fund ........................................... 2 2.1 Background.................................................................................................................................2

2.2 CAFObjectivesandPriorities.............................................................................................3

2.3 OverviewoftheCAFFundingSolicitation.....................................................................4

3.0 Feedback and Lessons Learned .............................................................. 4 3.1 CAFDesignandDevelopment............................................................................................5

3.2 FundingSolicitation................................................................................................................7

3.3 Communications.....................................................................................................................16

Appendix A: Summary of Survey Results ................................................... 18

Appendix B: Overview of CAF Solicitation Process ..................................... 22

UNCLASSIFIED / NON CLASSIFIÉ

ReviewoftheCommunityActionFund(CAF)ii

List of Key Acronyms

CAF CommunityActionFund

CCDIC CentreforCommunicableDiseasesandInfectionControl

CGC CentreforGrantsandContributions

Gs&Cs GrantsandContributions

LOI LetterofInterest

NACHA NationalAboriginalCouncilonHIV/AIDS

NCR NationalCapitalRegion

PHAC PublicHealthAgencyofCanada

PT Provincesandterritories

STBBI Sexuallytransmittedandblood‐borneinfections

Acknowledgements

ThereviewteamwouldliketothankthosewithinthePublicHealthAgencyofCanadawhosupportedthiswork.Theteamwouldalsoliketothanktheindividualswhotookthetimetocontributetothisreviewstudy,particularlyintervieweesandsurveyrespondentswhoprovidedinsightsandcommentsimportanttoidentifyinglessonslearned.

ReviewoftheCommunityActionFund(CAF)1

1.0 Introduction

ThepurposeofthisreportistosummarizethefindingsandidentifylessonslearnedfromareviewoftheprocessusedtodevelopandimplementtheHIVandHepatitisCCommunityActionFund(CAF).GossGilroyInc.(GGI)washiredbythePublicHealthAgencyofCanada(PHAC)toconductthisreviewin2017‐18.

1.1ObjectivesandScopeoftheReviewTheobjectivewastoconductareviewoftheprocessusedtodevelopandimplementCAFwithaviewofprovidingPHACwithlessonslearnedforfuturegrantsandcontributions(Gs&Cs)programdevelopmentandimplementation.

ThescopeofthereviewincludedthetimeperiodbeginninginMarch2012whenconsultationswereundertakenbyPHACtoinformthedevelopmentofCAF,tothefundingsolicitation1,andfinally,thesigningofcontributionagreements.Thereviewexaminedthreeareasspecifically:

EfficiencyandeffectivenessofthereviewandfeedbackprocessfortheLetterofIntent(LOI)andfullproposalprocess.

StakeholderEngagementinthedevelopment,designandimplementationofCAF,including: theextenttowhichstakeholderswereengagedintheprocess;and theextenttowhichPHACwasresponsivetostakeholderfeedback.

Communicationswithexternalstakeholdersincluding: timeliness,frequencyandclarityofcommunications.

1.2MethodologyThereviewofCAFwasconductedfortheCentreforCommunicableDiseasesandInfectionControl(CCDIC)atPHAC.CCDICassistedinidentifyingstakeholdersforthereview,aswellasprovidingdocumentationanddata.Thelinesofevidenceforthereviewincluded:

Reviewofdocumentsanddata:InordertobuildanunderstandingofCAF,documentsrelatedtoconsultationsandthefundingsolicitationwerereviewed(e.g.,webinarmaterials,applicationform,assessmentform).Inaddition,othermaterialsrelatedtothesolicitationincludinginternalbriefingmaterialsanddatawereexamined.Approximately140documentswerereviewed.

Abilingualweb‐basedconfidentialsurvey:CAFexternalstakeholdersweresurveyedtogatherfeedbackoncommunicationsandengagementduringthedevelopmentofCAF,the

1LetterofIntent(LOI)andfullproposalphase


fundingsolicitationandreviewprocesses.PHACprovidedalistofexternalstakeholdersincludingthosewhoparticipatedinwebinarconsultationsduringthedevelopmentofCAF,successfulandunsuccessfulapplicants,andexternalreviewers.Intotal,131individualscompletedthesurvey,foraresponserateof41%.(SeeAppendixAforsurveyresults).

Keyinformantinterviews:Thepurposeofthekeyinformantinterviewswastocollectin‐depthqualitativeinformationontheconsultation,solicitationandreviewprocessesandspecificallytosupplementevidencefromthedocumentreviewandtheweb‐basedsurvey.Intotal,48interviewswereconductedwith69individuals,including:PHACNationalCapitalRegion(NCR)andregionalmanagersandstaff;applicants(successfulandunsuccessful);and,externalreviewers.Ofthetotalnumberofinterviews,12wereconductedwithPHACrepresentativesand36withexternalstakeholders.2

1.3StudyLimitationsTwolimitationsofthisreviewshouldbenoted.First,thescope(timeframeandresources)ofthereviewdidnotpermitasystematicexaminationofthemanyproductsofthesolicitation–i.e.,thecontentofLOIs,fullproposalsorfeedbackforms.Therefore,respondentperceptionscouldnotalwaysbevalidatedagainstdocumentaryevidence.Second,whileasignificantnumberofinterviewsandsurveyswereconducted,somerespondentscouldnotanswerallquestionsbecausetheydidnotparticipateconsistentlythroughoutthe4‐yearperiod.Inthesurvey,theoverallresponseratewashigh,butthetotalnumberofcasesdidnotallowforextensivesub‐groupanalyses(e.g.,byregion).

2.0 Description of the Community Action Fund

2.1BackgroundCAFisaPHACGs&CsprogramthatamalgamatedfundingundertheFederalInitiativetoAddressHIV/AIDSinCanadaandtheHepatitisCPrevention,SupportandResearchProgram,whichwerepreviouslymanagedseparately.Launchedin2017,CAFprovidesfundingtocommunity‐basedorganizationstosupportthedomesticresponsestoHIV,hepatitisCandrelatedsexuallytransmittedandblood‐borneinfections(STBBIs).Inthedeliveryofthisprogram,PHACprovides

2Intheanalysisofkeyinformantinterviews,thefollowingdescriptivequalifiersareusedthroughoutthisreporttoindicatethefrequencyofresponses:“all/almostall”–findingsreflecttheviewsandopinionsof75%ormoreofinterviewees;“many/most”‐findingsreflecttheviewsandopinionsofatleast50%butlessthan75%ofinterviewees;“minority/some”‐findingsreflecttheviewsandopinionsofatleast25%butlessthan50%ofinterviewees;and“afew/several/smallnumber”‐findingsreflecttheviewsandopinionsofatleasttwointervieweesbutlessthan25%ofinterviewees.


$26.4millioninfundingannuallythroughGs&Cstocommunity‐basedorganizations(including$4millionindedicatedIndigenousfunding).

TheimpetustoamalgamatefundingforHIV/AIDSandhepatitisCoriginatedwiththe2008publicationoftheGovernmentofCanadaReportoftheIndependentBlueRibbonPanelonGrantandContributionProgramsthatconcludedtherewasaneedforfundamentalchangesinthewaythefederalgovernmentunderstands,designs,manages,andaccountsforGs&Cs,andtheneedtosimplifyadministrationtoensurestrengthenedaccountability.OtherdriversforchangeincludedadirectiveinBudget2012toincreaseadministrativeefficienciesandaccountabilityofGs&CsprogramsandanevaluationoftheFederalInitiativetoAddressHIV/AIDSinCanada(2008‐09to2012‐13)thathighlightedtheneedforincreasedcoherenceatnationalandregionallevelsandtheneedtobetterevaluatetheimpactsofinvestments.

In2014,PHACtransformedthemanagementanddeliveryofGs&CstoadoptacentralizedmodelforGs&CsadministrationwithPHACseekingtoenhanceefficiencies,harmonizefundingpracticesandtools,andreducethereportingburdenonrecipients.ThischangeresultedinGs&CsprogramauthoritiesandbudgetsshiftingfromindividualPHACregionalofficestotheNCR(CCDICandtheCentreforGrantsandContributions(CGC)).

2.2CAFObjectivesandPriorities3CAFwasintendedtomaximizeefficienciesandincreasetheeffectivenessofthecommunity‐basedinvestment.Recognisingtherealityofcommonriskbehaviours,transmissionroutesandat‐riskpopulations,CAFtakesanintegrated,holisticapproachtoaddressingHIV,hepatitisC,andotherrelatedSTBBIs,andrelatedaspectsofhealth,includingmentalhealth,aging,andsocialdeterminantsofhealth.

TheCAF’sobjectivesareto:

Increaseknowledgeofeffectiveinterventionsandpreventionevidence

Enhanceknowledgeapplicationincommunity‐basedinterventions

Strengthenthecapacityofprioritypopulations4andtargetedgroupstopreventinfectionandtobetterhealthoutcomes

Increaseuptakeofbehavioursthatpreventthetransmissionofinfections

3SectionadaptedfromtheHIVandHepatitisCCommunityActionFundwebsite:https://www.canada.ca/en/public‐health/services/funding‐opportunities/hiv‐hepatitis‐community‐action‐fund‐next‐steps.html#s34Prioritypopulationsinclude:Gaymenandothermenwhohavesexwithmen;peoplewhousedrugs;Indigenouspeople;ethno‐culturalcommunities,particularlythoserepresentingcountrieswithhighHIVorhepatitisCprevalence;peopleengagedinthesale,tradeorpurchaseofsex;peoplelivinginorrecentlyreleasedfromcorrectionalfacilities;transgenderpeople;peoplelivingwith,oraffectedby,HIVand/orhepatitisC;womenandyouthamongthesepopulations,asappropriate.


Increaseprioritypopulation’saccesstohealthandsocialservices.

Proposedactivitiesaretofocusonspecificpopulations,supportedbyevidenceofneed.Priorityactivitieseligibleforfundingincludedinterventionsdirectedatprioritypopulationsinordertopreventthetransmissionofnewinfectionsandtopromotehealthyoutcomes;adaptationorcreationofresources;capacityandskillsbuildingforprioritypopulations;andknowledgesynthesis,mobilizationandexchange.Ineligibleactivitieswerespecifiedasthoseoutsidethefederalmandate(e.g.,directhealthservicedeliveryactivities)oroutsidePHAC’sdepartmentalmandate(e.g.,activitieson‐reserve,pureresearch).TheAgencywasparticularlyseekingapplicationsforprojectsthatdemonstratesustainabilityandinnovation,andaddressidentifiedpriorities(e.g.,reducethenumberofpeoplewhoareunawareoftheirHIVorhepatitisCstatus,addressstigmarelatedtoHIVorhepatitisCorpopulationsaffectedbytheseinfections).

2.3OverviewoftheCAFFundingSolicitationInadditiontointegrationoftheresponse,throughthistransformation,PHACchangedthewayitadministeredthisfundingprogram.ThesolicitationprocessforCAFwasforthefirsttimeinmanyyears,implementedasanopencompetitiveprocess.PHACimplementedatwo‐stageapplicationprocessforCAF(seeAppendixAforavisualdepiction).ThefirststageofthesolicitationinvolvedacallforLOIswhichrequiredorganizationstodemonstratecertainmandatoryrequirements(Canadiannot‐for‐profit,atleasttwoyearsofexperiencebothwithSTBBIrelatedworkandmeaningfulengagementoftheprioritypopulation(s)identifiedintheproposedproject)andtodescribe,usinganevidence‐basedapproach,howtheirprojectswouldaddressCAFpriorities,meetobjectives,andimpactthetargetedhealthissuearea.5Nofundingceilingwasidentifiedforfundingagreementsandapplicantscouldapplyforamaximumoffiveyearsoffunding.

LOI’swerereviewedbybothinternalprogramstaffandexternalreviewersandsuccessfulLOIsmovedforwardtothefullproposalphase.Allfullproposalswerefunded,eitherassubmitted,orwithadjustmentstobudgets,workplans,evaluationplans,and/orprojectscope.

3.0 Feedback and Lessons Learned

Thefollowingsectionpresentsasummaryoffeedbackgatheredduringthereviewaswellas15lessonslearned.ThefeedbackispresentedlooselyinchronologicalorderassociatedwiththeCAFsolicitationprocess.Thelessonsareintendedtoinformthedevelopmentandimplementationof

5Anorganizationwithlessthantwoyearsofexperiencecouldapplyaspartofacommunityalliance,whichwasanewlyintroducedfeatureofCAF.Thecommunityalliancemodelinvolvesaproposalwheretwoormoreorganizationsagreetoworktogethertoachievetheobjectivesofasingleprojectinordertoincreasecoherenceoftheresponseandreduceduplicationandburdenonorganizationswhomaypoolresources.


futurefundingopportunitiesatPHAC,andmanyarerelevanttosolicitationsthatintroducechangestoafundingparadigm.

3.1CAFDesignandDevelopmentStakeholderEngagementToinformthedesignanddevelopmentofCAF,PHACdevelopedaStakeholderEngagementStrategy.Between2014and2016,toimplementthisStrategy,theProgramsandPartnershipsDivisionofCCDICconsultedwithstakeholdersonthebroadoutlineofthefundingdeliverymodel(firstphase)anddefiningthefundingpriorities,eligibilityofprojects,andfundingstreams(secondphase).Approximately40consultationswereheldthatweretailoredtodifferentstakeholdergroups:

Nationallyfundedorganizations,whoparticipatedinfacetofacemeetingsandteleconferenceswithPHAC.

NationalIndigenousorganizationsandrepresentativesfromtheNationalAboriginalCouncilonHIV/AIDS(NACHA)6participatedinatwo‐dayin‐personmeetingthatincludedarangeoforganizations.TheCanadianAboriginalAIDSNetworkwascontractedtoleadpiecesofthisengagementprocess.

Provincial‐levelhealthagenciesanddepartmentsparticipatedinbilateralteleconferencesandattendedmanywebinarstodiscussfundingintheirrespectiveprovinces.

Peoplewithlivedexperiencewereengagedthroughteleconferences.

Community‐basedorganizationsparticipatedininteractivewebinars,whichwereusedtodescribeandgatherfeedbackonCAFandthesolicitationprocess,includingtheLOIprocess.

PHACstaff(i.e.,NCR,regional,CGC,otherareas)alsoparticipatedininternal,facetofaceconsultations,includingdiscussionsaboutregionalallocations.

Mostexternalstakeholders(59%ofsurveyedstakeholders)agreedthattheirorganizationhadanopportunitytoaskquestionsand/ortoprovidefeedbackonthedesignofCAF.Similarly,intervieweesgenerallyapprovedofPHAC’soverallapproachtoconsultations,statingthatthematerialswereclearandappropriate.Thatsaid,someexternalstakeholdersfelttheconsultationsessionswere,attimes,scheduledonshortnoticeorwerenotsufficientlyinclusiveofkeyprioritypopulations,and/orlackedculturalorregionalsensitivity.Somealsofeltthattheengagementstrategywasnotsufficientlydrivenbytheregions.

6NationalAboriginalCouncilonHIV/AIDS(NACHA)ismandatedtoprovideIndigenous‐focussed,unbiased,non‐partisan,evidence‐informedandstrategicexpertadvicetothePHAConHIV/AIDS.HepatitisCandrelatedhealthfactorssuchassexuallytransmittedinfections,tuberculosis,aging/seniorsandmentalhealthastheyaffectIndigenousPeoples.


Intermsoftheconsultationchannels,surveyedexternalstakeholderswereaslikelytoagree(41%‐25%amongunsuccessfulapplicants)asdisagree(43%)thatthemethodsofengagementwereappropriatefortheirorganization.Somestakeholdersnotedparticularlimitationswiththewebinarformat(toomanyparticipantswithdiversemandates,poorFrenchtranslation,highlyspecificconcernsofafeworganizationsthatdominateddiscussions).

LESSON#1:Whenundertakingconsultationswithstakeholders,employmultiplechannelsandvariousconsultationapproachestoensuresufficientoutreachtodiversecommunitiestofullyparticipate.Consider:

usingknowledgeable/connectedthirdpartiestosupportengagement,asappropriate leveragingnetworksattheregionalandcommunitylevel addressinginherentlimitationsofwebinarsbystructuringtheseconsultationsto

focusonsmallergroupsand/orthosewithcommoninterestsRespondingtoStakeholderFeedbackBasedonthedocumentsandstaffinterviews,PHACheld14consultationswithstakeholderstosharehowstakeholderfeedbackwasincorporatedintotheCAFdesign.ThesesourcesciteanumberofexamplesofaspectsofCAFthatwereadaptedand/oraddedbasedonstakeholderfeedbackfromthefacetofaceandvirtualroundtableswithexternalstakeholders,suchasimplementingthetwo‐stagesolicitationprocess,expandingthebreadthofprioritypopulationsfromfourtoeight,incorporatingtheburdenofinfectionaspartofthefundingformulaandallowingforlongerfundingterms.

However,oftheelementsoftheconsultationsthatwereexamined,externalstakeholdersprovidedtheweakestratinginrelationtofeelingtheirorganization’sviewswereheardbyPHACduringtheconsultationprocess.Overall,30%ofstakeholdersagreedthattheyfeltheard;however,46%ofstakeholdersdisagreedthattheyfeltheard(29%ofsuccessfuland80%ofunsuccessfulapplicantsdisagreedthattheyfeltheard).Thissentimentwasechoedininterviews,wheresomestakeholderssaidtheydidn’treceivefollow‐upmaterialsontheresultsorconclusionoftheconsultations(e.g.,follow‐upQ&A).PHACintervieweesnotedthatthelengthoftheconsultationperiodlikelyimpactedthesecommunications.

SomestakeholdersexpressedinterestinunderstandingwhysomefeedbackofferedduringtheconsultationscouldnotbeincorporatedintothedesignofCAF.AccordingtoPHACprogrammanagers,suggestionsthatcouldnotbeincorporatedwereoftenproposedactivitiesfallingoutsidethefederalmandate(e.g.,permittingoperationalfunding,directhealthserviceactivities).Whiletherewasnoexpectationamongstakeholdersthatallfeedbackwouldbeincorporated,intheabsenceofcompleteorconsistentlyavailableresponsestostakeholderinput,theyperceivedtheconsultationsasinformationsharingratherthanmeaningfulengagement.


LESSON#2:Whilerecognizingthatprogrammingdecisionsmustalignwithstrategicpriorities,andfederalpoliciesandguidelines,respondtoconsultationinput/feedback(includingdetailsregardingwhether(ornot)inputwas/wasn’tincorporatedandwhy)inaconsistentandtimelymanner.

3.2FundingSolicitationLOI Launch FollowingtheconsultationperiodandfinalizedCAFdesign,thefundingsolicitationwaslaunchedinFebruary2016.ApplicantshadtwomonthstosubmittheirLOIs(mid‐Februarytomid‐April,2016).TheLOIwascompletedusinganonlineplatformandconsistedoftwoparts‐acoverletterwhereapplicantsweretoaddressmandatoryrequirements(PartA)andanapplicationformdescribingtheproject(PartB).

GuidanceandSupport

GuidanceforapplicantstopreparetheirLOIwasprovidedthroughwebinars,7whichmostapplicantsattended(83%basedonsurveyresponses),aswellasaguidancedocument/factsheetandcontentontheCAFwebsite.Overall,applicantshadmixedviewsaboutthesupportprovidedbyPHACduringthesolicitationprocess;oneinthreeapplicantssurveyed(30%)agreedthatPHACguidanceduringtheLOIsolicitationwasuseful(responsive,clear,consistent),whilealmostfourinten(38%)disagreed.Applicantsthatproposednationalprojects,successfulapplicantsandnewlyfundedorganizationsprovidedamorefavourableratingoftheguidanceprovided.

PositivecommentsfromapplicantsinthesurveyandinterviewsincludedthattheguidancewashelpfulandthatthechangesthatPHACintroducedinthiscallforproposals(e.g.,thatitwasopenandcompetitive,LOIsshouldbeevidence‐based,anddemonstrateimpactofprojectfunding)wereclearlycommunicated.

Others,however,perceivedthatsomeoftheguidanceduringthesolicitationwasunclear,particularlyaroundkeyconceptsthatdefinedthepriorityandeligibilityofactivitiesunderCAF.WhiletheLOIsupportmaterialsincludedinformationmeanttoclarifyactivitiesthatwereeligibleandnoteligibleforfunding,questionsreceivedduringthesolicitationprocessandsomeLOIsthatproposedactivitiesthatwerefoundtobeineligiblesuggestthereremainedsomeambiguityinthemessagingorthatmessagingwasnotsufficientlyunderstoodbysomeapplicants.

7ThewebinarsprovidedanoverviewofthefundingopportunityandrequirementsfortheLOI,andcoveredtopicssuchastheprioritypopulationsandactivitiesandprovidedamockLOIresponsetoillustratethetypeofinformationexpectedintheLOI.


LESSON#3:Inthecontextofchangingprogramdirection,includestrategiesto:

maximizeopportunitiesthroughcontinuouscommunicationsandvariousmeanstoreinforcenewconceptsanddriversforchange

ensuremessageclarity,precisionandconsistencytominimizeuncertainty/maximizecomprehension.

ResponsivenesstoQuestionsToencourageafairandlevelplayingfieldforthesolicitationprocessandtosupportconsistencyinresponses,applicants’questionsabouttheLOIweredirectedtoacentralizedemailaddressandreplieswereprovidedbynationalstaff.

Feedbackfromsomeapplicantsintheinterviewsandsurveysuggestsomechallengesinreceivingtimely,clearandsatisfactoryresponsestoquestionsposedduringthewebinarandviathecentralizedemail.WhilethePHACservicestandardis10daystoreplytoquestionsduringasolicitation,thisislikelylonginthecontextofan8‐weeksolicitation.Also,whilePHACreportedlyincreasedstaffresourcesforadditionalsurgecapacitytohelprespondtoincomingquestions,PHACwasnotabletoconsistentlymeetthestandardduringpeakperiods.

Thereviewfoundtheissueswithresponsivenessofthecentralizedemailsystemcontributedtoagapininformationandcommunicationsforsomeapplicantsduringatime‐sensitiveperiodinthesolicitation.

LESSON#4:Giventhetime‐sensitivenatureofopenandcompetitivesolicitations,mechanismstorespondtoapplicantquestionsshouldbalancethedesireforfairnessandconsistencywithhavingsufficientcapacitytoprovideclearandtimelyresponsestoquestions.

LOI Submission LOISubmissionFormThereviewfoundthattheLOIsubmissionformwascomprehensive,andmostsurveyedapplicantsagreedorwereneutral(67‐72%)thattheformanditsrequirementswereclear,easytouseandappropriate.FewadditionstotheLOIwereproposedbystakeholders;theexceptionwastheviewofafewapplicantsthatthetrackrecordoftheorganizationinprovidingacommunityresponsetoHIV/AIDS,hepatitisCorsexualhealthshouldhavebeenamoreimportantcriterionfortheselectionprocess.

OnereportedchallengeoftheLOIsubmissionformwasthemandatoryrequirementswhichwereexpectedtobedemonstratedinthecoverletter.Whilethemandatoryrequirements(i.e.,theorganizations’managementandgovernancestructure,andexperience)weremetbymost


applicants,45LOIs(oralmost20%)werescreenedoutbecauseoneormoretheserequirementswerenotadequatelydemonstrated(accordingtoreviewers,someoftheoneinfiveapplicationsthatwerescreenedoutoverlookedthisrequirement).

LESSON#5:Toenhancethecomplianceandqualityofapplications,ensurethatessentialormandatoryinformationrequirementsareevidenttoapplicantsthrough:

theuseofdedicatedresponseentriesintheapplicationform achecklistidentifyingrequiredcontentandmaterials.

LOITimelinesandLevelofEffortIntermsoftimelinesforsubmission,44%ofsurveyedapplicantsagreedthatthetimeavailabletopreparetheirsubmission(abouteightweeks)wassufficient(higheramonglargerorganizationsproposingnationalprojects),whileoneinthreeapplicants(33%)disagreed(higheramongcommunityallianceapplicants).Intheinterviews,someapplicantsnotedthatthetimelinesweretightastheLOIwaslaunchedinthelastquarterofthefiscalyear(abusytimeforcommunityorganizations).

Thequalitativefeedbackfromtheinterviewsalsosuggestedthat,despitethetwo‐stagesolicitation,manyorganizationsfoundthepreparationoftheirLOItobearduous(afewrespondentsindicatedthattheLOIrepresented75‐80%oftheworkofthefullproposal).Thiswasparticularlythecaseforsmall,regionalorvolunteer‐runorganizationsthatlackedthecapacitytoproducethekindofevidence‐basedsubmissionrequiredbyCAF,evenattheLOIstage.Thiswasconfirmedbysomeexternalreviewerswhoobservedthatlargerorganizationswithaccesstoresources/stafftosupportamorecomplete,higherqualityapplicationweregenerallymoresuccessfulintheLOIstage.

LESSON #6: Toensureorganizationswithlimitedcapacityarewellpositionedtofulfillsolicitationrequirements,considerstreamliningtheapplicationprocessandprovidingopportunitiesforcapacitydevelopment.

CommunityAllianceStreamOverall,thereviewfoundthatthecommunityalliancestreamwasviewedaspromisingbymanystakeholdersandaspresentingpotentialbenefitsforthecommunityresponsetoHIV/AIDSandhepatitisC(e.g.,scalingofactivitiestoabroaderpopulationorregion).However,communityallianceapplicantsfeltthattherewerealsochallengesincompletingthismorecomplexLOIandsomePHACstaffobservedalackofcommunityreadinessforthismoredemandingdeliverymodel.Comparedtotheindividualorganizationstream,requirementsfortheLOI(andfullproposal)forthecommunityalliancestreamweregenerallyfoundtobelessclearbythese


applicantsandbysomePHACstaffaswell.ConveningpartnersandpreparingtheLOIwasuniformlydescribedastimeconsuminganddifficultbyapplicants.Severalapplicantsnotedthattheirproposedcommunityalliancewasreconfiguredduringthefullproposalstageforavarietyofreasons.

LESSON #7: Whenintroducingnewand/ormulti‐organizationmodels/approaches(suchasthecommunityalliancemodel),assessreadinessofthecommunity/PHACandprovidemoreflexibilitytoensurethereissufficienttimeforcoordinationamongapplicants(e.g.,capacitydevelopment,staggereddeadlineswithindividualapplications,clearexpectations,processes/templates).

LOI Review OnceLOIsweresubmitted,thereviewphasecommenced.PHACreceived232LOIs.ForthefirstyearofCAFfunding,fundingrequestsvaluedat$63millionayearwerereceived,greatlyexceedingtheavailablefundingenvelopof$26.4millionperyear.

ThereviewofLOIsproceededonanumberoftracksthatincluded:

PHAC’sCGCreviewedtheLOIsforcompletenessandeligibility;

agroupofexternalreviewers(fromprovincesandterritories(PT),federalsubjectmatterexpertsandpersonswithlivedexperience)(about60intotal,includingsixpersonswithlivedexperience))wereprovidedwithLOIstoreviewthatwererelevanttotheirregionorareaofexpertise;

10podscomposedofPHAC(regionalandNCR)representativeseachreviewedasub‐setofrandomlyassignedLOIs,incorporatingexternalreviewerfeedbackasavailable.ThepodsdevelopedfundingrecommendationswhichwerediscussedamongthepodsandwiththeOversightCommitteeduringatwo‐dayteleconferencemeeting;and

asmallOversightCommittee(consistingoftwoseniorPHACmanagersintheNCR)reviewedallLOIsandpoddiscussionstomakethefinalrecommendationsforapprovalbytheprogramdirector.

LOIswerenotrankedbythepodsortheOversightCommittee,butwereratherassignedacode–recommended,recommendedwithchangesornotrecommended.

Accordingtointernalstaffandexternalreviewers:

someperceivedtherewasinsufficienttimeforathoroughreviewofLOIsand,amonginternalstaff,apoorformat(teleconference)forthepodstodiscusstheresultsofthereviewprocessandarriveatLOIfundingrecommendations;and


afewfoundtherewasalackofclarityorsharedunderstandingaboutaspectsofthesolicitation(e.g.,typesofactivitieseligibleunderthefederalmandate,responsibilityforpreparingfeedbackforms)orthereviewprocess(e.g.,whetherthepodswererecommendingfundingorrecommendingLOIstomoveontoafurthercompetitiveprocess,howthefundingrecommendationswouldbesensitivetoregionalcontexts).

Thereviewconductedbythepods(eachreviewedasubsetofthetotalLOIssubmitted)ledto129LOIsbeingrecommended(morethancouldbefunded).TheassessmentformandreviewformatdidnotlendthemselveswelltodiscussionandprioritizationoftheLOIsrecommendedbythepods.TheOversightCommittee’sreviewofLOIs(basedonalignmentwithpriorities,eligibilitycriteriaandavailablefunding)resultedinfinalfundingrecommendationsfor81LOIs.TheOversightCommitteemetwithregionaldirectorstodiscussfundingrecommendations,however,somepodreviewersindicatedtherewasalackofclarityabouthowthefinallistofrecommendedLOIswasdetermined,andwereconcernedaboutthepotentialforgapsacrosspopulationsandregionsinthefinaldistributionofrecommendedprojects.

LESSON#8:Toimprovethesolicitationandreviewprocess:

ensureclarity/understandingofthereviewers’rolesandresponsibilities establishbroad‐basedreviewbody(ies)tomakefundingrecommendations createtools/providetrainingforreviewerstosupportconsistentandinformed

assessmentofapplications ensuresufficienttimeandappropriateformattofacilitatediscussionsamong

reviewersaroundfundingrecommendations.

Someapplicantandstaffintervieweesraisedquestionsabouttheeffectiveness,consistencyandtransparencyofthereviewprocess.AssuccessfulLOIsmovedintothefullproposalphase,theextentofthechangesthatwererequestedorpermittedtosomeofthefullproposals(describedbelow)ledsomeunsuccessfulapplicantstoperceiveunfairnessinthisprocess.

LESSON#9:Ensurethattheestablishedparametersofthefundingsolicitationareappliedinanobjectiveandconsistentmanner.

LOIReviewCriteria

Overhalfofsurveyedapplicants(54%)disagreedthatthecriteriaandprocessthatwereusedtoassessLOIswereclear(successfulapplicantsweremorelikelytosaythecriteriaandprocesswereclear),while26%agreed.


Withrespecttothereviewcriteria,theguidanceforapplicantsexplainedthattheLOIswouldbereviewedbasedonquality,meritandrelevance.OntheCAFwebsite,forinstance,thematerialindicatesthatLOIswouldbeselectedonthebasisofhavingthegreatestpotentialtomakeanimpactonSTBBIs.Prioritywasalsotobegiventothoseprojectswhichincludedactivitiesthatdemonstratesustainabilityandscalabilityandthatdemonstrateoriginalityandinnovativeapproaches.

Thereviewfoundthatsomeoftheselectioncriteriaarticulatedontheweb‐sitewerenotclearlyfeaturedintheLOIwebinarmaterialsordisplayedprominentlyintheLOIsubmissionformitself(e.g.,theLOIformdidnotdirectapplicantstodemonstratesustainability,scalability,orinnovation,althoughthesecriteriawereusedintheassessment).

Fortheirreview,podsandtheOversightCommitteeusedanassessmentformthatparalleledtheselectioncriteriaintheguidanceandweb‐site.Theforminitiallyincludeda3‐pointscaletoassessLOIsintermsofprojectrelevance,involvementofprioritypopulations,evidence‐basedneed,projectimpacts,sustainabilityandscalability,originalityandinnovation,andattentiontoperformancemeasurementandevaluation.The3‐pointscaleratingwasabandonedduringtheLOIreviewprocess,thuslimitingtheabilityforLOIstobeclearlyprioritizedorranked.ManyreviewersalsosuggestedthatthetoolsandsupportsrequiredimprovementtosupportconsistencyacrossthepodsandbetweenthepodsandOversightCommitteeinhowtheyassessedtheLOIs.

LESSON#10:Establishclearselectioncriteriathatarecommunicatedconsistentlyduringthesolicitationandthereviewprocess.

Tosupportfundingrecommendations,establishaclearframeworktoprioritizeapplicationsthatarerecommendedforfunding.

TimelinessofFundingDecisionFollowingthereviewofLOIs,therecommendationsforfundingwerepreparedbytheendofJune2016(withintwomonthsafterthecloseoftheLOIphase).Thedecisionswerenotcommunicateduntilthreemonthslater.ThedelayinannouncingthefundingdecisionswasanissueraisedbyPHACstaffandmanyapplicants(60%ofapplicantsdisagreedthatthefundingdecisionsweretimely)(SeeLesson#14).Ininterviewsandtheopen‐endedsurveyfeedback,manyapplicantsdescribedthedelaysasstressful,frustratingandchaoticfortheirorganization,withseriousimplicationsforstaffingandplanning.InrecognitionofthecompressedtimeframefororganizationsthatwerenolongergoingtobefundedunderCAFtowinddowntheirprojects,itwasannouncedthattheseorganizationswouldreceivetransitionfundingforoneyear.


LOI Feedback AfeedbackformwascompletedforeachapplicationbothtoassistsuccessfulapplicantstopreparetheirfullproposalandforunsuccessfulapplicantstoexplainwhytheirLOIwasnotrecommended.Becauseoftherecentchangestotheprogram,internalstaffreportedthatthefeedbacktoCAFapplicantsdepartedfromthestandardizedformstypicallyusedbyCGCtocommunicatefundingdecisionsbyprovidingmoredetailsandincludingreviewerfeedbackontheLOIstrengthsandweaknesses.

Oftheapplicantswhorespondedtothesurvey,aminorityagreedthatthefeedbackaccuratelyreflectedtheirproposal(26%)orwasclear(20%)orcomprehensive(20%).Expectedly,dissatisfactionwiththefeedbackwasmoreprevalentamongunsuccessfulapplicantsalthoughamongorganizationsthatwerefunded,viewsabouttheLOIfeedbackwerealsomixed.InsomeinstancesapplicantsindicatedthatthefeedbackdidnotreflectthecontentorideasthatwereproposedintheirLOIorquestionedhowpositiveandnegativefeedbackwasreconciled,bothofwhichcreatedconfusionaboutthedecisionforsome.

FromtheperspectiveofsomeoftheLOIreviewers,thequalityofthecontentoftheLOIswasperceivedtobemixed.InternalandexternalreviewersobservedthatorganizationshadvaryinglevelsofunderstandingoftheLOIsubmissionrequirements,theprioritiesandselectioncriteria.WhereasmanyLOIswereclearandofferedinnovativeandevidence‐basedprojects,somereviewersalsonotedinstanceswhereapplicationswerenotwellwritten,lackedsubstantiationofneedoreffectivenessoftheintervention,werenotabletoclearlyarticulatetheiractivitiesorgoalsand/orincludedproposedactivitiesthatwerenotalignedwiththefederalmandate.

LESSON#11:Feedbacktoapplicantsshouldbebasedonastandardizedformatthat:

clearlyexplainswhytheapplicationwasnotsuccessfulreferencingtheselectioncriteria(unsuccessfulapplicants);or

providescleardirectiononrecommendedchanges,includingafundingceilingforfullproposals(successfulapplicants).

Full Proposal Submission, Review and Contribution Agreement Intotal,82LOIswereapprovedtomoveforwardtothefullproposalstageand85projectswereultimatelyapproved(someLOIsweresplitintoseparateprojects).Thefullproposalsrequiredthatapplicantselaborateinmoredetailontheirproject,includingadetailedworkplan,objectives,budgetandevaluationplan.IntermsofassistingsuccessfulLOIapplicantstopreparetheirfullproposal,applicantshadfeedbackontheirLOIandPHAChostedwebinarsandprovidedtemplatesforkeyrequirements.Atipsheetonpreparingaworkplanwasdevelopedandcirculatedinresponsetoquestionsfromapplicants.Asthisphaseofthesolicitationprocesswas


notcompetitive,applicantshadgreateraccesstoPHACregionalandnationalprogramconsultantstoseekclarificationwhichwasappreciatedbyanumberofapplicants.Thefullproposalsubmissionhadadeadlineand,uponsubmission,wereassessed,first,byCGCtoensure

adherencetobudgetandexpenditureeligibilitycriteria8andthenbythepodsandOversight

Committee.SomeproposalsweresharedwithPTrepresentativesforfeedback.

TimelinesforFullProposalSubmissionSimilartotheLOIphase,someapplicants(particularlythoseinvolvedincommunityallianceproposals)feltthatthetimelinetosubmittheirfullproposal(8weeksbetweenlaunchanddeadline)wastoocompressedandthatPHACwasnotpromptinprovidinginformationorclarificationinresponsetoquestionsabouttherequirementssuchastheSMART9objectiveswhichwerenotwellunderstood.AffectedapplicantswereaggravatedwhendelayedorinconsistentfeedbackfromPHACcompressedtheirowntimelines.Someorganizationsexperiencedfurtherdelaysinthereleaseoffunds(onlyoneinfourapplicants(21%)agreedthereleaseoffundswastimely).(RefertoLessons#14)Itshouldbenotedthatduringthistimeperiod,PHACwasexperiencingcapacityconstraintsduetotheannouncementoftransitionfundingandtheresultingneedtoamendexistingcontributionagreements,aswellasrespondingtoahighvolumeofcorrespondencefromunsuccessfulapplicants.

Guidance/SupportduringtheFullProposalPhaseAmongthevariousaspectsofthefullproposalphasethatwereexamined,applicantsagainhadmixedviewsontheguidanceandsupportprovidedbyPHACduringthisprocess,withsimilarproportionsagreeing(38%)anddisagreeing(43%)thattheguidancewasclearanduseful.OrganizationsthatwerenewlyfundedbyCAFhadmorefavourableopinionsofPHAC’ssupportduringtheprocess.

Accordingtointernalinterviewees,inordertoimprovetheeffectivenessofprojects,apriorityforPHACwastoensurethatorganizationsclearlymappedtheirplannedinterventionworkplanswithwell‐articulatedoutcomes.Asmentionedabove,PHAC’sexpectationsfortheserevisionsweresupportedbywebinars,templates/resources,andfeedbackfromPHACstaff.Thevastmajorityofsurveyedapplicantswhopreparedfullproposals(n=55)wererequestedtorevisetheirSMARTobjectives(91%)andtheirworkplan(84%).

Preparingtheworkplan,budget,SMARTobjectivesandevaluationplanwerefoundtobecomplexanddemandingbymostsuccessfulapplicants.Theprocesstofinalizethecontributionagreementwasdescribedbysomeasconfusingandheavyhanded,with‘changinggoalposts’andinconsistentmessaging(whichafewattributedtochangesinpersonnel)thatwasnotsufficientlydocumented.Accordingtotheseapplicants,therevisionsrangedfromexcessivewordsmithing

8Examplesofineligibleexpendituresincludecapitalcosts,rentalcharged,non‐projectrelatedorganizationalcosts.9Specific,Measurable,Achievable,ResultsBased,andTime‐Bound


(e.g.,toaligntextwithSMARTobjectives)torevisionsthatwereperceivedtobehighlyprescriptive.SomeapplicantsfeltthattheirproposalchangedconsiderablyfromtheprioritiesandevidenceofneedoriginallyoutlinedbytheorganizationintheLOIandwerealsoquiteprotracted.

Accordingtointernalinterviewees,someorganizationslackedreadinessforthisstageandtheynotedaswellthatmanyiterationsofthefullproposalwererequiredforsomeapplicantstoprepareafullproposalthatmettheAgency’srequirements(e.g.,revisionswererequiredtoremoveineligibleactivities,ensureworkplansandbudgetsconformedtorequirements).

LESSON#12:Tofinalizefullproposalsinatimelyfashion,provideclearguidanceandconsidersimplifyingworkplan/evaluationplanrequirements,whilerespectingGovernmentofCanadarequirements,policies/guidelines.

FinalFundingCommitmentAsmentionedabove,theannualCAFbudgetis$26.4millionperyear.Withinthebudgetedamount,therearenational,regionalandIndigenousfundingallocationstobeadheredto.Duringthefullproposalphase,stayingwithinthefundingallocationswaschallengingasanumberofapplicantssubmittedhigherbudgetsintheirfullproposalcomparedtotheirLOIs(whichmayhavebeeninresponsetosuggestionsintheirLOIfeedbacktoaddcomponentsorbecauseapplicantsunderestimatedthecostsoftheirprojectattheLOIstage).Forthisreasonandothers,almostallorganizations(91%)wereaskedtomakesubsequentadjustmentstotheirprojectbudgetinordertoalignwiththeprogrambudgetallocationsand/ortoremoveineligibleexpenditures.

AttheconclusionoftheCAFsolicitation,mostoftheCAFfundingwascommittedforthenext3years,makingitdifficultfororganizationsthatdidnotreceivefundingtore‐applyfromthissourceorforPHACtorespondtoemergingissues.

LESSON#13:Tominimizepotentialimpactstoorganizationsthatwerenotsuccessfulinthesolicitationandaddresspotentialgapsinthecommunityresponse,considerbuildingintimefororganizationstofindalternativefundingsourcesand/orstaggerfundingcalls.

Two‐stage Solicitation Thetwo‐stageLOIandfullproposalsolicitationwasnewtotheCAF2016fundingcall,institutedtodealwithananticipatedhighervolumeofapplicationswiththenewopenandcompetitiveformatandtobelessburdensomeforapplicants.Thetwo‐stageprocesswasalsomeanttoreduceduplicationandlessenthereviewprocess.


Overall,bothinternalstaffandapplicantswerereceptivetothetwo‐stagemodel.However,asmentionedabove,somePHACstaffandapplicantsnotedsomeconfusionduringthesolicitationaboutwhetherthefullproposalphasewascompetitive.EarlycommunicationsaboutthesolicitationindicatedthatbeingsuccessfulattheLOIstagedidnotguaranteefundingatthefullproposalstage.Thiswasinterpretedbysomestaffandapplicantstosuggestthatthefullproposalphasewouldbecompetitive.

Accordingtosomeinternalstaffandapplicants,thedisadvantageofthetwo‐stagesolicitationisthelongersolicitationperiod(twosubmissions,tworeviewperiods).Ashasbeenmentionedabove,therewerevariousstagesinthesolicitationwhendelayswereintroduced(releaseofLOIdecisions,signingofcontributionagreementsandreleaseoffunds).ThefinalCAFcontributionagreementwassignedinMarch2018,almostafullyearaftertheinitiallyproposedtimeline.

LESSON14:Inplanningthesolicitationprocess,developcontingencyplanstolimittheimpactofchallengesrelatedtointernalprocessesandcapacityshortfallsinordertomitigatedelays.

3.3CommunicationsOneoftheobjectivesofthisreviewwastoexaminetheclarity,timelinessandfrequencyofcommunicationsthroughouttheCAFsolicitationprocess.Afrequentthreadinthefeedbackdiscussedabovehasbeenrelatedtocommunications.ThereareseveralexamplesofcommunicationsthatPHACimplementedwithanambitiousnumberanddiversityofstakeholdersinahighlychallengingperiodofchange.Alistof4,000stakeholderswascompiledandnotifiedofkeyelementsofCAF(e.g.,launchofthesolicitation).Duringthesolicitationphase,therewereavarietyofchannelsusedtocommunicateaboutthesolicitation(webinars,web‐site,templates,resources)andmultiplecontactswithinPHAC–national,regional,CGC–involvedinthesolicitation.

However,intermsofclarityofcommunications,thereviewsuggestsanumberofkeyelementsoftheCAFsolicitationwherestakeholdersdidnothaveasufficientlyclearorpreciseunderstandingofthechangedvisionforCAF.AlthoughcommunicatedonanumberofoccasionsbyPHAC,particularlychallengingwasconveyingthemeaningofthefederalmandateinpublichealth,andtheimportantimplicationfortheactivitiesthatwereeligibleforfundingunderCAF.Thischangewasnotwell‐understoodorunderestimatedbysomeorganizations(andevenbysomereviewers).Inothercases,thereappearedtobesomegapsinclarityaroundoperationalaspectsofthesolicitation;whetherthefullproposalstageofthetwostagesolicitationwascompetitiveisanexample.

Intermsoffrequencyandtimelinessofcommunications,thereweresomereportedchallengesinresponsivenesstoquestionsduringtheLOIsolicitationandfullproposalstage.Finally,someapplicantsfeltthatcommunicationsaroundtheLOIfeedbackandfeedbackonfullproposalswereinconsistent.


Itshouldbenotedthatduringthedatacollection,theconsultantsencounteredmanystrongviewsabouttheCAFsolicitationonthepartofbothinternalandexternalstakeholders.CAFintroducedimportantmodificationstoafundingmodelthathadbeenunchangedforanumberofyears,andaspectsoftheCAFsolicitationandreviewprocessesandsomeofthefundingdecisionstriggeredavocalandcriticalreactionamongsomeHIV/AIDSorganizations.SomeorganizationsfeltalossoftrustandconfidenceinthesolicitationprocessandincommunicationsaboutCAFwhichhasnotyetbeenfullyrestored.

LESSON#15:Communicationsrelatedtofundingsolicitationsshouldbebasedonanapproachthat:

ensuresconsistmessagingthatisreadilyaccessibletoapplicants; prioritizecontinuityincontactswithapplicants; usepreciseandplainlanguage,andfocusoncriticalaspectsofthesolicitation; communicatefrequentlywithstakeholdersonstatusofthesolicitation,particularly

whendelaysareencountered; devoteattentiontorecordkeepinganddocumentingdecision


Appendix A: Summary of Survey Results

Figure 1: Stakeholder Engagement during the Development of CAF

Source:SurveyofCAFStakeholders,2018(n=68)

46%

43%

32%

29%

22%

24%

16%

22%

19%

19%

29%

41%

46%

52%

59%

2%

0% 20% 40% 60% 80% 100%

I felt my views/ views of my organization were heard byPHAC

The means of consultations/providing feedback wereappropriate for me/my organization

The purpose/ rationale for the changes to the HIVHepatitis C community‐based funding programs were

communicated clearly

After the consultation, I/my organization understood thekey elements of CAF (e.g., objectives, priorities)

I/My organization had opportunities to askquestions/provide feedback on the design of CAF

Engagement and Consultation during Development of CAF

Disagree Neither disagree nor agree Agree Don't know


Figure 2: Feedback on LOI Solicitation

Source:SurveyofCAFStakeholders,2018(n=84to90)

Figure 3: Feedback on Reviews of LOIs

Source:SurveyofCAFStakeholders,2018(n=90)

38%

30%

31%

33%

24%

30%

27%

27%

20%

28%

30%

40%

40%

44%

44%

2%

3%

2%

2%

3%

0% 20% 40% 60% 80% 100%

The guidance provided by PHAC to prepare the LOI wasuseful (e.g., responsive, clear, consistent)

The questions on the LOI application form wereappropriate to explain our proposal

The space available to respond to questions on the LOIapplication form was sufficient

The amount of time available to prepare and submitthe LOI was sufficient

The LOI application form questions and requirementswere clear

LOI Solicitation


60%

54%

28%

23%

18%

16%

21%

9%

19%

26%

48%

62%

3%

4%

3%

6%

0% 20% 40% 60% 80% 100%

Communications about the results of the LOI reviewwere timely

The criteria and process that were used to assessLOIs were clear

It was clear the funding decisions would be madebased on the information submitted in the LOI

It was clear that funding decisions were final

LOI Review



Figure 4: Stakeholder Views on Quality of LOI Feedback


58%

58%

54%

19%

13%

17%

20%

26%

26%

3%

3%

4%

0% 20% 40% 60% 80% 100%

Feedback on our LOI was comprehensive

Feedback on our LOI was clear

Feedback on our LOI was reflective of the selectioncriteria and content of our proposal

Feedback on LOI



Figure 5: Stakeholders Views on Full Proposal Stage


53%

50%

40%

43%

38%

47%

33%

16%

17%

26%

17%

19%

7%

16%

21%

29%

33%

38%

40%

43%

50%

3%

2%

2%

3%

3%

2%

0% 20% 40% 60% 80% 100%

Receipt of funding from PHAC was timely

Feedback from PHAC to complete the contributionagreement was timely

The guidance/support provided by PHAC to prepare theFull Proposal was useful/appropriate

The guidance/support provided by PHAC to prepare theFull Proposal was clear

Feedback from PHAC to complete the contributionagreement was clear

The amount of time available to prepare and submit theFull Proposal was sufficient

The questions and requirements to complete the FullProposal were clear (e.g., detailed workplan, budget and

evaluation plan)

CAF Full Proposal Stage



Appendix B: Overview of CAF Solicitation Process

review of the community action fund (caf)€¦ · (caf). goss gilroy inc. (ggi) was hired by the...

Documents