rigorously using evaluation strategies to improve reach … · evaluation strategies to improve...
TRANSCRIPT
![Page 1: RIGOROUSLY USING EVALUATION STRATEGIES TO IMPROVE REACH … · EVALUATION STRATEGIES TO IMPROVE REACH AND FOCUS OF NEW MEDIA OUTREACH Anthony Freeman, MA Senior Evaluation Specialist](https://reader031.vdocuments.net/reader031/viewer/2022013021/5f0620f07e708231d4166f2b/html5/thumbnails/1.jpg)
RIGOROUSLY USING EVALUATION STRATEGIES TO IMPROVE REACH AND
FOCUS OF NEW MEDIA OUTREACH
Anthony Freeman, MASenior Evaluation Specialist
New York City Department of Health and Mental Hygiene
![Page 2: RIGOROUSLY USING EVALUATION STRATEGIES TO IMPROVE REACH … · EVALUATION STRATEGIES TO IMPROVE REACH AND FOCUS OF NEW MEDIA OUTREACH Anthony Freeman, MA Senior Evaluation Specialist](https://reader031.vdocuments.net/reader031/viewer/2022013021/5f0620f07e708231d4166f2b/html5/thumbnails/2.jpg)
CONFLICT OF INTEREST DISCLOSUREANTHONY FREEMAN, MA
none.
![Page 3: RIGOROUSLY USING EVALUATION STRATEGIES TO IMPROVE REACH … · EVALUATION STRATEGIES TO IMPROVE REACH AND FOCUS OF NEW MEDIA OUTREACH Anthony Freeman, MA Senior Evaluation Specialist](https://reader031.vdocuments.net/reader031/viewer/2022013021/5f0620f07e708231d4166f2b/html5/thumbnails/3.jpg)
YAHOO
HOOTSUITE
JACK’D
GRINDR
CRAIGSLIST
SCRUFF
MEETUP
INSTAGRAM ADAM 4 ADAM
TUMBLR
PLENTY OF FISHHORNET
SNAPCHAT
YOUTUBE
TINDER
“NEW MEDIA” = SOCIAL MEDIA + DATING APPS
![Page 4: RIGOROUSLY USING EVALUATION STRATEGIES TO IMPROVE REACH … · EVALUATION STRATEGIES TO IMPROVE REACH AND FOCUS OF NEW MEDIA OUTREACH Anthony Freeman, MA Senior Evaluation Specialist](https://reader031.vdocuments.net/reader031/viewer/2022013021/5f0620f07e708231d4166f2b/html5/thumbnails/4.jpg)
CONTINUOUS QUALITY IMPROVEMENT
“The purpose of continuous quality improvement (CQI)
programs is to improve health care by identifying
problems, implementing and monitoring corrective action
and studying its effectiveness.”- National Commission on Correctional Health Care
![Page 5: RIGOROUSLY USING EVALUATION STRATEGIES TO IMPROVE REACH … · EVALUATION STRATEGIES TO IMPROVE REACH AND FOCUS OF NEW MEDIA OUTREACH Anthony Freeman, MA Senior Evaluation Specialist](https://reader031.vdocuments.net/reader031/viewer/2022013021/5f0620f07e708231d4166f2b/html5/thumbnails/5.jpg)
LEARNING OBJECTIVES
1. Explain how to evaluate a social media plan in the context of HIV
prevention
2. Understand how to measure your agency’s social media efforts
and return on investment
3. Describe how to use data to help inform programming and
strategic changes
![Page 6: RIGOROUSLY USING EVALUATION STRATEGIES TO IMPROVE REACH … · EVALUATION STRATEGIES TO IMPROVE REACH AND FOCUS OF NEW MEDIA OUTREACH Anthony Freeman, MA Senior Evaluation Specialist](https://reader031.vdocuments.net/reader031/viewer/2022013021/5f0620f07e708231d4166f2b/html5/thumbnails/6.jpg)
OVERVIEW
• Background: PlaySure Network and Social Media/New Media
• Developing Social Media/New Media Evaluation
– Creating Engaging Content
– Internal Workgroup
• Documenting Social Media efforts
– Content Calendar
– Social Media Reporting Log
• Preliminary Data
• Discuss Future Plans: Evaluation and Technical Assistance (TA)
![Page 7: RIGOROUSLY USING EVALUATION STRATEGIES TO IMPROVE REACH … · EVALUATION STRATEGIES TO IMPROVE REACH AND FOCUS OF NEW MEDIA OUTREACH Anthony Freeman, MA Senior Evaluation Specialist](https://reader031.vdocuments.net/reader031/viewer/2022013021/5f0620f07e708231d4166f2b/html5/thumbnails/7.jpg)
PLAYSURE NETWORK: NEW NETWORK, NEW MEDIA
1. Promote approaches to sexual healthand combination HIV prevention thatare appropriate for each client
2. Increase access to PrEP and PEP forpeople who may be exposed to HIV –especially priority populations who areuninsured
3. Link people who test positive for HIV,or those who are HIV-positive and outof care, to HIV care services so they canget treated and become virallysuppressed<40 agencies total
17 agencies with new media deliverables
![Page 8: RIGOROUSLY USING EVALUATION STRATEGIES TO IMPROVE REACH … · EVALUATION STRATEGIES TO IMPROVE REACH AND FOCUS OF NEW MEDIA OUTREACH Anthony Freeman, MA Senior Evaluation Specialist](https://reader031.vdocuments.net/reader031/viewer/2022013021/5f0620f07e708231d4166f2b/html5/thumbnails/8.jpg)
GOALS OF SOCIAL MEDIA
![Page 9: RIGOROUSLY USING EVALUATION STRATEGIES TO IMPROVE REACH … · EVALUATION STRATEGIES TO IMPROVE REACH AND FOCUS OF NEW MEDIA OUTREACH Anthony Freeman, MA Senior Evaluation Specialist](https://reader031.vdocuments.net/reader031/viewer/2022013021/5f0620f07e708231d4166f2b/html5/thumbnails/9.jpg)
EXAMPLE: FACEBOOK EVENT PROMOTION VOCES LATINA
![Page 10: RIGOROUSLY USING EVALUATION STRATEGIES TO IMPROVE REACH … · EVALUATION STRATEGIES TO IMPROVE REACH AND FOCUS OF NEW MEDIA OUTREACH Anthony Freeman, MA Senior Evaluation Specialist](https://reader031.vdocuments.net/reader031/viewer/2022013021/5f0620f07e708231d4166f2b/html5/thumbnails/10.jpg)
EXAMPLE:
AWARENESS OF SERVICES (PAID ADS)
MT. SINAI
![Page 11: RIGOROUSLY USING EVALUATION STRATEGIES TO IMPROVE REACH … · EVALUATION STRATEGIES TO IMPROVE REACH AND FOCUS OF NEW MEDIA OUTREACH Anthony Freeman, MA Senior Evaluation Specialist](https://reader031.vdocuments.net/reader031/viewer/2022013021/5f0620f07e708231d4166f2b/html5/thumbnails/11.jpg)
EXAMPLE:DATING APP OUTREACH (GRINDR)CAMBA
![Page 12: RIGOROUSLY USING EVALUATION STRATEGIES TO IMPROVE REACH … · EVALUATION STRATEGIES TO IMPROVE REACH AND FOCUS OF NEW MEDIA OUTREACH Anthony Freeman, MA Senior Evaluation Specialist](https://reader031.vdocuments.net/reader031/viewer/2022013021/5f0620f07e708231d4166f2b/html5/thumbnails/12.jpg)
SOCIAL MEDIA EVALUATION
• Strategies to evaluate the success ofsocial media/new media have laggedbehind scale-up
• We developed strategies to monitorand evaluate use of new media forHIV prevention outreach with the goalof increasing capacity through data-driven technical assistance (TA) to HIVprevention programs
![Page 13: RIGOROUSLY USING EVALUATION STRATEGIES TO IMPROVE REACH … · EVALUATION STRATEGIES TO IMPROVE REACH AND FOCUS OF NEW MEDIA OUTREACH Anthony Freeman, MA Senior Evaluation Specialist](https://reader031.vdocuments.net/reader031/viewer/2022013021/5f0620f07e708231d4166f2b/html5/thumbnails/13.jpg)
METHODS
• Workgroup and stakeholder engagement
• Content Calendar and Post Projections
• Log/Indicators
– Quality Improvement
– Examination of results
• Comparing log data to referral source from service data
![Page 14: RIGOROUSLY USING EVALUATION STRATEGIES TO IMPROVE REACH … · EVALUATION STRATEGIES TO IMPROVE REACH AND FOCUS OF NEW MEDIA OUTREACH Anthony Freeman, MA Senior Evaluation Specialist](https://reader031.vdocuments.net/reader031/viewer/2022013021/5f0620f07e708231d4166f2b/html5/thumbnails/14.jpg)
SOCIAL MEDIA WORK GROUP
![Page 15: RIGOROUSLY USING EVALUATION STRATEGIES TO IMPROVE REACH … · EVALUATION STRATEGIES TO IMPROVE REACH AND FOCUS OF NEW MEDIA OUTREACH Anthony Freeman, MA Senior Evaluation Specialist](https://reader031.vdocuments.net/reader031/viewer/2022013021/5f0620f07e708231d4166f2b/html5/thumbnails/15.jpg)
CONTENT CALENDAR
Date Platform Message Staff Responsible
![Page 16: RIGOROUSLY USING EVALUATION STRATEGIES TO IMPROVE REACH … · EVALUATION STRATEGIES TO IMPROVE REACH AND FOCUS OF NEW MEDIA OUTREACH Anthony Freeman, MA Senior Evaluation Specialist](https://reader031.vdocuments.net/reader031/viewer/2022013021/5f0620f07e708231d4166f2b/html5/thumbnails/16.jpg)
CONTENT PROJECTIONS
![Page 17: RIGOROUSLY USING EVALUATION STRATEGIES TO IMPROVE REACH … · EVALUATION STRATEGIES TO IMPROVE REACH AND FOCUS OF NEW MEDIA OUTREACH Anthony Freeman, MA Senior Evaluation Specialist](https://reader031.vdocuments.net/reader031/viewer/2022013021/5f0620f07e708231d4166f2b/html5/thumbnails/17.jpg)
SOCIAL MEDIA WORKFLOW – ADDING EVALUATION STRATEGIES AT EVERY STEP
PEOPLE
REACHED
ENGAGEMENT/
INTERACTION
PEOPLE
REFERRED
NEW
CLIENTSPOSTS
![Page 18: RIGOROUSLY USING EVALUATION STRATEGIES TO IMPROVE REACH … · EVALUATION STRATEGIES TO IMPROVE REACH AND FOCUS OF NEW MEDIA OUTREACH Anthony Freeman, MA Senior Evaluation Specialist](https://reader031.vdocuments.net/reader031/viewer/2022013021/5f0620f07e708231d4166f2b/html5/thumbnails/18.jpg)
SOCIAL MEDIA CORE INDICATORS
• Platforms utilized(list of sites and dating apps)
• # of posts
– # of banner ads(social media & dating apps)
• # of interactions/engagement
per post
– # of views/impressions per banner ad
• # of direct messages
• # of referrals to agency(documented through social media interaction)
• # of new clients(documented as referral source at enrollment)
process
outcome
![Page 19: RIGOROUSLY USING EVALUATION STRATEGIES TO IMPROVE REACH … · EVALUATION STRATEGIES TO IMPROVE REACH AND FOCUS OF NEW MEDIA OUTREACH Anthony Freeman, MA Senior Evaluation Specialist](https://reader031.vdocuments.net/reader031/viewer/2022013021/5f0620f07e708231d4166f2b/html5/thumbnails/19.jpg)
SOCIAL MEDIA MONTHLY LOG
![Page 20: RIGOROUSLY USING EVALUATION STRATEGIES TO IMPROVE REACH … · EVALUATION STRATEGIES TO IMPROVE REACH AND FOCUS OF NEW MEDIA OUTREACH Anthony Freeman, MA Senior Evaluation Specialist](https://reader031.vdocuments.net/reader031/viewer/2022013021/5f0620f07e708231d4166f2b/html5/thumbnails/20.jpg)
SOCIAL MEDIA WORKFLOW
PEOPLE
REACHED
ENGAGEMENT/
INTERACTION
PEOPLE
REFERRED
NEW
CLIENTSPOSTS
![Page 21: RIGOROUSLY USING EVALUATION STRATEGIES TO IMPROVE REACH … · EVALUATION STRATEGIES TO IMPROVE REACH AND FOCUS OF NEW MEDIA OUTREACH Anthony Freeman, MA Senior Evaluation Specialist](https://reader031.vdocuments.net/reader031/viewer/2022013021/5f0620f07e708231d4166f2b/html5/thumbnails/21.jpg)
AGGREGATE NEW MEDIA DATA(AUGUST 2017 – JULY 2018)
*PRELIMINARY DATA
PEOPLE
REACHED
ENGAGEMENT/
INTERACTION
PEOPLE
REFERRED
NEW
CLIENTSPOSTS
7118 totalRange: 0 – 125 per month
![Page 22: RIGOROUSLY USING EVALUATION STRATEGIES TO IMPROVE REACH … · EVALUATION STRATEGIES TO IMPROVE REACH AND FOCUS OF NEW MEDIA OUTREACH Anthony Freeman, MA Senior Evaluation Specialist](https://reader031.vdocuments.net/reader031/viewer/2022013021/5f0620f07e708231d4166f2b/html5/thumbnails/22.jpg)
AGGREGATE NEW MEDIA DATA(AUGUST 2017 – JULY 2018)
*PRELIMINARY DATA
PEOPLE
REACHED
ENGAGEMENT/
INTERACTION
PEOPLE
REFERRED
NEW
CLIENTSPOSTS
15,293,010 totalRange: 0 – 1,631,680 per post
161,559 totalRange: 0 – 6069 per post
829 totalRange: 0 – 50 per month
![Page 23: RIGOROUSLY USING EVALUATION STRATEGIES TO IMPROVE REACH … · EVALUATION STRATEGIES TO IMPROVE REACH AND FOCUS OF NEW MEDIA OUTREACH Anthony Freeman, MA Senior Evaluation Specialist](https://reader031.vdocuments.net/reader031/viewer/2022013021/5f0620f07e708231d4166f2b/html5/thumbnails/23.jpg)
SERVICE DATA: 243 INTAKES FROM SOCIAL MEDIA(AUGUST 2017 – JULY 2018)
*PRELIMINARY DATA
2
164
243
300
333
717
975
998
1077
1173
0 200 400 600 800 1000 1200 1400
PEP Call Center
Flyer/advertisement
Social media
Sexual Health Clinic
Word of mouth
Internet search
Other
Outreach Event
PlaySure Network Site
Existing client at the agency (in-reach)
number of PlaySure Network intakesAugust 2017 - July 2018
Ref
err
al S
ou
rce
![Page 24: RIGOROUSLY USING EVALUATION STRATEGIES TO IMPROVE REACH … · EVALUATION STRATEGIES TO IMPROVE REACH AND FOCUS OF NEW MEDIA OUTREACH Anthony Freeman, MA Senior Evaluation Specialist](https://reader031.vdocuments.net/reader031/viewer/2022013021/5f0620f07e708231d4166f2b/html5/thumbnails/24.jpg)
232/243 CLIENTS (95%) FROM SOCIAL MEDIA FROM A PRIORITY POPULATION(AUGUST 2017 – JULY 2018)
*PRELIMINARY DATA
social media
243
Priority
population*
232
All other
referral sources
5739
Non-priority
population
11
*MSM (including YMSM), TGNC, BLWSM, etc.
![Page 25: RIGOROUSLY USING EVALUATION STRATEGIES TO IMPROVE REACH … · EVALUATION STRATEGIES TO IMPROVE REACH AND FOCUS OF NEW MEDIA OUTREACH Anthony Freeman, MA Senior Evaluation Specialist](https://reader031.vdocuments.net/reader031/viewer/2022013021/5f0620f07e708231d4166f2b/html5/thumbnails/25.jpg)
65%Of the total clients who
entered the PlaySure Network
through social media and were
eligible for a PrEP prescription
received a PrEP prescription.**(clinical sites only)
(August 2017 – July 2018)
![Page 26: RIGOROUSLY USING EVALUATION STRATEGIES TO IMPROVE REACH … · EVALUATION STRATEGIES TO IMPROVE REACH AND FOCUS OF NEW MEDIA OUTREACH Anthony Freeman, MA Senior Evaluation Specialist](https://reader031.vdocuments.net/reader031/viewer/2022013021/5f0620f07e708231d4166f2b/html5/thumbnails/26.jpg)
SUMMARY
• Collecting and analyzing aggregate new media monitoring data was
critical in supporting agencies.
• Evaluation strategies:
– Helped contracted agencies complete their new media
activities
– Helped HD staff gauge whether agencies were meeting their
specific goals
– Helped centralize new media best practices
![Page 27: RIGOROUSLY USING EVALUATION STRATEGIES TO IMPROVE REACH … · EVALUATION STRATEGIES TO IMPROVE REACH AND FOCUS OF NEW MEDIA OUTREACH Anthony Freeman, MA Senior Evaluation Specialist](https://reader031.vdocuments.net/reader031/viewer/2022013021/5f0620f07e708231d4166f2b/html5/thumbnails/27.jpg)
SUCCESSES & CHALLENGES
Successes
• Standardized new media indicators for entire PlaySure Network
• Helped agencies do more with less
• Increased capacity of both DOHMH and agency staff
Challenges
• Diversity of capacity to execute social media among agencies
• Diversity of DOHMH staff knowledge/experience with social media
• Potential for error at every step (e.g. log data entry mistakes)
![Page 28: RIGOROUSLY USING EVALUATION STRATEGIES TO IMPROVE REACH … · EVALUATION STRATEGIES TO IMPROVE REACH AND FOCUS OF NEW MEDIA OUTREACH Anthony Freeman, MA Senior Evaluation Specialist](https://reader031.vdocuments.net/reader031/viewer/2022013021/5f0620f07e708231d4166f2b/html5/thumbnails/28.jpg)
FUTURE PLANS
• Compare performance by agency and service category
• Social Media 101 Training
• Evaluate new clients referred through social media
• Move log submission to web form to reduce submission burden
![Page 29: RIGOROUSLY USING EVALUATION STRATEGIES TO IMPROVE REACH … · EVALUATION STRATEGIES TO IMPROVE REACH AND FOCUS OF NEW MEDIA OUTREACH Anthony Freeman, MA Senior Evaluation Specialist](https://reader031.vdocuments.net/reader031/viewer/2022013021/5f0620f07e708231d4166f2b/html5/thumbnails/29.jpg)
ACKNOWLEDGEMENTS
• Contracted Agencies
• NYC DOHMH Prevention ProgramStaff
– Testing TA Unit
– Prevention TA Unit
– Research & Evaluation Unit
• Julie Myers
• Oni Blackstock
• Co-authors:
– Nancy Morisseau
– Stephen Jean III
– Lucila Wood
– Carolina Alcala
• Zoe Edelstein
• Divya Bisht
• Emily Appel
• Sarah Ramteke
• Fardina Miah