request for proposals: hopkins/usda participant research ......apps, web tools and platforms for...
TRANSCRIPT
USDA-FNS HPRIL
Request for Proposals:Hopkins/USDA Participant Research Innovation Lab for Enhancing WIC Services (HPRIL)March 29, 2019
USDA-FNS HPRIL
Webinar Logistics
This webinar is being recorded. The recording will be posted on the HPRIL website
All participants are currently muted
We will break for questions in the middle and at the end of the presentation
Want to ask a question?
▪ Click and we will unmute you
▪ Type your question in the chat box
USDA-FNS HPRIL
Agenda
▪ Background
▪ Project Details
▪ Eligibility Details
▪ Q&A
▪ Proposal Requirements
▪ Review Process
▪ Important Dates
▪ Q&A
USDA-FNS HPRIL
BackgroundWhat specific problem are we trying to solve and what’s been studied so far?
USDA-FNS HPRIL
WIC’s Child Retention Challenge
Source: U.S. Department of Agriculture, "National- and State-Level Estimates of WIC Eligibles and WIC Program
Reach in 2015," February 2018.
USDA-FNS HPRIL
Barriers to Retention
Lack of knowledge
about WIC/eligibility
Lack of
transportation
Lack of child care
Language barriers
Frequency of
certification
Quality of core
client services:
scheduling, timing,
wait times/time
burden, not liking
WIC foods
Quality of vendor
services:
Hard-to-find foods,
hours, problems
with purchasing,
and interactions
with vendors
Fewer visits with pediatric
care provider to enable
linkage/referral
WIC clinics are not child-
friendly
Immigration Issues
Conflicts with family
food preferences
Competing priorities with
work and other activities
USDA-FNS HPRIL
Positive and Negative Associations with Retention
▪ Missing benefits in the months leading up to 1st birthday
▪ Under redemption (<75%) of WIC benefits
▪ English language preference
▪ Mother employed after a child’s birth
▪ More educated household
▪ Higher income household
▪ Infants breastfed 6-12 months
▪ Prenatal intention to breastfeed
▪ Receipt of online nutrition education
▪ More months of prenatal enrollment in WIC
▪ Other family members receiving WIC
▪ Participation in Medicaid
USDA-FNS HPRIL
Existing Work Addressing Retention
▪ Apps, web tools and platforms for appointment reminders, education, breastfeeding support, and tracking benefits
▪ Web and mobile tools for eligibility and enrollment
▪ Clinics in non-traditional
spaces and mobile clinics
▪ Community partnerships and
co-location
▪ WIC messaging campaigns on
program and eligibility
USDA-FNS HPRIL
MIS: A Powerful Innovation Tool Demographic patterns
Client food behavior
Nutrition education compliance
Missed appointments
Non-redemption of benefits
Other data
It can help us:
▪ Identify risk for early termination
▪ Identify gaps in service
▪ Test and evaluate innovations at the local agency level with harvested data
A rich, often untapped database
USDA-FNS HPRIL
Current WIC MIS ActivityCONSORTIUM AGENCIES
WOW (WIC on the Web) MD, DE, and U.S. Virgin Islands
*MI updated the MD WOW system to adapt to EBT. This
system has been transferred to IN, and CT.
Crossroads AL, NC, VI, WV
HANDS (Health and Nutrition Delivery System) AZ, American Samoa, Guam, Navajo Nation, and the
Commonwealth Northern Mariana Islands
MPSC (Mountain Plains State Consortium ) CO, ND, UT, VT, WY
MSC (Multi-State Consortium) KS, Inter-Tribal Council of Arizona, NH
SPIRIT (Successful Partners in Reaching Innovative Technology) AL, AK, ME, MN, MS, MO, MT, 2 Maine ITOs, 6 New Mexico
ITOs, & 7 Oklahoma ITOs
State Developed Information System CA, DC, FL, GA, ID, IL, IA, KY, LA, MA, NE, NV, NJ, NM, NY,
OH, OR, OK, PA, RI, SC, SD, TN, TX, WA, WI
USDA-FNS HPRIL
HPRIL
USDA-FNS has funded Johns Hopkins University to create the Hopkins Participant Research Innovation Laboratory for Enhancing WIC Services (HPRIL)
The goal of HPRIL is to identify, fund, and support 3-5 local WIC agency projects to implement innovative tools and evaluate their impact on child retention using MIS data
USDA-FNS HPRIL
Project DetailsWhat will your responsibilities be during the project period?
USDA-FNS HPRIL
HPRIL Project OverviewFunded projects will:
▪ Implement and evaluate innovative tools addressing declining rates of WIC retention of children ages 1-4
▪ Implement and evaluate customer service best practice activities common across all funded projects; activities TBD based on funded agency capacities
▪ Use MIS data to identify target population and local retention challenge and assess the impact of the innovative tool on retention
▪ Receive up to $150,000 over 18 months between August 2019 and April 2021 for:▪ Approximately 3 months of planning
▪ Approximately 12 months of implementation
▪ Approximately 3 months of evaluation
USDA-FNS HPRIL
Project Roles OverviewHPRIL will… Local Agency Sub-grantees will…
Provide training and technical assistance to develop and strengthen
implementation and evaluation plans, including data collection
instruments
Develop implementation and evaluation plans
Provide opportunities for ongoing TA throughout project
implementation period
Implement the intervention and customer service best practice
activities
Receive MIS and other de-identified data from local agencies Collect relevant MIS and other process and outcome evaluation data
and send it to HPRIL, de-identified
Provide training and technical assistance on analyzing process and
outcome evaluation data
Work with HPRIL to analyze own agency’s process and outcome
evaluation data
Perform some of the data analysis for one or more local WIC
agencies as needed
Request additional assistance with data analysis, if needed
Provide training and technical assistance on interpreting data and
sharing results
Interpret data and share results
Collectively analyze evaluation data from all projects in a cross-
project evaluation and share results
Page 9 in RFP
USDA-FNS HPRIL
Project Focus:Child Retention
Child = 1-4 year olds
Retention = recertification within 14 months
Source: U.S. Department of Agriculture, "National- and State-Level Estimates of WIC
Eligibles and WIC Program Reach in 2015," February 2018.
USDA-FNS HPRIL
Identify Target Population and Local Retention ChallengeGroup at risk for termination within 1-4 year olds
Identified using:
• Retention data for all ages and your target population
• Other MIS and non-MIS data
• Local staff wisdom
USDA-FNS HPRIL
Identify and Implement an Innovative ToolA physical or digital product, instrument, skill, method, or protocol that is:
▪ Applied to WIC management, clinical, or retail services
▪ Logically linkable to possible impact on retention
▪ Innovative
▪ Scalable
▪ Relatively cost-neutral
▪ Culturally and socially sensitive
USDA-FNS HPRIL
What is Innovative?
“Mash-up” Innovation = existing tools used in new ways or for new populations
Example #1: Existing tool that has
not been studied for retention
Example #2: Tool minimally adapted
from existing one*
*<10% total award funds can go towards tool adaptation costs; given
the length of the project period, funds will not be available for new
tool development (i.e. app or curriculum development from scratch)
as all tools must be implementation-ready by the end of the 3-
month planning period
Example #3: New tool inspired by a
service or experience in another
discipline or industry that you can make
a good case for potential impact on
retention*
Pages 10-13 in RFP
USDA-FNS HPRIL
Management Tools
Examples:
Mobile apps with
information on
appointments and
resources
WIC brand awareness
initiatives
Partnerships between
WIC and other not-
for-profit agencies
USDA-FNS HPRIL
Examples:
On-call electronic
nutrition education
nutritionists (teleWIC)
Text messaging with
targeted message
content
Provision of WIC
services in non-
traditional settings
Clinic Tools
USDA-FNS HPRIL
Retail Tools
Examples:
Mobile apps aiding
the shopping
experience
Mobile apps
providing
information about
current benefits
Vendor training
development and
implementation
USDA-FNS HPRIL
Define Your SMART Objectives
Outcome Objectives
Describe the intended effect of implementing your innovative tool in the target population. They can be short-term, intermediate, or long-term
Example: Improve retention
rates of 2-year-olds exposed to
the innovative tool by 5%
between November 15, 2019 and
November 15, 2020
Process Objectives
Describe the activities you will deliver during project implementation
Example: Train all 20 client-
facing agency staff on customer
service best practice activities by
November 15, 2019
Specific, Measurable, Achievable, Realistic, Timebound
USDA-FNS HPRIL
Determine the Implementation and Evaluation Design
You will need a minimum of 500 children* in your target population.
Option #1: Pre/Post Analysis
with one or more clinics
Option #2: Intervention vs.
Comparison Group with at
least two clinics
Option #3: Propose a different
design
*May be achieved with one agency or a consortium of
agencies working together on the project
Pages 14-15 in RFP
USDA-FNS HPRIL
Implement Your Innovative Tool
Create and execute a feasible plan for project implementation and oversight.
Identify key staff and partners who will contribute to successful project implementation.
Plan for sustaining the tool beyond the project period.
USDA-FNS HPRIL
Implement Customer Service Best Practice Activities
Commit to continuing or implementing some customer service best practice activities
Common across all funded projects
List of activities to be finalized post-award, based on the capacities of funded agencies
Appointment
reminder
calls/texts
Missed
appointment
follow-up
calls/texts
Same day
appointments
Staff training
on customer
service
Participant-
centered
communication
and counseling
30-day
temporary
eligibility
approval
Accept
electronic
eligibility
documents
Customer Service
Best Practice
Activities
USDA-FNS HPRIL
Develop and Implement an Evaluation Plan
Process Evaluation
How you will determine whether project activities have been implemented as intended
Outcome Evaluation
How you will determine whether the project
has impacted the target population and
achieved or made progress towards desired
outcomes
USDA-FNS HPRIL
Develop and Implement an Evaluation PlanInvolves:
▪ Defining process and outcome evaluation questions related to your project objectives
▪ Deciding how you will collect data: with MIS data reports, surveys, interviews, focus groups, etc.
▪ Developing data collection instruments
▪ Outlining how you will analyze your data
▪ Determining personnel who will collect and analyze data
▪ Determining how HPRIL can provide training and technical assistance to support your evaluation activities
▪ Implementing your plan
USDA-FNS HPRIL
Collect and Report MIS Data to HPRIL
Required*
▪ Retention**, ages 1-4 in total
▪ Retention**, individual age groups
▪ Benefit issuance data, ages 1-4 in total
Dependent on Your Project
▪ Demographic characteristics
▪ Benefit redemption
▪ Kept appointments
▪ Nutrition risk priority level
▪ Nutrition education participation
▪ Breastfeeding status
▪ Household size
▪ Other variables of interest
De-identified data will be collected and submitted to HPRIL and will be analyzed in aggregate
*This is consistent with current reporting requirements for
all local WIC agencies
**Recertification within 14 months
USDA-FNS HPRIL
What will Hopkins do with your data?
We will work with each sub-grantee on their data analyses
We will perform a cross-project evaluation
USDA-FNS HPRIL
What Training and Technical Assistance Can You Expect from HPRIL?
Q&A opportunities for applicants throughout the proposal-development process
A kick-off training workshop at the beginning of the funding period for funded sub-grantees to refine implementation and evaluation plans
Ongoing TA support on project implementation and evaluation throughout the project period
USDA-FNS HPRIL
Additional Project Requirements
Attend kick-off training workshop in Baltimore in September 2019
Participate in monthly CoIN calls:▪ CoIN stands for collaborative innovation network
▪ Calls with HPRIL and all sub-grantee agencies for connecting, sharing successes and challenges, and troubleshooting issues
Submit quarterly technical and financial progress reports
Submit final report within 2 months after the project funding period
Attend a final reporting workshop at USDA/FNS in September 2021
USDA-FNS HPRIL
Eligibility DetailsWhat are the eligibility requirements for submitting a proposal?
USDA-FNS HPRIL
EligibilityMIS Capacity: Must have an MIS system with specific capabilities outlined in attachment #1 of application package
Assurances: Must meet specific assurances outlined in attachment #3 of application package
Must be a local WIC agency, Indian Tribal Organization (ITO) WIC Program, or U.S. Territory WIC Program
Must have letter of commitment from State Agency to help obtain MIS data reports
Must meet criteria outlined in the 2 application package attachments
One proposal per local agency
USDA-FNS HPRIL
Attachment #1: MIS system capabilities
USDA-FNS HPRIL
USDA-FNS HPRIL
Attachment #2: Assurances
USDA-FNS HPRIL
Q&A
USDA-FNS HPRIL
Proposal RequirementsWhat should you include in your proposals?
USDA-FNS HPRIL
Register for HPRIL Grant Submission PlatformSend us by May 21, 2019:
▪ Name, email address, and telephone number for Project Director
▪ Name of Agency
▪ Name and email for individual who will upload the application (if different from the Project Director)
▪ Send to Martelle Esposito at [email protected]
We’ll send you:
▪ A unique link to a secure, encrypted Johns Hopkins Box, so you can upload your application package
USDA-FNS HPRIL
Or Plan to Mail the Application Package Mail to:
Martelle Esposito
Senior Research Program Manager, HPRIL
Department of Population, Family, and Reproductive Health
615 N. Wolfe Street, E4031
Baltimore, MD 21205
410-502-0692
Please let Martelle Esposito know you plan to do so by sending her a note at [email protected]
USDA-FNS HPRIL
Application Package Components
Proposal
▪ Title/Cover Page
▪ Project Summary
▪ State or Local Agency Context
▪ Project Goals and Objectives
▪ Description of Proposed Innovative Tool
▪ Description of Implementation Plan
▪ Description of Evaluation Plan
▪ Description of Your Organization’s IRB* Process (if applicable)
Additional Documents
▪ Timeline
▪ Attachment #1: MIS Capabilities
▪ Attachment #2: Assurances
▪ Resumes
▪ Letter of Commitment from State Agency
▪ Budget form and justification narrative document
*IRB stands for institutional review board and is an administrative body
established to protect the rights and welfare of human research subjects
recruited to participate in research activities conducted under the auspices
of the institution with which it is affiliated.
USDA-FNS HPRIL
Cover Page
Please complete the HPRIL cover page form and include it with your application
USDA-FNS HPRIL
Local Agency Context
Tell us about:
▪ Your organization and client population
▪ Your target population for this project
▪ Why and how you chose your target population, using MIS and other data and local program wisdom
▪ Your current customer service best practice activities and capacity to implement others
▪ Your agency’s MIS reporting capabilities
▪ Any other projects or circumstances that the agency is/could be engaged in that could impact child retention, including EBT transition
USDA-FNS HPRIL
Project Goals and Objectives
Tell us about:
▪ Your project goals and SMART objectives
▪ Tell us how they link to your local context and retention challenge
▪ The implementation and evaluation design you have selected for your project and why you think it is a good fit
USDA-FNS HPRIL
Description of Proposed Innovative Tool
Tell us about:
▪ Your tool, its purpose, and its potential to address your challenge and impact child retention; link it to project goals and objectives
▪ How the tool is culturally and socially sensitive and relevant to your target population
▪ How innovative it is, and its stage of development
▪ Your plans to adapt, pilot-test, and refine your tool during the 3-month planning period
▪ Key partner engagement to success of your project
▪ Generalizability and scalability of the tool
▪ How it is relatively cost neutral to sustain once implemented
▪ How you will sustain the implementation of the tool beyond the project period
USDA-FNS HPRIL
Description of Implementation Plan
Tell us about:
▪ The appropriate environment for implementation (e.g. clinics, retail)
▪ The various implementation steps, including training, rollout, resources, management, and oversight; steps should be included in your timeline
▪ The feasibility of implementation in your agency during the project period
▪ Key staff, their expertise, and how they will contribute to successful implementation of the project
▪ How you will incorporate or sustain customer service best practice activities
▪ Key partner engagement that will contribute to the success of your project
USDA-FNS HPRIL
Description of Evaluation Plan
Tell us about:
▪ Your agency’s evaluation capacity and planned capacity-building activities
▪ Your plan for process and outcome evaluation, consistent with project goals and objectives
▪ The data you plan to collect to answer your process and evaluation questions and how you plan to collect it (e.g. MIS reports, surveys, interviews, focus groups, etc.)
▪ Data quality issues or concerns and procedures to maintain data quality and integrity of analyses
▪ How you envision HPRIL supporting your project’s implementation and evaluation activities, including data analysis needs
USDA-FNS HPRIL
Timeline
▪ Build your project timeline from the provided timeline template, which includes some of the key project dates
Year 1Activity
Month Number1 2 3 4 5 6 7 8 9
1
0
1
1
1
2
Quarterly progress reports to HPRIL x x xCoIN Telephone/WebX conference calls with sub-
grantees (monthly)X X X X X X X X X X X
X
Sub-grantee Evaluation Plan Development X
Coordinate and conduct a web-based post-competition
meeting with guidance on developing evaluation plan
X
Pre- Evaluation of Training Competencies Survey Sent
to Sub-grantees
X
CITI Completion Certificates from sub-grantees X
Training Workshop for sub-grantees Evaluation & MIS
Data in Baltimore
X
Evaluation plan developed by sub-grantee with
HOPKINS HPRIL
X
Innovation Project Implemented X X X X X X X X XEvaluation Plan Implemented X X X X X X X X XCommon Measures Data Collection X X X X X X X X X
Common Measures Baseline Data Due to HPRIL X
First Progress Report due to HPRIL X
X
Year 2Month Number 1
3
1
4
15 1
6
1
7
1
8
1
9
2
0
2
1
2
2
Quarterly progress reports to HPRIL X
CoIN Telephone/WebX conference calls with sub-
grantees (monthly)
X X X X X X X X X X
Second Progress Report from sub-grantees XInnovation Project Implemented X X X X XEvaluation Plan Implemented X X X X XCommon Measures Data Collection X X X X XCommon Measures Final Database due to HPRIL XFinal Report due XSlides for telecast webinar of sub-grantee findings due
to HPRIL
X
Telecast webinar of sub-grantees findings X
USDA-FNS HPRIL
Budget
Complete the budget form provided
Provide a narrative justification for each line item in a separate document
USDA-FNS HPRIL
Letter of Commitment from State Agency
A letter of commitment from your State WIC agency must state how they support you and how they will help you obtain data in a timely manner for this project, if necessary
USDA-FNS HPRIL
All Documents are Downloadable from the HPRIL Website
https://www.jhsph.edu/hpril
HPRIL→ Funding Opportunity→ How to Apply
USDA-FNS HPRIL
Review ProcessHow will your proposal be evaluated?
USDA-FNS HPRIL
Review Process
Your proposal will be independently reviewed and ranked by a panel of up to 5 reviewers external to HPRIL
The panel of reviewers will make recommendations for funding based on scores and state/regional diversity
Considering review panel recommendations, the HPRIL team will finalize a list of 3-5 agencies for the award
USDA-FNS HPRIL
Review Criteria
Eligibility screening questions
Scoring breakdown for a possible 100 points:
▪ 20 points for local agency context, goals, and SMART objectives
▪ 30 points for relevance, innovation, and significance of the proposed tool
▪ 30 points for implementation feasibility, including implementation plan, budget and justification, personnel, and timeline
▪ 20 points for evaluation approach
Pages 25-27 in RFP
USDA-FNS HPRIL
Important DatesWhat dates do you need to get on your calendar now?
USDA-FNS HPRIL
Important Dates▪ Registration for HPRIL grants submission platform: Between now and May 21,
2019
▪ Q&A period: March 29 – May 15, 2019
▪ Applications due: May 24, 2019 by 11:59 p.m. ET
▪ Award notification date: July 12, 2019
▪ Award start date: As early as August 1, 2019, defined by sub-grantee and contingent on signing contract
▪ Award period: August 2019 – April 2021, dependent on award start date
▪ Training workshop: Mid-September 2019
▪ Final reporting workshop date: September 2021
USDA-FNS HPRIL
Q&A Opportunities During Proposal Development▪ April 16, 2019 at 3:00 p.m. - 4:00 p.m. EDT: Attachments Q&A Session
▪ April 17, 2019 at 3:00 p.m. - 4:00 p.m. EDT: Local Agency Context Q&A Session
▪ April 18, 2019 at 3:00 p.m. - 4:00 p.m. EDT: Project Goals/Objectives, Description of Proposed Innovative Tool, and Implementation Plan Q&A Session
▪ April 23, 2019 at 3:00 p.m. - 4:00 p.m. EDT: Evaluation Approach Q&A Session
▪ April 24, 2019 at 3:00 p.m. - 4:00 p.m. EDT: Timeline and Budget Q&A Session
▪ April 25, 2019 at 3:00 p.m. - 4:00 p.m. EDT: Open Q&A Session
▪ May 8, 2019 at 3:00 p.m. - 4:00 p.m. EDT: Open Q&A Session
▪ May 15, 2019 at 3:00 p.m. - 4:00 p.m. EDT: Open Q&A Session
▪ Ongoing: Q&A posted to the HPRIL website (https://www.jhsph.edu/hpril HPRIL→ Funding Opportunity)
USDA-FNS HPRIL
Contact [email protected]
Martelle Esposito, MS, MPH
HPRIL Senior Research Program Manager
Department of Population, Family, and Reproductive Health
615 N. Wolfe Street, E4031
Baltimore, MD 21205
410-502-0692
USDA-FNS HPRIL
References▪ Slide 4: WIC’s Child Retention Challenge
▪ U.S. Department of Agriculture, "National- and State-Level Estimates of WIC Eligibles and WIC Program Reach in 2015," February 2018.
▪ Slide 5: Barriers to Retention▪ National Academies of Sciences, Engineering, and Medicine (NASEM). (2017). Review of WIC Food Packages:
Improving Balance and Choice: Final Report. NASEM; Health and Medicine Division; Food and Nutrition Board; Committee to Review WIC Food Packages. Washington, DC: National Academies.
▪ Geller, D.M., Harrington, M., Huang, G., et al. (2012) National Survey of WIC Participants II Volumes I – IV. U.S. Department of Agriculture, Food and Nutrition Service, Office of Research and Analysis.
▪ Slide 6: Positive and Negative Associations with Retention ▪ Jacknowitz, A., Tiehen, L. (2010). WIC Participation Patterns: An Investigation of Delayed Entry and Early Exit. U.S.
Department of Agriculture, Economic Research Service, Economic Research Report 102759.
▪ Whaley, S., Whaley, M., Au, L., Gurzo, K., & Ritchie, L. (2017). Breastfeeding Is Associated With Higher Retention in WIC After Age 1. Journal Nutrition Education and Behavior, 49, 810-816.
USDA-FNS HPRIL
References▪ Slide 7: Existing Work Addressing Retention
▪ Bensley, R., Hovis, A., Horton, K., Loyo, J., Bensley, K., Phillips, D., & Desmangles, C. (2014). Accessibility and Preferred Use of Online Web Applications Among WIC Participants With Internet Access. .Journal of Nutrition Education and Behavior,46(3), S87–S92.
▪ Brusk, J.J. & Bensley, R.J. (2016). A Comparison of Mobile and Fixed Device Access on User Engagement Associated With Women, Infants, and Children (WIC) Online Nutrition Education. JMIR Research Protocols, 5, e216.
▪ Colorado WIC Program. (2015). Texting for Retention Program. Final Report: WIC Special Project Grant, Fiscal Year 2014. Denver, CO: Colorado Department of Public Health and Environment.
▪ Hull, P., Emerson, J., Quirk, M., Canedo, J., Jones J., Vylegzhanina, V., Schmidt, D., Mulvaney, S., Beech, B., Briley, C., Harris C., & Husaini. B. (2017). A Smartphone App for Families With Preschool-Aged Children in a Public Nutrition Program: Prototype Development and Beta-Testing. JMIR Mhealth Uhealth. 5, e102.
▪ Illinois chapter of the American Academy of Pediatrics. (2015). WIC to 5. Pocket Guide for Health Care Providers.▪ Neuberger, Z. (2016). Modernizing and Streamlining WIC Eligibility Determination and Enrollment Processes. Center on
Budget and Policy Priorities. Washington, DC.▪ Rodriguez, A. (2018). SC DHEC debuts new mobile unit to help Upstate women & children. Fox Carolina (Meredith
Corporation).▪ Vermont State WIC Program (2017). WIC2FIVE: Using Mobile Health Education Messaging to Support Program
Retention. Vermont 2014 WIC Special Project Mini-Grant. Final Report. Burlington, VT: Vermont Department of Health WIC Program.
▪ Virginia WIC Program (2018). Virginia WIC on Wheels. Virginia WIC Newsletter, Virginia Department of Health. ▪ Whaley, S., Whaley, M., Au, L., Gurzo, K., & Ritchie, L. (2017). Breastfeeding Is Associated With Higher Retention in WIC
After Age 1. Journal Nutrition Education and Behavior, 49, 810-816.
USDA-FNS HPRIL
Q&A