Implementation and Evaluation Toolkit for Piloting Innovations in the Safety Net

October 2013

2

How can health care providers and start-up companies or entrepreneurs partner more effectively to pilot innovations within safety net systems?

The Implementation and Evaluation Toolkit for Piloting Innovations in the Safety Net highlights important considerations for healthcare organizations as they incorporate an innovative practice or technology and set shared expectations with companies. While the toolkit focuses on fostering external partnerships with start-up companies or entrepreneurs, much of the information could also apply to internally developed innovations within safety net systems. In the toolkit, you’ll find an overview of activities in the general planning phases that can apply to any innovative practice, along with more specific considerations once a solution has been identified. It also offers a few recommendations that any organization can apply regardless of their starting place. We will continue to update this toolkit with more resources as they become available.

How to use this guidebook:

“INTERNAL” indicates assessments within the organization

“EXTERNAL” indicates assessments to ask/request of the technology company/firm

indicates a checklist or question list that can be used for assessment

indicates an additional tool or resource on the topic (for deeper exploration)

3

Organizational culture

Readiness

Technology environment

Contracting and compliance

Company/innovation assessment• Health care focus• Safety net focus• Integration into existing

environment• Innovation stage• Timeline• Resources and finances

Internal assessment• Workflows• Staff involvement• Link to existing work/

programs• Resources and finances

Link to existing reporting

Timeline

Clinic/system metrics

Patient metrics

Comparison groups

Costs and benefits

Summary/Table of Contents

PLANNING IMPLEMENTATION

p. 4

5

7

9

p. 10

11

14

p. 16

EVALUATION

• Sample checklist for culture & readiness / p. 6

• Baseline Technology Assessment / p. 8• Readiness assessment• Sample HIPAA business use agreement,

business associate contract, and confidentiality MOU

TOOLS

• Sample application for understanding a company’s innovation / p. 13

• Sample questions for internally mapping out the innovation implementation / p. 15

•

•

•

• Upfront Evaluation Considerations / p. 17

•

Workflow assessment for health information technologyOnline ROI calculator

NHS spread tool

Survey tools

4

PLANNINGBefore safety-net healthcare organizations have selected specific innovations to pilot within their system, they should have an understanding of their organizational culture and readiness. This includes necessary leadership and stakeholder engagement, existing technology infrastructure in which innovations may interface, and internal processes for testing new ideas as well as working with outside companies or organizations. Particularly for public organizations, understanding the regulatory environment for working with new companies and technologies is essential.

The planning stage for piloting an innovation involves a broad, internal assessment of the organization.

Organizational culture

Readiness

Technology environment

Contracting and compliance

PLANNING

For a deeper dive on readiness assessment: http://bit.ly/lNUgRp

Sample checklist for culture and readiness / p. 6

Baseline Technology Assessment / p. 8

Sample contract templatesHIPAA business use agreement: http://bit.ly/15Wv5cfBusiness associate contract: http://bit.ly/1frTvQFConfidentiality MOU: http://bit.ly/194UPzq

5

1. Culture & Readiness

There are various roles which support innovation within a single institution:

Creative Geniuses are the individuals coming up with the new ideas and insights. Although they can be anywhere within the organization, they tend to be frontline staff directly engaging in clinic processes.

Champions create the practical means for innovation within an organization. They seek out the creative people and ideas and support them within an operational context. These individuals tend to serve as a bridging role between frontline staff and leadership.

Leaders refer to the senior management that set the overall organizational philosophies, expectations, and priorities for innovation.

Understanding these roles within the organization is essential when planning a successful pilot and implementation. These individuals can provide feedback about successful processes within the organization that can be harnessed to meet larger objectives of improving patient outcomes and reducing costs.

It is also critical to have a clear sense of the organizational culture and readiness, since piloting and spreading innovations requires external and internal alignment for change. The following page includes a sample checklist for assessing readiness.

PLANNING

Innovation Culture: Three Roles

Innovation Leaders

Expectations and Policy

Innovation Champions

Facilitate and Guide

Creative Genuises

Ideas and Know-How

Source: Langdon Morris, “Creating Innovation Culture,” 2007

6

Yes/No/Short Response

DEFINED NEED

1. What are the top organizational priorities and how is success defined for these priorities?

2. Have innovation/improvement priorities been mapped to larger organizational strategies (e.g., by conditions or outcomes of interest)?

3. Is there an external driver for innovation (i.e., policy change, reimbursement concern)?

CULTURE

4. Which key leaders in your organization support innovation?

5. Will leaders support innovation and the related effort required to implement and sustain it?

6. Does your organization emphasize central control of processes, or decentralization and/or flexibility?

7. Are there internal and external networks to support and spread the innovation?

TIME, RESOURCES & PERSONNEL

8. Has the staff with the necessary characteristics and skills been identified as project leaders?

9. Will these project leaders commit to providing time to serve in their role?

10. Does your organization have “champion(s)” (ideally including clinicians) who will lead the effort?

11. Do you have an organizational process for reviewing incoming requests for innovative pilots? Is this process the same whether the innovation emerges internally or externally?

SUSTAINING CHANGE

12. Will your organization be willing and able to measure and assess progress for innovations?

13. Will your organization be able to reward positive outcomes and improvements resulting from innovations?

PLANNING

Sample checklist for culture and readiness

7

PLANNING

2. Technology

Each healthcare organization must know the current state of their entire information technology infrastructure/environment. This includes the electronic health record (EHR) system as well as the connectivity capability and storage capacities. Innovation champions should work with their IT departments early in the planning process to document their systems and understand how an innovative solution would fit into the existing infrastructure. Use the checklist on the following as a starting point for this assessment.

Note: Not all pilots will require full integration with an EHR system, especially pilots with an innovation that is earlier in the development process.

8

PLANNING

Name of system(s): I can do the following with my information systems

I cannot do the following with my information systems

Notes (currently used frequently or infrequently, plans for upgrading, etc.)

Electronic health record• Separate for inpatient, outpatient,

behavioral health?• Review notes?• Track health maintenance items?• Help analyze outcomes of care?• Find patients with certain characteristics?

Patient portal

Practice management system • Both billing and scheduling?

Lab interface

Electronic prescribing

Electronic referral

Registry/population management software

Network and servers

Baseline Technology Assessment

9

PLANNING

3. Contracting and Compliance

Another major step in planning is to understand the contracting and compliance issues that emerge when working with external companies. This includes business use agreements, memoranda of understanding, HIPAA privacy and security agreements, and other contractual documents. Developing a standard process and referring to existing template documents used in previous work can be extremely useful in expediting these contractual processes.

10

IMPLEMENTATIONAfter the general planning stage is complete and a specific innovation has been identified to implement, the organization needs to focus on: knowing as much as possible from the company about the innovation, understanding their plan for integrating it within the organization and conducting an internal assessment on how the innovation will impact processes within the clinic or hospital.

Company/innovation assessment• Health care focus• Safety net focus• Integration into existing

environment• Innovation stage• Timeline• Resources and finances

Internal assessment• Workflows• Staff involvement• Link to existing work/

programs• Resources and finances

IMPLEMENTATION

Sample application for understanding a company’s innovation / p. 13

Sample questions for internally mapping out the innovation implementation / p. 15

Workflow assessment for health information technology: http://1.usa.gov/qCUWOB

Online ROI calculator: http://bit.ly/105JBgG

NHS spread tool: http://bit.ly/111ZFxn

11

IMPLEMENTATION

4. Understand and Assess the COMPANY’S Innovation

This section includes examples of information that safety net organizations should ask the company to provide before they make the decision to pilot the innovation. The questions below should be answered by the company.

Often there is a disconnect between how companies and healthcare organizations discuss solutions. Understanding the different types of partnership will help move discussions beyond a sales pitch to a meaningful conversation about how you can work together to meet patient and provider needs within resource-constrained settings.

The primary questions that safety net organizations have for companies are: • What specifically would I have to do to be able to pilot? • Are you still in development mode, or are you attempting to “sell” a solution that is ready to implement?• Will this partnership be free for us, at least in the feasibility testing phase?

1 | Early-stage pre-piloting

The innovation is actively evolving and the concepts are not finalized. This stage might include minimally functional prototypes rather than full-scale products. Piloting at this stage should be considered co-development as the providers, staff, and leadership from the healthcare organization will provide substantial and lasting input about the final innovation. The upfront investment from healthcare organizations is more likely to be time and energy rather than purchasing of the product. Concerns about scalability are not as high of a priority.

Stage of development assessment:

It is important to ask companies to report on the development stage of their innovation, since this will directly impact their ability to answer all of the assessment questions below in detail. Those later in the development process will have more answers as well as more specific responses. There are three broad categories for the company to self-assess their innovation’s stage of development:

12

3 | Adoption/Spread

A “plug-and-play” solution that has been piloted in several settings with positive results and possibly longer-term sustainability. The solution is ready to be fully integrated into the healthcare organization, without significant alterations. Often, the ROI has been established during the pilot and the financing models are more established.

IMPLEMENTATION

As an innovation progresses through these steps, earlier stages will involve more usability and beta- or prototype-testing (with a higher risk profile), and later stages will involve more iteration and validation to generate increasingly stronger types of evidence for the innovation’s effectiveness (with a lower risk profile). In addition, only those in later stages of development will likely have experience with previous contractual agreements and establishing financing options to implement their solution. In general, earlier stages of development and piloting should allow for more flexible pricing arrangements, such as no-cost partnerships in which the safety net organization’s primary investment is staff time and energy.

However, at all stages of development, it is important to remember that safety net organizations have bargaining power in these arrangements, not least of which is the importance of testing solutions with more diverse patient populations to increase the solution’s credibility for a wider market. Pilots in safety net settings may also provide companies with access to new payers and funders, such as Medicaid managed care plans and healthcare foundations.

2 | Piloting

The innovation has been tested previously within a few healthcare settings to establish its proof of concept and value for patients or systems. The partnership with the company will involve adaptation of the product or practice to fit within the healthcare setting, as well as a higher level of integration within the existing information and electronic record systems. Often, several iterative rounds of piloting within a variety of contexts are needed to formalize the innovation. As the pilot will be used to establish return on investment (ROI), the payment model with the company may include hybrid approaches, such as smaller tests to establish effectiveness, followed by a longer contract with wider implementation if successful.

13

IMPLEMENTATION

Sample application for understanding a company’s innovation:

Please provide brief, but specific answers to the following questions. Responses can be provided in bullets or short sentences/paragraphs, with the entire application remaining between 250-500 words.

Overview1. Indicate the stage of development for this innovation (that is, pre-pilot,

pilot, or spread – see definitions above). 2. Please describe your solution:

• Outline how the solution is relevant for a safety net patient population (considerations for language, literacy, etc.)

• Who will use the innovation (patients only, physicians, nurses, case managers, etc.)

• When will the solution be used (e.g., in the patients’ everyday lives, during a visit, as a follow-up to a healthcare encounter, etc.)

• The intended health outcomes (e.g., reduced emergency room use, increased control of specific chronic illnesses, reduced wait times for visits, etc.)

3. Is there protected health information (PHI) involved in the innovation? If so, how is that shared and what are the security protocols to protect privacy?

4. If this solution has been piloted with patients, providers, or systems previously, please describe the implementation process, including the number of individuals involved, timeline, preliminary outcomes, and whether any features can be changed.

Integration 5. Outline the technical integration of the innovation with our systems

(i.e., is the innovation a standalone system for patients/providers to access or does it interface with existing IT systems?).• If integration is required, who will cover the costs of the interface?

How long will this take to complete and how many internal IT resources are required?

• Are devices or software needed to support the new technology?

6. Please identify the staff member or project director who will oversee the work.

7. Please the discuss any plans for fixing barriers in early phases of implementation, including interoperability or functionality issues, staff training or resources for patients who face barriers.

Timeline8. Please outline the general timeline for this pilot.9. If possible, please describe the contingencies we can build into the

agreement to ensure significant delays are not incurred.

Resources and Finances10. Please describe the financing structure for this innovation. If applicable,

please provide evidence of an established ROI or business case.11. Please estimate how much clinic/hospital IT staff time is needed to install,

troubleshoot, and maintain operations during the pilot.12. After the pilot has ended, please describe any longer-term financing model

planned to sustain or expand a successful pilot.• Please be specific about financing structures that could include

reduced fees upfront for early adopters or initial testing, bundled purchasing options, etc.

Additional considerations13. If there is a patent or trademark on this solution, please provide language

about who “owns” this innovation after adapting or piloting.14. If applicable, please provide a standard contractual agreement that has

been used for previous collaborations with healthcare organizations.

14

IMPLEMENTATION

5. INTERNAL Assessments of Innovation, Given the Specific Healthcare Context

Each safety net system must weigh the feasibility to implement the specific innovation within their own healthcare organization. It is often more successful to partner with external companies if the healthcare organization can clearly identify their priorities and needs and develop a clear idea of how the solution may map onto these. Therefore, having an outline of anticipated steps and workflow changes may facilitate implementation as well as inform the ultimate evaluation (i.e., understand at which point the implementation succeeded and/or failed). This section focuses on the safety net system’s internal assessment of the innovation implementation within their organization.

?

15

IMPLEMENTATION

Sample questions for internally mapping out the innovation implementation:

Workflows> Does the innovation require integration within complex, interrelated processes?

> What is the expected change in clinic workflow? Is the scope of workflow changes expected to increase with widespread implementation?

• Are the expected workflow changes permanent or temporary?

> Can the number of steps required to implement be mapped out? • “Length” = the number of sequential sub-processes or steps for

using or implementing an intervention• “Breadth” = number of choices presented at decision points

Staff involvement> Is there a champion for this specific project that can secure the resources (from clinic staff to IT) and approval necessary to move a pilot forward?

• Is this person highly motivated and interested in investing in this solution?

• Same person cannot be the champion for all innovations

> How many organizational units (teams, clinics, departments) or types of people (providers, patients, managers) are affected?

> Who are the team members on the frontline involved with the implementation of this innovation? How many departments do they represent? Do they have clearly delineated roles and/or formal communication channels?

> What is the level of provider, staff, and patient engagement needed with the innovation?

> Is additional training for providers and staff needed?

Relationships with existing programs/initiatives> How does the innovation relate to existing patient self-management needs as well as existing patient education/skill-building programs?

> Is the timing right for implementing this innovation (i.e., will it compete with other strategies or needs)?

Finances and resources> Are the payment models aligned for implementing this innovation?

> Are there specific budget or grant cycles throughout the year that make piloting more feasible?

> Are the financial stakeholders involved in planning discussions, particularly the Medicaid managed care plans and/or other major health plans?

• That is, can the long-term funding model for this innovation be articulated upfront?

• The higher the health plan market share within your system, the stronger the case for partnering – even if the plan makes investment decisions conservatively and more often focused on successful spread of practices

> What resources, if any, have been allocated for the innovation?

> Other resources/costs include:• Staff time needed for testing, meetings• Leadership time• Lost productivity from testing, meetings• Cost of other equipment or needs to support the innovation

(connectivity, technology, new devices, change in work-space)

16

EVALUATIONOnce implementation has begun, there are several additional considerations for evaluating the impact of the innovation. Within healthcare settings, the importance of strong evidence to support the innovation cannot be overstated. While early pilots should not be held to the level of rigor of a randomized trial in evaluation, there are simple considerations for measurement and outcome selection that can make the business case for future implementation and longer term commitment.

The evaluation of outcomes is a joint INTERNAL and COMPANY process on which safety net healthcare organizations and external companies must agree.

Link to existing reporting

Timeline

Clinic/system metrics

Patient metrics

Comparison groups

Costs and benefits

EVALUATION

Upfront Evaluation Considerations: p. 17

Survey tools: http://1.usa.gov/18ztGui

17

EVALUATION

Upfront Evaluation Considerations

Overall> How does this innovation match to metrics for 1) overall organizational strategy, 2) quality improvement/measurement, and/or 3) patient experience initiatives?

• Can we choose primary outcomes that are already being measured and evaluated?

> Length of follow-up needed • 2-month metrics may be more focused on process, patient/staff

satisfaction or use, or behavioral intentions, while 6- to 12-month follow-up may include clinical outcomes – having a plan for both short and longer term evaluations can set the stage for the next round of implementation/spread

Clinic/system outcomes> Provider/clinic and/or system impacts?

• Reduced no-shows, increased access (including increases in appropriate follow-up appointments or referrals)

• Preventive screenings or disease testing rates at a population level• Overall management of chronic disease within a panel (i.e., proportion

of diabetes patients with A1c<8%)

> Efficiency • Time to appointment, time to deliver results, number of patients seen,

etc.

Patient or staff experience> Staff Satisfaction (e.g. Gallup)> Usability/Engagement: http://bit.ly/svg16> Patient Experience/Satisfaction (e.g. CAHPS survey): http://1.usa.gov/1aU0AaH> Percentile Top Box Scores for 12-month 4-point Adult: see Appendix, pg. 18> Qualitative feedback from comment cards

Patient self-reported health or lifestyle changes> Disease-specific

• Individual’s A1c change for diabetes, symptom management for COPD• Overall resource for measures: http://stanford.io/qcsv60

> Access, or receipt of timely visit (see CAHPS items above)

> Self-reported quality of life (http://1.usa.gov/d1hgSO) or depressive symptoms (http://stanford.io/17Ka0RF), patient activation(http://bit.ly/vx1F4H)

> Reduced out of pocket costs

> Self-efficacy or confidence in ability to perform chronic disease self-management tasks (http://stanford.io/r20g3h)

Control/comparison groups> There are a wealth of innovations that appear to be effective but have not spread with similar success

• Most patients will improve once they begin using the technology – but they would likely improve somewhat without the technology (statistically known as “regression to the mean”) – importance of a comparison group

• Reliable changes from baseline (among the same group of individuals)• Random waitlists or staggering of rollout to create quasi-experimental

control groups

Measurement> Are the intended changes in metrics/outcomes also related to state or national reporting requirements? If so, how are the data for those reports currently gathered and reported?

• If so, are these feasible baseline metrics to be able to examine changes over time?

> Are there accounting systems in place for capturing intervention costs (including investment, supply, and opportunity costs)?

• Or perhaps surrogate clinical measures that could indicate financial impact?

> Other benefits/value to potentially measure include:• Potential future partnerships with funders, grants, seed money, etc. • Reducing downstream costs, improved health outcomes, patient experience,

staff satisfaction

18

EVALUATION

Appendix: Percentile Top Box Scores for 12-month 4-point Adult

Composite/Item CAHPS DB Overall

90th Percentile

75th Percentile

50th Percentile

25th Percentile

Getting Timely Appointments, Care, and Information 63 80% 71% 66% 62%

Got appointment for urgent care as soon as needed 67 88% 78% 69% 64%

Got appointment for check-up or routine care as soon as needed 70 88% 81% 74% 68%

Got answer to phone question during regular office hours on same day 67 83% 76% 69% 63%

Got answer to phone question after hours as soon as needed 64 82% 74% 65% 53%

Wait time to be seen within 15 minutes of appointment time 49 69% 60% 54% 44%

How Well Doctors Communicate With Patients 83 89% 86% 83% 79%

Doctor explained things clearly 85 92% 88% 85% 80%

Doctor listened carefully 86 93% 89% 86% 81%

Doctor gave easy to understand instructions about taking care of health problems

84 92% 88% 83% 79%

Doctor knew important info about medical history 75 83% 80% 75% 69%

Doctor showed respect 89 94% 92% 89% 86%

Doctor spent enough time 81 89% 86% 81% 78%

Helpful, Courteous, and Respectful Office Staff 81 92% 87% 83% 80%

Office staff was helpful 76 89% 83% 79% 75%

Office staff showed courtesy and respect 86 94% 91% 88% 85%

Follow-up on Test Results 70 86% 83% 79% 67%

Patients’ Rating of the Doctor 79 89% 84% 80% 76%

19

FINAL ASSESSMENT OF THE INNOVATION

Although most systems would prefer to only pilot innovations that fall into the upper left quadrant of the figure below, we understand that this is not the reality. Hopefully these assessments for planning, implementation, and evaluation of innovations can help each safety net organization better understand where on the spectrum an innovation falls. We hope the tools included here can help organizations understand when an innovative solution may hold potential for higher impact and will encourage them to partner with companies as they pursue the Triple Aim objectives of improved health outcomes, increased patient satisfaction, and lower costs.

EVALUATION

Feasible to implement, high

impact

Hard to implement, high impact

Hard to implement, low impact

Feasible to implement, low

impact

20

Never worked with a company within your organization? Although all of the steps outlined above may feel overwhelming to tackle at once, there are several smaller steps that can be taken to get over the inertia and move toward testing innovative solutions:

1. In order to be on the same page about what type of solutions make sense within safety net settings, inviting a company to observe even a few hours at clinic may provide them with deep insight into the demands of meeting the needs of our patient populations.

2. It is often easiest to get leadership and payor approval by starting with innovations that have already been tested in other clinical settings and have demonstrated value/(cost)-effectiveness.• Organizations like the Center for Care Innovations can be a starting place for learning about existing solutions that meet

safety net system priorities – even opening up early dialogue with companies can be an important 1st step

3. Integration into clinical workflows and EHRs may be goals to work toward, but many innovative pilots don’t need integration to demonstrate success (e.g., patient-facing with a dedicated case manager)• Aim for “minimal footprint in operations” and simpler data collection on the 1 or 2 primary outcomes that are high strategic

priorities (costly outcomes, patient satisfaction)

4. Having early conversations with payors such as health plans about what type of metrics (cost and outcomes) are needed for long-term payment models and ultimate sustainability

GETTING STARTED

Please send us your feedback:veenu@careinnovations.orgwww.careinnovations.org

Contributors: Courtney Lyles, PhD (UCSF Division of General Internal Medicine at San Francisco General Hospital); Veenu Aulakh, Sarah Frankfurth, Roza Do (CCI)