Clinical Team Led Design for determining ROI on Clinical

Redesign Professor Jon Patrick

iCIMS and HLL

Brief CV

• 5 degrees with 2 in psychology and a PhD in Computer Science (ML-Analytics) and – Registered Psych for 17 years. Fellow of ACS, ACHI.

• Joined Massey Uni 1994 Chair of IS • Joined U Sydney in 1998 Chair of IS • Changed Chair to Language Technology 2003 • Won Eureka Prize in 2005 for Scamseek • Started in Health IT 2006 – subsequently: 100+ projects • Published critical review of Cerner Firstnet 2010 • Left University in 2012 to start up iCIMS and HLL

consultancies

Project Partners

• Cancer Council of Victoria (Registry-Radiology) • Cancer Institute NSW (Registry-Radiology) • RPAH (ED, ICU, Pathology, QUPP) • Prince of Wales (Pathology) • Westmead (Breast Cancer Inst, Aged care) • Womens, Melbourne, PMCC (GynaeOnc & Breast,

Ovarian and Cervical Research) • Nepean (ED) • Liverpool (Pathology) • Northern Sydney LHD (Antibiotic Stewardship) • Sydney Adventist Hospital ( modeling 6 systems)

Health IT Horror Stories

• Healthsmart – Victoria

• Cerner FirstNet – NSW

• NPfIT – UK

• Why did they fail ?

• What was supposed to be the ROI

• How was it determined

General IT Project Assessments

“There is little evidence that consultants, IT companies, public agencies, or many practitioners and academics have learned one of the key lessons of IS failure — that large and ambitious projects should be treated with great caution or avoided altogether.” S. Goldfinch, Pessimism, Computer Failure and IS Development. Public Admin Review, 2007.

More Issues

• “ In some cases, projects are reported to be 90 percent complete for half the duration of the project — an obvious impossibility. “

• “In a survey of IT auditors, it was found that the completion effect classified more than 70 percent of runaway projects.” Goldfinch op cit

Using External Consultants

“higher levels of contracting . . . were less likely to have the project delivered on time and within budget and achieved less benefit from the technology in productivity and performance, organization decision making, and customer service” Goldfinch op cit

Contracts and Outsourcing

“there is considerable evidence that contracts are not effective at controlling projects and providing sanctions when IS projects fail and that outsourcing has costs and dangers of its own.”

Goldfinch op cit

More recent analysis

“Big-bet IT projects have a history of causing trauma in the private sector as well.

A 2011 survey of business and IT executives found that 75 percent believed their projects were either usually or always doomed from the start.

Those expectations are firmly rooted in reality –research by the Standish Group International found that in 2012, only 10% of projects with value over $10 million were successfully completed on time and within budget.”

The seven deadly sins of HealthCare.gov Ars Technica, by Sean Gallagher - Oct 30 2013, 12:00am EST

HealthSmart – no accountability, no comparative studies

Although the funding agreement with health services required them to complete a benefits realisation plan prior to program implementation and to report on this before and after clinical ICT system implementation, funding was not allocated to assess the achievement of purported benefits. Further, DH has not studied the performance of clinical ICT systems at non-HealthSMART sites so that it could evaluate the benefits of the HealthSMART clinical ICT system compared to other systems. Consequently, DH is unable to evaluate the benefits and value for money of one system versus others.

Victorian Auditor General Oct 30th 2013

Cautionary Approach to Redesign

• If the decision for redesign is made to proceed with IS development, then the question becomes,

• how can this be done with – the least disruption,

– the least cost,

– and the least risk and uncertainty? Goldfinch (op cit)

• How can we achieve the greatest effect with the least change J. Patrick, iCIMS

Evidence based Solutions

• Believe that developments will work only when they can be shown to work.

• A recent claim by large HIT company says that you cannot show ROI on HIT with only 30% of their corporate software, rather you need have 90% of their software to show a ROI ($500 million)

• What is the failure here? – the first 30% of their software or the short-sightedness of the client.

Good HIT and Bad HIT

Good Health IT ("GHIT") provides a

good user experience,

enhances cognitive function,

puts essential information as effortlessly as possible into the physician’s hands,

can be easily, substantively and cost-effectively customized to the needs of medical specialists and subspecialists (Scot Silverstein, 2012)

Benefits Criteria of Good HIT

• Staff productivity

• Patient safety

• Audit and Analytics for all groups

Benefit Criterion of Good HIT

Improved staff productivity, but •Different specialties operate in different ways •The same specialty operates in different ways across departments – e.g GynaeOnc •Specialty teams collect 90-95% of the same data •The differences between departments of the same discipline are their PROCESSES, that is, HOW they collect that data. •What is done at the bottom of the health tree determines a great deal about what can be done at the top. •Failure to support processes at the bottom will lead to a failed supply of data to the top.

Benefit Criterion of Good HIT

Improved patient outcomes

•Patient satisfaction can be assessed more readily.

•Changes in patient outcomes due to HIT are undetectable against the backdrop of all the variables in clinical practice.

•Unless very specific issues are addressed – e.g. antibiotic stewardship at Northern Sydney LHD – 20% reduction in antibiotics over 2 years

– This is not even a direct measurement

Benefit Criterion of Good HIT

Audit/Administration/Research Analytics

•All are mostly aggregation functions of the clinical data - in an IT sense.

•Optimal data collection at the clinical coalface is the biggest single determinant of what you can understand about the system when looking from the top. Hence administration wants data.

•However, better provision of statistics serves management mostly. Clinicians don’t see the point of helping to collect them.

Definition of ROI for Clinical Redesign

• The ROI is defined as the time taken to create the redesigned system versus the duration of time the department has to operate to recover that time, because the staff are saving time using the redesigned system.

• E.g. if the time spent in the cost in redesign is 1000 hours and the redesign saves 10 hours per day for staff for a typical load then they recover the time spent on redesign in 100 days of operation of the department.

• If the time saving is 30 minutes then the duration to recover that time is 2000 days or 5.5 years.

iCIMS Methodology for determining ROI

• Identify current practices reliably, for roles and processes. • Measurement 1 - Measure the time to perform a statistically

sufficient range of the practices to reliably reproduce typical patient journey profiles

• This output defines the baseline model • Identify the changes clinicians consider will improve their work • Require them to establish why they think the redesign will improve

their work by the criteria: – Greater staff productivity – Efficiency & Feedback – Greater patient safety – Correct & Appropriate – Better clinical audit/analytics – data origins & transformations

• This output provides the predictive model and can be used to determine if the clinical judgment is correct.

iCIMS Methodology for determining ROI (contd)

• Use the clinical team to redesign the current practices with an iCIMS modeling tool

• Review 1 - A range of departmental staff bench test the redesigned practices

• Revision 1 - Revise the model based on their feedback

• Measurement 2 – repeat all the the tasks in Measurement 1 in an in vivo setting

• Compile the resulting comparative measures between M1 and M2

iCIMS Methodology for determining ROI (contd)

• Review 2 - Assess the differences between the models - review the redesigned model

• Revision 2 - Revise the model • Measurement 3 - remeasure the model for those

revisions • This stage determines the fully redesigned model • Assess quantitatively by statistical testing the

comparative times between the two models • Arrive at a final assessment, costing the effort to

change the system against the benefits and disadvantages introduced in the model, both quantitatively and qualitatively

Experiments in Determining ROI

• Two Case Studies

• Nepean Hospital ED – NEDIMS vs FirstNet

• Royal Womens Hospital Mel – GynaeOnc & Breast cancer

Features of iCIMS technology

• Design is completed using a design tool

• Design is automatically compiled into a run-time system – No programming

• Real-time modification

• Multiple systems run in the one software installation

NEDIMS Clinical Redesign Case Study

• Can we build a better ED system to make staff more productive and more easily follow the patient journey

• Can staff be effectively engaged in system design • Can the staff design be turned into a working

system • How do we assess the cost of creating such a

system • How do we assess the cost of changing the

system

NEDIMS Outcomes

• NEDIMS minimum of 40% better for time and clicks vs FirstNet

• Initial modeling required 500 person hours

• Subsequent Redesign time costs recovered in less than 6 days of clinicians time

• Numerous workarounds eliminated

• A number of paper processes incorporated into the NEDIMS hence improving auditability and analytics

Calculating ROI on NEDIMS

• 1140 hours to build and test model • Staff productivity gain minimally 21.74 hours per

day (for an av. 150 pts per day) • 52.4 Maximum department days to recover build

and test time • Plus 10 hours to modify model - 6 staff days to

recover this time • If the design of the base system is amortised

over say 5 sites the the recovery occurs in 10.5 department days, plus the cost of modifications for local processes.

GynaeOnc & Breast Case Study – Joint Design Processes

• Replace decaying and moribund systems built on the same underlying technology

• Aim was to replicate existing processes • Add some innovations and extensions • But extend only to the level of acceptable usage by each clinical

team, and • Emphasized the different levels of functionality with GynaeOnc

wanting a more extensive solution than Breast • Significant value of sharing of design ideas across the disciplines • Dynamic adaptation of design as the project progressed • The speed of modeling was: 2 clinical and 3 research systems in 9

months

What do clinicians need?

• A mechanism for continuous clinical redesign improvement (CCRI)

• CCRI requires a number of things – A baseline measurement of process efficiency

– a means of testing changes to processes

– a means of assessing the value of the changes

– a low cost method of enabling change to their IT

– a lead team that others can aspire to and learn from - co-operation with rivalry – e.g. Clinical Quality Audit Registries

Determinants of ROI - Local efficiency feeds global analytics

• All departments of the one specialty collect the 90-95% the same data

• The differences are in their processes not their data • All gains will come from improving their processes • If clinicians are given the tools to improve their

processes they will embrace them enthusiastically • If you give the clinical teams the means of process

improvement they will collect the information needed for global assessment and planning.

• Each department has its own level of computerisation and needs to identify its own next level.

Key Elements of System Process (Re-)Design

Hence systems should be designed to supply "not-quite-enough" functionality, and then,

be "readily-expandable" (this implies cheaply expandable)

when the clinicians are confident they know what computerisation will be of the "next-best-benefit ”.

A procedure for establishing ROI as de rigour for clinical redesign projects

• Identify leaders in the field

• Perform a process analysis and clinical redesign assessment on the leaders delivering an ROI evaluation

• Move to support the specialties that give the highest ROI first.

• Watch the others jump into the queue when they see the effects.

END