evaluation of innovative health it applications ...€¦ · evaluation of innovative health it...

HFMI 2009 – Sonoma, CA, USA, November 12-13 2009

Evaluation of innovative health IT applications:

Importance of usability studies in hospital settings

Marie-Catherine Beuscart-Zéphir, Régis Beuscart

Inserm CIC-IT - Evalab, University Hospital – Lille, France

CIC-IT 807

Biocapteurs et e-santé


Part One

Why evaluate?


Why evaluate ?

• Scientific Objectives

• Clinical Objectives

• Economic Objectives

• At the end: commercialization, payment,

reimbursement …

Innovative

device

CE Marking

Innovative

device

CE Marking

Notified Body

Innovative

device

CE Marking

Can be sold to hospitals

HC organisations

patients


For CE Marking: follow the rules

• The Company:• Is responsible for the conception, manufacturing, and labeling of a

product, for the norms and standards.

• Notified Body • controls the company, verifies the dossier, and realizes the

necessary controls

• Competent National Agency• Controls the notified bodies,

• Can perform complementary controls

• Disseminate the results

• CE Mark allows for commercialization, but

not for reimbursement


More Evaluations for payment or

reimbursement

• Pre-clinical studies:• In silico modeling, In vitro studies

• In vivo trials on small or large animal

• Usability Studies• How the device (procedure) can be used ? Integrated ?

• Clinical Trials (if possible blind and

comparative clinical trials)

Innovative

device

CE Marking

File for

device

reimbursement

Payment through Reimbursement

Innovative

device

CE Marking

File for

device

reimbursement

Certification

by

the National

Agency

Reimbursement

of the device


Innovative

device

CE Marking

File for

device

reimbursement

Certification

by

the National

Agency

File of

evaluation of

the device

Reimbursement

of the device

Payment through DRG


Innovative

device

CE Marking

File for

device

reimbursement

Certification

by

the National

Agency

File of

evaluation of

the device

Inclusion of

the device

In a

DRG

DRG

Payment

Reimbursement

of the device

Payment through DRG



The payment or the reimbursement of

devices is a long and complicated process

• 2-5 years

• Needing different levels of evaluation

• Clinical validation by good clinical trials

becomes mandatory

• Difficult for SMEs, start-ups, even

international companies.


In the ICT domain

• Current examples:

– Telemedicine

– Tele-radiology

– Tele-cardiology

– Communicating pace-makers and defibrillators

– Computer-added surgery


CIC-IT: Technology Innovation Centers

Improve transfer of biomedical technologies from

labs to industrial companies

Promote the cooperation between academic

hospitals and industrial companies for innovation

and evaluation.

Provide evaluation methodologies and

participate to the validation of innovative

devices and procedures

Contribute to translational research, pre-

clinical and clinical studies.

Contribute to the training of students and

physicians in the domain of Healthcare

Technologies


CIC-IT: Technology Innovation Centers

“ Provide evaluation methodologies and

participate to the validation of innovative

devices and procedures “

Objectives:

Improve the methodology and the quality of:

• Technical validation and pre-clinical

studies -> CE Marking

• Clinical trials : payment /

reimbursement of innovative

technologies in healthcare

•Certification of procedures or ICT

Applications

City specialization

Besançon Micro-nano-

technologies

Bordeaux Biomaterials

Grenoble Assisted Surgery

Lille Biosensors and

e-Health

Nancy NMR Imaging

Paris Devices for

handicapped

Rennes Cardiologic

devices

Tours Ultrasonography

8 CIC-IT

Lille

Rennes

Nantes Tours

Nancy

Lyon

MontpellierMarseille

Bordeaux

Toulouse

Grenoble

Paris -

Garches

Dijon

Brest

Besançon

In academic hospitals

Scientific - Economic

• Biosensors for

monitoring, telemedicine,

e-health

• Technological validation

and clinical evaluation

Innovation Usage

Valuation

• Usability and acceptance of ICT by

Healthcare ¨Professionals and Patients

• Take into account users constraints in

the design of the application


• Human Factors for the optimization of healthcare work systems and usability of healthcare IT applications

• Academic hospital of Lille (3000 Beds) and University / Faculty of Medicine: research and consultancy projects

• Accredited by the Ministry of Health and the INSERM (national agency) for evaluation and optimization of work systems and IT applications in healthcare

http://www.univ-lille2.fr/evalab

The Evalab


Part two

Illustration

The Metrodoloris case study


The project

• Research: a new index to support the monitoring of the patient’s pain during anesthesia; a new monitor to visualize the index

• Clinical studies: positive results

• European Patent, gold medal of the 2009 Innovation Awards from the French Ministry of Research

• Creation of SME, CE marking as next objective

• Evaluation questions:– Demonstrate the benefits for the patients (clinical trials)

– Usability and acceptance? Usability file for CE marking


Société en

création

ANALGESIA MONITORINGGold medal of the Ministry of Research Innovation Awards 2009


The problem

• General anesthesia (GA):– hypnotic drugs (e.g.. Propofol)

– analgesic drugs (e.g. opioid)

• Reliable index supporting the monitoring of the depth of hypnosis (BIS index)

• But only few and unreliable ways to assess the analgesia/nociception balance in an unconscious patient– hemodynamic reactivity (increase of heart rate or systolic blood

pressure, sweat, pupils diameter change)

• Objective of the project = establish a reliable index supporting the analgesia / nociception monitoring


Analgesia / Nociception monitoring system

• General principle:

– Analgesia / Nociception monitoring system

based on instantaneous heart rate

variability analysis.

– Analgesia / Nociception Index (ANI) related to

the effect of pain on the autonomous nervous

system.


EKG

acquisition

Heart rate

computation

ANI

Computation

Visualization

interface


The monitoring system

Prototype for developing

and tuning the index (ANI)

1st prototype of the monitoring

system

Visualization of the

index


Benefits for the patients?


Basic principle

• Patients recover faster when anesthesia is

appropriate

– Avoid excessive doses of analgesic drugs

– Avoid pain during unconscious period


Pre-clinical trials

• Study the evolution of the index according to:

– The time of occurrence of painful stimulations

– The level of anesthesia:

• Deep Analgesia (DA) = appropriate

• Light Analgesia (LA) = insufficient

• assessed through hemodynamic reactivity, defined as an

increase of heart rate or systolic blood pressure of more than

30% from the baseline

– Anesthetists actions to correct the anesthesia level

(boluses of opioid)


Video


The pre-clinical trial

• N = 39 patients undergoing surgery with General

Anesthesia

• Anesthesia protocol:– Propofol using a DiprifusorTM (TCI-Diprifusor™, Zeneca Ltd,)

– Propofol dose is continuously adapted (target [40-60]) to the measured

Bispectral Index (BISTM, Aspect A-2000 XPTM v3.31)

– Opioid (Remifentanyl) administered at induction with Propofol; additional

boluses in case of hemodynamic reactivity

• 2 periods analyzed :

– DA deep analgesia = adequate

– LA Light Analgesia = insufficient


Preliminary results

•** p<0.01 : effect DA vs LA

(anova)

ANI index

ANI decreases significantly in case of painful stimulation on anesthetized

patients

10mn before visible patient’s reactions provoking

anesthetist’s actions (boluses)


Benefits for the patient?

• Promising results of pre-clinical trial

• Clinical trials in progress

– orthopedic surgery, pediatric surgery, neurosurgery,

obstetrics

• On this basis: patent, innovation award, creation

of an SME, CE marking file.

• Impact on outcomes such as “rapidity of

recovering” to be evaluated in phase B trials


Users, usage, usability and

acceptance


Usability evaluation

• Analysis of the work system:

– Existing knowledge (Anesthesia Clinical

Information System)

– Completed by focused observations

• Usability inspection of the prototype

• On-site usability tests (portable labs)?

• Focus groups?


Usability inspection

• In-lab evaluation (system installed in lab, stand-

alone, fake database, simulations)

• Ergonomics criteria (INRIA)


Ergonomic criteria (INRIA)

1. Guidancea. Prompting

b. Grouping/Distinction of Items

c. Immediate Feedback

d. Legibility

2. Workloada. Brevity

b. Information Density

3. Explicit Controla. Explicit User Action

b. User Control

4. Adaptabilitya. Flexibility

b. User Experience

5. Error Managementa. Error Protection

b. Quality of Error Messages

c. Error Correction

6. Consistency.

7. Significance of Codes

8. Compatibility


Usability inspection

• 1st iteration on the computer-prototype:

– Intermediary tool:

• Objective = optimize the monitor under development

– Quick and dirty evaluation

• Two ergonomists

• No severity rating

– Frequent debriefings with developers


Results of first usability inspection

• 27 problems

• examples


Legibility


Workload / information density(time scale in seconds / length of anesthesia)


Guidance, prompting (zone for text entering)

Workload (documenting free text during anesthesia)

Guidance (click on “ajouter” to save)


Results of first usability inspection

• Mostly HCI problems

• Question: who is the intended user?

– Researcher / anesthetist in charge of documenting the events in order to validate the index

– Anesthetists during routine anesthesia?

• Problem with the model of work integrated in the system


Second usability inspection: prototype monitor

• Problems with plugging and starting the monitor, initial

erroneous values, discrepancy with values on other

monitors (cardiac frequency)

• HCI problems


Guidance / consistency:

No unit mentioned for the scales, while

the curves are dynamically generated


Error Management / Error CorrectionOnce entered, an event cannot be modified

When several events are entered in a row, they are not visible


Conclusion on usability inspections

• Continuous improvement of the HCI,

evolution of the product


First prototype realized for index development and pre-clinical trial

1st prototype of the monitoring system

EKG Sensor

2nd prototype of the monitoring system


Other usage related problems

• Ambiguity of the model of work integrated in the

product:

– Monitoring the ANI in routine anesthesia

– Continuous validation of the index, extended clinical

trials

• The work environment of the anesthetists is

already overcrowded with monitors of all kinds:

– Duplicate information (e.g. cardiac frequency, EKG)

• Implementation of an anesthesia CIS integrating

the information form various monitors


Acceptance problems

• Not all anesthetists are convinced it is necessary to monitor the analgesia level– Senior anesthetists ≠ junior anesthetists

• Additional workload– EKG duplicate, specific EKG sensors required

• Suggestions: – retrieve the EKG signal from existing monitor when available

• Technically possible

• Economically unfeasible

– Integrate the index in the CIS

• Limited space

• To be discussed with a board of anesthetists

• Further research required on HF questions


Usability file for CE marking?

• Collateral standard on usability for IEC 60601-1-6 for medical electrical equipment

• Related to:– ISO 9241(10-17) (ergonomics requirements for office work with

visual display terminals)

– ISO 13407 (Human-centered design process for interactive systems)

– ISO 18529 (ergonomics of human-system interactions, human-Centered lifecycle process descriptions)

– CIF (Common Industry Format) for reports

• In the present case: mostly usability evaluation of the final product (usability inspection and tests)

• HCI OK, but fundamental problems remain: to which extent do we reflect those problems in the report?

Innovative

device

CE Marking

File for

device

reimbursement

Certification

by

the National

Agency

File of

evaluation of

the device

Inclusion of

the device

In a

DRG

DRG

Payment

Reimbursement

of the device

Payment through DRG


Can be sold to

Hospitals

HC organisations


Conclusion

• Next challenge: integration of the usability

dimension in the certification process of e-

prescribing applications incorporating CDS

functions in hospital setting

• International cooperation in defining the

protocol?

evaluation of innovative health it applications ...€¦ · evaluation of innovative health it...

Documents