redesign of the columbia university infobutton manager james j. cimino, beth e. friedmann, kevin m....

51
Redesign of the Columbia University Infobutton Manager James J. Cimino, Beth E. Friedmann, Kevin M. Jackson, Jianhua Li, Jenia Pevzner, Jesse Wrenn Department of Biomedical Informatics Columbia University College of Physicians and Surgeons

Post on 19-Dec-2015

216 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Redesign of the Columbia University Infobutton Manager James J. Cimino, Beth E. Friedmann, Kevin M. Jackson, Jianhua Li, Jenia Pevzner, Jesse Wrenn Department

Redesign of the Columbia University Infobutton Manager

James J. Cimino, Beth E. Friedmann, Kevin M. Jackson, Jianhua Li, Jenia Pevzner, Jesse Wrenn

Department of Biomedical InformaticsColumbia University

College of Physicians and Surgeons

Page 2: Redesign of the Columbia University Infobutton Manager James J. Cimino, Beth E. Friedmann, Kevin M. Jackson, Jianhua Li, Jenia Pevzner, Jesse Wrenn Department

Infobutton Manager Redesign

• What is an infobutton manager?

• Why does it need redesign (metrics)?

• What methods were used?

• What happened?

Page 3: Redesign of the Columbia University Infobutton Manager James J. Cimino, Beth E. Friedmann, Kevin M. Jackson, Jianhua Li, Jenia Pevzner, Jesse Wrenn Department

Overview

• Infobuttons – context specific links from one system (clinical) to another (resource)

Page 4: Redesign of the Columbia University Infobutton Manager James J. Cimino, Beth E. Friedmann, Kevin M. Jackson, Jianhua Li, Jenia Pevzner, Jesse Wrenn Department
Page 5: Redesign of the Columbia University Infobutton Manager James J. Cimino, Beth E. Friedmann, Kevin M. Jackson, Jianhua Li, Jenia Pevzner, Jesse Wrenn Department
Page 6: Redesign of the Columbia University Infobutton Manager James J. Cimino, Beth E. Friedmann, Kevin M. Jackson, Jianhua Li, Jenia Pevzner, Jesse Wrenn Department
Page 7: Redesign of the Columbia University Infobutton Manager James J. Cimino, Beth E. Friedmann, Kevin M. Jackson, Jianhua Li, Jenia Pevzner, Jesse Wrenn Department

Overview

• Infobutton Manager (IM) – provides a set of context-specific links

• Infobuttons – context-specific links from one system (clinical) to another (resource)

Page 8: Redesign of the Columbia University Infobutton Manager James J. Cimino, Beth E. Friedmann, Kevin M. Jackson, Jianhua Li, Jenia Pevzner, Jesse Wrenn Department
Page 9: Redesign of the Columbia University Infobutton Manager James J. Cimino, Beth E. Friedmann, Kevin M. Jackson, Jianhua Li, Jenia Pevzner, Jesse Wrenn Department
Page 10: Redesign of the Columbia University Infobutton Manager James J. Cimino, Beth E. Friedmann, Kevin M. Jackson, Jianhua Li, Jenia Pevzner, Jesse Wrenn Department

Overview

• Utilization statistics from log files– Usage – how often is it used– “Selection rate” – when used, how often is topic selected– “Perusal time” – how long does it take to select a topic

• Gold standard: Health Resources (HR) page

• Infobuttons – context-specific links from one system (clinical) to another (resource)

• Infobutton Manager (IM) – provides a set of context-specific links

Page 11: Redesign of the Columbia University Infobutton Manager James J. Cimino, Beth E. Friedmann, Kevin M. Jackson, Jianhua Li, Jenia Pevzner, Jesse Wrenn Department
Page 12: Redesign of the Columbia University Infobutton Manager James J. Cimino, Beth E. Friedmann, Kevin M. Jackson, Jianhua Li, Jenia Pevzner, Jesse Wrenn Department

Overview

• Utilization statistics from log files– Usage – how often is it used– “Selection rate” – when used, how often is topic selected– “Perusal time” – how long does it take to select a topic

• Gold standard: Health Resources (HR) page

• Infobuttons – context-specific links from one system (clinical) to another (resource)

• Infobutton Manager (IM) – provides a set of context-specific links

Page 13: Redesign of the Columbia University Infobutton Manager James J. Cimino, Beth E. Friedmann, Kevin M. Jackson, Jianhua Li, Jenia Pevzner, Jesse Wrenn Department

0

1000

2000

3000

4000

5000

6000

7000

8000

HR

IM

Usage through October 2006

Page 14: Redesign of the Columbia University Infobutton Manager James J. Cimino, Beth E. Friedmann, Kevin M. Jackson, Jianhua Li, Jenia Pevzner, Jesse Wrenn Department

Selection Rate through October 2006

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

HR

IM

Page 15: Redesign of the Columbia University Infobutton Manager James J. Cimino, Beth E. Friedmann, Kevin M. Jackson, Jianhua Li, Jenia Pevzner, Jesse Wrenn Department

Perusal Time through October 2006

0

5

10

15

20

25

HR

IM

Page 16: Redesign of the Columbia University Infobutton Manager James J. Cimino, Beth E. Friedmann, Kevin M. Jackson, Jianhua Li, Jenia Pevzner, Jesse Wrenn Department

Methods

• Navigation study of IM version 1

• Redesign of IM architecture

• Redesign of user interface

• Navigation study of IM version 2

• Log file analysis of actual use

Page 17: Redesign of the Columbia University Infobutton Manager James J. Cimino, Beth E. Friedmann, Kevin M. Jackson, Jianhua Li, Jenia Pevzner, Jesse Wrenn Department

Navigation Study of IM Version 1

• Seven typical clinical questions from NLM– One diagnosis query– Two lab queries– Three in-patient medication queries– Three out-patient medication queries– “Does cranberry juice alter the action of warfarin?”

• Subjects shown sample clinical system• Evoke IM version 1• Attempt to answer the questions• Usability analysis software to record sessions (Morae)

Page 18: Redesign of the Columbia University Infobutton Manager James J. Cimino, Beth E. Friedmann, Kevin M. Jackson, Jianhua Li, Jenia Pevzner, Jesse Wrenn Department

Redesign of IM Architecture

• Version 1: CGI returned set of links (HTML)

• Version 2:

– Decouple user interface design from back end

– Support platform-specific user interfaces

Page 19: Redesign of the Columbia University Infobutton Manager James J. Cimino, Beth E. Friedmann, Kevin M. Jackson, Jianhua Li, Jenia Pevzner, Jesse Wrenn Department

Redesign of User Interface• Workflow (clicks): HR=many, IM=2• Aesthetics: similar font, color, graphics, layout• Efficiency:

– Scanability– Concise language– Keyword menus and buttons– Design consistency– Memorability (easy of reestablishing mastery)

• IM version 1:– Many concepts– Questions vary with concept and context– Questions are lengthy– Scrolling often required

Page 20: Redesign of the Columbia University Infobutton Manager James J. Cimino, Beth E. Friedmann, Kevin M. Jackson, Jianhua Li, Jenia Pevzner, Jesse Wrenn Department

Navigation Study of IM Version 2

• Repeated method used to study Version 1

Page 21: Redesign of the Columbia University Infobutton Manager James J. Cimino, Beth E. Friedmann, Kevin M. Jackson, Jianhua Li, Jenia Pevzner, Jesse Wrenn Department

Log File Analysis of Actual Use

• Logfile provides record of clicks on:– HR page– Links in HR page– Infobutton icons– Links in IM

• Also provides immediate prior action

• Time stamp, user, IP address

Page 22: Redesign of the Columbia University Infobutton Manager James J. Cimino, Beth E. Friedmann, Kevin M. Jackson, Jianhua Li, Jenia Pevzner, Jesse Wrenn Department

Results

• Navigation study of IM version 1– Convenience sample of six clinicians– Long persual times– Review of sessions confirmed wordiness and

layout inconsistency were factors

• Redesign of IM Architecture: AJAX

• Redesign of User Interface

Page 23: Redesign of the Columbia University Infobutton Manager James J. Cimino, Beth E. Friedmann, Kevin M. Jackson, Jianhua Li, Jenia Pevzner, Jesse Wrenn Department
Page 24: Redesign of the Columbia University Infobutton Manager James J. Cimino, Beth E. Friedmann, Kevin M. Jackson, Jianhua Li, Jenia Pevzner, Jesse Wrenn Department
Page 25: Redesign of the Columbia University Infobutton Manager James J. Cimino, Beth E. Friedmann, Kevin M. Jackson, Jianhua Li, Jenia Pevzner, Jesse Wrenn Department

Results

• Navigation study of IM version 1– Convenience sample of six clinicians– Long persual times– Review of sessions confirmed wordiness and

layout inconsistency were factors

• Redesign of IM Architecture: AJAX• Redesign of User Interface• Redesign of User Interface: scanability,

conciseness, keywords, consistency

• Navigation study of IM version 2

Page 26: Redesign of the Columbia University Infobutton Manager James J. Cimino, Beth E. Friedmann, Kevin M. Jackson, Jianhua Li, Jenia Pevzner, Jesse Wrenn Department

0

2

4

6

8

10

12

14

1 6 11 16 21 26 31 36 41 46

IM-2 IM-1Time in Seconds

Count

Perusal Times in Navigation Studies

Page 27: Redesign of the Columbia University Infobutton Manager James J. Cimino, Beth E. Friedmann, Kevin M. Jackson, Jianhua Li, Jenia Pevzner, Jesse Wrenn Department

Results

• Navigation study of IM version 1– Convenience sample of six clinicians– Long persual times– Review of sessions confirmed wordiness and

layout inconsistency were factors

• Redesign of IM Architecture: AJAX• Redesign of User Interface: scanability,

conciseness, keywords, consistency

• Navigation study of IM version 2• Log file analysis of actual use

Page 28: Redesign of the Columbia University Infobutton Manager James J. Cimino, Beth E. Friedmann, Kevin M. Jackson, Jianhua Li, Jenia Pevzner, Jesse Wrenn Department

0

1000

2000

3000

4000

5000

6000

7000

8000

HR

IM

Usage through October 2006

Page 29: Redesign of the Columbia University Infobutton Manager James J. Cimino, Beth E. Friedmann, Kevin M. Jackson, Jianhua Li, Jenia Pevzner, Jesse Wrenn Department

0

1000

2000

3000

4000

5000

6000

7000

8000

HR

IM

Usage through October 2007

Page 30: Redesign of the Columbia University Infobutton Manager James J. Cimino, Beth E. Friedmann, Kevin M. Jackson, Jianhua Li, Jenia Pevzner, Jesse Wrenn Department

Selection Rate through October 2006

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

HR

IM

Page 31: Redesign of the Columbia University Infobutton Manager James J. Cimino, Beth E. Friedmann, Kevin M. Jackson, Jianhua Li, Jenia Pevzner, Jesse Wrenn Department

Selection Rate through October 2007

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

HR

IM

Page 32: Redesign of the Columbia University Infobutton Manager James J. Cimino, Beth E. Friedmann, Kevin M. Jackson, Jianhua Li, Jenia Pevzner, Jesse Wrenn Department

Perusal Time through October 2006

0

5

10

15

20

25

HR

IM

Page 33: Redesign of the Columbia University Infobutton Manager James J. Cimino, Beth E. Friedmann, Kevin M. Jackson, Jianhua Li, Jenia Pevzner, Jesse Wrenn Department

0

5

10

15

20

25

HR

IM

Perusal Time through October 2007

Page 34: Redesign of the Columbia University Infobutton Manager James J. Cimino, Beth E. Friedmann, Kevin M. Jackson, Jianhua Li, Jenia Pevzner, Jesse Wrenn Department

Average IM and HR Monthly Statistics

Time PeriodNumber of Uses (SD)

Selection Rate (SD)

Perusal Time (SD)

IM-1 1/06-10/06 1937 (250) 46.3 (2.5) 14.8 (1.4)

IM-2 11/06-2/07 1789 (154) 46.9 (2.9) 14.5 (1.5)

HR 1/06-10/06 5887 (268) 88.4 (0.9) 6.0 (0.5)

HR 11/06-2/07 6272 (535) 88.6 (0.5) 5.7 (0.4)

Page 35: Redesign of the Columbia University Infobutton Manager James J. Cimino, Beth E. Friedmann, Kevin M. Jackson, Jianhua Li, Jenia Pevzner, Jesse Wrenn Department

Usage in Lab Contexts

0

500

1000

1500

2000

2500

HR-LabDetail

IM-LabDetail

Page 36: Redesign of the Columbia University Infobutton Manager James J. Cimino, Beth E. Friedmann, Kevin M. Jackson, Jianhua Li, Jenia Pevzner, Jesse Wrenn Department

Usage in Out-Patient Drug Contexts

0

50

100

150

200

250

300

350

400

HR-OutPatientDrugs

IM-OutPatientDrugs

Page 37: Redesign of the Columbia University Infobutton Manager James J. Cimino, Beth E. Friedmann, Kevin M. Jackson, Jianhua Li, Jenia Pevzner, Jesse Wrenn Department

Usage in In-Patient Drug Contexts

0

100

200

300

400

500

600

700

Aug

-02

Nov

-02

Feb

-03

May

-03

Aug

-03

Nov

-03

Feb

-04

May

-04

Aug

-04

Nov

-04

Feb

-05

May

-05

Aug

-05

Nov

-05

Feb

-06

May

-06

Aug

-06

Nov

-06

Feb

-07

May

-07

Aug

-07

HR-InPatientDrugs

IM-InPatientDrugs

Page 38: Redesign of the Columbia University Infobutton Manager James J. Cimino, Beth E. Friedmann, Kevin M. Jackson, Jianhua Li, Jenia Pevzner, Jesse Wrenn Department

Usage in Diagnosis Context

0

100

200

300

400

500

600

700

800

900

Aug

-02

Nov

-02

Feb

-03

May

-03

Aug

-03

Nov

-03

Feb

-04

May

-04

Aug

-04

Nov

-04

Feb

-05

May

-05

Aug

-05

Nov

-05

Feb

-06

May

-06

Aug

-06

Nov

-06

Feb

-07

May

-07

Aug

-07

HR-Diagnoses

IM-Diagnoses

Page 39: Redesign of the Columbia University Infobutton Manager James J. Cimino, Beth E. Friedmann, Kevin M. Jackson, Jianhua Li, Jenia Pevzner, Jesse Wrenn Department

Usage in Lab Order Entry Context

0

20

40

60

80

100

120

140

HR-LabOrder

IM-LabOrder

Page 40: Redesign of the Columbia University Infobutton Manager James J. Cimino, Beth E. Friedmann, Kevin M. Jackson, Jianhua Li, Jenia Pevzner, Jesse Wrenn Department

Usage in InPat Drug Order Entry

0

50

100

150

200

250

300

350

Aug

-02

Nov

-02

Feb

-03

May

-03

Aug

-03

Nov

-03

Feb

-04

May

-04

Aug

-04

Nov

-04

Feb

-05

May

-05

Aug

-05

Nov

-05

Feb

-06

May

-06

Aug

-06

Nov

-06

Feb

-07

May

-07

Aug

-07

HR-DrugOrder

IM-DrugOrder

Page 41: Redesign of the Columbia University Infobutton Manager James J. Cimino, Beth E. Friedmann, Kevin M. Jackson, Jianhua Li, Jenia Pevzner, Jesse Wrenn Department

Selection Rate for OutPat Drug Context

0

0.2

0.4

0.6

0.8

1

1.2

HR-OutPatientDrugs

IM-OutPatientDrugs

Page 42: Redesign of the Columbia University Infobutton Manager James J. Cimino, Beth E. Friedmann, Kevin M. Jackson, Jianhua Li, Jenia Pevzner, Jesse Wrenn Department

Selection Rate for InPat Drug Contexts

0

0.2

0.4

0.6

0.8

1

1.2

Aug

-02

Nov

-02

Feb

-03

May

-03

Aug

-03

Nov

-03

Feb

-04

May

-04

Aug

-04

Nov

-04

Feb

-05

May

-05

Aug

-05

Nov

-05

Feb

-06

May

-06

Aug

-06

Nov

-06

Feb

-07

May

-07

Aug

-07

HR-InPatientDrugs

IM-InPatientDrugs

Page 43: Redesign of the Columbia University Infobutton Manager James J. Cimino, Beth E. Friedmann, Kevin M. Jackson, Jianhua Li, Jenia Pevzner, Jesse Wrenn Department

Perusal Time for InPat Drug Contexts

0

5

10

15

20

25

Aug

-02

Nov

-02

Feb

-03

May

-03

Aug

-03

Nov

-03

Feb

-04

May

-04

Aug

-04

Nov

-04

Feb

-05

May

-05

Aug

-05

Nov

-05

Feb

-06

May

-06

Aug

-06

Nov

-06

Feb

-07

May

-07

Aug

-07

HR-InPatientDrugs

IM-InPatientDrugs

Page 44: Redesign of the Columbia University Infobutton Manager James J. Cimino, Beth E. Friedmann, Kevin M. Jackson, Jianhua Li, Jenia Pevzner, Jesse Wrenn Department

What Happened?• IM version 1 had a lower success rate and

higher perusal time than the HR page• This was confirmed in the laboratory• Inspection of the user interface showed

several possible problems• IM version 2 appears, on inspection, to

address the observed problems• This was also confirmed in the laboratory• The field experience did not confirm the

laboratory experience

Page 45: Redesign of the Columbia University Infobutton Manager James J. Cimino, Beth E. Friedmann, Kevin M. Jackson, Jianhua Li, Jenia Pevzner, Jesse Wrenn Department

Why Did This Happen?

• Execution time adds a few seconds• The HR page is static, while the IM is, by

definition, dynamic• IM uses a different paradigm (pull-push)• Results varied by context, but not user type

or level of experience• Actual users are different, however

Page 46: Redesign of the Columbia University Infobutton Manager James J. Cimino, Beth E. Friedmann, Kevin M. Jackson, Jianhua Li, Jenia Pevzner, Jesse Wrenn Department

Users of HR vs. Users of IM

HR3: 1408 IM3: 715

HR<3, IM<3 1160

248

Total Users 1/07-10/07: 2933

Page 47: Redesign of the Columbia University Infobutton Manager James J. Cimino, Beth E. Friedmann, Kevin M. Jackson, Jianhua Li, Jenia Pevzner, Jesse Wrenn Department

Now What?

• Educational e-mails• New user interface may improve question-

answering time• Observational studies may be the only way to

identify the rate-limiting step• IM approach remains more efficient• Users prefer IM over HR in some situations• Version 3 (thanks, AJAX!)

Page 48: Redesign of the Columbia University Infobutton Manager James J. Cimino, Beth E. Friedmann, Kevin M. Jackson, Jianhua Li, Jenia Pevzner, Jesse Wrenn Department
Page 49: Redesign of the Columbia University Infobutton Manager James J. Cimino, Beth E. Friedmann, Kevin M. Jackson, Jianhua Li, Jenia Pevzner, Jesse Wrenn Department
Page 50: Redesign of the Columbia University Infobutton Manager James J. Cimino, Beth E. Friedmann, Kevin M. Jackson, Jianhua Li, Jenia Pevzner, Jesse Wrenn Department

Conclusions

• Good user interface design principles work

• Laboratory experience may not reflect reality

• Users do invest extra effort

• Selection rate and perusal time are not the only metrics of success

Page 51: Redesign of the Columbia University Infobutton Manager James J. Cimino, Beth E. Friedmann, Kevin M. Jackson, Jianhua Li, Jenia Pevzner, Jesse Wrenn Department

Acknowledgments

This work was supported in part by National Library of Medicine Grant R01LM07593.

The authors thank the test subjects for their time and effort.

The authors also thank Dr. Rick Gallagher for providing the log files.