redesign of the columbia university infobutton manager james j. cimino, beth e. friedmann, kevin m....
Post on 19-Dec-2015
216 views
TRANSCRIPT
Redesign of the Columbia University Infobutton Manager
James J. Cimino, Beth E. Friedmann, Kevin M. Jackson, Jianhua Li, Jenia Pevzner, Jesse Wrenn
Department of Biomedical InformaticsColumbia University
College of Physicians and Surgeons
Infobutton Manager Redesign
• What is an infobutton manager?
• Why does it need redesign (metrics)?
• What methods were used?
• What happened?
Overview
• Infobuttons – context specific links from one system (clinical) to another (resource)
Overview
• Infobutton Manager (IM) – provides a set of context-specific links
• Infobuttons – context-specific links from one system (clinical) to another (resource)
Overview
• Utilization statistics from log files– Usage – how often is it used– “Selection rate” – when used, how often is topic selected– “Perusal time” – how long does it take to select a topic
• Gold standard: Health Resources (HR) page
• Infobuttons – context-specific links from one system (clinical) to another (resource)
• Infobutton Manager (IM) – provides a set of context-specific links
Overview
• Utilization statistics from log files– Usage – how often is it used– “Selection rate” – when used, how often is topic selected– “Perusal time” – how long does it take to select a topic
• Gold standard: Health Resources (HR) page
• Infobuttons – context-specific links from one system (clinical) to another (resource)
• Infobutton Manager (IM) – provides a set of context-specific links
0
1000
2000
3000
4000
5000
6000
7000
8000
HR
IM
Usage through October 2006
Selection Rate through October 2006
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
HR
IM
Perusal Time through October 2006
0
5
10
15
20
25
HR
IM
Methods
• Navigation study of IM version 1
• Redesign of IM architecture
• Redesign of user interface
• Navigation study of IM version 2
• Log file analysis of actual use
Navigation Study of IM Version 1
• Seven typical clinical questions from NLM– One diagnosis query– Two lab queries– Three in-patient medication queries– Three out-patient medication queries– “Does cranberry juice alter the action of warfarin?”
• Subjects shown sample clinical system• Evoke IM version 1• Attempt to answer the questions• Usability analysis software to record sessions (Morae)
Redesign of IM Architecture
• Version 1: CGI returned set of links (HTML)
• Version 2:
– Decouple user interface design from back end
– Support platform-specific user interfaces
Redesign of User Interface• Workflow (clicks): HR=many, IM=2• Aesthetics: similar font, color, graphics, layout• Efficiency:
– Scanability– Concise language– Keyword menus and buttons– Design consistency– Memorability (easy of reestablishing mastery)
• IM version 1:– Many concepts– Questions vary with concept and context– Questions are lengthy– Scrolling often required
Navigation Study of IM Version 2
• Repeated method used to study Version 1
Log File Analysis of Actual Use
• Logfile provides record of clicks on:– HR page– Links in HR page– Infobutton icons– Links in IM
• Also provides immediate prior action
• Time stamp, user, IP address
Results
• Navigation study of IM version 1– Convenience sample of six clinicians– Long persual times– Review of sessions confirmed wordiness and
layout inconsistency were factors
• Redesign of IM Architecture: AJAX
• Redesign of User Interface
Results
• Navigation study of IM version 1– Convenience sample of six clinicians– Long persual times– Review of sessions confirmed wordiness and
layout inconsistency were factors
• Redesign of IM Architecture: AJAX• Redesign of User Interface• Redesign of User Interface: scanability,
conciseness, keywords, consistency
• Navigation study of IM version 2
0
2
4
6
8
10
12
14
1 6 11 16 21 26 31 36 41 46
IM-2 IM-1Time in Seconds
Count
Perusal Times in Navigation Studies
Results
• Navigation study of IM version 1– Convenience sample of six clinicians– Long persual times– Review of sessions confirmed wordiness and
layout inconsistency were factors
• Redesign of IM Architecture: AJAX• Redesign of User Interface: scanability,
conciseness, keywords, consistency
• Navigation study of IM version 2• Log file analysis of actual use
0
1000
2000
3000
4000
5000
6000
7000
8000
HR
IM
Usage through October 2006
0
1000
2000
3000
4000
5000
6000
7000
8000
HR
IM
Usage through October 2007
Selection Rate through October 2006
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
HR
IM
Selection Rate through October 2007
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
HR
IM
Perusal Time through October 2006
0
5
10
15
20
25
HR
IM
0
5
10
15
20
25
HR
IM
Perusal Time through October 2007
Average IM and HR Monthly Statistics
Time PeriodNumber of Uses (SD)
Selection Rate (SD)
Perusal Time (SD)
IM-1 1/06-10/06 1937 (250) 46.3 (2.5) 14.8 (1.4)
IM-2 11/06-2/07 1789 (154) 46.9 (2.9) 14.5 (1.5)
HR 1/06-10/06 5887 (268) 88.4 (0.9) 6.0 (0.5)
HR 11/06-2/07 6272 (535) 88.6 (0.5) 5.7 (0.4)
Usage in Lab Contexts
0
500
1000
1500
2000
2500
HR-LabDetail
IM-LabDetail
Usage in Out-Patient Drug Contexts
0
50
100
150
200
250
300
350
400
HR-OutPatientDrugs
IM-OutPatientDrugs
Usage in In-Patient Drug Contexts
0
100
200
300
400
500
600
700
Aug
-02
Nov
-02
Feb
-03
May
-03
Aug
-03
Nov
-03
Feb
-04
May
-04
Aug
-04
Nov
-04
Feb
-05
May
-05
Aug
-05
Nov
-05
Feb
-06
May
-06
Aug
-06
Nov
-06
Feb
-07
May
-07
Aug
-07
HR-InPatientDrugs
IM-InPatientDrugs
Usage in Diagnosis Context
0
100
200
300
400
500
600
700
800
900
Aug
-02
Nov
-02
Feb
-03
May
-03
Aug
-03
Nov
-03
Feb
-04
May
-04
Aug
-04
Nov
-04
Feb
-05
May
-05
Aug
-05
Nov
-05
Feb
-06
May
-06
Aug
-06
Nov
-06
Feb
-07
May
-07
Aug
-07
HR-Diagnoses
IM-Diagnoses
Usage in Lab Order Entry Context
0
20
40
60
80
100
120
140
HR-LabOrder
IM-LabOrder
Usage in InPat Drug Order Entry
0
50
100
150
200
250
300
350
Aug
-02
Nov
-02
Feb
-03
May
-03
Aug
-03
Nov
-03
Feb
-04
May
-04
Aug
-04
Nov
-04
Feb
-05
May
-05
Aug
-05
Nov
-05
Feb
-06
May
-06
Aug
-06
Nov
-06
Feb
-07
May
-07
Aug
-07
HR-DrugOrder
IM-DrugOrder
Selection Rate for OutPat Drug Context
0
0.2
0.4
0.6
0.8
1
1.2
HR-OutPatientDrugs
IM-OutPatientDrugs
Selection Rate for InPat Drug Contexts
0
0.2
0.4
0.6
0.8
1
1.2
Aug
-02
Nov
-02
Feb
-03
May
-03
Aug
-03
Nov
-03
Feb
-04
May
-04
Aug
-04
Nov
-04
Feb
-05
May
-05
Aug
-05
Nov
-05
Feb
-06
May
-06
Aug
-06
Nov
-06
Feb
-07
May
-07
Aug
-07
HR-InPatientDrugs
IM-InPatientDrugs
Perusal Time for InPat Drug Contexts
0
5
10
15
20
25
Aug
-02
Nov
-02
Feb
-03
May
-03
Aug
-03
Nov
-03
Feb
-04
May
-04
Aug
-04
Nov
-04
Feb
-05
May
-05
Aug
-05
Nov
-05
Feb
-06
May
-06
Aug
-06
Nov
-06
Feb
-07
May
-07
Aug
-07
HR-InPatientDrugs
IM-InPatientDrugs
What Happened?• IM version 1 had a lower success rate and
higher perusal time than the HR page• This was confirmed in the laboratory• Inspection of the user interface showed
several possible problems• IM version 2 appears, on inspection, to
address the observed problems• This was also confirmed in the laboratory• The field experience did not confirm the
laboratory experience
Why Did This Happen?
• Execution time adds a few seconds• The HR page is static, while the IM is, by
definition, dynamic• IM uses a different paradigm (pull-push)• Results varied by context, but not user type
or level of experience• Actual users are different, however
Users of HR vs. Users of IM
HR3: 1408 IM3: 715
HR<3, IM<3 1160
248
Total Users 1/07-10/07: 2933
Now What?
• Educational e-mails• New user interface may improve question-
answering time• Observational studies may be the only way to
identify the rate-limiting step• IM approach remains more efficient• Users prefer IM over HR in some situations• Version 3 (thanks, AJAX!)
Conclusions
• Good user interface design principles work
• Laboratory experience may not reflect reality
• Users do invest extra effort
• Selection rate and perusal time are not the only metrics of success
Acknowledgments
This work was supported in part by National Library of Medicine Grant R01LM07593.
The authors thank the test subjects for their time and effort.
The authors also thank Dr. Rick Gallagher for providing the log files.