Lu Xu. A Visual Analysis of EHR Flowsheet to Assist Clinicians’ Interpretation of Health-related Data. A Master’s Paper for the M.S. in I.S degree. April, 2019. 32 pages. Advisor: David Gotz

This project designed and implemented visualization interface of EHR Systems based on the requirements of the doctors. The visualizations combined both clinic-reported data and patient self-reported data to provide a better representation of patient health related information for the doctor to make decisions. The visualizations highlighted the trend of values, the outliers and make it possible to compare across time and measures. The user study of ten participants suggests that the visualization interface helped them find the information in an efficient way.

Headings:

Visual Analysis

EHR

Flowsheet

Health-related Data

Patient Self-reported Data

brought to you by COREView metadata, citation and similar papers at core.ac.uk

provided by Carolina Digital Repository

A VISUAL ANALYSIS OF EHR FLOWSHEET TO ASSIST CLINICIANS’ INTEPRETATION OF HEALTH-RELATED DATA

by Lu Xu

A Master’s paper submitted to the faculty of the School of Information and Library Science of the University of North Carolina at Chapel Hill

in partial fulfillment of the requirements for the degree of Master of Science in

Information Science.

Chapel Hill, North Carolina

April 2019

Approved by

_______________________________________

David Gotz

1

Table of Contents

INTRODUCTION ................................................................................................................... 3

LITERATURE REVIEW............................................................................................................ 6

1.1 Doctor Interaction with EHR Systems ........................................................................ 6

1.2 Data Visualization ...................................................................................................... 7

1.3 Data Analysis ............................................................................................................. 7

1.4 Brief Summary ........................................................................................................... 8

METHODS.......................................................................................................................... 9

1. Data Collection............................................................................................................... 9

2. Research Design ........................................................................................................... 11

2.1 Data Analysis and Visualization ............................................................................... 11

2.2 Evaluation ................................................................................................................ 12

2.2.1 Survey ...................................................................................................................... 12

2.2.2 Semi – structured Interview..................................................................................... 13

3. Limitations ................................................................................................................... 14

SYSTEM DESIGN................................................................................................................ 15

1. User Requirement ........................................................................................................ 15

2. System Functions ......................................................................................................... 15

2.1 Measure Types ........................................................................................................ 16

2.2 Measure Display ...................................................................................................... 16

2.3 Self-reported Measures ........................................................................................... 17

USER STUDY AND ANALYSIS ................................................................................................ 19

1. Participants .................................................................................................................. 19

2. Analysis of Survey Results ............................................................................................ 20

3. Limitations and Future Work ....................................................................................... 21

CONCLUSION .................................................................................................................... 22

2

REFERENCE ...................................................................................................................... 24

APPENDIX ........................................................................................................................ 28

3

INTRODUCTION

Diabetes Mellitus is a chronic disease. Management for Diabetes is concentrated in

keeping blood glucose level close to normal. For this purpose, monitoring and control

blood glucose, weight, pulse and other vital signs to a health level are important.

The current Electronic Health Records (EHR) systems that UNC Hospital are using to

record patients’ vital signs and other measures includes two dimensional tables of

variables and value pairs. The patient’s measure values of each visit to the clinic/hospitals

are recorded in the EHR systems, and the patient self-generated health data are not

collected in the systems. The medicine records and measures are in different tables and

are not well connected with each other, so it is almost impossible for clinicians to check

the medicine records and the corresponding measures after each medicine record during

the patient visit time (around 15 min).

In this project, I am interested in data visual analysis of EHR Systems for diabetes

patients. Two types of EHR are visualized and analyzed: the clinical records and patients

self-generated data. Patient self-reported data is generally more frequent than clinical-

reported data. Patients can track their blood pressure, temperature, pulse, weight and

other measures of their body’s basic functions once or multiple times a day by a simple

wearable or fixed equipment. Clinical data, however, can only be measured/recorded

4

when patients visit clinics. If the equipment can upload data automatically and/ or user

can also type in the values, the information provided by both clinical visit data and

patient-self reported data can be much more informative for doctors to understand the

patient’s status.

The client group of the proposed visualization interface is doctors with M.D. degree, who

work in emergency as well as non-emergency environment in the hospitals or clinics, and

are using current EHR systems (such as Epic) to record patients’ health related

information.

The research question for this project is how to visualize health related measures of

diabetes patients, combined both patient self-reported and clinical reported information,

to assist with clinicians’ understanding of these data and help them to make decisions.

The following are the questions about patient self-report data and clinic-reported data

need to be figured out before the design of visualizations:

1. What measures are important for analysis and are visualized in the chart for the

clinicians to view?

2. Both patient self-reported data and clinical data have different frequency,

different categories, different ranges and different measures. Can they be

displayed in the same chart?

3. If they can be displayed in the same chart, how is the time axis divided? (use the

time point of the patient visit clinic?)

5

4. Are there some good examples to visualize them?

5. How to make unusual values obvious for clinicians to find out?

After meetings with Dr. Arlene Chung and Dr. David Gotz, we selected the measures that

most important for the diabetes patients. Dr. Chung is a UNC Lineberger Comprehensive

Cancer Center member and an Assistant Professor of Medicine and Pediatrics and

Associate Director of Health and Clinical Informatics for the UNC School of Medicine.

The variables we decied to visualize are Blood Glucose (including Glucose POC

(fingerstick) and Glucose Bld (blood draw)), Vital Signs (Weight, Blood Pressure, Pulse,

Pulse Oxygen), and Medicine. After the design and implement of the visualizations, I

recruited ten persons with M.D. degree or with B.S / B.A degrees to participate in user

experiment on clinicians and other participants to evaluate the visualization interface of

information systems that I designed and implemented, in order to find out whether the

visualizations are able to help doctors understand the patient’s health-related information.

6

LITERATURE REVIEW

There are several fields which have relevant studies, such as Electronic Health Records

(EHR), how to visualize health-related data, doctor interaction with EHR, and data

visualization. Following are the theories and opinions which are helpful and necessary to

look at.

1.1 Doctor Interaction with EHR Systems Montague (2013) mentions that EHR improves patient’s satisfaction. However, the

results show that technology-centered physicians spent the most time inputting and

extracting information, which may limit their time to interact with the technology at high

levels or recall information about the patient. For this situation, find an efficient way to

help improve the interactions between the doctors and EHR Systems are necessary.

Benjamin et al. (2018) mentions that EHRs are often perceived by users as difficult to

use. There are issues with difficult-to-read interfaces, confusing displays and iconography

that lacks consistency and intuitive meaning. These usability issues can often lead to

heavy cognitive load.

The difficulties that EHR currently have are hard-to-read interfaces, confusing displays

that lacks of intuitive meaning, and requiring more time to extract the data required.

7

Correspondingly, the information needs of the clients are simple, efficient access of the

patient health related information to help them to have a better understanding of the

patients and distinguish some outliers and abnormal patterns of the patients’ vital signs.

1.2 Data Visualization Data visualization is a very broad concept, and according to the definition by Azzam et

al. (2013), it is a process which is based on qualitative and quantitative data and results in

an image that represents the raw data, and the image is readable by viewers and supports

exploration, examination and communication of the data, while the theory of this

definition is originally from Kosara (2007).

While the technique of data visualization gets much developed and advanced, there are

still concerns and challenges of this field. Kovalerchuk (2017) studies the lossless

visualization of n-D data and considers it better than 2-D in perceptual and cognitive

abilities for discovering visual pattern, but human are limited in their ability to discover

patterns in n-D data using a naked eye, which therefore motivates the development of 2-

D visual representations of n-D data.

1.3 Evaluation of Visualization The evaluation of information visualization is necessary because the visualization

interface is designed and developed to meet the users’ requirements.

8

Carpendale (2008) mentioned that usability questions, perceptual and comprehensibility

questions are important in assessing the appropriateness of a representational encoding

and the readability of visuals.

Studies on data visualization and visual analytics in health informatics applications

commonly use task-based measurements, user feedback, and surveys with both

qualitative responses and scoring scales in a lab setting (Gotz et al, 2019). The PIOS

framework (Participants, Interventions, Comparators, Outcomes, and Study Design)

proposed by Gotz et al. also provide a framework for this project.

The evaluation methods that I applied are task-based questions, surveys about how user

feel about the visualization interface and semi-structured interview to effectiveness,

efficiency and satisfaction.

1.4 Brief Summary Current research has already discussed about the difficulties of current EHR Systems,

advantages of 2-D data visualization over n-D data visualization for human, and

evaluation of visualization approaches for health informatics, which is very informative

for the basic context of this project. But there are very few papers for visualizations of

EHR systems, especially how to visualize both clinical data and patient self-reported

data.

9

METHODS

This project includes both quantitative and qualitative methods for design and

constructing the visualization interface and user study for visualization interface

evaluation.

1. User Interview To collect the user requirements for this project, Dr. Chung who is experienced with

diabetes patients are interviewed. The questions answered include what data is needed,

where the data is from, what kinds of insights are needed by a doctor.

2. Data Data used for the visualization is mostly artificial dataset. The real patient clinical data

are not considered because it is very sensitive. The patient-self reported data is not

currently available in EHR systems, and it is the future features adding to the current

EHR systems to help inform doctors about what happens between infrequent doctor

visits, so that they are not available for this study. The examples online and Dr. Chung

who has a good knowledge of the range of the normal values helped me to improve the

artificial data quality and made it more realistic for visualization development and user

evaluation.

10

The periodic clinical data with doctor’s office was also captured in today’s modern

flowsheets, and user-generated data captured with high frequency in between visits.

The normal range of measures that included in the visualizations are as follows:

• Weight

Weight has no upper or lower limits because the weight does not change dramatically.

Thus, there are no boundaries and outliers for weight in the visualizations.

• Pulse

Normal pulse rates at rest, in beats per minute (BPM):

newborn

(0–3 months

old)

infants

(3 – 6

months)

infants

(6 – 12

months)

children

(1 – 10

years)

children over 10

years

& adults,

including seniors

well-

trained

adult

athletes

99-149 89–119 79-119 69–129 59–99 39–59

This project selected 59 and 99 (children over 10 years & adults, including seniors) as

lower and upper limit to find out outliers.

• Blood Pressure

Category systolic, mmHg diastolic, mmHg

Normal 90–119

90–129

60–79

60–84

11

• Pulse Oxygen

Oxygen saturation (SO2): Normal range is 75-100 mmHg. Below 60 mmHg indicates

needs for supplemental oxygen. Values under 90 are considered low.

• A1C

Normal Range are between 5.7 - 6.4 percent.

• Glucose

A fasting blood glucose is in between 100-125 mg/dL. Any value between 140 mg/dL

and 199 mg/dL during a two-hour 75g oral glucose tolerance test. Glucose is the same

in terms of units/scale for both POC and BLD. The only difference would be if

showing the hgA1C which is a percentage with an associated average rage of glucose

with it.

• INR

This measure is weekly to a month if done at home; done after change in medication

usually. INR below 1.1 is normal. Often INR intentionally kept higher (e.g., 2.0-3.0

for people with afib). High INR means blood will clot more slowly.

• Medication

The visualization interface added medication of each doctor visit.

3. Research Design

3.1 Data Analysis and Visualization The main contribution of this project will be a visualization interface to show both the

values of clinic-reported health-related data and patient self-reported data, so that the data

analysis will be mainly about extracting and filtering data by different measures, finding

12

out the abnormal values and trend, then visualize them in the interface. The detailed

description of system design is in the next section.

Other than cleaning the data and finding out the attributes needed for visualization, the

design of visualization/interface is also one of the essential parts, which is followed by

actually working on the code.

What’s more, after building the visualization, some analysis will also be included as

illustrating findings and trends of this visualization, mainly for the purpose of identifying

abnormal values and outliers.

3.2 Evaluation As for constructing a data visualization project, it is necessary to have a final part as

evaluation in order to test the efficiency of using this visualization. For this project I

conducted questionnaire and semi-structured interview to evaluate the design of the

interface and collect people’s opinions and suggestions while using the visualization

interface. The evaluation of visualization will be in the User Study and Analysis section.

3.2.1 Participants Subjects of the evaluation are the target users of the project – clinicians in UNC Hospital.

The size of user study group includes ten subjects. But it is difficult to recruit ten

clinicians or people with M.D. degrees, we decided to recruit at least two participants

with M.D. degree, and the rest should have B.S. or B.A. degrees or even higher education

background. The sampling method is convenience sampling. I recruit participants by

13

sending requests to the people I am able to reach and have background in medicine study

or related fields, then ask them to recommend people they can reach out to me.

3.2.2 Questionnaire

I send out a questionnaire with the visualization webpage link

(https://opal.ils.unc.edu/~luxu/ehr/interface.html) to ask participants about the general

view of the visualization interface and to see how the users feel about it and how efficient

it is when using the visualization results to make decisions. I inserted some signals in the

dataset to see whether the participants can find out them.

There are two parts in the questionnaire. The first half is for all participants, and the

second half is for participants with M.D. degree only. For each part, there are 2 to 4

comparatively easy task-based questions tasks (for example: which value do you think is

abnormal?), followed by survey questions about how participants think about this

visualization.

3.2.3 Semi – structured Interview The questionnaire about users’ understanding of the visualizations is not sufficient for

this user study, thus a semi-structured interview follows up to get more detailed feedback

for the visualizations. According to participant’s answers of questionnaire, I asked

questions such as why did they choose this answer, and what do they think about this

visualization interface and why.

14

4. Limitations To engage more real users, the visualization interface should be tested by the real user

and be tested with complex, large and real dataset.

The first limitation of this project is that it used small-size simulated dataset. There are

potentials that some important signals and patterns of the health-related information are

missing in this dataset, so that we cannot evaluate whether the visualization interface

works for such signals and patterns. My approach to use the normal range of measures to

generate artificial data, then add outliers and trend according to the samples can make the

dataset more realistic, but still it cannot be compared with the real, large-size and

complex dataset.

The second limitation is using convenient sampling method to recruit participants for user

study. The participants for the visualization interface tasks, surveys and user feedback are

mostly made up with the Master Students and the Ph.D. candidates who are real users and

are not fully representative for the entire user group.

15

SYSTEM DESIGN

The visualization of the system interface is finished in JavaScript, HTML, CSS

programming languages. The JavaScript Libraries that applied to the project include D3,

Simple Statistics, jQuery and Bootstrap.

1. User Requirement

a. The visualization interface should be easy to understand and require minimum

efforts to learn how to use and interpret it.

b. The visualizations of this project are clear and simple, without very fancy

animations.

c. The visualization interface should display all the measures mentioned above in

the data collection part.

d. It should be easy to find the outliners and abnormal trend.

2. System Functions The table of the interface has the measures taken from each clinic visit and are displayed

in the column corresponding to the date of visit. In between the two clinical visit columns

are the scatter plots of self-reported data measured between these two doctor visits.

16

Figure 1 Visualization Interface

2.1 Measure Types The measures in the table can be selected by the doctors based on the patient type. For

this visualization project particularly for diabetes patients, the measures selected are

mentioned in the data collection part. Among them are the measures of vital signs such as

weight, Blood Pressure, Pulse and measures, and for the diabetes patients specifically,

Blood Glucose. The measures displayed in the interface meet requirement c.

2.2 Measure Display The frequency of user-reported data is more frequent than clinical reported data. Thus,

between each two measures which recorded in the clinic/ hospital, there are a scatter plot

17

of the data measured by the patient themselves or the equipment the they wear. This

function applied two-dimensional visualization, which is easy to interpret for users and

satisfies the requirements a and b.

Figure 2 self-reported data and clinical-reported data

2.3 Self-reported Measures The scatter plots start and end with the clinic-reported data before and after that period.

Each circle is a data point that the patient record.

For the measures with upper and lower limits, such as Blood Pressure, Pulse, Pulse

Oxygen, AIC and Glucose, there are blue dashed lines in the chart for the limits. The

outliers beyond the upper or lower limits are in red color.

Figure 3 Self-reported data

18

The line in gray is a smoothing line connecting all the circles to emphasize how the trend

of the measure changes.

Some measures are single value, but Blood Pressure has compounded values (two

values), which include systolic blood pressure and diastolic blood pressure. The measure

with two values is displayed as scatter plots with same y-axis scales.

Figure 4 Measure of two values

Users can mouse over the circle to read more details, containing the exact value and date

time.

Figure 5 Mouseover function

The functions mentioned above meet the requirements a and d. Thus, all aspects of the

function design meet the four user requirements.

19

USER STUDY AND ANALYSIS

The user study is the evaluation part for the visualization interface of the information

system. The user study includes questionnaire of task-based questions and survey, and

semi-structured interview for each participant. I take around five minutes to walk through

the visualization interface, introducing the participant how each of the function works.

Then they are asked to answer the questionnaire made up of single-answer-selection

questions and open-ended questions. The process of finding answer for each question is

similar to the process that the doctors look into the patient’s data and find out more

insights. Afterwards I interview them about how did they answer the questionnaire and

what are their opinions for the visualizations. In general, it takes around 20 min to finish

all the user study parts. The recruit letter, questionnaire and semi-structured interview

questions are in the appendix.

1. Participants Ten participants were recruited for the questionnaires and semi-structured interviews.

Among them, two are M.D. or M.D. Candidates, four are Computer Science Ph.D.

Candidates, one is Library and Information Science Ph.D. Candidate, one is Ph.D.

Candidate in Phycology, one is Ph.D. Candidate in Education, and one is Master of the

School of Government.

20

2. Analysis of Evaluation Results Q1. Find out the Glucose value of this patient on the day of November 28, 2017.

All participants found the correct answer, which indicates that it is not difficult to learn

and find values from the scatter plot.

Q2. Is this Glucose value similar to the Glucose value in about three months, February

10, 2018?

Only 30% of participants selected the correct answer for this question. In the follow-up

interviews, I found it was mostly because the participants have different understanding of

“similar to”. This question should be more specific and provide explanation of what is the

meaning of “similar to”.

Q3. Is the trend of Glucose values during the last two clinical visits (March 4, 2018 and

May 17, 2018) similar to the trend of pulse values during the same time period?

All participants found the correct answer, which indicates that trend of the scatter plots is

not difficult to figure out.

Q4. Does the patient self-reported Glucose data match the readings from doctor’s office

during December 30, 2017 to February 2, 2018?

All participants found the correct answer, which indicates that comparison of clinic

reported data and patient self-reported data are obvious.

Q5. Was it easy to find values for a specific date?

21

Figure 6 Difficulty of finding values

Most participants (90%) think it is not difficult to find specific values.

3. Limitations and Future Work According to the question 7 (What do you find most difficult or limited about this

visualization?) and the user study part, the limitations are as follows:

• The scatter plots are overlapping each other and hard to read if the time interval in

between is very small.

• Users are not able to read all the measures at one time. Instead, they have to hover

each plot to read the value.

• The scale of y-axis is based on the observed maximum and minimum value, thus

the variance of different measures cannot be compared.

0% 10%0%

50%

40%

very hard somewhat hard neutral somewhat easy very easy

22

Future work can be focused on how to solve the overlapping problems if there are too

many values during a short period time. Also, the moved lines can be added so that it will

be easier to align the values and compare with each other.

23

CONCLUSION

The research question for this project is how to visualize health related measures of

diabetes patients, combined both patient self-reported and clinical-reported information,

to assist with clinicians’ understanding of these data and help them to make decisions.

The doctor was interviewed to collect user requirements for the visualization design. The

dataset for visualization design and user study is artificial, because the sensitivity of

patient clinical data and the absence of user self-reported data in current EHR systems.

After design and development of visualization interface, the user study including task-

based questions, surveys and semi-structured interview was conducted to test the

efficiency, effectiveness and satisfaction of the visualization interface.

Overall, the visualization interface of EHR System satisfies the requirements of the

doctors. The way it combines the clinic-reported data and patient self-reported data is

clear and easy to learn. But there are still some functions need to be added to make it

work better.

24

REFERENCE

1. Ashish K. Jha, Catherine M. DesRoches, Eric G. Campbell, Karen Donelan, Sowmya

R. Rao, Timothy G. Ferris, Alexandra Shields, Sara Rosenbaum, and David Blumenthal.

“Use of Electronic Health Records in U.S. Hospitals”, April 16, 2009, N Engl J Med

2009; 360:1628-1638, DOI: 10.1056/NEJMsa0900592

2. A. Hoerbst, E. Ammenwerth. “Electronic Health Records: A Systematic Review on

Quality Requirements”. Methods Inf Med 2010; 49(04): 320-336, DOI: 10.3414/ME10-

01-0038.

3. David Nau, “Effects of a Community Pharmacist-Based Diabetes Patient-Management

Program on Intermediate Clinical Outcome Measures”. J Manag Care Spec Pharm, 2002

Jan; 8(1):48-53.

4. Yixuan Zhang, Kartik Chanana, Cody Dunne. “IDMVis: Temporal Event Sequence

Visualization for Type 1 Diabetes Treatment Decision Support”. IEEE Transactions on

Visualization and Computer Graphics (Volume: 25, Issue: 1, Jan. 2019).

5. S. E. Inzucchi, R. M. Bergenstal, J. B. Buse, M. Diamant, E. Ferrannini, M. Nauck,

A. L. Peters, A. Tsapas, R. Wender, D. R. Matthews. “Management of hyperglycaemia in

type 2 diabetes: a patient-centered approach. Position statement of the American Diabetes

Association (ADA) and the European Association for the Study of Diabetes (EASD)”.

2012.

6. Nancy DBrener, John O.GBilly, William R, GradyM.A.b. “Assessment of factors

affecting the validity of self-reported health-risk behavior among adolescents: evidence

from the scientific literature”. Journal of Adolescent Health, Volume 33, Issue 6,

December 2003, Pages 436-457

7. Seppo Miilunpalo, Correspondence information about the author Seppo Miilunpalo,

Ilkka Vuori, Pekka Oja, Matti Pasanen, Helka Urponen. “Self-rated health status as a

health measure: The predictive value of self-reported health status on the use of physician

25

services and on mortality in the working-age population”. UKK Institute for Health

Promotion Research, Tampere, Finland. May 1997Volume 50, Issue 5, Pages 517–528.

8. Benedetto, Simone; Caldato, Christian; Bazzan, Elia; Greenwood, Darren

C; Pensabene, Virginia; et al. “Assessment of the Fitbit Charge 2 for monitoring heart

rate”. PLoS One; San Francisco Vol. 13, Iss. 2, (Feb 2018): e0192691.

DOI:10.1371/journal.pone.0192691.

9. Azzam, T., Evergreen, S., Germuth, A. A. and Kistler, S. J. (2013), Data Visualization

and Evaluation. New Directions for Evaluation, 2013: 7–32. doi:10.1002/ev.20065

10. Kosara, R. (2007, July). Visualization criticism—The missing link between

information visualization and art. Information Visualization, IV’07. 11th International

Conference (pp. 631–636). doi: 10.1109/IV.2007.2

11. Melissa Mallon (2015) Data Visualization, Public Services Quarterly, 11:3, 183-192,

DOI: 10.1080/15228959.2015.1060147

12. Olsen, Kai A. "Data Visualization." Computer Sciences, edited by K. Lee Lerner and

Brenda Wilmoth Lerner, 2nd ed., Macmillan Reference USA, 2013. Science in Context,

link.galegroup.com/apps/doc/CV2642250148/SCIC?u=unc_main&xid=495a6d14.

Accessed 6 Sept. 2017.

13. T. Munzner (2015). Visualization analysis and design. CRC Press

14. Kovalerchuk, B. (2017). Visual Cognitive Algorithms for High-Dimensional Data

and Super-intelligence Challenges. Cognitive Systems Research.

15. Nicoletta Di Blas , Mirjana Mazuran , Paolo Paolini , Elisa Quintarelli , Letizia

Tanca, Exploratory computing: a challenge for visual interaction, Proceedings of the

2014 International Working Conference on Advanced Visual Interfaces, May 27-29,

2014, Como, Italy [doi>10.1145/2598153.2600037]

16. Pomarede, D; Courtois, HM; Hoffman, Y (2017). Cosmography and Data

Visualization, PUBLICATIONS OF THE ASTRONOMICAL SOCIETY OF THE

PACIFIC, Volume 129, Issue 975

17. Wang Y, White W, Andersen E (2017). PathViewer: Visualizing Pathways through

Student Data. Proceedings of the 2017 CHI Conference on Human Factors in Computing

Systems. pp: 960-964

26

18. Logre I, Mosser S, Riveill M (2015). Composition challenges for sensor data

visualization. Companion Proceedings of the 14th International Conference on

Modularity, pp: 25-26

19. Klomklao T, Ratanarungrong P, Phithakkitnukoon S (2016). Tweets of the nation:

tool for visualizing and analyzing global tweets. Proceedings of the 2016 ACM

International Joint Conference on Pervasive and Ubiquitous Computing Adjunct, pp:

1349-1357

20. Devitt, P.(2015). Research essentials: Understanding quantitative data analysis.

Nursing Children and Young People, Volume 27, Issue 7

21. Lucko, Gunnar; Mitchell, Zane W (2010). Quantitative Research: Preparation of

Incongruous Economic Data Sets for Archival Data Analysis. Journal of Construction

Engineering and Management. 01/2010, Volume 136, Issue 1

22. Marron, J. S. and Alonso, A. M. (2014), Overview of object oriented data analysis.

Biom. J., 56: 732–753. doi:10.1002/bimj.201300072

23. Shruti Kohli and Himani Singal. 2014. Data Analysis with R. In Proceedings of the

2014 IEEE/ACM 7th International Conference on Utility and Cloud Computing (UCC

'14). IEEE Computer Society, Washington, DC, USA, 537-538.

DOI=http://dx.doi.org/10.1109/UCC.2014.81

24. Danyel Fisher. 2016. Big data exploration requires collaboration between

visualization and data infrastructures. In Proceedings of the Workshop on Human-In-the-

Loop Data Analytics (HILDA '16). ACM, New York, NY, USA, Article 16, 5 pages.

DOI: https://doi.org/10.1145/2939502.2939518

25. Peter Norvig. 2011. Internet scale data analysis. In Proceedings of the 17th ACM

SIGKDD international conference on Knowledge discovery and data mining (KDD '11).

ACM, New York, NY, USA, 2-2. DOI: https://doi.org/10.1145/2020408.2020412

26. Rahul C. Basole, Mark L. Braunstein, and Jimeng Sun. 2015. Data and Analytics

Challenges for a Learning Healthcare System. J. Data and Information Quality 6, 2-3,

Article 10 (July 2015), 4 pages. DOI: https://doi.org/10.1145/2755489

27. Richard A. Becker and John M. Chambers. 1984. Design of the S system for data

analysis. Commun. ACM 27, 5 (May 1984), 486-495.

DOI=http://dx.doi.org/10.1145/358189.358078

27

28. Tova Milo and Amit Somech. 2016. REACT: Context-Sensitive Recommendations

for Data Analysis. In Proceedings of the 2016 International Conference on Management

of Data (SIGMOD '16). ACM, New York, NY, USA, 2137-2140. DOI:

https://doi.org/10.1145/2882903.2899392

29. Shyr, J. and Spisic, D. (2014), Automated data analysis. WIREs Comput Stat, 6: 359–

366. doi:10.1002/wics.1318

30. Carpendale, S. (2008). Evaluating information visualizations. In Information

visualization (pp. 19-45). Springer, Berlin, Heidelberg.

31. Wu, D. T., Chen, A. T., Manning, J. D., Levy-Fix, G., Backonja, U., Borland, D., ...

& Kandaswamy, S. (2019). Evaluating visual analytics for health informatics

applications: a systematic review from the American Medical Informatics Association

Visual Analytics Working Group Task Force on Evaluation. Journal of the American

Medical Informatics Association, 26(4), 314-323.

28

APPENDIX

Questionnaire 1. Find out the Glucose value of this patient on the day of November 28, 2017.

____________

2. Is this Glucose value similar to the Glucose value in about three months, February 10, 2018?

Yes No

3. Is the trend of Glucose values during the last two clinical visits (March 4, 2018 and May 17, 2018) similar to the trend of pulse values during the same time period?

Yes No

4. Does the patient self-reported Glucose data match the readings from doctor’s office during December 30, 2017 to February 2, 2018?

Yes No

5. Was it easy to find values for a specific date? ----------+--------------------+-------------------+-------------------+-------------------+---------- Completely not Probably not Not sure likely Of course 6. What do you like most about this visualization?

_______________________________________________________________ 7. What do you find most difficult or limited about this visualization?

________________________________________________________________

29

The following questions are for participants with M.D. only.

8. Can you find out any outliers and unusual patterns of the values before Jan 4, 2018 which may indicate that the patient’s blood glucose will rise on February 3, 2018?

----------+--------------------+--------------------+--------------------+--------------------+---------- Completely not Probably not Not sure likely Of course

9. After the Medicine started form January 2, 2018, do you think the patient’s glucose

was under control? ----------+--------------------+--------------------+--------------------+--------------------+---------- Completely not Probably not Not sure likely Of course

10. Would you find this visualization useful if it becomes part of the Epic systems? ________________________________________________________________

11. When would you use this visualization? ________________________________________________________________

12. What would you change this visualization?

________________________________________________________________

30

Recruit letter Hello, I am doing a research project on flowsheet visualizations to help clinicians have a better understanding of diabetes’ data. Can you join me walk through the visualizations and then conduct a questionnaire and semi-structured interview, which will in total cost around 20 min? Thank you so much. Your effort will be greatly appreciated! Best regards, Lu