exploratory data analysis - ohsu informatics · exploratory data analysis visualizing your data...
TRANSCRIPT
![Page 1: Exploratory Data Analysis - OHSU Informatics · Exploratory Data Analysis Visualizing your data Shannon McWeeney, PhD 14th January 2016](https://reader033.vdocuments.net/reader033/viewer/2022052801/5f16b21b7b41fc222e3350c8/html5/thumbnails/1.jpg)
Exploratory Data AnalysisVisualizing your data
Shannon McWeeney, PhD14th January 2016
![Page 2: Exploratory Data Analysis - OHSU Informatics · Exploratory Data Analysis Visualizing your data Shannon McWeeney, PhD 14th January 2016](https://reader033.vdocuments.net/reader033/viewer/2022052801/5f16b21b7b41fc222e3350c8/html5/thumbnails/2.jpg)
Exploratory Data Analysis (EDA)
2
1st step in a 2‐step process
![Page 3: Exploratory Data Analysis - OHSU Informatics · Exploratory Data Analysis Visualizing your data Shannon McWeeney, PhD 14th January 2016](https://reader033.vdocuments.net/reader033/viewer/2022052801/5f16b21b7b41fc222e3350c8/html5/thumbnails/3.jpg)
Main Objectives• ASSESS Assumptions
• SUPPORT Selection
• PROVIDE Basis
3
![Page 4: Exploratory Data Analysis - OHSU Informatics · Exploratory Data Analysis Visualizing your data Shannon McWeeney, PhD 14th January 2016](https://reader033.vdocuments.net/reader033/viewer/2022052801/5f16b21b7b41fc222e3350c8/html5/thumbnails/4.jpg)
• Examines distributions + relationships
• Utilizes visualization + numerical summaries
4
EDA Features
![Page 5: Exploratory Data Analysis - OHSU Informatics · Exploratory Data Analysis Visualizing your data Shannon McWeeney, PhD 14th January 2016](https://reader033.vdocuments.net/reader033/viewer/2022052801/5f16b21b7b41fc222e3350c8/html5/thumbnails/5.jpg)
• Examines distributions + relationships
• Utilizes visualization + numerical summaries
5
EDA FEATURES
![Page 6: Exploratory Data Analysis - OHSU Informatics · Exploratory Data Analysis Visualizing your data Shannon McWeeney, PhD 14th January 2016](https://reader033.vdocuments.net/reader033/viewer/2022052801/5f16b21b7b41fc222e3350c8/html5/thumbnails/6.jpg)
Everything is done in framework of the analysis plan!
6
Context is Key
![Page 7: Exploratory Data Analysis - OHSU Informatics · Exploratory Data Analysis Visualizing your data Shannon McWeeney, PhD 14th January 2016](https://reader033.vdocuments.net/reader033/viewer/2022052801/5f16b21b7b41fc222e3350c8/html5/thumbnails/7.jpg)
Data File Asessment:#variables#subjectsRange
% Missing
7
Starting Point
![Page 8: Exploratory Data Analysis - OHSU Informatics · Exploratory Data Analysis Visualizing your data Shannon McWeeney, PhD 14th January 2016](https://reader033.vdocuments.net/reader033/viewer/2022052801/5f16b21b7b41fc222e3350c8/html5/thumbnails/8.jpg)
R Commands:Summary()
Dim()
8
Assessing the File
![Page 9: Exploratory Data Analysis - OHSU Informatics · Exploratory Data Analysis Visualizing your data Shannon McWeeney, PhD 14th January 2016](https://reader033.vdocuments.net/reader033/viewer/2022052801/5f16b21b7b41fc222e3350c8/html5/thumbnails/9.jpg)
9
OHSU Resources• R Boot‐camp: Created by Dr. Ted Laderas, Division of Bioinformatics and
Computational Biology , Department of Medical Informatics and Clinical Epidemiology
https://www.coursesites.com/s/_Rbootcamp(Coursesites registration required)
![Page 10: Exploratory Data Analysis - OHSU Informatics · Exploratory Data Analysis Visualizing your data Shannon McWeeney, PhD 14th January 2016](https://reader033.vdocuments.net/reader033/viewer/2022052801/5f16b21b7b41fc222e3350c8/html5/thumbnails/10.jpg)
• Improved data quality• Improved trust in data integrity• Improved documentation and control• Reduced data redundancy• Reuse of data• Consistency in data use• Easier data analysis• Improved decision making based on better data• Simpler programming• Enforcement of standards
10
Benefits of a Data dictionary*
*From Ahima.org
![Page 11: Exploratory Data Analysis - OHSU Informatics · Exploratory Data Analysis Visualizing your data Shannon McWeeney, PhD 14th January 2016](https://reader033.vdocuments.net/reader033/viewer/2022052801/5f16b21b7b41fc222e3350c8/html5/thumbnails/11.jpg)
First Example
11
Source: AHIMA.ORG
![Page 12: Exploratory Data Analysis - OHSU Informatics · Exploratory Data Analysis Visualizing your data Shannon McWeeney, PhD 14th January 2016](https://reader033.vdocuments.net/reader033/viewer/2022052801/5f16b21b7b41fc222e3350c8/html5/thumbnails/12.jpg)
12
2nd Example
https://tcga‐data.nci.nih.gov/docs/dictionary/
![Page 13: Exploratory Data Analysis - OHSU Informatics · Exploratory Data Analysis Visualizing your data Shannon McWeeney, PhD 14th January 2016](https://reader033.vdocuments.net/reader033/viewer/2022052801/5f16b21b7b41fc222e3350c8/html5/thumbnails/13.jpg)
• Provide visual information
• Examine relationships; distribution
13
DISPLAYING Data: GRAPHICS
![Page 14: Exploratory Data Analysis - OHSU Informatics · Exploratory Data Analysis Visualizing your data Shannon McWeeney, PhD 14th January 2016](https://reader033.vdocuments.net/reader033/viewer/2022052801/5f16b21b7b41fc222e3350c8/html5/thumbnails/14.jpg)
14
Assessing: Relationships
-2 -1 0 1 2
-2-1
01
23
dat$Germinal.center.B.cell.signature
dat$
Lym
ph.n
ode.
sign
atur
e
R Commands:Plot()Cor()
mple..LYM
1.0 1.8 1.0 1.8 1.0 3.5 -2 1 -2 1 -1 2
030
0
1.0
1.8
nalysis.Se
ow.up..yea
015
1.0
1.8
us.at.follow
Subgroup
1.0
2.5
1.0
3.5
IPI.Group
enter.B.ce
-21
-21 h.node.sig
ration.sign
-11
-21 BMP6
class.II.sig
-30
0 300
-12
0 15 1.0 2.5 -2 1 -1 1 -3 0
me.predicto
![Page 15: Exploratory Data Analysis - OHSU Informatics · Exploratory Data Analysis Visualizing your data Shannon McWeeney, PhD 14th January 2016](https://reader033.vdocuments.net/reader033/viewer/2022052801/5f16b21b7b41fc222e3350c8/html5/thumbnails/15.jpg)
15
Assessing: Distributions
R Commands:Hist() ( also histogram() in lattice library)
boxplot()
ABC GCB Type III
05
1015
20
dat$Follow.up..years
Perc
ent o
f Tot
al
0
20
40
60
0 5 10 15 20
Alive
0 5 10 15 20
Dead
![Page 16: Exploratory Data Analysis - OHSU Informatics · Exploratory Data Analysis Visualizing your data Shannon McWeeney, PhD 14th January 2016](https://reader033.vdocuments.net/reader033/viewer/2022052801/5f16b21b7b41fc222e3350c8/html5/thumbnails/16.jpg)
• Layout
• “Stand alone”
• Comparison of interest/focus
16
Displaying Data: TABLES
![Page 17: Exploratory Data Analysis - OHSU Informatics · Exploratory Data Analysis Visualizing your data Shannon McWeeney, PhD 14th January 2016](https://reader033.vdocuments.net/reader033/viewer/2022052801/5f16b21b7b41fc222e3350c8/html5/thumbnails/17.jpg)
17
Sample Inspection
6RINng=15156x3252ng=16441x8005
Inf_Status=MInf_Status=W
Timepoint=D12Timepoint=D2
Timepoint=D21Timepoint=D28
Timepoint=D4Timepoint=D7
Lab=GLab=L
Tissue=BrTissue=Sp
−2
−1
0
1
2
3
R Commands:Heatmap()
![Page 18: Exploratory Data Analysis - OHSU Informatics · Exploratory Data Analysis Visualizing your data Shannon McWeeney, PhD 14th January 2016](https://reader033.vdocuments.net/reader033/viewer/2022052801/5f16b21b7b41fc222e3350c8/html5/thumbnails/18.jpg)
18
Sample Inspection
69
RIN
Mating=15156x3252Mating=16441x8005
Inf_Status=MInf_Status=W
Timepoint=D12Timepoint=D2
Timepoint=D21Timepoint=D28
Timepoint=D4Timepoint=D7
Lab=GLab=L
Tissue=Sp
−6
−4
−2
0
2
4
6
![Page 19: Exploratory Data Analysis - OHSU Informatics · Exploratory Data Analysis Visualizing your data Shannon McWeeney, PhD 14th January 2016](https://reader033.vdocuments.net/reader033/viewer/2022052801/5f16b21b7b41fc222e3350c8/html5/thumbnails/19.jpg)
Display data accurately and clearly
Good and bad data visualization
![Page 20: Exploratory Data Analysis - OHSU Informatics · Exploratory Data Analysis Visualizing your data Shannon McWeeney, PhD 14th January 2016](https://reader033.vdocuments.net/reader033/viewer/2022052801/5f16b21b7b41fc222e3350c8/html5/thumbnails/20.jpg)
20
![Page 21: Exploratory Data Analysis - OHSU Informatics · Exploratory Data Analysis Visualizing your data Shannon McWeeney, PhD 14th January 2016](https://reader033.vdocuments.net/reader033/viewer/2022052801/5f16b21b7b41fc222e3350c8/html5/thumbnails/21.jpg)
21
![Page 22: Exploratory Data Analysis - OHSU Informatics · Exploratory Data Analysis Visualizing your data Shannon McWeeney, PhD 14th January 2016](https://reader033.vdocuments.net/reader033/viewer/2022052801/5f16b21b7b41fc222e3350c8/html5/thumbnails/22.jpg)
22
![Page 23: Exploratory Data Analysis - OHSU Informatics · Exploratory Data Analysis Visualizing your data Shannon McWeeney, PhD 14th January 2016](https://reader033.vdocuments.net/reader033/viewer/2022052801/5f16b21b7b41fc222e3350c8/html5/thumbnails/23.jpg)
23
![Page 24: Exploratory Data Analysis - OHSU Informatics · Exploratory Data Analysis Visualizing your data Shannon McWeeney, PhD 14th January 2016](https://reader033.vdocuments.net/reader033/viewer/2022052801/5f16b21b7b41fc222e3350c8/html5/thumbnails/24.jpg)
24
![Page 25: Exploratory Data Analysis - OHSU Informatics · Exploratory Data Analysis Visualizing your data Shannon McWeeney, PhD 14th January 2016](https://reader033.vdocuments.net/reader033/viewer/2022052801/5f16b21b7b41fc222e3350c8/html5/thumbnails/25.jpg)
Bad Data Viz• Not informative
• Data is obscured (Tufte’s “Chart junk”*)
• Pie charts (3d!!)
• Issues of scale
25
*Tufte, E. R. The visual display of quantitative information
![Page 26: Exploratory Data Analysis - OHSU Informatics · Exploratory Data Analysis Visualizing your data Shannon McWeeney, PhD 14th January 2016](https://reader033.vdocuments.net/reader033/viewer/2022052801/5f16b21b7b41fc222e3350c8/html5/thumbnails/26.jpg)
• WHAT IS THE STORY?
• WHAT DO YOU NEED TO KNOW TO INTERPRET IT?
26
Graphical Proficiency
![Page 27: Exploratory Data Analysis - OHSU Informatics · Exploratory Data Analysis Visualizing your data Shannon McWeeney, PhD 14th January 2016](https://reader033.vdocuments.net/reader033/viewer/2022052801/5f16b21b7b41fc222e3350c8/html5/thumbnails/27.jpg)
Interactive Visualization
Examples & Tools you can use
![Page 28: Exploratory Data Analysis - OHSU Informatics · Exploratory Data Analysis Visualizing your data Shannon McWeeney, PhD 14th January 2016](https://reader033.vdocuments.net/reader033/viewer/2022052801/5f16b21b7b41fc222e3350c8/html5/thumbnails/28.jpg)
28
INTERACTIVE GRAPHICS
GAPMINDER.ORG
![Page 29: Exploratory Data Analysis - OHSU Informatics · Exploratory Data Analysis Visualizing your data Shannon McWeeney, PhD 14th January 2016](https://reader033.vdocuments.net/reader033/viewer/2022052801/5f16b21b7b41fc222e3350c8/html5/thumbnails/29.jpg)
29
Interactive Data: Path Models
![Page 30: Exploratory Data Analysis - OHSU Informatics · Exploratory Data Analysis Visualizing your data Shannon McWeeney, PhD 14th January 2016](https://reader033.vdocuments.net/reader033/viewer/2022052801/5f16b21b7b41fc222e3350c8/html5/thumbnails/30.jpg)
30
Google Charts
https://developers.google.com/chart/?csw=1
![Page 31: Exploratory Data Analysis - OHSU Informatics · Exploratory Data Analysis Visualizing your data Shannon McWeeney, PhD 14th January 2016](https://reader033.vdocuments.net/reader033/viewer/2022052801/5f16b21b7b41fc222e3350c8/html5/thumbnails/31.jpg)
31
Shiny (Rstudio)
http://shiny.rstudio.com/
![Page 32: Exploratory Data Analysis - OHSU Informatics · Exploratory Data Analysis Visualizing your data Shannon McWeeney, PhD 14th January 2016](https://reader033.vdocuments.net/reader033/viewer/2022052801/5f16b21b7b41fc222e3350c8/html5/thumbnails/32.jpg)
32
Data Driven (D3JS)
http://d3js.org
![Page 33: Exploratory Data Analysis - OHSU Informatics · Exploratory Data Analysis Visualizing your data Shannon McWeeney, PhD 14th January 2016](https://reader033.vdocuments.net/reader033/viewer/2022052801/5f16b21b7b41fc222e3350c8/html5/thumbnails/33.jpg)
“If you don’t think you have a quality problem with your data, you haven’t
looked at it”
Every data set has quirks.
![Page 34: Exploratory Data Analysis - OHSU Informatics · Exploratory Data Analysis Visualizing your data Shannon McWeeney, PhD 14th January 2016](https://reader033.vdocuments.net/reader033/viewer/2022052801/5f16b21b7b41fc222e3350c8/html5/thumbnails/34.jpg)
• Denial
• Anger
• Bargaining
• Depression
• Acceptance (+ Hope!)
34
5 Stages of Data Grief
![Page 35: Exploratory Data Analysis - OHSU Informatics · Exploratory Data Analysis Visualizing your data Shannon McWeeney, PhD 14th January 2016](https://reader033.vdocuments.net/reader033/viewer/2022052801/5f16b21b7b41fc222e3350c8/html5/thumbnails/35.jpg)
• Software shouldn’t dictate the Visual
• Tell a story
• Follow best practices (be mindful)
35
Visual Points to remember