interaction breakthroughs in wrangling data · interaction breakthroughs in wrangling data joe...
TRANSCRIPT
![Page 1: Interaction Breakthroughs in Wrangling Data · Interaction Breakthroughs in Wrangling Data Joe Hellerstein Jeffrey Heer Sean Kandel . BACKGROUND: A ... data project is preparing](https://reader030.vdocuments.net/reader030/viewer/2022041005/5eaa0cdd99a32944900082f9/html5/thumbnails/1.jpg)
Interaction Breakthroughs in Wrangling Data Joe Hellerstein Jeffrey Heer Sean Kandel
![Page 2: Interaction Breakthroughs in Wrangling Data · Interaction Breakthroughs in Wrangling Data Joe Hellerstein Jeffrey Heer Sean Kandel . BACKGROUND: A ... data project is preparing](https://reader030.vdocuments.net/reader030/viewer/2022041005/5eaa0cdd99a32944900082f9/html5/thumbnails/2.jpg)
BACKGROUND: A SMALL PROJECT
dp = datapeople
http://deepresearch.org
![Page 3: Interaction Breakthroughs in Wrangling Data · Interaction Breakthroughs in Wrangling Data Joe Hellerstein Jeffrey Heer Sean Kandel . BACKGROUND: A ... data project is preparing](https://reader030.vdocuments.net/reader030/viewer/2022041005/5eaa0cdd99a32944900082f9/html5/thumbnails/3.jpg)
Computing is free. Storage is free. Data is abundant. The remaining bottlenecks lie with people. Research imperative: Dramatically simplify labor-intensive tasks in the analytic lifecycle.
![Page 4: Interaction Breakthroughs in Wrangling Data · Interaction Breakthroughs in Wrangling Data Joe Hellerstein Jeffrey Heer Sean Kandel . BACKGROUND: A ... data project is preparing](https://reader030.vdocuments.net/reader030/viewer/2022041005/5eaa0cdd99a32944900082f9/html5/thumbnails/4.jpg)
dp (c. 2012)
Jeff Heer Stanford
Tapan Parikh Berkeley
Maneesh Agrawala Berkeley
Joe Hellerstein Berkeley
Sean Diana Ravi Kandel MacLean Parikh
Kuang Nicholas Wesley Chen Kong Willett
![Page 5: Interaction Breakthroughs in Wrangling Data · Interaction Breakthroughs in Wrangling Data Joe Hellerstein Jeffrey Heer Sean Kandel . BACKGROUND: A ... data project is preparing](https://reader030.vdocuments.net/reader030/viewer/2022041005/5eaa0cdd99a32944900082f9/html5/thumbnails/5.jpg)
THE ANALYTIC LIFECYCLE
ACQUISITION
TRANSFORMATION
ANALYSIS
VISUALIZATION
COLLABORATION
ACQUISITION
![Page 6: Interaction Breakthroughs in Wrangling Data · Interaction Breakthroughs in Wrangling Data Joe Hellerstein Jeffrey Heer Sean Kandel . BACKGROUND: A ... data project is preparing](https://reader030.vdocuments.net/reader030/viewer/2022041005/5eaa0cdd99a32944900082f9/html5/thumbnails/6.jpg)
THE ANALYTIC LIFECYCLE
ACQUISITION
TRANSFORMATION
ANALYSIS
VISUALIZATION
COLLABORATION
ACQUISITION
![Page 7: Interaction Breakthroughs in Wrangling Data · Interaction Breakthroughs in Wrangling Data Joe Hellerstein Jeffrey Heer Sean Kandel . BACKGROUND: A ... data project is preparing](https://reader030.vdocuments.net/reader030/viewer/2022041005/5eaa0cdd99a32944900082f9/html5/thumbnails/7.jpg)
THE ANALYTIC LIFECYCLE
$57M+ in venture funding to date
Open Source partnership with EMC/Pivotal
Top 10 most popular project on GitHub
![Page 8: Interaction Breakthroughs in Wrangling Data · Interaction Breakthroughs in Wrangling Data Joe Hellerstein Jeffrey Heer Sean Kandel . BACKGROUND: A ... data project is preparing](https://reader030.vdocuments.net/reader030/viewer/2022041005/5eaa0cdd99a32944900082f9/html5/thumbnails/8.jpg)
THE ANALYTIC LIFECYCLE
ACQUISITION
TRANSFORMATION
ANALYSIS
VISUALIZATION
COLLABORATION
ACQUISITION
![Page 9: Interaction Breakthroughs in Wrangling Data · Interaction Breakthroughs in Wrangling Data Joe Hellerstein Jeffrey Heer Sean Kandel . BACKGROUND: A ... data project is preparing](https://reader030.vdocuments.net/reader030/viewer/2022041005/5eaa0cdd99a32944900082f9/html5/thumbnails/9.jpg)
WHERE DOES THE TIME GO IN ANALYTICS?
PROCESSING
ANALYTICS
80% of the work in any
data project is preparing the data.
Patil, Data Jujitsu, 2012. Kandel et al. “Enterprise Data Analysis and Visualization: An Interview Study”, IEEE VAST, 2012.
![Page 10: Interaction Breakthroughs in Wrangling Data · Interaction Breakthroughs in Wrangling Data Joe Hellerstein Jeffrey Heer Sean Kandel . BACKGROUND: A ... data project is preparing](https://reader030.vdocuments.net/reader030/viewer/2022041005/5eaa0cdd99a32944900082f9/html5/thumbnails/10.jpg)
Interview study of 35 analysts:
25 companies Healthcare Retail, Marketing Social networking Media Finance, Insurance
Various titles Data analyst Data scientist Software engineer Consultant Chief technical officer
[Kandel et al., VAST12]
KANDEL SURVEY
![Page 11: Interaction Breakthroughs in Wrangling Data · Interaction Breakthroughs in Wrangling Data Joe Hellerstein Jeffrey Heer Sean Kandel . BACKGROUND: A ... data project is preparing](https://reader030.vdocuments.net/reader030/viewer/2022041005/5eaa0cdd99a32944900082f9/html5/thumbnails/11.jpg)
“I spend more than half of my time integrating, cleansing and transforming data without doing any actual analysis. Most of the time I’m lucky if I get to do any ‘analysis’ at all.”
Friction
“Most of the time once you transform the data ... the insights can be scarily obvious.”
Lost potential
![Page 12: Interaction Breakthroughs in Wrangling Data · Interaction Breakthroughs in Wrangling Data Joe Hellerstein Jeffrey Heer Sean Kandel . BACKGROUND: A ... data project is preparing](https://reader030.vdocuments.net/reader030/viewer/2022041005/5eaa0cdd99a32944900082f9/html5/thumbnails/12.jpg)
“It’s easy to just think you know what you are doing and not look at data at every intermediary step.
An analysis has 30 different steps. It’s tempting to just do this then that and then this. You have no idea in which ways you are wrong and what data is wrong.”
Interactivity and Visualization
![Page 13: Interaction Breakthroughs in Wrangling Data · Interaction Breakthroughs in Wrangling Data Joe Hellerstein Jeffrey Heer Sean Kandel . BACKGROUND: A ... data project is preparing](https://reader030.vdocuments.net/reader030/viewer/2022041005/5eaa0cdd99a32944900082f9/html5/thumbnails/13.jpg)
![Page 14: Interaction Breakthroughs in Wrangling Data · Interaction Breakthroughs in Wrangling Data Joe Hellerstein Jeffrey Heer Sean Kandel . BACKGROUND: A ... data project is preparing](https://reader030.vdocuments.net/reader030/viewer/2022041005/5eaa0cdd99a32944900082f9/html5/thumbnails/14.jpg)
…
THE DATA TRANSFORMATION PROBLEM
DATA TRANSFORMATION
Business System Data
Machine Generated Data
Log Data Data Visualization
Fraud Detection
Recommendations
DATA SOURCE Complexity
DATA PRODUCT Simplicity
…
A PROGRAMMING PROBLEM
![Page 15: Interaction Breakthroughs in Wrangling Data · Interaction Breakthroughs in Wrangling Data Joe Hellerstein Jeffrey Heer Sean Kandel . BACKGROUND: A ... data project is preparing](https://reader030.vdocuments.net/reader030/viewer/2022041005/5eaa0cdd99a32944900082f9/html5/thumbnails/15.jpg)
POTTER’S WHEEL (2001): ENTER THE VISUAL
➔ Step-by-step DSL ➔ Immediate visual feedback
➔ Ongoing discrepancy detection
➔ Data lineage, redo/undo
Problem: Remaining burden of specification for users.
[Raman & Hellerstein, VLDB11]
![Page 16: Interaction Breakthroughs in Wrangling Data · Interaction Breakthroughs in Wrangling Data Joe Hellerstein Jeffrey Heer Sean Kandel . BACKGROUND: A ... data project is preparing](https://reader030.vdocuments.net/reader030/viewer/2022041005/5eaa0cdd99a32944900082f9/html5/thumbnails/16.jpg)
DSL
VISUAL
VISUAL SPECIFICATION
Data Vis Visual Results
visualization compilation
Data Results coding
interaction
![Page 17: Interaction Breakthroughs in Wrangling Data · Interaction Breakthroughs in Wrangling Data Joe Hellerstein Jeffrey Heer Sean Kandel . BACKGROUND: A ... data project is preparing](https://reader030.vdocuments.net/reader030/viewer/2022041005/5eaa0cdd99a32944900082f9/html5/thumbnails/17.jpg)
TRADITIONAL BURDEN OF SPECIFICATION
Visualization and Interaction
Data Transformation Code
User authors a draft transformation script
User tests the script on a small amount of data
User inspects output data to assess effects
1. 2.
3.
![Page 18: Interaction Breakthroughs in Wrangling Data · Interaction Breakthroughs in Wrangling Data Joe Hellerstein Jeffrey Heer Sean Kandel . BACKGROUND: A ... data project is preparing](https://reader030.vdocuments.net/reader030/viewer/2022041005/5eaa0cdd99a32944900082f9/html5/thumbnails/18.jpg)
HINTS OF INTELLIGENT INTERFACES
Type-ahead uses context and data to predict your search term — and preview results
![Page 19: Interaction Breakthroughs in Wrangling Data · Interaction Breakthroughs in Wrangling Data Joe Hellerstein Jeffrey Heer Sean Kandel . BACKGROUND: A ... data project is preparing](https://reader030.vdocuments.net/reader030/viewer/2022041005/5eaa0cdd99a32944900082f9/html5/thumbnails/19.jpg)
WRANGLER (2011): ADD INTELLIGENCE
[Kandel, et al. CHI 11]
[Guo, et al. UIST11]
➔ Automatic inference of transforms
➔ Semantic data types
➔ Interactive history
➔ User Studies
http://vis.stanford.edu/wrangler
![Page 20: Interaction Breakthroughs in Wrangling Data · Interaction Breakthroughs in Wrangling Data Joe Hellerstein Jeffrey Heer Sean Kandel . BACKGROUND: A ... data project is preparing](https://reader030.vdocuments.net/reader030/viewer/2022041005/5eaa0cdd99a32944900082f9/html5/thumbnails/20.jpg)
DSL
Visual
PREDICTIVE INTERACTION
Data Vis Visual Results
visualization compilation
Data Results coding
ambiguous interaction Response
Visualization of probable Next Steps
disambiguation Prediction guide decide
![Page 21: Interaction Breakthroughs in Wrangling Data · Interaction Breakthroughs in Wrangling Data Joe Hellerstein Jeffrey Heer Sean Kandel . BACKGROUND: A ... data project is preparing](https://reader030.vdocuments.net/reader030/viewer/2022041005/5eaa0cdd99a32944900082f9/html5/thumbnails/21.jpg)
PREDICTIVE INTERACTION™
Visualization and Interaction
Data Transformation Code
User highlights features of a data visualization
Data previews allow user to choose, adjust and confirm
ML methods predict distribution over DSL statements
1. 3.
2.
![Page 22: Interaction Breakthroughs in Wrangling Data · Interaction Breakthroughs in Wrangling Data Joe Hellerstein Jeffrey Heer Sean Kandel . BACKGROUND: A ... data project is preparing](https://reader030.vdocuments.net/reader030/viewer/2022041005/5eaa0cdd99a32944900082f9/html5/thumbnails/22.jpg)
Demo
![Page 23: Interaction Breakthroughs in Wrangling Data · Interaction Breakthroughs in Wrangling Data Joe Hellerstein Jeffrey Heer Sean Kandel . BACKGROUND: A ... data project is preparing](https://reader030.vdocuments.net/reader030/viewer/2022041005/5eaa0cdd99a32944900082f9/html5/thumbnails/23.jpg)
IN SUM
➔ Focus on the human bottlenecks in data analysis! ➔ High impact work to be done here ➔ Trifacta customers: order-of-magnitude efficiency gains
➔ Tech transfer via many paths ➔ Startups ➔ Open-source collaborations with industry ➔ Campus open source
➔ Design challenges: Subtle interplay of three aspects ➔ People: Visualization and Interaction ➔ Data: DSL of “actual” behaviors ➔ Computation: prediction of DSL steps
Co-design these three!