7 dimensions of agile analytics by ken collier
TRANSCRIPT
7 DIMENSIONS OF AGILE ANALYTICS Ken Collier, PhD Director, Agile Analytics @theagilist #thoughtworks
1
Busi
ness
Val
ue
Analytical Complexity
What happened?
Descriptive Analytics
Why did it happen?
Diagnostic Analytics
What will happen?
Predictive Analytics Can we influence
what happens?
Prescriptive Analytics
Busi
ness
Val
ue
Analytical Complexity
What happened?
Descriptive Analytics
Why did it happen?
Diagnostic Analytics
What will happen?
Predictive Analytics How can we
make it happen?
Prescriptive Analytics
3
Business Intelligence
Data Science
THE DIFFERENCE
Data Engineering
Lean Learning
Streaming data pipelines &
adaptive architectures
Continuously challenge your assumptions by
measuring results.
Discovery of patterns and
signals hidden in data
Agile Delivery
Data Science
Deliver business value early and
often. Build your platform over time,
not all up front. Your Business
Questions
=
Fast results & Early Value
Data Guided Market
Advantage
Agile Analytics
Data Engineering
Solutions Thinking
Ethics
Agile Delivery Lean
Learning
Impact
Advanced Analytics
Agile Analytics
Data Engineering Solutions
Thinking
Ethics
Agile Delivery Lean
Learning
Impact
Advanced Analytics
Adaptive Architecture
Streaming Data
Polyglot Persistence
Strictly Private and Confidential © 2015 ThoughtWorks, Ltd. All rights reserved.
REMOVING THE DATA BOTTLE NECK
Data Warehouse Incoming data is cleaned and
organised into a single schema up front.
Data Lake Incoming data goes into the lake
in its raw form.
Long development times to create new value from data.
Analysis activities are distributed across technologists and business
users.
Analysis is done directly on the curated warehouse data.
Data is selected, structured, and organize as needed, when
needed.
DATA LAKE DONE RIGHT
8
Operational systems communicate directly
with each other via services
Operational systems push data to the lake
via topical queues
Data scientists explore the lake for
potential insights
Lakeshore marts and services curate and
organize the data for self-service analysis
Multi-tiered data lake for processing,
distribution, serving
ADAPTIVE ARCHITECTURE PRINCIPLES
9
Enable low latency data streaming
Store raw, low-level, historized data
Enable NoSQL presentation
Enable inexpensive scaling
Simplify data ingestion
Drive logic closer to the business
Enable emergent design
Enable easy recreation of data
DATA ENGINEERING CONCERNS
10
Streaming
Distributed MPP architecture
Data Strategy Elastic cloud computing
Reactive architecture
Master data
Data Governance
ETL techniques
Advanced Analytics
Agile Analytics
Solutions Thinking
Ethics
Agile Delivery Lean
Learning
Impact
Data Engineering
Adaptive Architecture
Streaming Data
Polyglot Persistence
Data Science
Machine Learning
Statistics
Discover & Explore
Analyze & Act
Data Convergence Analytical Divergence
Discover
Harvest
Filter
Integrate Augment Analyze
Act
Analytical Opportunities
HOW DATA SCIENCE WORKS Can we anticipate what the customer
will want to do next?
THE “DATA SCIENTIST”
Machine Learning Statistical Modeling
Artificial Neural Networks
Decision Tree Learning
Support Vector Machines
Unsupervised Learning
…and many more…
Bayesian Classification
Monte Carlo Simulation
Logistic Regression
K-Nearest Neighbor
…and many more…
Feature Engineering
Feature Extraction
Dimension Reduction
Domain expertise
Programming Skills
Functional Programming
Data “Wrangling”
Map/Reduce, SQL, & NoSQL
Objective Truth
Discoverable Truth
Uninterpretable
Irrelevant Noise
Not Actionable
Actionable Signals
MAKING
“BIG DATA”
INTO
“LITTLE DATA”
Advanced Analytics
Data Science
Visual Storytelling
Machine Learning
Statistics Agile Analytics
Solutions Thinking
Ethics
Agile Delivery Lean
Learning
Impact
Data Engineering
Volume Velocity
Variety
Adaptive Architecture
Streaming Data
Polyglot Persistence
Advanced Analytics
Data Science
Visual Storytelling
Machine Learning
Statistics Agile Analytics
Solutions Thinking
Ethics
Agile Delivery Lean
Learning
Impact
Continuous Integration
Collaboration Evolve
Continuous Delivery
Hypothesis
Build Learn
Measure
Data Reduction
Data Engineering
Volume Velocity
Variety
Adaptive Architecture Streaming
Data Polyglot
Persistence
drones.pitchinteractive.com
Data Visualization
drones.pitchinteractive.com
Advanced Analytics
Agile Analytics
Solutions Thinking
Ethics
Agile Delivery Lean
Learning
Impact
Hypothesis
Build Learn
Measure
Data Engineering
Adaptive Architecture
Streaming Data
Polyglot Persistence
Data Science
Machine Learning
Statistics
Visual Storytelling
Typical Timeline
3-6 months 1-2 months 2-4 months
22
Data Convergence Analytical Divergence
Discover
Harvest
Filter
Integrate Augment Analyze
Act
Analytical Opportunities
CONVENTIONAL DATA SCIENCE If we knew X, we could do Y
Analytical Divergence
Analytical Opportunities If we knew X, we could do Y
Data Convergence
Discover
Harvest
Filter
Integrate Augment Analyze
Act
Repeat this cycle solving small problems every few days
LEARN
MEASURE
BUILD
LEAN DATA SCIENCE
Advanced Analytics
Agile Analytics
Solutions Thinking
Ethics
Agile Delivery
Impact
Reflect & Improve
Collaborate Evolve
Deliver
Data Engineering
Adaptive Architecture
Streaming Data
Polyglot Persistence
Data Science
Machine Learning
Statistics
Visual Storytelling
Lean Learning
Hypothesis
Build Learn
Measure
Retain high value customers
Problem solved or continue?
High value business goal
What’s the smallest, simplest thing we can do?
Is it useful & actionable? Repeat!
What leads to customers leaving?
LIKE THIS EXAMPLE… Common features of
defectors?
Shopping behaviors of defectors?
What do defectors say about us?
Customers’ sentiment before defecting?
What encourages customers to stay?
Do incentives reduce defection rates?
Retain high value customers
High value business goal
Like this example…
What’s the smallest, simplest thing we can do?
Retain high value customers
Like this example… Common features of
defectors?
Is it useful & actionable?
Retain high value customers
Like this example… Common features of
defectors?
Repeat! Retain high value customers
Like this example… Common features of
defectors?
Shopping behaviors of defectors?
Retain high value customers
Like this example… Common features of
defectors?
What leads to customers leaving?
Shopping behaviors of defectors?
What do defectors say about us?
Customers’ sentiment before defecting?
What encourages customers to stay?
Do incentives reduce defection rates?
Problem solved or continue?
What leads to customers leaving?
Like this example… Common features of
defectors?
Shopping behaviors of defectors?
What do defectors say about us?
Customers’ sentiment before defecting?
What encourages customers to stay?
Do incentives reduce defection rates?
Advanced Analytics
Data Science
Visual Storytelling
Machine Learning
Statistics Agile Analytics
Solutions Thinking
Ethics
Agile Delivery Lean
Learning
Impact
Hypothesis
Build Learn
Measure
Insight
Knowledge
Action
Disruption
Data Engineering
Volume Velocity
Variety
Adaptive Architecture Streaming
Data Polyglot
Persistence
Reflect & Improve
Collaborate Evolve
Deliver
Advanced Analytics
Data Science
Visual Storytelling
Machine Learning
Statistics Agile Analytics
Solutions Thinking
Ethics
Agile Delivery Lean
Learning
Impact
Hypothesis
Build Learn
Measure
Insight
Knowledge
Action
Disruption
Business First
Evolve the Platform
Monitor & Measure
Data Engineering
Adaptive Architecture Streaming
Data Polyglot
Persistence
Reflect & Improve
Collaborate Evolve
Deliver
QUESTIONS FIRST, DATA SECOND
34
“We built a platform for self service, and now we’re trying to get the business to use it.”
From this… …to this
Advanced Analytics
Data Science
Visual Storytelling
Machine Learning
Statistics Agile Analytics
Solutions Thinking
Ethics
Agile Delivery Lean
Learning
Impact
Hypothesis
Build Learn
Measure
Insight
Knowledge
Action
Disruption
Business First
Evolve the Platform
Monitor & Measure
Privacy Controls Radical Transparency
Data Democracy
Open Data
Data Engineering
Volume Velocity
Variety
Adaptive Architecture Streaming
Data Polyglot
Persistence
Reflect & Improve
Collaborate Evolve
Deliver
38 http://bit.ly/DataAndPrivacy
Jonny Leroy, NA Head of Technology, ThoughtWorks
39
40
41
42
43
44
45
46
PRIVACY IS DEAD
DON’T BE EVIL
47
48
49
50
51
52
Advanced Analytics
Data Science
Visual Storytelling
Machine Learning
Statistics Agile Analytics
Solutions Thinking
Ethics
Agile Delivery Lean
Learning
Impact
Hypothesis
Build Learn
Measure
Insight
Knowledge
Action
Disruption
Business First
Evolve the Platform
Monitor & Measure
Privacy Controls Radical Transparency
Data Democracy
Open Data
Data Engineering
Volume Velocity
Variety
Adaptive Architecture Streaming
Data Polyglot
Persistence
Reflect & Improve
Collaborate Evolve
Deliver
Ken Collier, Director, Agile Analytics [email protected]
Value Creation
Cool New Technologies +
Sophisticated Analytics +
Lean Learning Principals +
Fast Agile Delivery =