7 dimensions of agile analytics by ken collier

56
7 DIMENSIONS OF AGILE ANALYTICS Ken Collier, PhD Director, Agile Analytics @theagilist #thoughtworks 1

Upload: thoughtworks

Post on 17-Feb-2017

1.318 views

Category:

Technology


0 download

TRANSCRIPT

Page 1: 7 Dimensions of Agile Analytics by Ken Collier

7 DIMENSIONS OF AGILE ANALYTICS Ken Collier, PhD Director, Agile Analytics @theagilist #thoughtworks

1

Page 2: 7 Dimensions of Agile Analytics by Ken Collier

Busi

ness

Val

ue

Analytical Complexity

What happened?

Descriptive Analytics

Why did it happen?

Diagnostic Analytics

What will happen?

Predictive Analytics Can we influence

what happens?

Prescriptive Analytics

Page 3: 7 Dimensions of Agile Analytics by Ken Collier

Busi

ness

Val

ue

Analytical Complexity

What happened?

Descriptive Analytics

Why did it happen?

Diagnostic Analytics

What will happen?

Predictive Analytics How can we

make it happen?

Prescriptive Analytics

3

Business Intelligence

Data Science

Page 4: 7 Dimensions of Agile Analytics by Ken Collier

THE DIFFERENCE

Data Engineering

Lean Learning

Streaming data pipelines &

adaptive architectures

Continuously challenge your assumptions by

measuring results.

Discovery of patterns and

signals hidden in data

Agile Delivery

Data Science

Deliver business value early and

often. Build your platform over time,

not all up front. Your Business

Questions

=

Fast results & Early Value

Data Guided Market

Advantage

Page 5: 7 Dimensions of Agile Analytics by Ken Collier

Agile Analytics

Data Engineering

Solutions Thinking

Ethics

Agile Delivery Lean

Learning

Impact

Advanced Analytics

Page 6: 7 Dimensions of Agile Analytics by Ken Collier

Agile Analytics

Data Engineering Solutions

Thinking

Ethics

Agile Delivery Lean

Learning

Impact

Advanced Analytics

Adaptive Architecture

Streaming Data

Polyglot Persistence

Page 7: 7 Dimensions of Agile Analytics by Ken Collier

Strictly Private and Confidential © 2015 ThoughtWorks, Ltd. All rights reserved.

REMOVING THE DATA BOTTLE NECK

Data Warehouse Incoming data is cleaned and

organised into a single schema up front.

Data Lake Incoming data goes into the lake

in its raw form.

Long development times to create new value from data.

Analysis activities are distributed across technologists and business

users.

Analysis is done directly on the curated warehouse data.

Data is selected, structured, and organize as needed, when

needed.

Page 8: 7 Dimensions of Agile Analytics by Ken Collier

DATA LAKE DONE RIGHT

8

Operational systems communicate directly

with each other via services

Operational systems push data to the lake

via topical queues

Data scientists explore the lake for

potential insights

Lakeshore marts and services curate and

organize the data for self-service analysis

Multi-tiered data lake for processing,

distribution, serving

Page 9: 7 Dimensions of Agile Analytics by Ken Collier

ADAPTIVE ARCHITECTURE PRINCIPLES

9

Enable low latency data streaming

Store raw, low-level, historized data

Enable NoSQL presentation

Enable inexpensive scaling

Simplify data ingestion

Drive logic closer to the business

Enable emergent design

Enable easy recreation of data

Page 10: 7 Dimensions of Agile Analytics by Ken Collier

DATA ENGINEERING CONCERNS

10

Streaming

Distributed MPP architecture

Data Strategy Elastic cloud computing

Reactive architecture

Master data

Data Governance

ETL techniques

Page 11: 7 Dimensions of Agile Analytics by Ken Collier

Advanced Analytics

Agile Analytics

Solutions Thinking

Ethics

Agile Delivery Lean

Learning

Impact

Data Engineering

Adaptive Architecture

Streaming Data

Polyglot Persistence

Data Science

Machine Learning

Statistics

Page 12: 7 Dimensions of Agile Analytics by Ken Collier

Discover & Explore

Analyze & Act

Data Convergence Analytical Divergence

Discover

Harvest

Filter

Integrate Augment Analyze

Act

Analytical Opportunities

HOW DATA SCIENCE WORKS Can we anticipate what the customer

will want to do next?

Page 13: 7 Dimensions of Agile Analytics by Ken Collier

THE “DATA SCIENTIST”

Machine Learning Statistical Modeling

Artificial Neural Networks

Decision Tree Learning

Support Vector Machines

Unsupervised Learning

…and many more…

Bayesian Classification

Monte Carlo Simulation

Logistic Regression

K-Nearest Neighbor

…and many more…

Feature Engineering

Feature Extraction

Dimension Reduction

Domain expertise

Programming Skills

Functional Programming

Data “Wrangling”

Map/Reduce, SQL, & NoSQL

Page 14: 7 Dimensions of Agile Analytics by Ken Collier

Objective Truth

Discoverable Truth

Uninterpretable

Irrelevant Noise

Not Actionable

Actionable Signals

MAKING

“BIG DATA”

INTO

“LITTLE DATA”

Page 15: 7 Dimensions of Agile Analytics by Ken Collier

Advanced Analytics

Data Science

Visual Storytelling

Machine Learning

Statistics Agile Analytics

Solutions Thinking

Ethics

Agile Delivery Lean

Learning

Impact

Data Engineering

Volume Velocity

Variety

Adaptive Architecture

Streaming Data

Polyglot Persistence

Page 16: 7 Dimensions of Agile Analytics by Ken Collier

Advanced Analytics

Data Science

Visual Storytelling

Machine Learning

Statistics Agile Analytics

Solutions Thinking

Ethics

Agile Delivery Lean

Learning

Impact

Continuous Integration

Collaboration Evolve

Continuous Delivery

Hypothesis

Build Learn

Measure

Data Reduction

Data Engineering

Volume Velocity

Variety

Adaptive Architecture Streaming

Data Polyglot

Persistence

Page 17: 7 Dimensions of Agile Analytics by Ken Collier

drones.pitchinteractive.com

Data Visualization

Page 18: 7 Dimensions of Agile Analytics by Ken Collier

drones.pitchinteractive.com

Page 19: 7 Dimensions of Agile Analytics by Ken Collier
Page 20: 7 Dimensions of Agile Analytics by Ken Collier
Page 21: 7 Dimensions of Agile Analytics by Ken Collier

Advanced Analytics

Agile Analytics

Solutions Thinking

Ethics

Agile Delivery Lean

Learning

Impact

Hypothesis

Build Learn

Measure

Data Engineering

Adaptive Architecture

Streaming Data

Polyglot Persistence

Data Science

Machine Learning

Statistics

Visual Storytelling

Page 22: 7 Dimensions of Agile Analytics by Ken Collier

Typical Timeline

3-6 months 1-2 months 2-4 months

22

Data Convergence Analytical Divergence

Discover

Harvest

Filter

Integrate Augment Analyze

Act

Analytical Opportunities

CONVENTIONAL DATA SCIENCE If we knew X, we could do Y

Page 23: 7 Dimensions of Agile Analytics by Ken Collier

Analytical Divergence

Analytical Opportunities If we knew X, we could do Y

Data Convergence

Discover

Harvest

Filter

Integrate Augment Analyze

Act

Repeat this cycle solving small problems every few days

LEARN

MEASURE

BUILD

LEAN DATA SCIENCE

Page 24: 7 Dimensions of Agile Analytics by Ken Collier

Advanced Analytics

Agile Analytics

Solutions Thinking

Ethics

Agile Delivery

Impact

Reflect & Improve

Collaborate Evolve

Deliver

Data Engineering

Adaptive Architecture

Streaming Data

Polyglot Persistence

Data Science

Machine Learning

Statistics

Visual Storytelling

Lean Learning

Hypothesis

Build Learn

Measure

Page 25: 7 Dimensions of Agile Analytics by Ken Collier

Retain high value customers

Problem solved or continue?

High value business goal

What’s the smallest, simplest thing we can do?

Is it useful & actionable? Repeat!

What leads to customers leaving?

LIKE THIS EXAMPLE… Common features of

defectors?

Shopping behaviors of defectors?

What do defectors say about us?

Customers’ sentiment before defecting?

What encourages customers to stay?

Do incentives reduce defection rates?

Page 26: 7 Dimensions of Agile Analytics by Ken Collier

Retain high value customers

High value business goal

Like this example…

Page 27: 7 Dimensions of Agile Analytics by Ken Collier

What’s the smallest, simplest thing we can do?

Retain high value customers

Like this example… Common features of

defectors?

Page 28: 7 Dimensions of Agile Analytics by Ken Collier

Is it useful & actionable?

Retain high value customers

Like this example… Common features of

defectors?

Page 29: 7 Dimensions of Agile Analytics by Ken Collier

Repeat! Retain high value customers

Like this example… Common features of

defectors?

Shopping behaviors of defectors?

Page 30: 7 Dimensions of Agile Analytics by Ken Collier

Retain high value customers

Like this example… Common features of

defectors?

What leads to customers leaving?

Shopping behaviors of defectors?

What do defectors say about us?

Customers’ sentiment before defecting?

What encourages customers to stay?

Do incentives reduce defection rates?

Page 31: 7 Dimensions of Agile Analytics by Ken Collier

Problem solved or continue?

What leads to customers leaving?

Like this example… Common features of

defectors?

Shopping behaviors of defectors?

What do defectors say about us?

Customers’ sentiment before defecting?

What encourages customers to stay?

Do incentives reduce defection rates?

Page 32: 7 Dimensions of Agile Analytics by Ken Collier

Advanced Analytics

Data Science

Visual Storytelling

Machine Learning

Statistics Agile Analytics

Solutions Thinking

Ethics

Agile Delivery Lean

Learning

Impact

Hypothesis

Build Learn

Measure

Insight

Knowledge

Action

Disruption

Data Engineering

Volume Velocity

Variety

Adaptive Architecture Streaming

Data Polyglot

Persistence

Reflect & Improve

Collaborate Evolve

Deliver

Page 33: 7 Dimensions of Agile Analytics by Ken Collier

Advanced Analytics

Data Science

Visual Storytelling

Machine Learning

Statistics Agile Analytics

Solutions Thinking

Ethics

Agile Delivery Lean

Learning

Impact

Hypothesis

Build Learn

Measure

Insight

Knowledge

Action

Disruption

Business First

Evolve the Platform

Monitor & Measure

Data Engineering

Adaptive Architecture Streaming

Data Polyglot

Persistence

Reflect & Improve

Collaborate Evolve

Deliver

Page 34: 7 Dimensions of Agile Analytics by Ken Collier

QUESTIONS FIRST, DATA SECOND

34

“We built a platform for self service, and now we’re trying to get the business to use it.”

From this… …to this

Page 35: 7 Dimensions of Agile Analytics by Ken Collier
Page 36: 7 Dimensions of Agile Analytics by Ken Collier
Page 37: 7 Dimensions of Agile Analytics by Ken Collier

Advanced Analytics

Data Science

Visual Storytelling

Machine Learning

Statistics Agile Analytics

Solutions Thinking

Ethics

Agile Delivery Lean

Learning

Impact

Hypothesis

Build Learn

Measure

Insight

Knowledge

Action

Disruption

Business First

Evolve the Platform

Monitor & Measure

Privacy Controls Radical Transparency

Data Democracy

Open Data

Data Engineering

Volume Velocity

Variety

Adaptive Architecture Streaming

Data Polyglot

Persistence

Reflect & Improve

Collaborate Evolve

Deliver

Page 38: 7 Dimensions of Agile Analytics by Ken Collier

38 http://bit.ly/DataAndPrivacy

Jonny Leroy, NA Head of Technology, ThoughtWorks

Page 39: 7 Dimensions of Agile Analytics by Ken Collier

39

Page 40: 7 Dimensions of Agile Analytics by Ken Collier

40

Page 41: 7 Dimensions of Agile Analytics by Ken Collier

41

Page 42: 7 Dimensions of Agile Analytics by Ken Collier

42

Page 43: 7 Dimensions of Agile Analytics by Ken Collier

43

Page 44: 7 Dimensions of Agile Analytics by Ken Collier

44

Page 45: 7 Dimensions of Agile Analytics by Ken Collier

45

Page 46: 7 Dimensions of Agile Analytics by Ken Collier

46

PRIVACY IS DEAD

DON’T BE EVIL

Page 47: 7 Dimensions of Agile Analytics by Ken Collier

47

Page 48: 7 Dimensions of Agile Analytics by Ken Collier

48

Page 49: 7 Dimensions of Agile Analytics by Ken Collier

49

Page 50: 7 Dimensions of Agile Analytics by Ken Collier

50

Page 51: 7 Dimensions of Agile Analytics by Ken Collier

51

Page 52: 7 Dimensions of Agile Analytics by Ken Collier

52

Page 53: 7 Dimensions of Agile Analytics by Ken Collier
Page 54: 7 Dimensions of Agile Analytics by Ken Collier
Page 55: 7 Dimensions of Agile Analytics by Ken Collier

Advanced Analytics

Data Science

Visual Storytelling

Machine Learning

Statistics Agile Analytics

Solutions Thinking

Ethics

Agile Delivery Lean

Learning

Impact

Hypothesis

Build Learn

Measure

Insight

Knowledge

Action

Disruption

Business First

Evolve the Platform

Monitor & Measure

Privacy Controls Radical Transparency

Data Democracy

Open Data

Data Engineering

Volume Velocity

Variety

Adaptive Architecture Streaming

Data Polyglot

Persistence

Reflect & Improve

Collaborate Evolve

Deliver

Page 56: 7 Dimensions of Agile Analytics by Ken Collier

Ken Collier, Director, Agile Analytics [email protected]

Value Creation

Cool New Technologies +

Sophisticated Analytics +

Lean Learning Principals +

Fast Agile Delivery =