why big data will survive the hype - and change the way we work

Why Big Data Will Survive the Hypeand Change the Way We Work

(and live) (and think)

The City University of New York

New York CityDecember 4, 2013

@BlairReeves

Blair ReevesProduct Lead, IBM Digital Analytics

IBM.com/digitalmarketing

I live here:Durham, North Carolina

@BlairReeves

@BlairReeves

“The Year of Big Data”

… is every year. From now on.Credit: Gartner Research

@BlairReeves

The Value of Data is Increasing

@BlairReeves

The Value of Data… is still being decided

Book Value: $13 billion

Market Value: $114 billion= 1.3 billion MAUs ~500 terabytes of data added… per day

$101 billion in data

@BlairReeves

A Short History of Data

Great Library of Alexandria (Egypt)

300 B.C.

Al-Azhar University (Egypt)

Invention of the Gutenberg printing press

Copyright law is born

Press freedom guarantees; pamphleteering

Cambridge University owns 122 books

Martin Luther translates the Latin Bible, accelerating mass literacy

Watson

970 A.D.

1400

1450s

1520s

17101770s

Herman Hollerith invents machine-readable data for U.S. Census

~2.8 billion global internet users(40% of world’s population)

ARPANET – first TCP/IP Protocol

1969

1890

2013

@BlairReeves

The Way We Use Data Will Change

Trade Exactitude for Size

Why Sample?

Correlation Over Causality

@BlairReeves

1 – Trade Exactitude for Size

Precision < Size

More data > Better algorithms

@BlairReeves

250 word pairs

1954 1990

3 million word pairs2006

>100 billion word pairs(and counting)

1 – Trade Exactitude for Size

@BlairReeves

2 – Why Sample?

• Sampling relies on randomness

• Difficult to drill down into subcategories

• Requires careful pre-planning

@BlairReeves

2 – Why Sample?

• Sumo wrestlers• Google Flu• Non-linear

relationships (social media)

@BlairReeves

3 – Correlation Over Causality

When does knowing “why” matter?

Data rather than hypotheses

Correlations are value

@BlairReeves

3 – Correlation Over Causality

A/B Testing Attribution

@BlairReeves

“Everything is obvious once you know the answer.”

- Duncan Watts

@BlairReeves

IBMBigDataHub.com

IBM.com/digitalmarketing

Thanks!BReeves@us.ibm.com