Interesting ways Big Data is used today

Daniel Sarbe May 2015, Big Data Romanian Tour - Timisoara

Agenda

1. Source of (Big)Data2. Why now?3. Interesting patterns of using BigData4. BigData – Big Opportunities

“There is a big data revolution.

But it is not the quantity of data that is revolutionary.The big data revolution is that now we can do something with the data.”

Gary King, professor at Harvard University

“In God we Trust, all others bring data”William Edwards Deming - American statistician

“If we have data, let’s look at data. If all we have are opinions, let’s go with mine.”

Jim Barksdale, former Netscape CEO

Source of (Big) Data

The 3+1Vs of Big Data

Just how Big is the Big Data market?

Big Data Market Forecast, 2011-2020(in $ billion)

Source: Wikibon 2015

Interest in BigData

Why now?

1) So much data generated that we cannot store and analyze with conventional tools

Big Data in Aviation Industry

2) Companies realized the potential and started to invest money

Companies sources of data analysis

Source: captricity.com

Data Mining & Machine Learning

• Data Mining - The process of discovering meaningful correlations, patterns and trends by sifting through large amounts of data

• Machine Learning is the study of computer algorithms that improve automatically through experience▫ Supervised machine learning -  The program is “trained” on a pre-defined set

of “training examples”, which then facilitate its ability to reach an accurate conclusion when given new data.

▫ Unsupervised machine learning - The program is given a bunch of data and must find patterns and relationships therein.

BigData use-cases

Source: IBM

Analytics Maturity

Predictive Analysis

• Predictive analytics is an area of data mining that deals with extracting information from data and using it to predict trends and behavior patterns. 

• The accuracy and usability of results will depend greatly on the level of data analysis and the quality of assumptions

BigData used for predictions – 2012 US Election

The 2012 Election: A Big Win for Big Data• Statistician Nate Silver, gave Barack Obama

over a 90 percent chance of victory in the Electoral College. 

• Algorithm 538 name - number of electors in US

• In 2008 his mathematical model correctly called 49 out of 50 states, missing only Indiana (which went to Obama by 0.1%.) (John McCain vs Barack Obama)

• In 2012 Silver's model has correctly predicted 50 out of 50 states. 

• Incorporated hundreds of state-level polls into his analysis. Economic variables, demographics, electoral outcome, historical polls, economic data and party registration figures were also incorporated

• While some analysts might cherry-pick data

sources according to whether they were qualitatively "reliable" or "unbiased", Silver incorporated them all. Silver's model instead looked at trends over time

BigData used for predictions - 2014 Sochi Winter Olympics

•“Canada will enjoy their best Olympics ever, while the U.S. and host Russia will struggle." 

BigData used for predictions - 2014 Sochi Winter Olympics

• The analysts used publicly available data on all Winter Olympic Games from 1924 forward

• The model's inputs are Gross Domestic Product(GDP), year, if the country is communist or not, if the country is a host or not, population of that country, and its historical performances and medal counts in previous Olympics.

• All variables are given the same weight in the model• The medal count prediction is based on a linear regression model• The algorithm is based on historical data, and doesn’t necessarily reflect more

current information such as emerging stars, recent funding boosts, and an unexpectedly large addition of new events to the program.

• “Based on the above mentioned data and analysis, the analysts predict that Canadian athletes will grab the most medals and the United States will finish seventh. Germany, Norway, Austria, China and Russia will rank second to sixth respectively.”

Big Data used in other sports

Germany Uses Big Data to Crush Brazil in World Cup Semifinal• Forget about Moneyball - Germany has now used serious Big Data to win a World Cup

match. • Soccer, a more fluid game, was thought to be less amenable than baseball to Big

Data's wiles.• According to assistant coach Hansi Flick, team managers combed through years of

research about the Brazilian team compiled by students at Cologne's Sports University, looking for any advantage to be gained over the Brazilian team.

• The compiled information included a detailed analysis of all Brazil's players--their favorite moves, how they deal with high pressure scenarios, their reactions when fouled, and even how they sprint for the ball.

•3) Cost of cloud/hardware and full-grown of software solutions (Hadoop ecosystem)

Cost per GB of disk

Hadoop – The platform for BigData

• Hadoop became a very stable and mature platform (and faster)

Hadoop 1.0 to 2.0

Hadoop myths debunked

Hadoop isn’t enterprise ready

Hadoop isn’t stable, cluster go down

You lose data on HDFSData cannot be shared across

the organizationHadoop is not secured

NameNode do not scaleSoftware upgrades are rareHadoop use cases are limitedI need expensive servers to

get moreHadoop is so dead

Source: Sumeet Singh - Yahoo

Cost of BigData vs Traditional DBs

Hadoop Providers

◦ 1. Cloudera - $4B market value

◦ - 1,000+ paying customers

◦ 2. Hortonworks - $1B market value - 800+ paying customers

◦◦ 3. MapR - $1B

market value◦ - 700+ paying

customers

Open Data Platform

The Open Data Platform Initiative (ODP) is a shared industry effort focused on promoting and advancing the state of Apache Hadoop® and Big Data technologies for the enterprise.

Other (interesting) Big Data real use-cases

Netflix

Netflix collects a lot of data to understand how its users behave and what their preferences are• It collects metrics including what people watch, when they watch, where

they watch, what devices they use, ratings, searches, when users pause or stop watching, etc.

• Netflix made the House of Cards decision by identifying that subscribers who watched the original British version of House of Cards were very likely to watch movies starring Kevin Spacey or directed by David Fincher

• Netflix made ten different versions of the trailer for House of Cards geared towards different audiences▫ Fans of Kevin Spacey watched trailers that were focused on him while people

who liked female-oriented movies saw trailers that highlighted the women in the show.

Verizon

• 103.3 million wireless customers, 6.2 million Internet users and 5.3 million TV subscribers.

• Data collected:▫ Calls(order flowers) or accessing some pages▫ Locations in City + Roaming▫ Home + Mobile web pages + Television

• Formed a Precision Marketing division – e.g. Event attendance information▫ Migrate from iPhone 5 to iPhone 6 – resulted in a plan data increase or not? ▫ Some migrated from Android to iPhone and huge data plan consumtion 3x-5x more

Notes:• Customers can choose not to participate in the program by going to their privacy choices page on

MyVerizon or by calling 866-211-0874• Verizon’s business and government customers are not part of the Precision program

The Perfect Milk - Digital Cow - The internet of cows

• Embaded sensos in cow stomachs • If cow is seek, sensor will let a veterinar know while there is

time to treat the disease• Sensor to detect the presence of E.coli bacteria• Vital Herd, a Texas-based start-up - e-Pill - collect information

about the animal: breathing rate, heart rate, temperature, rumination time, rumen acidity and estrogen levels

The City of Las Vegas

• archaic records and inaccurate information

• took advantage of smart data to develop a living model of its utilities network

• aggregate data from various sources into a single real-time 3D model created with Autodesk technology for both avove and below ground utilities

Google Flu Trends

BigData – Big Opportunities

• Big data means big IT job opportunities -- for the right people

Big Opportunities

• Gartner predicted in 2013 that by 2015, Big Data demand will generate 4.4 million jobs in the IT Industry all around the world.

• 1.9 million IT jobs will be created just in the U.S. That is how Big Data directly affects the IT Industry.

• Only 1/3rd of these jobs will be fulfilled, due to lack of skills in the individuals

What is needed?• A Curious Mind Is Key - The most important qualifications for these positions

aren't academic degrees, certifications, job experience or titles. Rather, they seem to be soft skills: a curious mind, the ability to communicate with nontechnical people, a persistent -- even stubborn -- character and a strong creative bent.

• The CIA is hiring data scientists : “We are looking for curious, creative individuals interested in serving their country through the field of data science.”

“I keep saying that the sexy job in the next 10 years will be statisticians, and I’m not kidding.”  

Hal Varian, chief economist at Google

“Without big data, you are blind and deaf and in the middle of a freeway.”

Geoffrey Moore, author and consultant

Thank you!Twitter: @danielsarbe