big data velocity: leveraging high speed event streams

Neil McGovern, SAP Product Marketing, Real-Time Data Platform

Streaming Analytics

© 2012 SAP AG. All rights reserved. 2

Speaker Introduction

Neil McGovern is a Senior Director of Marketing

at SAP responsible the SAP Real-Time Data

Platform which is based on the flagship SAP

HANA .

Neil has over 25 years experience in Business

Enterprise Software and has a degree in

Computer Science and an MBA.


Big Data and Streaming Analytics

Big Data – Velocity

Genesis – Financial Services

Expansion of Streaming Analytics

Streaming Analytics Gets Smart

The Future is Going to be Amazing


Data Doubles

Every 18 months

Many Opportunities are

Short-lived

Information Is a Strategic

Corporate Asset

Big Data: Volume, VELOCITY, Variety

Web

Data Marts

Enterprise

Applications

Sensors

Smart Machines

Twitter

Data Warehouses

Databases

Events

Tracking


Why Real Time? Real-Time is a competitive advantage

Source: compare Hackathorn, 2002

Business

Value

Time

Business-relevant Event occurs

Event data

stored

Analysis information

delivered Action

taken




Business

Value

Time Data Analysis Action


Event data

stored


delivered Action

taken

Latency:




Business

Value

Time Data Analysis Action


Event data

stored


delivered Action

taken

Value

lost

through

latency

Latency:


NYSE – Before


NYSE – After


Sudden Explosion in Quote Data M

essages p

er

second

-

50,000

100,000

150,000

200,000

250,000

300,000

350,000

400,000

2000 2001 2002 2003 2004 2005 2006


Sudden Explosion in Quote Data

-

500,000

1,000,000

1,500,000

2,000,000

2,500,000

2000 2001 2002 2003 2004 2005 2006 2007 2008 2009

Messages p

er

second


Traditional Business Intelligence Architecture

Collect

Data

Data

Warehouse

ETL

ETL

ETL

ETL

Regularly

Consolidate

Query

& Analyze

Lost Time, Missed Opportunity

BI Tool


A New Paradigm – Event Stream Processing

Traditional Database Queries Continuous Queries

Step 1:

Store the data

Step 2:

Query the data Step 1:

Define the

continuous

queries and

the dataflow

Step 2:

Wait for data to

arrive. As it arrives,

it flows through the

continuous queries

to produce

immediate results


Big Data – Velocity Challenge

“ We're faced with analyzing data as it passes by

on a high-speed conveyor belt. We don't have the luxury of

structuring it and putting it into a database first, because

we want to be able to classify it within 2 to 3 seconds. ”

– Alex Philp, CEO, TerraEchos


The Challenge:

Extracting Insight from the flood of raw data


A Major Change in Telecommunications


Legacy Architecture

RAW CDRs Processed CDRs

Import

Transformation

Scripts

Queries

Data Warehouse


New Architecture

RAW CDRs

Summary Files

SAP Sybase

ESP

Queries

Queries

ALARMS /SNMP Traps

Import

Data Warehouse


Monitoring Networks

Business Challenges

BICS has rapidly growing data that needs to be analyzed in real-time to

deliver network status reports on demand to other telecommunication users

BICS wanted to add additional services to their standard reporting

Technical Challenges

Data could not be loaded into data warehouse quickly enough

Data growth (over 700 million records per day) had overwhelmed their data

warehouse, so reports were 45 minutes out-of-date

Peak rates of 10,000 data items per second

User queries were overwhelming data warehouse

Benefits

Greater efficiency, quality and speed of services to internal and external

clients through real-time reporting and shorter response time

Clear visualization of emergency signals and peak periods

Savings in machine power to generate improved ROI

700 Million

records/day and peak rates of

10,000 per second

Real-time

alerts now

available

On Time

Analytics

“Our ultimate choice was prompted just as much by the excellent guidance and support of the development team at the time and the

transparency of the documentation available, as it was by the short learning curve for the tool,”

Yves Beddegenoots

IT manager at BICS

“ ”

http://bics.com/


Businesses of All Types See Value in Faster Response to Changing Conditions

eCommerce firm: “I want to have a live dashboard that

shows me the top ten links that are attracting the most

clicks at all times”

BPM implementation: “I want to be alerted when average

response times are outside defined service goals”

Telecom provider: “I want to alert Customer Service when

an individual customer has just experienced their fourth

dropped call in a one-week period”

Online gaming operator: “I want to shut down accounts

immediately when they exhibit patterns indicating possible

fraud”

Spot emerging

threats or

opportunities

before it’s too

late

React to

changing

conditions

sooner

Make decision

based on more

timely

information


Public safety: alerting authorities when a bridge is unsafe

The problem:

• Immediately following an earthquake,

identify unsafe bridges

The solution:

• NTT Data Corporation BRIMOS bridge

monitoring system

• Bridges equipped with > 10,000 optical

vibration sensors

• Monitor vibration data to detect abnormal

vibration indicating the bridge may be

unsafe

Result:

• immediately identify bridges exhibiting

excessive vibration


INTERNET OF THINGS Enabling Next Generation Applications

Overview

The Internet of Things (IoT) is a concept that describes the increase in

devices that are attached to the internet.

The “Internet of Things” is the next generation of “Big Data” and the hype is

growing

Internet of Things is a GREAT fit for the SAP RTDP

There are expected to be 25 billion devices connected to the Internet by

2015, and 50 billion devices by 2020

Technical Challenges

Managing the “things”

Dealing with the massive amounts of data

Analyzing data from the “things” in real-time

Historical analysis

Benefits

Single platform for management, real-time analytics and historical analytics

Scalability and performance to be able to deal with the

Massive data

sets including

structured and

unstructured data

Billions of data sources

(“things”)

Complex historical analysis

big data velocity: leveraging high speed event streams

Documents