big data at cme group: challenges and opportunities

25
Big Data at CME Group: Challenges and Opportunities Rick Fath & Slim Baltagi 9/18/2012

Upload: slim-baltagi

Post on 15-Jul-2015

287 views

Category:

Data & Analytics


1 download

TRANSCRIPT

Big Data at CME Group: Challenges and Opportunities

Rick Fath & Slim Baltagi 9/18/2012

© 2012 CME Group. All rights reserved

Agenda

1. CME Group Overview

2. Big Data at CME Group

3. Big Data Use Cases at CME Group

4. Big Data Challenges and Opportunities at CME Group

5. Big Data Key Learning’s at CME Group

6. Questions

1. CME Group Overview

3

© 2012 CME Group. All rights reserved 4

CME Group Overview

• #1 futures exchange in the

U.S. and globally by 2011

volume

• 2011 Revenue of $3.3bn

• 11.8 million contracts per

day

• Strong record of growth,

both organically and

through acquisitions

– Dow Jones Indexes

– S&P Indexes

– BM&FBOVESPA

– CME Clearing Europe

– CME Europe Ltd

Combination is greater

than the sum of its parts

© 2012 CME Group. All rights reserved 5

Forging Partnerships to Expand Distribution,

Build 24-Hour Liquidity, and Add New Customers

CBOT Black

Sea Wheat

Partnerships include:

• Equity investments

• Trade matching services

• Joint product development

• Order routing linkages

• Product licensing

• Joint marketing

• European clearing services

• Developing capabilities globally

• Expanding upon global benchmark products

• Positioned well within key strategic closed markets

•Recently announced application to

FSA for CME Europe Ltd. –

expected launch mid-2013

Big Data: Transactions + Interactions + Observations

2. Big Data at CME Group

© 2012 CME Group. All rights reserved

When you Data gets too Big!!

7

© 2012 CME Group. All rights reserved

Big Data and CME Group: a natural fit

Definition (Strategy) of Big Data at CME Group

• CME Group is a Big Data factory! 11 million

contracts a day…

• Data is not merely a byproduct, but an

essential deliverable

• Daily market data generated by the exchange

and also historical data

• Growing partnerships around the world

• Meeting new regulatory requirements (CFTC,

SEC)

• Trading shift from floor to electronic

• Algorithmic trading improvements-high

volumes, low latency, and historical trends

© 2012 CME Group. All rights reserved

Solving Big Data at CME Group

• Leverage legacy tools vs. adoption of new Big Data technologies

• Transactional data (trades, market data, order data, etc.)

• Missed opportunities with existing Legacy Big Data applications:

Not all data is being stored and quality is lacking. Learn from this

lesson

• Raw data being sold without analytics

• Risk and Maturity of new Big Data Technologies

• Quality of Historical data: different protocol, format....

9

3. Big Data Use Cases at CME Group

10

© 2012 CME Group. All rights reserved

Use Case 1: CME Group's BI solution

Description: Provide complex reporting and data analysis for internal Business

teams.

Challenges:

• Performance Needed: Reports and Analysis are time sensitive. (Existing queries

taking 18+ hours)

• Established tools and partnerships (Informatica, Business Objects, and

established business users)

• Mission critical queries, timely analytics, complex aggregation required

• Risk adverse business customers

11

© 2012 CME Group. All rights reserved

Use Case 1: CME Group's BI solution

Opportunities:

• Structured RDBMS

• Minimize integration impact

• Enhance customer analytics

• Deliver faster time to market of new Market Regulation requirements

• Improve and increase fact based decision making

• Reduce batch processes execution duration

Solution:

12

© 2012 CME Group. All rights reserved

Why we chose a Data Warehouse Appliance?

• No need to re-engineer the existing Oracle Database applications

• Based on CME evaluation of Data Warehouse Appliances, Exadata is

best suited to CME environment

• DWA performs consistently better than our DB production environment.

• Product maturity

• Faster queries compared to Oracle DB

13

© 2012 CME Group. All rights reserved

Description: Reduce reliance on SAN while increasing query performance.

Challenges:

• Expensive storage cost.

• Oracle queries cannot keep up with inserts and do not meet SLAs

Opportunities:

• Reduce storage cost with non specialized hardware. “not commodity”

• Fast Parallel queries

Solution:

14

Use Case 2: CME Group’s Rapid Data Platform

© 2012 CME Group. All rights reserved

Description: Provide historical Market Data to exchange customers who want to

understand market trends, conduct market analysis, and build\test trading

algorithms.

Challenges:

• Expensive storage cost.

• Legacy downstream application with limited support

• Historical data from acquisitions and mergers…”Don’t look for problems

because you will find them”

• 100TB data and growing….

• Data Redundancy and Quality (limited awareness)

• Business users dependent on Technology staff to use the data.

15

Use Case 3: CME Group’s Historical Data Platform

© 2012 CME Group. All rights reserved

Opportunities:

• Reduce storage cost with non specialized hardware. “not

commodity”

• Reduce duplication of data (derive data on demand fast)

• Improve Data Quality with better interrogation

• Grow the business (new datasets, mash-ups, analytics)

• Separate Delivery from Storage: Store everything, deliver what you

need.

• Enable Business users (Ad Hoc queries, define new datasets)

• Reduce TCO (support, improve reliability)

Solution:

16

Use Case 3: CME Group’s Historical Data Platform

© 2012 CME Group. All rights reserved

Why we chose Hadoop?

• Solution built for scale and growth

• Structure of data doesn’t matter. Data is the Data.

• Hadoop as ETL solution

• Performance POC with Oracle – Beat Oracle queries and

removes duplication

• Decouple Storage and Delivery. Store raw data, deliver

enhanced data.

• Ecosystem reduces development time with proven solutions to

common problems

• Structure is fluid, perfect for historical data

17

4. Big Data Challenges and Opportunities at CME Group

18

© 2012 CME Group. All rights reserved

Enterprise Challenges

• Open Source Paradigm

• Enterprise Maturity- built for failure

• Operations Readiness

• Framework can mask problems

• Educate the enterprise about Hadoop

• Evolution of Hadoop

19

© 2012 CME Group. All rights reserved

Deployment Challenges • Selection of Hadoop distribution

• Growing cluster equals growing support fees. How to lower

maintenance fees

• Future Project candidates (super cluster vs. individual clusters)

• Distributions support and alignment

• Maximize investment in IT assets (Informatica, BusinessObjects,

ExaData)

• DR and Backup Solutions

20

© 2012 CME Group. All rights reserved

Potential Hadoop Use Cases for Future

21

• Compliance and regulatory reporting

• Trade surveillance

• Abnormal trading pattern analysis

• Integrate disparate datasets and unlock the value at the intersection

of data.

• Identify new Big Data projects as a new source of Revenue

• Reduce enterprise data redundancy (ETL, storage, analytics),

• Use as alternative for costly CEP solutions where applicable

5. Big Data Key Learning’s at CME Group

22

© 2012 CME Group. All rights reserved

What we Learned

• Embrace open source- “We are not the first ones to solve this problem!!”

• Enterprise readiness: DR, Backup, Security, HA • Solving operational maturity (server to admin ratio) • Leverage existing IT investments • Adapt standards to make cluster more supportable • Tackling the learning curve: keep close to community • Leverage the Hadoop Ecosystem… • Capture everything (More capture, support for structured and semi-

structured • Capture data first and then figure out new opportunities (schema-

less) • Hadoop can be complimentary to existing IT assets.

23

6. Questions

24

© 2012 CME Group. All rights reserved 25

Futures trading is not suitable for all investors, and involves the risk of loss. Futures are a

leveraged investment, and because only a percentage of a contract’s value is required to trade,

it is possible to lose more than the amount of money deposited for a futures position. Therefore,

traders should only use funds that they can afford to lose without affecting their lifestyles. And

only a portion of those funds should be devoted to any one trade because they cannot expect to

profit on every trade.

The Globe Logo, CME®, Chicago Mercantile Exchange®, and Globex® are trademarks of

Chicago Mercantile Exchange Inc. CBOT® and the Chicago Board of Trade® are trademarks of

the Board of Trade of the City of Chicago. NYMEX, New York Mercantile Exchange, and

ClearPort are trademarks of New York Mercantile Exchange, Inc. COMEX is a trademark of

Commodity Exchange, Inc. CME Group is a trademark of CME Group Inc. All other trademarks

are the property of their respective owners.

The information within this presentation has been compiled by CME Group for general purposes

only. CME Group assumes no responsibility for any errors or omissions. Although every attempt

has been made to ensure the accuracy of the information within this presentation, CME Group

assumes no responsibility for any errors or omissions. Additionally, all examples in this

presentation are hypothetical situations, used for explanation purposes only, and should not be

considered investment advice or the results of actual market experience.

All matters pertaining to rules and specifications herein are made subject to and are superseded

by official CME, CBOT, NYMEX and CME Group rules. Current rules should be consulted in all

cases concerning contract specifications.