the economics of sql on hadoop

© 2013 Datameer, Inc. All rights reserved.

The Economics of SQL on Hadoop

Watch the Recording of this Webinar

View the entire recorded webinar at:

http://info.datameer.com/Slideshare-Economics-SQL-Hadoop.html

About our Speakers

John Myers !John Myers joined Enterprise Management Associates in 2011 as senior analyst of the business intelligence (BI) practice area. John has 10+ years of experience working in areas related to business analytics in professional services consulting and product development roles, as well as helping organizations solve their business analytics problems, whether they relate to operational platforms, such as customer care or billing, or applied analytical applications, such as revenue assurance or fraud management. !

Slide 3 © 2013 Datameer, Inc. All rights reserved.

About our Speakers Stefan Groschupf!!▪  Stefan Groschupf is the co-founder and CEO of

Datameer. He is one of the original contributors to Nutch, the open source predecessor of Hadoop, Stefan has been at the forefront of the Hadoop and Big Data market.�Prior to Datameer, Stefan was the co-founder and CEO of Scale Unlimited, which implemented custom Hadoop analytic solutions for HP, Sun, Deutsche Telekom, Nokia and others. Earlier, Stefan was CEO of 101Tec, a supplier of Hadoop and Nutch-based search and text classification software to industry-leading companies such as Apple, DHL and EMI Music. Stefan has also served as CTO at multiple companies, including Sproose, a social search engine company.

Slide 4 © 2013 Datameer, Inc. All rights reserved.

About our Speakers

Matt Schumpert!!Matt has been working in enterprise software of over 10 years in various capacities, including sales engineering, strategic alliances and consulting.  !!Matt currently runs the pre-sales engineering team at Datameer, supporting all technical aspects of customer engagement through roll-out of customers into production. ! !Matt holds a BS in Computer Science from the University of Virginia.!

Slide 5 © 2013 Datameer, Inc. All rights reserved.

Agenda ▪  EMA on Current State of the Big Data Industry!

–  Online Archiving in Practice!–  SQL on NoSQL: Metadata!–  Exploratory Use Cases!–  Late Binding Schemas better for Discovery!–  Economics of Hadoop!

▪  Datameer on how to solve these problems!–  Use Case #1: Semi-Structured Data !–  Use Case #2: Text Analytics data!–  Use Case #3: Path Analysis!

▪  Takeaways; and Question and Answer!

Slide 6 © 2013 Datameer, Inc. All rights reserved.

© 2013 Datameer, Inc. All rights reserved.

State of Big Data Industry

Online Archiving is the majority use case for Big Data projects

© 2013Enterprise Management Associates, Inc. Slide 8

Moving Beyond select * from tablename SQL requires a managed set of metadata

© 2013Enterprise Management Associates, Inc. Slide 9

Big Data Platforms have Multiple Uses: Discovery is a significant portion

© 2013Enterprise Management Associates, Inc. Slide 10

Late Binding Schemas are good for Discovery

© 2013Enterprise Management Associates, Inc. Slide 11

Free as a Free puppy…

Slide 12 © 2013 Enterprise Management Associates, Inc.

© 2013 Datameer, Inc. All rights reserved.

Datameer Demos

Use Case #1: Semi-Structured Data ▪ Noisy, log-structured data à signal

Slide 14 © 2013 Datameer, Inc. All rights reserved.

Use Case #1: Semi-Structured Data ▪ Noisy, log-structured data à signal ▪ Extract, cast, & define fields on demand

Slide 15 © 2013 Datameer, Inc. All rights reserved.

Use Case #1: Semi-Structured Data ▪ Noisy, log-structured data à signal ▪ Extract, cast, & define fields on demand ▪ Painful/impossible without inspection

Slide 16 © 2013 Datameer, Inc. All rights reserved.

Use Case #1: Semi-Structured Data ▪ Noisy, log-structured data à signal ▪ Extract, cast, & define fields on demand ▪ Painful/impossible without inspection ▪ “One-offs” are possible with SQL+UDFs ▪ But better to collaborate with shared “views”

Slide 17 © 2013 Datameer, Inc. All rights reserved.

Use Case #1: Semi-Structured Data ▪ Noisy, log-structured data à signal ▪ Extract, cast, & define fields on demand ▪ Painful/impossible without inspection ▪ “One-offs” are possible with SQL+UDFs ▪ But better to collaborate with shared “views”

▪ Examples: ▪  “User-agent” string ▪ URL Parameters ▪  JSON

Slide 18 © 2013 Datameer, Inc. All rights reserved.

Use Case #2: Text Analytics ▪ Few/no known fields

Slide 19 © 2013 Datameer, Inc. All rights reserved.

Use Case #2: Text Analytics ▪ Few/no known fields ▪ Notion of a record is nebulous / fluid

Slide 20 © 2013 Datameer, Inc. All rights reserved.

Use Case #2: Text Analytics ▪ Few/no known fields ▪ Notion of a record is nebulous / fluid ▪ Wrangling and mining

Slide 21 © 2013 Datameer, Inc. All rights reserved.

Use Case #2: Text Analytics ▪ Few/no known fields ▪ Notion of a record is nebulous / fluid ▪ Wrangling and mining ▪ “Bag-of-Words” is a sensible start

Slide 22 © 2013 Datameer, Inc. All rights reserved.

Use Case #2: Text Analytics ▪ Few/no known fields ▪ Notion of a record is nebulous / fluid ▪ Wrangling and mining ▪ “Bag-of-Words” is a sensible start ▪ Again, frequent inspection is key

Slide 23 © 2013 Datameer, Inc. All rights reserved.

Use Case #3: Path Analysis ▪ Key component of clickstream analysis

Slide 24 © 2013 Datameer, Inc. All rights reserved.

Use Case #3: Path Analysis ▪ Key component of clickstream analysis ▪ Compares each record to the next/previous

Slide 25 © 2013 Datameer, Inc. All rights reserved.

Use Case #3: Path Analysis ▪ Key component of clickstream analysis ▪ Compares each record to the next/previous ▪ Defines/summarizes transitions, not events

Slide 26 © 2013 Datameer, Inc. All rights reserved.

Use Case #3: Path Analysis ▪ Key component of clickstream analysis ▪ Compares each record to the next/previous ▪ Defines/summarizes transitions, not events ▪ Supported by list/array types

Slide 27 © 2013 Datameer, Inc. All rights reserved.

Use Case #3: Path Analysis ▪ Key component of clickstream analysis ▪ Compares each record to the next/previous ▪ Defines/summarizes transitions, not events ▪ Supported by list/array types ▪ Requires multi-pass queries

Slide 28 © 2013 Datameer, Inc. All rights reserved.

© 2013 Datameer, Inc. All rights reserved.

Takeaways

When NOT to use SQL on Hadoop ▪ Structured Schemas

or “Schema on Write”

Slide 30 © 2013 Datameer, Inc. All rights reserved.

When NOT to use SQL on Hadoop ▪ Structured Schemas

or “Schema on Write” ▪  “Realtime” Query

SLAs for operational or reporting tasks

Slide 31 © 2013 Datameer, Inc. All rights reserved.

When NOT to use SQL on Hadoop ▪ Structured Schemas

or “Schema on Write” ▪  “Realtime” Query

SLAs for operational or reporting tasks

▪ Highly detailed SQL query requirements (SQL-2003)

Slide 32 © 2013 Datameer, Inc. All rights reserved.

When to use SQL on Hadoop ▪ Unstructured

Datasets and “Schema on Read”

Slide 33 © 2013 Datameer, Inc. All rights reserved.

When to use SQL on Hadoop ▪ Unstructured

Datasets and “Schema on Read”

▪ Discovery tasks designed to find new connections and new business value

Slide 34 © 2013 Datameer, Inc. All rights reserved.

When to use SQL on Hadoop ▪ Unstructured

Datasets and “Schema on Read”

▪ Discovery tasks designed to find new connections and new business value

▪  Lower level SQL queries (SQL-99)

Slide 35 © 2013 Datameer, Inc. All rights reserved.

Summary ▪  EMA on Current State of the Big Data Industry

–  Online Archiving in Practice –  SQL on NoSQL: Metadata –  Exploratory Use Cases –  Late Binding Schemas better for Discovery

▪  Datameer on how to solve these problems –  Use Case #1: Semi-Structured Data –  Use Case #2: Text Analytics –  Use Case #3: Path Analysis

Slide 36 © 2013 Datameer, Inc. All rights reserved.

Call To Action ■ Visit our website

–  www.datameer.com

■ Download our Trial –  http://www.datameer.com/Datameer-trial.html

Slide 37 © 2013 Datameer, Inc. All rights reserved.