Making your Apps Smarter with Azure HDInsightMatt Winkler, @mwinklePrincipal Program Manager3-529

What is HDInsight?Programming HadoopIntegrating with your apps

Agenda

Elastic – what you want, when you want itSimple – 3 clicks, 10 minutes to HadoopSecure – isolated, secured by default

Big data processing on top of Azure Storage

On demand Apache Hadoop Clusters

Windows Azure HDInsight

Demo: HDInsight 0 – 60

Storage + proliferation of compute models for data processing at scaleBegan life as an open source implementation of Google’s Map/Reduce and GFS papersIn use at many major web companies at massive scale (1000’s of node, PB’s of storage)

“Hadoop is a distributed system for counting words.”

scalding readme

Hadoop there it is

Drill

Pig

Map/Reduce

Oozie

Sqoop

MahoutPegasus

Storm

Scalding

Cascading

Hive

Scoobi

Cloud as global aggregation point for sensors & devicesData born in the cloud in apps and servicesIn your app, what data are you not collecting? Why? What could you do with it?

“we should stop acting as if our goal is to author extremely elegant theories, and instead embrace complexity and make use of the best ally we have: the unreasonable effectiveness of data. ”

Peter Norvig, et al

But I don’t have big data!

Acquire – Compute – Publish – Consume

Acquire Publish

Azure BlobStorage

Shape

QueryExperiment

Acquire Publish

Azure BlobStorage

Shape

QueryExperiment

Consume

Authoring Jobs App Integration

Core Hadoop

Consistent REST API’s

Breadth of Clients (Java, JS, .NET, etc)

Authoring frameworks and languages

End User Tooling (IDE’s, Analyst tools, Command lines)

ConnectivityProgrammabilitySecurityLoosely coupled

LightweightLow cost to extendScenario oriented

Innovation flows upward

New compute modelsPerf enhancements

Extend breadth & depthEnable new scenariosIntegrate with current tool chains

App/Services writing to BlobData available via servicesPush to blob (on-prem => cloud)Copy To Blob

Acquire

No one tool to rule them all

Preprocessing – Cleansing / Shaping / EnrichingTraditional Analytics – Query Advanced Analytics – Experiments / Models

SDK & Command line tools for submitting and managing these jobs

Compute

Back to BlobsInto reporting engineInto “online” store – SQL, Mongo, etcAs another table

Publish

Data ExplorerODBC/JDBCREST API’sDirect from Blob StorageAdditional Hadoop jobs

Consume Results

Scheduled executionMore sophisticated analysisAnalyze impact of changesIncorporate other data sources

What Next?

HDInsight StatusAzure Public PreviewAvailable in 2 Azure RegionsDiscounted compute pricingSDK & CLI tools available

HDInsight, What’s Next? Committed SLAGlobally availableContinuing to improve Hadoop (including Stinger)Continued Investments in Cluster Configuration, Management, Integration and Developer tooling

Resourcesbig data on azurehdinsight getting startedhadoop sdk for .net (includes PowerShell for now) node.js sdk x-plat cli

Evaluate this session

Scan this QR code to evaluate this session and be automatically entered in a drawing to win a prize!

© 2013 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.