automating big data with the automic hadoop agent

Automic World 2015

Automating Big Data with the Hadoop Agent

Dave KellermannsChief Automation Architect


Every day, we create 2.5 quintillion (18 zeroes !) bytes of data

So much that 90% of the data in the world today has been created in the last two years alone. This data comes from everywhere: sensors used to gather climate information, posts to social media sites, digital pictures and videos, purchase transaction records, and cell phone GPS signals to name a few. This is called “Internet of the Things”. Connect all together. But the data is called

BIG DATA

What is Big Data ?

Source.Forbes.com


Think you can avoid Big Data?

The Big Data technology and services market represents a fast-growing multibillion-dollar worldwide opportunity [...] that will grow at a 26.4% compound annual growth rate to $41.5 billion through 2018, or about six times the growth

rate of the overall information technology market […]

IDC - 2015


• Make better, more quantitative decisions

• Reach new levels of profits, efficiently

• Predict with unprecedented accuracy to influence

business outcomes

• Deliver highly personalized customer experiences at

massive scale

• Make new discoveries using massive amounts of data

• Recognize new revenue streams from digital exhaust

Why are companies focused right now on Big Data ?


Where does Big Data fit into the Enterprise?


• Big data technologies must be integrated with more traditional data systems and sources

• Efficient Dev-Test-Prod change control needs to be implemented end-to-end

• Administration, development, operations, and analytics must all need tools tailored to their roles to maximize

• Automation is a core requirement for making these complex systems accessible. It has to be easy to use and customizable

Simplifying user experience and procedures


A conflict in the skillset of analysts vs data engineers

People running the data platform

<workflow-app xmlns="uri:workflow:0.4" name="hive-add-partition-searchevents-wf"> <start to="hive-add-partition-searchevents" /> <action name="hive-add-partition-searchevents" retry-max="1" retry-interval="1"> <hive xmlns="uri:oozie:hive-action:0.4"> <job-tracker>${jobTracker}</job-tracker> <name-node>${nameNode}</name-node> ... ... <script>add_partition_hive_searchevents_script.q</script> <param>YEAR=${YEAR}</param> <param>MONTH=${MONTH}</param> <param>DAY=${DAY}</param> <param>HOUR=${HOUR}</param> </hive> <ok to="end" /> <error to="fail" /> </action><bundle-app name='BundleApp-LoadAndIndexTopCustomerQueries' xmlns='uri:oozie:bundle:0.2'> <controls> <kick-off-time>${jobStart}</kick-off-time> </controls> <coordinator name='CoordApp-LoadCustomerQueries' > <app-path>${coordAppPathLoadCustomerQueries}</app-path> </coordinator>

<coordinator name='CoordApp-IndexTopQueriesES' > <app-path>${coordAppPathIndexTopQueriesES}</app-path> </coordinator></bundle-app>....<coordinator-app name="CoordApp-LoadCustomerQueries" frequency="${coord:days(1)}" start="${jobStart}" end="${jobEnd}" timezone="UTC" xmlns="uri:oozie:coordinator:0.2"> ... <action> <workflow> <app-path>${workflowRoot}/hive-action-load-customerqueries.xml </app-path> </workflow> </action></coordinator-app>...<coordinator-app name="CoordApp-IndexTopQueriesES" frequency="${coord:days(1)}" start="${jobStartIndex}" end="${jobEnd}" timezone="UTC" xmlns="uri:oozie:coordinator:0.2"> ... <action> <workflow>

Automic helps to bridge the gap between the skillsets of the people who need the tool and the skillsets required to run the tool

People wanting data


Hadoop Open Source

“The Apache™ Hadoop® project develops open-source software for reliable, scalable, distributed computing.”

“Open source as a development model promotes a universal access via a free license to a product's design or blueprint, and universal redistribution

of that design or blueprint, including subsequent improvements to it by anyone”


Many people work on Hadoop


3 Releases of the Hadoop Platform


New capabilities keep on coming


APIs do change constantly


© Automic. All rights reserved.

Configuration & Objects


Proven value for Data Automation

Improve Decisions

Business & Operational Intelligence

Data Warehousing

Big Data

Call centre performance

Hadoop Big Data

automation

Data Ingestion

across IaaS

Fast Cognos Analytics delivery

POS data mining, ETL

& MFT


Proven Value for Data Automation

Self-service platform for

data scientists

We use Automic in our data center to define dependencies between various jobs between our data center and the cloud, and run them as ‘process flows’.

Automic ensures that the right data is delivered on time to Data Scientists. This requires approximately 6,000 jobs per day.

Ashi ShethManger of Enterprise Services, Netflix


Business Benefit to NetflixTo “Give Viewers What They Want”

Collect hundreds of terabytes of data daily

Petabyte-scale

Platform Engineers

… build templates and workflows using ONE Automation

… enable data scientists to perform all kinds of ad hoc analysis without having to deal with the complexity of the underlying data infrastructure

Automic

1 2

• >50m subscribers• >40 countries

Recommendation EngineData Scientists… perform data-driven experiments and tests on a daily basis

… and many other tools

using… to improvethe quality of recommendations

… resultingin happycustomers!

3 4


eBay relies on Automic

If Automic goes down eBay loses 70% of their web traffic to Amazon

– Automic automates Hadoop for eBay which provides all of their business intelligence for optimized SEO

– Automic moves data, schedules the map reduce, schedules the analytics and then pushes the output to Google


Automating ebay Data Warehouse Platforms

ebay DW environment

Teradata:– Mozart: 2.6PB(used storage)/6.6PB(total storage)– Martini: 1.4PB used, 8.5PM total – EDW concurrent queries: 500+

Singularity (eBay specific TD):– Vivaldi: 9.5PB(used storage) /16.9PB (total storage)– Davinci:2.5PM used, 3.4PB total

• SG concurrent queries:100+

Hadoop:– Hadoop Total: 71.5PB /91.9PB (used storage / total storage)– Hadoop Ares: 29.5PB /41.4PB, Hadoop Apollo: 32.2PB /37.8PB,

Hadoop Artemis: 9.8PB/11.9PB– Hadoop concurrent jobs running: 1000+ Source: http://www.slideshare.net/madananil/hadoop-at-ebay


Automic’s Value to Big Data

• We help our customers to get out of the scripting business by abstracting the APIs from the

user by using Hadoop templates

• Current functionality can be extended by Automic and Users alike and in turn distributed via

Automic’s Marketplace, so there is no need to wait for vendors to catch up and release a

new Agent for new APIs (think Falcon, Ranger, Knox, Ambari, Cloudbreak, etc.)

• Automic and it’s Objects are agnostic – templates work with Hortonworks, Cloudera, MapR

– they can even help you transition between Hadoop distributions


Contact

Dave KellermannsChief Automation Architect

[email protected]

+1 (720) 440-2838

Thank you!

automating big data with the automic hadoop agent

Presentations & Public Speaking