future of apache ambari

Future of Apache Ambari

DataWorks Summit 2017

San Jose, USA

06/14/2017

2 © Hortonworks Inc. 2011 – 2017. All Rights Reserved

Speakers

Apache Ambari PMC

Staff Engineer @ Hortonworks

[email protected]

Apache Ambari PMC

Principal Engineer @ Hortonworks

[email protected]

Jaimin Jetly Jayush Luniya

mailto:[email protected]

mailto:[email protected]


Agenda


Agenda

Ambari Overview

v.Next Architecture

Management Packs

Multi Everything

Upgrades

UI Enhancements

Other Improvements


Ambari Overview


Apache Ambari

Ambari ServerProvision | Manage | Monitor

Ambari REST API

Ambari Stacks

Apache Ambari is a 100% open sourceplatform for provisioning, managing andmonitoring Apache Hadoop clusters

Other Operations Tools

Am

bar

i Age

nts


Lifecycle Management


Ambari Architecture


LimitationsOne-One Relationships

Monolithic Single monolithic stack

Entire cluster upgrade

Ambari upgrades for refreshing stack definitions

Ambari Cluster

Cluster Stack

StackService Version

ClusterService

Instance

HostComponent

Instance

AmbariHosting Platform


v.Next Architecture


Vision

Refactor at service level

Replace monolithic stacks

Decouple Ambari core release

Independent release cycles

Modular upgrades

3rd party onboarding platform

Multiple Clusters

Multiple Stacks

Multiple Services

Multiple Service Versions

Multiple Host Components

Multiple Hosting Platforms

Modular Multi Everything


New Concepts

Deployment artifact

Individual ownership of services

Independent release cycles

Examples:o HDFS-3.0.0-packlet

o SPARK-2.0.0-packlet

Distribution artifact

Composed of multiple packlets

Modular release & upgrades

Examples:o HDP-3.0.0-mpack

o HDF-3.1.0-mpack

Packlets Management Packs (Mpacks)

Mpack Repository

Catalog of management packs

Compatibility between mpacks

Examples:o Hortonworks Mpack Repository


New Concepts

Runtime concept

Logical grouping of services

Examples:o Core Service Group

o Stream Service Group

Defined in management packs

Prescribes how to deploy cluster

Examples:o Shared v/s Dedicated Zookeeper

Service Groups Deployment Models


Cluster2

Cluster1

Ambari Hosting Platform

v.Next Architecture

ZK1 HDFS YARN HBASE

Core SG

NIFIZK2 KAFKA

Stream SG

Mpack Repository

HDP3.0.0

HDF3.1.0HDF3.2.0

Ambari

BROKER-1

BROKER-2

BROKER-3

HBASE2

Yapp SG

YarnApp1.0.0

Yarn Hosting Platform

HMaster

RServer-1

RServer-2


Management Packs


Management Packs

Decouple Ambari Core release

Replacement for monolithic stacks

Composition of verified combination of packlets

Support dynamic installation

Mpacks released and upgraded independently

Release Management


Management Packs

Support multiple mpack repositories

Search and discover management packs

Compatibility between management packs

Recommendations for mpack bundles

Add-on Mpacks First class support for 3rd party services

Mpacks provide a third party onboarding platform

Independently upgradable services

Mpack Repositories


Multi Everything


Multi Everything

Logical grouping of services

Service Groups

Cluster v/s Service Group

Cluster – Uniquely identifies collection of physical hosts

Service Group – Logical grouping of correlated services

Mpack v/s Service Group

Mpack – Delivery concept

Service Group – Runtime concept

Inheritance to recommend service dependencies

Service Group Inheritance


Multi Everything

Defined in management packs

Enriched user deployment experience

Prescriptions for deploying a clustero Service groups to create

o Service dependencies

Examples:

Deployment Models

Core SG

Stream SG

ZK HDFS

KAFKA

Core SG

Stream SG

ZK1 HDFS

ZK2 KAFKASAM SAM

Shared Zookeeper Dedicated Zookeeper


Multi Everything

Talk: Apache Hadoop Yarn: Present and Future - Vinod Vavilapalli

Long Lived Yarn Services - YARN-896

First class support for Yarn Services - YARN-4692 (Yarn Assemblies)

Multiple Hosting Platformso Ambari Hosting Platform

o Yarn Hosting Platform

Ambari should enable first class support for Yarn services

Mpacks provide delivery platform for Yarn Services & Yarn Assemblies

Yarn Hosted Services

https://issues.apache.org/jira/browse/YARN-896

https://issues.apache.org/jira/browse/YARN-4692


Upgrades


Upgrades

Modular upgrades

Ability to upgrade subset of services in a cluster

Patch services without affecting other parts of the cluster

Performance improvements

Goals

Mpack Upgrades Upgrade packs defined at mpack level

Upgrade subset of services in a cluster that belong to a mpack

Download new version of a mpack and kickoff an upgrade


Upgrades

Upgrade mechanism for hotfix releases

Selectively patch components without restarting other services

Purely Version Definition File (VDF) driven; no configuration changes

Does not require a new management pack; uses same upgrade pack

Applied patches can be reverted; no finalization step

Patch Upgrades

Service Upgrades

VDF driven if no configurations changes needed

Mpack driven for more complicated service upgrades

No revert capability (similar to current cluster upgrades)


UI Enhancements


UI Enhancements

Cosmetic changeso New style for all UI elements

o New design for all pages

Consistent experienceo New bootstrap skin

Utilize available screen spaceo Expandable Navigation bar

New Look & Feel


UI Enhancements – Install Wizard


UI Enhancements – Dashboard


UI Enhancements – Alerts


UI Enhancements

Ambari REST API explorer

Swagger Integration

Interactive console to execute REST APIs

REST API documentation


UI Enhancements – REST API ExplorerDemo


Other Improvements


Other Improvements

Topic: Tuning Apache Ambari Performance for Big Data at Scale

Speakers: Alejandro, Aravindan

Performance: Ambari Scale Testing

Upgrade: Host Ordered Upgrades

AMS: Anomaly Detection

AMS: Extensible - Support other storage choices

AMS: Scalability

Talk


Summary


mpacksmodular packlets

service groups multi

yarn hostedadd-ons

performance


Apache JIRAs

AMBARI-14714 – Multi Everything ArchitectureUmbrella Apache EPIC

AMBARI-19621 – Mpack Based Operations Model

AMBARI-20463 – Multi Service Instances

AMBARI-20465 – Multi Host Component Instances

AMBARI-17353 – Yarn Hosted Services

AMBARI-12556 – Patch Upgrades

AMBARI-20466 – Multi Cluster

AMBARI-20435 - Swagger Integration

Other Apache EPICs

Release Timelines Features will be incrementally delivered in Apache Ambari 3.x releases.

https://issues.apache.org/jira/browse/AMBARI-14714









Thank You

future of apache ambari

Technology