manage add-on services in apache ambari

1 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Manage Add-on Services in Apache Ambari

Jayush LuniyaDataworks Summit Sydney, 2017


Speakers

Apache Ambari PMC

Engineering Manager @ Hortonworks

[email protected]

Jayush Luniya

mailto:[email protected]


Agenda

Ambari Overview

Management Packs – Mpacks v1

Add-On Services – Primer

vNext Architecture – Ambari 3.x

Mpack Based Operations Model – Mpacks v2

Summary


Ambari Overview


Apache Ambari

Ambari ServerProvision | Manage | Monitor

Ambari REST API

Ambari Stacks

Apache Ambari is a 100% open sourceplatform for provisioning, managing andmonitoring Apache Hadoop clusters

Other Operations Tools

Am

bar

i Age

nts


Apache Ambari Themes

• Deliver the core operational capabilities to provision, manage and monitor Hadoop clusters at scale.

Operate Hadoop at

Scale

• Robust API for integration with existing enterprise systems

Integrate with the

Enterprise

• Provide extensible platform for Customers, Partners and the Community (Stacks, Views)

Extend for the

Ecosystem


Ambari Architecture


Extensibility Features

• No vendor lock-in with Ambari

• Customize a stack for customer specific environments

• Add new Services (ISV or otherwise) beyond vendor stackStacks

• Use Ambari for automating cluster installations

• Share best practices on layout and cluster configurationBlueprints

• Extend and customize the Ambari Web UI

• Add new capabilities, customize existing capabilitiesViews


Lifecycle Management

Deploy

Configure

MonitorSecure

Upgrade

Extend


Management Pack – V1


Ambari Management Packs

Motivation



Decouple stack releases from Ambari core release

Delivery artifact to release stacks and services

Release vehicle for add-on services

Tarballs with metadata of how to install the mpack

EPIC: AMBARI-14854

Ambari Wiki

Release: 2.4.0.0

https://issues.apache.org/jira/browse/AMBARI-14854

https://cwiki.apache.org/confluence/display/AMBARI/Management+Packs


Management Pack Use Cases

Hortonworks Data Flow (HDF)

Open Data Platform (ODP)

Hortonworks CyberSecurity (HCS)

Lucidworks HDP Search (Solr)

Stack Mpacks Add-on Service Mpacks


Add-On Services


Add-on Service

Mpack Definition– Specify services, stacks in mpack– Format: mpack.json

Service Definition– Specify components, dependencies, configs– Format: metainfo.xml

Lifecycle Management– Scripts for install/start/stop/upgrade commands– Format: python scripts

Role Command Order– Dependency order for start/stop commands – Format: role_command_order.json

Service Repo – Where to install the service binaries from?– Format: repoinfo.xml

Configurations– Key, value, description, allow empty, password, etc.– Format: <config-type>.xml

Starter Pack


Add-on Service

Enhanced UI– Enhanced UI configuration and dashboard layouts– Format: theme.json, widgets.json

Service Advisor– Recommend/validate configurations– Format: service-advisor.py

Alerts– Defines periodic checks to perform and alert on– Format: alerts.json

Metrics– Defines metrics to be sent from your service to Ambari– Format: metrics.json

Kerberos– Defines service principals, keytab & rules for configuration changes – Format: kerberos.json

Service Upgrade Pack– Steps to perform to upgrade the service (express, rolling)– Format: upgrades/*.xml

Starter Pack


Mpack Definition

Mpack.json{

"name": "my-ambari-mpack",

"version": “1.0.0.0",

"description": "Ambari Management Pack for MyService-1 & MyService-2",

"prerequisites": {

"min-ambari-version": "2.5.0.0",

"min-stack-versions": [

{

"stack_name": "HDP",

"stack_version": "2.5.0"

}

]

},

"artifacts": [

{

"name": "my-service-definitions",

"type" : "service-definitions",

"source_dir" : "common-services"

}

],


Mpack DefinitionMpack.json

{

"name" : "my-addon-services",

"type" : "stack-addon-service-definitions",

"source_dir": "addon-services",

"service_versions_map": [

{

"service_name" : "MyService-1",

"service_version" : "2.1.1",

"applicable_stacks" : [

{

"stack_name" : "HDP",

"stack_version" : "2.5"

}

]

},

{

"service_name" : "MyService-2",

"service_version" : "1.0.1",

"applicable_stacks" : [

{

"stack_name" : "HDP",

"stack_version" : "2.6"

}

]

}

]


Service DefinitionMetainfo.xml<service>

<name>SAMPLESRV</name>

<displayName>New Sample Service</displayName>

<comment>A New Sample Service</comment>

<version>1.0.0</version>

<components>

<component>

<name>SAMPLESRV_MASTER</name>

<displayName>Sample Srv Master</displayName>

<category>MASTER</category>

<cardinality>1</cardinality>

<commandScript>

<script>scripts/master.py</script>

<scriptType>PYTHON</scriptType>

<timeout>600</timeout>

</commandScript>

</component>

<component>

<name>SAMPLESRV_SLAVE</name>

<displayName>Sample Slave</displayName>

<category>SLAVE</category>

<cardinality>1+</cardinality>

...


Service DefinitionMetainfo.xml...

<commandScript>

<script>scripts/slave.py</script>


</commandScript>

<customCommands>

<customCommand>

<name>MY_CUSTOM_COMMAND</name>

<commandScript>

<script>scripts/my_custom_command.py</script>


<timeout>1200</timeout>

</commandScript>

</customCommand>

<customCommands>

<dependencies>

<dependency>

<name>HDFS/HDFS_CLIENT</name>

<scope>cluster | host</scope>

<auto-deploy>

<enabled>true | false</enabled>

</auto-deploy>

</dependency>

<dependencies>


Service DefinitionMetainfo.xml...

</component>

</components>

<requiredServices>

<service>HDFS</service>

</requiredServices>

<configuration-dependencies>

<config-type>service-env</config-type>

<config-type>service-site</config-type>

</configuration-dependencies>

<osSpecifics>

<osSpecific>

<osFamily>any</osFamily>

<packages>

<package>

<name>rpm_apt_pkg_name</name>

</package>

</packages>

</osSpecific>

</osSpecifics>

</service>

</services>

</metainfo>


Lifecycle Management

import sys

from resource_management import Script

class Master(Script):

def install(self, env):

print 'Install the Sample Srv Master'

def stop(self, env):

print 'Stop the Sample Srv Master'

def start(self, env):

print 'Start the Sample Srv Master'

def status(self, env):

print 'Status of the Sample Srv Master'

def configure(self, env):

print 'Configure the Sample Srv Master'

if __name__ == "__main__":

Master().execute()

Python Scripts


Lifecycle ManagementRole Command Order{

"_comment" : "Record format:",

"_comment" : "blockedRole-blockedCommand: [blockerRole1-

blockerCommand1, blockerRole2-blockerCommand2, ...]",

"general_deps" : {

"_comment" : “Dependencies for SAMPLESRV",

"SAMPLESRV_SLAVE-START": ["SAMPLESRV_MASTER-START"],

"SAMPLESRV_SLAVE-STOP": ["SAMPLESRV_MASTER-STOP"]

}

}


Enhanced Configurations

Too many configurations– Which ones are important?

Too easy to mess up– What are valid/reasonable values?

– What are the units?

– Ok, what about dependencies?

Gets harder with combinations of services, host assignments, enabled features, CPU/RAM/disks, etc– Any recommendations? What am I doing wrong?

Solution: – Enhanced Configurations

– Stack/Service Advisor

Hadoop Configuration Challenges


Enhanced Configurations - UI

Customizable layout- Tabs- Sections- Sub-sections- Simple grid layout

(Advanced Tab contains remaining configurations)

New Widgets

Sliders- Recommended- Minimum- Maximum- Increment StepCombos- Enumerated valuesToggles- Binary optionsSpinners- Splits value into multiple

controls. - Time in milliseconds split

into days, hours, minutes.

Lists- Enumerated values- Single select- Multi select


Service Advisor

Replaces monolithic stack advisors

Defined at service level

Config recommendations and validations

Scenarios– Kerberos Enabled/Disabled

– HTTPS Enabled/Disabled

– Zookeeper Quorum

– Memory Settings

– High Availability Enabled/Disabled


Alerting Framework

Alert Type Description Thresholds (units)

WEB Connects to a Web URL. Alert status is

based on the HTTP response code

Response Code (n/a)

Connection Timeout

(seconds)

PORT Connects to a port. Alert status is based

on response time

Response (seconds)

METRIC Checks the value of a service metric.

Units vary, based on the metric being

checked

Metric Value (units vary)

Connection Timeout

(seconds)

AGGREGATE Aggregates the status for another alert % Affected (percentage)

SCRIPT Executes a script to handle the alert

check

Varies

SERVER Executes a server-side runnable class to

handle the alert check

Varies


Ambari Metrics System

Metric Sinks – HDFS, YARN, HBase, Storm etc.

Metric Monitors – lightweight daemon for system metrics

Collector – API daemon + HBase (embedded / distributed)

Phoenix schema designed for fast reads

Grafana support from version 2.2.2

Ambari

Collector API

Grafana

Phoenix

HDP

Services

System

MO

NIT

OR

SS

INK

S

Metrics Collector


Kerberos

Ambari manages Kerberos principals and keytabs

Works with existing MIT KDC or Active Directory

Once kerberized, handles

– Adding host components

– Adding hosts

– Adding services

– Moving components


vNext Architecture


LimitationsOne-One Relationships

Monolithic Stacks are monolithic

Entire cluster upgrade

Ambari upgrades for refreshing stack definitions

Ambari Cluster

Cluster Stack

StackService Version

ClusterService

Instance

HostComponent

Instance

AmbariHosting Platform


Vision

Replace monolithic stacks

Modular upgrades

Decouple Ambari core release

Independent release cycles

Refactor at service level

3rd party onboarding platform

Multiple Clusters

Multiple Stacks

Multiple Services

Multiple Service Versions

Multiple Host Components

Multiple Hosting Platforms

Modular Multi Everything


Cluster2

Cluster1

Ambari Hosting Platform

v.Next Architecture

ZK1 HDFS YARN HBASE

Core SG

NIFIZK2 KAFKA

Stream SG

Mpack Repository

HDP3.0.0

HDF3.1.0HDF3.2.0

Ambari

BROKER-1

BROKER-2

BROKER-3

HBASE2

Yapp SG

YarnApp1.0.0

Yarn Hosting Platform

HMaster

RServer-1

RServer-2


Cluster2

Cluster1

Ambari Hosting Platform

v.Next Architecture

ZK1 HDFS YARN HBASE

Core SG

NIFIZK2 KAFKA

Stream SG

Mpack Repository

HDP3.0.0

HDF3.1.0HDF3.2.0

Ambari

BROKER-1

BROKER-2

BROKER-3

HBASE2

Yapp SG

YarnApp1.0.0

Yarn Hosting Platform

HMaster

RServer-1

RServer-2

Modular

Multi-Everything


Mpack Based Operations Model


New Concepts - Delivery

Deployment artifact

Individual ownership of services

Release container for services

Examples:o HDFS-3.0.0-packlet

o SPARK-2.0.0-packlet

Distribution artifact

Composed of multiple packlets

Modular release & upgrades

Examples:o HDP-3.0.0-mpack

o HDF-3.1.0-mpack

Packlets Management Packs (Mpacks)

Mpack Repository

Catalog of management packs

Compatibility between mpacks

Examples:o Hortonworks Mpack Repository


Management Packs – V2

Decouple Ambari Core release

Replacement for monolithic stacks

Composition of verified combination of packlets

Support dynamic installation

Mpacks released and upgraded independently

Release Management


Management Packs – V2

Support multiple mpack repositories

Search and discover management packs

Compatibility between management packs

Recommendations for mpack bundles

Add-on Mpacks First class support for 3rd party add-on services

Provide a third party onboarding platform

Independently upgradable services

Mpack Repositories


Upgrades

Modular upgrades

Ability to upgrade subset of services in a cluster

Patch services without affecting other parts of the cluster

Performance improvements

Goals

Mpack Upgrades

Upgrade packs defined at mpack level

Upgrade subset of services in a cluster that belong to a mpack

Download new version of a mpack and kickoff an upgrade


Upgrades

Upgrade mechanism for hotfix releases

Selectively patch components without restarting other services

Purely Version Definition File (VDF) driven; no configuration changes

Does not require a new management pack; uses same upgrade pack

Applied patches can be reverted; no finalization step

Patch Upgrades

Service Upgrades

VDF driven if no configuration changes needed

Mpack driven for more complicated service upgrades

No revert capability (similar to current cluster upgrades)


Summary


Key Takeaways

Stack Based Operations Model

Refactoring at service level

Mpacks v1 provided delivery vehicle for add-on services

Architecture limitations remain

Mpack Based Operations Model

Re-architect from ground up

First class support for add-on services

Modular deployments & upgrades

Current Future

A

M P


Resources

https://cwiki.apache.org/confluence/display/AMBARI/Stacks+and+Services

https://cwiki.apache.org/confluence/display/AMBARI/Custom+Services


https://cwiki.apache.org/confluence/display/AMBARI/Enhanced+Configs

https://cwiki.apache.org/confluence/display/AMBARI/Enhanced+Service+Dashboard

https://cwiki.apache.org/confluence/display/AMBARI/Stack+Defined+Metrics

https://cwiki.apache.org/confluence/display/AMBARI/Alerts

https://cwiki.apache.org/confluence/display/AMBARI/Quick+Links

https://cwiki.apache.org/confluence/display/AMBARI/Automated+Kerberizaton

https://cwiki.apache.org/confluence/display/AMBARI/Service+Upgrade

https://cwiki.apache.org/confluence/display/AMBARI/Role+Command+Order

https://cwiki.apache.org/confluence/display/AMBARI/Stacks+and+Services

https://cwiki.apache.org/confluence/display/AMBARI/Custom+Services


https://cwiki.apache.org/confluence/display/AMBARI/Enhanced+Configs

https://cwiki.apache.org/confluence/display/AMBARI/Enhanced+Service+Dashboard

https://cwiki.apache.org/confluence/display/AMBARI/Stack+Defined+Metrics

https://cwiki.apache.org/confluence/display/AMBARI/Alerts

https://cwiki.apache.org/confluence/display/AMBARI/Quick+Links

https://cwiki.apache.org/confluence/display/AMBARI/Automated+Kerberizaton

https://cwiki.apache.org/confluence/display/AMBARI/Service+Upgrade

https://cwiki.apache.org/confluence/display/AMBARI/Role+Command+Order


Apache JIRAs - vNext

AMBARI-14714 – Multi Everything ArchitectureUmbrella Apache EPIC

AMBARI-19621 – Mpack Based Operations Model

AMBARI-20463 – Multi Service Instances

AMBARI-20465 – Multi Host Component Instances

AMBARI-17353 – Yarn Hosted Services

AMBARI-12556 – Patch Upgrades

AMBARI-20466 – Multi Cluster

AMBARI-20435 - Swagger Integration

Other Apache EPICs

Release Timelines Features will be incrementally delivered in Apache Ambari 3.x releases.

Join the Apache Ambari Community










Thank You

manage add-on services in apache ambari

Technology