APACHE AMBARI STACK EXTENSIBILITY

Juanjo Marron @ IBMApache Ambari Committer

Jayush Luniya @ HortonworksApache Ambari PMC

Agenda• Ambari Stack Framework• Current State of Affairs• New Extensibility Features

• Stack Featurization• Service Level Extension Points• Stack Extensions• Ambari Management Packs

• Future Goals• Q&A

AMBARI STACK FRAMEWORK

Anatomy of Ambari Extension Points

Stack TerminologiesTerm Definition ExamplesSTACK Defines a set of Services, where to obtain

the software packages and how to manage the lifecycle.

HDP-2.3, HDP-2.2

SERVICE Defines the Components that make-up the service.

HDFS, YARN

COMPONENT The building-blocks of a Service, that adhere to a certain lifecycle.

NAMENODE, DATANODE, OOZIE_SERVER

CATEGORY The category of Component. MASTER, SLAVE, CLIENT

REPO Repository metadata where the artifacts reside

http://public-repo-1.hortonworks.com/HDP/centos6/2.x/GA/2.3.0.0

Ambari Stacks• Stacks define Services + Repo

• What is a stack composed of and where to get the bits• Each service has a definition

• What components are part of the Service• Each service has defined lifecycle commands

• start, stop, status, install, configure• Lifecycle is controlled via command scripts

Ambari Server

Stack

Service Definitions

Command Scripts

xml python

Ambari Agents

Repos

Ambari Stack Framework• Ambari’s extensible stack framework allows different stack vendors to

create their own custom stacks.

CURRENT STATE OF AFFAIRS

Current State• Extensible Framework

• Provides ability for onboarding new Hadoop distributions• Key role in avoiding vendor lock-in

• Stack driven• Service lifecycle management• Custom service commands• UI layouts and themes• Upgrade packs• Stack advisors• Metrics• Kerberos

• Stack Inheritance• Code reuse between stack versions

• Common Service Definitions• Code reuse between multiple stacks

• ODPi Operations Spec

Limitations – Common Services• AMBARI-7201 focused on adding notion of common services• Service definitions scripts were not refactored• Common service scripts still tightly coupled with HDP distribution• Hardcoded references for stack name (HDP), stack version checks

(HDP-2.2+), stack tools (hdp-select, conf-select), installation paths (/usr/hdp), method names (format_hdp_stack_version) etc.

Limitations – Custom Services• Rudimentary approach to add third party custom services

Ambari Wiki• No release vehicle for building, releasing and deploying custom

services• No service level extension points for stack features

• Role command order• Stack advisor• Upgrade packs• Repositories

• Custom service definitions not self-contained and require hacks inside stack definitions

Limitations – Release Management• Ambari core and stack definitions released together• Bug fix in stack definition requires Ambari release• New stack version requires Ambari release• Stack releases and Ambari releases have to be coordinated• Need to decouple stack definition releases from Ambari core release

NEW EXTENSIBILITY FEATURES

Stack Featurization• Provides a basic framework for refactoring stack-specific hardcoded logic in

common service definitions.• Features defined at stack-level based on which execution logic in service

definition is triggered. • All common service definitions have been stack featurized and HDP-specific

hardcodings have been removed. • Stacks similar to HDP can now use common service definitions.• EPIC: AMBARI-13363

Apache JIRA Description Target Release

AMBARI-15420 Generalize resource management library 2.4.0.0

AMBARI-13364 Parameterize stack information used by common services 2.4.0.0

AMBARI-15329 Remove HDP hardcoding 2.4.0.0

Stack FeaturizationNew Config Propertiesambari-server/src/main/resources/stacks/HDP/2.0.6/configuration/cluster-env.xml <property> <name>stack_tools</name> <value></value> <description>Stack specific tools</description> <property-type>VALUE_FROM_PROPERTY_FILE</property-type> <value-attributes> <property-file-name>stack_tools.json</property-file-name> <property-file-type>json</property-file-type> </value-attributes> </property> <property> <name>stack_root</name> <value>/usr/hdp</value> <description>Stack root folder</description> </property> <property> <name>stack_features</name> <value></value> <description>Stack specific tools</description> <property-type>VALUE_FROM_PROPERTY_FILE</property-type> <value-attributes> <property-file-name>stack_features.json</property-file-name> <property-file-type>json</property-file-type> </value-attributes> </property>

Stack FeaturesWithout Stack Featurization:if(Script.is_stack_greater_or_equal("2.2”) && Script. is_stack_less_than("2.5")) { audit_jdbc_url = format('jdbc:mysql://{db_host}/{ranger_auditdb_name}'}

With Stack Featurization:ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_features.json{ "stack_features": [ { "name": "ranger_audit_db_support", "description": "Ranger Audit to DB support", "min_version": "2.2.0.0", "max_version": "2.5.0.0" } ]}

if(check_stack_feature(StackFeature.RANGER_AUDIT_DB_SUPPORT, stack_version_formatted)) { audit_jdbc_url = format('jdbc:mysql://{db_host}/{ranger_auditdb_name}'}

Stack ToolsWithout Stack Featurization:packages.append('hdp-select')

command = format('{sudo} /usr/bin/hdp-select set all `ambari-python-wrap /usr/bin/hdp-select versions | grep ^{version_to_select} | tail -1`’) only_if_command = format('ls -d /usr/hdp/{version_to_select}*')

With Stack Featurization:ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_tools.json{ "stack_selector": ["hdp-select", "/usr/bin/hdp-select", "hdp-select"], "conf_selector": ["conf-select", "/usr/bin/conf-select", "conf-select"]}

stack_selector_package = stack_tools.get_stack_tool_package(stack_tools.STACK_SELECTOR_NAME)packages.append(stack_selector_package)

stack_root = Script.get_stack_root()(stack_selector_name, stack_selector_path, stack_selector_package) = stack_tools.get_stack_tool(stack_tools.STACK_SELECTOR_NAME) command = format('{sudo} {stack_selector_path} set all `ambari-python-wrap {stack_selector_path} versions | grep ^{version_to_select} | tail -1`’) only_if_command = format('ls -d {stack_root}/{version_to_select}*’)

Service Level Extension Points

Apache JIRA Description Ambari Wiki Target Release

AMBARI- 15388

Upgrade Pack Extensions Ambari Wiki 2.4.0.0

AMBARI-15226 Stack Advisor Extensions Ambari Wiki 2.4.0.0

AMBARI-9363 Role command order Extensions Ambari Wiki 2.4.0.0

AMBARI-11268 Quick Links for custom services Ambari Wiki 2.4.0.0

AMBARI-15538 Service-specific repo definitions 2.X.X.X

• Facilitate integration of third-party custom services to a stack definition• Self-contained service definitions without requiring changes to stack definition• Upgrade Pack Extensions

• Service-level upgrade packs that specify how to upgrade the custom service • Defines how to integrate into the stack upgrade pack

• Stack Advisor Extensions• Service Advisors (service_advisor.py) define recommendations and validations for the service• Stack framework extends stack advisors with service advisors

• Role Command Order• Define role command order at service level• Stack framework merges all role command orders to create dependencies

• Quick Links• Define quick links (quicklinks.json) in the service definition that is used by Ambari Web UI• No hardcoded quick links in the Ambari Web UI

• Repo Definitions• Service specific repositories so that third party artifacts can reside on their own repositories• Under development

EPIC: AMBARI-15537

Upgrade Pack Service ExtensionWithout Upgrade Pack Extensions:

- Upgrade logic defined at stack level- Custom services need to modify the stack's upgrade-packs in order to integrate themselves into the cluster

With Upgrade Pack Extensions:

- Each service can define upgrade-packs XML files placed in the service's upgrades/ folder

ambari-server/src/test/resources/stacks/HDP/2.0.5/services/HDFS/upgrades/HDP/2.2.0/upgrade_test_15388.xml

<target>2.4.*</target> <target-stack>HDP-2.4.0</target-stack> <type>ROLLING</type> <prerequisite-checks> <check>org.apache.ambari.server.checks.FooCheck</check> </prerequisite-checks> <order> <group xsi:type="cluster" name="PRE_CLUSTER" title="Pre {{direction.text.proper}}"> <add-after-group-entry>HDFS</add-after-group-entry> <execute-stage service="FOO" component="BAR" title="Backup FOO"> <task xsi:type="manual"> <message>Back FOO up.</message> </task> </execute-stage> </group> ….

Service AdvisorWithout Service Advisor:

- Stack advisor defined at stack level- Custom services need to modify stack advisor files in order to recommend/validate dynamically service configurations and the layout of the service on cluster

With Service Advisor:

- Each service can choose to define its own service advisor and explicitly extend parent's service-advisor script

ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.pyambari-server/src/main/resources/common-services/PXF/3.0.0/service_advisor.py

def getServiceComponentLayoutValidations(self, services, hosts): componentsListList = [service["components"] for service in services["services"]] componentsList = [item["StackServiceComponents"] for sublist in componentsListList for item in sublist] hawqMasterHosts = self.getHosts(componentsList, "HAWQMASTER") hawqStandbyHosts = self.getHosts(componentsList, "HAWQSTANDBY") hawqSegmentHosts = self.getHosts(componentsList, "HAWQSEGMENT") datanodeHosts = self.getHosts(componentsList, "DATANODE")

# Generate WARNING if any HAWQSEGMENT is not colocated with a DATANODE mismatchHosts = sorted(set(hawqSegmentHosts).symmetric_difference(set(datanodeHosts))) if len(mismatchHosts) > 0: hostsString = ', '.join(mismatchHosts) message = "HAWQ Segment must be installed on all DataNodes. " \

Quick LinksWithout Quick Links Extensions:

- Hardcoded quick links in the Ambari Web UI

With Quick Links Extensions :

- A service can add a list of quick links to the Ambari web UI by adding predefined JSON format file to metainfo.xml

ambari-server/src/main/resources/stacks/HDP/2.3/services/HBASE/metainfo.xmlambari-server/src/main/resources/stacks/HDP/2.3/services/HBASE/quicklinks/quicklinks.json

<quickLinksConfigurations> <quickLinksConfiguration> <fileName>quicklinks.json</fileName> <default>true</default> </quickLinksConfiguration> </quickLinksConfigurations>

mapQuickLinks: function (finalJson, item){ if(!(item && item.ServiceInfo)) return; var quickLinks = { OOZIE: [19], GANGLIA: [20], STORM: [31], FALCON: [32], RANGER: [33], SPARK: [34], MY_CUSTOM_SERVICE: [35] };

{ "name": "default", "description": "default quick links configuration", "configuration": { "protocol": { "type":"http" }, "links": [ { "name": "hbase_master_ui", "label": "HBase Master UI", ….. } }, { "name": "hbase_logs", "label": "HBase Logs", …

Stack Extensions• A stack extension is a collection of custom services which are packaged

together.• Provides a REST API for installing extensions.• Extensions are staged at /var/lib/ambari-server/resources/extensions• After installing extensions requires explicit linking of extensions with the

supported stack versions.• Custom services contained in the extension may be added to the cluster like

any other service in the stack.• EPIC: AMBARI-12885

Stack ExtensionsStructure|_ extensions |_ <extension_name> |_ <extension_version> metainfo.xml |_ services |_ <service_name> metainfo.xml metrics.json |_ configuration {configuration files} |_ package {files, scripts, templates}

Stack Applicability<metainfo> <prerequisites> <min-stack-versions> <stack> <name>HDP</name> <version>2.4</version> </stack> <stack> <name>OTHER</name> <version>1.0</version> </stack> </min-stack-versions> </prerequisites></metainfo>

Ambari Management Packs Motivation

Ambari Management Packs• Release artifact to bundle stack definitions, service definitions, add-on service

definitions, views etc.• Decouple stack definition releases from Ambari core release.• Also provide a release vehicle for add-on services.• Released as tarballs but contains metadata that describes applicability and

contents of the management pack• Staging Location: /var/lib/ambari-server/resources/mpacks• Final stack definition can be an overlay of multiple management packs.• EPIC: AMBARI-14854• Ambari Wiki• Target Release: 2.4.0.0

Overlay of Management Packs

Stack Management Pack{ "type" : "full-release", "name" : ”stack1-ambari-mpack", "version": “1.0.0.0", "description" : ”Stack1 Ambari Management Pack", "prerequisites": { "min-ambari-version" : ”2.4.0.0", "max-ambari-version" : “" }, "artifacts": [ { "name" : ”stack1-service-definitions", "type" : "service-definitions", "source_dir": "common-services" }, { "name" : ”stack1-stack-definitions", "type" : "stack-definitions", "source_dir": "stacks" } ]}

hdp-ambari-mpack-1.0.0.0├── mpack.json├── common-services│   └── NEWSERVICE│   └── 1.0.0│   ├── configuration└── stacks └── STACK1 └── 2.0 ├── metainfo.xml ├── repos │ └── repoinfo.xml ├── role_command_order.json └── services │   ├── KAFKA │   │   ├── configuration │   │   │   ├── ranger-kafka-audit.xml │   │   │   └── ranger-kafka-policymgr-ssl.xml │   │   └── metainfo.xml │   ├── NEWSERVICE │   │   └── metainfo.xml │   └── ZOOKEEPER │   └── metainfo.xml └── widgets.json

Addon Service Management Pack{ "type": "full-release", "name": "myservice-ambari-mpack", "version": "1.0.0.0", "description": "MyService Ambari Mpack" , "prerequisites": { "min-ambari-version": "2.4.0.0", "min-stack-versions": [ { "stack_name": "STACK1", "stack_version": ”2.1" } ] }, "artifacts": [ { "name": "myservice-common-services", "type" : "service-definitions", "source_dir" : "common-services" },

{ "name" : "myservice-addon-services", "type" : "stack-addon-service-definitions", "source_dir": "stack-addon-services", "service_versions_map": [ { "service_name" : "MyService", "service_version" : ”0.8.0.0", "applicable_stacks" : [ { "stack_name" : "STACK1", "stack_version" : "2.1" } ] } ] } ]}

Future Goals• Service level repos• Management Pack++

• Short Term Goals (Ambari 2.4.0.0)o Release vehicle for stacks

o HDP management pack, IOP management packo Release vehicle for add-on services (custom services)

o Microsoft-R management packo Retrofit in existing stack processing infrastructureo Command line to update stack and service definitions

• Long Term Goals (Ambari 2.4+)o Release HDP stacks as mpackso Build management pack processing infrastructureo Dynamic creation of stack definitions by processing mpackso Rest API for adding/removing mpacks

• UI wizards stack driven

Q&A