apache ambari

© Hortonworks Inc. 2011 – 2015. All Rights Reserved

Apache Ambari 2.0.0

April 2015

What’s New


What is Ambari?

Apache Ambari is the open source

operational platform to provision, manage

and monitor Hadoop clusters


What’s New in Ambari 2.0.0

Core Platform

Ambari Alerts (AMBARI-6354)

Ambari Metrics (AMBARI-5707)

Simplified Kerberos Setup (AMBARI-7204)

Automated Rolling Upgrade (AMBARI-7804)

HA Stack Support

Hive Metastore HA (AMBARI-6684)

HiveServer2 HA (AMBARI-8906)

Oozie HA (AMBARI-6683)

Ambari Platform

Handle umask 027 setting (AMBARI-7796)

Ambari Agent non-root (AMBARI-1596)

Blueprints API

Add Host (AMBARI-8458)

For a complete list of changes:

https://issues.apache.org/jira/browse/AMBARI/fixforversion/12327486

https://issues.apache.org/jira/browse/AMBARI/fixforversion/12327486


Ambari Alerts


Ambari Alerts

• Ambari Web provides centralized management of Health Alerts

• Alerts are configured by default based on the Services in your cluster


Viewing Host Alerts

• View all Health Alerts for each host


Modifying Alerts

• Control thresholds, check intervals and response text


Alert Notifications

• What: Create and manage

multiple notification targets

• Control who gets notified when

• Why: Filter by severity

• Send only certain notifications to certain

targets based on severity

• How: Control dispatch method

• Support for EMAIL + SNMP


Alert Groups

• Create and manage groups of

alerts

• Group alerts further controls what alerts

are dispatched which notifications

• Assign group to notifications

• Only dispatch to interested parties


Terminology

Term Definition

Alert Definition Defines the alert including the description, check interval, type and

thresholds.

Type The type of alert, such as PORT, METRIC, WEB, AGGREGATE, SCRIPT.

State Indicates the state of an alert definition (enabled or disabled). When

disabled, no alert instances are created.

Alert Instance Represents the specific alert instances based on an alert definition. For

example: the alert definition for “DataNode process” will have an alert

instance per DataNode component in the cluster.

Status The status (or “severity”) of the alert instance, such as: OK, CRIT.

Threshold The thresholds assigned to each alert status (dependent on type).

Alert Group Grouping of alert definitions, useful for handling notifications targets.

Notification A notification target for when an alert instance status changes. Methods of

notification include EMAIL and SNMP.


Alert Types

Type Description StatusThresholds

Configurable?

PORTWatches a port based on a configuration property as the uri.

Example: Hive Metastore ProcessOK, WARN, CRIT Yes (seconds)

METRICWatches a metric based on a configuration property.

Example: ResourceManager RPC LatencyOK, WARN, CRIT Yes (variable)

AGGREGATEAggregate of status for another alert definition.

Example: percentage NodeManagers AvailableOK, WARN, CRIT Yes (percentage)

WEBWatches a Web UI and adjusts status based on response.

Example: App Timeline Web UIOK, WARN, CRIT No

SCRIPTUses a custom script to handle checking.

Example: NodeManager Health SummaryOK or CRIT No


Ambari Alerting System

1. User creates or modifies cluster

2. Ambari reads alert definitions

from Stack

3. Ambari sends alert definitions to

Agents and Agent schedules

instance checks

4. Agents reports alert instance

status in the heartbeat

5. Ambari responds to alert instance

status changes and dispatches

notifications (if applicable)

Ambari

Server

1

2

4

Stack definition

alerts.json

5

Ambari

Agent(s)

3

email

snmp


Notable Alert REST APIs

REST Endpoint Description

/api/v1/clusters/:clusterName/alert_definitions The list of alert definitions for the cluster.

/api/v1/clusters/:clusterName/alerts The list of alert instances for the cluster. Use partial response syntax to filter.

Example: find all alert instances that are CRITITAL/api/v1/clusters/c1/alerts?Alert/state.in(CRITICAL)

Example: find all alert instances for “ZooKeeper Process” alert def/api/v1/clusters/c1/alerts?Alert/definition_name=zookeeper_server

_process

/api/v1/clusters/:clusterName/alert_groups The list of alert groups.

/api/v1/clusters/:clusterName/alert_history The list of alert instance status changes.

/api/v1/alert_targets/ The list of configured alert notification targets for Ambari.


Configuring Alerts During Cluster Install

• Ambari Alerts are configured by

default based on the Services

included in your cluster

• Setting-up email notification is

optional

• Customize Services > Misc


Ambari Alerts <-> Nagios Integration Patterns

Pattern Description Ambari<->Nagios Integration

1Use Ambari Alerts for

definitions, instances and

instance status.

• Integrate with Ambari REST API.

• Generate Nagios cfgs based on alert information from Ambari.

• Nagios watches Ambari.

• Nagios dispatches notifications based on Ambari reported status.

2Use Ambari Alerts for definition

and instances.

• Integrate with Ambari REST API.

• Generate Nagios cfgs based on alert information from Ambari.

• Nagios watches the cluster.

• Nagios dispatches notifications based on status determined by Nagios.

3 Do not use Ambari Alerts.

• Use Ambari REST API to obtain cluster information.

• Define and write your own alerts for Nagios cfgs.

• Nagios watches cluster.

• Nagios dispatches notifications based on status determined by Nagios.


EXAMPLE: /api/v1/clusters/MyCluster/alert_definitions/15

{

"href" : "http://c6401.ambari.apache.org:8080/api/v1/clusters/MyCluster/alert_definitions/15",

"AlertDefinition" : {

"cluster_name" : "MyCluster",

"component_name" : "ZOOKEEPER_SERVER",

"description" : "This host-level alert is triggered if the ZooKeeper server process cannot be determined",

"enabled" : true,

"id" : 15,

"ignore_host" : false,

"interval" : 1,

"label" : "ZooKeeper Server Process",

"name" : "zookeeper_server_process",

"scope" : "ANY",

"service_name" : "ZOOKEEPER",

"source" : {

"default_port" : 2181.0,

"reporting" : {

"ok" : {

"text" : "TCP OK - {0:.3f}s response on port {1}"

},

"warning" : {

"text" : "TCP OK - {0:.3f}s response on port {1}",

"value" : 1.5

},

"critical" : {

"text" : "Connection failed: {0} to {1}:{2}",

"value" : 5.0

}

},

"type" : "PORT",

"uri" : "{{zookeeper-env/clientPort}}"

}

}

}


EXAMPLE: /api/v1/clusters/MyCluster/alerts/1

{

"href" : "http://c6401.ambari.apache.org:8080/api/v1/clusters/MyCluster/alerts/1",

"Alert" : {

"cluster_name" : "MyCluster",

"component_name" : "AMBARI_AGENT",

"definition_id" : 42,

"definition_name" : "ambari_agent_disk_usage",

"host_name" : "c6401.ambari.apache.org",

"id" : 1,

"instance" : null,

"label" : "Ambari Agent Disk Usage",

"latest_timestamp" : 1425530280340,

"maintenance_state" : "OFF",

"original_timestamp" : 1425484200341,

"scope" : "HOST",

"service_name" : "AMBARI",

"state" : "OK",

"text" : "Capacity Used: [1.34%, 7.0 GB], Capacity Total: [525.3 GB]"

}

}


Pattern #1: Alert definitions, instances and status

1. Ambari reads alert definitions and

creates instances for the cluster

2. Use Ambari REST API to read

alert instance information and

generate Nagios .cfg files

3. Nagios watches Ambari REST API

for alert instance status changes

4. Ambari receives alert instance

status info from Agents

5. On status changes, Nagios

dispatches notifications

Nagios

Ambari

Server1

2

.cfg

4

3

5email

snmp

CLUSTER

(Optional: Not shown) Configure additional alerts in Nagios to watch

components of the system (Ambari Server, Agents, Hosts)


Pattern #2: Alert definitions in Ambari

1. Ambari reads alert definitions and

creates alert instances for cluster


alert definition information


cluster configuration information

for generating Nagios .cfg files

4. Nagios watches the cluster


dispatches notificationsNagios

Ambari

Server1

3

.cfg

4

5email

snmp

CLUSTER

2




Pattern #3: No alerts in Ambari

1. Define your own alerts and use

Ambari REST API to read cluster

configuration information for

generating Nagios .cfg files for

those alerts

2. Nagios watches the cluster


dispatches notifications

Nagios

Ambari

Server

1

.cfg

2

3email

snmp

CLUSTER




Ambari Metrics


Ambari Metrics System

Built using Hadoop technologies

Default uses local filesystem for

metrics storage (“embedded”) **

Local Filesystem **

HBaseATS

Phoenix

** Tech Preview “distributed” storage option to use existing HDFS


Terminology

Term Definition

Ambari Metrics (“AMS”) The built-in metrics collection system for Ambari (“AMS”).

Metrics Collector The standalone server that collects metrics, aggregates metrics, serves

metrics from the Hadoop service sinks and the Metrics Monitor. Analogous

to gmetad.

Metrics Monitor Installed on each host in the cluster to collect system-level metrics and

forward to the Collector. Analogous to gmond.

Metrics Hadoop Sinks Plugs into the Service sinks to send Hadoop metrics to the Collector.


Collection System

1. Metric Monitors send system-

level metrics to Collector

2. Sinks send Hadoop-level

metrics to Collector

3. Metrics Collector service

stores and aggregates metrics

4. Ambari exposes REST API for

metrics retrieval

Ambari

Server

Metrics

Monitor

Metrics

Collector

Host1

Sink(s)

3

Metrics

Monitor

Host1

Sink(s)Metrics

Monitor

Hosts

Sink(s)

1 2

4


Simplified Kerberos Setup


Simplified Kerberos Setup

• Wizard driven and automated

Kerberos support

• Works with existing Kerberos

infrastructure, including Active

Directory

• Eliminates time consuming

administration of Kerberos


Automated Kerberos Operations

Once enabled, Ambari works directly with the Kerberos infrastructure to

automate these common tasks:

• Add/Delete Host

• Add Service

• Add/Delete Component

• Regenerate Keytabs

• Disable Kerberos


Terminology

Term Definition

Service Principals Principals required for HDP Service Components.

Ambari PrincipalsHeadless principals used by Ambari to perform “smoke tests” and “health

alert checks”.

KDC Admin AccountAn administrative account that will be used by Ambari to create principals

and generate keytabs in KDC.


Principal and Keytab Generation and Distribution

1. User provides KDC Admin

Account credentials to Ambari

2. Ambari connects to KDC, creates

principals (Service and Ambari)

needed for cluster

3. Ambari generates keytabs for the

principals

4. Ambari distributes keytabs to

Ambari Server and cluster hosts

5. Ambari discards the KDC Admin

Account credentials

Ambari

Server KDC

1 2

4

3

5

HDP

Cluster


Ambari + Service Keytab Files

Ambari

Server

HDP

Cluster

Hosts

Keytabs for

Ambari

Principals

Keytabs for

Service +

Ambari

Principals

KDC

Service Principals

Ambari Principals

Ambari and

Service

Principals


Customizable Principal Attributes

• Customize the directory principal attributes for your environment

• Particularly useful when using Active Directory for Kerberos


Manage Kerberos Clients

• Ambari installs Kerberos clients

on cluster hosts

• Optional to not have Ambari

manage krb5.conf client config

OS Client

RHEL/CentOS/OEL krb5-workstation

SLES 11 krb5-client

Ubuntu 12 krb5-user, krb5-config


Configure Ambari and Service Identities


Ambari Automated Rolling Upgrade

For HDP Stack


Rolling Upgrades

Minimized

Service Disruption &

Degradation

Minimizes service disruption by avoiding services

entirely not being available for a period of time, or

suffering performance degradation.

Reliable, Safe, Efficient

Process

Dependable, certified, check-pointed process.

Keep data, configurations, key operational state safe.


Process: Rolling Upgrade

HDP has a certified process

for Rolling Upgrades

Services are switched over to

new version in rolling fashion

ZooKeeper

Ranger

Core Masters

Core Slaves

Hive

Oozie

Falcon

Clients

Kafka

Knox

Storm

Slider

Flume

Finalize

HDFS, YARN, MR,

Tez, HBase, Pig.

Hive

HDFS

YARN

HBase


Process: Rolling Downgrade

ZooKeeper

Ranger

Core Masters

Core Slaves

Hive

Oozie

Falcon

Clients

Kafka

Knox

Storm

Slider

Flume

DowngradeV

V

V

V

V

V

V

V

V

V

V

V

V

Finalize


Packaging: Side-by-Side Versions

/usr/hdp/current/usr/hdp/2.2.4.0-2633

/usr/hdp/2.2.0.0-2041

/hadoop-hdfs-namenode/hadoop-hdfs

/hadoop-yarn

/hive

/hadoop-hdfs

/hadoop-yarn

/hive

/hadoop-hdfs-datanode


Ambari Automated Rolling Upgrade

Register InstallPerform

UpgradeFinalize


Register New Version

• Register new version with

Ambari

• Provide the Repository Base

URLs for new version


UpgradeFinalize


New Version Available Register InstallPerform

UpgradeFinalize


Install Packages

Click “Install Packages” to start the install of the version across hosts


UpgradeFinalize


Perform Upgrade Register InstallPerform

UpgradeFinalize


Perform Upgrade

Click “Perform Upgrade” to start the Rolling Upgrade Wizard


UpgradeFinalize


Upgrade In Process

Badge indicates upgrade “In Process” or “Action Required”


UpgradeFinalize


Wizard Driven Experience Register InstallPerform

UpgradeFinalize

With

verification

and validation


Rolling Upgrade – Success! Register InstallPerform

UpgradeFinalize


Ambari Agent Non-Root

For HDP Stack


Non-Root Ambari Agent

Agent Runs Commands From the Ambari Server

• Configuration Change

• Service Start

• Service Stop

Some Command require root level access• /bin/su hdfs -l -s /bin/bash -c /usr/hdp/current/hadoop-client/sbin/hadoop-

daemon.sh --config /etc/hadoop/conf start datanode

Sudo Leveraged

• Configuration for:

– Customizable Users (su hdfs, yarn, etc.)

– Non-Customizable Users (su mysql)

– Commands (yum, mkdir, touch, test, etc.)

Ambari

AgentAmbari

AgentAmbari

Agent

python


Configuring Agent for Non-Root

1. Create and configure a sudoer account

2. Manually bootstrap Ambari Agents

3. Set run_as_user in ambari-agent.ini for the sudoer account


Updated umask Handling


What about umask?

Unix Permissions Basics: (user, group, other)

4 – read

2 – write

1 – execute

rwxr-xr-x == 755

Previous Behavior:

• If (umask > 022); Warning during agent pre-req check

• Installations would fail if ignored

New Behavior:

• If (umask > 027); Warning during agent pre-req check

• Installation will fail if ignored


Blueprints

Add Host


Blueprints Add Host

Add hosts to a cluster based on a

host group from a Blueprint

Add one or more hosts with a

single call

POST /api/v1/clusters/MyCluster/hosts

{

"blueprint" : "myblueprint",

"host_group" : "workers",

"host_name" : "c6403.ambari.apache.org"

}

POST /api/v1/clusters/MyCluster/hosts

[

{




},

{




}

]

https://issues.apache.org/jira/browse/AMBARI-8458

https://issues.apache.org/jira/browse/AMBARI-8458


High Availability

For HDP Stack


High Availability Support

HDP Stack Mode Ambari 2.0.0

HDFS: NameNode HDP 2.0+ Active/Standby

YARN: ResourceManager HDP 2.1+ Active/Standby

HBase: HBaseMaster HDP 2.1+ Multi-master

Hive: HiveServer2 HDP 2.1+ Multi-instance

Hive: Hive Metastore HDP 2.1+ Multi-instance

Oozie: Oozie Server* HDP 2.1+ Multi-instance

* Oozie Server must not use Derby database and needs external load balancer to complete HA solution.


Hive HA

Services > Hive > Service Actions

+ Add Hive Metastore

+ Add HiveServer2


Learn More

Page 58

Resource Location

Apache Ambari Project Page http://ambari.apache.org

Ambari Project Wiki https://cwiki.apache.org/confluence/display/AMBARI

Ambari Project JIRA https://issues.apache.org/jira/browse/AMBARI

http://ambari.apache.org

https://cwiki.apache.org/confluence/display/AMBARI

https://issues.apache.org/jira/browse/AMBARI

apache ambari - what's new in 2.0.0

Technology

ambari metrics ambari

setting ambari

alert instancerepresents

ambari agent nonroot

oozie ha ambari

ambari platformhandle

alert status dependent