disclaimer - meetupfiles.meetup.com/1642258/ambari-overview.pdfdisclaimer • this document may...

© Hortonworks Inc. 2014

Disclaimer •  This document may contain product features and technology directions

that are under development or may be under development in the future.

•  Technical feasibility, market demand, user feedback, and the Apache Software Foundation community development process can all affect timing and final delivery.

•  This document’s description of these features and technology directions does not represent a contractual commitment from Hortonworks to deliver these features in any generally available product.

•  Product features and technology directions are subject to change, and must not be included in contracts, purchase orders, or sales agreements of any kind.

1


Apache Ambari Provisioning, Managing and Monitoring Hadoop

October 2014


Agenda

• Ambari Overview and the Community • Provisioning Clusters and Managing Services • Extensibility Features • Futures • Learn More


How do you Operate a Hadoop Cluster?

Apache Ambari is a platform to

provision, manage and monitor

Hadoop clusters


Apache Ambari Themes

Operate Hadoop at

Scale

Integrate with the Enterprise

Extend for the

Ecosystem

Deliver the core opera-onal capabili-es to

provision, manage and monitor Hadoop clusters at scale.

Robust API for integra-on with

exis@ng enterprise systems, such as

Teradata Viewpoint and MicrosoD SCOM.

Provide extensible plaForm for

Customers, Partners and the Community to, such as Stacks and

Views.


Enterprise Operations Platform

Apache Ambari is a 100% open source platform for provisioning, managing and monitoring Apache Hadoop clusters

AMBARI WEB

Others

compute &

storage . . .

. . .

. . compute &

storage

.

. EXTEND

AMBARI REST API

AMBARI SERVER PROVISION | MANAGE | MONITOR

Integra@on with Exis@ng Tools

OPERATE

AMBARI STACKS


Innovation at the Speed of Community

An active 100% open source Community: •  60+ Contributors / 30+ Committers •  250+ Ambari User Group Members

2014

April

Graduates to Apache Top Level Project

2013

Dec

Apache Ambari 1.5 Released Adds support for Hortonworks Data Platform 2.1 Adds new operational and extensibility capabilities


What’s New in Ambari 1.5

• Support for HDP 1.3, 2.0 and 2.1 Stacks • Rolling Restarts • Maintenance Mode • Decommission TTs, NMs, RSs • Service and Component Restarts • Bulk Host Operations • Add Service • Ambari Blueprints – Tech Preview • For a complete list of changes:

https://issues.apache.org/jira/browse/AMBARI/fixforversion/12325019


What’s New in Ambari 1.6.0

• PostgreSQL Support – For Ambari DB, Hive Metastore and Oozie

• Ambari Blueprints – GA

• For a complete list of changes



What’s New in Ambari 1.6.1 • New Host Checks

– Custom JDK Path Check – Hostname Resolution Check – Reverse Hostname Lookup Check

•  Improved External Database Support – Simplified JDBC Driver Setup – Database Connection “Test”

• Stack Repository Management Enhancements – Manage Stack repositories from Ambari Web – Template-driven .repo files

• Performance and Scale – 2,000 node cluster testing

• For a complete list of changes


Provisioning


Choose Services

• Core Hadoop plus Ecosystem Services • Popular Open Source Metrics + Monitoring systems


Master Component Assignment

• Highly customizable targeting of Master components


Slave Component Assignment

• Host-level control to target Slave components • Designate “gateway” hosts with client configurations


Auto Agent Bootstrap (via SSH)

• Configure to use “root” or a “sudoer” user for SSH • More flexibility in root-restricted environments


Manual Agent Bootstrap

• Ambari Agent “manual” bootstrap option • Useful when SSH restricted or not available


Pre-Flight Host Checks •  Examine hosts, check for

conflicts before installation

•  Checks for existing files, directories, packages and processes

Management


Cluster Operations

• Start, Stop and Smoke Test Services • Apply configura@on changes and inject custom proper@es

19


Configuration Management

• Ability to update core-site.xml properties • Add and Remove custom properties


Host Level Visibility + Controls

• View System Metrics and Host Information • Manage Host Components, including start/stop/decommission

• “Add Hosts” to expand cluster capacity

21


Enable NameNode HA

• Post-cluster install, wizard driven experience • For Hadoop 2 Stack (QJM)


NameNode HA Alerts


Moving Masters

•  Move master components to different hosts – NameNode – SecondaryNameNode –  TaskTracker (Hadoop 1) – ResourceManager (Hadoop 2)

•  Move individual NameNodes in Hadoop 2 HA configuration – Hadoop 2


HBase Multi-Master

•  Add multiple HBase Masters

•  During Cluster Install – Assign 1+ HBase Masters

•  Post Cluster Install – Add HBase Master to a host


Host Configuration Groups

• Set custom configuration properties for one or more host groups (e.g. “host overrides”)

• Important for handing “heterogeneous” HW clusters – Different memory, mount points, directories

26

HEAPSIZE= 1024 HEAPSIZE= 512


Staged Configurations Changes

• Make configuration changes and save in Ambari w/o stopping services

• Stage the changes on Ambari, restart components to pickup configurations

Monitoring


Cluster Heatmaps

• Gain health visibility of entire cluster • Quickly identify host outliers and potential issues • Includes both System and Hadoop Metric Heatmaps

29


Hadoop Metrics

• Hadoop Service metrics such as HDFS, MapReduce and HBase

• Visibility info DFS capacity, file operations, RPC latency and JVM performance

30


Hadoop + System Monitoring

31

•  Watch key Hadoop operational metrics for all services

•  Master service availability, CPU utilization, network latency, storage capacity

•  Host-level alerting •  Component service processes

and storage availability


Customizable Metrics Dashboard

• Watch key System and Hadoop metrics


YARN Dashboard Widgets + Heatmaps


YARN Alerts

Core Platform


Platform Support

• Support for popular Linux platforms such as RHEL, CentOS, Oracle Linux and SLES

• Support for Oracle JDK 1.6 and 1.7 • Support for Open JDK 7


Database Support

Hive Metastore Oozie Ambari DB

MySQL 5.x

Oracle 11g

PostgreSQL 8, 9

Default PostgreSQL MySQL Derby


Secure Authentication

• Pluggable LDAP and AD, Default RDBMS • Supports AmbariAdmin and AmbariUser roles


Ambari Extensibility Stacks, Blueprints and Views


Extensibility Features

•  To add new Services (ISV or otherwise) beyond HDP Stack •  To customize a Stack for customer specific environments

•  To use Ambari for automating cluster installations •  To share best practices on layout and cluster configuration

•  To extend and customize the Ambari Web UI •  Add new capabilities, customize existing capabilities

Stacks

Blueprints

Views

Goal: Extend Ambari without hard-coding in Ambari


Community Links

Capability Link

Stacks https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=38571133

Blueprints https://cwiki.apache.org/confluence/display/AMBARI/Blueprints

Views https://cwiki.apache.org/confluence/display/AMBARI/Views


Anatomy of Ambari Extension Points

Ambari Server

Ambari Agent Ambari

Agent Ambari Agent

Ambari Web

Stacks Stacks

Stacks

java!js! python!

Ambari Views Ambari Stacks


Ambari Stacks


Ambari Stacks

• Goal: Reduce time + effort to new Services to Ambari

• Defines a consistent Service lifecycle management interface that can be extended

• Dynamically add Stack + Service definitions • Bring companion technologies together with “Hadoop”

AMBARI {rest}!

<ambari-web>!

Stacks HDFS YARN MR2

Hive

Pig

Oozie

NEW

NEW

NEW


Stack Terminology

Term Definition Examples

STACK Defines a set of Services, where to obtain the software packages and how to manage the lifecycle.

HDP-2.0, HDP-1.3.3

SERVICE Defines the Components that make-up the Service. HDFS, NAGIOS, YARN

COMPONENT The building-blocks of a Service, that adhere to a certain lifecycle.

NAMENODE, DATANODE, OOZIE_SERVER

CATEGORY The category of Component. MASTER, SLAVE, CLIENT


Stack Mechanics

• Stacks define Services + Repos – What is in the Stack, and where to get the bits

• Each Service has a definition – What Components are part of the Service

• Each Service has defined lifecycle commands – start, stop, status, install, configure

• Lifecycle is controlled via command scripts • Ability to define “custom” commands

AMBARI SERVER

Stack

Command Scripts

Service Defini@ons

AMBARI AGENT/S AMBARI AGENT/S AMBARI AGENT/S

python!xml!

Repos


Stacks Support Inheritance

HDP 2.0 Stack

HDP 2.1 Stack

•  Defines a set of Service definitions •  Default service configurations and command

scripts

•  Overrides any Service definitions, commands and configurations

•  Adds new Services specific to this Stack


Service Lifecycle Commands

Component Category Install Start Stop Configure Status Custom

MASTER

SLAVE

CLIENT


Ambari Views


Ambari Views

• Goal: enable the delivery of custom UI experiences in Ambari Web

• Developers can extend the Ambari Web interface •  Views expose custom UI features for Services

• Ambari Admins can authorize Views to Ambari users •  Entitlements framework for controlling access to Views


View Concepts

Views Framework

Views

Core to Ambari

Built by Hortonworks, Community,

Partners


Example Views

Capacity Scheduler Queue Manager

Hive Tez Query Analyzer


View Components

•  Deploy client-side assets (such as HTML + JavaScript)

•  Expose server-side resources (such as REST endpoints)


Views Deployment

•  For Hadoop Operators: •  Deploy Views in an Ambari Server that is managing a Hadoop

cluster

•  For Data Workers: •  Run Views in a dedicated “standalone” Ambari Server

Ambari Server

HADOOP Store & Process

Ambari Views Server

Operators manage the cluster, may have Views deployed

Data Workers use the cluster and use the Ambari Views Server for Views


View Versions and Instances

•  Deploy multiple versions and create multiple instances of a view

•  Manage accessibility and usage


Terminology Term Description

View Definition Describes the view resources and core view properties such as name, version and any necessary configuration properties.

View Package Packages the view client and server assets (and dependencies)

View Deployment

Deploying a view into Ambari

View Version A specific version of a view

View Instance An instantiation of a specific view version

Framework Services

View context, instance data, configuration properties and events


View Packaging

• View Definition: view.xml • Resource / Service classes : JAX-RS annotated • UI classes : html, Servlets deployed as web app (WEB-INF/web.xml).

• Application logic : Supporting classes • Dependencies : 3rd party jars or classes

├── WEB-INF │ └── web.xml ├── org │ └── apache │ └── ambari │ └── view │ └── weather │ ├── CityResource.class │ ├── CityResourceProvider.class │ ├── CityService.class │ └── WeatherServlet.class └── view.xml

view.jar


Learn More

Resource Location

Apache Ambari Project Page http://ambari.apache.org

Ambari Project Wiki https://cwiki.apache.org/confluence/display/AMBARI

Ambari Project JIRA https://issues.apache.org/jira/browse/AMBARI


Extras


Ambari SCOM Overview

Ambari SCOM Mgmt Pack

Cluster Hosts (virtualized or physical)

Ambari Server

HADOOP Store & Process

1

2

3

1

2

3

Ambari aggregates and exposes Hadoop metrics

Ambari SCOM displays cluster layout and Hadoop metric graphs

Ambari SCOM monitors Hadoop health and alerts in case of problems

disclaimer - meetupfiles.meetup.com/1642258/ambari-overview.pdfdisclaimer • this document may...

Documents