© Hortonworks Inc. 2014

Apache Ambari Provisioning, Managing and Monitoring Hadoop

© Hortonworks Inc. 2014

Agenda

• Ambari Overview and the Community • Ambari Architecture • Provisioning Clusters and Services • Extensibility Features

– Ambari Stacks – Ambari Views – Ambari Blueprints

• Releases • Learn More • Q & A

Page 2

© Hortonworks Inc. 2014

Hadoop Operations with Ambari

Provision: Simplified deployment across platforms

Manage: Consistent controls across the Stack

Monitor: Visibility into key cluster metrics -  Single pane of glass for Hadoop & System status -  Pre-configured metrics & alerts

-  Single point for cluster operations -  Advanced configurations and host controls

-  Wizard-driven cluster install experience -  Cloud, virtual and physical environments

Apache Ambari is a 100% open source framework for provisioning, managing and monitoring Hadoop clusters

© Hortonworks Inc. 2014

100% Apache Open Source

• Active Community •  60+ Contributors / 30+ Committers •  200+ Ambari User Group Members

Page 4

2013

Oct Apache Ambari 1.4.1 Released Adds operations for Hadoop 2 Stack

Ambari Ambari 1.2.0 Released Operations for Hadoop 1 Stack

2013

Jan

2013

Dec Apache Ambari Graduates to Top Level Project

© Hortonworks Inc. 2014

Ambari Release Cadence

Page 5

Ambari 1.4.3 Jan 2014

Ambari 1.4.4 Feb 2014

Ambari 1.5.0 Apr 2014

Ambari 1.7.0 2H2014

Ambari 1.6.0 May 2014

© Hortonworks Inc. 2014

Demo #1 Cluster Management

Page 6

© Hortonworks Inc. 2014

System Architecture

7

Ambari Server

Host

Agent

Host

Agent

Host

Agent

Ambari Web

DB

/clusters!

Host

Agent

© Hortonworks Inc. 2014

Ambari Server Architecture

8

DB Orchestrator Monitoring

REST API

Request Dispatcher

Ambari Web

Ambari Server

Ambari Agents

Metrics Providers

Auth P

rovider

/clusters!

User!Repo!

java!

python!

JS!

RDBMS!

LDAP!

REST API

Configurable Auth Provider

Cluster Configurations and Topology

Web Client

AD!

© Hortonworks Inc. 2014

Installing Ambari

• Install Ambari Server • Setup Options

– JDK (Oracle JDK 6, Oracle JDK 7, Custom JDK) – Database (PostgreSQL, MySQL or Oracle) – Server Port and HTTPS – User Authentication (Local, LDAP or AD) – Server-Agent Two-way SSL

• Start Ambari Server • Browse to Ambari Web for Cluster Install Wizard • Install Ambari Agents

– Option #1: Automate via SSH – Option #2: Manual

Page 9

© Hortonworks Inc. 2014

Installing Agents

Page 10

OPTION 1!automate!

provide SSH Key!

OPTION 2!manual!

© Hortonworks Inc. 2014

Option #1: Ambari Agent via SSH

Page 11

Ambari Server Ambari Agent

1) Configure Ambari Repo!

2) Copy Ambari Agent Setup script!

3) Execute Ambari Agent Setup script!

4) Install ambari-agent!

5) Configure ambari-agent.ini!

6) Start ambari-agent!

7) Begin Ambari Agent Registration!

Uses the SSH

© Hortonworks Inc. 2014

Option #2: Ambari Agent via Manual

Page 12

Ambari Server Ambari Agent

2) Install ambari-agent!

3) Configure ambari-agent.ini!

4) Start ambari-agent!

5) Begin Ambari Agent Registration!

1) Configure Ambari repo!

** User executes all ops on the Agent host(s). No requirement for SSH Private Key.

No SSH

© Hortonworks Inc. 2014

Ambari Agent Registration

Page 13

Ambari Server Ambari Agent

11) Agent Heartbeat Begins!

1) Connect on Handshake port 8441!

4) Sign Agent Cert!

2) Download Server Cert!

6) Connect on Registration port 8440!

7) Perform 2WAY auth using Agent Cert!

10) Complete Host Registration!

Agent Host

8) Get FQDN!

9) Register host!

8a) Host

Script!

3) Request to Sign Agent Cert!

5) Download Agent Cert + Disconnect!

© Hortonworks Inc. 2014

Demo #2 Cluster Install

Page 14

© Hortonworks Inc. 2014

Extensibility Features Ambari Stacks Ambari Blueprints Ambari Views

Page 15

© Hortonworks Inc. 2014

Ambari Stacks

Page 16

© Hortonworks Inc. 2014

Ambari Stacks

• Separate “the Stack” from “Ambari”

• Make it easier to bring more than a traditional “Hadoop Stack” under Ambari management

• Defining a consistent Service lifecycle management interface that can be extended

• Dynamically add Stacks + Services definitions

Page 17

© Hortonworks Inc. 2014

Stack Details • Stacks define Services + Repos

– What is in the Stack, and where to get the bits • Each Service has a definition

– What Components are part of the Service • Each Service has defined lifecycle commands

– start, stop, status, install, configure • Lifecycle is controlled via command scripts • Ability to define “custom” commands • Ability to “extend” Stacks

Page 18

AMBARI  SERVER  

Stack

Command  Scripts  

Service  Defini:ons  

AMBARI  AGENT/S  AMBARI  AGENT/S  AMBARI  AGENT/S  

python!xml!

Repos

© Hortonworks Inc. 2014

Stack Mechanics

• Ambari Server reads Stack definitions on start

• Ambari Server sends a command to Agents

• Agents download Stack definition + command scripts

• Agent executes command

• If the Stack definition changes, Agent will request latest Stack definition + command scripts

Page 19

© Hortonworks Inc. 2014

Ambari Blueprints

Page 20

© Hortonworks Inc. 2014

Ambari Blueprints

Page 21

STACK   HOSTS   BLUEPRINT  

© Hortonworks Inc. 2014

Automating Installs •  Blueprint defines a cluster layout and

component configuration for a Stack

•  Provide Blueprint to Ambari to perform cluster installation (no wizard required)

•  Export Blueprint from existing cluster

BLUEPRINT! Ambari  

Submit Blueprint to Ambari Hadoop  

Cluster

Ambari provisions cluster

BLUEPRINT!<stack>!<host>!

<service>!<component>!<config>!

HOST!MANIFEST!<host>!<meta>!

SERVICE!CONFIGS!<props>!

Page 22

© Hortonworks Inc. 2014

Ambari + OpenStack

Page 23

OpenStack  Infrastructure  

Horizon

Savanna

AMBARI  SERVER  

Hadoop  Cluster

© Hortonworks Inc. 2014

Ambari Views

Page 24

© Hortonworks Inc. 2014

Ambari Views

• Goal: Customize the Ambari Web experience

• Views compliment Stack Extensibility – Stack Extensibility makes custom Stack Services available to Ambari – Views expose custom UI features for Services

• Ambari Admins can entitle “views” to Ambari Web users

– Entitlements framework for finer-grained permissions control for Ambari users

Page 25

© Hortonworks Inc. 2014

View Usage

• Developer Creates a View

• Deploys View to Ambari Server

• Ambari Admin creates instances of Views

• Ambari Admin entitles Views to users

Page 26

© Hortonworks Inc. 2014

View Configuration + Entitlement

Page 27

© Hortonworks Inc. 2014

Customized Views

Page 28

“admin” gets operator controls “joe.user” gets specific views based on entitlements

© Hortonworks Inc. 2014

Releases What’s Up Next

Page 29

© Hortonworks Inc. 2014

What’s New in Ambari 1.5.0

• Maintenance Mode • Rolling Restarts • Bulk Host Operations • Decommission TTs, NMs, RSs • Add Service • Refresh Client Configs • Default JDK 7 • For a complete list of changes:

https://issues.apache.org/jira/browse/AMBARI/fixforversion/12325019

Page 30

© Hortonworks Inc. 2014

Ambari Futures

• Operations – Ambari Blueprints – Component log access + search – Ubuntu and Windows support

• Extensibility – Finer-grain user entitlements – Ambari Views

• Insight – Troubleshooting + Performance Tuning – Capacity Planning

Page 31

© Hortonworks Inc. 2014

Learn More

Page 32

Resource Location

Apache Ambari Project Page http://ambari.apache.org

Ambari Project Wiki https://cwiki.apache.org/confluence/display/AMBARI

Ambari Project JIRA https://issues.apache.org/jira/browse/AMBARI

© Hortonworks Inc. 2014

Q & A

Page 33