manage hadoop cluster with ambari

Mr. Charnsilp Chinprasert Email: charnsilp.chin@gmail.com

Apache Ambari A Hadoop management simpler

Apache Hadoop

The Apache Hadoop software library is a framework that allows for the distributed processing of large data sets across clusters of computers using simple programming models.

Hadoop Cluster

Install Cluster ยังไง?

แก้ไข Config ทั้ง Cluster ยังไง?

Hadoop Ecosystem

Install แต่ละ Service ยังไง?

Config ยังไง

แต่ละ Service ไม่เหมือนกัน?

จัดการแต่ละ Service ยังไง?

จะรู้ได้ยังไงว่า Service ยังปกติ?

What is Apache Ambari?

The Apache Ambari a 100% open source platform provisioning, managing and monitoring Apache Hadoop Cluster

Hadoop Ecosystem

Apache Ambari

Provisioning

Monitoring Management

Apache Ambari• Agent Installation• Service Installation

• Service Management• Cluster Configuration

• OverAll Dashboard• By Host• By Service• Alert Trigger

Provisioning

Monitoring Management

Welcome Ambari!

Install Cluster ยังไง?

System Architecture

Ambari Server

Linux Server

Linux Server

Ambari AgentLinux

Server

Ambari Agent

Linux Server

Ambari Agent

Ambari Agent

Port: 8440/8441(SSL)

Ambari Web

Ambari Api

Installation Agent❖ Provisioning by remote Secure Shell (SSH)

❖ OS Support

❖ RHEL (Redhat Enterprise Linux)

❖ SLES (SuSE Linux Enterprise Server)

❖ Ubuntu

❖ Debian

Ambari Server

Linux Server

Ambari Server

Linux Server

Ambari Agent

SSH SSH

ambari-server start

Setup Cluster

Installation Agent

• Cluster name• Stacks version

• Remote Repos• Local Repos

Installation Agent• Target Hosts

• FQDN• SSH private key

Installation AgentHost Checks

• มีกระบวนการตรวจสอบและconfig OS ให้พร้อมกัน Environment ของHadoop

• ทำให้ Linux ที่นำมาติดตั้งไม่จำเป็นต้อง ลงApplication ใดๆเลย • ถ้ายังไม่ได้ Install หรือ Config Ambari จะจัดการให้• ใช้ Puppet

Install & Configure ServicesSelect

Services

Assign Masters

Assign Slaves & Clients

Customize Services

Review & Confirm

Install & Configure ServicesSelect

Services

Assign Masters

Assign Slaves & Clients

Customize Services

Review & Confirm

Install & Configure ServicesSelect

Services

Assign Masters

Assign Slaves & Clients

Customize Services

Review & Confirm

Install & Configure ServicesSelect

Services

Assign Masters

Assign Slaves & Clients

Customize Services

Review & Confirm

Install & Configure ServicesSelect

Services

Assign Masters

Assign Slaves & Clients

Customize Services

Review & Confirm

Installing

Installing

Installing

Welcome Ambari!

จัดการแต่ละ Service ยังไง?

Service ManagementHDFS Yarn

Service Management

Cluster Configuration

Move namenode ได้

Move namenodeService

Management

Cluster Configuration

Move namenodeService

Management

Cluster Configuration

Move namenodeService

Management

Cluster Configuration

Move namenodeService

Management

Cluster Configuration

Move namenodeService

Management

Cluster Configuration

แก้ไขConfig ทั้ง Cluster ยังไง?

Cluster Configuration

Config ที่เดียว มีผลทั้งCluster

Basic View สำหรับ Configส่วนที่จำเป็นและสำคัญ!!

Service Management

Cluster Configuration

Cluster Configuration

ถ้ามีแก้ไขConfig ระบบจะแนะนำConfigตัวอื่นๆที่ตวรแก้ไขตาม

Service Management

Cluster Configuration

Cluster Configuration

Compare Code ระหว่าง version ได้!!!

Service Management

Cluster Configuration

Configuration History LogService

Management

Cluster Configuration

จะรู้ได้ไงว่าที่เราConfigไปแล้วมันดี?

จะรู้ได้ยังไงว่า Service ยังปกติ?

Monitoring

DashboardDashboard

By Service

By Host

Alert Trigger

By ServiceDashboard

By Service

By Host

Alert Trigger

By HostDashboard

By Service

By Host

Alert Trigger

Alert TriggerDashboard

By Service

By Host

Alert Trigger

Apache Ambari• Agent Installation• Service Installation

• Service Management• Cluster Configuration

• OverAll Dashboard• By Host• By Service• Alert Trigger

Provisioning

Monitoring Management

Q&A

• https://ambari.apache.org• https://issues.apache.org/jira/secure/attachment/12559939/Ambari_Architecture.pdf• http://www.slideshare.net/hortonworks/ambari-agentregistrationflow-17041261• https://cwiki.apache.org/confluence/display/AMBARI/Technology+Stack• http://www.throughwave.co.th/2011/11/10/enterprise-apache-hadoop/• http://www.slideshare.net/hortonworks/apache-ambari-whats-new-in-200

Thank you