accumulo summit 2014: monitoring apache accumulo
DESCRIPTION
Speaker: Ravi Mutyala When we started using Apache Accumulo on large scale, our key concern was on monitoring the health of the cluster. Accumulo exposes metrics through JMX. Ganglia and Nagios are the de-facto metrics and monitoring tools for hadoop clusters. We identified that integration with ganglia, nagios and Apache Ambari will provide ease of use both for monitoring and managing Accumulo clusters. We started with ganglia and nagios integration which helps reuse all the hadoop monitoring infrastructure for Accumulo. Our next target is Apache Ambari integration for Accumulo. In this talk, we focus on why we need to integrate and how this can be done. We will show a Hands On for ganglia and nagios integration and share the status of ambari integration.TRANSCRIPT
© Hortonworks Inc. 2014
Monitoring Apache Accumulo
© Hortonworks Inc. 2014
Your Presenter
Ravi MutyalaSystems Architect
Hortonworks
© Hortonworks Inc. 2014
Agenda
• Metrics
• Alerts
• Provisioning and Managing
• Q & A
Page 3
© Hortonworks Inc. 2014
Metrics
• JMX
• Ganglia
• Jmxtrans
Page 4
© Hortonworks Inc. 2014
Alerts
• Nagios
–Monitoring Ports/Processes– TabletServers
– GC
– Tracer
– Masters
–Monitoring Metrics
Page 5
© Hortonworks Inc. 2014
Hadoop Operations with Ambari
Provision: Simplified deployment across platforms
Manage: Consistent controls across the Stack
Monitor: Visibility into key cluster metrics- Single pane of glass for Hadoop & System status- Pre-configured metrics & alerts
- Single point for cluster operations- Advanced configurations and host controls
- Wizard-driven cluster install experience- Cloud, virtual and physical environments
Apache Ambari is a 100% open source framework for
provisioning, managing and monitoring Hadoop clusters
© Hortonworks Inc. 2014
100% Apache Open Source
• Active Community• 60+ Contributors / 30+ Committers• 200+ Ambari User Group Members
Page 7
2013
OctApache Ambari 1.4.1 ReleasedAdds operations for Hadoop 2 Stack
Ambari Ambari 1.2.0 ReleasedOperations for Hadoop 1 Stack
2013
Jan
2013
DecApache Ambari Graduates to Top Level Project
© Hortonworks Inc. 2014
Ambari Release Cadence
Page 8
Ambari 1.4.3Jan 2014
Ambari 1.4.4Feb 2014
Ambari 1.5.0Apr 2014
Ambari 1.7.02H2014
Ambari 1.6.0May 2014
© Hortonworks Inc. 2014
System Architecture
9
Ambari Server
Host
Agent
Host
Agent
Host
Agent
Ambari Web
DB
/clusters
Host
Agent
© Hortonworks Inc. 2014
Ambari Server Architecture
10
DBOrchestrator Monitoring
REST API
Request Dispatcher
Ambari Web
Ambari
Server
Ambari
Agents
Metrics
Providers
Auth P
rovider
/clusters
UserRepo
java
python
JS
RDBMS
LDAP
REST API
Configurable Auth Provider
Cluster Configurations and Topology
Web Client
AD
© Hortonworks Inc. 2014
Installing Ambari
• Install Ambari Server• Setup Options
– JDK (Oracle JDK 6, Oracle JDK 7, Custom JDK)–Database (PostgreSQL, MySQL or Oracle)–Server Port and HTTPS–User Authentication (Local, LDAP or AD)–Server-Agent Two-way SSL
• Start Ambari Server• Browse to Ambari Web for Cluster Install Wizard• Install Ambari Agents
–Option #1: Automate via SSH–Option #2: Manual
Page 11
© Hortonworks Inc. 2014
Ambari Agent Registration
Page 12
Ambari Server Ambari Agent
11) Agent Heartbeat Begins
1) Connect on Handshake port 8441
4) Sign Agent Cert
2) Download Server Cert
6) Connect on Registration port 8440
7) Perform 2WAY auth using Agent Cert
10) Complete Host Registration
Agent Host
8) Get FQDN
9) Register host
8a) Host Script
3) Request to Sign Agent Cert
5) Download Agent Cert + Disconnect
© Hortonworks Inc. 2014
Extensibility FeaturesAmbari StacksAmbari BlueprintsAmbari Views
Page 13
© Hortonworks Inc. 2014
Ambari Stacks
Page 14
© Hortonworks Inc. 2014
Ambari Stacks
• Separate “the Stack” from “Ambari”
• Make it easier to bring more than a traditional “Hadoop Stack” under Ambari management
• Defining a consistent Service lifecycle management interface that can be extended
• Dynamically add Stacks + Services definitions
Page 15
© Hortonworks Inc. 2014
Stack Details
• Stacks define Services + Repos– What is in the Stack, and where to get the bits
• Each Service has a definition– What Components are part of the Service
• Each Service has defined lifecycle commands– start, stop, status, install, configure
• Lifecycle is controlled via command scripts• Ability to define “custom” commands• Ability to “extend” Stacks
Page 16
AMBARISERVER
Stack
Command Scripts
Service Definitions
AMBARIAGENT/SAMBARI
AGENT/SAMBARIAGENT/S
pythonxml
Repos
© Hortonworks Inc. 2014
Stack Mechanics
• Ambari Server reads Stack definitions on start
• Ambari Server sends a command to Agents
• Agents download Stack definition + command scripts
• Agent executes command
• If the Stack definition changes, Agent will request latest Stack definition + command scripts
Page 17
© Hortonworks Inc. 2014
Ambari Blueprints
Page 18
STACK HOSTS BLUEPRINT
© Hortonworks Inc. 2014
Automating Installs
• Blueprint defines a cluster layout and component configuration for a Stack
• Provide Blueprint to Ambari to perform cluster installation (no wizard required)
• Export Blueprint from existing cluster
BLUEPRINT Ambari
Submit Blueprint
to Ambari HadoopCluster
Ambari provisions
cluster
BLUEPRINT<stack><host>
<service><component><config>
HOSTMANIFEST<host><meta>
SERVICECONFIGS<props>
Page 19
© Hortonworks Inc. 2014
Ambari Views
• Goal: Customize the Ambari Web experience
• Views compliment Stack Extensibility– Stack Extensibility makes custom Stack Services available to Ambari– Views expose custom UI features for Services
• Ambari Admins can entitle “views” to Ambari Web users– Entitlements framework for finer-grained permissions control for Ambari users
Page 20
© Hortonworks Inc. 2014
View Usage
• Developer Creates a View
• Deploys View to Ambari Server
• Ambari Admin creates instances of Views
• Ambari Admin entitles Views to users
Page 21
© Hortonworks Inc. 2014
Ambari Futures
• Operations–Ambari Blueprints–Component log access + search–Ubuntu and Windows support
• Extensibility–Finer-grain user entitlements–Ambari Views
• Insight–Troubleshooting + Performance Tuning–Capacity Planning
Page 22
© Hortonworks Inc. 2014
Learn More
Page 23
Resource Location
Apache Ambari Project Page http://ambari.apache.org
Ambari Project Wiki https://cwiki.apache.org/confluence/display/AMBARI
Ambari Project JIRA https://issues.apache.org/jira/browse/AMBARI
© Hortonworks Inc. 2014
Q & A
Page 24