hortonworks technical workshop: apache ambari

Apache Ambari WorkshopStacks, Blueprints, Views

We Do Hadoop

Ajay Singh, Ali Bajwa, Sean RobertsPartner Solution Engineering

March-2015

Who

Ajay Singh- Director- [email protected]

Ali Bajwa- Engineer- [email protected]

Sean Roberts- Engineer, London & EMEA- @seano

Schedule

● Ambari Overview● Stacks● Blueprints● Stack Services● Views

© Hortonworks Inc. 2011 – 2015. All Rights Reserved

How do you Operate a Hadoop Cluster?

Apache Ambari is a platform to provision, manage and monitor

Hadoop clusters


Apache Ambari Themes

Operate Hadoop at Scale

Deliver the core operational capabilities to provision, manage and monitor Hadoop clusters at scale.

Integrate with the Enterprise

Robust API for integration with existing enterprise systems, such as Teradata Viewpoint and Microsoft SCOM.

Extend for the Ecosystem

Provide extensible platform for Customers, Partners and the Community to, such as Stacks and Views.


Hadoop Operations Platform

Apache Ambari is a 100% open source platform for provisioning, managing and monitoring Apache Hadoop clusters

AMBARI WEB

Others

compute&

storage. . .

. . .

. . compute&

storage

.

. EXTEND

AMBARI REST API

AMBARI SERVERPROVISION | MANAGE | MONITOR

Integration with Existing Tools

OPERATE

AMBARI STACKS


Innovation at the Speed of CommunityAn active 100% open source Community:

• 90+ Contributors / 45+ Committers

• 270+ Ambari User Group Members

2014

Dec

Graduates to Apache Top Level Project2013

Dec

Apache Ambari 1.7 ReleasedAdds support for Hortonworks Data Platform 2.2Adds new operational and extensibility capabilities


Recent Ambari Releases

Ambari 1.6.0May 2014

Ambari 1.5.1Apr 2014

Ambari 1.5.0Apr 2014

Ambari 1.6.1Jul 2014

IntroducedAmbari Stacks

IntroducedAmbari Blueprints

HDP2.1 GA

Ambari 1.7.0Dec 2014

HDP2.2 GA

IntroducedAmbari Views


Extensibility Features

• To add new Services (ISV or otherwise) beyond HDP Stack• To customize a Stack for customer specific environments

• To use Ambari for automating cluster installations• To share best practices on layout and cluster configuration

• To extend and customize the Ambari Web UI• Add new capabilities, customize existing capabilities

Stacks

Blueprints

Views

Goal: Extend Ambari without hard-coding in Ambari

1.5.0

1.7.0

1.6.0


Anatomy of Ambari Extension Points

AmbariServer

AmbariAgentAmbari

AgentAmbariAgent

AmbariWeb

StacksStacks

Stacks

javajs python

Ambari Views

Ambari Stacks

Ambari Blueprints

Ambari API


Partners Integrating With Ambari

Ambari Community

Apache Ambari: http://ambari.apache.org ● wiki: https://cwiki.apache.org/confluence/display/AMBARI/Ambari ● mail lists: https://ambari.apache.org/mail-lists.html ● issues: https://issues.apache.org/jira/browse/AMBARI

Hortonworks:● forum: http://hortonworks.com/community/forums/forum/ambari/● social media: twitter, facebook, slideshare, linkedin, ...

http://ambari.apache.org

https://cwiki.apache.org/confluence/display/AMBARI/Ambari

https://ambari.apache.org/mail-lists.html

https://issues.apache.org/jira/browse/AMBARI

http://hortonworks.com/community/forums/forum/ambari/

Partner Resources

This Workshop:● github: http://git.io/jmz4 (https://github.com/abajwa-hw/ambari-workshops)● blog: http://hortonworks.com/blog/apache-ambari-technical-workshop/

Past & Future Workshops:● http://hortonworks.com/partners/learn/

Upcoming meetups and events:● http://hortonworks.com/events/

http://git.io/jmz4

https://github.com/abajwa-hw/ambari-workshops

http://hortonworks.com/blog/apache-ambari-technical-workshop/

http://hortonworks.com/partners/learn/

http://hortonworks.com/partners/learn/

http://hortonworks.com/events/

http://hortonworks.com/events/

Ambari Stacks


Stack TerminologyTerm Definition Examples

STACK Defines a set of Services, where to obtain the software packages and how to manage the lifecycle. HDP-2.0, HDP-1.3.3

SERVICE Defines the Components that make-up the Service. HDFS, NAGIOS, YARN

COMPONENT The building-blocks of a Service, that adhere to a certain lifecycle.

NAMENODE, DATANODE, OOZIE_SERVER

CATEGORY The category of Component. MASTER, SLAVE, CLIENT


Stack MechanicsStacks define Services + ReposWhat is in the Stack, and where to get the bits

Each Service has a definitionWhat Components are part of the Service

Each Service has defined lifecycle commandsstart, stop, status, install, configure

Lifecycle is controlled via command scriptsAbility to define “custom” commands

AMBARISERVER

Stack

Command Scripts

Service Definitions

AMBARIAGENT/S

AMBARIAGENT/S

AMBARIAGENT/S

pythonxml

Repos


Stacks Support Inheritance

HDP 2.0 Stack

HDP 2.1 Stack

• Defines a set of Service definitions• Default service configurations and command scripts

• Overrides any Service definitions, commands and configurations

• Adds new Services specific to this Stack

Deploying clusters of any scale with ease

Ambari Blueprints

Before Ambari

● Manual installation● Manual configuration● Manual distribution of configuration● XML● XML● XML ...

Manual Install with Ambari Wizard

Automation with Ambari Blueprints

Stack DefinitionComponent

Layout& Configuration

BLUEPRINT

BLUEPRINT INSTANTIATE CLUSTER

2 HTTP calls for a Deployed Cluster

{ "Blueprints": { "stack_name": "HDP", "stack_version": "2.2" }, "host_groups": [ { "name": "workers", "components": [ { "name": "DATANODE" }, { "name": "JOURNALNODE" }, { … … } }, { "name": "master-1", "components": [ { "name": "NAMENODE" }, { … … } }]}

{ "Blueprints": { "blueprint": "myblueprint", "default_password": "awesome", "configurations": [ { "hive-site": { "properties": { "hive.execution.engine": "tez" }}}], "host_groups": [ { "name": "master-1", "hosts": [ { "fqdn": "hdp00" } ]}, { "name": "workers" "hosts": [ { "fqdn": "hdp01" }, { "fqdn": "hdp01" }, { "fqdn": "hdp.." }, { "fqdn": "hdp99" }]}]}

1) POST /api/v1/blueprints/myblueprint 2) POST /api/v1/clusters/mycluster

Export Cluster to BlueprintGET /api/v1/clusters/my_cluster?format=blueprint

{ "configurations" : [ { "cluster-env" : { "cluster_name" : "sandbox", ... }}, { "hive-site" : { "hive.execution.engine" : "tez", ... }} ], "host_groups" : [ { "name" : "host_group_1", "configurations" : [ ], "components" : [ { "name" : "NAMENODE" }, { "name" : "HIVE_SERVER"}, ... ], "cardinality" : "1" }], "Blueprints" : { "stack_name" : "HDP", "stack_version" : "2.2" }}

Live Demo

Command snippets: http://nbviewer.ipython.org/github/seanorama/ambari-bootstrap/blob/master/api-examples/ambari-workshop-blueprints.ipynb

For API examples including Python versions in an ipython notebook:https://github.com/seanorama/ambari-bootstrap/tree/master/api-examples

http://nbviewer.ipython.org/github/seanorama/ambari-bootstrap/blob/master/api-examples/ambari-workshop-blueprints.ipynb



https://github.com/seanorama/ambari-bootstrap/tree/master/api-examples

https://github.com/seanorama/ambari-bootstrap/tree/master/api-examples

Ambari Stack Services


Files needed to create an Ambari service

• Service definition file (metainfo.xml)

• Service configuration file (myservice-config.xml)

• Service configuration parameters (params.py)

• Service master python script (master.py)

Not always needed:

• Service status configuration file (myservice-env.xml)

• Service status configuration parameters (status_params.py)


Service Definition (metainfo.xml)● name – the internal name of the service (must be unique)● displayName – the public display name of the service● version – the version of the service● component category – is the component a MASTER or CLIENT (one or more)● script – the script that will be executed during service install, start, stop● package – packages to be installed via yum during install

<service> <name>MAVEN</name> <displayName>Maven</displayName> <comment>Apache Maven Project Build Tool</comment> <version>1.0.0</version> <components> <component> <name>MAVEN_CLIENT</name> <displayName>Maven</displayName> <category>CLIENT</category> <commandScript> <script>scripts/client.py</script> <scriptType>PYTHON</scriptType>


Service Configuration (myservice-config.xml)

● name – the internal name of property (must be unique)● value – default value for property● property-type – e.g. whether property is password (optional)● description – tool tip help text for property

<configuration> <property require-input="true"> <name>vnc.password</name> <value></value> <property-type>PASSWORD</property-type> <description>VNC password</description> </property> <property> <name>vnc.geometry</name> <value>1400x860</value> <description>VNC screen geometry/resolution</description> </property>


Service Status Configuration (myservice-env.xml)

● status related parameters (e.g. location of file containing pid of service)● name – the internal name of property (must be unique)● value – default value for property● description – tooltip help text for property

<configuration> <property> <name>pid_file</name> <value>/var/run/ipython-notebook/notebook.pid</value> <description>Process ID File</description> </property> </configuration>


Service Configuration Params (params.py)

Python script to read configuration properties from myservice–config.xml

from resource_management import *

# server configurationsconfig = Script.get_config()

user = config['configurations']['ipython-config']['ipython.user']group = config['configurations']['ipython-config']['ipython.group']


Service Status Config Params (status_params.py)

Python script to read status related configuration properties from myservice–env.xml

from resource_management import *

config = Script.get_config()pid_file = config['configurations']['ipython-env']['pid_file']


Service master python script (master.py) - simpleIf the Ambari service is a just wrapper around a Linux service,

the start/stop/status methods as as easy as…

def start(self, env): Execute("service myservice start")

def status(self, env): Execute("service myservice status")

def stop(self, env): Execute("service myservice stop")


Service master python script (master.py)● def install – called to install service● def start – called to start service● def stop – called to stop the service● def status – called to get status of service● def configure – called to configure service

def install(self, env): #Install packages from metainfo.xml self.install_packages(env) self.configure(env) #code to install service def start(self, env): import params import status_params

#code to start service

def configure(self, env): import params env.set_params(params) #code to re-configure service

def stop(self, env): import status_params env.set_params(status_params) self.configure(env) Execute (format('kill `cat {stack_pidfile}` >/dev/null 2>&1')) Execute (format("rm -f {stack_pidfile}"))

def status(self, env): import status_params env.set_params(status_params) check_process_status(status_params.stack_pidfile)

Let’s build some services!

Demo


Demo• Build a MASTER service

• Ntpd

• Sample CLIENT service walk through• Maven

• Use VNC service to setup a developer sandbox • Installs Eclipse, IntelliJ, Maven, Spark

• Real world scenarios• Data Science: iPython service to easily manage platform for data science on Hadoop• Security: OpenLDAP, KDC, NSLCD services to easily setup security with minimal CLI work

Ambari Views


Ambari Views Framework

Goal: enable the delivery of custom UI experiences in Ambari Web

Developers can extend the Ambari Web interface• Views expose custom UI features for Hadoop Services

Ambari Admins can entitle Views to Ambari Web users• Entitlements framework for controlling access to Views


Views Framework

Views Framework vs. Views

Page 38

Views

Core to Ambari

Built by Hortonworks, Community,

Partners


Views Framework

Views Framework vs. Views

Page 39

Views

Core to Ambari

Built by Hortonworks, Community,

Partners


View Components

• Serve client-side assets (such as HTML + JavaScript)

• Expose server-side resources (such as REST endpoints)

VIEWClient-side

assets(.js, html)

AMBARI WEB

VIEWServer-side resources

(java)

AMBARI SERVER

{rest}

Hadoop and

other systems


View Delivery

1. Develop the View (just like you would for a Web App)

2. Package as a View (basically a WAR)

3. Deploy the View into Ambari

4. Ambari Admins create + configuration view instance(s) and give access to users + groups

Develop DeployPackageCreate

Instance(s)


Versions and Instances

• Deploy multiple versions and create multiple instances of a view

• Manage accessibility and usage


Choice of Deployment Model

• For Hadoop Operators:Deploy Views in an Ambari Server that is managing a Hadoop

cluster

• For Data Workers:Run Views in a “standalone” Ambari Server

AmbariServer

HADOOPStore & Process

AmbariServer

Operators manage the cluster, may have Views deployed

Data Workers use the cluster and use a “standalone” Ambari Server for Views


Views REST API

Resource Description

GET /api/v1/views List of available views

GET /api/v1/views/{viewName}/versions List of available view versions

GET /api/v1/views/{viewName}/versions/{version}/instances

List of view instances in a given version

GET /api/v1/views/{viewName}/versions/{version}/instances/{instanceName} Details about a specific view instance.


View Deployment

Place view-package.jar on Ambari Server/var/lib/ambari-server/resources/views

Restart Ambari Server to start deployment

ambari-server restart

Deployment registers the view with Ambari and extracts the view/var/lib/ambari-server/resources/views/work

Once complete, the view is available to create instances

Dev Tip: UI assets in extracted view can be modified w/o restart


Views Framework Terminology

Term Description

View Definition Describes the view resources and core view properties such as name, version and any necessary configuration properties.

View Package Packages the view client and server assets (and dependencies)

View Deployment Deploying a view into Ambari

View Version A specific version of a view

View Instance An instantiation of a specific view version

Framework Services View context, instance data, configuration properties and events


View Definition (view.xml)name – the internal name of the view (must be unique)label – the public display name of the viewversion – the version of the viewparameter – (optional) metadata about view instance propertiesresource – (optional) the names of the classes required to support a view sub-resourceinstance – (optional) property sets which define static view instances

<view> <name>MYVIEW</name> <label>My View</label> <version>1.0.0</version> <parameter> <name>scope</name> <description>The scope of the instance.</description> </parameter></view>

Dev Tip: Use Static instances for development


Maven build file (pom.xml)

artifactId – determines name of jar

ambari-views 1.7.0.0 dependency ensures jar not required locally<modelVersion>4.0.0</modelVersion> <artifactId> hello-world-view</artifactId> <packaging>jar</packaging> <name>Hello World View</name> <groupId>com.github.hortonworks</groupId> <version>1.0-SNAPSHOT</version>

<dependencies> <dependency> <groupId>org.apache.ambari</groupId> <artifactId> ambari-views</artifactId> <version> 1.7.0.0</version> </dependency>


View Package

View Definition: view.xml

Resource / Service classes : JAX-RS annotated

UI classes : html, Servlets deployed as web app (WEB-INF/web.xml).

Application logic : Supporting classes

Dependencies : 3rd party jars or classes├── WEB-INF│ └── web.xml├── org│ └── apache│ └── ambari│ └── view│ └── weather│ ├── CityResource.class│ ├── CityResourceProvider.class│ ├── CityService.class│ └── WeatherServlet.class└── view.xml

view-package.jar


View Context

• Available to the view server-side through injection.• Provides access to instance configuration.• Provides access to the view and instance attributes.• Provides access to run time information about the current

execution context, including authenticated user principal.public interface ViewContext { public String getUsername(); public String getViewName(); public String getInstanceName(); public Map<String, String> getProperties(); public void putInstanceData(...); public String getInstanceData(...);}


Views DemoLet’s build some views!


Demo• Build example view with only HTML

• Hello World• iFrame view

• Sample views with HTML and javascript w/o frameworks• REST API explorer • Google map

• Sample views using javascript framework• HDFS Visualizer

• Sample servlet view (uses java) • Hive query view

The End & Questions

The End & Questions

This Workshop:● github: https://github.com/abajwa-hw/ambari-workshops● blog: http://hortonworks.com/blog/apache-ambari-technical-workshop/

Apache Ambari: http://ambari.apache.org/ ● wiki: https://cwiki.apache.org/confluence/display/AMBARI/Ambari ● mail lists: https://ambari.apache.org/mail-lists.html ● issues: https://issues.apache.org/jira/browse/AMBARI

Hortonworks:● forum: http://hortonworks.com/community/forums/forum/ambari/● social media: twitter, facebook, slideshare, linkedin, ...

https://github.com/abajwa-hw/ambari-workshops

http://hortonworks.com/blog/apache-ambari-technical-workshop/

http://ambari.apache.org/

https://cwiki.apache.org/confluence/display/AMBARI/Ambari

https://ambari.apache.org/mail-lists.html

https://issues.apache.org/jira/browse/AMBARI

http://hortonworks.com/community/forums/forum/ambari/

Appendix


Ambari Views: Where Can I Learn More?

https://cwiki.apache.org/confluence/display/AMBARI/Views

https://github.com/apache/ambari/blob/trunk/ambari-views/docs/index.md

https://github.com/apache/ambari/tree/trunk/ambari-views/examples

https://github.com/apache/ambari/tree/trunk/contrib/views










Ambari Views: Developer resources

https://nodesource.com/blog/your-first-nodejs-package

https://docs.angularjs.org/tutorial/

http://www.jeromecukier.net/blog/2012/09/04/getting-to-hello-world-with-d3/

http://jsbin.com (e.g. http://jsbin.com/hirirazu/3/edit?html,output)

More sample views:









http://jsbin.com

http://jsbin.com/hirirazu/3/edit?html,output

http://jsbin.com



hortonworks technical workshop: apache ambari

Technology

apache ambari workshopstacks

ambari web ui

hortonworks data platform

rights reservedinnovation

rights reservedpartners

rights reservedhow

open source platform

monitor hadoop clusterspage