ambari meetup: apis and spis of ambari

© Hortonworks Inc. 2013

AmbariAPI's and SPI's of Ambari

Tom Beerbower @ Hortonworks

April 02, 2013


Agenda

• API Overview• Monitoring• API Constructs• Management• Error Handling• SPI Overview• SPI Usage


API Overview - Features

• REST• Monitoring and Management of Hadoop Cluster• Partial Response• Query Predicates


Monitoring

• Read state of Hadoop resources.

Example:api/v1/clusters/cluster1/services/HDFS/components/NAMENODE{ "href" : "http://ec2…1/services/HDFS/components/NAMENODE", "metrics" : { "boottime" : 1.364912931E9, "process" : { "proc_total" : 752.333333333, "proc_run" : 1.64444444444 }, "rpc" : { "rpcAuthorizationSuccesses" : 141, "SentBytes" : 251758, "rpcAuthorizationFailures" : 0,…


API Overview – Resource Types

• Collection Resource: This resource type doesn’t refer to any specific resource; rather it refers to a collection of resources.

For example: api/v1/clusters/cluster1/services

Returns a collection of services

• Instance Resource: This resource type refers to a single specific resource.

For example: api/v1/clusters/cluster1/services/HDFS

Refers to the service resource identified by the id “HDFS”.


API Overview - Response

• JSON• Collection resource

• Always includes request href.• Inlcludes “items” of the collection.• Each item includes href and primary id fields.

• Instance resource• Always includes request href.• Always includes primary id fields.• May include metrics and properties for instance. • May include href and primary id fields for sub-resources.



Collection Resource Response Example:

api/v1/clusters/cluster1/services/ { "href" : "http://ec2…api/v1/clusters/cluster1/services/", "items" : [ { "href" : "http://ec2…/clusters/cluster1/services/HDFS", "ServiceInfo" : { "cluster_name" : "cluster1", "service_name" : "HDFS" } }, { "href" : "http://ec2…/cluster1/services/MAPREDUCE",…



Instance Resource Response Example: api/v1/clusters/cluster1{ "href" : "http://ec2...com:8080/api/v1/clusters/cluster1", "Clusters" : { "cluster_name" : "cluster1", "cluster_id" : 1, "version" : "HDP-1.2.1" }, "services" : [ { "href" : "http://ec2...com:8080/api/v1/clusters/cluster1/services/MAPREDUCE", "ServiceInfo" : { "cluster_name" : "cluster1", "service_name" : "MAPREDUCE" } }, …


API Constructs – Partial Response

• Used to control which fields are returned by a query.– restrict which fields are returned.– reach down and return data from sub-resources.

• Properties, categories and sub-resources can be specified.

• The wildcard ‘*’ can be used to show all…– categories, fields and sub-resources.–provides ‘expand’ functionality for sub-components.

• Primary id fields of a resource are always shown regardless of the specifies partial response.



Query Specific Field Example:api/v1/clusters/cluster1/services/MAPREDUCE/components/JOBTRACKER?fields=metrics/rpc/SentBytes

{ "href" : "http://…/api/v1/clusters/cluster1/services/MAPREDUCE/components/JOBTRACKER?fields=metrics/rpc/SentBytes", "metrics" : { "rpc" : { "SentBytes" : 2763921 } }, "ServiceComponentInfo" : { "cluster_name" : "cluster1", "component_name" : "JOBTRACKER", "service_name" : "MAPREDUCE" }}



Expand Sub-Resource Example:api/v1/clusters/cluster1/services/MAPREDUCE?fields=components/metrics/jvm/gcCount{ "href" : "http://…/services/MAPREDUCE?fields=components/metrics/jvm/gcCount", … }, "components" : [ { "href" : "http://…/MAPREDUCE/components/JOBTRACKER", "metrics" : { "jvm" : { "gcCount" : 47 } },…


API Constructs – Query Predicates

• Limits the set of resources returned by a query.• Consists of at least one relational expression.• Can only be applied to collection resources.• Relational operators ( =, !=, <, >, <=, >= )• Logical operators ( |, &, !)• Functions (in(), isEmpty())• Brackets can be used to provide explicit grouping of expressions. Expressions within brackets have the highest precedence.


API Constructs – Query Predicates

Query For Started Services Example:api/v1/clusters/cluster1/services?ServiceInfo/state=STARTED{ "href" : "http://…/clusters/cluster1/services?ServiceInfo/state=STARTED", "items" : [ { "href" : "http://…/clusters/cluster1/services/NAGIOS", "ServiceInfo" : { "cluster_name" : "cluster1", "state" : "STARTED", "service_name" : "NAGIOS" } },…


Management - Create

• Create a resource.

Example:

Create a cluster named ‘c1’ with the property ‘Clusters/version’ = ‘HDP-1.2.0’

POST http://…:8080/api/v1/clusters/c1{ "Clusters": { "version" : "HDP-1.2.0” }}


Management - Update

• Update a resource.

Example:

Update the state of all ‘INSTALLED’ services to be ‘STARTED’PUT http://…:8080/api/v1/clusters/c1/services?ServiceInfo/state=INSTALLED { "ServiceInfo": { "state" : "STARTED” }}


Management - Delete

• Delete a resource.

Example:

Delete the cluster named ‘c1’

DELETE http://…:8080/api/v1/clusters/c1


Error Handling

HTTP Code Description

200 OK

400 Bad Request

401 Unauthorized

403 Forbidden

404 Not Found

500 Internal Server Error


Error Handling - Examples

api/v1/clusters/BAD_CLUSTER_NAME

404 Not Found{ "status" : 404, "message" : "The requested resource doesn't exist: Cluster not found,clusterName=BAD_CLUSTER_NAME"}

api/v1/clusters/cluster1?fields=BAD_FIELD

400 Bad Request{ "status" : 400, "message" : "The properties [BAD_FIELD] specified in the request or predicate are not supported for the resource type Cluster."}


SPI Overview

• Service framework for handling requests for the various resources that it manages and monitors.

ClusterController

Handler

ClusterResourceProvider

ServiceResourceProvider

HostResourceProvider

ComponentResourceProvider

HostComponentResourceProvider

/api/v1/clusters/c1/hosts/host1/host_components

getResources(Resource.Type.HostComponent, request, predicate)

getResources(request, predicate)


SPI Use Case

• Pluggable resource providers allow API usage with a GSInstaller installed cluster.

API

Ambari DB

Default ResourceProviders

SPI

GSInstaller Artifacts

GSInstaller ResourceProviders


Q & A

ambari meetup: apis and spis of ambari

Documents

service resource

requested resource

post http

c1delete http

asingle specific resource

request href

started services example

collection resources