Application Timeline Server- Past, Present & Future NAGANARASIMHA G R & VARUN SAXENA

Agenda Who we are ?

Why we need History Server?

Application History Server

Timeline Server V1

Timeline Server V2

Who we are ?Naganarasimha G R

Senior Technical Lead @ Huawei

Active Apache Hadoop Contributor.

Currently working in Hadoop Platform Dev team

Earlier worked in Reporting Domain

Varun Saxena Technical Lead @ Huawei

Active Apache Hadoop Contributor.

Currently working in Hadoop Platform Dev team

Earlier worked in Telecom Data Network Domain

Both of us are currently participating in ATS V2 development

Agenda Who we are ?

Why we need History Server?

Application History Server

Timeline Server V1

Timeline Server V2

Need for new History Server Job History server is only for MR app, YARN supports

many Applications.

YARN level Events and Metrics are not captured.

Storage is HDFS only, Not good for adhoc analysis.

JHS is only for historical or completed jobs.

On failures of Application Master, Data for current running application is lost.

Storage is very MR specific- Counters- Mappers and Reducers

Agenda Who we are ?

Why we need History Server?

Application History Server

Timeline Server V1 & V1.5

Timeline Server V2

Application History Server

Separate Process

Resource Manager directly writes to Storage(HDFS)

Aggregated Logs

Separate UI, CLI and Rest End Point

Data stored :- Application level data (queue, user etc…)- List of ApplicationAttempts- Information about each ApplicationAttempt- List of containers for ApplicationAttempt- Generic information about each container.

CLI and REST Query interfaces were supported

Drawbacks :

Storing Application specific custom data not supported

RM crashes, HDFS files are not readable

Hard limit no number of Files

Upgrades / Update

Supports only completed jobs.

Agenda Who we are ?

Why we need History Server?

Application History Server

Timeline Server V1

Timeline Server V2

Application Timeline Service Motivation : YARN takes care of it

- Relieving the application from monitoring service Application diversity

- Framework specific metadata/metrics

ATS V1 : Data Model Timeline Domain

- Namespace for Timeline server which supports isolations users and applications

- Timeline server Security is defined at this level Timeline Entity

- An abstract concept of anything- Defines the relationship between entities- Can be an application, an application attempt, a

container or any user-defined object- contains Primary filters which will be used to index the

entities in the Timeline Store.- uniquely identified by an EntityId and EntityType.

Timeline Event

- Event that is related to a specific Timeline Entity of an application

- Users are free to define what an event means, such as starting an application, getting allocated a container,

ATS V1 : Architecture

Separate Process

Pluggable store – defaults to LevelDB

REST Interfaces

ATS V1 : Level DB Key- value store

Lightweight

Open source Compatible license

Used to store - TimelineStore : Domain, Entity, Events and metrics - TimelineStateStore : Security Tokens

Supports Data Retention

ATS V1 : Client & API Timeline client

- Wrap over REST POST method- POJO objects

TimelineEntity TimelineEvent

- In Client/AM/Container

Rest APIs, JSON as the media- Get timeline entities

http://localhost:8188/ws/v1/timeline/{entityType}

- Get timeline entityhttp://localhost:8188/ws/v1/timeline/{entityType}/{entityId}

- Get timeline eventshttp://localhost:8188/ws/v1/timeline/{entityType}/events

ATS V1 : SecurityHTTP SPNEGO

Kerberos Authentication

Delegation Token- Performance- AM/Container no Kerberos

Access Control- Admin/owner- Timeline entity-level

ATS V1 : Use cases

Agenda Who we are ?

Why we need History Server?

Application History Server

Timeline Server V1

Timeline Server V2

Why ATSv2 ? Scalability• Single global instance of writer/reader• ATSv1 uses local disk based LevelDB storage

Usability• Handle flows as first-class concepts and model aggregation.• Elevate configuration and metrics to first-class members.• Better support for queries.

Reliability• Data is stored only in a local disk .• Single daemon so single point of failure.

Existing external tooling: hRaven, Finch, Dr. Elephant, etc. As new Hadoop versions are rolled out, maintenance of these tools becomes an issue.

Key Design Points Distributed writers (per app and per node)• Per App Writer/Collector launched as part of RM.• Per Node Collector/Writer launched as an auxiliary service in NM.• In future, will support standalone writers.

Scalable and reliable backend storage (HBase)

A new object model API with flows built into it.

Separate reader instance(s). Currently have a single reader instance.

Aggregation i.e. rolling up the metric values to the parent.• Online aggregation for apps and flow runs.• Offline aggregation for users, flows and queues.

Timeline Reader

Timeline Reader

ATSv2 ComponentsApplication

Master

Node Manager

Timeline Writer

App Events / Metrics

Container Events / Metrics

Storage

Resource Manager

Timeline WriterTimeline Reader

User Queries

Timeline Reader Pool

App / Container Events

Resource Manager

RMApp

Distributed Writers / Collectors

Node Manager 1

{app_1_collector_info….}

List of app collectors

App Master3. Launch App Master

App Collector

App Collector

Aux Service

4. Notify Aux Service to bind new collector 5. Bind new collector

NODE 1

HBase

NM Collector Service

6. Register new collector

RM Events

Heartbeat with collector info

App Collector

App Collector

Node Manager 2

Node Manager X

1. User submits app

Heartbeat with collector info

2. RMApp launches companion app collector on new app submission

7. Report new collector info. (IP + Port)

Container Events

AM reports events to app collector notified in HB by RM.

NM reports events to app collector notified in HB by RM.

{app_1_collector_infoapp_2_collector_info….}

App 1

App 2

App 3 App 4

Run

Run at 9:00 pm

Flow

Script / Program(eg. HIVE Query /

Pig Script)

App 1

App 2

App 3 App 4Run

Run at 7:30 pm

Joe

Data ModelEntity

ID + Type

Configurations

Metadata(Info)

Parent-Child Relationships

Metrics

Events

Cluster

Type

Cluster Attributes

Flow

Type

User

Flow Runs

Flow Attributes

Flow Run

Type

User

Running apps

Flow Run Attributes

Application

Type

User

Flow + Run

Queue

Attempts

Attempt

Type

Application

Queue

Containers

Container

Type

Attempt

Attributes

Entities of first class citizens

User

Username(ID)

Aggregated metrics

Queue

Queue(ID)

Sub queues

Aggregated metrics

Aggregation

Event

ID

Metadata

Timestamp

Metric

ID

Metadata

Single Value or Time Series(with

timestamps)

HBase vs Phoenix evaluationBased on the evaluation of both Hbase and Phoenix, it was decided that HBase will be used on write path. With Hbase, much higher throughput, a lower IO wait and far lower CPU load was witnessed.

Test descript ion

Map tasks

Entities per mapper

Total entities written

Phoenix Transaction Rate (per mapper) ops/sec

HBase Transaction Rate (per mapper) ops/sec

Phoenix Write Time (job counter TIMELINE_ SERVICE_ WRITE_TIME)

Hbase Write Time (job counter TIMELINE _SERVICE _WRITE_TIME)

Synthetic Data 170 1k 170k 112.83 2285.13 1506704 74394

Synthetic Data 170 10k 1.7M 53.029 636.41 32057957 2671241

Synthetic Data 1 50k 50k 196.67 19770.66 254225 2529

9 History Files 33 - 85k 319.19(write errors)

962.32 265460 88049

555 History Files

33 - 810k 206.25(write errors)

927.62 4102364 874151

Aggregation Aggregation basically means rolling up metrics from child entities to parent entities. We can perform different operations such as

SUM, AVG ,etc. while rolling them up and store them in the parent.

App level aggregation will be done by app collector as and when it receives different metrics.

Online or real time aggregation for apps would be a simple SUM of metrics of child entities. Additional metrics will also be stored which indicate AVG, MAX, AREA(time integral) etc. More on this in next slide.

App to flow run aggregation will be done via a HBase coprocessor on the read path. Cell tags used to achieve this.

For user/flow, aggregation happens periodically(not real time i.e. offline). For this, Phoenix tables will be used. To achieve offline aggregation, a MR job is run which reads application table and writes to user and flow aggregation tables

Container A1(CPUCoresMillis = 400)

Container A2(CPUCoresMillis = 300)

Container B1(CPUCoresMillis = 200)

App A(CPUCoresMillis = 700)

App B(CPUCoresMillis = 200)

Flow(CPUCoresMillis = 900)

Accumulation While aggregating, we also accumulate metric values along the time dimension. This is especially useful for gauges. Consider the

table below which displays the CPU utilization for containers belonging to an app(in terms of cores). Here t1…t16 represents time 10ms. apart. This table shows how values are aggregated for an app and how they are accumulated and averages calculated. Trapezoidal integration rule is used to calculate area under the curve i.e. Area under the curve = ((valuet1 + valuet2)/2) * Dt

t1 t2 t3 t4 t5 t6 t7 t8 t9 t10 t11 t12 t13 t14 t15 t16

Container 1 1 1 1 0.5 0.5 0.5 0.5 0.5

Container 2 0.5 1 1 1 1 1 0.5 0.5 0.5 0.5 0.5

Container 3 0.5 1 1 1 1 1 1 1 1 1 0.5 0.5

Container 4 0.5 1 1 1 1 1 1 1 1 1 1 0.5 0.5

Container 5 0.5 0.5 1 0

Application

Area(CoreMillis)Average

t1 t2 t3 t4 t5 t6 t7 t8 t9 t10 t11 t12 t13 t14 t15 t16

Container 1 1 1 1 0.5 0.5 0.5 0.5 0.5

Container 2 0.5 1 1 1 1 1 0.5 0.5 0.5 0.5 0.5

Container 3 0.5 1 1 1 1 1 1 1 1 1 0.5 0.5

Container 4 0.5 1 1 1 1 1 1 1 1 1 1 0.5 0.5

Container 5 0.5 0.5 1 0

Application 1

Area(CoreMillis)Average

t1 t2 t3 t4 t5 t6 t7 t8 t9 t10 t11 t12 t13 t14 t15 t16

Container 1 1 1 1 0.5 0.5 0.5 0.5 0.5

Container 2 0.5 1 1 1 1 1 0.5 0.5 0.5 0.5 0.5

Container 3 0.5 1 1 1 1 1 1 1 1 1 0.5 0.5

Container 4 0.5 1 1 1 1 1 1 1 1 1 1 0.5 0.5

Container 5 0.5 0.5 1 0

Application 1 2.5 4 4 4 3.5 3 3 3 3 2 1.5 1 1 1 0

Area(CoreMillis)

-

Average -

t1 t2 t3 t4 t5 t6 t7 t8 t9 t10 t11 t12 t13 t14 t15 t16

Container 1 1 1 1 0.5 0.5 0.5 0.5 0.5

Container 2 0.5 1 1 1 1 1 0.5 0.5 0.5 0.5 0.5

Container 3 0.5 1 1 1 1 1 1 1 1 1 0.5 0.5

Container 4 0.5 1 1 1 1 1 1 1 1 1 1 0.5 0.5

Container 5 0.5 0.5 1 1

Application 1 2.5 4 4 4 3.5 3 3 3 3 2 1.5 1 1 1 1

Area(CoreMillis)

- 15

Average - 1.5

t1 t2 t3 t4 t5 t6 t7 t8 t9 t10 t11 t12 t13 t14 t15 t16

Container 1 1 1 1 0.5 0.5 0.5 0.5 0.5 0

Container 2 0.5 1 1 1 1 1 0.5 0.5 0.5 0.5 0.5 0

Container 3 0.5 1 1 1 1 1 1 1 1 1 0.5 0.5 0

Container 4 0.5 1 1 1 1 1 1 1 1 1 1 0.5 0.5 0

Container 5 0.5 0.5 1 0

Application 1 2.5 4 4 4 3.5 3 3 3 3 2 1.5 1 1 1 0

Area(CoreMillis)

- 15 42

Average - 1.5 2.1

t1 t2 t3 t4 t5 t6 t7 t8 t9 t10 t11 t12 t13 t14 t15 t16

Container 1 1 1 1 0.5 0.5 0.5 0.5 0.5 0

Container 2 0.5 1 1 1 1 1 0.5 0.5 0.5 0.5 0.5 0

Container 3 0.5 1 1 1 1 1 1 1 1 1 0.5 0.5 0

Container 4 0.5 1 1 1 1 1 1 1 1 1 1 0.5 0.5 0

Container 5 0.5 0.5 1 0

Application 1 2.5 4 4 4 3.5 3 3 3 3 2 1.5 1 1 1 0

Area(CoreMillis)

- 15 42 82 122 160 192 222 252 282 307 325 335 345 355 360

Average - 1.5 2.1 2.7 3.1 3.2 3.2 3.1 3.1 3.1 3.1 3 2.8 2.6 2.5 2.4

Series10

0.5

1

1.5

2

2.5

3

3.5

4

4.5

CPU Cores for App Avg

HBase Table Schema Entity Table – Used for storing Timeline Entity object. Contains configs, metrics and other info (events,

parent child relationships, etc.). Row Key : clusterId!user!flowId!flowRunId!appId!entityType!entityId

Application Table – Used for storing YARN Application entity. Contains configs, metrics and other info.

Same as entity table but added for better performance. Row Key : clusterId!user!flowId!flowRunId!appId

App To Flow Table – Used for getting flowId and flowRunId information based on cluster and app. This is

helpful in querying entity table on the basis of just the cluster and app information. Row Key : clusterId! appId

Flow Run Table – Stores flow run information aggregated across apps. Row Key : clusterId!user!flowId!flowRunId

HBase Table Schema (Contd.) Flow Activity Table – Used for storing daily activity records for a flow. For quick lookup of flow level info.

Row Key : clusterId!inverted top of the day timestamp!user!flowId

Phoenix Tables for Offline Aggregation :

Flow Aggregation Table – Stores aggregated metrics at flow level. Metrics are aggregated from application table.

Primary Key : user, cluster, flowId

User Aggregation Table – Stores aggregated metrics at user level. Metrics are aggregated from application table.

Primary Key : user, cluster

Querying ATSv2 ATSv2 offers major enhancement over ATSv1 in terms of queries supported. Efficient queries around

flows, flow runs, apps, etc. are possible. Moreover, ATSv2 can support complex queries to filter out results.

ATSv1 offered only primary filters and secondary filters for filtering out entities. ATSv2 offers ability to filter out entities based on config values, metric values, entity parent child relationships and events. It also supports returning only certain configurations and metrics in the result.

ATSv1 queries supported only “equal to” match for primary and secondary filters. But for metrics this does not quite make sense. A user would while filtering on the basis of metric values would more likely be using relational operators such as >=, <=, != etc. All these relational operators are supported in ATSv2 for metrics. In addition to this different predicates in filters can be combined using “AND” and “OR” operators.

All in all this gives ATSv2 a very powerful query interface.

Querying ATSv2 (Contd.) ATSv2, like ATSv1 supports a REST API interface with JSON as the media. Some examples are given below.

Get Entities – Returns a set of TimelineEntity objects based on cluster, app and entity type. The query also supports multiple optional query parameters such as limit on number of entities to be returned, configurations and metrics to be returned, filter on the basis of created and modified time window, config filters, metric filters and event filters.

http://localhost:8188/entities/{clusterId}/{appId}/{entityType}

Example : - http://localhost:8188/entities/cluster1/application_1334432321_0002/YARN_CONTAINER?limit=5&metrics=memory,cpu

Get Entity – Returns a Timeline Entity object based on cluster, app, entity type and entityId.

http://localhost:8188/entity/{clusterId}/{appId}/{entityType}/{entityId}

Possible use cases Cluster utilization and inputs for capacity planning. Cluster can learn from flow’s/application’s historical data.

Mappers / reducers optimizations.

Application performance over time.

Identifying job bottlenecks.

Ad-hoc troubleshooting and identification of problems in cluster.

Complex queries possible at flow, user and queue level. For instance, queries like % of applications which ran more than 10000 containers.

Full DAG from flow to flow run to application to container level can be seen.

Team Members Sangjin Lee, Vrushali C and Joep Rottinghuis (Twitter)

Junping Du, Li Lu and Vinod Kumar Vavillapalli (Hortonworks)

Zhijie Shen (formerly Hortonworks)

Varun Saxena and Naganarasimha G R (Huawei)

Robert Kanter and Karthik Kambatla (Cloudera)

Inputs from LinkedIn, Yahoo! and Altiscale.

Feature Status Distributed per-app and per-node writers (as Aux Service)

RM Companion writer

NM, RM and AM writing events and metrics to ATS

File based readers and writers for test

HBase and Phoenix writer implementations

Performance evaluation of these writers

HBase based reader implementation

Support for flows

App and flow run level online Aggregation

Offline Aggregation

Query Interface

Feature Status (Contd.) Standalone timeline writer

Distributed timeline readers and a reader pool

ATSv2 UI

Security

Support for migration

Click icon to add picture

Thank You !