ambari meetup: yarn

© Hortonworks Inc. 2013

YARN and AmbariYARN service management using Ambari

Srimanth Gunturi

September 25th, 2013


Agenda

• YARN Overview• Installing• Monitoring• Configuration• Capacity Scheduler• MapReduce2• Future

Architecting the Future of Big Data


YARN Overview


• Yet Another Resource Negotiator (YARN)• Purpose - Negotiating resources (Memory, CPU, Disk, etc.) on a cluster• What was wrong with original MapReduce?

– Cluster not open to non-MapReduce paradigms– Inefficient resource utilization (Map slots, Reduce slots)– Upgrade rigidity

• MapReduce (Hadoop 1.0) -> YARN + MapReduce 2 (Hadoop 2.0)


YARN Overview - Applications


• MapReduce1 applications are fully compatible with MapReduce2 – Same JARs can be used– Binary compatibility with org.apache.hadoop.mapred API– Source compatibility with org.apache.hadoop.mapreduce API


YARN Overview - Architecture


• ResourceManager• NodeManagers• Containers• Applications (ApplicationMasters)


Installing



Monitoring



Monitoring – NodeManager Summary


NodeManagers Status• Active – In communication with RM• Lost – Not communicating with RM• Unhealthy – Flagged by custom health

check script identified by propertyyarn.nodemanager.health-checker.script.path

• Rebooted – Automatically restarted due to internal problems

• Decommissioned – RM ignoring communications from host. Host placedin yarn.resourcemanager.nodes.exclude-path file.


Monitoring – Container Summary


Containers• Allocated – Containers which have

been created with requested resources.• Pending – Containers, whose resources

will become available and are pending creation.

• Reserved – Containers, whose resources are not yet available.

Examples10 GB Cluster• Request three 5GB containers

• 2 allocated, 1 pending.• Request three 4GB containers

• 2 allocated, 1 reserved (2GB)


Monitoring – Applications Summary


Applications• Submitted – Application requests made

to YARN.• Running – Application with Masters

which have been created and are running.• Pending – Application requests which

are pending creation.• Completed – Applications which have

completed running. They could havebeen successful, killed or failed.

• Killed – Applications which have beenterminated by user

• Failed – Applications which have failedto run due to internal failures.


Monitoring – Memory Summary


Cluster Memory• Used – Memory resource currently

being used across the cluster• Reserved – Memory resources that

are set aside for being allocated.• Total – Memory resource available

across entire cluster


Monitoring – Alerts


Service Alerts- ResourceManager health- % NodeManagers alive

Host Alerts- NodeManager health- NodeManager process check


Monitoring – Graphs



Configuration



Configuration – Capacity Scheduler


QueuesRooto Ao Bo C

o C1o C2

o default

• Hierarchical Queues• Capacity Guarantees

• Capacity (%)• Maximum-am-resource-percent (%)

• Elasticity• Maximum-capacity (%)

• Access Control


MapReduce2


YARN-321: Generic application history service


Future


• Support more YARN applications• Improve per application-type information• Improve Capacity Scheduler configuration• Better health checks

ambari meetup: yarn

Technology

future of big data yarn

future of big data containers

monitoring page

yarn overview page

configuration page

yarn applications

mapreduce2 page

monitoring alerts page