apache ambari - hdp cluster upgrades operational deep dive and troubleshooting

Apache Ambari - HDP Cluster UpgradesOperational Deep Dive and Troubleshooting

DATAWORKS Summit, Munich

April 5, 2017

Presenters

• Venkatraman Poornalingam (vpoornalingam@hortonworks.com)• Principal Automation Engineer, Technical Support Team Hortonworks• Part of Ambari and Upgrades SME team

• Vivek Sharma (vsharma@hortonworks.com)

• Staff Software Engineer, Ambari Quality Engineering Team

• Specializing on Ambari Upgrades and Views

Agenda

• Use Case• Prerequisites for upgrade• Upgrades Deep Dive

• Express Vs Rolling• Internals

• Troubleshooting • Ambari 2.5 Upgrades new Feature• Best Practices

Sam’s Upgrade Story

• Sam is a Hadoop Administrator working with WBC Inc.• Manages several HDP clusters using Ambari• Is planning to upgrade a cluster with following config:

• 300 nodes, HDP-2.3.6, Ambari-2.2.2.0• Hive, Spark, HBase, Oozie, Kerberos-Managed by Ambari• Interested in Hive LLAP for his applications, Oozie Workflow View

Sam reviews HDP Stack

Sam’s Upgrade Plan

• After reviewing Hortonworks current product stack• Discusses with his CIO/Team• Decides to upgrade to the following

• Ambari 2.5• HDP 2.6

• Sam has to research / plan for• A Runbook consisting of

• Prerequisites• Upgrade Method• Troubleshooting in case of issues• Complete upgrade• Downtime

• Identifies appropriate Ambari user roles for the upgrade• New Stack registration can be done only by Ambari Administrator role• Upgrade can be done by Ambari Administrator and Cluster Administrator

Sam in Research mode …

Ambari Upgrade Workflow

Post Ambari upgrade, complete upgrade for AMS, Infra, SmartSense and Logsearch

HDP Cluster Upgrade Workflow

Upgrade Planning• Backup of configs, Databases - Hive, Oozie,Ranger

• Important to have DB access available to Ambari Administrator• Check 3rd party software compatibility with newer HDP version• Handling Tech Preview services / Custom Services• Ensure Ambari pre-checks pass

• API:/api/v1/clusters/c1/rolling_upgrades_check?fields=*&UpgradeChecks/repository_version=2.6.0.3-8&UpgradeChecks/upgrade_type=NON_ROLLING

• Disk space availability:• New software installation (in /usr/hdp/)• Backups during Upgrade (/tmp/)

• Check and ensure software dependencies are resolved• Example, yum check dependencies; echo $?, Should return 0

• Identify list of hosts which are• In maintenance mode• To be decommissioned• Has software installation failures

Sam decides to Deep Dive

Express Upgrade Orchestration

Upgrade Pack Location on Ambari server:/var/lib/ambari-server/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.6.xml

Config pack:/var/lib/ambari-server/resources/stacks/HDP/2.3/upgrades/config-upgrade.xml

Magic of Symbolic Links!

● hdp-select /usr/hdp/current/$comp-name/ -> /usr/hdp/$version/$compExample:

● conf-select /etc/$comp/conf -> /usr/hdp/$version/$comp/conf -> /etc/$comp/$version/0Example:

– Syntax:– hdp-select set hive-server2-hive2 2.6.0.3-8

– conf-select create-conf-dir --package hive --stack-version 2.6.0.3-8 --conf-version 0

– conf-select set-conf-dir --package hive --stack-version 2.6.0.3-8 --conf-version 0

Pre-Upgrade /usr/hdp/current/hive-server2-hive2 -> /usr/hdp/2.5.3.0-37/hive2

Post-Upgrade /usr/hdp/current/hive-server2-hive2 -> /usr/hdp/2.6.0.3-8/hive2

Pre-upgrade /etc/hive2/conf -> /usr/hdp/current/hive-server2-hive2/conf -> /etc/hive2/2.5.3.0-37/0

Post-upgrade /etc/hive2/conf -> /usr/hdp/current/hive-server2-hive2/conf -> /etc/hive2/2.6.0.3-8/0

Rolling upgrade orchestration

Upgrade Pack Location on Ambari server:/var/lib/ambari-server/resources/stacks/HDP/2.3/upgrades/upgrade-2.6.xml

Config pack:/var/lib/ambari-server/resources/stacks/HDP/2.3/upgrades/config-upgrade.xml

EU Vs RU Performance (Controlled Environment)

Service Configurations - Mergesproperty_x property_y property_z property_x

HDP 2.3 foo (default) 120 Didn’t exist foobar

HDP 2.6 bar (default) deprecated baz bar

Post Upgrade bar Property deleted baz foobar

Sam decides to upgrade Dev Cluster

Development Cluster Upgrade

● 50 Node cluster● Starts a Runbook

● Completes Pre-requisites identified during planning phase; keeps a watch on the time taken

● Upgrades Ambari (yum upgrade, ambari-server upgrade; takes about 45 minutes)● Verifies cluster is operational● Completes registration and installation of new HDP version (ahead of time, takes about 30 minutes to

complete)● Runs API to do pre-check● Allocates 4 Hours for the upgrade● Starts Express Upgrade at the scheduled time

Troubleshooting

● Checks ○ ambari-server.log○ namenode logs○ ambari-agent.log in Namenode

● And then… ambari-agent.log → ambari-agent status

Troubleshooting is no different compared to any other Ambari Issues

Upgrade Completed!

• Finalize Later – for Application verification• Suggests Application team to run basic application testing and finalizes within 2 days (including 3rd

party applications)• If cluster isn’t finalized, the space usage on HDFS would increase and could lead to severe performance issues

• Checks for version details in Ambari UI and finds all in place!

Sam in Research mode…

Fine Tuning Upgrade parameters

• Support for auto-retry of tasks• Fault tolerance options at the start and during Upgrade - skip service check failures, skip slave failures• Batch size during package installation is controlled via a config in ambari.properties

• agent.package.parallel.commands.limit=100

• In the Express upgrade packs, the batch size can be modified from the default value:

<parallel-scheduler><max-degree-of-parallelism>100</max-degree-of-parallelism>

</parallel-scheduler>

Ambari Upgrade – Failure due to DB inconsistencies

• Ambari upgrade - constraint violation• Review Ambari logs• Identify table reporting the violation• Restore Ambari DB• Fix the violation• Restart Ambari Upgrade

• DB Consistency check introduced from Ambari 2.4• Verify if DB consistency is being skipped while starting Ambari

• In Previous versions, this could happen due to• Failed installation / deletion using API’s

Ambari Schema Changes during HDP Upgrade

Performance issues during upgrade

● Save namespace takes too long○ Older versions with large heap size○ Attempt save namespace before upgrade and ensure it works good○ Increase agent.task.timeout in ambari.properties if required

● Too many entries in host_role_command○ It may be necessary to remove entries from the host_role_command table if the size of the table has grown excessively

large in order to reduce the query times for "IN_PROGRESS" requests.

○ This operation can’t be performed during upgrade

How to get summary of current upgrade status?

• Invoke the following Ambari API call:• http://<ambari-server>:8080/api/v1/clusters/c1/upgrades

• From the output of above, identify the latest upgrade id• http://<ambari-server>:8080/api/v1/clusters/c1/upgrades/441

• To get information upto upgrade_item level:• http://<ambari-

server>:8080/api/v1/clusters/c1/upgrades/441?fields=upgrade_groups/upgrade_it

ems/UpgradeItem/status,upgrade_groups/upgrade_items/UpgradeItem/context,upgra

de_groups/UpgradeGroup/title

• To get information up to task level:• http://<ambari-

server>:8080/api/v1/clusters/c1/upgrades/441?fields=upgrade_groups/upgrade_it

ems/tasks/Tasks/status,upgrade_groups/upgrade_items/tasks/Tasks/command_detai

l,upgrade_groups/upgrade_items/tasks/Tasks/stderr

Upgrade States

"upgrade_items" : [

"href" : "http://vpamb2010.novalocal:8080/api/v1/clusters/Ambari21/upgrades/441/upgrade_groups/106/upgrade_items/1",

"UpgradeItem" : {

"cluster_name" : "Ambari21",

"context" : "Restarting NodeManager on vpamb2012.novalocal",

"group_id" : 106,

"request_id" : 441,

"stage_id" : 1,

"status" : "HOLDING_FAILED"

Upgrade States:●IN_PROGRESS●HOLDING●FAILED/HOLDING_FAILED/SKIPPED_FAILED●TIMEDOUT/HOLDING_TIMEDOUT●ABORTED●PENDING/QUEUED●COMPLETED

Service fails to start due to Circular Symlink issue

STDERR while starting Oozie service:packages/resource_management/core/environment.py", line 124, in run_action provider_action() File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 177, in action_create raise Fail("Applying %s failed, looped symbolic links found while resolving %s" % (self.resource, path))resource_management.core.exceptions.Fail: Applying Directory'/usr/hdp/current/oozie-client/conf' failed, looped symbolic links found while resolving /usr/hdp/current/oozie-client/conf

conf-select create-conf-dir --package oozie-client --stack-version $version --conf-version 0

conf-select set-conf-dir --package oozie-client --stack-version $version --conf-version 0

ln -s /etc/oozie/2.3.2.0-2950/0 /usr/hdp/2.3.2.0-2950/oozie/conf

Post RU, Hive applications are failing

● Hive is started with port number 10010 instead of 10000 post upgrade

● Either Configurations need to be updated or HiveServer2 needs to be restarted with the older port number

● Rolling upgrade is not supported for Hive from HDP 2.6

○ Ambari 2.5 would give a warning while upgrading - “HiveServer2 does not currently support rolling upgrades. HiveServer2 will be upgraded, however existing queries which

have not completed will fail and need to be resubmitted after HiveServer2 has

been upgraded.”

What’s new in Ambari 2.5 for upgrades?

● Auto Start of services● Delete older version of the Software

● AMBARI-18435 Releases space used by older versions post upgrade. Previously this had to be done manually. For eg,

curl 'http://c6401.ambari.apache.org:8080/api/v1/clusters/cl1/requests' -u admin:admin -H "X-Requested-By: ambari" -X POST -

d'{"RequestInfo":{"context":"remove_previous_stacks", "action" : "remove_previous_stacks", "parameters" : {"version":"2.5.0.0-

1245"}}, "Requests/resource_filters": [{"hosts":"c6403.ambari.apache.org, c6402.ambari.apache.org"}]}'

● Upgrade history● Pulls all data about upgrades/downgrades from Ambari DB and displays in UI

Sam’s Runbook for Cluster upgrade at WBC

Customized Upgrade Runbook

• Sam writes up a Runbook for WBC Inc. cluster upgrades which includes• Upgrade Planning• Installing packages ahead of time• Checking disk space in hosts• Choosing the right Upgrade method

• Deleting older versions if not required (keep the current and new one intact)• Backup method for Databases and Configurations• Stopping any Jobs which would restart services in the system and disable AUTO_RESTART of services in

Ambari• Upgrading Development cluster

• Table to document issues faced during Development• Time taken for the Upgrade activity

• Documents prerequisites including• No changes to stack during upgrade• No new installation / No new hosts etc• Reviewing list of supported Databases in documentation

Thanks

apache ambari - hdp cluster upgrades operational deep dive and troubleshooting

Technology

aws ambari

ambari: using a local repository

bk using ambari book-20140401

apache ambari: past, present, future

apache ambari tutorial -...

hortonworks data platform - apache ambari minor upgrade ·...

upgrading from-hdp-21-to-hdp-24

ホールダウンパイプ（hdpタイプ）加工φ22寸法図...

installing apache rangerinstalling ranger using ambari...

ambari meetup: namenode ha

apache ambari stack extensibility

discover hdp 2.1: using apache ambari to manage hadoop...

ambari user views: tech preview · apache ambari...

bk using ambari book-20130114

emc isilon$hdp 2.3$and ambari$2.1$$installation guide ·...

accumulo summit 2015: ambari and accumulo: hdp 2.3 upcoming...

hortonworks technical workshop: apache ambari

on demand hdp clusters using cloudbreak and ambari

manage hadoop cluster with ambari

hortonworks data platform - apache ambari minor...