troubleshooting your suse® openstack cloud tut19873

Click here to load reader

Post on 15-Feb-2017




1 download

Embed Size (px)


  • Troubleshooting Your

    SUSE OpenStack Cloud TUT19873

    Adam SpiersSUSE Cloud/HA Senior Software Engineer

    Dirk MllerSUSE OpenStack Senior Software Engineer

  • SUSE OpenStack Cloud ...

  • SUSE OpenStack Cloud

  • SUSE OpenStack Cloud


  • SUSE OpenStack Cloud


  • SUSE OpenStack Cloud


  • SUSE OpenStack Cloud Troubleshooting

  • Billling VM Mgmt Image Tool Portal App Monitor Sec & Perf




    Cloud APIs


    Message QDatabase



    HypervisorXen, KVM

    Vmware, HyperV


    Operating System

    Physical Infrastructure: x86-64, Switches, Storage

    OpenStack Management Tools OS and Hypervisor







    Physical InfrastructureSUSE Cloud Adds





    HypervisorXen, KVM

    SUSE Linux Enterprise Server 11 SP3

    SUSE Product

    Physical Infrastructure: x86-64, Switches, Storage

    Billling Portal App Monitor Sec & Perf

    Adapters Adapters Vmware, HyperV

    Partner Solutions



    SUSE OpenStack Cloud


  • Non-HA SUSE OpenStack Cloud

  • HA SUSE OpenStack Cloud

  • Crowbar and Chef

  • #

    Generic SLES Troubleshooting

    All Nodes in SUSE OpenStack Cloud are SLES


    Watch out for typical issues:

    dmesg for hardware-related errors, OOM, interesting kernel


    usual syslog targets, e.g. /var/log/messages

    Check general node health via:

    top, vmstat, uptime, pstree, free

    core files, zombies, etc

  • #


    supportconfig can be run on any cloud node


    installed on all SUSE OpenStack Cloud nodes automatically

    collects precious cloud-specific information for further analysis

  • #

    Admin Node: Crowbar UI

    Useful Export Page

    available in the

    Crowbar UI in order to

    export various log files

  • Cloud Install

    screen install-suse-cloud --verbose



  • SUSE OpenStack Cloud Admin

    SLES 11 SP3

    SUSE Cloud Addon

    Crowbar UI

    Crowbar Services


    Repo Mirror

    Install logs:/var/log/crowbar/install.log




    Crowbar repo server:/var/log/apache2/provisioner*log



  • #


    Cloud uses Chef for almost everything:

    All Cloud and SLES non-core packages

    All config files are overwritten

    All daemons are started

    Database tables are initialized

  • Admin Node: Using Chef

    knife node list

    knife node show

    knife search node "*:*"

  • #

    SUSE OpenStack Cloud Admin

    Populate ~root/.ssh/authorized_keys

    prior install

    Barclamp install logs:


    Node discovery logs:


    Syslog of crowbar installed nodes sent via rsyslog to:


  • #

    Useful Tricks

    Root login to the Cloud installed nodes should be

    possible from admin node (even in discovery stage)

    If admin network is reachable:


    host 192.168.124.*

    StrictHostKeyChecking no

    user root

  • #

    SUSE OpenStack Cloud Admin

    If a proposal is applied, chef client logs are at:


    Useful crowbar commands:

    crowbar machines help

    crowbar transition

    crowbar proposal list|show

    crowbar proposal delete default


    crowbar_reset_proposal default

  • #

    Admin Node: Crowbar Services

    Nodes are deployed via PXE boot:


    Installed via AutoYaST; profile generated to:


    Can delete & rerun chef-client on the admin node

    Can add useful settings to autoyast.xml:


    (dont forget to chattr +i the file)

  • #

    Admin Node: Crowbar UI

    Raw settings in barclamp

    proposals allow access to

    "expert" (hidden) options

    Most interesting are:

    debug: true

    verbose: true

  • Admin Node: Crowbar Gotchas

  • #

    Admin Node: Crowbar Gotchas

    Be patient

    Only transition one node at a time

    Only apply one proposal at a time

    Cloud nodes should boot from:

    1. Network

    2. First disk

  • SUSE OpenStack Cloud

    Cloud Node

    SLES 11 SP3

    SUSE Cloud Addon

    Chef Client

    All managed via Chef:


    rcchef-client status

    chef-client can be invoked manually

    Node specific services

  • SUSE OpenStack Cloud Control Node

    Control Node

    SLES 11 SP3

    SUSE Cloud Addon

    Chef Client

    Just like any other cloud node:


    rcchef-client status


    Chef overwrites all config files it touches

    chattr +i is your friend

    OpenStack API


  • High Availability

  • 30

    What is High Availability?

    Availability = Uptime / Total Time

    99.99% (4 nines) == ~53 minutes / year

    99.999% (5 nines) == ~5 minutes / year

    High Availability (HA)

    Typically accounts for mild / moderate failure scenarios

    e.g. hardware failures and recoverable software errors

    automated recovery by restarting / migrating services

    HA != Disaster Recovery (DR)

    Cross-site failover

    Partially or fully automated

    HA != Fault Tolerance

  • 31

    Internal architecture

  • 32

    Resource Agents

    Executables which start / stop / monitor resources

    RA types:

    LSB init scripts

    OCF scripts (~ LSB + meta-data + monitor action + ...)


    Legacy Heartbeat RAs (ancient, irrelevant)

    systemd services (in HA for SLE12+)

  • 33

    Results of resource failures

    If fail counter is exceeded, clean-up is required:

    crm resource cleanup $resource

    Failures are expected:

    when a node dies

    when storage or network failures occur

    Failures are not expected during normal operation:

    applying a proposal

    starting or cleanly stopping resources or nodes

    Unexpected failures usually indicate a bug!

    Do not get into the habit of cleaning up and ignoring!

  • 34

    Before diagnosis

    Understand initial state / context

    crm configure graph is awesome!


    Which fencing devices are in use?

    What's the network topology?

    What was done leading up to the failure?

    Look for first (relevant) failure

    Failures can cascade, so don't confuse cause and effect

    Watch out for STONITH

  • 35

    crm configure graph FTW!

  • 36


    What failed?



    Orchestration via Crowbar / chef-client?

    cross-cluster ordering

    Pacemaker config? (e.g. incorrect constraints)

    Corosync / cluster communications?

    chef-client logs are usually a good place to start

    More on logging later

  • 37

    Symptoms of resource failures

    Failures reported via Pacemaker UIsFailed actions:

    neutron-ha-tool_start_0 on d52-54-01-77-77-01 'unknown error' (1): call=281,

    status=complete, last-rc-change='Thu Jun 4 16:15:14 2015', queued=0ms,


    neutron-ha-tool_start_0 on d52-54-02-77-77-02 'unknown error' (1): call=259,

    status=complete, last-rc-change='Thu Jun 4 16:17:50 2015', queued=0ms,


    Services become temporarily or permanently unavailable

    Services migrate to another cluster node

  • 38

    Symptoms of node failures

    Services become temporarily or permanently unavailable, or migrated to another cluster node

    Node is unexpectedly rebooted (STONITH)

    Crowbar web UI may show a red bubble icon next to a controller node

    Hawk web UI stops responding on one of the controller nodes (should still be able to use the others)

    ssh connection to a cluster node freezes

  • 39

    Symptoms of orchestration failures

    Proposal / chef-client failed

    Synchronization time-outs are common and obviousINFO: Processing crowbar-pacemaker_sync_mark[wait-keystone_database] action guess

    (keystone::server line 232)

    INFO: Checking if cluster founder has set keystone_database to 5...

    FATAL: Cluster founder didn't set keystone_database to 5!

    Find synchronization mark in recipe:crowbar_pacemaker_sync_mark "wait-keystone_database"

    # Create the Keystone Database

    database "create #{node[:keystone][:db][:database]} database" do


    So node timed out waiting for cluster founder to create keystone database

    i.e. you're looking at the wrong log! So ...

    [email protected]:~ # knife search node founder:true -i

  • 40


    All changes to cluster configuration driven by chef-client

    either from application of barclamp proposal

    admin node: /var/log/crowbar/chef-client/$NODE.log

    or run by chef-client daemon every 900 seconds

    /var/log/chef/client.log on each node

    Remember chef-client often runs in parallel across nodes

    All HAE components log to /var/log/messages on each cluster node

    Nothing Pacemaker-related on admin node

  • 41

    HAE logs

    Which nodes' log files to look at?

    Node failures:

    /var/log/messages from DC

    Resource failures:

    /var/log/messages from DC and node with failed resource

    but remember the DC can move around (elections)

    Use hb_report or crm history or Hawk to assemble chronological cross-cluster log

    Saves a lot of pain strongly recommended!

  • 42

    Syslog messages to look out for

    Fencing going wrongpengine[16374]: warning: cluster_status: We do not have quorum -

    fencing and resource management disabled

    pengine[16374]: warning: stage6: Node d52-54-08-77-77-08 is unclean!

    pengine[16374]: warning: stage6: Node d52-54-0a-77-77-0a is unclean!

    pengine[16374]: notice: stage6: Cannot fence unclean nodes until quorum is

    attained (or no-quorum-policy is set to ignore)

    Fencing going rightcrmd[16376]: notice: te_fence_node: Executing reboot fencing operation (66)

    on d52-54-0a-77-77-0a (timeout=60000)

    stonith-ng[16371]: notice: handle_request: Client crmd.16376.f6100750 wants to

    fence (reboot) 'd52-54-0a-77-77-0a' with device '(any)'

    Reason for fencing is almost always earlier in log.

    Don't forget all the possible reasons for fencing!

    Lots more get used to reading /var/log/messages!

  • 43

    Stabilising / recovering a cluster

    Start with a single node

    Stop all others

    rcchef-client stop

    rcopenais stop

    rccrowbar_join stop

    Clean up any failures

    crm resource cleanup

    crm_resource -C is buggy

    crm_resource -o | \

    awk '/\tStopped |Timed Out/ { print $1 }' | \

    xargs -n1 crm resource cleanup

    Make sure chef-client is happy

  • 44

    Stabilising / recovering a cluster (cont.)

    Add another node in

    rm /var/spool/corosync/block_automatic_start

    service openais start

    service crowbar_join start

    Ensure nothing gets fenced

    Ensure no resource failures

    If fencing happens, check /var/log/messages to find out why, then rectify cause

    Repeat until all nodes are in cluster

  • #

    Degrade Cluster for Debugging

    crm configure location fixup-cl-apache cl-apache \

    rule -inf: '#uname' eq $HOSTNAME

    Allows to degrade an Activate/Activate resource to

    only one instance per cluster

    Useful for tracing Requests

  • #

    TL; DR: Just Enough HA

    crm resource list


    crm resource restart

    crm resource cleanup

  • Now Coming to OpenStack

  • OpenStack Architecture Diagram

  • OpenStack Block diagram

    Keystone: SPOF

    Accesses almost everything

  • #

    OpenStack Architecture

    Typically each OpenStack component provides:

    an API daemon / service

    one or many backend daemons that do the actual work

    openstack / command line client to access the API

    -manage client for admin-only functionality

    dashboard ("Horizon") Admin tab for a graphical view on the


    uses an SQL database for storing state

  • #

    OpenStack Packaging Basics

    Packages are usually named:


    usually a subpackage for each service (-api, -scheduler, etc)

    log to /var/log//.log

    each service has an init script:

    dde-ad-be-ff-00-01:~# rcopenstack-glance-api status

    Checking for service glance-api ...running

  • #

    OpenStack Debugging Basics

    Log files often lack useful information without verbose enabled

    TRACEs of processes are not logged without verbose

    Many reasons for API error messages are not logged unless debug is turned on

    Debug is very verbose (>10GB per hour)

  • OpenStack Architecture

    Keystone: SPOF

    Accesses almost everything

  • #

    OpenStack Dashboard: Horizon




    Get the exact URL it tries to


    Enable debug in Horizon


    Test components


  • #

    OpenStack Identity: Keystone

    Needed to access all services

    Needed by all services for checking authorization

    Use keystone token-get to validate credentials

    and test service availability

  • #

    OpenStack Imaging: Glance

    To validate service availability:

    glance image-list

    glance image-download > /dev/null

    glance image-show

    Dont forget hypervisor_type property!

  • OpenStack Compute: Nova

    nova-manage service list

    nova-manage logs errors

    nova show shows compute node

    virsh list, virsh dumpxml

  • Nova Overview




    "Launches" go to Scheduler; rest to Conductor

  • Nova Booting VM Workflow

  • #

    Nova: Scheduling a VM

    Nova scheduler tries to select a matching compute

    node for the VM

  • #

    Nova Scheduler

    Typical errors:

    No suitable compute node can be found

    All suitable compute nodes failed to launch the VM

    with the required settings

    nova-manage logs errors

    INFO nova.filters [req-299bb909-49bc-4124-8b88-

    732797250cf5 c24689acd6294eb8bbd14121f68d5b44

    acea50152da04249a047a52e6b02a2ef] Filter

    RamFilter returned 0 hosts

  • #

    OpenStack Volumes: Cinder




  • #

    OpenStack Cinder: Volumes

    Similar syntax to Nova:

    cinder-manage service list

    cinder-manage logs errors

    cinder-manage host list

    cinder list|show

  • #

    OpenStack Networking: Neutron

    Swiss Army knife for SDN

    neutron agent-list

    neutron net-list

    neutron port-list

    neutron router-list

    There's no neutron-manage

  • Basic Network Layout


    Networking with OVS: Compute Node

  • Networking with LB: Compute Node

  • #

    Neutron Troubleshooting

    Neutron uses IP Networking Namespaces on the

    Network node for routing overlapping networks

    neutron net-list

    ip netns list

    ip netns exec qrouter- bash



    ip ro..


  • #


    Thank you

  • Bonus Material

  • OpenStack Orchestration: Heat

  • #

    OpenStack Orchestration: Heat

    Uses Nova, Cinder, Neutron to assemble complete

    stacks of resources

    heat stack-list

    heat resource-list|show

    heat event-list|show

    Usually necessary to query the actual OpenStack

    service for further information

  • #

    OpenStack Imaging: Glance

    Usually issues are in the configured glance backend

    itself (e.g. RBD, swift, filesystem) so debugging

    concentrates on those




    ceph -w

    rbd -p ls

  • #

    SUSE OpenStack Cloud

  • Unpublished Work of SUSE. All Rights Reserved.

    This work is an unpublished work and contains confidential, proprietary, and trade secret information of SUSE.

    Access to this work is restricted to SUSE employees who have a need to know to perform tasks within the scope of

    their assignments. No part of this work may be practiced, performed, copied, distributed, revised, modified, translated,

    abridged, condensed, expanded, collected, or adapted without the prior written consent of SUSE.

    Any use or exploitation of this work without authorization could subject the perpetrator to criminal and civil liability.

    General Disclaimer

    This document is not to be construed as a promise by any participating company to develop, deliver, or market a

    product. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making

    purchasing decisions. SUSE makes no representations or warranties with respect to the contents of this document,

    and specifically disclaims any express or implied warranties of merchantability or fitness for any particular purpose.

    The development, release, and timing of features or functionality described for SUSE products remains at the sole

    discretion of SUSE. Further, SUSE reserves the right to revise this document and to make changes to its content, at

    any time, without obligation to notify any person or entity of such revisions or changes. All SUSE marks referenced in

    this presentation are trademarks or registered trademarks of Novell, Inc. in the United States and other countries. All

    third-party trademarks are the property of their respective owners.