KVM for IBM z Systems In an Open Source Ecosystem

Scott Loveland Michael Tebolt Johanna Husta 12/02/2016

Trademarks The following are trademarks of the International Business Machines Corporation in the United States and/or other countries. BigInsights BlueMix CICS* COGNOS* DB2*

IBM LinuxONE IBM LinuxONE Emperor IBM LinuxONE Rockhopper XIV* PartnerWorld*

PR/SM DFSMShsm DFSORT DS6000* DS8000*

* Registered trademarks of IBM Corporation

Notes: Performance is in Internal Throughput Rate (ITR) ratio based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput that any user will experience will vary depending upon considerations such as the amount of multiprogramming in the user's job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve throughput improvements equivalent to the performance ratios stated here. IBM hardware products are manufactured from new parts, or new and serviceable used parts. Regardless, our warranty terms apply. All customer examples cited or described in this presentation are presented as illustrations of the manner in which some customers have used IBM products and the results they may have achieved. Actual environmental costs and performance characteristics will vary depending on individual customer configurations and conditions. This publication was produced in the United States. IBM may not offer the products, services or features discussed in this document in other countries, and the information may be subject to change without notice. Consult your local IBM business contact for information on the product or services available in your area. All statements regarding IBM's future direction and intent are subject to change or withdrawal without notice, and represent goals and objectives only. Information about non-IBM products is obtained from the manufacturers of those products or their published announcements. IBM has not tested those products and cannot confirm the performance, compatibility, or any other claims related to non-IBM products. Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products. Prices subject to change without notice. Contact your IBM representative or Business Partner for the most current pricing in your geography. This information provides only general descriptions of the types and portions of workloads that are eligible for execution on Specialty Engines (e.g, zIIPs, zAAPs, and IFLs) ("SEs"). IBM authorizes customers to use IBM SE only to execute the processing of Eligible Workloads of specific Programs expressly authorized by IBM as specified in the “Authorized Use Table for IBM Machines” provided at www.ibm.com/systems/support/machine_warranties/machine_code/aut.html (“AUT”). No other workload processing is authorized for execution on an SE. IBM offers SE at a lower price than General Processors/Central Processors because customers are authorized to use SEs only to process certain types and/or amounts of workloads as specified by IBM in the AUT.

The following are trademarks or registered trademarks of other companies. Adobe, the Adobe logo, PostScript, and the PostScript logo are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States, and/or other countries. Cell Broadband Engine is a trademark of Sony Computer Entertainment, Inc. in the United States, other countries, or both and is used under license therefrom. Intel, Intel logo, Intel Inside, Intel Inside logo, Intel Centrino, Intel Centrino logo, Celeron, Intel Xeon, Intel SpeedStep, Itanium, and Pentium are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States and other countries. IT Infrastructure Library is a registered trademark of the Central Computer and Telecommunications Agency which is now part of the Office of Government Commerce. ITIL is a registered trademark, and a registered community trademark of the Office of Government Commerce, and is registered in the U.S. Patent and Trademark Office. Java and all Java based trademarks and logos are trademarks or registered trademarks of Oracle and/or its affiliates. Linear Tape-Open, LTO, the LTO Logo, Ultrium, and the Ultrium logo are trademarks of HP, IBM Corp. and Quantum in the U.S. and Linux is a registered trademark of Linus Torvalds in the United States, other countries, or both. Microsoft, Windows, Windows NT, and the Windows logo are trademarks of Microsoft Corporation in the United States, other countries, or both. OpenStack is a trademark of OpenStack LLC. The OpenStack trademark policy is available on the OpenStackwebsite. TEALEAF is a registered trademark of Tealeaf, an IBM Company. Windows Server and the Windows logo are trademarks of the Microsoft group of countries. Worklight is a trademark or registered trademark of Worklight, an IBM Company. UNIX is a registered trademark of The Open Group in the United States and other countries. * Other product and service names might be trademarks of IBM or other companies.

FICON* FlashSystems Storwize* IBM* IBM (logo)*

RACF* Rational* Redbooks* REXX SmartCloud*

System z10* Tivoli* UrbanCode WebSphere* z13

z13s zEnterprise* zSecure z Systems z/VM*

Agenda

l  The Journey Begins: Timeline and Architecture

l  Deployment & Automation

KVM for IBM System z: The Journey Begins

•  1Q2014: our team was approached about owning System Test for a newly proposed KVM for IBM z offering, that would ship in Fall of 2015

•  We form a small core team to begin planning and preparing for this test

•  2Q2014 – 4Q2014: The team builds several configurations from scratch, studies and selects various tools, and begins developing automation

•  1Q2015: Testing begins

TestEnvironmentDesignPhilosophy(1)

§ Our Business Model –  Test in a large, horizontally-scaled environment for nearly the same cost as a small-scale

environment

§ Architectural First Principles –  Design for cloud-scale efficiencies

•  Highly standardized configurations to reduce management costs •  Identify and leverage commonalities to achieve economies of scale •  Drive down cost of management by orders of magnitude

–  Enable rapid and dynamic change –  Keep it simple – Make it easy to use –  Demonstrate KVM for IBM z Systems fits well in an open source ecosystem

TestEnvironmentDesignPhilosophy(2)

§ Key Design Characteristics – Automate everything

•  Provisioning of virtual machines, operating systems, middleware, workloads •  Configuration and re-configuration •  Test execution scripts (where applicable) •  Ideally, only debug should be done manually, and if that leads to adjustments they should then be applied via

automation

– Make it consumable •  On demand self-service by testers via easy-to-use interfaces •  Ubiqutous network access across pools of virtual servers •  Location-independent resource pooling (storage, network access, IP addresses) within a given cloud

– Make it repeatable •  Standardized configurations. Customer-available tools •  Version control for an entire configuration, with ability to revert to older configuration versions for recreates

– Provide rapid scalability & elasticity •  Stateless workload spraying across virtual servers

– Monitor for failures and bottlenecks •  Monitor both the virtualization infrastructure and guest OS‘s •  Performance, logs, events •  Alert testers to failures or anomalies needing investigation

7

IBMCloudCompu>ngReferenceArchitecture:CloudEnabledDataCenter(adapted)…

Access Layer

Interface Layer

Cloud Mgmt UIs

Service Governance Layer

Service Orchestration Layer

Service Management Layer

Service Delivery Layer

Infrastructure Layer

End user self-service portal Cloud Admin UI

Cloud Mgmt APIs Cloud service interaction APIs Cloud Admin APIs

Accounting & chargeback

Threat & vulnerability mgmt Orchestration VMs pattern

provisioning

Monitoring

Security compliance

Backup/restore

Metering

Patch mgmt

Failure Detection (Logs, Alerting, etc)

Network provisioning & configuration

Storage provisioning & configuration

User and tenant administration

Cloud resource administration

User authentication & authorization

Image mgmt

Image creation

VMs (

Virtualization Environment

Compute

Storage

Network

Authenticzation / Authorization

Cloud Administrator Tester / End User Operator

Workload Mgmt

8 8

Access Layer

Interface Layer

Cloud Mgmt UIs

Service Governance Layer

Service Orchestration Layer

Service Management Layer

Service Delivery Layer

Infrastructure Layer

End user self-service portal Cloud Admin UI

Cloud Mgmt APIs Cloud service interaction APIs Cloud Admin APIs

Accounting & chargeback

Threat & vulnerability mgmt

Orchestration VMs pattern provisioning

Monitoring

Security compliance

Backup/restore

Metering Patch mgmt

Failure Detection (Logs, Alerting, etc)

Network provisioning & configuration

Storage provisioning & configuration

User and tenant administration

Cloud resource administration

User authentication & authorization

Image mgmt

Image creation

VMs (SLES, etc)

Virtualization Environment

Compute

Storage

Network

OpenStack

Chef

Cloud Administrator Tester / End User Operator

Workload Mgmt

TSM

SELinux ELK, rsyslog

OpenStack / MCM

OpenStack

Rundeck

Ceilometer

zHPM

Nagios, Kibana

Rundeck

Shell-script based custom solution

(MCM)

Qemu-img

…wastheFounda>onUponwhichtheSolu>onTestEnvironmentwasDesigned

Authenticzation / Authorization

zEC12, z13

DS8700, V7000, SVC, XIV, FlashSystems

OSA, uniper, etc

LDAP

Flowviewer

KVM Solution Test Ecosystem

Distributions Hypervisor Transactions Database AnalyticsAvailabilityManagement

zAware

FlowViewer Ginger

IBM Confidential

Deployment & Automation

11

Category Function Recommended Limit Maximum Limit

CPU Overcommit 10:1 N/A

Host CPUs z13: 28-36 (1 drawer) 101 (zEC12) 141 (z13)

Per Guest vCPUs Guest vCPUs <= Host CPUs 64

Memory Overcommit 2:1 N/A

Maximum Host 1 TB (zEC12/z13) 1 TB (z12), 8TB (z13)

Networking OSA CHPIDs per Host 16 N/A

Virtual NICs per Host (with OpenVSwitch)

4096 8192

Virtual NICs per Guest 8 32

Storage Total attached FCP LUNs 4096 64K

Total attached ECKD Devices 4096 64K

Virtual Block Disks per Host 4096 64K

Virtual Block Disks per Guest 500 1024

Others Guests per Host 512 4096

*

Limits in KVM for LinuxONE

Scale 4000+GuestsLiveGuest

Migra4on

MemoryLimits Fixpack

IOStress

EverysupportedIBMstoragetype&2000+SCSILUNs

Installer

zHPM Security/Crypto

HAcluster

Pacemaker/Corosync

OpenStackclouds

•  40 zKVM LPARs across 5 CPCS •  2 CPCs in DPM mode

•  Multiple KVM releases in parallel

•  4000+ guest scaling environment •  Disk-based (5 types) •  File-based (2 types) •  SLES, Ubuntu

Automated InfrastructureThe automation builds the infrastructure

Also, x86 Linux server pool (OpenStack controllers, Virt-manager testing, Chef server, etc)

Opera4onsManagement

Toolsinves4gated,Rundeckchosen

90communityjobs,522projectjobs• Commonrepeatabletasks,scheduledcomplexruns,workloadmanagement,Chefintegra4on,MCMintegra4on

CloudProvisioning

OpenStackandahome-grownsolu4onplanned

Full-func4onsolu4oninplace• 40combina4onsofinstall,network,distrotypes• OpenStackintegra4on

Configura4onManagement Chefinplace

46cookbooks,upto20versionupdates,51KLOC• Configuresworkloads(includingauto-recovery),monitoringplug-ins,HPMcustomtooling,KVMhosta_ributes,etc

Monitoring Toolsunderinves4ga4on:Ganglia,Nagios,ELK

Toolsfullyintegrated• Ganglia,FlowViewer:performancemonitoring&metricscapture• Nagios:visualhealthcheck&addi4onalhooks• ELK,zAware:reac4veandproac4veloganaly4cs,capture,andemailalerts

Workloads13thrasherandmiddleware-drivenworkloadschosen,

deployedonSLESWorkloadsportedtoUbuntu

Task Flow

Linux on z

KVM for IBM z Host

Chef server (Linux on x86)

Cookbooks: Middleware Workload Monitor-nagios/ganglia

Provisioning Automation Summary (non-Openstack) •  User goes to Web page and selects “Create Guests”, clicking on choice of Linux distribution, KVM

host, number of LUNS required, type of network attachment, swap space, and “roles” the guest(s) will perform

•  Tool selects IP address and LUN(s) for the VM from a pool, and ssh’es to target KVM host, where it invokes the provisioning tool

•  Rundeck starts the newly defined VM •  Rundeck contacts new VM, installs Chef client in it and associates it with requested roles •  Once VM is personalized by Chef client, user can select from a menu of applications/services to initiate

on the VM

•  Generates domain XML file to define the guest •  Issues (virsh) commands to instantiate the VM

VM Provisioning tool

Virtual Machine

Chef client •  Contacts Chef server. Pulls down and

executes cookbooks to personalize VM for assigned roles

•  Updates configuration, Install tools, middleware, monitors, etc

•  Personalizes guest networking & hostname

15

UbiquitousNetworkAccessacrossPoolsofVirtualServers

Route 172.16.60.0 > 10.20.92.91

9.12.23.xx

10.20.92.70

ovsbr0 172.16.60.1

GRE0

ovsbr1 172.16.80.1

Route 172.16.80.0 > 10.20.92.91

Route 172.16.60.0 > ovsbr0

GRE1

Route 172.16.80.0 > ovsbr1

10.20.92.91

GRE0

10.20.92.60

GRE1

10.20.101.241 10.20.101.247

Guest default gw = 172.16.60.1 Guest default gw = 172.16.80.1

guest 172.16.60.2

guest 172.16.60.3

guest guest guest guest 172.16.60.4 172.16.60.5

172.16.80.2 guest guest

172.16.80.3 172.16.80.4 172.16.80.5

•  Access from Private Network OVS Guests to Automation Hub Servers: Rundeck, Chef, Nagios, Ganglia, Distribution Mirrors •  ssh from Automation Hub system to any guest

Automation Hub System

GRE / VXLAN Tunnels KVM Host KVM Host

Guest default gw = 172.16.60.1 Guest default gw = 172.16.80.1

16

qcow2 images

qcow2 images Guest xml

NFS Exports

OSA NFS

OSA Guests

OSA STONITH

vm1, vm2, … vm50

OSA NFS

OSA Guests

OSA STONITH

Migrate Virtual Domain Resources

Virtual Domain Resources / Remote Nodes

Virtual Domain Resources vm101, vm102,…vm150

destroy/ create

VM201

IPaddr2 apache

VM202 Migrate VM Or Move Resources

OSA Ring1

OSA Ring2

Shared LUNs for disk-based VMs HMC

Loca4onConstraint

STONITH Target

Both Corosync interfaces must route to VM remote nodes

OSA Ring2

OSA Ring1

Virtual Domain Resources vm51, vm52, … vm100

Virtual Domain Resources vm151, vm152,…vm200

VM203

Pacemaker/CorosyncforHAofVirtualMachines

17

Managed Systems Management & Monitoring Servers User Interfaces

KVM for IBM z Host

VM Test appl chef-client

VM WAS ,DB2

chef-client nrpe

… 4000 VMs …

chef-client

rsyslog

Host sFlow

gmetric

Logstash Collect, parse, forward

ElasticSearch Analyze, search, store

Curator Prune

Email alerts (throttled)

Kibana Visualize

Nagios

hostgroups

hosts

check-ganglia-metric

check-service

Event handler

check-ping

Ganglia

restart

Chef server

Rundeck

Add new VM to chef role(s)

Add V

M to host &

groups

Email alerts

restart

Config maint.

Custom metrics

Test appl

Std metrics

Up?

services

Portal

Hypervisor Performance Manager

Prune

SiLK FlowViewer Open vSwitch sFlow

check-nrpe check-http

zAware Predictive Analytics

rsyslog

ManagementandMonitoring

18

Logstashemailalerts

Alert on ZS93KB: <1>May 11 12:16:58 zs93kb kernel: [325826.809838] User process fault: interruption code 0x4003B in libvirt.so.0.1002.12[3fffcf07000+339000]

19

Nagios–monitor/recoveryofDaytrader‘triplets’

Statusofmiddleware–HTTPServer,WebSphereApplica4onServer,DB2,JMeter

20

Nagios–monitor/recoveryofDaytrader‘triplets’

Statusofmiddleware–HTTPServer,WebSphereApplica4onServer,DB2,JMeterAferStoppingWebSphereonguestzs93kbg100107

WebSphere is down

Jmeter has stopped with errors:

Generate Summary Results = 171356 in 340.0s = 503.9/s Avg: 13 Min: 1 Max: 3358 Err: 600 (0.35%) Tidying up ... @ Sun Nov 27 14:31:38 EST 2016 (1480275098929) ... end of run

21

Nagios–monitor/recoveryofDaytrader‘triplets’

Statusofmiddleware–HTTPServer,WebSphereApplica4onServer,DB2,JMeterAferStoppingWebSphereonguestzs93kbg100107

Nagios has restarted WebSphere: nagios : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/opt/wasprofile/AppSrv01/bin/startServer.sh server1

Jmeter has been restarted

22

Nagios–monitor/recoveryDaytrader‘triplets’

Statusofmiddleware–HTTPServer,WebSphereApplica4onServer,DB2,JMeterAferStoppingWebSphereonguestzs93kbg100107

Recovery complete

23

Ganglia–Host&Guestmetrics

24

Ganglia–zHPMServiceClassPerformanceIndexandaverageCPUshares

# date; zhpm config --cpu-mgmt off Fri Mar 4 16:19:40 EST 2016 zHPM CPU Management is off

25

ToolsandProductswehavetestedwithKVMforIBMzSystems

Open Source Tools running on Linux for z, interacting with KVM for IBM z Systems •  Rundeck 2.6.1 •  Nagios-core 3.5.1 •  Nagios plugins 2.0.3 (Nagios plugins ping guests and hostsm tests middleware availability. Nagios also runs the

check-ganglia plugin). •  Nagios NRPE 2.15 (Nagios Remote Plugin Executor (npre) runs in guests and send metrics back to Nagios

server on Linux on z) •  Ganglia web frontend 3.5.12 / Ganglia Monitoring Core 3.6 •  Host sFlow 1.26.2 (running on KVM host feeding metrics to ganglia server on Linux on z, Open vSwitch feeding

metrics to SiLK for viewing w/FlowViewer on Linux on x) •  gmetric 3.1 (running on KVM host feeding metrics to Ganglia server on Linux on z)

Open Source Tools running on x86 Linux, interacting with KVM for IBM z Systems •  Chef server 12.2 (Chef server on x86) •  Chef-client 12.0.3 (chef-client running on KVM hosts and guests. Chef-client for z is not available from the Chef

community, so we built our own from source. For instructions, see https://github.com/linux-on-ibm-z/docs/wiki/Building-Chef-client-12.1.2

•  logstash 1.4.2 (rsyslog on KVM hosts shipped logs to logstash on x86) •  ElasticSearch 1.4.2 •  Kibana 3.1.2 •  SiLK 3.11, FlowViewer 4.6 (Open vSwitch metrics) •  IBM Cloud Manager with OpenStack v4.3 FP4 (Controller on x, Compute nodes & Cinder host on z)

26

ForMoreInforma>on

Sources of Further Information •  Getting Started with KVM for IBM z Systems, SG24-8332-01. Chapter 4 discusses Nagios, ElasticSearch,

Logstash, Kibana, and more.

•  KVM for IBM z Systems v 1.1.2: Planning and Installation, SC27-8236-02.

•  KVM for IBM z Systems v1.1.2: System Administration, SC27-8237-02. Chapter 11 discusses Pacemaker and Corosync.

27

Ques>ons?

28

KVM for z Systems (1.1.1) and LinuxONE supports

Servers

IBM z13™

IBM z13s™

IBM LinuxONE Rockhopper™

IBM LinuxONE Emperor™

IBM zEnterprise® zEC12 IBM zEnterprise® zBC12

Guest Operating systems supported

SUSE Linux Enterprise Server (SLES12 SP1) Ubuntu 16.04 LTS for LinuxONE and for IBM z Systems

Networking features supported (NICs)

IBM OSA-Express5S IBM OSA-Express4S

Crypto Coprocessor supported

Crypto Express4S Crypto Espress5S

Storage devices are supported .

ECKD™ DASD §  DS8000® (FICON®-attached)

FCP SCSI disks: §  XIV®

§  Storwize® V7000, V5000, V3700, V3500 §  FlashSystems™ §  SAN Volume Controller §  DS8000 (FCP-attached) §  DS8880 (FCP-attached)

Note:RefertotheKVMforIBMzSystems:PlanningandInstalla>onGuide(SC27-8236)forthemostcurrentinforma>on