silo busting removing the wall between system and storage

© Copyright IBM Corporation 2016. Technical University/Symposia materials may not be reproduced in whole or in part without the prior written permission of IBM.

Silo Busting

Removing the wall between system and storage administrators

Chuck Laing Senior Technical Staff Member (STSM) IBM GTS SO/IS Delivery

Link: http://www.ibmsystemsmag.com/power/infrastructure/storage/silo_busting/

1 IBM Systems Technical Events | ibm.com/training/events

© Copyright IBM Corporation 2016. Technical University/Symposia materials

may not be reproduced in whole or in part without the prior written permission of

IBM.

Objectives

• Considerations that trigger Crit Sits and Performance issues

• Share and Share Alike – Top Ten things to know

• Example 1 - Concepts for solutions and example 4 step migration process

• Example 2 - Physical/Logical Considerations Understand the underlying

disk structure of LUNs

• Understand the virtual nature of SVC, DS8K, V7000 and XIV Storage

• Making it Work – communicating key points

• My object is to teach:

• Examples that foster communication between towers

• Concepts that will make you better

• Concepts that will make your job easier




IBM.

Problem statement

• Storage and System Administrators often clash in the common goal to

achieve data performance and availability, leading to:

• Too many logical configuration related outages

• Performance related enhancements not working to specification

• Leading causes:

• Lack of understanding configurations

• No cohesiveness between the logical and physical implementations

• Lack of communication between System and Storage Administrators

• Resulting in:

• A lack of data reliability and IO throughput

• Customer dissatisfaction

2




IBM.

Addressing considerations that trigger

Crit Sits and Performance issues

• The 3 most common issues that cause performance

degradation and application outages are:

• Hardware component failures

• Configuration changes

• IO load shifts - caused by anything ranging from imbalance through data growth to

failure of components

• How do I address and avoid:

• Data Safety

• Outages

• Extended or unexpected downtime

• Data corruption

• Missing data or data loss

• Application performance issues

• Technical compatibility issues

• Risk associated with changes in the environment




IBM.

• As System’s Administrators – we don’t always KNOW what we don’t know

about storage

• Ask for storage, leveraging what you know

• Avoid bottlenecks

• Use tools available

• Speed problem isolation

• Make more informed architectural decisions

• As Storage Administrators – we don’t always KNOW how the storage will

be utilized

• Make more informed architectural decisions

• Ask what is needed for best performance and IO separation

• What we are NOT going to do today:

• Try to turn Sys Admins into Storage Admins or vice versa

• Boil the ocean

Share and Share Alike - Knowledge is Power




IBM.

Systems Administrators should know about Storage

1. What should I be aware of/what should I avoid? (Tips & Pitfalls-Tuning)

2. Storage Overview - what's inside?

• What is the Physical makeup – drive types – RAID - Size?

• What is the Virtual makeup (good throughput design tips - IOPs)?

• What is a Storage Pool ?

• Where do I place data?

3. Connectivity- Picking the right drivers

4. Host Attachment Kits

5. How to Improve Performance using LVM

6. Documentation - why it matters

7. Topology Diagrams

8. Disk Mapping (view at a glance)

9. Easy Storage Inquiry Tools

10. How to avoid Bottlenecks

Storage Admins should know

about Hosts

1. What should I be aware of/what should I avoid? (Tips & Pitfalls-Tuning)

2. Host Operating System (OS) type

3. Hdisk Volume - LUN Layout - Purpose?

• DB type

• Access Patterns

• Number of spindles required

• Stripe

• Spread

• Mirror

4. What is a Volume Group (VG)?

5. What is a Logical Volume (LV)?

6. Disk Mapping (view at a glance)

7. Easy Storage Inquiry Tools

8. Cluster Properties

9. LPAR FC HBA port mappings

10. What causes Bottlenecks

The top ten things to know

5




IBM.

Plan

Execute

Validate

Evaluate

Example 1

Four Steps to communicate pattern for any process

Solutions Process


Evaluate

• Data and Storage cross pollination for Power Systems




IBM.

Example: Evaluate the data migration process

• Migrating data is always a disruptive process. Whatever the migration

technique used, it always affects to some degree the normal

operations of the system.

• Selecting the appropriate technique depends on:

• The criticality of the data being moved

• The resources available

• Other business constraints and requirements.

• Note: Risks should be identified depending on the migration technique

used. We strongly recommend that you consider selecting the

technique that is the best compromise between efficiency and the least

impact to the system users.




IBM.

Disk consolidation can trigger data migration of storage when:

• You want to change computer systems

• You need to upgrade to new products to

stay competitive

• New functions of evolving technology are

introduced

• Database growth

• You need newer, faster, higher density

devices

• Taking advantage of the ever improving

price to performance ratio of new storage

devices

• You just require more flexibility

• You need to relocate you data center

• You need to reduce the foot print of your

storage subsystem within the data center

• Leveraging data migration to provide

Disaster Recovery solutions

Storage migration can trigger LVM

data movement when:

• You want to spread IO evenly across all the disks in the VG

• You need to align IO access patterns • Random access

• Sequential access

• You want to protect the data integrity

• Database growth

• Database refreshes

• You need to consolidate the space in a VG or multiple VGs

• You need to troubleshoot an ailing volume for • Performance

• Availability (failure boundary)

• You need to separate data into separate LVs

Evaluating migration triggers




IBM.

Evaluate the migration technique summary for risks

• Make a list of Pros and cons (each offering strengths and limitations)

Migration technique Pros Cons

– Host-based – LVM

– LDM

– Add-on software such

as VxVM

– Volume (block) level

– TDMF

– Generally lowest initial implementation

cost

– Leverages existing and IP network

– LVM or LDM tools available

– Storage device-agnostic

– Leverages existing Operating System

skills

– Migration can happen on-line during peak

hours

– Consumes host resources

– Operating system specific

– Management can become complex and

time consuming

– Each host is its own island – no central

management console

– May cause an initial outage to install the

utility or software if it is not already

existing on the host

– Network-based – Fabric

– TDMF-IP

– Supports heterogeneous environments –

servers and storage

– Single point of management for replication

services

– Higher initial cost due to hardware &

replication software

– Requires proprietary hardware and may

require implementation of Storage

– Application-based – SVC

– Migration can happen on-line during peak

hours

– Supports heterogeneous environments –

servers and storage

– Single point of management for migration

– Requires an initial outage to bring the

host volumes on-line to SVC

– Requires the host to reboot to load or

upgrade the multipathing drivers

– Tape based – TSM

– Etc

– Does not require additional special tools,

software or hardware

– Does not require additional skills or

training

– Requires the disruption of the

applications and down time

– Slow and cumbersome


Plan





IBM.

Planning phase – Data Protection & identify risks

Example migration methodology plan

Action Item Assigned Status Date

– Establish a migration management team

– Gather availability and production schedules

– Document Change Control procedures and

incorporate into the plan

– Document the time line for migration activities

– Announce the migration at least 30 days prior to the

intended target migration date

– Gather information about the storage server

environment and applications (lists, commands,

scripts and/or drawings)

– Schedule a pre-migration rehearsal that includes all

the members on the migration team and a data

sampling that will enable the application groups to

appropriately conduct the pre- and post migration

verification process

– Establish a “Migration Status” call-in process

– Utilize a “Migration Planning Checklist” to assure

that all of the pre migration planning steps have

been executed




IBM.

Planning phase

• A successful data migration always requires substantial evaluation and

planning

• Adequate planning is the critical success factor in a migration project

• Develop a high level migration plan

• Develop a detailed migration plan




IBM.

Planning phase – design requirements

Action

Item

Application Environment

Databases to be moved (DB2, Informix®, Oracle®, SQL, Sybase)

Database version

Database size

Availability requirements of databases (any existing SLA’s, downtime issues

to consider)

Cluster environment (MSCS, Veritas, Sun, HACMP™, MC/Service Guard,

etc.)

Understanding the requirements may help simplify migration process

Action

Item

Network Environment (if applicable)

Topology

Speed of network




IBM.

Planning phase – design requirements

Action

Item

Storage Environment

Storage Vendor and model (EMC, HDS, IBM, STK, Dell, HP)

Channel type (ESCON, FICON, Fibre, iSCSI, SAN)

SAN HBA & Model (Qlogic, Emulex, JNI)

Number of Channel Paths

Logical to Physical mapping (i.e. RAID-1 vs. RAID-5)

Number of Source volumes to be migrated

Volume sizes

Identify Target volumes to receive source data

Understanding the requirements may help simplify migration process




IBM.

Planning summary – avoiding data corruption

Example migration methodology approach

Action

Item Migration and validation methodology checklist

Based on the information gathered in the planning phase, structure the

migration architecture to match the production requirements

Use checklists to ensure any operating patches and software are at

the correct levels

Build detailed migration procedures following the chosen architecture

Put together a schedule of events with time lines to implement the

migration procedures

Establish an initial test plan to validate the initial installation of all

required components

Develop a cooperative deployment plan

Write and configure any automation scripts that will speed up the

process

Run a simple initial test plan that validates the migration process

Implement the migration procedures and time line built in the design

phase

Verify the migration completion by checking the successful completion

and status of the migration jobs

Utilize a “Migration Planning Checklist” to assure that all of the pre migration planning steps

have been executed.


Execute





IBM.

Execute

• During the migration phase, you will need to:

• Communicate your plans

• Obtain, install and configure any necessary:

• Hardware

• Software

• Automation scripts and tools (to perform the actual data migration)




IBM.

Establish good communication with towers and teams

• Team members may include but are not limited to:

– Project manager

– Client (account) manager

– DBA/Application owners

– System administrator

– Network administrator

– Security administrator

– Firewall administrator

– Disk storage administrator

– Backup/Recovery administrator

– SAN fabric administrator

– Hardware CE

– Floor planner

– Cable vendor

– Disaster/Recover administrator

– IT Architect




IBM.

Execute

An example migration may go as follows:

–This high level illustration is the execution migratepv –l


Validate





IBM.

Validate

• It is important to validate that you have the same data and functionality

of the application after the migration

• You should make sure that the application runs with the new LUNs,

that performance is still adequate, that operations and scripts work

with the new system




IBM.

A sample validation list may include

but not be limited to the following items:

• Compile migration statistics

• Prepare a report to highlight: • What worked

• What didn’t work

• Lessons learned

• Share the report with all members of the migration team

• These types of reports are critical in building a repeatable and consistent process through

continuous process improvement, building on what worked and fixing or changing what

didn’t work. Further, documenting the migration process can help you train your staff, and

simplify or streamline the next migration you do, reducing both expense and risk




IBM.

Migration Methodology Summary

Validate Evaluate Plan Execute

– Analyze business

impact

– Risks

– Business

interviews

– Criticality of data

being moved

– Performance

– Migration types

– Key factors

– Multi-vendor

environment

requirements

– Application down

time

– Determine

migration

requirements

– Identify existing

environment

– Define future

environment

– Create migration

plan

– Develop design

requirements

– Migration types

– Create migration

architecture

– Develop test plan

– Obtain software

tools and licenses

– Communicate

deployment plan

– Validate HW & SW

requirements

– Customize

Migration

procedures

– Install & configure

– Run pre-validation

test

– Perform migration

– Verify migration

completion

– Run Post validation

test

– Perform knowledge

transfer

– Communicate

Project information

– Create report on

migration statistics

– Conduct migration

close out meeting




IBM.

Share and Share Alike

• If the storage expert and the OS expert share information and produce

vital documentation together, the environment is less vulnerable to

performance woes:

• Missing dependencies

• Outages

• Risks introduced through change

• Data corruption

• Performance degradation

• Etc…etc…etc


Example 2





IBM.

27

Grid Building Block

-Data Module (1-15)

-CPU

-Memory (360GB/720GB)

-12 disk drives (1, 2, 3, 4, 6TB)

-Optional SSD 360/720 cache

External Connect

-Interface/Data Module (4-9)

-24 FC ports – 8Gb

-iSCSI ports – 22 GbE or 12 10GbE (Model 214)

Internal Interconnect

-2 Infiniband Switches

-3 UPSs Gen 3 Spectrum Accelerate

Example 2 - Physical/Logical Considerations

Build with a Strong Foundation

• Model 961 base and model 96E expansion

• 961 with up to 3 x 96E expansion frames

• 2.5” small-form-factor drives; 3.5” nearline; 1U High Performance Flash Enclosures

• 6 Gb/s SAS (SAS-2)

• Maximum of four frames

• Maximum of 1,536 drives plus 240 Flash cards

• Top of rack exit option for power and cabling

• Each frame has its own redundant set of power cords

• Two POWER7+ servers 4.228 GHz processors

• 2, 4, 8, and 16 core processor options per storage controller

• DS8870 has dual active/active controllers

• Up to 1TB of processor memory

• Host adapters

• Up to 64 16 Gb/sec or 128 8 Gb/sec ports or combination of 8/16 Gb/sec ports

• Each port supports FCP and FICON at the port level

• Base frame and first expansion frame allows 16 adapters. For 8 Gb/sec, both 4 and 8 port host adapter cards available. For 16 Gb/sec, 4 port host adapter card available

• All Flash configuration has all host adapters in the base frame

• Efficient front-to-back cooling (cold aisle/hot aisle)

V7000 V9000 • Great for

multiple mixed

workloads that

drive huge I/O

• Scale out for

more all flash

capacity, IOPS

and bandwidth

• Up to 2.5M

IOPS, 200µs

(.2ms)

• Up to 228TB

usable, 1.1PB

Effective




IBM.

Making it work

• Fully document LPAR to hdisks, databases to LPARs, HBA, paths and so forth.

• Document how virtualization is implemented and underlying physical structures to clarify the solution.

• Map out application-specific requirements or mappings introduced by mirroring or availability solutions.

• List parameters that differ from defaults (and why).

• Pipe historical output (for example network, path, virtualization results, data path queries Fibre channel statistics) to an alternative server to keep a history of changes. This output can be quite useful in diagnostic endeavors after a change.

• Diagram storage pools and how striping and mirroring are implemented. (Striping twice or incorrectly is a very negative performance factor.)

• Create switch and cable diagrams and network diagrams. It’s essential.

• Produce job-flow diagrams to pinpoint data-access patterns.

28




IBM.

Foundation - Build a SAN Environment

Seems simple enough right?

• Build the Pools, at the Storage Device • Choose your storage type by disk characteristics, speeds and feeds

• Create Volumes from those pools

• Use ET, Compression, technology to render best Performance

• Connect Hosts to the Storage though the SAN Fabric

• Zone for redundancy and resiliency

• Configure settings to Best Practices

• Configure hosts to take advantage of the Storage Foundation

• Configure VIOS, LPARs, VMs, etc.

• Distribute virtual aspects appropriately

• Map the volumes to the hosts

• Create the file systems , LPs, PPs, PVs, LPs, VGs, etc

• Place the applications on the configured hosts




IBM.

Foundation - Slow Performance or Outage Occurs

Now What?

• You followed the recipe • You took advantage of all the technology, features and functions by:

• Minimizing and automatically migrating volume IO hotspots – using ET in Pools

• Dual connecting all ports from the Storage to the Hosts

• Using good redundant performing storage foundation building blocks

• What happened?

• The cookies came out of the oven with clumps of: • Salt, baking soda and brown sugar in spots.




IBM.

What automated storage inquiry tools can

help me understand my setup?

• Storage tools

• Gathers information such as, but not limited to:

• LUN layout

• LUN to Host mapping

• Storage Pool maps

• Fabric connectivity

• SAT - (SATHC) - (SATPD) - (HW/SW Currency) – (ECM) –Cisco – Brocade-SVC-

DS8K,-DS6K-DS5K-DS4k-XIV-NetApp-EMC VMAX-Quantastor-V7000b-V7000u-

• Go to the following Website to download the tool:

• http://bldgsa.ibm.com/projects/s/storage_automation/sat/index.html

• DS8QTOOL –DS8000 type


• http://congsa.ibm.com/~dlutz/public/ds8qtool/index.htm

• SVCQTOOL – Spectrum Virtualize


• http://congsa.ibm.com/~dlutz/public/svcqtool/index.htm

http://congsa.ibm.com/~dlutz/public/ds8qtool/index.htm




IBM.

© Copyright IBM Corporation 2014

DEV#: 81 DEVICE NAME: hdisk81 TYPE: 2145 ALGORITHM: Load Balance

SERIAL: 60050768019002F4A8000000000005C7

======================================================================

Path# Adapter/Path Name State Mode Select Errors

0 fscsi0/path0 FAILED NORMAL 89154 2

1* fscsi0/path1 FAILED NORMAL 63 0

2 fscsi1/path2 OPEN NORMAL 34014 3

3* fscsi1/path3 OPEN NORMAL 77 0

Using the right tools -

Mapping Virtual LUNS to Physical Disks

• On the host server using SDD

LUN

to

Pool

to

Array

• Ask the StoAdmin to find disk/device UID or Raid-group in Storage Pool

• StorAdmin cross-references Storage Pool UID with Controller’s Arrays in Pools




IBM.

Throughput and Performance

Key Optimization Factors

• Throughput

• Spreading and balancing IO across

hardware resources

• Controllers

• Ports & zoning connections

• PCI’s Cards

• CPUs, RAM

• Disks spindles

• Compression

• Thin Provisioning

• Easy Tier - SSD

• Etc….

• IO Performance Tuning

• Using utilities, functions and features

to tweak (backend, frontend)

• Qdepths

• HBA transfer rates

• FC adapters

• LVM striping vs spreading

• Data Placement

• Random versus sequential

• Spreading versus Isolation

• Application characteristics

Configuring throughput optimally increases potential performance scalability




IBM.

Troubleshooting - Sys Admin -

How do I improve disk performance on the Host?

• Reduce the number of IOs

– Bigger caches

• Application, file system, disk subsystem

– Use caches more efficiently

– No file system logging

– No access time updates

• Improve average IO service times

– Better data layout

– Reduce locking for IOs

– Buffer/queue tuning

– Use SSDs or RAM disk

– Faster disks/interfaces, more disks

– Short stroke the disks and use the outer edge

– Smooth the IOs out over time

• Reduce the overhead to handle IOs




IBM.

Troubleshooting- StorAdmin –

How do I improve disk performance?

• Data layout affects IO performance more than any tunable IO parameter

• If a bottleneck is discovered, then some of the things you need to do are:

– Identify the hardware resources the heavy hitting volumes are on

• Identify which D/A pair the rank resides on

• Identify which I/O enclosure the D/A pair resides on

• Identify which host adapters the heavy hitting volumes are using

• Identify which host server the problem volumes reside on

• Identify empty non used volumes on other ranks – storage pools

– Move data off the saturated I/O enclosures to empty volumes residing on less used

ranks/storage pools

– Move data off the heavy hitting volumes to empty volumes residing on less used hardware

resources and perhaps to the another Storage Device

– Balance LUN mapping across

• Backend and host HBAs

• SVC IOgrps

• SVC preferred nodes

– Change Raid type.




IBM.

Summary

• Knowing - what's inside will help you make informed decisions?

• You should make a list of the things you don’t know

– Talk to the Storage Administrator or those who do know

• A better Admin understands • The backend physical makeup

• The backend virtual makeup

• What's in a Storage Pool for better data placement

• Avoids the Pitfalls associated with IO Tuning

• Knows where to go to get right device drivers

• Knows why documentation matters

• Keeps Topology Diagrams

• Keeps Disk Mapping documentation

• Is able to use Storage Inquiry Tools to find answers

Understands how to troubleshoot storage performance bottlenecks Silo Busting article in IBM System Magazine ->

http://www.ibmsystemsmag.com/power/infrastructure/storage/silo_busting/

http://www.ibmsystemsmag.com/power/infrastructure/storage/silo_busting






IBM.

Questions and Answers


Definitions





IBM.

IOPS

• IOPS (Input/Output Operations Per Second) is a common performance

measurement used to benchmark computer storage devices like hard disk

drives (HDD), solid state drives (SSD), and storage area networks (SAN).

• IOPS is the standard unit of measurement for the maximum number of

reads and writes to non-contiguous storage locations. The RAID type

defines the number of IOPS




IBM.

Throughput

• Throughput is a measurement of the average number of megabytes transferred within a period of time for a specific file size. Back in the day this was performed using a single computer making a single request for a disk, but in today’s age with large storage arrays that are providing storage to a number of clients we need to measure based on a lot of small read/writes verses a single computer making a large request.

• IO size (random) x IOPS = Throughput

• IOPS = Throughput/IO (all value in kb) or IOPS=[MB/KB]*1024 or IOPS = (MBps Throughput / KB per IO) * 1024 [since 1mb=1024kb] So using the above, if I wanted to configure an IOPS limit to satisfy a 10 MBps throughput using a 8KB IO request size I would require to set the IOPS limit to 1280. First let us convert 10MBps to kbps 10*1024=10240 IOPS = (10240/8) = 1280




IBM.

What is Latency?

• For storage subsystems, latency refers to how long it takes for a single

data request IO ( input/output) to be received and the right data found and

accessed from the storage media.

• In a disk drive, read latency is the time required for the controller to find

the proper data blocks and place the heads over those blocks (including

the time needed to spin the disk platters) to begin the transfer process.

• The reality is that three factors intertwine to impact overall storage

performance: IOPS, latency, and bandwidth.

• The effect of low latency storage I/O is more data flows through the same

hardware infrastructure in the same amount of time as legacy storage

systems




IBM.

The Storage Performance Ecosystem

• Storage systems have four basic components that create an ecosystem.

• First is the media on which users store and access data.

• The second component is the storage software. It controls how data is

written to the media as well as providing advanced features like data

protection, snapshots, and replication. This software component should

also dispatch and schedule I/O traffic to back-end media.

• The third component is the CPU processing that drives the storage

software.

• Finally, there is the storage network. It transfers data back and forth to the

application tier.




IBM.

LUNs, Volumes and SAN

• LUNs are logical drives

• Storage subsystems have their physical disk partitioned into logically addressed

portions that allow host servers to access them. This partition is called a LUN. For

example, most PC users will be familiar with the partition of a single disk into a C: drive

• Volumes are groups of LUNs

• LUN and volume are frequently used interchangeably. But it is worth noting that volume

is also often used to describe groups of several LUNs created with volume manager

software.

• SAN zoning and masking maintains security on the fabric

• Provisioning LUNs and volumes is only one part of storage provisioning. The SAN

fabric must also be configured so that drive arrays and LUNs are managed, and

security on the SAN is achieved by ensuring only those servers that have authorization

can access specific LUNs. For this we use SAN zoning and masking.

• For more details on zoning best practices please go the following link ;

http://www-01.ibm.com/support/docview.wss?uid=tss1wp102488&aid=1




IBM.

1. StorAdmin -The stressing of the data layout for Oracle, SQL Databases and other application data especially for

customers using SVC based Storage with XIV, since Storage Administrators don't really have the ability to set where

the data is located, so the DB Admins need to organize the data and spread the data better...

2. StorAdmin - I think somehow we need to get equal cooperation from both sides when dealing with issues, as I said its

usually on us as a Storage group to prove that issue isn't with the SAN or Storage..

3. StorAdmin - Dynamic disks(Intel OS) can be created using one or more LUNS and in case of performance issues we

cannot move the data of one single LUN which is a part of the Dynamic Disk. So we always recommend the system

admins to use one single bigger LUN or the LUNs from the same class of disk from the back-end to get the similar

performance or else we get into the performance issues. -Bottlenecks

4. StorAdmin – Server Activation & Deactivation (SA&D) --This is not being followed by the system admins in several

accounts. When the server is getting deactivated, they should request the SAN team to un-map and release the

storage from the server during the deactivation process. Otherwise we run into SAN slow drain issues ( TSM servers

on the SAN and not being in use would cause this ). This would eliminate the Orphan LUNs as well from our side.

5. StorAdmin - With the deployment of ASP - Automated Storage Provisioning the System Administrators should have a

crash course for basic storage for their respective accounts.

6. StorAdmin - a simple deck - called it Storage fundamentals maybe 8 slides should be presented to the System Admins

Sample Content :

1. What's IOPS?

2. What's LUN?

3. When do we use bigger LUN size versus smaller?

4. What the sequence to follow to remove a LUN?

5. What are the different drive type ( speed) and when to use them?

6. What is the proper zoning best practice?

7. What makes a SAN storage environment ( just stick to the basics)?

What are Storage Administrators saying?

Feedback examples




IBM.

1. GTS TI&A Architect -Slide 28 - With automated provisioning, the tool will be taking over a lot of what is currently being

done manually and documented (or not) by a storage admin. Would like your input on what the tool ought to

automatically create in terms of documentation. Also, where should the automatically created documentation be sent

and/or stored?

2. GTS TI&A Architect - Slide 44 item 5 - I agree it would be good to have a "crash course" in storage for server admins in

accounts where automated storage provisioning is being deployed. Do you have an outline or deck we could start

from?

What are the TI&A team saying?

Feedback examples




IBM.

1. ??

2. ??

What are the Sys-Admins saying?

Feedback examples




IBM.

Session Evaluations

YOUR OPINION MATTERS!

Submit four or more session

evaluations by 5:30pm Wednesday

to be eligible for drawings!

*Winners will be notified Thursday morning. Prizes must be picked up at

registration desk, during operating hours, by the conclusion of the event.

1 2 3 4




IBM.

Continue growing your IBM skills

ibm.com/training

provides a comprehensive

portfolio of skills and career

accelerators that are designed

to meet all your training needs.

If you can’t find the training that is right for you with our

Global Training Providers, we can help.

Contact IBM Training at [email protected]

Global Skills Initiative

mailto:[email protected]

silo busting removing the wall between system and storage

Documents