storage consumption and chargeback

© 2009 IBM Corporation

Developing a SAN Storage Consumption Solution

CMG 2009


Trademarks & Disclaimer

The following terms are trademarks of the IBM Corporation:

Enterprise Storage Server® - Abbreviated: ESS

TotalStorage® Expert TSE

FAStT/DS4000/DS8000

AIX®

IBM SAN Volume Controller

IBM Flash Copy is a registered trademark of IBM

TPC is a registered trademark of IBM

Other trademarks appearing in this report may be considered trademarks of their respective companies.

UNIX is a registered trademark in the United States and other countries, licensed exclusively through X/Open Company Limited.

Microsoft, Windows, Windows NT, and the Windows logo are trademarks of Microsoft Corporation in the United States, other countries, or both.

EMC is a registered trademark of EMC Inc.

BCV is a registered trademark of EMC Inc.

StorageScope is a registered trademark of EMC Inc.

CLARiiON is a registered trademark of EMC.

HP-UX is a registered trademark of HP Inc.

Solaris is a registered trademark of SUN Microsystems, Inc

Linux is a registered trademark of Linus Torvalds in the United States, other countries, or both.

UNIX is a registered trademark of The Open Group in the United States and other countries.

NetApp is a registered trademark of Network Appliance Inc.

NetApp Operations Manager is a registered trademark of Network Appliance Inc.

Disclaimer

The views in this presentation are those of the author and are not necessarily those of IBM


Abstract

The information in this paper is based on my experience ‘fixing’ the storage consumption process for a large a managed infrastructure services environment. The names have been changed to protect the innocent but the processes, tools, and data issues are very real. The accuracy of storage consumption reporting used for chargeback is essential in maintaining IT’s reputation and customer trust.

This presentation discusses the challenges, processes and tools involved when establishing a robust SAN storage consumption process.


Agenda

Storage consumption basics

Requirements and objectives

The environment

The processes

The tools

Lessons learned


Storage Consumption Basics: What are we measuring?


Solution Objectives

Process:

– Fix orphan creating processes

– Develop process for identifying and remediating data issues

– Develop operational processes for automating problem identification

Tools

– Automate process of storage consumption reporting

– Develop re-usable tools for data collection and post-processing

– Provide minimal manual processing of data from sources

– Improve coverage of storage platforms through use of new tools and data sources

Data– Report 99% of storage allocated

– Identify secondary sources of data for reconciliation

– Remediate data issues

Business– Provide justification of variances based on change tickets


Storage Consumption Challenges


Environment


Environment Size – Data Complexity

Array Count

EMC 36

HP 4

IBM 19

NAS Filers 15

Other Logistics

Server Count 900

ECC/StorageScope Servers 3

Number of SAN Fabric Locations 6

Number of Array Migrations 54

Number of Input data Sources 17


Environment - Data InputsInput Description Input Source Description

NAS Base Composite Ops Mgr Report template – Manually ran

EMC NAS EMC StorageScope 6.0 Report template - Manually ran

Host Allocation Billing database Automatically generated via query

EMC Configuration Configuration database Automatically generated via query

Clariion Configuration Configuration database Automatically generated via query

DS8K Configuration Configuration database Automatically generated via query

Symbcv output EMC CLI Automatically gathered by shell scripts

Symdev list - fibre - nofibre

Configuration database Automatically generated via queries

Host_Consumption_Query EMC StorageScope 6.0 Report template - Manually ran

HP-Compaq Host-LUN HP-Compaq CLI Manually created

Host Status, LOB Asset Management Database Automatically generated report

Exclusion Host Names Asset Management Database Automatically generated report

Array Location Capacity report Manually created

Backup Media Servers Backup reporting team Manually created

Application Correlation Application table Manually created

Flash copy relationships Array management tools Manually created

Change Tickets Problem and Change Management Tools Automatically generated report


Process Section


Storage Consumption/Chargeback High Level Process Flow


Processes that Impact Storage Consumption


Migration Process

Yes YesYes

Yes

No No

Identify Scope

Perform Detailed Data

Collection

Analyze for Remediation

Remediation Required?

Proceed to Host Prep

Can it be integrated with TDMF

Install?


Schedule Outage for Remediation

SAN Migration Process

Host PrepSchedule HP

Outage Window

Complete Allocation Request

Perform Solution Review

Perform Remediation


Schedule Final Cutover

Window

Perform Host Prep Review

Execute Host Prep

Final CutoverMonitor

Migration Status

Perform Solution Review

SAN De-Allocation

Execute Final Cutover

Validate SAN

removal


Server Decommission Process


Host Rename/Host Repurpose


BCV Removal


Server/HBA Re-use


Storage Change Process


SRM Operational Processes


Exception Reporting and Remediation Process


Sample Exception Report

Server Name

Location

Status Usable GB

In Asset DB

IBM Supported

In Billing DB?

In TPC? IBM & EMC?

server10 Mars 1,280.00 no yes yes no no

server11 Pluto 1,121.00 no yes yes no no

server12 Mars Production 1,121.00 yes yes yes no no

server13 Pluto Production 851.35 yes yes yes no no

server14 Mars Production 851.35 yes yes yes no no

server15 Pluto Decommissioned 864.00 yes yes yes yes no

server16 Mars Decommissioned 1,120.00 yes yes yes yes no

server17 Pluto Decommissioned 385.00 yes yes yes no yes

server18 Mars Decommissioned 385.00 yes yes yes no yes

server19 Pluto Production 770.00 yes yes yes no no


Tools


Tool/Reporting Requirements

Legend

Reporting

Tool


Change Management Required Fields

Server Name

Purpose of Request

Location or datacenter

Physical Server name to SAN

Tier

Amount of usable storage being requested

New number of SAN ports


Historical Consumption Analyst Process Logical Flow


Improved Consumption Analyst Process Logical Flow


Sample Storage Consumption Summary ReportServer/Cluster Name Location Tier Type Raw Total

(GB) Usable Total (GB)

Change Ticket

Server1 Mars T1 IBM 431.81 376.00 57955

Server1 Mars T2 IBM 1,083.62 944.00 25892

Server2 Venus T2 IBM 588.19 514.00 44874

Server3 Pluto T1 IBM 1,281.17 1,121.00 80509

Server4 Pluto T1 EMC 1,442.60 851.35 39607

Server5 Pluto T1 EMC 121.14 106.00 15571



Server8 Pluto T2 EMC 1,303.62 1,120.00 31390

Server9 Uranus T1 IBM 1,189.71 1,025.00 34741

Server10 Uranus T2 IBM 7,100.43 6,154.00 16917

Server11 Uranus T2 IBM 295.62 256.00 19851

Server12 Uranus T2 EMC 4,712.76 4,097.00 48131

Server13 Uranus T2 EMC 443.05 385.00 27053




Sample Detailed Consumption Report

Server/Cluster Name Array Location

Array SN Tier Type LUN LUN Type

Sum of Raw (GB)

Sum of Usable (GB)

Server3_Server4 Pluto 1234567 T1 EMC 1000 STD 67.43 33.72

T1 EMC 1001 STD 67.43 33.72

T1 EMC 1002 STD 67.43 33.72

T1 EMC 1003 STD 67.43 33.72

T1 EMC 1004 STD 67.43 33.72

T1 EMC 1005 STD 67.43 33.72

T1 EMC 1006 STD 67.43 33.72

T2 EMC 3001 BCV 38.53 33.72

T2 EMC 3002 BCV 38.53 33.72

T2 EMC 3003 BCV 38.53 33.72

T2 EMC 3004 BCV 38.53 33.72

T2 EMC 3005 BCV 38.53 33.72

T2 EMC 3006 BCV 38.53 33.72

T2 EMC 3007 BCV 38.53 33.72


Sample InvoiceSupplier

NameInvoice Number

Invoice Date Tower Application ID Rate Element Unit of Measure

Rohan Riders 88305 5/14/2011 Orthanc Oracle Disk Storage GBs Gigabyte

Rohan Riders 88305 5/14/2011 Orthanc WebSphere Disk Storage GBs Gigabyte

Rohan Riders 88305 5/14/2011 Orthanc Oracle Disk Storage GBs Gigabyte

Rohan Riders 88305 5/14/2011 Barad-dur WebSphere Disk Storage GBs Gigabyte

Rohan Riders 88305 5/14/2011 Barad-dur Oracle Disk Storage GBs Gigabyte

Rohan Riders 88305 5/14/2011 Barad-dur WebSphere Disk Storage GBs Gigabyte

Server Name

Equipment Location Useable Quantity

LOB Rate Per GB

Amount

server45 IBM Isengard 500.00 Urak-hai $1.94 $969.31

server46 IBM Isengard 500.00 Urak-hai $1.94 $969.31

server49 IBM Isengard 500.00 Palantir $1.94 $969.31

server50 EMC Mordor 1,000.00 Shelob $1.94 $1,938.61



Continued -


Lessons Learned

Get Sponsored

True-Up Candidate

Project Management

Create Baseline

Avoid Surprises

Persevere


Definitions

Storage consumption – How much space is associated with each server

SAN – Storage area network

Tier – Classification of storage based on hardware attributes such as disk size, speed, capacity, controller type or RAID configuration.

HBA – Host Bus Adapter provides SAN connectivity from a host to the backend storage device.

WWPN – World Wide Port Name is a unique identifier tied to the HBA hardware. It is used to manage the connectivity access from the host to the backend storage targets.

LUN – SAN volumes assigned to hosts

BCV – Business Continuance Volumes are an EMC specific technology used for creating copies of production volumes.


Appendix A – What is a SAN?

Edge Switch - A

Edge Switch - B

Core Switch - A

Core Switch - B

Links Links

Storage Switch - A

Storage Switch - B

Servers Fabric Storage Servers

storage consumption and chargeback

Documents