Download - SPW Lab Guide
EMC Education Services
Symmetrix Performance Workshop
April 2012
Copyright
Copyright © 1996, 2000, 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012 EMC Corporation. All Rights Reserved. EMC believes the information in this publication is accurate as of its publication date. The information is subject to change without notice.
THE INFORMATION IN THIS PUBLICATION IS PROVIDED “AS IS.” EMC CORPORATION MAKES NO REPRESENTATIONS OR WARRANTIES OF ANY KIND WITH RESPECT TO THE INFORMATION IN THIS PUBLICATION, AND SPECIFICALLY DISCLAIMS IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
Use, copying, and distribution of any EMC software described in this publication requires an applicable software license.
EMC2, EMC, Data Domain, RSA, EMC Centera, EMC ControlCenter, EMC LifeLine, EMC OnCourse, EMC Proven, EMC Snap, EMC SourceOne, EMC Storage Administrator, Acartus, Access Logix, AdvantEdge, AlphaStor, ApplicationXtender, ArchiveXtender, Atmos, Authentica, Authentic Problems, Automated Resource Manager, AutoStart, AutoSwap, AVALONidm, Avamar, Captiva, Catalog Solution, C-Clip, Celerra, Celerra Replicator, Centera, CenterStage, CentraStar, ClaimPack, ClaimsEditor, CLARiiON, ClientPak, Codebook Correlation Technology, Common Information Model, Configuration Intelligence, Configuresoft, Connectrix, CopyCross, CopyPoint, Dantz, DatabaseXtender, Direct Matrix Architecture, DiskXtender, DiskXtender 2000, Document Sciences, Documentum, elnput, E-Lab, EmailXaminer, EmailXtender, Enginuity, eRoom, Event Explorer, FarPoint, FirstPass, FLARE, FormWare, Geosynchrony, Global File Virtualization, Graphic Visualization, Greenplum, HighRoad, HomeBase, InfoMover, Infoscape, Infra, InputAccel, InputAccel Express, Invista, Ionix, ISIS, Max Retriever, MediaStor, MirrorView, Navisphere, NetWorker, nLayers, OnAlert, OpenScale, PixTools, Powerlink, PowerPath, PowerSnap, QuickScan, Rainfinity, RepliCare, RepliStor, ResourcePak, Retrospect, RSA, the RSA logo, SafeLine, SAN Advisor, SAN Copy, SAN Manager, Smarts, SnapImage, SnapSure, SnapView, SRDF, StorageScope, SupportMate, SymmAPI, SymmEnabler, Symmetrix, Symmetrix DMX, Symmetrix VMAX, TimeFinder, UltraFlex, UltraPoint, UltraScale, Unisphere, VMAX, Vblock, Viewlets, Virtual Matrix, Virtual Matrix Architecture, Virtual Provisioning, VisualSAN, VisualSRM, Voyence, VPLEX, VSAM-Assist, WebXtender, xPression, xPresso, YottaYotta, the EMC logo, and where information lives, are registered trademarks or trademarks of EMC Corporation in the United States and other countries.
All other trademarks used herein are the property of their respective owners.
© Copyright 2012 EMC Corporation. All rights reserved. Published in the USA.
Revision Date: March 28, 2012 Revision Number: MR-1CP-SPERET 1.5
EMC Education Service
3 EMC Education Service
Document Revision History
Rev # File Name Date
1.4 SPW_lab_guide.docx March 2012
1.5 SPW_lab_guide.docx April 2012
4 EMC Education Service
Table of Contents
COPYRIGHT............................................................................................................. 2
DOCUMENT REVISION HISTORY ............................................................................. 3
LAB EXERCISE 1: TOOLS .......................................................................................... 5
LAB 1: PART 1 – USING SYMMETRIX PERFORMANCE ANALYZER.............................................6
LAB 1: PART 2 – LAUNCHING PERFORMANCE MANAGER....................................................14
LAB 1: PART 3 – MAPPING LOCAL ARCHIVES ...................................................................16
LAB 1: PART 4 – PERFORMANCE MANAGER FEATURES ......................................................18
LAB EXERCISE 2: ANALYSIS.................................................................................... 24
LAB 2: PART 1 – WORKLOAD CHARACTERIZATION ............................................................25
LAB EXERCISE 3: SYMMETRIX FRONT-END ............................................................ 28
LAB 3: PART 1 – EFFECT OF I/O SIZE ..............................................................................29
LAB 3: PART 2 – GUIDED ANALYSIS ................................................................................31
LAB 3: PART 3 – AUTO DESIGN FRONT-END PERFORMANCE ...............................................34
LAB 3: PART 4 –24/7 FRONT-END PERFORMANCE ..........................................................36
LAB EXERCISE 4: SYMMETRIX CACHE .................................................................... 38
LAB 4: PART 1 – EFFECTS OF RANDOMNESS ....................................................................39
LAB 4: PART 2 – GUIDED ANALYSIS ................................................................................41
LAB 4: PART 3 – ONE HOST REPORTING PROBLEMS ...........................................................44
LAB 4: PART 4 – AUTO DESIGN CACHE PERFORMANCE .......................................................46
LAB 4: PART 5 – 24/7 CACHE PERFORMANCE .................................................................48
LAB EXERCISE 5: SYMMETRIX BACK-END .............................................................. 50
LAB 5: PART 1 – GUIDED ANALYSIS ................................................................................51
LAB 5: PART 2 – AUTO DESIGN BACK-END PERFORMANCE ..................................................54
LAB 5: PART 3 – PREFETCH ANALYSIS .............................................................................56
LAB 5: PART 4 – 24/7 BACK-END PERFORMANCE .............................................................58
LAB EXERCISE 6: BUSINESS CONTINUITY ............................................................... 60
LAB 6: PART 1 – GUIDED ANALYSIS ................................................................................61
LAB 6: PART 2 – UNEXPECTED NIGHT DELAYS ..................................................................63
5 EMC Education Service
Lab Exercise 1: Tools
Purpose:
To become familiar with the use of Symmetrix Performance
Analyzer and Performance Manager
Tasks: In this lab, you will perform the following tasks:
Use Symmetrix Performance Manager to examine workload
characteristics
Map and open archives in Performance Manager
Use Views and Metrics to explore the archive
Create new views, change the appearance of graphs, and export
data
EMC Education Service
6 EMC Education Service
Lab 1: Part 1 – Using Symmetrix Performance Analyzer
Step Action
1 Launch Symmetrix Performance Analyzer by double-clicking the desktop icon URL, or by
entering this URL in Internet Explorer:
https://hostname:8443/spa
If it warns you about security or certificate issues, just confirm in a way that continues
the program launch.
Log in using the user and password your instructor has provided, or with the default:
Login: smc Password: smc
It may take up to a minute after the initial launch for the program to become active. Be
patient.
7 EMC Education Service
Step Action
2 Explore the default Dashboard.
Note that the default view when you log in to SPA is the default Dashboard – Heat Map.
Examine the Dashboard and identify the components that are actively performing work.
a. Record any components that are color coded red – indicating 100% utilization.
FA/RA Directors: __________________________________
Cache (Cache Partitions if any): ______________________
DA Directors: ______________________________________
Disks: ____________________________________________
b. Click on the components identified above and record their utilization.
FA/RA Directors: __________________________________
Cache (Cache Partitions if any): ______________________
DA Directors: ______________________________________
Disks: ____________________________________________
8 EMC Education Service
Step Action
3 Overall Symmetrix Performance.
Select the Snapshot view and click on the local Symmetrix in the left.
Use the “Utilization Distribution” to find out what the overall condition of the Symmetrix
components is. Do any of the components have greater than 25% utilization? If so, which
components (Hint: click on the bar in the snapshot to show more details)?
FA Dirs: ______________________________
Cache: ______________________________
BE Dirs: _____________________________
Disks (Disk Groups): ___________________
Use the “Throughput Distribution” to determine the Front end MB/s.
FA: _________________________MB/s
Are there any Device and/or Storage groups listed in the Snapshot view. Use the “Device
Groups – I/Os Per Second & Response Time Distribution” to find the device group (do not
include ungrouped devices) having the highest utilization. What rating is the group’s
response time?
Group:_______________________________ Response time rating: ________________
What is the overall IOs Per Second to this Symmetrix? ___________________
9 EMC Education Service
Step Action
4 Symmetrix Storage Group Snapshots(1).
Use the navigation panel on the left to locate and click on the Storage Group that you
identified in the previous part. Make sure you are still viewing Snapshots.
Use the “Storage Group Profile” to find IO profile for this Group:
Determine if the workload is predominantly reads or writes (%): ___________
Is the workload predominantly random or sequential: ____________________
The overall Hit rate and Miss rate: __________________________
What are the average read and write IO sizes: ______________________
Does this Storage Group contain Thinly Provisioned devices: ______________
10 EMC Education Service
Step Action
5 Symmetrix Storage Group Snapshots(2).
Use the navigation panel on the left to locate and click on the Storage Group that is
Virtually Provisioned. Make sure you are still viewing Snapshots.
Use the “Storage Group Profile” to find IO profile for this Group:
Determine if the workload is predominantly reads or writes (%): ___________
Is the workload predominantly random or sequential: ____________________
The overall Hit rate and Miss rate: __________________________
What are the average read and write IO sizes: ______________________
Record the information relevant to VP for this Storage Group:
IO Density by Tier: _____________________________
Symmetrix Tier Capacity Allocation %: _________________________
BE IOs by Tier: ____________________________
BE Throughput by Tier: _______________________
IOs Per Second to this Storage Group:_________________________
6 Symmetrix Front-End Directors Snapshots(1).
Open the sub-folders under the Symmetrix array in the navigation pane on the left and
click on the “FE Directors” folder. Make sure you are still viewing Snapshots.
Identify the Director(s) that are performing the most work (IOs Per Second and
Utilization. _________________________________
Identify the Port(s) that are performing the most work (MBs Per Second)._____________
11 EMC Education Service
Step Action
7 Symmetrix Front-End Directors Snapshots(2).
Open the “FE Director” folder and choose just one front-end director identified in
step(6). Make sure you are viewing Snapshots.
What is the IO contribution (%) of this director to the overall workload on the Symmetrix:
______________________
Is this director servicing more reads or more writes: ________________________
What is the Throughput contribution (% and MB) of this director to the overall workload
on the Symmetrix: _____________________________
Are both the Front-end ports on this director equally utilized: ___________________
8 Symmetrix Back-End Directors Snapshots.
Click on the “BE Directors” folder, and make sure you are viewing Snapshots.
Use the Snapshot views to monitor the balance of the back-end directors.
Are some directors being utilized more than others? __________
Can you detect similar patterns of usage across two or more directors?
_______________________
If you detected imbalance on the front-end directors, but mostly balance on the back-
end directors, what does this say about the arrangement of data in the array?
12 EMC Education Service
Step Action
9 Symmetrix Disks Snapshots.
Click the “Disks” folder, and make sure you are viewing Snapshots.
Use the Snapshot views to determine if any of the disk groups have an average utilization
of more than 25%:
_________________________________________________________________________
Do any of the disk groups have a peak utilization of more than 25%? ______________
Drill down under the “Disks” folder to click on a single disk group. Are the disks in the
group being utilized fairly evenly? __________________________________________
Compare a few of the disks in your group to see if they have similar traffic patterns.
10 Diagnostic view.
Select Diagnostic view. Observe the overall dashboard that is presented for the
Symmetrix.
Double-click on the Symmetrix and all the available Diagnostic view tabs are presented.
Explore the different components.
Select the FE Director tab. Then select the FE identified as busy in the earlier part of this
exercise. Switch to the Explore tab. Select both the Average Read Response Time and
Average Write Response Time and plot the graph.
Is there any abrupt change in the response times: ________________________
Diagnostic view can help in quickly identifying changes that have occurred recently.
Explore the other components for which Diagnostic view is available.
13 EMC Education Service
Step Action
11 Real Time view (1).
Select Real Time view. The overall metrics for the Symmetrix in the past hour with finer
granularity is displayed.
Select the Symmetrix. Then select FE Reqs/sec and CTRL select BE Reqs/sec. Plot the
graph.
Which of the two is consistently higher: ______________________________
We will discuss the reason for these later in the course.
12 Real Time view (2).
Double-click on the Symmetrix. This will provide three tables for FE, BE and RDF
Directors. Take some time to explore the metrics available for these components in the
Real Time view.
End of Lab 1 – Part 1
EMC Education Service
14 EMC Education Service
Lab 1: Part 2 – Launching Performance Manager
Performance Archive Used In This Part
Symmetrix/000194900180/interval/20091124.btp. All performance archives are available on the
desktop of your classroom workstation. They can also be found on the student CD.
Step Action
1 Launch Performance Manager by double-clicking the desktop icon, or by using
the EMC > ControlCenter > Performance Manager menu of the Windows Start
menu.
Click the “Work Offline” box in the login dialog and then “OK” to launch the
program.
If a “Data Providers” dialog appears, just click “Cancel.”
At the “Data Selection” dialog, choose the “Specific File” radio button and
browse to locate and open the archive used in this part (listed at the start of the
exercise).
15 EMC Education Service
Step Action
2 Viewing Vital Signs.
Click on the Vital Signs icon, and answer these questions using the graphs. You
might use the Window > Cascade or Window > Tile menus to view the Vital Signs
better.
What was the maximum I/O rate to the Symmetrix during the collection period?
_________
What was the maximum % Hit during the collection period? ________
Does the % Write tend to correspond with the % Hit measure? ________
3 Use the Window > Close All menu to close all the views.
Close Performance Manager in preparation for the next Part.
End of Lab 1 – Part 2
16 EMC Education Service
Lab 1: Part 3 – Mapping Local Archives
Performance Archive Used In This Part
Symmetrix/000194900180/interval/20091124.btp. All performance archives are available on your
classroom workstation, or the student CD.
Step Action
1 Launch Performance Manager using the desktop icon or the Windows Start
menu.
At the “Data Providers” dialog, click the “Add Group” button.
2 In the “Data Provider” dialog, Choose “Symmetrix” from the “Class” drop down
menu, or type in the word if it is not an available choice.
Browse to the “Symmetrix” folder of your archive set—it should be on the
Desktop. Click directly on the top-level folder named “Symmetrix,” not the folder
for a particular Symmetrix ID.
Click “OK” to save the selection and exit the “Data Provider” dialog.
If you did this correctly, a list of Symmetrix archives should appear in the “Data
Providers” dialog. If not, use the dialog to delete any mistakes and try again.
Click “OK” to exit the “Data Providers” dialog.
17 EMC Education Service
Step Action
3 Use the “Data Provider” and “Archives” parts of the “Data Selection” dialog—not
the “Specific File” part—to launch the archive used in this part. You can now
retrieve any archives used in the class using the “Data Provider” and “Archives”
part of the “Data Selection” dialog.
You may leave this archive open for the next Part.
End of Lab 1 – Part 3
18 EMC Education Service
Lab 1: Part 4 – Performance Manager Features
Performance Archive Used In This Part
Symmetrix/000194900180/interval/20091124.btp
Step Action
1 Launch Performance Manager and open the archive used with this part, if you have
not already done so.
2 Plot the All Fibre Directors – IO Rates graph. This graph (View) can be found in the
“Dir-Fibre” Folder. By default this graph will be plotted as a Line graph.
3 Graph Wizard—Ribbon Graph.
(a) Select the graph you opened (make sure its title bar is not gray) and click
the “Graph Wizard” icon ( ).
(b) Change the Chart Type to Ribbon, and click “OK”.
(c) Are all the directors performing equal amount of IO?______________
(d) Can you detect grouping of directors performing equal amounts of
IO?_______
(e) What are the groups? _______________
19 EMC Education Service
Step Action
4 Graph Wizard—Fill Style.
(a) Click the Graph Wizard icon again.
(b) Select the “Fill Style” tab.
(c) Change the Fill Style for the Legend, Display, and Title, to None.
(d) Click on “Apply.”
(e) Note the change in appearance of the graph.
(f) Close the graph and then launch it again. Were the changes you made
retained? __________
(g) Close the graph.
Explore other features of the Graph Wizard throughout the rest of the lab
exercises.
20 EMC Education Service
Step Action
5 Creating a New Data View.
(a) Right-click on the “System” folder and select “New Data View.”
(b) In the “Data View Definition” dialog, leave the Class as Symmetrix and
change the Identifier to an asterisk (*). What effect does changing the
Identifier to an asterisk have?
____________________________________________________________
(c) Select “Dir-DA” as the Category, “ios per sec” for the Metric, for All objects.
Check the “Sum Across Selected Objects” and click the “Add to Contents”
button.
(d) Repeat (c) for Category “Dir-Fibre”.
(e) Choose Ribbon as the Graph Style.
(f) Enter a Name (this will appear under the Systems folder when the
definition is created), and a Title. If you are connected to a Repository,
check the Public Views box and click OK. What functionality does checking
the Public Views box provide?
_____________________________________________________________
___
(g) Plot the graph from the “Systems” folder.
(h) The Ribbon format does not add any utility when the two lines are so
dissimilar. Right-click on the View, choose Modify Data View, change the
Graph Style to Line, and click “OK.”
(i) Plot the graph again to verify the change.
21 EMC Education Service
Step Action
6 Metrics Tab.
(a) Click on the “Dir-Fibre” folder in the top left panel. Select any director. In
the metrics panel, click on “ios per sec” and CTRL-click on “requests per
sec.”
(b) Click on the “Graph per Object” button ( ). The resulting graph now
displays both “ios per sec” and “requests per sec”.
(c) Click on the “Insert the graph into Views” button ( ).
(d) In the “Save Graph as Data View” dialog, check both the “Generalize Class
IDs” and “Generalize Objects” boxes, and give it a name.
(e) Click “OK” to save the new view.
(f) What category in the Views tab did the new view appear under? Why?
22 EMC Education Service
Step Action
7 Single-point Analysis Mode.
(a) Launch the “System View – Total Throughput to-from hosts” view (or any
similar line graph view) from the Views tab.
(b) Right-click on any part of the graph and enable Single Point Analysis mode.
(c) Select the Metrics tab and choose the “Dir-Port” folder in the top panel.
(d) Shift-click to choose all of the director ports that show a non-zero I/Os per
second value in the middle panel.
(e) Click on the “ios per sec” measure in the bottom panel.
(f) Click the “Graph histogram from sorted metric” button ( ). The resulting
histogram shows the I/O per second of each port at a given time in the time
graph.
(g) Click the time graph at the point of peak throughput. The histogram will
show the status of those objects at that time.
What is the time: __________ and approximate peak throughput?
______________
Record the top two ports and their I/O per sec at this time:
____________________________________________________________________
What is the time (or a time) of the lowest throughput? ____________________
Are the ports receiving any I/Os at that time? ___________________
Experiment with this feature by adding another histogram from the Metrics tab
and changing the time.
When you are done experimenting, you can close all the graphs you created to end
Single Point Analysis mode.
23 EMC Education Service
Step Action
8 Are the values in the graphs sometimes difficult to read? Open any Views or
Metrics graph, and then use the “Table” option. from the “Tools” menu to show
the time and data points that make up the graph. Try this now.
You can copy the data out of a table to spreadsheet or text editor, or you can just
save the graph as a CSV file. Select the graph and use “File > Save as csv > Graph”
to save the data. View the output in a spreadsheet program or Notepad.
Try the “File > Save Graph as” menu to save an image of the graph. Double-click
the image to view it.
End of Lab Exercise 1
24 EMC Education Service
Lab Exercise 2: Analysis
Purpose:
To become familiar with the use of Symmetrix Performance
Analyzer and Performance Manager
Tasks: In this lab, you will perform the following tasks:
Analyze the archive to determine average workload
characteristics
Analyze the archive to determine profile of usage, or the
changes in the workload over time
Analyze the archive to determine component usage, or the way
in which the workload is distributed over the components
25 EMC Education Service
Lab 2: Part 1 – Workload Characterization
Performance Archive Used In This Part
Symmetrix/000284500356/daily/20021215d.btp
Workload Characteristics
Examine the archive used in this part and record the following information about the Workload Characteristics—the
basic measures for the overall system. Use any graph from the Views or Metrics tab of Performance Manager to get
your answers. To get an average number for a measure that is constantly changing, display the measure as a
histogram.
Characteristic Average Value
Average System I/O per second
Average System Throughput
Average System I/O Size (divide Throughput by I/O per second)
Average System Hit Percentage
Average System Write Percentage
Average System Write Pendings
Average Back-End I/O per second (sum I/O per second over all disk directors)
26 EMC Education Service
Profiles of Usage
Examine the archive to detect usage changes over time. Consider the measures you looked at in the previous
part, especially the I/O per second measure. Are there noticeable changes in the characteristics at certain
times? Do your best to identify the time periods that differentiate the workload, and record them here. Also
note what evidence led you to conclude that a change in the traffic had happened here: increased I/O per
second, change in write percentage, etc.
Period Start - End Evidence
1
2
3
4
Component Usage
Examine the archive to determine what components are being utilized. Answer each of the following
questions by examining the I/O per second measure for the indicated components
Step Action
1 List any front end ports that are mostly unused:
2 Are any of the front end ports “grouped?” Do they seem to be sharing the
workload for one I/O stream? If so, list the ports:
3 List any back end directors that are mostly unused:
27 EMC Education Service
Step Action
4 List any physical disks that are mostly unused (if there are many, just write the
count out of the total number: 10/30):
5 Are any of the disks “grouped?” Do they seem to be sharing the workload for
one I/O stream? If so, list the disks (if there are many, just write the count out of
the total number: 10/30):
6 Are any of the devices “grouped?” Do they seem to be sharing the workload for
one I/O stream? If so, list the devices (if there are many, just write the count out
of the total number: 10/30):
End of Lab Exercise 2
28 EMC Education Service
Lab Exercise 3: Symmetrix Front-end
Purpose:
To use guided and free-format techniques to detect front-end
issues.
Tasks: In this lab, you will perform the following tasks:
Relate I/O size to performance
Use guided steps to analyze front-end issues
Analyze front-end issues using techniques you learned in the
course
EMC Education Service
29 EMC Education Service
Lab 3: Part 1 – Effect of I/O size
Performance Archive Used In This Part
IOSize: Symmetrix/000184501731/analyst/iosize02.btp
This Part uses the IOSize archive. This archive is not real-world data, nor is it intended to simulate any real-
world environment. It is simply a test of the effects of I/O characteristics on Fibre Channel and SCSI ports.
The activity was generated by single-threaded I/O generation programs (one for each device) that have only
one task: trigger a new I/O request immediately after the completion of the previous one. Four distinct
traffic cycles were issued, each signified by a change in one of the I/O characteristics.
Examine the archive and fill in the following table regarding the Symmetrix-wide activity.
Host Port:______________________________
Metric Cycle 1 Cycle 2 Cycle 3 Cycle 4
I/O per second
Throughput
Host Port:______________________________
Metric Cycle 1 Cycle 2 Cycle 3 Cycle 4
I/O per second
Throughput
Since these applications are simply waiting to send a new I/O request after the completion of the previous
one, any increase in traffic rate or volume indicates a performance improvement; any decrease, a reduction
in performance.
30 EMC Education Service
Questions How do you explain the wide difference in I/O rate across the four cycles? When was the actual data throughput at its highest: when the I/O rate was at its lowest or at its highest? The applications used to generate the activity on both ports are the same. Suggest a reason why the performance on both ports is not identical.
End of Lab 3 – Part 1
31 EMC Education Service
Lab 3: Part 2 – Guided analysis
Performance Archive Used In This Part
Email SRDF: Symmetrix/000284701058/daily/20040420d.btp
This archive was recorded at a site that uses Solaris hosts to support an Email application. The application is
replicated to disaster recovery site using four SRDF RA1 ports. Another four RA2 ports receive data from the
disaster recovery site.
Step Action
1 Look at the “Dir-Fibre > All Fibre Directors IO Rates” View (in the Views tab).
What director[s] peak at the highest I/O activity?
______________________________
What is the peak I/O rate, and when does it occur?
_____________________________
Do the hosts appear to be attached by 2 or more balanced ports? If so, list the port
pairings: ______________________________________________________________
32 EMC Education Service
Step Action
2 Look at the “Dir-Fibre>Fibre Directors Utilization” View (or a similar View).
What is the peak utilization of all Fibre directors? _________________________
When does this peak occur? ____________________________
Since director utilization is calculated from the I/O per second measure, the director
causing the highest utilization must be the one with the peak I/O rate that you
recorded earlier. From this information, determine how many I/O per second the
director would have to be processing to reach 100% utilization:
___________________________
3 Look at the “Dir-Port>Host Ports % Utilization” View.
What port[s] peak at the highest port utilization?
_______________________________
What is the peak utilization, and when does it occur?
___________________________
33 EMC Education Service
Step Action
4 Turn to the Metrics tab and plot the throughput for all of the Fibre ports.
What is the peak throughput of all ports, and when does it occur?
_________________
How much throughput would the port have to be processing to reach 100%
utilization?
_____________________________________________________________________
How can the port with the peak utilization not be attached to the director with the
peak utilization?
____________________________________________________________
Plot the measure that supports your answer. What did you plot?
______________________________________________________________________
End of Lab 3 – Part 2
34 EMC Education Service
Lab 3: Part 3 – Auto Design Front-end Performance
Performance Archive Used In This Part
Automobile Design: Symmetrix/000284500972/revolving/200204181502.btp
This archive was taken from an array using 72GB RAID-1 drives. To help visualize the effects of adding a new
database application, a load operation (write, read back and verify) was performed while this archive was
created to capture the performance. Your analysis will explain the effects to the IT group and help them plan
for future applications to be added to the array
Analyze the front-end performance of the “Automobile Design” archive. Thoroughly examine any issue (good
or bad performance) related to the topics discussed in this module. Put off detailed analysis of the other
components until later in the class—you will revisit this archive again once you finish the other modules. Be
prepared to back up any claims with appropriate evidence.
You might want to start with a complete characterization as outlined during the Characterization exercise you
did previously. This will give you some idea of the basic environment of this archive, including the general
workload characteristics and component usage.
Following this, observe the Roadmap measures outlined in the module to detect issues and problems.
Finally, look for any minor issues discussed in the module (off-roadmap topics).
Here are some questions you should be able to answer once you have done your analysis:
35 EMC Education Service
Step Action
1 Is the utilization of any of the front end components overly high?
2 Is the balance across the front end components good?
3 Is there room for addition of the new applications?
4 What is the best strategy for adding new applications with regard to the front
end hardware?
End of Lab 3 – Part 3
36 EMC Education Service
Lab 3: Part 4 –24/7 Front-end Performance
Performance Archive Used In This Part
24/7: Symmetrix/000190100720/interval/20080312.btp
This archive shows full-sized DMX array that is used for a variety of applications around the clock. In this
growing environment, the IT staff would be very interested in any recommendations for adding new
applications to the array.
Analyze the front-end performance of the archive. Thoroughly examine any issue (good or bad performance)
related to the topics discussed in this module. Put off detailed analysis of the other components until later in
the class—you will revisit this archive again once you finish the other modules. Be prepared to back up any
claims with appropriate evidence.
You should be able to answer these questions once you have finished your analysis:
37 EMC Education Service
Step Action
1 Is the utilization of any of the front-end components overly high?
2 Is the balance across the front end components good?
3 Is there room for addition of the new applications?
4 What is the best strategy for adding new applications with regard to the front
end hardware?
5 If there are any issues, what recommendations do you have for resolving them?
End of Lab 3 – Part 4
End of Lab Exercise 3
38 EMC Education Service
Lab Exercise 4: Symmetrix Cache
Purpose:
To use guided and free-format techniques to detect cache
issues.
Tasks: In this lab, you will perform the following tasks:
Relate random I/O to performance
Use guided steps to analyze cache issues
Detect performance problem caused by shared resources
Analyze cache issues using techniques you learned in the
course
39 EMC Education Service
Lab 4: Part 1 – Effects of Randomness
Performance Archive Used In This Part
Random: Symmetrix/000184501731/analyst/random01.btp
This Part uses the “Random” archive. This archive is not real-world data, nor is it intended to simulate any
real-world environment. It is simply a test of the effects of I/O characteristics on Fibre Channel and SCSI
ports. The activity was generated by single-threaded I/O generation programs (one for each device) that
have only one task: trigger a new I/O request immediately after the completion of the previous one. Three
distinct traffic cycles were issued, each signified by a change in one of the I/O characteristics.
Examine the archive and fill in the following table regarding the Symmetrix-wide activity.
Metric Cycle 1 Cycle 2 Cycle 3
I/O per second
Throughput
Since these applications are simply waiting to send a new I/O request after the completion of the previous
one, any increase in traffic rate or volume indicates a performance improvement; any decrease, a reduction
in performance.
40 EMC Education Service
Step Action
1 How do you explain the differences in I/O rate across the three cycles? What
metrics support your explanation?
2 In this example, approximately how much of an impact did this characteristic
have on the overall I/O rate (give as a percentage increase between the first and
last cycle)?
3 Is the effect more predominant for reads or writes? Explain why.
End of Lab 4 – Part 1
41 EMC Education Service
Lab 4: Part 2 – Guided Analysis
Performance Archive Used In This Part
Email SRDF: Symmetrix/000284701058/daily/20040420d.btp
This archive was recorded at a site that uses Solaris hosts to support an Email application. The application is
replicated to disaster recovery site using four SRDF RA1 ports. Another four RA2 ports receive data from the
disaster recovery site.
Step Action
1 Look at the “System>System Cache Utilization” View.
Estimate the average Hit % and Write % from this View:
_______________________
From this view alone, answer this question: Is the write hit percentage equal to
100%?
2 Look at the “System>System Total IO Rate” View.
What is the peak total I/O rate, and when does it occur?
________________________
Which is predominant I/O type for this array: reads, sequential reads, or writes?
42 EMC Education Service
Step Action
3 Use the Metrics tab to plot the slot collisions. Generate a graph that shows the
total slot collisions across the whole array.
Describe how you generated this graph:
Roughly what percentage of the overall I/Os cause a slot collision?
_______________
4 Plot the “Device>System bus Kbytes per sec” for all devices, and view the result
in an Area graph.
What is the peak internal system throughput, and when does it occur?
_____________
5 Use the Metrics tab to plot the “System>system max wp limit” and
“System>number write pending tracks” on the same graph.
Is there any evidence of reaching system write pending limits? __________
What is the system write pending limit for this array? _________________
43 EMC Education Service
Step Action
6 Plot the “Device>write pending count” for all devices on the array. Since this is a
pre-DMX-2 array, use the technique discussed in class to determine the base
device write pending limit.
Is there any evidence of reaching device write pending limits? _________
If so, which devices seem to be hitting the limits the most?
______________________
What is the base device write pending limit for this array?
_______________________
7 Plot the “Dir-Fibre>ios per sec” and “Dir-Fibre>requests per sec” for each of the
Fibre directors individually—if you plot several directors in the same graph, it
may be too difficult to match the measures up.
What is the approximate ratio of requests / I/Os for the busiest (highest I/O) pair
of directors? _______________________________
Is large I/O size a factor in increasing the number of requests in this case?
__________
What graph supports your opinion?
_________________________________________
What pair of directors shows the highest (peak) requests to I/O ratio?
______________
Is this high ratio sustained or temporary?
_____________________________________
Is large I/O size a factor in the large number of requests for these directors?
_________
End of Lab 4 – Part 2
44 EMC Education Service
Lab 4: Part 3 – One host reporting problems
Performance Archive Used In This Part
One Host: Symmetrix/000182503445/revolving/200304081410.btp
This part uses the One Host archive. In this simulated environment, three hosts are connected to a single
Symmetrix. Each has a different application profile, and is maintained by a different application
administrator. At around 13:30 of this day, sun220 experienced a prolonged slowdown. Average response
time increased by 40% on this already heavily utilized server, causing a corresponding reduction in the
number of records processed. You will be able to identify this period by observing the drop in I/O rate for
sun220.
Analyze the archive and report your findings back to the administration team so that they are aware of the
cause of the slowdown and any potential solutions.
Basic Configuration Information
sun220 using ports: SA01A-0, SA16A-0
hpd370a using ports: FA05B-0, FA12B-0
f50a using ports: FA05A-0, FA12A-0
45 EMC Education Service
Step Action
1 Describe the event (change in traffic issued from hosts) that caused the
performance problem:
2 Why did this event cause a performance issue?
3 Why did none of the other hosts connected to the same array report
performance issues during this period?
4 What can be done to avoid the problem in the future?
End of Lab 4 – Part 3
46 EMC Education Service
Lab 4: Part 4 – Auto design Cache performance
Performance Archive Used In This Part
Automobile Design: Symmetrix/000284500972/revolving/200204181502.btp
This archive was taken from an array using 72GB RAID-1 drives. To help visualize the effects of adding a new
database application, a load operation (write, read back and verify) was performed while this archive was
created to capture the performance. Your analysis will explain the effects to the IT group and help them plan
for future applications to be added to the array
Analyze the cache performance of the “Automobile Design” archive. Thoroughly examine any issue (good or
bad performance) related to the topics discussed in this module. Put off detailed analysis of the other
components until later in the class—you will revisit this archive again once you finish the other modules. Be
prepared to back up any claims with appropriate evidence.
You began analyzing the front-end performance of this archive in the previous Lab Exercise. Revisit your
findings. Then refer to the Roadmap measures to begin your analysis and consider the issues discussed in the
module.
Here are some questions you should be able to answer once you have done your analysis:
Step Action
1 Is the hit percentage unusually low for an open systems environment?
47 EMC Education Service
Step Action
2 Are there any System or Device write pending issues?
3 Is there any evidence of data misalignment?
4 If there are any issues, what recommendations do you have for resolving them?
End of Lab 4 – Part 4
48 EMC Education Service
Lab 4: Part 5 – 24/7 Cache performance
Performance Archive Used In This Part
24/7: Symmetrix/000190100720/interval/20080312.btp
This archive shows full-sized DMX array that is used for a variety of applications around the clock. In this
growing environment, the IT staff would be very interested in any recommendations for adding new
applications to the array.
Analyze the system and cache performance of this archive. Thoroughly examine any issue (good or bad
performance) related to the topics discussed in this module. Put off detailed analysis of the other
components until later in the class—you will revisit this archive again once you finish the other modules. Be
prepared to back up any claims with appropriate evidence.
You should be able to answer these questions once you have finished your analysis:
49 EMC Education Service
Step Action
1 Is the hit percentage unusually low for an open systems environment?
2 Are there any System or Device write pending issues?
3 Is there any evidence of data misalignment?
4 If there are any issues, what recommendations do you have for resolving them?
End of Lab 4 – Part 5
50 EMC Education Service
Lab Exercise 5: Symmetrix Back-end
Purpose:
To use guided and free-format techniques to detect back-end
issues.
Tasks: In this lab, you will perform the following tasks:
Analyze an archive and look for issues related to the
Symmetrix back end components
Plan for the deployment of a new database on an existing
Symmetrix
Examine the effectiveness of prefetching
51 EMC Education Service
Lab 5: Part 1 – Guided analysis
Performance Archive Used In This Part
Email SRDF: Symmetrix/000284701058/daily/20040420d.btp
This archive was recorded at a site that uses Solaris hosts to support an Email application. The application is
replicated to disaster recovery site using four SRDF RA1 ports. Another four RA2 ports receive data from the
disaster recovery site.
Step Action
1 Look at the “Dir-DA>Disk Directors Utilization” View.
Would you say that any of the directors is heavily utilized? _________
The “average” line is very close to the “max” line in this graph. What does that
mean about the variance between the directors—how different are their
individual utilizations?
___________________________________________________________
2 Look at the “Dir-DA>All Disk Directors – IOs per Sec” View.
Do the DAs look balanced? ______________
Does the traffic on the DAs generally correlate with the front end traffic? Are
the peak and valley times roughly the same? ______________
52 EMC Education Service
Step Action
3 Look at the “Disks>Disks Utilization” View.
What is the maximum utilization of any disk, and when does it occur?
______________
4 Use the Metrics tab to show a histogram of the “Devices>total ios per sec” for all
active devices.
Try to use this display to detect groupings of devices—devices involved in the
same workload. Verify your guesses by displaying the “Devices>total ios per
sec” in a normal time graph. If the devices are all part of the same workload,
they will often have similar workload patterns.
5 Use the Metrics tab to show the “Disks>total SCSI commands per sec”
Are the disks well balanced? _________
53 EMC Education Service
Step Action
6 Pick the most heavily utilized disk, and plot the 6 measures which are summed
to generate the “total SCSI commands per sec” measure all on the same graph.
Do the same with the least heavily utilized disk.
What are the top two measures that make up most of the workload for the
heavily utilized disk?
___________________________________________________________
What is the top measure that makes up the workload for the lightly utilized disk?
_________________________________________________________________
Is XOR activity a large factor in the performance of any of the array’s disks?
________
7 Plot the “Dir-DA>prefetched tracks per sec,” “Dir-DA>tracks not used per sec”
and “Dir-DA>tracks used per sec” on the same graph for one of the disk
directors.
Is the prefetching activity constant or temporary? ______________________
When prefetching does trigger, is it generally effective? _________________
How can the “tracks used per sec” be higher than the “prefetched tracks per
sec?”
______________________________________________________________
End of Lab 5 – Part 1
54 EMC Education Service
Lab 5: Part 2 – Auto design back-end performance
Performance Archive Used In This Part
Automobile Design: Symmetrix/000284500972/revolving/200204181502.btp
This archive was taken from an array using 72GB RAID-1 drives. To help visualize the effects of adding a new
database application, a load operation (write, read back and verify) was performed while this archive was
created to capture the performance. Your analysis will explain the effects to the IT group and help them plan
for future applications to be added to the array
Analyze the back-end performance of the “Automobile Design” archive. You have already examined the front-
end and cache performance of this archive, now examine any issue (good or bad performance) related to the
topics discussed in this module. Refer to the back-end road map for guidance. Be prepared to back up any
claims with appropriate evidence.
Here are some questions you should be able to answer once you have done your analysis:
55 EMC Education Service
Step Action
1 Are any back end directors and disks being over utilized?
2 Is the traffic on the back end directors and disks balanced?
3 Are there significant back end resources for additional traffic?
4 If there are any issues, what recommendations do you have for resolving them?
End of Lab 5 – Part 2
56 EMC Education Service
Lab 5: Part 3 – Prefetch analysis
Performance Archive Used In This Part
SAP/Oracle: Symmetrix/000187401250/interval/20040707.btp
This exercise uses an earlier snapshot of the “SAP/ORACLE” archive to investigate some basic prefetch
metrics. The effectiveness of prefetching in the late night backup job is of the most concern.
Step Action
1 Compare the system-wide “total ios per sec” measure with the system-wide
“prefetched tracks per sec” measure.
Are any changes in the measures evident during the late night backup period?
What do these changes signify?
What is the approximate hit rate during this time?
57 EMC Education Service
Step Action
2 Compare the system-wide Prefetched tracks per second measure with the
system-wide Tracks not used measure. Since the second of these is not available
at the system level, you will have to plot it at the Dir-DA level. You will find it
easiest to create a custom view that sums the measures for all disk directors to
produce just one “Tracks not used” line.
Does the difference between prefetched tracks and tracks not used indicate an
improvement in the effectiveness of prefetching during the backup job?
What is the approximate percentage of tracks not used to prefetched tracks
during the backup job?
End of Lab 5 – Part 3
58 EMC Education Service
Lab 5: Part 4 – 24/7 Back-end performance
Performance Archive Used In This Part
24/7: Symmetrix/000190100720/interval/20080312.btp
Analyze the overall back end performance of this archive. Thoroughly examine any issue (good or bad
performance) related to the topics discussed in this module. Be prepared to back up any claims with
appropriate evidence.
You might have thoroughly examined the back end performance of this archive during previous exercises in
this Lab. If so, just make sure you have not missed anything.
Here are some questions you should be able to answer once you have done your analysis:
Step Action
1 Are any back end directors and disks being over utilized?
2 Is the traffic on the back end directors and disks balanced?
59 EMC Education Service
Step Action
3 Are back-end resources capable of handling additional applications?
4 Do you have any recommendations of how best to add new applications?
5 If there are any issues, what recommendations do you have for resolving them?
End of Lab 5 – Part 4
End of Lab Exercise 5
60 EMC Education Service
Lab Exercise 6: Business Continuity
Purpose:
To use guided and free-format techniques to detect business
continuity issues.
Tasks: In this lab, you will perform the following tasks:
Examine SRDF/S communications
Investigate Business Continuance issues and detect
effects on overall Symmetrix performance
61 EMC Education Service
Lab 6: Part 1 – Guided Analysis
Performance Archive Used In This Part
Email SRDF: Symmetrix/000284701058/daily/20040420d.btp
This exercise covers the “Email SRDF” archive. This lab will simply ask a lot of questions about the archive;
use Performance Manager to get the answers.
This 8730 archive was recorded at a site that uses Solaris hosts to support an Email application. The
application is replicated to disaster recovery site using four SRDF RA1 ports. Another four RA2 ports receive
data from the disaster recovery site. They are using SRDF/S to replicate all data.
Step Action
1 View the I/O traffic on the four RA1 ports in one graph, and the I/O traffic on the four
RA2 ports in another graph.
Do the ports within an RA group appear balanced? ____________
Do the two groups appear to be carrying different traffic loads? ____________
62 EMC Education Service
Step Action
2 Compare the I/O traffic on the front-end ports to the traffic on the two RA groups.
Which front-end ports are responsible for the outgoing SRDF traffic?
_________________
What is the average I/O size on these ports? _____________________________
What is the approximate write percentage of those ports?
___________________________
Does the sum of the I/Os on the two front-end ports appear to be equal to the sum of
the I/Os on the RA group? ___________
What would you expect to see on arrays using SRDF/A mode?
________________________________________________________________________
3 Compare the I/O traffic of a few of the most-utilized devices to the traffic on the front-
end ports you considered in the previous step.
Can you quickly identify devices that are carrying the outgoing SRDF load?
End of Lab 6 – Part 1
63 EMC Education Service
Lab 6: Part 2 – Unexpected Night Delays
Performance Archive Used In This Part
Symmetrix/000284500356/daily/20021214d.btp
Symmetrix/000284500356/daily/20021215d.btp
In this Part, you will be investigating application slowdowns that are occurring late in the night. Activity from
around 18:00 to 03:00 is noticeably slower than the rest of the day.
Investigate the archive and answer the questions.
Step Action
1 What event is most likely causing the change in response time?
2 What effects are these events likely to be having on the Symmetrix? Support your
answer with evidence whenever possible.
3 What software solution[s] might help reduce the negative performance effects?
End of Lab 6 – Part 2
End of Lab Exercise 6