#ibmiod

Proactive Performance Tuning: A Day in the Life of a DBA at ADP, Inc Session Number IDZ-1620

Rachel Niedzwiecki, ADP

Saghi Amirsoleymani, IBM

#ibmiod

Please note

IBM’s statements regarding its plans, directions, and intent are subject to change or withdrawal without notice at IBM’s sole discretion.

Information regarding potential future products is intended to outline our general product direction and it should not be relied on in making a purchasing decision.

The information mentioned regarding potential future products is not a commitment, promise, or legal obligation to deliver any material, code or functionality. Information about potential future products may not be incorporated into any contract. The development, release, and timing of any future features or functionality described for our products remains at our sole discretion.

Performance is based on measurements and projections using standard IBM

benchmarks in a controlled environment. The actual throughput or performance

that any user will experience will vary depending upon many factors, including

considerations such as the amount of multiprogramming in the user’s job stream,

the I/O configuration, the storage configuration, and the workload processed.

Therefore, no assurance can be given that an individual user will achieve results

similar to those stated here.

#ibmiod

Acknowledgements and Disclaimers:

© Copyright IBM Corporation 2012. All rights reserved.

– U.S. Government Users Restricted Rights - Use, duplication or disclosure restricted by GSA ADP Schedule Contract

with IBM Corp.

– Please update paragraph below for the particular product or family brand trademarks you mention such as WebSphere, DB2,

Maximo, Clearcase, Lotus, etc

IBM, the IBM logo, ibm.com, [IBM Brand, if trademarked], and [IBM Product, if trademarked] are trademarks or registered trademarks of

International Business Machines Corporation in the United States, other countries, or both. If these and other IBM trademarked terms are

marked on their first occurrence in this information with a trademark symbol (® or ™), these symbols indicate U.S. registered or common

law trademarks owned by IBM at the time this information was published. Such trademarks may also be registered or common law

trademarks in other countries. A current list of IBM trademarks is available on the Web at “Copyright and trademark information” at

www.ibm.com/legal/copytrade.shtml

f you have mentioned trademarks that are not from IBM, please update and add the following lines:

[Insert any special 3rd party trademark names/attributions here]

Other company, product, or service names may be trademarks or service marks of others.

Availability. References in this presentation to IBM products, programs, or services do not imply that they will be available in all

countries in which IBM operates.

The workshops, sessions and materials have been prepared by IBM or the session speakers and reflect their own views. They are

provided for informational purposes only, and are neither intended to, nor shall have the effect of being, legal or other guidance or advice

to any participant. While efforts were made to verify the completeness and accuracy of the information contained in this presentation, it is

provided AS-IS without warranty of any kind, express or implied. IBM shall not be responsible for any damages arising out of the use of,

or otherwise related to, this presentation or any other materials. Nothing contained in this presentation is intended to, nor shall have the

effect of, creating any warranties or representations from IBM or its suppliers or licensors, or altering the terms and conditions of the

applicable license agreement governing the use of IBM software.

All customer examples described are presented as illustrations of how those customers have used IBM products and the results they may

have achieved. Actual environmental costs and performance characteristics may vary by customer. Nothing contained in these

materials is intended to, nor shall have the effect of, stating or implying that any activities undertaken by you will result in any specific

sales, revenue growth or other results.

#ibmiod

Agenda

• Automatic Data Processing, Inc. - Who are we?

• DB2 Query Monitor (QM) – Overview

• Optim Query Workload Tuner (OQWT)- Overview

• How ADP uses these tools to help with both reactive and proactive performance resolutions.

• ADP Performance Success Stories using these tools.

• QUESTIONS?

3

#ibmiod

Automatic Data Processing, Inc.

Automatic Data Processing, Inc. (NASDAQ: ADP), • About $10 billion in revenues

• Approximately 570,000 clients

• Is one of the world's largest providers of business outsourcing solutions.

• Leveraging over 60 years of experience, ADP offers a wide range of human resource, payroll, tax and benefits administration solutions from a single source.

• ADP's easy-to-use solutions for employers provide superior value to companies of all types and sizes.

• ADP is also a leading provider of integrated computing solutions to auto, truck, motorcycle, marine, recreational vehicle, and heavy equipment dealers throughout the world.

http://www.adp.com

4

#ibmiod

DB2 Query Monitor v3.1

• SQL Monitoring Tool

o Not Thread or Subsystem monitor

• Low overhead SQL statement monitor

• Identify problem SQL activity

• Determine the tables and indexes used by SQL statement

#ibmiod

InfoSphere Optim Query Workload Tuner Requirements

Benefits

Get expert, actionable advice to

improve SQL workload

performance

Improved performance

Improved productivity

Reduced costs

Reduced risks

Optimize

Performance

Reduce skill

requirements and

increase productivity for

tuning

Analyze entire workloads

rather than query at a

time

Provide expert query

tuning advice that

analyzes tradeoffs

#ibmiod

Streamlined Analysis with expert advisors

• Statistics

o Get recommendations on the best statistics to capture to influence access path selection

• Access Path

o Get recommendations on

how to influence the

access path selection

• Query

o Get recommendations regarding how to rewrite the query for better efficiency

• Index

o Get recommendations on indexes changes that can reduce database scans

#ibmiod

How ADP Uses Query Monitor & Query Workload Tuner

• Reactive Performance issues: Top 25 CPU Usage Report : identifies poor performing jobs with high CPU usage or support calls for jobs running longer than expected.

• Proactive performance issues: SQL Reviews

8

#ibmiod

Top 25 CPU Usage

• ADP runs this report daily to identify our top 25 jobs with the highest CPU usage. These jobs are reviewed and each one starting with the worst and working our way down are evaluated using QM and OQWT.

• With QM we are able to quickly drill down and identify the problem SQL(s) for the specific programs run in the job.

• Using OQWT we are able to run the various advisors to help us quickly identify possible improvements.

9

#ibmiod

TOP 25 CPU Users Report

10

• We produce these reports daily using data that we store in DB2 in our PM History Database.

#ibmiod

Query Monitor: Identify the “culprit”

• We use Query Monitor to then look up the job that was identified in the TOP 25 report to identify the query(ies) and their respective program(s) that utilized the majority of the CPU.

11

#ibmiod

Query Monitor: Continued

12

#ibmiod

Query Monitor: List SQL statements for a given Job

13

#ibmiod

QM: Identify Poor SQL and its Program

14

#ibmiod

QM: Identifying the “problem” Query

15

#ibmiod

OQWT: Execute Advisors

16

#ibmiod

OQWT: Single Query: Continued

17

#ibmiod

OQWT: Single Query: Continued

18

#ibmiod

OQWT: Single Query Analysis results

19

Recommendations by severity

#ibmiod

OQWT: Stats Advisor Recommendation

20

At this point we can try running the Stats and redo the advisors

Generates

RUNSTATS control statements

Improve Statistics Quality and Collection

#ibmiod

OQWT: Access Path Advisor

21

Interrogate the plan table to discover access plan flags

#ibmiod

OQWT: Index Advisor

22

Estimated performance

improvement

Indexing Advice to Improve Query Efficiency

#ibmiod

OQWT: Tuning Report (shows all advisor recommendations)

23

You can save the

report to your PC

#ibmiod

TOP 25 CPU Report Success Stories (1)

• Identified a job that was first on the report for months. Sent summary report to developers and verified with them the expected performance improvement for implementing some of the advisor recommendations: Here are the results of the recommended index change

o DSN_STATEMNT_TABLE ESTIMATED COST CHANGE - OLD EST SVC UNITS 823,974

o DSN_STATEMNT_TABLE ESTIMATED COST CHANGE - OLD EST CPU 18,952

o DSN_STATEMNT_TABLE ESTIMATED COST CHANGE - NEW EST SVC UNITS 163,377

o DSN_STATEMNT_TABLE ESTIMATED COST CHANGE - NEW EST CPU 3,758

• 80% reduction in CPU cost

24

#ibmiod

TOP 25 CPU Report Success Stories (2)

• We have a job that runs all day Tuesday through Saturday and has peak processing every Wednesday and Thursday. Its accounting information shows that it spends almost all of the time in DB2 and all application CPU is same as DB2 CPU time.

• Through QM we quickly identified the most expensive SQL where we verified that it had a bad access path doing a tablespace scan for some of its query blocks.

• Then we generated OQWT advisor reports for this problem query. We implemented the recommendation of a new index on the table being scanned and performance of the query improved by as much as 70% : ( the query is a SELECT COUNT(*)…… DELETE FROM…….)

25

#ibmiod

What to do next …SQL Reviews

• At ADP we conduct “DART” (DB2 Application Review Template) reviews for new and modified SQL for our upcoming application releases.

• At these reviews Queries are reviewed along with their explain output. If an access path is not acceptable (TBL or IX scans, IX/MC 1 on a low cardinality column, etc.) then we provide an OQWT summary report with recommended improvements.

• In most cases the recommendation is a change to the query, in some cases it was a change or addition of an index.

26

#ibmiod

SQL Reviews: Example • This is taken from an OQWT Query Advisor report:

27

Highlights relevant components of the

query

Recommendation and rationale

#ibmiod

OQWT Summary report: Access Plan Summary

28

#ibmiod

SQL Review: Results

• After running several iterations of the Query with recommended changes, we put together the cost as shown in the OQWT summary reports to determine the best choice for the Query:

29

We verified that this is the best case by running two DSNTEP2 jobs, one in

the morning and one in the afternoon using the original query and the

modified query for option 2 and the results seen in QM for ET, CPU and

I/O were very close to the results shown in the summary reports for cost.

#ibmiod

Analyze DB2 10 Workload Success Story

30

Best practice for getting workload tuning advice

– Improve statistics quality

– Improve query design

– Improve database design (index & access path )

Speed up analysis, optimize design,

and balance resource usage

Workload : Multiple SQL statements defined by user

•The effort for tuning the whole application with good performance by evaluating every statement is overwhelming . Optimization decisions are based on tradeoffs

•Statistics – CPU costs vs. query savings •Indexing – query speed vs resource and transaction

•When application data grows, allows you to do proactive application health check periodically to find potential problems earlier before costly application outages

#ibmiod

Compare current access path with DB2 10

31

•Adding ‘OPTIMIZE FOR 1 ROW’ clause to the SQL statement for the DAT

Cursor

•creating a new index to match the predicates and here are the cost results:

#ibmiod

Improve Query Design

32

Recommendation and rationale

Guard against errors and

oversights:

Further constrain query,

increase index utilization, and

reduce data reads

#ibmiod

Using QM & OQWT: Big Savings

• One of our DBA’s who is primarily using these tools to research these poor performing jobs and recommend improvements stated the following with regard to how much time on average it takes to find these improvements (this is as recent as work done on 6/19/2012):

• “Using QM, PM and OQWT took one hour. Testing indices took 6 hours as the query was complex and took time to understand which query block is bad performer”.

• So in total, it takes on average less than a day for one DBA to identify improvements for poor performing queries. Identifying a solution has been reduced significantly for us.

33

#ibmiod

Summary

Meet SLAs Increase

Performance

Lower

Infrastructure

Costs

Minimize slowdowns which impact End User Productivity

Avoid Problems with proactive query tuning

Improved HW and DB Utilization & Cost Savings

Database Administrators do more with Less

Faster Time to Problem Resolution

KEY SOLUTION: "IBM InfoSphere Optim Query Workload Tuner for z/OS“

& "IBM DB2 Query Monitor for z/OS"

Lower

Operational

Costs

#ibmiod 35

#ibmiod

Resources

• Integrated Data Management Community

o http://www.ibm.com/developerworks/spaces/optim

• IBM Optim Query Workload Tuner web page

o https://www-01.ibm.com/software/data/db2imstools/db2tools/opti-expert-zos/

• IBM DB2 Query Monitor for z/OS

o http://www-01.ibm.com/software/data/db2imstools/db2tools/db2querymon.html

• SQL Tuning, Not just for hardcore DBAs anymore

o http://www.ibmdmmagazinedigital.com/dmmagazine/

• Redbooks at www.redbooks.ibm.com

o DB2 9 for z/OS Performance Topics SG24-7473

o IBM DB2 9 for z/OS: New Tools for Query Optimization SG24-7421

Contact

Saghi Amirsoleymani, Optim Enablement Specialist, amirsole@us.ibm.com

#ibmiod 37

Thank You for Joining Us today!

If you would take a moment to fill out the feedback form which will display on the next slide, it would be greatly appreciated. Your comments are very important to us.

Go to www.ibm.com/software/systemz/events/calendar to:

Replay this webcast

View previously broadcast webcasts

Register for upcoming webcasts

#ibmiod

Communities

• On-line communities, User Groups, Technical Forums, Blogs, Social networks, and more

o Find the community that interests you …

• Information Management bit.ly/InfoMgmtCommunity

• Business Analytics bit.ly/AnalyticsCommunity

• Enterprise Content Management bit.ly/ECMCommunity

• IBM Champions

o Recognizing individuals who have made the most outstanding contributions to Information Management, Business Analytics, and Enterprise Content Management communities

• ibm.com/champion

#ibmiod

Thank You!

Your Feedback is Important!

• Access SmartSite to complete your session surveys

o Any web or mobile browser at iodsmartsite.com

o Any SmartSite kiosk onsite

o Each completed session survey increases your chance to win an Apple TV with daily drawing sponsored by Alliance Tech