enterprise software performance engineering · pdf fileenterprise software performance...

© 2012 Collaborative Consulting

Enterprise Software Performance Engineering

Presented By: Walter Kuketz

Scaling your career..

2 2012 Collaborative Consulting

Software Performance matters - everywhere

DTG

Facebook IPO


Who is responsible for end-to-end system performance?

• When a website slows down?

• When the marketing team runs a big event?

• When the Storage subsystem slows down, which

slows down the database, which slows down the

application servers, and slows down the iPad?

Where do the performance goals

come from?


Agenda – Building systems that scream…

Review the current challenges of being a performance engineer

Overview of the five knowledge areas of Software performance engineering body of knowledge

Roles and career path overview

Case study – 10X event (Communicate to Sr. Mgt.)


No clear standards in the market for PE

Performance Engineer Performance tester

Capacity planner

Performance Architect

Test automation technician

Performance analyst

Performance Data engineer

Troubleshooting Benchmarks

?

?

?

?

?

?

? Software Engineer

Performance test


Opportunity to take SPE to the next level..

Define the profession of performance

engineering Define profession

Establish a body of knowledge for

performance engineering

Establish Body of

Knowledge

Establish a common language for roles,

responsibilities, and practice areas using

the form of Knowledge Areas

Establish

common

language

Define the competencies shared across

the knowledge areas

Define

competencies

Capture common techniques for sharing

across the industry, or your company

Capture common

techniques


The SPE Body of Knowledge K

now

ledge a

reas

Planning, coordination, information sharing, and control

SDLC &

architecture

(SA)

Application

performance

management

(APM)

Problem

detection

and

resolution

(PDR)

Underlying competencies and roles for delivering

Techniques supporting the knowledge areas

Performance

validation and testing

(PVT)

Capacity planning

(CP)

For the Enterprise


The Enterprise PE team: There must be a practice leader

Operations & PT

Architecture and

production

Business direction

Development teams

Communication


The Enterprise PE team: There must be a practice leader

A shared service and clearing house for all performance, scalability and

capacity planning information. Transforms performance information to

business knowledge and advantage.

Communicates across the teams to share information

For instance, the production workload and the

performance test workload – are they in sync?

Make sure the architecture team knows how the

new release, or newest build behave in production.

The business risks are clearly identified and

communicated. Application risk profile.

Can provide performance guidelines to the

developers, must influence the unit testing process.

Operations & PT

Architecture and

production

Business direction

Development teams


PE: Knowledge areas

PVT

Performance testing and

validation

CP

Capacity planning

APM

Application performance

management

PDR

Problem detection and

resolution

SA

Software development

lifecycle and architecture


PE: Software development lifecycle and architecture

PVT


validation

CP

Capacity planning

APM


management

PDR


resolution

SA




PE: SA (software development lifecycle)

Define Design Develop Functional

test

Performance

test Deploy

Waterfall methodologies

Need to plan for

performance testing

Build code to performance

requirements

Define

Design

Build Test

Deploy Agile

methods

Can a system be designed to support 10 TPS and 1 second response time work @ 100 TPS and 1 second?


Performance budgets

PROXY

Datacenter

WEB

MQ/ESB

DCOM

CORBA

SERVER

APP SVR

LDAP

Web

Services

Code runs

everywhere.

Respond to the user

in 2 seconds.

Do your developers know how much time they have for their piece of the transaction?


SA



PE: Performance testing and validation

CP

Capacity planning

APM


management

PDR


resolution

PVT


validation


PE: PTV (performance testing and validation)

What is changing? Where does it need to be validated? What (all) is being measured?

The project: Why are we testing and what are the success criteria?

Prerequisites:

Business goals and non-functional requirements are defined

Workload and scenarios defined

Environment defined (where are we testing)

Application in a stable state

Configuration management and release management ready

Core performance team identified

Supporting team committed


• Deep understanding of

the application being

tested

• Deep understanding of

the technical

architecture of the

application

• Understanding of the

business area

supported by the

applications

Business and

application area

knowledge

• Performance engineers

involved in design

sessions

• Understand the nature

of the data required for

testing

• Able to understand

technical gaps to meet

requirements not

supported by the tool

Early involvement

with the applications

team

• Well-understood

workload model and

characterization

• All components in the

test are monitored and

reported on

• A defined triage process

for troubleshooting

• A process to manage

the inevitable

unplanned tests and

test archive exists

Test execution

• Testing environment is

a close approximation

of production, including

database size

• Differences between

test and production are

clearly known

• Release and

configuration

management in place

and procedures

enforced

Environment

Organization: Performance team value

You still need to explain to the rest of the organization what you do

Performance team value


SA



PE: Capacity planning

PVT


validation

APM


management

PDR


resolution

CP

Capacity planning


PE: CP (capacity planning)

The right system at the right price!

Workloads drive system transactions, which drive resource utilization

Determine

capacity

requirements

Analyze

current

capacity

Plan for

future • Forecast workloads

• Plan for usage

• Track utilization of

existing systems

• Track workload

• Define system to

support workloads

• Test

• Define service

levels


Capacity: Use of resources

PROXY

Datacenter

WEB

MQ/ESB

DCOM

CORBA

SERVER

APP SVR

LDAP

Web

Services

The overall system has a capacity; each component has a capacity.


SA



PE: Application performance management

PVT


validation

CP

Capacity planning

PDR


resolution

APM


management


Five elements of APM (Gartner)

Monitoring the performance of complex distributed applications

End user experience measurement – browser/mobile

device

Create a model of the run-time environment (discovery)

Profile the performance and behavior of user-defined

transactions

Performance metrics from each of the

applications/systems technical components (Webserver,

App server, Database, etc.)

Application performance management database

1

2

3

4

5


Mobile

Track key user experience metrics

Network latency

Browser rendering time

Total end-user response time

Renderin

g

Mobile Real User Monitoring


And see every hop of the transaction

Mobile

App server

Web server

Proxy

server

Database

Mainframe

Data warehouse gateway

Dynamic Transaction

Path Detection

Renderin

g


SA



PE: Problem detection and resolution

PVT


validation

CP

Capacity planning

APM


management

PDR


resolution


When problems occur

Flash web site events (all at once)

New code released causes problems

Infrastructure upgrade

Consolidation of applications and servers

New workload model (batch and online)


Cause and Effect


SPE Competencies

Career path


PE roles: The team – many possible roles

Managing and developing the core team members

Technical Project manager Performance Architect

Sr. Performance Engineer

Performance Engineer

Sr. Test automation technician

Test automation technician

Performance data engineer

Sr. Capacity planner

Capacity planner Workload and capacity

Modeler


Underlying competencies

Analytical

thinking

problem solver

Application and

tool knowledge

Workload

model

development

Statistics

Test data

management

Project

management

Communication

skills

Behavioral

characteristics

Test execution

process Forecasting

Business

Knowledge

Monitoring and

tuning

Underlying

competencies


Technical competencies

Technical

competencies

Code

profiling

tools (Static analysis,

diagnostics)

Database

monitoring

tools

End to end

performance

tools

Deep dive

tools (HP Diagnostics,

CA Wily, etc,

Jprobe, etc)

Test design

and

execution

tools

Comprehensive

results collection

and reporting

Resource

utilization

system

level

Queuing

and

modeling

tools

Statistics and

Distributions

Test data

creation

DB tuning

SQL tuning


PE Techniques

Brainstorming Data modeling

Creating a workload model Performance testing

Triage approach Static code analysis

System metrics collection Logging and

instrumentation


Career path: Performance engineer foundation

Grow your breadth of capabilities

Grow your

leadership and

communication

Performance

Engineer

Performance

testing

Lead

technical

role

Workload

models

Communication

Production

triage

Technical

architecture

Additional

tech

competency

Core

technical

competency


Upcoming Marketing event


How do these Knowledge areas work together (10X)

Forecast the

workload for

the event

Measure the current

workload and

resource utilization

Plan and execute

PT for forecast

workload with

support for

troubleshooting

SA PVT

APM PDR

CP CP

Architecture and

Tune application

based on results

Responsibilities


Questions?

Thank you!

[email protected]


• Development methodologies

Agile

Iterative

RUP

Relational Data models

Object oriented

• Implementation concepts

Swing

Hibernate

JMS

Web Services

Windows

RESTful services

SOAP

• Integrated Development Environments

Eclipse, Windows, SQL Developer


• Operating systems

Windows Platform

Unix and all its varient

iOS

Z/OS

• Database platforms

Oracle

Microsoft

DB/2

• Application servers

Microsoft

Jboss

Oracle AS

Weblogic

Websphere

• Application frameworks

.NET

J2EE

Spring

Struts

• Programming languages

Java

C#

XML

C/C++

COBOL

Ruby/

• Reporting technologies

• Integrations technologies

• Software managements technologies


CP: Workload model

Step Transaction – by product Percentage

1 Login 100

2 Search product 100

3 View product detail 100

4 Add to shopping cart 50

5 Confirm purchase method 50

6 Shipping information 50

7 Send confirmation email 50

CP

The CP team uses this information to forecast the

new workload 10X.


Review current Production workload, and the under lying system

utilization.

• Average workload

• Current peak workload

CP: Analyze current capacity

System utilization 100%

Response

time

Service level

CP


• Use the workload model from CP

• Enable APM during testing

• Work with the Application Architect

PVT: planning, executing, and analysis

Methodology is iterative and tactical

Review infrastructure

& architecture

Define business

activity profiles &

service levels

• Identify risk areas

• Review configuration

settings, topology, & sizing

• Define points of

measurement

• Types & numbers of

users

• Business activities &

frequencies

Iterate testing & tuning

• Test data generation

• Create test scripts

• User & transaction profiles

• Infrastructure configuration

Design & build tests

PVT

enterprise software performance engineering · pdf fileenterprise software performance...

Documents