performance testing in the agile enterprise
DESCRIPTION
According to the presenter, performance testing is an inherently agile process, even though many teams and organizations experience significant difficulty integrating performance into their existing agile processes. In this session, learn how performance can be seamlessly integrated and how to employ and manage performance testing on Agile projects and within Agile enterprises. After this session, students will have the tools and knowledge they need to overcome many of the common difficulties organizations face when trying to make performance an integral part of their Agile processes.TRANSCRIPT
www.PerfTestPlus.com © 2006-11 PerfTestPlus, Inc. All rights reserved. Performance Testing in Agile Page 1
Performance Testing in the Agile Enterprise
Chief Technologist, PerfTestPlus, Inc. [email protected] www.perftestplus.com
Co-Founder: Workshop On Performance and Reliability
www.performance-workshop.org
by: Scott Barber
Co-Author: Author: Contributing Author:
Books: www.perftestplus.com/pubs About me: about.me/scott.barber
www.PerfTestPlus.com © 2006-11 PerfTestPlus, Inc. All rights reserved. Performance Testing in Agile Page 2
Have you ever wondered…
teams have no idea where performance “fits” in an Agile
development lifecycle?
www.PerfTestPlus.com © 2006-11 PerfTestPlus, Inc. All rights reserved. Performance Testing in Agile Page 3
Because performance
“fit” in an Agile development lifecycle…
www.PerfTestPlus.com © 2006-11 PerfTestPlus, Inc. All rights reserved. Performance Testing in Agile Page 4
Performance is (or should be)
in an Agile development lifecycle!
www.PerfTestPlus.com © 2006-11 PerfTestPlus, Inc. All rights reserved. Performance Testing in Agile Page 5
Overview
Performance Testing is inherently Agile, but...
Integrating Performance Testing
into Agile Efforts effectively is not inherently easy!
www.PerfTestPlus.com © 2006-11 PerfTestPlus, Inc. All rights reserved. Performance Testing in Agile Page 6
Agility in Performance Testing Performance Testing is a repeating cycle of:
Complicated by unknowns, estimations, &
approximations
www.PerfTestPlus.com © 2006-11 PerfTestPlus, Inc. All rights reserved. Performance Testing in Agile Page 7
Agility in Software Development Agile is a repeating cycle of:
Gated by inexact and
variable notions of acceptance
www.PerfTestPlus.com © 2006-11 PerfTestPlus, Inc. All rights reserved. Performance Testing in Agile Page 8
Agile Perf Testing in Agile Dev Putting them together is…
…a tad complicated
www.PerfTestPlus.com © 2006-11 PerfTestPlus, Inc. All rights reserved. Performance Testing in Agile Page 9
Conceptual Models “On Demand”
• A.K.A. Center of Excellence • Call ‘em when you (think you) need ‘em • “Pay” only for what you (think you) need
“On Retainer”
• Performance Tester on team, but mostly independent • Always available
“Full Immersion” • Performance is everyone’s job • Coordinated by embedded Performance Test Specialist • Spans entire lifecycle
www.PerfTestPlus.com © 2006-11 PerfTestPlus, Inc. All rights reserved. Performance Testing in Agile Page 10
On Demand Roles & Interactions:
• Similar to outsourcing except that… • you expect familiarity with the project/product (unlikely) • you expect familiarity with the process (less likely) • you won’t get 100% focus on your project • the assigned resource(s) also report to someone who can
trump the project manager • Minimal interaction with team • In general, they collect info, run tests, & report results.
Strengths: • Fewer performance testers can service more projects • Pooled tools and environments can save time and money • Performance testers benefit from peer collaboration
www.PerfTestPlus.com © 2006-11 PerfTestPlus, Inc. All rights reserved. Performance Testing in Agile Page 11
Performance Test Cycle; On Demand
www.PerfTestPlus.com © 2006-11 PerfTestPlus, Inc. All rights reserved. Performance Testing in Agile Page 12
On Demand Challenges:
• Service tends to be “generic” • Skills/tools/environments may not work for *your* project • Availability often restricted to highest priority projects • Performance testing tends to be both late and infrequent • Validation of tuning often has to wait until the next “on
demand cycle” • No continuity release to release • A core team member must serve as performance lead • Core team responsible for performance exploration
∆ from non-Agile efforts: • Difficult to schedule • System under test a faster moving target • Testing can’t keep up with continuous production delivery
www.PerfTestPlus.com © 2006-11 PerfTestPlus, Inc. All rights reserved. Performance Testing in Agile Page 13
On Retainer Roles & Interactions:
• Performance testing specialist • is embedded into the team full time • may be managed by Project Manager, Product Manager,
or Sr. Engineer • Performance is not (yet) integral to project team culture
Strengths: • Performance testing specialist
• knows project/product/culture/environment details • customizes service to those details • builds relationships that enhance “performance culture”
• Always available • Many more tests conducted than On Demand • Performance exploration and tuning validation “in scope”
www.PerfTestPlus.com © 2006-11 PerfTestPlus, Inc. All rights reserved. Performance Testing in Agile Page 14
Performance Test Cycle; On Retainer
www.PerfTestPlus.com © 2006-11 PerfTestPlus, Inc. All rights reserved. Performance Testing in Agile Page 15
On Retainer Challenges:
• Performance tester(s), tool(s), and environment(s) needed for every project
• Performance testers • Are often seen as “outsiders” • Tend to loose peer collaboration • Might not be fully utilized
• Performance assessments still tend to occur at end of sprints
• No additional support during “crunch time” ∆ from non-Agile efforts:
• Planning is necessarily more reactive • Tool and environment sharing virtually impossible • Testing done in smaller segments
www.PerfTestPlus.com © 2006-11 PerfTestPlus, Inc. All rights reserved. Performance Testing in Agile Page 16
Full Immersion Roles & Interactions:
• Performance specialist coordinates performance related activities including:
• Creation of performance stories • Code profiling and performance unit testing • Performance assessment • Collaboration with production support/capacity planning
• Performance is part of everyone’s job, every day Strengths:
• Performance is, or is becoming, an integral part of the team culture
• No delays, performance testing is done in line with dev • Cross-functional & cross-departmental collaboration • Best chance of no performance nightmares
www.PerfTestPlus.com © 2006-11 PerfTestPlus, Inc. All rights reserved. Performance Testing in Agile Page 17
Performance Test Cycle; Full Immersion
See role and task details in additional resource slides included at the end of this deck
www.PerfTestPlus.com © 2006-11 PerfTestPlus, Inc. All rights reserved. Performance Testing in Agile Page 18
Full Immersion Challenges:
• Performance tester(s), tool(s), and environment(s) needed for every project
• Organizational culture shift typically required • Demands an “architect level” performance specialist • Getting team dynamics “right” takes time – often 2-3
projects ∆ from non-Agile efforts:
• Performance testing at many application levels, daily • Root cause analysis dramatically easier • Tuning is continual • Unity of purpose and goals is essential • The only “external” gates are acceptance testing and
executive business decisions
www.PerfTestPlus.com © 2006-11 PerfTestPlus, Inc. All rights reserved. Performance Testing in Agile Page 19
First Steps Involve management in evolving performance culture Get performance into user stories
• Response times • Resource allocations
Involve developers • Code profiling • Integrate performance measurement into unit tests
Make performance part of story acceptance • Quantitative • Qualitative
Involve performance testers • Throughout the cycle • Gain respect as advisors as well as testers
www.PerfTestPlus.com © 2006-11 PerfTestPlus, Inc. All rights reserved. Performance Testing in Agile Page 20
Valuable Resources Performance Testing Guidance for Web Applications
• Microsoft patterns&practices 2009 • *especially* Chapters 1-7 • Free from http://perftestingguide.codeplex.com/
How to Reduce the Cost of Testing • CRC Press, 2011 • Chapter 16 (free from PerfTestPlus.com) • Order book on Amazon
Other Articles and Websites • An Explanation of Performance Testing on an Agile
Team (part 1, part 2) • What is performance testing? • Software performance testing: There is no 'I' in 'team'
www.PerfTestPlus.com © 2006-11 PerfTestPlus, Inc. All rights reserved. Performance Testing in Agile Page 21
Questions
www.PerfTestPlus.com © 2006-11 PerfTestPlus, Inc. All rights reserved. Performance Testing in Agile Page 22
Contact Info
Scott Barber Chief Technologist
PerfTestPlus, Inc E-mail: [email protected]
Web Site: www.PerfTestPlus.com
www.PerfTestPlus.com © 2006-11 PerfTestPlus, Inc. All rights reserved. Performance Testing in Agile Page 23
Oversee, Review and/or
Approve
• Executives set the tone • Articulate performance goals and risks to the business • The rest of the team must
• Achieve goals • Mitigate risks • Assess & report both -- continually
Roles & Tasks (Reference)
www.PerfTestPlus.com © 2006-11 PerfTestPlus, Inc. All rights reserved. Performance Testing in Agile Page 24
Build Capacity Model for n+2 Design n+2Architect &
Model n+2Supervise,
Troubleshoot, Validate and/
or Update Architecture
Model
Observe, Advise,
Troubleshoot, Validate and/
or UpdateArchitecture
Model
Cross-Validate n+2
Capacity Predictions, Architecture
Model Predictions,
Perf Test Results in Staging
• Design to achieve goals & mitigate risk • Establish assessment targets, goals & budgets • Set & enforce sound performance practices • Troubleshoot, tune, update design & refactor
Roles & Tasks (Reference)
www.PerfTestPlus.com © 2006-11 PerfTestPlus, Inc. All rights reserved. Performance Testing in Agile Page 25
Baseline n+1 in Prod
Build Capacity Model for n+2
Reverse Cross-Validate Release
Readiness Perf Tests in Staging
Validate n+2
Capacity Predictions
in Prod
Cross-Validate n+2
Capacity Predictions, Architecture
Model Predictions,
Perf Test Results in Staging
Provide Current Production Info & Models, Consult When Needed
• Be “hardware architects” • Design to achieve goals & mitigate risk • Establish assessment targets, goals & budgets • Set & enforce sound performance practices • Provide production feedback • Troubleshoot, tune, update design & refactor
Roles & Tasks (Reference)
www.PerfTestPlus.com © 2006-11 PerfTestPlus, Inc. All rights reserved. Performance Testing in Agile Page 26
Add Performance & Profiling Stories
Develop Stories; Profile; Unit
Level Performance
Testing
Design n+2
• Use coding practices that support good performance • Code to achieve goals, targets, budgets & mitigate risk • Assess components continually • Troubleshoot, tune, & refactor
Roles & Tasks (Reference)
www.PerfTestPlus.com © 2006-11 PerfTestPlus, Inc. All rights reserved. Performance Testing in Agile Page 27
Roles & Tasks (Reference)
Reverse Cross-Validate Release
Readiness Perf Tests in Staging
Project Investigative Performance
Testing
SystemInvestigative Performance
TestingCross-
Validate n+2 Capacity
Predictions, Architecture
Model Predictions,
Perf Test Results in Staging
Observe, Consult and
Assist Where
Appropriate
Update Project Performance
Strategy; Design/Develop
Investigative Performance
Tests
Prod Readiness
Perf Testing (System)
• Manage whole system assessment • Own simulation (usage) model(s) • Contribute to component assessments • Cross-validate across environments