performance engineering bob dugan, ph.d. computer science department rensselaer polytechnic...
TRANSCRIPT
Performance Performance EngineeringEngineering
Bob Dugan, Ph.D.Bob Dugan, Ph.D.
Computer Science DepartmentComputer Science Department
Rensselaer Polytechnic InstituteRensselaer Polytechnic Institute
Troy, New York 12180Troy, New York 12180
The Nightmare ScenarioThe Nightmare Scenario
Product pre-sold by marketing as carrier scalable
Demos are flashy, fast and successful
Product is supposed to ship to big name customers like GM, Fidelity, and AT&T a week after QA
During QA product is performance tested
Performance tests uncover serious scalability problems
Analysis shows a fundamental architecture flaw
Months of redesign and testing necessary to fix
OverviewOverview
Background Methodology Resources
Incorporate performance into software’s entire life cycleto achieve performance goals.
BackgroundBackground
What is software performance?
BackgroundBackground
Response Time
ThroughputResource Utilization
Background: Response Background: Response TimeTime
How long does it take for a request to execute? Example:
Web page takes 100ms to return to browser after
request.
Interactive applications require 2000ms or less. Tells us a lot about how system is performing. Response time has big impact on the holy grail
of performance THROUGHPUT.
Background: ThroughputBackground: Throughput How many requests per second can be processed? Example:
A server has throughput of 30 requests/sec Supports roughly 1 million requests/10 hour day Assume average user makes 10 requests/day Server will support approximately 100,000 users
Inverse of response time on lightly loaded system. Combined with user model, can be used for performance requirements, capacity planning, sales, and marketing.
Background: Resource Background: Resource UtilizationUtilization
Resources consumed by code processing request. Examples: CPU, memory, network, disk In a closed system, as load increases:
Throughput rises linearly Resources are consumed Response time remains near constant
When a resource is completely consumed: Throughput remains constant Resource utilization remains near constant Response time rises linearly with load
Background: Resource Background: Resource UtilizationUtilization
Virtual UsersVirtual Users Response Response TimeTime
ThroughpuThroughputt
CPU CPU UtilizationUtilization
11 100ms100ms 10 req/sec10 req/sec 25%25%
22 110ms110ms 19 req/sec19 req/sec 53%53%
44 130ms130ms 38 req/sec38 req/sec 96%96%
88 300ms300ms 37 req/sec37 req/sec 98%98%
1616 640ms640ms 39 req/sec39 req/sec 99%99%
Resource utilization is critical to determining throughput/response time relationships.
During performance testing, resource utilization helps identify the cause of a performance problem.
Performance Performance Engineering Engineering MethodologyMethodology
Incorporate performance into software’s entire life cycle to achieve performance goals.
Software Life CycleSoftware Life Cycle
Requirements
Specification
Design
Implementation
Integration
Test
Release
Maintenance
RequirementsRequirements
Functional requirements identified. What are the performance
requirements? Do any functional requirements
interfere with performance requirements?
Performance Performance RequirementsRequirements
What is the capacity planning guide for the system?
How much is a customer willing to pay for performance and scalability? Hardware Software licensing (e.g. OS, Oracle, etc.) System Administration
Example: Internet BankExample: Internet Bank
View accounts Search for specific transaction Transfer money between accounts Export account to Quicken 10 million potential users
Performance ModelPerformance Model
Make some assumptions (refine later) Three tier system: browser, web farm, database
server Database updated nightly with day’s transactions
(e.g. read mostly) User logs in once per 5 day work week, between
8AM-6PM EST Logins evenly distributed Typical user does 3 things, then logs off About 20% of customers will actually use online
banking
Performance ModelPerformance Model
10,000,000 users x 20% adoption rate = 2,000,000 users/week2,000,000 x 3 requests per user = 6,000,000 requests/week6,000,000 / 5 day work week = 1,200,000 requests/day1,200,000 / 10 hour day = 120,000 requests/hour120,000 / 60 minutes per hour = 2000 requests / minute2000 / 60 seconds per hour =
33 requests per second33 requests per second
Performance ModelPerformance Model
Performance Performance RequirementsRequirements
The customer wants to pay as little as possible for the system hardware.
Your company wants the system to perform well, but there’s a development cost.
YOU must find the balance. What are reasonable service times and
throughput for web and database servers?
Performance Performance RequirementsRequirements
DescriptionDescription TimeTime ThroughputThroughput
Web/App Service Web/App Service TimeTime
< 1000 ms< 1000 ms 1 req/sec per 1 req/sec per processorprocessor
Database Service Database Service TimeTime
< 100 ms< 100 ms 10 req/sec per 10 req/sec per processorprocessor
Total Response Total Response TimeTime
< 1100 ms< 1100 ms 33 req/sec33 req/sec
RequirementsRequirements
Goal: Identify/eliminate performance problems before theyget into Functional/Design/UI specifications.
Functional/Design/UIFunctional/Design/UI
Goal: Eliminate performance problems before writing a line of code.Example:
Requirements say that users should be able to search on account activity using any combination of activity fields (e.g. date, payee, amount, check#).
Functional/Design specification describes an ad-hoc query mechanism with pseudocode that allows users to conduct this search using a single database query.
Performance analysis of prototype ad-hoc query shows a throughput of 2 req/sec with 100% CPU utilization on a two processor database server.
PrototypingPrototyping Great time to play Investigate competing architectures Don’t forget performance!
Example: HTML Tag Processing Engine for Internet Bank Initial performance analysis showed 5 tags/sec. Web server
CPU 100%. Dependency on size of page. Second iteration improved to 20 tags/sec. Still too slow!
Service time allotted completely consumed by tag processing.
Third iteration at 60 tags/sec. No page size dependency.
ImplementationImplementation
Long duration Break into drops Performance assessment of drops Track progress A maturing system increases in complexity and
jeopardizes performance Use instrumentation!
Goal: Identify and eliminate performance problems before they are discovered in QA.
InstrumentationInstrumentation
Code must be instrumented by development Allows self-tuning Provides execution trace for debugging Aids performance analysis in lab Useful for monitoring application in
production
Example: InstrumentationExample: Instrumentation
PUNCHIN eCal::Metrics::TableStatisticsDB::unitTest []
|PUNCHIN eCal::Metrics::TableStatisticsDB::tableSelect []
||PUNCHIN eCal::Oracle::prepare []
||PUNCHOUT eCal::Oracle::prepare [] 131.973028182983 msecs
|PUNCHOUT eCal::Metrics::TableStatisticsDB::tableSelect [] 642.809987068176 msecs
PUNCHOUT eCal::Metrics::TableStatisticsDB::unitTest [] 643.355011940002 msecs
sub unitTest {
eCal::Metrics->new()->punchIn();
my $tableName;
my $result = tableSelect("users");
print $result."\n";
eCal::Metrics->new()->punchOut();
}
eCal::Metrics->new()->setEnabled("true");
eCal::Metrics->new()->setShowExecutionTrace("true");
unitTest;
Sample code
Sample instrumentation output
Activating instrumentation
TestingTesting
Goal: Identify and eliminate performance problems before they get into production.
Performance testing and analysis must occur throughput development!!!
In late cycle QA, should be a formality with no surprises.
A surprise at this point will delay product release or potentially kill a product.
MaintenanceMaintenance
Goal: Identify and eliminate performance problems before they are detected by users.
Management console for resource monitoring
Metrics pages Instrumentation
ConclusionConclusion
Incorporate performance into software’s
entire life cycle to achieve performance goals.
Resources: BooksResources: Books
Smith/Williams, “Software Performance Engineering” Jain, “The Art of Computer Systems Performance Analysis” Tannenbaum, “Modern Operating Systems” Elmasri/Navathe, “Fundamentals of Database Systems” Baase, “Computer Algorithms: An Introduction to Design
and Analysis”
Resources: SoftwareResources: Software Resource Monitoring:
Task Manager, Perfmon Sar/iostat/netstat/stdprocess, SE Toolkit BMC Best/1, HP OpenView, Precise Insight
Load Generation LoadRunner, SilkPerformer Webload
Automated Instrumentation Numega True Time, Jprobe Tkprof, Explain Plan, Precise In Depth for Oracle
Resources: Literature/WebResources: Literature/Web www.perfeng.comwww.perfeng.com - Dr. Connie Smith’s Website www.spec.orgwww.spec.org - Benchmarks for computer hardware www.tpc.orgwww.tpc.org - Benchmarks for databases Computer Management Group – annual conference in
December. Workshop on Software Performance – semi-annual
conference in late summer/early fall ACM SIGMETRICs – annual conference in early summer. ACM SIGSOFT/SIGMETRICS publications – periodically
feature papers on performance engineering.
Case StudiesCase Studies
Case Study: Microsoft Case Study: Microsoft VBScriptVBScript
resp = resp & “<ul>”resp = resp & “<ul>”
I=0I=0
while (I<MAX) {while (I<MAX) {
resp = resp & “<li> List Element” & I & oneKString resp = resp & “<li> List Element” & I & oneKString
}}
resp = resp & “</ul>”resp = resp & “</ul>”
Website uses IIS, Microsoft ASP, VBScriptWebsite uses IIS, Microsoft ASP, VBScript Critical page takes 3000 ms, CPU boundCritical page takes 3000 ms, CPU bound Instrumentation shows 2500 ms in a single subroutineInstrumentation shows 2500 ms in a single subroutine Subroutine executed just before html returned to browserSubroutine executed just before html returned to browser Approximate size of HTML page is 64KApproximate size of HTML page is 64K
Case Study: Microsoft Case Study: Microsoft VBScriptVBScript
MAXMAX Response Response TimeTime
Average Time per Average Time per IterationIteration
1010 10ms10ms 1ms1ms
100100 800ms800ms 8ms8ms
10001000 50000ms50000ms 50ms50ms
1000010000 2,000,000m2,000,000mss
200ms200ms
The more the loop iterates, the longer each iteration takes.The more the loop iterates, the longer each iteration takes. VBScript does not support string concatenation VBScript does not support string concatenation Each string operation results in a malloc(), copy, and free which is Each string operation results in a malloc(), copy, and free which is
dependent on the current size of the html stringdependent on the current size of the html string Why is that so bad?Why is that so bad?
Case Study: Microsoft Case Study: Microsoft VBScriptVBScript
I = 1
n
oneK string malloc()cost of malloc() =
Sn = 1 + 2 + … + (n-1) + nSn = n + (n-1) + … + 2 + 1
2Sn = (n+1) + (n+1) + … + (n+1) +(n+1)
Sn = n(n+1)/2
Case Study: Microsoft Case Study: Microsoft VBScriptVBScript
SolutionSolutions?s?