response time difference analysis of performance testing tools
TRANSCRIPT
15/04/2023 CSPA Seminar 1
Performance testing : Analyzing Differences of Response Time between
Performance Testing Tools
BySpoorthi Sham1PI14SSE12
15/04/2023 CSPA Seminar 2
Performance Testing
• Performance testing is the process of determining the speed or effectiveness of a computer, network, software program or device.
• This process can involve quantitative tests done in a lab, such as measuring the response time or the number of MIPS (millions of instructions per second) at which a system functions.
15/04/2023 CSPA Seminar 3
How is it performed?Tools used..
15/04/2023 CSPA Seminar 4
Issues with the tools• Several issues have been observed related to tools when conducting
performance testing: tools compatibility with the software under test tools installation tools setup tools flexibility in doing test both for client and server side response time generated by the tools
To demonstrate and prove that response time from different performance testing
tools is different
To suggest potential reasons or root
cause behind response time
differences
To answer the question: “Why do different performance testing tools produce different
response time?”
ResearchFocus
15/04/2023 CSPA Seminar 5
• Most previous work on performance testing tools comparison ignored on different result reported by each tools
• There is no work so far to understand why they are different against tools
• Each tool claims they are better than the others but none able to justify the performance testing results against the real world.
Related Works
15/04/2023 CSPA Seminar 6
Tool A• Open source tool purely
developed on Java platform• Sits as desktop-based tool• Serves functional, load and stress
testing which is extensible to write own test to suit the scenario.
• Can simulate heavy load on the application, server and even the network
• Able to give instant visual feedback and capable to do load and stress testing via distributed approach
• Supports protocols such as HTTP, JMS, JDBC, FTP, SOAP as well as LDAP
• Used across platforms and supports full multithreading framework
• Allows caching and offline analysis with replaying of test results
Tool B• Open source load testing
tool• Developed using C++
language• Can perform heavy load
tests using scripted HTTP and HTTPS
• Feature-rich GUI-based web server benchmarking tool
• Only runs on Windows-based platform
• Performance scripts are recorded using own proprietary language
• Support custom functions, variable scopes, and random or sequential lists
Tool C• Proprietary tool (one of the
established performance tools in the market)
• Built on Eclipse and Java• Offers automated
performance testing for web and served based application
• Can be used across platforms (Windows, UNIX and Linux)
• Capable to create code free tests, automate test data variation and enables insertion of custom Java code for flexible test customization.
• Supports operating systems such as Windows, Linux, and z/OS
Overview of Performance Testing Tools
15/04/2023 CSPA Seminar 7
Test Environment Setup
Hardware Specification (Both Machines)CPU/processor : Intel Pentium D 3.4 GHzRAM/memory : 2 GBHDD storage : 80 GBNetwork Card : Integrated 10/100/1000 Ethernet
Server machineOperating system : Windows Server 2003 Enterprise Edition SP1Java JDK : JDK 1.6.0 update 21Web server : Internet Information Services 6HTML page size : 65.8 KB (Page: 7 KB; Image 1: 25.2 KB; Image 2: 33.6 KB)
Client machineOperating system : Windows XP SP2Java JDK : JDK 1.6.0 update 21Tool : Tool A (open source);
Tool B (open source); Tool C (proprietary)
15/04/2023 CSPA Seminar 8
Result of Tool A
Findings
15/04/2023 CSPA Seminar 9
Result of Tool B
Findings
15/04/2023 CSPA Seminar 10
Result of Tool C
Findings
15/04/2023 CSPA Seminar 11
Comparison of Performance Test Results between ToolsFindings
15/04/2023 CSPA Seminar 12
Comparison of Performance Test Results between ToolsFindings
15/04/2023 CSPA Seminar 13
Comparison of Performance Test Results between ToolsFindings
15/04/2023 CSPA Seminar 14
Potential Reasons for Response Time Differences
• Some fundamental reasons: capturing and simulating the load used for the
performance testmethod of calculating metrics gathered by each
toollanguage to develop the toolsarchitecture of the respective tools
• Architecture differs greatly:Tool A and C developed by using Java and they
require JVM to run so the value setting for Java Heap Size plays a role to generate the best user load without putting extra burden to the client
Tool B architecture relies on web relay daemon facility allowing CORBA-based communication to be transmitted between machines during executing the performance test
Findings
15/04/2023 CSPA Seminar 15
Conclusion• Different performance testing tools do give
a different response time. • Currently, there is no tool able to tell us if
application is fast enough in term of user experience in a reality
• It is crucial for performance testers to understand that there is no tool that is able to automate and tell us about the full picture of the application's performance that is going to be in a real world
• It is back to human brain to analyze the information given and performance testing tools is just one of the way that can be used to achieve that.
15/04/2023 CSPA Seminar 16
Thank You