jagger: Сервер непрерывного тестирования...
DESCRIPTION
Доклад Ильи Кацова, SQA Days-13, 26-27 апреля 2013, www.sqadays.comTRANSCRIPT
Scalable eCommerce Platform Solutions
Jagger: Continuous Performance Testing of Mission-Critical Applications
Ilya Katsov, Grid Dynamics
Scalable eCommerce Platform Solutions
04/08/2023 2
Performance Testing
Tool
System Under
Test
Report
What’s wrong?
Scalable eCommerce Platform Solutions
04/08/2023 3
What we do• Develop backend services for 3 of Top-30 US Retailers• Production clusters of 40 nodes and more• Load up to 50K TPS• Heavy computations
in navigational queries:
Scalable eCommerce Platform Solutions
04/08/2023 4
Antipatterns of Performance Testing • Acceptance performance testing• Independent functional and performance testing• Poor tracking of testing results• Insufficient introspection of the system under test• Failover is tested in production
Scalable eCommerce Platform Solutions
04/08/2023 5
Performance Testing
Tool
System Under
Test
Report
What’s wrong?
Scalable eCommerce Platform Solutions
04/08/2023 6
Performance Testing Ecosystem
Development Environment
Performance Environment
QA Environment
Production Environment
Functional
Tests
Profiling Performance Tests
Monitoring
Release Manager
Performance Test Lead
Performance Architect
Technical Lead
Scalable eCommerce Platform Solutions
04/08/2023 7
Slave Slave Slave SlaveSlave
Master
SuT
SuT
Agent
SuT
SuT
Agent
SuT
SuT
Agent
SuT
SuT
Agent
SuT
SuT
Agent
SuT
SuT
Agent
SuT
SuT
Agent
Jagger Technical Architecture Web UI
HDFS
MySQLZookeeper
Scalable eCommerce Platform Solutions
04/08/2023 8
Lessons learned: Automate. Automate to scale.• Completely automated flows• Jenkins plugin for automatic deployment• Results warehousing in the database
Scalable eCommerce Platform Solutions
04/08/2023 9
Lessons learned: Automate. Automate to scale.• Decision makers that can access all collected data• Session comparison• VCS-friendly configuration in plain text files
Scalable eCommerce Platform Solutions
04/08/2023 10
Lessons learned: Monitoring is everything.• OS-level and JVM-level metrics• Ability to load external metrics• All metrics can be involved into automatic decision
making• Jagger can be used to monitor an external process:
ActivityJagger AgentJagger Cluster
CI Server
Scalable eCommerce Platform Solutions
04/08/2023 11
Lessons learned: Profile system under test• Performance issues investigation – a typical flow:
Problem ResolvedFix Performance Testing
Performance Problem Detected
Scalable eCommerce Platform Solutions
04/08/2023 12
Lessons learned: Profile system under test• Sampling profiler for JVM (stacktraces via JMX)• Profiler contains intelligence to detect hot spots
Performance Problem Detected Problem ResolvedFix
Profiling results
Performance Testing
Scalable eCommerce Platform Solutions
04/08/2023 13
Lessons learned: Validate responses • Custom validators• Record results of single-user test and use for validation
Scalable eCommerce Platform Solutions
04/08/2023 14
Lessons learned: Manage your data well.• Statistics from one of our accounts:• About 100 test cases• 20 scheduled test sessions weekly + ad hoc testing • 30GB of test results, more than 1500 retained sessions
Scalable eCommerce Platform Solutions
04/08/2023 15
Lessons learned: Manage your data well.• Reports - PDF and web UI
Scalable eCommerce Platform Solutions
04/08/2023 16
Results and Roadmap
Successful:• Continuous testing• Monitoring• Validation• Data management
Not very successful:• Continuous robustness
testing• Integration with production
monitoring
Scalable eCommerce Platform Solutions
04/08/2023 17
https://jagger.griddynamics.netDistribution and Documentation