performance testing of a road tolling...

Performance Testing of a Road Tolling System

Oliver BaendlSiegfried Goeschl

Montag, 15. April 13

Gauteng Open Road TollingSouth Africa


42 Tolling PointsMontag, 15. April 13

1.500.000 registered road-userMontag, 15. April 13

11 Satellite CentersMontag, 15. April 13

41 E-Toll Shops


20 Mobile Payment UnitsMontag, 15. April 13

Central Operation CenterMontag, 15. April 13

750 Concurrent CRM Client Users


162 vehicle passges per secondMontag, 15. April 13

400 concurrent web usersMontag, 15. April 13

Two Data Centers


Master Control Program


Project Overview

• Vehicles are passing through a tolling point

• The vehicle is identified either using the license plate number (via OCR) or a tag

• Vehicle passage are transmitted to Open Road Tolling Back Office (ORTBO)

• 90% video passages with three images

• 10% tag passages


Project Overview

• Vehicle passages are collected and assigned to road-user accounts in the Transaction Clearing House (TCH)

• TCH responsible for billing the customer

• Passages of unregistered or black-listed cars are processed by the Violation Processing Center (VPC)


Project Overview

• Overall system contains multiple Oracle RACs and MSSQL clusters

• Interfaces between the subsystems mostly web services

• Backup data center replicates the “Central Operation Center” over dark fiber


Schedule Phase One

• Performance testing of CRM Server simulating 750 concurrent & active CRM client sessions

• Performance testing of public-facing web portal simulating 400 concurrent & active web user sessions

• On success the CRM subsystem goes live and has real user accounts


Schedule Phase Two

• Simulating up to 162 vehicle passage per second

• Validating the end-to-end processing under load (passage processing, customer billing and violation processing)

• On success the overall system goes live


Performance Test Scope

• End to end performance acceptance test

• Mandatory and client-witnessed test

• Strict acceptance criteria regarding

‣ throughput

‣ number and severity of errors

‣ validation of test runs


Performance Test Types

Baseline Test One hour average load

Stress Test One hour peak load

Endurance Test 24 hours maximum load


Performance Test Hardware

• Six dedicated load injectors

‣ Windows 2008 Server

‣ 3 physical and 3 virtual boxes

‣ Dual quad-cores with 8 GB RAM

• No direct access outside the data center

‣ Citrix over RDP over RDP


Performance Test Tools• All performance test tools have issues

‣ Make sure that they work for you

• We prefer JVM-based test tools

‣ Easy to migrate between different OS

‣ We are Java developers

• We prefer command-line invocation

‣ RDP over slow network is really slow


Performance Test Tools

• We use a non-distributed installation

‣ Complex network and routing

‣ Firewall not under our control

• Check the license and price tags

‣ Virtual users can be expensive

‣ Connectivity to license server?


The Test Tools We Used

Subsystem Interface Planned Delivered

CRM Server WebService

Public Web Portal HTML

Vehicle PassageProcessing WebService


Apache JMeter

• Open Source and free

• Implemented a new reporting backend to scope with huge result files

• Rock solid - only minor issues

• Extensible using scripting and Java libraries

• Rough GUI compared to commercial tools

• Unsuitable for complex ASP.NET web sites


WAPT Pro

• Excellent value for money

• Good support

• ASP.NET module saved my day

• Extensible through Javascript

• One major issue with generation of test values under high load (requires restart)


soapUI Pro

• Excellent for prototyping and functional testing of web services

• WebService mocking saved my day

• Extensible using Groovy and Java libraries

• Various issues during load testing

• soapUI was replaced with JMeter for performance tests


• No software installation

• Take two (or more)

• Automation is your friend

• Performance test for everyone

• Performance tests are a valuable asset

Some More Thoughts


No Software Installation


No Software Installation

• Portable Apps on USB Stick

‣ Run tests directly from USB Stick

‣ Clone USB Stick to hard disk

• Have all your tools on the USB stick

‣ Java, JMeter, Editor, Git, ...


Take Two (or more)

• A minimum of two share-nothing load injectors are required to prove that the production servers is causing the performance bottleneck (and not your test tool)

• We used a maximum of five share-nothing load injectors simultaneously


• Performance test were run mostly at night and over the weekend

• Test execution managed by Hudson

• Test failures triggers email notification

• Test protocols are copied to FTP server

Automation Is Your Friend


Performance Tests for Everyone

• All test scenarios are configured as Hudson jobs

‣ Baseline, stress & endurance test

‣ Some other internal tests

• Everyone can start and monitor a performance test scenario over the web browser

• Even developers can run performance tests


• Performance tests can be used in creative and un-planned ways

‣ Smoke test for new deployment

‣ Testing database failover

‣ Testing data center replication & failover

‣ Testing different database setups

Performance Tests as Asset


The few things you should take home


Lessons Learned


Lessons Learned

• Tests tools are buggy


Lessons Learned




Lessons Learned



• Creation and maintenance of complex performance test suite is a project on its own


Questions & Answers


Resources

JMeter SLA Report https://github.com/sgoeschl/jmeter-sla-report

JAMon API http://jamonapi.sourceforge.net/

JMeter ANT Tasks http://www.programmerplanet.org/pages/projects/jmeter-ant-task.php

WAPT Pro http://www.loadtestingtool.com

soapUI http://www.soapui.org/


https://github.com/sgoeschl/jmeter-sla-report

https://github.com/sgoeschl/jmeter-sla-report

http://jamonapi.sourceforge.net

http://jamonapi.sourceforge.net

http://www.programmerplanet.org/pages/projects/jmeter-ant-task.php




http://www.loadtestingtool.com

http://www.loadtestingtool.com

http://www.soapui.org

http://www.soapui.org

The Software Stack

JMeter

Cacti

JSPWiki

Hudson

WAPT Pro soapUI Pro

Git

Portable Apps

Scripting


The Challenge Ahead


The Challenge Ahead• A 100+ million EUR project



‣ High complexity



‣ High complexity

‣ Prohibitive contractual penalties



‣ High complexity


• Multi-site Development



‣ High complexity



‣ Teams located in Austria, Sweden, Argentina and South Africa



‣ High complexity



‣ Teams located in Austria, Sweden, Argentina and South Africa

• Development & QA overloaded


The Challenge Ahead


The Challenge Ahead

• Performance testing on critical path


The Challenge Ahead


• Lots of politics and finger-pointing


The Challenge Ahead



‣ “Your performance tests are broken”


The Challenge Ahead




• Complex test scenarios


The Challenge Ahead




• Complex test scenarios

‣ Often broken by new deployment


“Plans are invitation to disappointment.”


“Two Data Centers, altogether 300 servers, MSSQL clusters,

Oracle RACs, large SAN storage, 1.5 million registered road-users, 750 CRM users, backup data

center over dark fiber”


Things We Did To JMeter


Things We Did To JMeter• Three million WS requests over HTTPS

requests where the server closes the socket after each request




• Sending up to 400 SOAP requests / sec





• Sending 400 KB SOAP requests






‣ 70 MByte/sec over the wire







• Using up to 750 worker threads per JVM







• Using up to 750 worker threads per JVM

‣ 64-bit JVM with -Xmx=2048M


performance testing of a road tolling...

Documents