@gmuecke

Performance TestingWITH GATLING

1

@gmuecke

Workshop mission

Theory

What? Why? How?

Introduction to Gatling

Test Design

Structure

Load Modelling

(Diagnostics)

Practice

The Juice Shop Story

Record

Edit &

Execute Load Tests with

Gatling

2

@gmuecke

By the end of this Workshop

You should have a local test setup with

An example Application

A Gatling Test Script

Basic Monitoring

So you could experiment for yourself afterwards

You should know

How to create a Simulation using the Recorder (Record & Replay)

Waht the elements of a Gatling Load Test are

Know the Basics for Designing a Load Test

3

@gmuecke

How to do Performance Testing?

4

@gmuecke

Type of Load Test

Batch Online

5

• Process the most events in the

least time

• Event Driven

• Focus on Low Response Times

• Get the most done in the least time

and resources

• Data Driven

• Focus on Maximum Troughput

@gmuecke

Input to Load Test Scenarios

Performance Requirements

Target Users (Concurrent, per Duration, Total)

Response Time Targets (90%, 95%, 99%)

Throughput

Historical Data

Number of Total Users per Duration

Number of Concurrent Users

Peak Loads (Peak Month/Day/Hour/Minute)

Request Logs

Educated Guesses / Gut Feeling

6

@gmuecke

Example

Requirements

The system is capable of

Serving 1000 concurrent users with

an average Response Time of 1.5s

Source: The Project Manager

What are the most relevant

information?

This Photo by Unknown Author is licensed under CC BY-NC-SA

7

@gmuecke

Numbers need a Context

Concurrent Users (fit into CPU)

vs. Concurrent Sessions (fit into Memory)

vs. Users per Period (fit into System)

Average

vs. perceived Average (90 %)

vs. Percentiles (95%,99%,99.9%)

1000 Conc.Users, avg < 1.5s

vs 1000 Users/h, 90% < 1.5s

This Photo by Unknown Author is licensed under CC BY-NC-SA

8

@gmuecke

Load Testing Practices

Soak Testing

Discover Leaks

SLA Regressions

Stress Testing

Testing Stability

Overload / Recovery

Benchmarking

Discover Regressions between different Versions or Configurations

9

@gmuecke

A typical load test: Constant Load

Constant Load

+ Ramp up / down

Allowing the System to adapt to Load (warm up)

Distributes Load (virtual users)

Good for:

Finding latent bugs, i.e.Memory or Resource Leaks

Precise Measurements

Finding Regressions

Finding Stability Issues

Statistical Response Times

Known Load Distributions

Load

%

t

10

@gmuecke

Why do a ramp up?

System Warm-up

Allow JIT to optimize code

Allow Caches to be populated

Fetch or Initiate Resources

(i.e. Database Connections)

Allow Queues and Buffers to fill to a stable level

Distribute Load evenly

System’s Performance Characteristics arenon-linear during ramup

11

This Photo by Unknown Author is licensed under CC BY-NC-ND

@gmuecke

What do we test for?

Capacity

Oversubscribed resources

CPU

Memory

Bandwith

Storage

Queues

…

Response times under load

Stability

Resources leaks

Memory

Network connections

Filehandle

System crashes

System recovery

…

12

@gmuecke

Any Heuristics to sum it up?

13

@gmuecke

CCD IS EARI

Context: Project context is central to successful performance testing.

Criteria: Business, project, system, & user success criteria.

Design: Identify system usage, and key metrics; plan and design tests.

Install: Install and prepare environment, tools, & resource monitors.

Script: Implement test design using tools.

Execute: Run and monitor tests. Validate tests, test data, and results.

Analyze: Analyze the data individually and as a cross-functional team.

Report: Consolidate and share results, customized by audience.

Iterate: "Lather, rinse, repeat" as necessary.

14

http://scott-barber.blogspot.com/2007/05/performance-testing-core-principles-ccd.html

@gmuecke

IVECTRAS

INVESTIGATION or VALIDATION

of END-TO-END or COMPONENT

response TIMES and/or RESOURCE consumption

under ANTICIPATED or STRESSFUL conditions

15

http://scott-barber.blogspot.com/2007/08/this-is-second-installment-of-currently.html

@gmuecke

GatlingAN INTRODUCTION

16

@gmuecke

Gatling Basics

Written in Scala

And so are the Load Tests

Asynchronous, Non-Blocking, Event-Driven Concurrency Model

Virtual Users are «Messages», not threads

Less Resource Consumption & Higher Loads

Supports Open and Closes Workload Modells

Standalone or Integration with Maven / Gradle

Useful in Continuous Integration Pipelines

Visual Reports

17

@gmuecke

Pros & Cons

Pro

Testing Framework

Small, lightweight → reduced to

max

No installation:

Maven / Gradle Plugin

Download Standalone

Highly Scalable

Event Based Load Generation

Documentation & Community

Free & OpenSource

Cons

No UI

only for Recorder

Requires programming skills

Scala

Limited Protocol Support

HTTP & JMS

Basic reporting

18

@gmuecke

Threads vs EventsLOAD GENERATION BASICS

19

@gmuecke

Thread based Load Generation

Thread 1

Virtual User

20

@gmuecke

Thread based Load Generation

Thread 1

Request

Virtual User

21

@gmuecke

Thread based Load Generation

Thread 1

Thread WAIT

Virtual User

22

@gmuecke

Thread based Load Generation

Thread 1

Response

Virtual User

23

@gmuecke

Thread based Load Generation

Thread 1

Think Time

Virtual User

24

@gmuecke

Thread based Load Generation

Thread 1

User Scenario

Virtual User

25

@gmuecke

Thread based Load Generation

Thread 1

Pace time

Virtual User

26

@gmuecke

Thread based Load Generation

Thread 1

Virtual User

Real User 1 Real User 2

27

@gmuecke

Thread based Load Generation

Thread 1

Thread 2

28

Thread 3

@gmuecke

Thread based Load Generation

Characteristics

Each Simulated Real User depends on it’s previous user

Each consecutive Request depends on Response for previous Request

Requires lot of Resource

Suffers from Coordinated Omission (SUT throttles Load Generator)

Rampup ensures that generated load is evenly distributed

Pacing ensures that load distribution remains stable

Good for

Closed User Groups (i.e. Employees, Named Users), Dependant Users

Determine Capacity

Tool Example: Apache JMeter

29

@gmuecke

Event Based Load Generation

Event

Thread

Timer Event Response

Event

AsyncRequest

Async

Response

30

@gmuecke

Event Based Load Generation

Event

Thread

31

More

Users

@gmuecke

Event Based Load Generation

Characteristics

Simulated Users are Independent

Sent Requests are independent from previous Requests

Requests/Responses are handled asynchronously

Requires less resources for same load as thread based systems

No Coordinated Omission

Limited by the Processing Capacity of the Event-Thread

Ramp up / down is defined by change in User Rate

Good for

Independent Users

Open User Groups (i.e. for public web sites)

User Experience Rating

Tool-Example: Gatling

32

@gmuecke

Three

elements of

a Load Test

• How groups of usersnavigate throughyour application

User Flows

• How to fetch a typical page and it’sresources

Page Scripts

• How the users arrive at your page

Load Function

33

@gmuecke

Three

elements of

a Load Test

User Flows

Page Scripts

Load Function

34

GET /index.html

GET /favicon.ico

GET /someScript.js

GET /img/background.jpg

@gmuecke

Test Development with Gatling 35

Scenarios Load ModelsPages

@gmuecke

Test Development with Gatling 36

val page1 = exec(

http("request_0")

.get("/")

.headers(acceptHtml)

.resources(...))

Pages Scenarios

val scn =

scenario("Simple")

.exec(page1)

Load Models

scn.inject(

constantUsersPerSec(500)

during (1 minute)

)

@gmuecke

ExerciseRECORD & REPLAY SCRIPT WITH GATLING

37

@gmuecke

The Boutique de Jus story

We want to revolutionize the way people consume healthy food. We’re going to launch several service offerings, starting with ourultimate Juice-Shop.

Business goal:

Our investors expect profit at peak of 1.5Mio$ / month (30days)

Assuming a revenue 5$/order and profit margin of 10%

→ we need 100k orders per day

We expect 70k-100k visits on our site per hour.

We assume that all visits occur within 10 hours per day-

Conversion rate of 10% (submitted orders)

Response time per page should be 1.5s – 4s

38

@gmuecke

CCD IS EARI

Context: Startup, Webshop: boutique de jus

Criteria: 100k user/h, 10k orders/h, Response times: 1.5s-4s

Design: → see excercises

Install: → see excercise 1

Script: → see excercises

Execute: execute scripts & monitor with VisualVM

Analyze: see gatling reports & VisualVM

Report: create a brief report in exercise 3 using PROOF

Iterate: in three excercises

39

@gmuecke

Test Mission

Criteria:

Set the technical baseline

Are you able to test with Gatling?

Design: Record a very simple flow for load testing

Install: setup the test lab package on your machines

40

@gmuecke

Designing Performance Tests

41

@gmuecke

FIBLOTS

Frequent: Common application usage.

Intensive: i.e. Resource hogging activities.

Business Critical: Even if these activities are both rare and not risky

Legal: Stuff that will get you sued or not paid.

Obvious: Stuff that is likely to earn you bad press

Technically Risky: New technologies, old technologies, places where

it’s failed before, previously under-tested areas

Stakeholder Mandated: Don’t argue with the boss (too much).

42

http://scott-barber.blogspot.com/2011/09/model-workloads-for-performance-testing.html

@gmuecke

What is a Load Model?

How Load is distributed over time

Defined by a rate of events in a period

Users arriving at the application

Requests

Actions / Clicks

Bytes

43

@gmuecke

Equi-Distribution

Completely Synthetic

Suitable for

long-running stability Tests

Benchmarking

Load should aim at 80% systemcapacity

Good for tracking long-term effects

Resource / Memory Leaks

Gargbage Collection Singularities

Native support in Gatling

44

@gmuecke

Ramp-up

Slowly increasing Load on System

System can Warm-up

Populate Caches

Compile & Optimize Code (Java ->

JIT)

Users & Requests get distributed

Native support in Gatling

45

@gmuecke

Open vs Closed Workload

Open

Load Generator and System Under

Test have no Feedback Loop

SUT can be overloaded

Modelled by Arrival Rate

Suitable for Unknown User Group

and Statistical Distributions

Supported by Gatling

Closed

Load Generator and System under

Test in a Feedback Loop

SUT throttles LG

Load System establishes a balance

Modelled by Concurrent Users

Suitable for Closed User Groups

and Known Loads

Since Gatling 3.0

46

@gmuecke

Analysis &

Reporting

48

Photo: Calvinius

@gmuecke

What do we need for a deeper analysis?

Precise reporting of your observations

Baseline configuration of the system

Model of the system (architecture) to visualize the flow

The right people available

49

@gmuecke

External Observations

Response Times

Averages, Min, Max

Percentile (90% is perceived average!)

Outliers

Histogram

Error Responses

Which page fails most often?

Live Uses (Users that have not finished their scenario)

Piling up?

Constant

50

@gmuecke

Internal Observations

CPU Consumption

System

I/O Activity

Disc

Network

Context Switching

User

Threads & Contention

GC Activity

Algorithms

Hardware Counters

Garbage Collection Activity

GC Logs

Allocation Rates

GC Generational Behavior

Memory Consumption

Heap

Non-Heap

Thread Stacks

Direct Allocations

Code Cache

Metaspace

Leaks

51

@gmuecke

Java Performance:

Diagnostic Model*

Hardware/OS

JVM

Application

Usage Patterns / Actors

52

@kcpeppe

Dynamics

No Dynamics

@gmuecke

Java Performance:

Diagnostic Process

Application

JVM

Threads

System CPU: sys > 10% user?

User <100%?

Memory efficient? GC Logs

App/CPU Profiling

53

IO, Disk, Network,…

system profiling: netstat,

mpstat, iostat, sar, strace, ...

Thread Dump

Thread Starvation

GC Tuning, Collectors, Pool Sizes,

Memory Profiling, Frequency,

Lifespan …

CPU Profiling

Algorithms & Data Structures

@kcpeppe

@gmuecke

Reporting: PROOF

Past. What happened during the session?

Results. What was achieved during the session?

Obstacles. What got in the way of good testing?

Outlook. What still needs to be done?

Feelings. How does the tester feel about all this?

54

http://www.satisfice.com/articles/sbtm.pdf

@gmuecke

Summary

55

@gmuecke

Warning!

THE NEXT SLIDE CONTAINS A FLASHY PICTURE!

VIEWER DISCRETION IS ADVISED.

56

@gmuecke

57

Forget everything what you have heard so far, now comes the real important stuff!

@gmuecke

Test Development with Gatling 58

Scenarios Load ModelsPages

@gmuecke

Test Development with Gatling 59

val page1 = exec(

http("request_0")

.get("/")

.headers(acceptHtml)

.resources(...))

Pages Scenarios

val scn =

scenario("Simple")

.exec(page1)

Load Models

scn.inject(

constantUsersPerSec(500)

during (1 minute)

)

# BaselOne18