analysing test results over time listen beyond the pass/fail · analysing test results over time...

Listen beyond the pass/fail

Analysing test results over time

Lena Wiberg@LenaPejgan

Lena Wiberg

SCI-FI/Fantasy & Games

Patterns & trends

Visual models

Dev. => Tester => Manager

Awesome new job!

Vision! Reality…

@LenaPejgan

It works fine when I run it locally

@LenaPejgan

The test environment was locked by another

user

@LenaPejgan

We hit a random duality check

@LenaPejgan

It was just an unexpected popup

@LenaPejgan

System/service X had the wrong data/version

@LenaPejgan

Oh, it’s just because of bad data

@LenaPejgan

Run

Analyze

Setup

Fix @LenaPejgan

Identify the problem

@LenaPejgan

NOISE!

@LenaPejgan

@LenaPejgan

FALSE SECURITY

Feedback

Stability

Planning

Investment Is it stable enough to explore?

Did my changes break something?

Is this version ready for release?

Are we spending out money right?

@LenaPejgan

It works fine when I run it

locally

Oh, it’s just because of bad data

The test environment was locked by another user

It was just an unexpected

popup

We hit a random

duality check

The integrating system had the

wrong backup/version

+@LenaPejgan

Operations & Hardware

@LenaPejgan

”Our firewall doesn’t allow that”

”Oh, that’s when we do backups”

”We do patches every 3rd Monday”

Test environments & Test data

@LenaPejgan

”The data was used up by another test”

”Someone else was using the environment”

”Service X was down/had the wrong data”

Processes and communication

@LenaPejgan

”We didn’t have time to update the tests”

”We can only test version X”

”We didn’t know the code has changed”

FIX!…Or an ACTUAL bug!

@LenaPejgan

TEAM AWESOME

Summary•Don’t get (too) caught up in the now •Look past the obvious •Kon Mari those tests! Do they spark joy?•Don’t assume flaky can’t be fixed•Reuse is awesome. Except when it’s not…

@LenaPejgan

lena.wiberg@lemontree.se

@LenaPejgan

http://testing.pejgan.se/