wix automation - test state pattern - 11.9.16
TRANSCRIPT
01 / The Problem
E2E tests have more breaking points.
Writing a stable E2E test is
difficult
New E2E tests are even more fragile.
Why?
Pass locally, fail remotely
●Timing issues●Remote connection issues●Bad luck
We try to avoid …False positives Causes lack of trust
NoiseAnalysis becomes difficult
Shame
@Test@Test
@Test
Release Unstable tests
running in production
02 / Solution: @BetaTest
@BetaTest@BetaTest
@BetaTest
Beta @Test
@Test
@Test
Release
Split the tests into 2 “run groups”
No Beta tests - Less coverage
@BetaTest(owner = "tzipi")
public void newTest() {
…
}
@BetaTest@BetaTest
@BetaTest
Beta @Test
@Test
@Test
Release
Results
Beta run group reports results to server
@BetaTest@BetaTest
@BetaTest
Beta @Test
@Test
@Test
Release
Run groups in different configurations
Results
Run X times per night
@Override
public void beforeTest(Description description) {
if (isBetaTestInReleaseRunMode(description) ||
isNonBetaTestInBetaRunMode(description)) { TestUtils.ignoreTest();
}
}
@BetaTest@BetaTest
@BetaTest
Beta @Test
@Test
@Test
Release
“Approve” stable BetaTests
Results
How do we find the stable ones?
Results
Tests that pass X times in a row, with maximum Y failures.
How do we find the stable ones?
So, for each stable test notify the owner to replace @BetaTest with @Test
03 / We Can Do Better
@Test@Test
@Test
Beta @Test
@Test
@Test
Release
No need for special @BetaTest annotation
Results
Beta
Each test begins with a Beta state
Beta
Remains in Beta state for some time
Beta
And (hopefully) changes to stable
Stable
pass X times
@Override
public void beforeTest(Description description) {
TestState state = getTestState(description); if (isBetaTestInReleaseRunMode(description, state) ||
isNonBetaTestInBetaRunMode(description, state)) {
TestUtils.ignoreTest();
}
}
Run tests according to their state
What happens if a test keeps failing?
Betafail X times
What happens if a test keeps failing?
Beta
Failed
Beta
fail X times
Failed beta - manual fix
Beta
Failed
Beta
fail X times
manual fix
Beta
What happens if a stable test starts to “act weird”?
Stable
pass X times
Beta Stable
FlakyFlaky
Become unstable
pass X times
pass X times
What happens if a stable test starts to “act weird”?
Beta
Full pattern
Failed
Beta
fail X timesStabl
e
FlakyFlaky
Become unstable
pass X times
pass X times
manual fix
Stable automation is possible!
Results
@Test@Test
@Test
Beta @Test
@Test
@Test
Release@Test
@Test
@Test
Flaky
Failed Beta
04 / Architecture
API Core Web DB
Web View
Java Client
Test Observer
- testStarted- testIgnored- testFailed- testPassed
Test
- Name*- State**- Creation date- Update date- Annotations
Object modeling
Run Result
- Test Name- Passed / failed- Creation date- Configuration**
*
1 n
05 / Live Demo
This is where you are going to present your final words.This slide is not meant to have a lot of text.Thank You!