What You are Doing Wrong With Automated Testing

by Shawn Faunce and Martin Folkoff

OwnershipDisplacement

Ice CreamCone

AcceptingFailure Recorders Monolith

Coverage=

QualitySunk Cost

Underthe

Hood

“An anti-pattern is just like a pattern, except that instead of a solution, it gives something that looks superficially like a solution but isn't one.”

Andrew Koenig, former director of the IBM Watson Laboratory and author

99%UI Tests

Ice CreamCone

70%Unit & Integration Tests

20%API Tests

10%UI Tests

ExploratoryTesting

Looks like a good solution• Easy to create• Mimics user interactions• You see something happening

Anti-Pattern: Automated testing strategy relies mostly or exclusively on User Interface (UI) testing.

But it introduces problems• Maintenance nightmare• Slow feedback loops• Complex debugging

Why this anti-pattern does more harm than good: UI dependency introduces fragility, which results in ignored test failures.Benefit(s) from the correct approach: Confidence, Fast Feedback

10%Unit & Integration Tests

20%API Tests

70%UI Tests

ExploratoryTesting

Looks like a good solution• There are bigger fish to fry• Only happens occasionally• Development is not slowed

Anti-Pattern: Accepting failure by Ignoring intermittent or low-priority bugs because they are easily ignored or not noticed by users.

But it introduces problems• Lose confidence in your tests• Flaky tests• Increases technical debt

AcceptingFailure

Why this anti-pattern does more harm than good: Invalidates the information you get from your regression test suite.Benefit(s) from the correct approach: Confidence

Looks like a good solution• Tools make it quick and easy• No programming required• Leverage manual testers

Anti-Pattern: Exclusive use of a UI recorder to build automated tests, instead of frameworks built on programming.

But it introduces problems• Extremely fragile• Dependent on having a UI• Slow feedback loop

Recorders

Why this anti-pattern does more harm than good: UI dependency creates slow feedback loops and unnecessary test maintenance complexity.Benefit(s) from the correct approach: Fast Feedback

Looks like a good solution• Avoid managing dependencies• Easy to report on• Just keep appending new tests

Anti-Pattern: Organizing and running all of your individual tests as a single test suite.

But it introduces problems• Slow feedback loop• Longer build and execution times• Encourages sloppy engineering

Monolith

Why this anti-pattern does more harm than good: Complex tests lead to complex debugging. Benefit(s) from the correct approach: Focused Feedback, Scalability, Fast Feedback

OwnershipDisplacement Looks like a good solution

• Testers test, and this is a test• Frees up developers• Clear lines of ownership

Anti-Pattern: Assigning ownership of code quality to somebody other than the person who wrote the code, like a tester.

But it introduces problems• Missing feedback to developer• Creates ownership issues• Cannot divorce quality

Why this anti-pattern does more harm than good: The missing feedback to the developer doesn’t prevent reoccurrences of the same defect.Benefit(s) from the correct approach: Defect Prevention, Decrease in Test Flakiness

Looks like a good solution• Can’t get any better than 100%• Everything appears to be tested• The dashboard is all green

Anti-Pattern: Measuring quality primarily based on the percentage test coverage of code.

But it introduces problems• Are the tests good tests?• Can lead to assertion free testing• Generate tests vs. meaningful tests

Coverage=

Quality

“I expect a high level of coverage. Sometimes managers require one. There’s a subtle difference.”

“People optimize their performance according to how they’re measured”

“If a part of your test suite is weak in a way that coverage can detect, it’s likely also weak in a way coverage can’t detect.”

-- Brian MarickHow to Misuse Code Coveragehttp://www.exampler.com/testing-com/writings/coverage.pdf

Why this anti-pattern does more harm than good: You are under investing in higher risk areas. Benefit(s) from the correct approach: Insights

Looks like a good solution• Something’s better than nothing• We’ll fix them someday• Can’t justify throwing them away

Anti-Pattern: Your automated tests are not working but you don’t want to abandon them because of the investment you made in them.

But it introduces problems• Builds technical debt• Confidence loss• Inefficient testing

Sunk Cost

Why this anti-pattern does more harm than good: Good money chases bad tests, meanwhile you are getting no benefit from these tests all the while losing confidence.Benefit(s) from the correct approach: Confidence, Test Efficiency

Anti-Pattern: You are using a COTS product so you resort to UI level tests exclusively because you can’t get under the hood.Under

TheHood Looks like a good solution

• It is your only available option• You are only configuring• You are only testing your code

But it introduces problems• Complex UI testing• Tightly coupled architecture• Only option is manual testing

Why this anti-pattern does more harm than good: Enables aversion to critical thinking and scalable testing solutions.Benefit(s) from the correct approach: Scalability, Fast Feedback

Data

base

Appl

icat

ion

Before

Data

base

Serv

ice

Appl

icat

ion

After

Before:• Hard to Test, SQL hidden

in the application layer• Embedded query logic• Complex UI Level Testing

After• Testable, SQL exposed• Added service layer• Easily inspected • Better architecture

?