santiago stareast 2016

May 4th, 2016, Orlando, Florida Analyze, Diagnose, and Prevent Test FlakinessMay 5th, 2016, Orlando, FL

Dionny Santiago

2Analyze, Diagnose, and Prevent Test FlakinessDionny Santiago 201

6

Agenda1. Automation Benefits & Challenges2. Metric Analysis3. Owning Full-Stack Testing4. UI Testing Recommendations5. Coding Recommendations

Demo:


6

Testing Landscape• 200 engineers• 70 test engineers• 30+ million lines of code• 400,000 documented tests• Highly complex business rules• Sensitive domain


6

Why automate?• Hands-off

regression testing• Focus on

exploratory• Reduce testing cost• Find defects


6

Consistent

Meaningful

Quick Feedback

Ideal Automation

Detects Defects

Maintainable

Evolves with System


6

UI Automation Challenges

• End-to-end Complexity• Asynchronous Content• Dynamic Content• DOM Structure Evolution


6

UI Automation Challenges

• High test volume• Multiple platforms• Multiple browsers


6

Inconsistent

Unstable

Delayed Feedback

“Flaky” Automation

False Negatives

Unmaintainable


6

“Flaky” Automation


6

Analyze and Prevent Flakiness: MetricsPerformance Metrics Complexity Metrics


6

Execution Performance

Failure % Over Time

Average Duration

Bug Detection Accuracy


6

Failure % Over Time

Useful?

Failure Percentage

# Fi

xtur

es


6

ComplexityMetric Indicator ofTest Fixture Size Test organizationTest Method Size Test case designAssertion Count Over-testing and coupling


6

Inheritance Complexity

0 1

2

2

6*

Depth within the inheritance hierarchy

*Difficult to refactor. May encounter unintended behavior.


6

Cyclomatic Complexity

Start

X > 10

Y < 10

callA() callB()

callC() callD()

Edges - Nodes + 2

Edges = 9

Nodes = 8

CC = 3

Linearly independent paths through a program

*Indicator of branch/path complexity and non-determinism.


6

Tracking and Diagnosing

Continuous Integration Tracking

Combine continuous execution with tracking software


6

Tracking Metrics


6

Tracking Metrics – Run Totals


6

Unit

System

Integration

10%

20%

70%

Complexity

Runtime

Excessive UI AutomationResults in tests that are error-prone and difficult to maintain


6

Unit

Integration

System

70%

20%

10%Complexi

ty

Runtime

Owning Full-Stack TestingQA must be more involved in lower-level testing


6

Unit

Component

System

UI Test Breakdown• Reduce UI tests via unit tests• HTTP Mocking

[End-to-end]


6


6

Demo


6

UI Testing Recommendations

• End-to-end Complexity• Asynchronous Content• Dynamic Content• DOM Structure Evolution


6

UI Testing Architecture

Test Abstraction*

Application Under Test

Application Model*Specifies tests against

Executes tests against

*Reduces end-to-end complexity*Encapsulates DOM Structure and handles dynamic content*Delegates to various platforms and browsers*Handles asynchronous content


6

Application Modeling

Page Objects ElementsControl Objects

• Atomic• Modeled using

IDs, CSS selectors, XPath

• Runtime execution

• Containers• Promote

organization• Compile-time

• Containers• Promote DRY• Compile-time


6

Control

Page ObjectControl

Elements


6

A note on element selectors…

• CSS selectors are faster than XPath on Chrome/Firefox

• Avoid presentation elements (class names)

• Avoid hierarchical paths (div > div > tr > div…)

• Avoid structural elements (div, p, table, span)


6

Element Selectors


6

Coding RecommendationsMaintainability Stability


6

MaintainabilityOrganization of tests promotes readability

Excessive nestingToo flat


6

MaintainabilityEstablish and adhere to a coding style standard


6

Maintainability

Not focused and difficult to debug

Strive for short, focused test cases


6

MaintainabilityFavor composition over inheritance for code reuse


6

StabilityExplicit delays may result in non-determinism and wasted execution time


6

StabilityFavor implicit delays over explicit delays


6

Stability• Implicit vs. explicit delays


6

StabilityRaw exception handling introduces logic and affects readability


6

StabilityInstead, use built-in unit testing constructs


6

StabilityAvoid conditionals and loops

*Recall cyclomatic complexity


6

Summary• Collect and analyze key metrics• Own full stack testing• Implement best practices• Prevent flakiness• Reduce testing cost• Find defects, faster


6

Thank you!

Tariq KingGabriel NunezAdam CandoEduardo Pena

Robert Vanderwall

Troy ThomasJorge Martinez

Michael Mattera

Acknowledgements

Sample Project Availablegithub.com/dionny

santiago stareast 2016

Documents