subunit2sql: Tracking 1 Test Result in Millions

Matthew Treinishmtreinish@kortar.org

June 15, 2015

OpenStack CI by the Numbers

One proposed change generates:

I 4-25 Devstacks VMs

I ˜1.5k tempest tests per devstack

I ˜130 second level guests started per tempest run

Looking at December:

I ˜12K gating tempest jobs

I ˜15M tempest tests run in the gate

Looking at Kilo:

I ˜70K gating tempest jobs

I ˜>80M tempest tests run in the gate

What was missing?

I An aggregate view of individual tests

I Success and failure rates for tests over time

I Run times for each test over time

I All of this in a programatically queriable location

What is subunit2sql?

I A python library and utilities to aggregate and store subunit streams in a SQL DBI Currently 3 utilities:

I subunit2sql - Take in a subunit stream and store it in a subunit2sql DBI sql2subunit - Output a subunit stream from a subunit2sql DBI subunit2sql-db-manage - Setup and manage the DB schemaI subunit2sql-graph - Tool to generate graphs for analysis with data from a

subunit2sql DB

I Provides a python DB api for interacting with the DB which provides basic iooperations and some analysis tooling

Catching Performance RegressionsGraphing a single test’s run time:

Or when they’re fixed

Tracking an individual test’s failure rateYou can use subunit2sql-graph to figure out failure rates for tests:

which also displays percentages:

I Fail Percentage: 0.2045%

I Success Percentage: 99.7955%

I Skip Percentage: 0.0000%

Where to get more information

I subunit2sql Repository

I subunit2sql Documentation

I Infra Documentation for subunit2sql

I IRC #openstack-qa or #openstack-infra channel on Freenode