11

Auto Regression Analysis

Shuang HeIntel Linux Graphics Validation Team

Open Source Technology Center

2011-08-16

22

Legal DisclaimerINFORMATION IN THIS DOCUMENT IS PROVIDED IN CONNECTION WITH INTEL® PRODUCTS.  EXCEPT AS PROVIDED IN INTEL'S TERMS AND CONDITIONS OF SALE FOR SUCH PRODUCTS, INTEL ASSUMES NO LIABILITY WHATSOEVER, AND INTEL DISCLAIMS ANY EXPRESS OR IMPLIED WARRANTY RELATING TO SALE AND/OR USE OF INTEL PRODUCTS, INCLUDING LIABILITY OR WARRANTIES RELATING TO FITNESS FOR A PARTICULAR PURPOSE, MERCHANTABILITY, OR INFRINGEMENT OF ANY PATENT, COPYRIGHT, OR OTHER INTELLECTUAL PROPERTY RIGHT.

Intel may make changes to specifications, product descriptions, and plans at any time, without notice.  All dates provided are subject to change without notice.Intel is a trademark of Intel Corporation in the U.S. and other countries.*Other names and brands may be claimed as the property of others.Copyright © 2007-2011, Intel Corporation. All rights are protected.

33

Agenda• Overview• Architecture• Auto Nightly Test Infrastructure• Auto Nightly Analysis• Plan• Reference

Catch and analyze every regression!

44

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 310

50

100

150

200

250

3002011 Intel linux graphics driver commit

(week)

(commits)

55

Background• Integrated Test suites:

– Intel-gpu-tools (11 cases)– Rendercheck (10 cases)– Piglit (7500+ cases)– Cairo_test (1272 cases)– OpenGL conformance (4565 cases)– OpenGL ES 2.0 conformance (1198 cases)– Webglc (5685 cases)

66

Architecture

DATABASE

Auto Build

Auto Test

Auto Analysi

s

Test Result Page

gfx-ab-at

infrastructure

auto_bisect

77

Architecture

`

Testing Machine-1

Testing Machine-2

Testing Machine-3

Building MachineExternal Source Repository Internal Source Repository

IntranetInternet

Running AutoBuild & AutoTest

Running test case with Infrastructure

Running test case result analysis with

auto_bisectBisect machines

88

Information required for analysis

• Hardware information– GPU, CPU, Memory

• Build information– Software version

• Test information– Test case version– Test case result

99

Auto Nightly Build

• Build & Distribute Xorg components• Build & Distribute Kernel package• Build & Distribute Test suites• Archives

– X11R7.*.tgz– Kernel.* package

• Build info -> DATABASE– Component branch/commit– Infrastructure branch/commit– Test suite branch/commit

1010

Auto Nightly Test

• Run test cases one by one• Test log collection

– Run time– Output– Dmesg per case

• test result ->DATABASE• Test status watchdog

– TIMEOUT -> killed– GPU hang -> reboot -> resume to next case– System hang -> reboot -> resume to next case

• Pre/post-condition check

1111

Auto Nightly Test Process

Target Test machine

GPU hang check

TIMEOUT control

Running test case

Build/control machine

System hang check

TIMEOUT control

Resume after reboot

Auto Test Control

Init & control

Finish & collect logs

1212

Auto Nightly Analysis

• Track result change– Regression– Improvement

• Locate culprit component– Xorg component– Infrastructure– Testsuite

• Locate culprit commit– Xorg component– Infrastructure– Testsuite

1313

Auto Nightly Analysis Process

Test suite regression analysis

Infrastructure regression analysis

Kernel regression analysis

Auto bisect

Component regression

analysis

package regression

analysis

1414

Component regression analysis

• Purpose– Locating culprit component– uncommon component break

• difficulties– Too many components– Library dependency

1515

Locate culprit component

1616

Locate culprit component

• Reduce analysis effort – Only analyze libraries in use– Only analyze changed libraries

• Reduce dependency– Group related library

1717

Locate culprit commit

1818

bad branch first algorithmIf A is bad commit, and C fixed it, and then F is bad again, A -> B -> C -> D -> E -> F -> G -> H (master)

\ \ / O -> P... Q -> R -> S->T (feature 1) Start with H as bad commit, and D as good commit, it's possible git-bisect would jump to Q, and it will lead to wrong direction

1919

Bad branch first algorithmEXAPMLE 2: Given this condition: A -> B -> C -> D -> E -> F -> G -> H (master) \ O -> P -> Q -> R -> S / (feature 1) \ X -> Y -> Z/ (feature 2) start with H as bad commit, and A as good commit, if y is the target bad commit.

2020

2121

Auto Nightly Analysis Archi965 platforms

Build/control machine

Test machine Bisect machine

Test machine

Test machine

enqueue

enqueue

enqueue

Summary report

Test report&Bisect report

2222

Auto Nightly Analysis

Build machine

Test machine

Regression analysis

Engineers

Build&dist

Case 1

Analyze case 1

Analysis report

request

Case 2

All case completeTest

result Test report summary

Case n

Analyze case n

Analysis report

request

regression

regression

2323

Nightly Test Result Page

2424

Analysis enhancement

• Analysis accuracy– Verification

• Make sure all parents commit are good

• Analysis acceleration– Archive built binaries– Analyze regression in batch– Archive testing result

2525

Some results

Regressionauto

bisected

Improvementauto

bisected

2011-Aug 62 633

2011-Jul 202 381

2011-Jun 247 329

2011-May 97 35

2011-Apr 45 258

2011-Mar 90 102

2011-Feb 19 87

2011-Jan 19 33

2626

Risk

• Unstable– System may hang– GPU hang– Test case unstable

2727

Plan• Email triggered patch test system• PRTS (Patch Regression Test Suite)

– Select test case set based on code modification• Nightly Auto Test

– Capture more info when case crashed• Auto bisect for kernel/GPU hang• Nightly test result page

– Single case test result query– Quality trend indicators

• Pass rate• Regression rate• …

2828

Q&A