![Page 1: Efficient Dependency Detection for Safe Java Test Acceleration · ESEC/FSE @_jon_bell_ September 4, 2015 Efficient Dependency Detection for Safe Java Test Acceleration Jonathan](https://reader030.vdocuments.net/reader030/viewer/2022041100/5ed7d6e18e533201dc4aea30/html5/thumbnails/1.jpg)
@_jon_bell_ESEC/FSE September 4, 2015
Efficient Dependency Detection for Safe Java
Test AccelerationJonathan Bell, Gail Kaiser, Eric Melski and Mohan Dattatreya
Columbia University & Electric Cloud, Inc
![Page 2: Efficient Dependency Detection for Safe Java Test Acceleration · ESEC/FSE @_jon_bell_ September 4, 2015 Efficient Dependency Detection for Safe Java Test Acceleration Jonathan](https://reader030.vdocuments.net/reader030/viewer/2022041100/5ed7d6e18e533201dc4aea30/html5/thumbnails/2.jpg)
@_jon_bell_ESEC/FSE September 4, 2015
Simplified Software Lifecycle
Make changes to code
Build & test
Commit
How long is too long of a build?
1 day? 6 hours? 10 minutes?
![Page 3: Efficient Dependency Detection for Safe Java Test Acceleration · ESEC/FSE @_jon_bell_ September 4, 2015 Efficient Dependency Detection for Safe Java Test Acceleration Jonathan](https://reader030.vdocuments.net/reader030/viewer/2022041100/5ed7d6e18e533201dc4aea30/html5/thumbnails/3.jpg)
@_jon_bell_ESEC/FSE September 4, 2015
Simplified Software Lifecycle
• Compile sources
• Generate documentation
• Run tests
• Package
Make changes to code Build & test Commit
![Page 4: Efficient Dependency Detection for Safe Java Test Acceleration · ESEC/FSE @_jon_bell_ September 4, 2015 Efficient Dependency Detection for Safe Java Test Acceleration Jonathan](https://reader030.vdocuments.net/reader030/viewer/2022041100/5ed7d6e18e533201dc4aea30/html5/thumbnails/4.jpg)
@_jon_bell_ESEC/FSE September 4, 2015
Testing Dominates Build Times
20%
38%
41%
351 projects from GitHub
Testing
Other
Compiling
![Page 5: Efficient Dependency Detection for Safe Java Test Acceleration · ESEC/FSE @_jon_bell_ September 4, 2015 Efficient Dependency Detection for Safe Java Test Acceleration Jonathan](https://reader030.vdocuments.net/reader030/viewer/2022041100/5ed7d6e18e533201dc4aea30/html5/thumbnails/5.jpg)
@_jon_bell_ESEC/FSE September 4, 2015
Testing Dominates Build Times
14%
26%60%
Projects taking > 10 minutes to build (69)
Testing
Other
Compiling
![Page 6: Efficient Dependency Detection for Safe Java Test Acceleration · ESEC/FSE @_jon_bell_ September 4, 2015 Efficient Dependency Detection for Safe Java Test Acceleration Jonathan](https://reader030.vdocuments.net/reader030/viewer/2022041100/5ed7d6e18e533201dc4aea30/html5/thumbnails/6.jpg)
@_jon_bell_ESEC/FSE September 4, 2015
Testing Dominates Build Times
2%8%
90%
Projects taking > 1 hour to build (8)
Testing
OtherCompiling
![Page 7: Efficient Dependency Detection for Safe Java Test Acceleration · ESEC/FSE @_jon_bell_ September 4, 2015 Efficient Dependency Detection for Safe Java Test Acceleration Jonathan](https://reader030.vdocuments.net/reader030/viewer/2022041100/5ed7d6e18e533201dc4aea30/html5/thumbnails/7.jpg)
Faster tests = Faster builds
![Page 8: Efficient Dependency Detection for Safe Java Test Acceleration · ESEC/FSE @_jon_bell_ September 4, 2015 Efficient Dependency Detection for Safe Java Test Acceleration Jonathan](https://reader030.vdocuments.net/reader030/viewer/2022041100/5ed7d6e18e533201dc4aea30/html5/thumbnails/8.jpg)
Test acceleration is well studied
![Page 9: Efficient Dependency Detection for Safe Java Test Acceleration · ESEC/FSE @_jon_bell_ September 4, 2015 Efficient Dependency Detection for Safe Java Test Acceleration Jonathan](https://reader030.vdocuments.net/reader030/viewer/2022041100/5ed7d6e18e533201dc4aea30/html5/thumbnails/9.jpg)
@_jon_bell_ESEC/FSE September 4, 2015
Regression Test Selection
Test 1 Test 2 Test 3
Test 8 Test 9 Test 10
Test 4 Test 5 Test 6 Test 7
Test 11 Test 12 Test 13 Test 14
Gligoric et al. [ISSTA ’15], Orso et al. [FSE ’04], Harrold et al. [OOPSLA ’01]
Changeset
![Page 10: Efficient Dependency Detection for Safe Java Test Acceleration · ESEC/FSE @_jon_bell_ September 4, 2015 Efficient Dependency Detection for Safe Java Test Acceleration Jonathan](https://reader030.vdocuments.net/reader030/viewer/2022041100/5ed7d6e18e533201dc4aea30/html5/thumbnails/10.jpg)
@_jon_bell_ESEC/FSE September 4, 2015
Regression Test Selection
Test 1 Test 3
Test 8
Test 4
Gligoric et al. [ISSTA ’15], Orso et al. [FSE ’04], Harrold et al. [OOPSLA ’01]
Changeset
Tests not relevant to changeset: skipped
![Page 11: Efficient Dependency Detection for Safe Java Test Acceleration · ESEC/FSE @_jon_bell_ September 4, 2015 Efficient Dependency Detection for Safe Java Test Acceleration Jonathan](https://reader030.vdocuments.net/reader030/viewer/2022041100/5ed7d6e18e533201dc4aea30/html5/thumbnails/11.jpg)
@_jon_bell_ESEC/FSE September 4, 2015
Test Suite Minimization
Test 1 Test 2 Test 3
Test 8 Test 9 Test 10
Test 4 Test 5 Test 6 Test 7
Test 11 Test 12 Test 13 Test 14
< /> Code
Hao et al. [ICSE ’12]; Orso et al. [ICSE ’09]; Jeffrey et al. [TSE ’07]; Tallam et al. [PASTE ’05]; Jones et al. [TOSEM ’03]; Harrold et al. [TOSEM ’93]; Chen et al. [IST ’98]; Wong et al. [ICSE ’95] and more
![Page 12: Efficient Dependency Detection for Safe Java Test Acceleration · ESEC/FSE @_jon_bell_ September 4, 2015 Efficient Dependency Detection for Safe Java Test Acceleration Jonathan](https://reader030.vdocuments.net/reader030/viewer/2022041100/5ed7d6e18e533201dc4aea30/html5/thumbnails/12.jpg)
@_jon_bell_ESEC/FSE September 4, 2015
Test Suite Minimization
Test 3
Test 8 Test 9 Test 10
Test 4 Test 5 Test 6 Test 7
Test 11 Test 13 Test 14
< /> Code
Hao et al. [ICSE ’12]; Orso et al. [ICSE ’09]; Jeffrey et al. [TSE ’07]; Tallam et al. [PASTE ’05]; Jones et al. [TOSEM ’03]; Harrold et al. [TOSEM ’93]; Chen et al. [IST ’98]; Wong et al. [ICSE ’95] and more
Redundant tests: removed
![Page 13: Efficient Dependency Detection for Safe Java Test Acceleration · ESEC/FSE @_jon_bell_ September 4, 2015 Efficient Dependency Detection for Safe Java Test Acceleration Jonathan](https://reader030.vdocuments.net/reader030/viewer/2022041100/5ed7d6e18e533201dc4aea30/html5/thumbnails/13.jpg)
@_jon_bell_ESEC/FSE September 4, 2015
Test Parallelization
Test 1 Test 2 Test 3
Test 8 Test 9 Test 10
Test 4 Test 5 Test 6 Test 7
Test 11 Test 12 Test 13 Test 14
![Page 14: Efficient Dependency Detection for Safe Java Test Acceleration · ESEC/FSE @_jon_bell_ September 4, 2015 Efficient Dependency Detection for Safe Java Test Acceleration Jonathan](https://reader030.vdocuments.net/reader030/viewer/2022041100/5ed7d6e18e533201dc4aea30/html5/thumbnails/14.jpg)
@_jon_bell_ESEC/FSE September 4, 2015
Test Parallelization
Test 1 Test 2 Test 3
Test 8 Test 9
Test 10
Test 4
Test 5 Test 6 Test 7
Test 11 Test 12 Test 13 Test 14
![Page 15: Efficient Dependency Detection for Safe Java Test Acceleration · ESEC/FSE @_jon_bell_ September 4, 2015 Efficient Dependency Detection for Safe Java Test Acceleration Jonathan](https://reader030.vdocuments.net/reader030/viewer/2022041100/5ed7d6e18e533201dc4aea30/html5/thumbnails/15.jpg)
@_jon_bell_ESEC/FSE September 4, 2015
Controlled Regression Testing Assumption
Tests </> Code
![Page 16: Efficient Dependency Detection for Safe Java Test Acceleration · ESEC/FSE @_jon_bell_ September 4, 2015 Efficient Dependency Detection for Safe Java Test Acceleration Jonathan](https://reader030.vdocuments.net/reader030/viewer/2022041100/5ed7d6e18e533201dc4aea30/html5/thumbnails/16.jpg)
@_jon_bell_ESEC/FSE September 4, 2015
Controlled Regression Testing Assumption
Tests </> Code
External Factors
External Factors External Factors
External Factors
![Page 17: Efficient Dependency Detection for Safe Java Test Acceleration · ESEC/FSE @_jon_bell_ September 4, 2015 Efficient Dependency Detection for Safe Java Test Acceleration Jonathan](https://reader030.vdocuments.net/reader030/viewer/2022041100/5ed7d6e18e533201dc4aea30/html5/thumbnails/17.jpg)
@_jon_bell_ESEC/FSE September 4, 2015
Controlled Regression Testing Assumption
Tests </> Code
External Factors
External Factors External Factors
External Factors
Test 1
![Page 18: Efficient Dependency Detection for Safe Java Test Acceleration · ESEC/FSE @_jon_bell_ September 4, 2015 Efficient Dependency Detection for Safe Java Test Acceleration Jonathan](https://reader030.vdocuments.net/reader030/viewer/2022041100/5ed7d6e18e533201dc4aea30/html5/thumbnails/18.jpg)
@_jon_bell_ESEC/FSE September 4, 2015
Controlled Regression Testing Assumption
Tests </> Code
External Factors
External Factors External Factors
External Factors
Test 1Test 2
![Page 19: Efficient Dependency Detection for Safe Java Test Acceleration · ESEC/FSE @_jon_bell_ September 4, 2015 Efficient Dependency Detection for Safe Java Test Acceleration Jonathan](https://reader030.vdocuments.net/reader030/viewer/2022041100/5ed7d6e18e533201dc4aea30/html5/thumbnails/19.jpg)
@_jon_bell_ESEC/FSE September 4, 2015
Controlled Regression Testing Assumption
Tests </> Code
External Factors
External Factors External Factors
External Factors
Test 1Test 2Test 3
![Page 20: Efficient Dependency Detection for Safe Java Test Acceleration · ESEC/FSE @_jon_bell_ September 4, 2015 Efficient Dependency Detection for Safe Java Test Acceleration Jonathan](https://reader030.vdocuments.net/reader030/viewer/2022041100/5ed7d6e18e533201dc4aea30/html5/thumbnails/20.jpg)
Not sound in practice
![Page 21: Efficient Dependency Detection for Safe Java Test Acceleration · ESEC/FSE @_jon_bell_ September 4, 2015 Efficient Dependency Detection for Safe Java Test Acceleration Jonathan](https://reader030.vdocuments.net/reader030/viewer/2022041100/5ed7d6e18e533201dc4aea30/html5/thumbnails/21.jpg)
@_jon_bell_ESEC/FSE September 4, 2015
Test Dependencies
Test 1 Test 2 Test 3 Test 4Test 1 Test 2
Shared File
Write, Value “A”
Test 4
ReadWrite, Value “B”
Value: B
Test 3
Read
![Page 22: Efficient Dependency Detection for Safe Java Test Acceleration · ESEC/FSE @_jon_bell_ September 4, 2015 Efficient Dependency Detection for Safe Java Test Acceleration Jonathan](https://reader030.vdocuments.net/reader030/viewer/2022041100/5ed7d6e18e533201dc4aea30/html5/thumbnails/22.jpg)
@_jon_bell_ESEC/FSE September 4, 2015
Test Dependencies
Test 1 Test 2 Test 3Test 4
Shared File
![Page 23: Efficient Dependency Detection for Safe Java Test Acceleration · ESEC/FSE @_jon_bell_ September 4, 2015 Efficient Dependency Detection for Safe Java Test Acceleration Jonathan](https://reader030.vdocuments.net/reader030/viewer/2022041100/5ed7d6e18e533201dc4aea30/html5/thumbnails/23.jpg)
@_jon_bell_ESEC/FSE September 4, 2015
Test Dependencies
Test 1 Test 2 Test 3Test 4Test 1 Test 2 Test 3
Shared File
Write, Value “A”
Test 4
Write, Value “B”
Read, Expect Value “A”
Value: B
Read
![Page 24: Efficient Dependency Detection for Safe Java Test Acceleration · ESEC/FSE @_jon_bell_ September 4, 2015 Efficient Dependency Detection for Safe Java Test Acceleration Jonathan](https://reader030.vdocuments.net/reader030/viewer/2022041100/5ed7d6e18e533201dc4aea30/html5/thumbnails/24.jpg)
@_jon_bell_ESEC/FSE September 4, 2015
Test Dependencies
Test 1 Test 2 Test 3Test 4Test 1 Test 2 Test 3
Shared File
Write, Value “A”
Test 4
Write, Value “B”
Read, Expect Value “A”
Value: B
A manifest test dependency
Read
![Page 25: Efficient Dependency Detection for Safe Java Test Acceleration · ESEC/FSE @_jon_bell_ September 4, 2015 Efficient Dependency Detection for Safe Java Test Acceleration Jonathan](https://reader030.vdocuments.net/reader030/viewer/2022041100/5ed7d6e18e533201dc4aea30/html5/thumbnails/25.jpg)
@_jon_bell_ESEC/FSE September 4, 2015
Test Dependencies: A Clear and Present Danger
• Really exist in practice (Zhang et al. found 96, Luo et al. found 14)
• Hard to specify - if we could specify, would be safe to accelerate
• Rarely: isolated (especially not in long building projects)
• Existing technique to detect: combinatorially run tests [Zhang, et al ’14]
![Page 26: Efficient Dependency Detection for Safe Java Test Acceleration · ESEC/FSE @_jon_bell_ September 4, 2015 Efficient Dependency Detection for Safe Java Test Acceleration Jonathan](https://reader030.vdocuments.net/reader030/viewer/2022041100/5ed7d6e18e533201dc4aea30/html5/thumbnails/26.jpg)
@_jon_bell_ESEC/FSE September 4, 2015
Brute Force Dependency Detection
• Looked at feasibility on 10 large open source test suites
• Exhaustive approach: > 10300 years to find all dependencies
• Pairwise approach: Average 31,882 executions of the entire test suite to find (incomplete) dependencies
• Problem: How do we safely accelerate test suites in the presence of unknown dependencies?
![Page 27: Efficient Dependency Detection for Safe Java Test Acceleration · ESEC/FSE @_jon_bell_ September 4, 2015 Efficient Dependency Detection for Safe Java Test Acceleration Jonathan](https://reader030.vdocuments.net/reader030/viewer/2022041100/5ed7d6e18e533201dc4aea30/html5/thumbnails/27.jpg)
@_jon_bell_ESEC/FSE September 4, 2015
Manifest Test Dependencies
• Definition: a data dependence between tests T1, T2 that results in the outcome of T2 changing
• All manifest dependencies are data dependencies
• Not all data dependencies are manifest dependencies
![Page 28: Efficient Dependency Detection for Safe Java Test Acceleration · ESEC/FSE @_jon_bell_ September 4, 2015 Efficient Dependency Detection for Safe Java Test Acceleration Jonathan](https://reader030.vdocuments.net/reader030/viewer/2022041100/5ed7d6e18e533201dc4aea30/html5/thumbnails/28.jpg)
@_jon_bell_ESEC/FSE September 4, 2015
Data Dependencies
Test 1 Test 2 Test 3 Test 4Test 1 Test 2
Shared File
Write, Value “A”
Test 4
ReadWrite, Value “B”
Test 3
Read
Present Dependencies:Test 1 must run before 2 and 3 Test 4 must run after 2 and 3
![Page 29: Efficient Dependency Detection for Safe Java Test Acceleration · ESEC/FSE @_jon_bell_ September 4, 2015 Efficient Dependency Detection for Safe Java Test Acceleration Jonathan](https://reader030.vdocuments.net/reader030/viewer/2022041100/5ed7d6e18e533201dc4aea30/html5/thumbnails/29.jpg)
Key Insight: Dependencies don’t need to be precise,
but must be sound
![Page 30: Efficient Dependency Detection for Safe Java Test Acceleration · ESEC/FSE @_jon_bell_ September 4, 2015 Efficient Dependency Detection for Safe Java Test Acceleration Jonathan](https://reader030.vdocuments.net/reader030/viewer/2022041100/5ed7d6e18e533201dc4aea30/html5/thumbnails/30.jpg)
@_jon_bell_ESEC/FSE September 4, 2015
IntuitionTest 1 Test 2 Test 3
Test 8 Test 9 Test 10
Test 15
Test 4 Test 5 Test 6 Test 7
Test 11 Test 12 Test 13 Test 14
Idle extra capacity
![Page 31: Efficient Dependency Detection for Safe Java Test Acceleration · ESEC/FSE @_jon_bell_ September 4, 2015 Efficient Dependency Detection for Safe Java Test Acceleration Jonathan](https://reader030.vdocuments.net/reader030/viewer/2022041100/5ed7d6e18e533201dc4aea30/html5/thumbnails/31.jpg)
@_jon_bell_ESEC/FSE September 4, 2015
IntuitionTest 1 Test 2 Test 3 Test 8 Test 9 Test 10 Test 15
Test 4 Test 5 Test 6 Test 7 Test 11 Test 12 Test 13
Test 14
Test 1 Test 2 Test 3 Test 8 Test 9 Test 10 Test 15
Test 4 Test 5 Test 6 Test 7 Test 11 Test 12 Test 13
Test 14
Idle extra capacity
A lot of dependencies, but still a 2x speedup
![Page 32: Efficient Dependency Detection for Safe Java Test Acceleration · ESEC/FSE @_jon_bell_ September 4, 2015 Efficient Dependency Detection for Safe Java Test Acceleration Jonathan](https://reader030.vdocuments.net/reader030/viewer/2022041100/5ed7d6e18e533201dc4aea30/html5/thumbnails/32.jpg)
@_jon_bell_ESEC/FSE September 4, 2015
ElectricTest - Detecting Data Dependencies in Java
• Tracks in-memory dependencies (JVMTI plugin)
• Tracks file and network dependencies (IO-Trace agent)
• Implemented entirely within the Oracle or OpenJDK JVM, no specialized drivers, etc required
• Captures stack traces when dependencies occur to support debugging
• Generates dependency trees to enable sound test acceleration
![Page 33: Efficient Dependency Detection for Safe Java Test Acceleration · ESEC/FSE @_jon_bell_ September 4, 2015 Efficient Dependency Detection for Safe Java Test Acceleration Jonathan](https://reader030.vdocuments.net/reader030/viewer/2022041100/5ed7d6e18e533201dc4aea30/html5/thumbnails/33.jpg)
@_jon_bell_ESEC/FSE September 4, 2015
Identifying Heap Dependencies
After each test, garbage collect; traverse heap to map objects back to static fields.
Class A
End of test 1
![Page 34: Efficient Dependency Detection for Safe Java Test Acceleration · ESEC/FSE @_jon_bell_ September 4, 2015 Efficient Dependency Detection for Safe Java Test Acceleration Jonathan](https://reader030.vdocuments.net/reader030/viewer/2022041100/5ed7d6e18e533201dc4aea30/html5/thumbnails/34.jpg)
@_jon_bell_ESEC/FSE September 4, 2015
Identifying Heap Dependencies
After each test, garbage collect; traverse heap to map objects back to static fields.
Class A
End of test 1
![Page 35: Efficient Dependency Detection for Safe Java Test Acceleration · ESEC/FSE @_jon_bell_ September 4, 2015 Efficient Dependency Detection for Safe Java Test Acceleration Jonathan](https://reader030.vdocuments.net/reader030/viewer/2022041100/5ed7d6e18e533201dc4aea30/html5/thumbnails/35.jpg)
@_jon_bell_ESEC/FSE September 4, 2015
Identifying Heap Dependencies
After each test, garbage collect; traverse heap to map objects back to static fields.
Class A
W1
W1
W1
W1
W1
W1
W1
W1
W1
static fieldsta
tic fie
ld static field
static field
End of test 1
![Page 36: Efficient Dependency Detection for Safe Java Test Acceleration · ESEC/FSE @_jon_bell_ September 4, 2015 Efficient Dependency Detection for Safe Java Test Acceleration Jonathan](https://reader030.vdocuments.net/reader030/viewer/2022041100/5ed7d6e18e533201dc4aea30/html5/thumbnails/36.jpg)
@_jon_bell_ESEC/FSE September 4, 2015
Identifying Heap Dependencies
During test execution, monitor accesses to existing objects
Class A
W1W1 W1 W1
W1
W1W1
W1
W1
static fieldsta
tic fie
ld static field
static fieldDuring Test 2
![Page 37: Efficient Dependency Detection for Safe Java Test Acceleration · ESEC/FSE @_jon_bell_ September 4, 2015 Efficient Dependency Detection for Safe Java Test Acceleration Jonathan](https://reader030.vdocuments.net/reader030/viewer/2022041100/5ed7d6e18e533201dc4aea30/html5/thumbnails/37.jpg)
@_jon_bell_ESEC/FSE September 4, 2015
Identifying Heap Dependencies
During test execution, monitor accesses to existing objects
Class A
W1W1 W1 W1
W1
W1W1
W1
W1
static fieldsta
tic fie
ld static field
static fieldW2
W2 Write!
During Test 2
![Page 38: Efficient Dependency Detection for Safe Java Test Acceleration · ESEC/FSE @_jon_bell_ September 4, 2015 Efficient Dependency Detection for Safe Java Test Acceleration Jonathan](https://reader030.vdocuments.net/reader030/viewer/2022041100/5ed7d6e18e533201dc4aea30/html5/thumbnails/38.jpg)
@_jon_bell_ESEC/FSE September 4, 2015
Identifying Heap Dependencies
During test execution, monitor accesses to existing objects
Class A
W1W1 W1 W1
W1
W1W1
W1
W1
static fieldsta
tic fie
ld static field
static fieldW2
W2
W1
Read!
During Test 2
Dependency!
![Page 39: Efficient Dependency Detection for Safe Java Test Acceleration · ESEC/FSE @_jon_bell_ September 4, 2015 Efficient Dependency Detection for Safe Java Test Acceleration Jonathan](https://reader030.vdocuments.net/reader030/viewer/2022041100/5ed7d6e18e533201dc4aea30/html5/thumbnails/39.jpg)
@_jon_bell_ESEC/FSE September 4, 2015
Identifying External Dependencies
Application under test Network
Filesystem
Log remote host address
Log path
![Page 40: Efficient Dependency Detection for Safe Java Test Acceleration · ESEC/FSE @_jon_bell_ September 4, 2015 Efficient Dependency Detection for Safe Java Test Acceleration Jonathan](https://reader030.vdocuments.net/reader030/viewer/2022041100/5ed7d6e18e533201dc4aea30/html5/thumbnails/40.jpg)
ElectricTest enables sound exploitation of existing test acceleration techniques
![Page 41: Efficient Dependency Detection for Safe Java Test Acceleration · ESEC/FSE @_jon_bell_ September 4, 2015 Efficient Dependency Detection for Safe Java Test Acceleration Jonathan](https://reader030.vdocuments.net/reader030/viewer/2022041100/5ed7d6e18e533201dc4aea30/html5/thumbnails/41.jpg)
@_jon_bell_ESEC/FSE September 4, 2015
Safe Test ParallelizationTest 1 Test 2 Test 3
Test 8 Test 9 Test 10
Test 15
Test 4 Test 5 Test 6 Test 7
Test 11 Test 12 Test 13 Test 14
Test 1 Test 2 Test 3 Test 4 Test 5 Test 6 Test 7
Test 8 Test 9 Test 10 Test 11 Test 12 Test 13 Test 14
Test 15
![Page 42: Efficient Dependency Detection for Safe Java Test Acceleration · ESEC/FSE @_jon_bell_ September 4, 2015 Efficient Dependency Detection for Safe Java Test Acceleration Jonathan](https://reader030.vdocuments.net/reader030/viewer/2022041100/5ed7d6e18e533201dc4aea30/html5/thumbnails/42.jpg)
@_jon_bell_ESEC/FSE September 4, 2015
Safe Test ParallelizationTest 1 Test 2 Test 3 Test 8 Test 9 Test 10 Test 15
Test 4 Test 5 Test 6 Test 7
Test 11 Test 12 Test 13Test 14
Test 1 Test 2 Test 3 Test 8 Test 9 Test 10 Test 15
Test 4 Test 5 Test 6 Test 7
Test 11 Test 12 Test 13Test 14
![Page 43: Efficient Dependency Detection for Safe Java Test Acceleration · ESEC/FSE @_jon_bell_ September 4, 2015 Efficient Dependency Detection for Safe Java Test Acceleration Jonathan](https://reader030.vdocuments.net/reader030/viewer/2022041100/5ed7d6e18e533201dc4aea30/html5/thumbnails/43.jpg)
@_jon_bell_ESEC/FSE September 4, 2015
Safe Test Selection
Test 15
Single test selected to be executed
![Page 44: Efficient Dependency Detection for Safe Java Test Acceleration · ESEC/FSE @_jon_bell_ September 4, 2015 Efficient Dependency Detection for Safe Java Test Acceleration Jonathan](https://reader030.vdocuments.net/reader030/viewer/2022041100/5ed7d6e18e533201dc4aea30/html5/thumbnails/44.jpg)
@_jon_bell_ESEC/FSE September 4, 2015
Safe Test Selection
Test 15Test 1 Test 2 Test 3
Single test selected to be executed with its dependencies
![Page 45: Efficient Dependency Detection for Safe Java Test Acceleration · ESEC/FSE @_jon_bell_ September 4, 2015 Efficient Dependency Detection for Safe Java Test Acceleration Jonathan](https://reader030.vdocuments.net/reader030/viewer/2022041100/5ed7d6e18e533201dc4aea30/html5/thumbnails/45.jpg)
@_jon_bell_ESEC/FSE September 4, 2015
Understanding Dependencies
• What should a developer do about test dependencies?
• Might be intentional (e.g. cache shared state)
• Might be unintentional but OK (e.g. loggers)
• Might be unintentional and bad (e.g. bug)
![Page 46: Efficient Dependency Detection for Safe Java Test Acceleration · ESEC/FSE @_jon_bell_ September 4, 2015 Efficient Dependency Detection for Safe Java Test Acceleration Jonathan](https://reader030.vdocuments.net/reader030/viewer/2022041100/5ed7d6e18e533201dc4aea30/html5/thumbnails/46.jpg)
@_jon_bell_ESEC/FSE September 4, 2015
Assisting Debugging
Debugging information reported by the previous technique
Test 3 Test 1Depends on
![Page 47: Efficient Dependency Detection for Safe Java Test Acceleration · ESEC/FSE @_jon_bell_ September 4, 2015 Efficient Dependency Detection for Safe Java Test Acceleration Jonathan](https://reader030.vdocuments.net/reader030/viewer/2022041100/5ed7d6e18e533201dc4aea30/html5/thumbnails/47.jpg)
@_jon_bell_ESEC/FSE September 4, 2015
Assisting DebuggingException*in*thread*"main"*edu.columbia.cs.psl.testdepends.DependencyException:*Static*Field*ClassA.FieldA*member*was*previously*written*by*Test*1,*read*here.*! at!edu.columbia.cs.psl.testdepends.test.Example$NestedExample.dragons(Example.java:20)!! at!edu.columbia.cs.psl.testdepends.test.Example.moreMagic(Example.java:12)!! at!edu.columbia.cs.psl.testdepends.test.Example.magic(Example.java:8)!! at!edu.columbia.cs.psl.testdepends.test.Example.main(Example.java:15)!
Test that wrote value
Stack trace shows use
Value that is read
![Page 48: Efficient Dependency Detection for Safe Java Test Acceleration · ESEC/FSE @_jon_bell_ September 4, 2015 Efficient Dependency Detection for Safe Java Test Acceleration Jonathan](https://reader030.vdocuments.net/reader030/viewer/2022041100/5ed7d6e18e533201dc4aea30/html5/thumbnails/48.jpg)
@_jon_bell_ESEC/FSE September 4, 2015
Evaluation
• RQ1: Recall (accuracy)
• RQ2: Runtime overhead
• RQ3: Impact on acceleration
![Page 49: Efficient Dependency Detection for Safe Java Test Acceleration · ESEC/FSE @_jon_bell_ September 4, 2015 Efficient Dependency Detection for Safe Java Test Acceleration Jonathan](https://reader030.vdocuments.net/reader030/viewer/2022041100/5ed7d6e18e533201dc4aea30/html5/thumbnails/49.jpg)
@_jon_bell_ESEC/FSE September 4, 2015
RQ1: Recall
Project
Joda
XMLSecurity
Crystal
Synoptic
Dependencies DetectedGround Truth
Ground Truth
2
4
18
1
ElectricTestWriters
15
3
15
10
Readers
121
103
39
117
![Page 50: Efficient Dependency Detection for Safe Java Test Acceleration · ESEC/FSE @_jon_bell_ September 4, 2015 Efficient Dependency Detection for Safe Java Test Acceleration Jonathan](https://reader030.vdocuments.net/reader030/viewer/2022041100/5ed7d6e18e533201dc4aea30/html5/thumbnails/50.jpg)
@_jon_bell_ESEC/FSE September 4, 2015
RQ1: Recall
Project
Joda
XMLSecurity
Crystal
Synoptic
Dependencies DetectedGround Truth
Ground Truth
2
4
18
1
ElectricTestWriters
15
3
15
10
Readers
121
103
39
117
ElectricTest SharedResource Locations
App
39
3
4
3
Library
12
15
19
14
![Page 51: Efficient Dependency Detection for Safe Java Test Acceleration · ESEC/FSE @_jon_bell_ September 4, 2015 Efficient Dependency Detection for Safe Java Test Acceleration Jonathan](https://reader030.vdocuments.net/reader030/viewer/2022041100/5ed7d6e18e533201dc4aea30/html5/thumbnails/51.jpg)
@_jon_bell_ESEC/FSE September 4, 2015
RQ2: Overhead• Selected 10 projects with > 10 minutes of tests
• Also included projects studied by Zhang et al, averaging < 10 seconds of testing
• Previous exhaustive approach slowdown: >10300X
• Previous heuristic approach slowdown: 31,882X
• ElectricTest slowdown: 36X (885X faster than previous approach)
![Page 52: Efficient Dependency Detection for Safe Java Test Acceleration · ESEC/FSE @_jon_bell_ September 4, 2015 Efficient Dependency Detection for Safe Java Test Acceleration Jonathan](https://reader030.vdocuments.net/reader030/viewer/2022041100/5ed7d6e18e533201dc4aea30/html5/thumbnails/52.jpg)
@_jon_bell_ESEC/FSE September 4, 2015
0X 1,000X 2,000X 3,000X 4,000X 5,000X 6,000X 7,000X 8,000X 9,000X 10,000X mongo%java%driver-
tachyon-
spring%data%mongodb-
xml-security-
ne8y-
je8y.project-
crystal-
crunch-
camel-
:tan-
synop:c-
hazelcast-
mule-
joda%:me-
ElectricTest Slowdown Pairwise Slowdown
*418,000X
RQ2: Overhead
On average, ElectricTest is 885X faster than running all tests pairwise
Slowdown relative to a single test suite execution (lower is better)
![Page 53: Efficient Dependency Detection for Safe Java Test Acceleration · ESEC/FSE @_jon_bell_ September 4, 2015 Efficient Dependency Detection for Safe Java Test Acceleration Jonathan](https://reader030.vdocuments.net/reader030/viewer/2022041100/5ed7d6e18e533201dc4aea30/html5/thumbnails/53.jpg)
@_jon_bell_ESEC/FSE September 4, 2015
0X 50X 100X 150X 200X 250X 300X mongo%java%driver-
tachyon-spring%data%mongodb-
xml-security-ne8y-
je8y.project-crystal-crunch-camel-:tan-
synop:c-hazelcast-
mule-joda%:me-
RQ2: Overhead
Average 36X
A lot of fast running tests:Runtime dominated by pauses
between tests (gc)
Slowdown relative to a single test suite execution (lower is better)
![Page 54: Efficient Dependency Detection for Safe Java Test Acceleration · ESEC/FSE @_jon_bell_ September 4, 2015 Efficient Dependency Detection for Safe Java Test Acceleration Jonathan](https://reader030.vdocuments.net/reader030/viewer/2022041100/5ed7d6e18e533201dc4aea30/html5/thumbnails/54.jpg)
@_jon_bell_ESEC/FSE September 4, 2015
0X 5X 10X 15X 20X 25X 30X
camel&
crunch&
hazelcast&
je/y.project&
mongo5java5driver&
mule&
ne/y&
spring5data5mongodb&
tachyon&
:tan&
Safe UnsafeSpeedup (higher is better)
RQ3: Impact on Acceleration
Average (Unsafe) 19x
Average (Safe) 7x
![Page 55: Efficient Dependency Detection for Safe Java Test Acceleration · ESEC/FSE @_jon_bell_ September 4, 2015 Efficient Dependency Detection for Safe Java Test Acceleration Jonathan](https://reader030.vdocuments.net/reader030/viewer/2022041100/5ed7d6e18e533201dc4aea30/html5/thumbnails/55.jpg)
@_jon_bell_ESEC/FSE September 4, 2015
![Page 56: Efficient Dependency Detection for Safe Java Test Acceleration · ESEC/FSE @_jon_bell_ September 4, 2015 Efficient Dependency Detection for Safe Java Test Acceleration Jonathan](https://reader030.vdocuments.net/reader030/viewer/2022041100/5ed7d6e18e533201dc4aea30/html5/thumbnails/56.jpg)
@_jon_bell_ESEC/FSE September 4, 2015
Efficient Dependency Detection for Safe Java
Test AccelerationJonathan Bell, Gail Kaiser, Eric Melski and Mohan Dattatreya
Columbia University & Electric Cloud, Inc