scalameter performance regression testing framework aleksandar prokopec, josh suereth
TRANSCRIPT
![Page 1: ScalaMeter Performance regression testing framework Aleksandar Prokopec, Josh Suereth](https://reader035.vdocuments.net/reader035/viewer/2022070408/56649e585503460f94b5156c/html5/thumbnails/1.jpg)
ScalaMeterScalaMeterPerformance regression testing Performance regression testing
frameworkframework
Aleksandar Prokopec, Josh SuerethAleksandar Prokopec, Josh Suereth
![Page 2: ScalaMeter Performance regression testing framework Aleksandar Prokopec, Josh Suereth](https://reader035.vdocuments.net/reader035/viewer/2022070408/56649e585503460f94b5156c/html5/thumbnails/2.jpg)
GoalGoal
List(1 to 100000: _*).map(x => x * x)
26 ms
![Page 3: ScalaMeter Performance regression testing framework Aleksandar Prokopec, Josh Suereth](https://reader035.vdocuments.net/reader035/viewer/2022070408/56649e585503460f94b5156c/html5/thumbnails/3.jpg)
GoalGoal
List(1 to 100000: _*).map(x => x * x)
class List[+T]extends Seq[T] {
// implementation 1
}
26 ms
![Page 4: ScalaMeter Performance regression testing framework Aleksandar Prokopec, Josh Suereth](https://reader035.vdocuments.net/reader035/viewer/2022070408/56649e585503460f94b5156c/html5/thumbnails/4.jpg)
GoalGoal
List(1 to 100000: _*).map(x => x * x)
class List[+T]extends Seq[T] {
// implementation 1
}
26 ms
![Page 5: ScalaMeter Performance regression testing framework Aleksandar Prokopec, Josh Suereth](https://reader035.vdocuments.net/reader035/viewer/2022070408/56649e585503460f94b5156c/html5/thumbnails/5.jpg)
GoalGoal
List(1 to 100000: _*).map(x => x * x)
class List[+T]extends Seq[T] {
// implementation 2
}
49 ms
![Page 6: ScalaMeter Performance regression testing framework Aleksandar Prokopec, Josh Suereth](https://reader035.vdocuments.net/reader035/viewer/2022070408/56649e585503460f94b5156c/html5/thumbnails/6.jpg)
First exampleFirst example
def measure() { val buffer = mutable.ArrayBuffer(0 until 2000000: _*) val start = System.currentTimeMillis() var sum = 0 buffer.foreach(sum += _) val end = System.currentTimeMillis() println(end - start) }
![Page 7: ScalaMeter Performance regression testing framework Aleksandar Prokopec, Josh Suereth](https://reader035.vdocuments.net/reader035/viewer/2022070408/56649e585503460f94b5156c/html5/thumbnails/7.jpg)
First exampleFirst example
def measure() { val buffer = mutable.ArrayBuffer(0 until 2000000: _*) val start = System.currentTimeMillis() var sum = 0 buffer.foreach(sum += _) val end = System.currentTimeMillis() println(end - start) }
![Page 8: ScalaMeter Performance regression testing framework Aleksandar Prokopec, Josh Suereth](https://reader035.vdocuments.net/reader035/viewer/2022070408/56649e585503460f94b5156c/html5/thumbnails/8.jpg)
First exampleFirst example
def measure() { val buffer = mutable.ArrayBuffer(0 until 2000000: _*) val start = System.currentTimeMillis() var sum = 0 buffer.foreach(sum += _) val end = System.currentTimeMillis() println(end - start) }
![Page 9: ScalaMeter Performance regression testing framework Aleksandar Prokopec, Josh Suereth](https://reader035.vdocuments.net/reader035/viewer/2022070408/56649e585503460f94b5156c/html5/thumbnails/9.jpg)
First exampleFirst example
def measure() { val buffer = mutable.ArrayBuffer(0 until 2000000: _*) val start = System.currentTimeMillis() var sum = 0 buffer.foreach(sum += _) val end = System.currentTimeMillis() println(end - start) }
![Page 10: ScalaMeter Performance regression testing framework Aleksandar Prokopec, Josh Suereth](https://reader035.vdocuments.net/reader035/viewer/2022070408/56649e585503460f94b5156c/html5/thumbnails/10.jpg)
First exampleFirst example
def measure() { val buffer = mutable.ArrayBuffer(0 until 2000000: _*) val start = System.currentTimeMillis() var sum = 0 buffer.foreach(sum += _) val end = System.currentTimeMillis() println(end - start) }measure()
![Page 11: ScalaMeter Performance regression testing framework Aleksandar Prokopec, Josh Suereth](https://reader035.vdocuments.net/reader035/viewer/2022070408/56649e585503460f94b5156c/html5/thumbnails/11.jpg)
First exampleFirst example
def measure() { val buffer = mutable.ArrayBuffer(0 until 2000000: _*) val start = System.currentTimeMillis() var sum = 0 buffer.foreach(sum += _) val end = System.currentTimeMillis() println(end - start) }measure()
26 ms
![Page 12: ScalaMeter Performance regression testing framework Aleksandar Prokopec, Josh Suereth](https://reader035.vdocuments.net/reader035/viewer/2022070408/56649e585503460f94b5156c/html5/thumbnails/12.jpg)
The warmup problemThe warmup problem
def measure() { val buffer = mutable.ArrayBuffer(0 until 2000000: _*) val start = System.currentTimeMillis() var sum = 0 buffer.foreach(sum += _) val end = System.currentTimeMillis() println(end - start) }measure()measure()
26 ms, 11 ms
![Page 13: ScalaMeter Performance regression testing framework Aleksandar Prokopec, Josh Suereth](https://reader035.vdocuments.net/reader035/viewer/2022070408/56649e585503460f94b5156c/html5/thumbnails/13.jpg)
The warmup problemThe warmup problem
def measure() { val buffer = mutable.ArrayBuffer(0 until 2000000: _*) val start = System.currentTimeMillis() var sum = 0 buffer.foreach(sum += _) val end = System.currentTimeMillis() println(end - start) }measure()measure()
26 ms, 11 ms
Why?Mainly:- JIT compilation- dynamic optimization
![Page 14: ScalaMeter Performance regression testing framework Aleksandar Prokopec, Josh Suereth](https://reader035.vdocuments.net/reader035/viewer/2022070408/56649e585503460f94b5156c/html5/thumbnails/14.jpg)
The warmup problemThe warmup problem
def measure() { val buffer = mutable.ArrayBuffer(0 until 2000000: _*) val start = System.currentTimeMillis() var sum = 0 buffer.foreach(sum += _) val end = System.currentTimeMillis() println(end - start) }
45 ms, 10 ms26 ms, 11 ms
![Page 15: ScalaMeter Performance regression testing framework Aleksandar Prokopec, Josh Suereth](https://reader035.vdocuments.net/reader035/viewer/2022070408/56649e585503460f94b5156c/html5/thumbnails/15.jpg)
The warmup problemThe warmup problem
def measure2() { val buffer = mutable.ArrayBuffer(0 until 4000000: _*) val start = System.currentTimeMillis() buffer.map(_ + 1) val end = System.currentTimeMillis() println(end - start) }
![Page 16: ScalaMeter Performance regression testing framework Aleksandar Prokopec, Josh Suereth](https://reader035.vdocuments.net/reader035/viewer/2022070408/56649e585503460f94b5156c/html5/thumbnails/16.jpg)
The warmup problemThe warmup problem
def measure2() { val buffer = mutable.ArrayBuffer(0 until 4000000: _*) val start = System.currentTimeMillis() buffer.map(_ + 1) val end = System.currentTimeMillis() println(end - start) }
241, 238, 235, 236, 234
![Page 17: ScalaMeter Performance regression testing framework Aleksandar Prokopec, Josh Suereth](https://reader035.vdocuments.net/reader035/viewer/2022070408/56649e585503460f94b5156c/html5/thumbnails/17.jpg)
The warmup problemThe warmup problem
def measure2() { val buffer = mutable.ArrayBuffer(0 until 4000000: _*) val start = System.currentTimeMillis() buffer.map(_ + 1) val end = System.currentTimeMillis() println(end - start) }
241, 238, 235, 236, 234, 429
![Page 18: ScalaMeter Performance regression testing framework Aleksandar Prokopec, Josh Suereth](https://reader035.vdocuments.net/reader035/viewer/2022070408/56649e585503460f94b5156c/html5/thumbnails/18.jpg)
The warmup problemThe warmup problem
def measure2() { val buffer = mutable.ArrayBuffer(0 until 4000000: _*) val start = System.currentTimeMillis() buffer.map(_ + 1) val end = System.currentTimeMillis() println(end - start) }
241, 238, 235, 236, 234, 429, 209
![Page 19: ScalaMeter Performance regression testing framework Aleksandar Prokopec, Josh Suereth](https://reader035.vdocuments.net/reader035/viewer/2022070408/56649e585503460f94b5156c/html5/thumbnails/19.jpg)
The warmup problemThe warmup problem
def measure2() { val buffer = mutable.ArrayBuffer(0 until 4000000: _*) val start = System.currentTimeMillis() buffer.map(_ + 1) val end = System.currentTimeMillis() println(end - start) }
241, 238, 235, 236, 234, 429, 209, 194, 195, 195
![Page 20: ScalaMeter Performance regression testing framework Aleksandar Prokopec, Josh Suereth](https://reader035.vdocuments.net/reader035/viewer/2022070408/56649e585503460f94b5156c/html5/thumbnails/20.jpg)
The warmup problemThe warmup problem
Bottomline: benchmark has to be repeated until the running time becomes “stable”.
The number of repetitions is not known in advance.
241, 238, 235, 236, 234, 429, 209, 194, 195, 195
![Page 21: ScalaMeter Performance regression testing framework Aleksandar Prokopec, Josh Suereth](https://reader035.vdocuments.net/reader035/viewer/2022070408/56649e585503460f94b5156c/html5/thumbnails/21.jpg)
Warming up the JVMWarming up the JVM
241, 238, 235, 236, 234, 429, 209, 194, 195, 195,194, 194, 193, 194, 196, 195, 195
Can this be automated?
Idea: measure variance of the running times.When it becomes sufficiently small, the test has stabilized.
![Page 22: ScalaMeter Performance regression testing framework Aleksandar Prokopec, Josh Suereth](https://reader035.vdocuments.net/reader035/viewer/2022070408/56649e585503460f94b5156c/html5/thumbnails/22.jpg)
The interference The interference problemproblem
val buffer = ArrayBuffer(0 until 900000: _*)buffer.map(_ + 1)
val buffer = ListBuffer(0 until 900000: _*)buffer.map(_ + 1)
![Page 23: ScalaMeter Performance regression testing framework Aleksandar Prokopec, Josh Suereth](https://reader035.vdocuments.net/reader035/viewer/2022070408/56649e585503460f94b5156c/html5/thumbnails/23.jpg)
The interference The interference problemproblem
Lets measure the first map 3 times with 7 repetitions:
val buffer = ArrayBuffer(0 until 900000: _*)buffer.map(_ + 1)
val buffer = ListBuffer(0 until 900000: _*)buffer.map(_ + 1)
61, 54, 54, 54, 55, 55, 56
186, 54, 54, 54, 55, 54, 53
54, 54, 53, 53, 53, 54, 51
![Page 24: ScalaMeter Performance regression testing framework Aleksandar Prokopec, Josh Suereth](https://reader035.vdocuments.net/reader035/viewer/2022070408/56649e585503460f94b5156c/html5/thumbnails/24.jpg)
The interference The interference problemproblem
val buffer = ArrayBuffer(0 until 900000: _*)buffer.map(_ + 1)
val buffer = ListBuffer(0 until 900000: _*)buffer.map(_ + 1)Now, lets measure the list buffer map in between:
61, 54, 54, 54, 55, 55, 56
186, 54, 54, 54, 55, 54, 53
54, 54, 53, 53, 53, 54, 51
59, 54, 54, 54, 54, 54, 5444, 36, 36, 36, 35, 36, 3645, 45, 45, 45, 44, 46, 4518, 17, 18, 18, 17, 292, 1645, 45, 44, 44, 45, 45, 44
![Page 25: ScalaMeter Performance regression testing framework Aleksandar Prokopec, Josh Suereth](https://reader035.vdocuments.net/reader035/viewer/2022070408/56649e585503460f94b5156c/html5/thumbnails/25.jpg)
The interference The interference problemproblem
val buffer = ArrayBuffer(0 until 900000: _*)buffer.map(_ + 1)
val buffer = ListBuffer(0 until 900000: _*)buffer.map(_ + 1)Now, lets measure the list buffer map in between:
61, 54, 54, 54, 55, 55, 56
186, 54, 54, 54, 55, 54, 53
54, 54, 53, 53, 53, 54, 51
59, 54, 54, 54, 54, 54, 5444, 36, 36, 36, 35, 36, 3645, 45, 45, 45, 44, 46, 4518, 17, 18, 18, 17, 292, 1645, 45, 44, 44, 45, 45, 44
![Page 26: ScalaMeter Performance regression testing framework Aleksandar Prokopec, Josh Suereth](https://reader035.vdocuments.net/reader035/viewer/2022070408/56649e585503460f94b5156c/html5/thumbnails/26.jpg)
Using separate JVMUsing separate JVM
Bottomline: always run the tests in a new JVM.
![Page 27: ScalaMeter Performance regression testing framework Aleksandar Prokopec, Josh Suereth](https://reader035.vdocuments.net/reader035/viewer/2022070408/56649e585503460f94b5156c/html5/thumbnails/27.jpg)
Using separate JVMUsing separate JVM
Bottomline: always run the tests in a new JVM.
This may not reflect a real-world scenario, but it gives a good idea of how different several alternatives are.
![Page 28: ScalaMeter Performance regression testing framework Aleksandar Prokopec, Josh Suereth](https://reader035.vdocuments.net/reader035/viewer/2022070408/56649e585503460f94b5156c/html5/thumbnails/28.jpg)
Using separate JVMUsing separate JVM
Bottomline: always run the tests in a new JVM.
It results in a reproducible, more stable measurement.
![Page 29: ScalaMeter Performance regression testing framework Aleksandar Prokopec, Josh Suereth](https://reader035.vdocuments.net/reader035/viewer/2022070408/56649e585503460f94b5156c/html5/thumbnails/29.jpg)
The List.map exampleThe List.map example
val list = (0 until 2500000).toListlist.map(_ % 2 == 0)
![Page 30: ScalaMeter Performance regression testing framework Aleksandar Prokopec, Josh Suereth](https://reader035.vdocuments.net/reader035/viewer/2022070408/56649e585503460f94b5156c/html5/thumbnails/30.jpg)
The List.map exampleThe List.map example
val list = (0 until 2500000).toListlist.map(_ % 2 == 0)
37, 38, 37, 1175, 38, 37, 37, 37, 37, …, 38, 37, 37, 37, 37, 465, 35, 35, …
![Page 31: ScalaMeter Performance regression testing framework Aleksandar Prokopec, Josh Suereth](https://reader035.vdocuments.net/reader035/viewer/2022070408/56649e585503460f94b5156c/html5/thumbnails/31.jpg)
The garbage collection The garbage collection problemproblem
val list = (0 until 2500000).toListlist.map(_ % 2 == 0)
37, 38, 37, 1175, 38, 37, 37, 37, 37, …, 38, 37, 37, 37, 37, 465, 35, 35, …
This benchmark triggers GC cycles!
![Page 32: ScalaMeter Performance regression testing framework Aleksandar Prokopec, Josh Suereth](https://reader035.vdocuments.net/reader035/viewer/2022070408/56649e585503460f94b5156c/html5/thumbnails/32.jpg)
The garbage collection The garbage collection problemproblem
val list = (0 until 2500000).toListlist.map(_ % 2 == 0)
37, 38, 37, 1175, 38, 37, 37, 37, 37, …, 38, 37, 37, 37, 37, 465, 35, 35, … -> mean: 47 ms
This benchmark triggers GC cycles!
![Page 33: ScalaMeter Performance regression testing framework Aleksandar Prokopec, Josh Suereth](https://reader035.vdocuments.net/reader035/viewer/2022070408/56649e585503460f94b5156c/html5/thumbnails/33.jpg)
The garbage collection The garbage collection problemproblem
val list = (0 until 2500000).toListlist.map(_ % 2 == 0)
37, 38, 37, 1175, 38, 37, 37, 37, 37, …, 38, 37, 37, 37, 37, 465, 35, 35, … -> mean: 47 ms
This benchmark triggers GC cycles!
37, 37, 37, 647, 37, 36, 38, 37, 36, …, 36, 37, 36, 37, 36, 37, 534, 36, 33, … -> mean: 39 ms
![Page 34: ScalaMeter Performance regression testing framework Aleksandar Prokopec, Josh Suereth](https://reader035.vdocuments.net/reader035/viewer/2022070408/56649e585503460f94b5156c/html5/thumbnails/34.jpg)
The garbage collection The garbage collection problemproblem
val list = (0 until 2500000).toListlist.map(_ % 2 == 0)
37, 38, 37, 1175, 38, 37, 37, 37, 37, …, 38, 37, 37, 37, 37, 465, 35, 35, … -> mean: 47 ms
This benchmark triggers GC cycles!
37, 37, 37, 647, 37, 36, 38, 37, 36, …, 36, 37, 36, 37, 36, 37, 534, 36, 33, … -> mean: 39 ms
![Page 35: ScalaMeter Performance regression testing framework Aleksandar Prokopec, Josh Suereth](https://reader035.vdocuments.net/reader035/viewer/2022070408/56649e585503460f94b5156c/html5/thumbnails/35.jpg)
The garbage collection The garbage collection problemproblem
val list = (0 until 2500000).toListlist.map(_ % 2 == 0)
Solutions:-repeat A LOT of times –an accurate mean, but takes A LONG time
![Page 36: ScalaMeter Performance regression testing framework Aleksandar Prokopec, Josh Suereth](https://reader035.vdocuments.net/reader035/viewer/2022070408/56649e585503460f94b5156c/html5/thumbnails/36.jpg)
The garbage collection The garbage collection problemproblem
val list = (0 until 2500000).toListlist.map(_ % 2 == 0)
Solutions:-repeat A LOT of times –an accurate mean, but takes A LONG time-ignore the measurements with GC – gives a reproducible value, and less measurements
![Page 37: ScalaMeter Performance regression testing framework Aleksandar Prokopec, Josh Suereth](https://reader035.vdocuments.net/reader035/viewer/2022070408/56649e585503460f94b5156c/html5/thumbnails/37.jpg)
The garbage collection The garbage collection problemproblem
val list = (0 until 2500000).toListlist.map(_ % 2 == 0)
Solutions:-repeat A LOT of times –an accurate mean, but takes A LONG time-ignore the measurements with GC – gives a reproducible value, and less measurements
- how to do this?
![Page 38: ScalaMeter Performance regression testing framework Aleksandar Prokopec, Josh Suereth](https://reader035.vdocuments.net/reader035/viewer/2022070408/56649e585503460f94b5156c/html5/thumbnails/38.jpg)
The garbage collection The garbage collection problemproblem
val list = (0 until 2500000).toListlist.map(_ % 2 == 0)
- manually - verbose:gc
![Page 39: ScalaMeter Performance regression testing framework Aleksandar Prokopec, Josh Suereth](https://reader035.vdocuments.net/reader035/viewer/2022070408/56649e585503460f94b5156c/html5/thumbnails/39.jpg)
Automatic GC detectionAutomatic GC detection
val list = (0 until 2500000).toListlist.map(_ % 2 == 0)
- manually - verbose:gc- automatically using callbacks in JDK7
37, 37, 37, 647, 37, 36, 38, 37, 36, …, 36, 37, 36, 37, 36, 37, 534, 36, 33, …
![Page 40: ScalaMeter Performance regression testing framework Aleksandar Prokopec, Josh Suereth](https://reader035.vdocuments.net/reader035/viewer/2022070408/56649e585503460f94b5156c/html5/thumbnails/40.jpg)
Automatic GC detectionAutomatic GC detection
val list = (0 until 2500000).toListlist.map(_ % 2 == 0)
- manually - verbose:gc- automatically using callbacks in JDK7
37, 37, 37, 647, 37, 36, 38, 37, 36, …, 36, 37, 36, 37, 36, 37, 534, 36, 33, …
raises a GC event
![Page 41: ScalaMeter Performance regression testing framework Aleksandar Prokopec, Josh Suereth](https://reader035.vdocuments.net/reader035/viewer/2022070408/56649e585503460f94b5156c/html5/thumbnails/41.jpg)
The runtime problemThe runtime problem
- there are other runtime events beside GC – e.g. JIT compilation, dynamic optimization, etc.
- these take time, but cannot be determined accurately
![Page 42: ScalaMeter Performance regression testing framework Aleksandar Prokopec, Josh Suereth](https://reader035.vdocuments.net/reader035/viewer/2022070408/56649e585503460f94b5156c/html5/thumbnails/42.jpg)
The runtime problemThe runtime problem
- there are other runtime events beside GC – e.g. JIT compilation, dynamic optimization, etc.
- these take time, but cannot be determined accurately- heap state also influences memory allocation patterns
and performance
![Page 43: ScalaMeter Performance regression testing framework Aleksandar Prokopec, Josh Suereth](https://reader035.vdocuments.net/reader035/viewer/2022070408/56649e585503460f94b5156c/html5/thumbnails/43.jpg)
The runtime problemThe runtime problem
- there are other runtime events beside GC – e.g. JIT compilation, dynamic optimization, etc.
- these take time, but cannot be determined accurately- heap state also influences memory allocation patterns
and performance
val list = (0 until 4000000).toListlist.groupBy(_ % 10)
(allocation intensive)
![Page 44: ScalaMeter Performance regression testing framework Aleksandar Prokopec, Josh Suereth](https://reader035.vdocuments.net/reader035/viewer/2022070408/56649e585503460f94b5156c/html5/thumbnails/44.jpg)
The runtime problemThe runtime problem
- there are other runtime events beside GC – e.g. JIT compilation, dynamic optimization, etc.
- these take time, but cannot be determined accurately- heap state also influences memory allocation patterns
and performance
val list = (0 until 4000000).toListlist.groupBy(_ % 10)
120, 121, 122, 118, 123, 794, 109, 111, 115, 113, 110
![Page 45: ScalaMeter Performance regression testing framework Aleksandar Prokopec, Josh Suereth](https://reader035.vdocuments.net/reader035/viewer/2022070408/56649e585503460f94b5156c/html5/thumbnails/45.jpg)
The runtime problemThe runtime problem
- there are other runtime events beside GC – e.g. JIT compilation, dynamic optimization, etc.
- these take time, but cannot be determined accurately- heap state also influences memory allocation patterns
and performance
val list = (0 until 4000000).toListlist.groupBy(_ % 10)
120, 121, 122, 118, 123, 794, 109, 111, 115, 113, 110
affects the mean – 116 ms vs 178 ms
![Page 46: ScalaMeter Performance regression testing framework Aleksandar Prokopec, Josh Suereth](https://reader035.vdocuments.net/reader035/viewer/2022070408/56649e585503460f94b5156c/html5/thumbnails/46.jpg)
Outlier eliminationOutlier elimination
120, 121, 122, 118, 123, 794, 109, 111, 115, 113, 110
![Page 47: ScalaMeter Performance regression testing framework Aleksandar Prokopec, Josh Suereth](https://reader035.vdocuments.net/reader035/viewer/2022070408/56649e585503460f94b5156c/html5/thumbnails/47.jpg)
Outlier eliminationOutlier elimination
120, 121, 122, 118, 123, 794, 109, 111, 115, 113, 110
109, 110, 111, 113, 115, 118, 120, 121, 122, 123, 794
sort
![Page 48: ScalaMeter Performance regression testing framework Aleksandar Prokopec, Josh Suereth](https://reader035.vdocuments.net/reader035/viewer/2022070408/56649e585503460f94b5156c/html5/thumbnails/48.jpg)
Outlier eliminationOutlier elimination
120, 121, 122, 118, 123, 794, 109, 111, 115, 113, 110
109, 110, 111, 113, 115, 118, 120, 121, 122, 123, 794
sort
inspect tail and its variance contribution
109, 110, 111, 113, 115, 118, 120, 121, 122, 123
![Page 49: ScalaMeter Performance regression testing framework Aleksandar Prokopec, Josh Suereth](https://reader035.vdocuments.net/reader035/viewer/2022070408/56649e585503460f94b5156c/html5/thumbnails/49.jpg)
Outlier eliminationOutlier elimination
120, 121, 122, 118, 123, 794, 109, 111, 115, 113, 110
109, 110, 111, 113, 115, 118, 120, 121, 122, 123, 794
sort
inspect tail and its variance contribution
109, 110, 111, 113, 115, 118, 120, 121, 122, 123
109, 110, 111, 113, 115, 118, 120, 121, 122, 123, 124
redo the measurement
![Page 50: ScalaMeter Performance regression testing framework Aleksandar Prokopec, Josh Suereth](https://reader035.vdocuments.net/reader035/viewer/2022070408/56649e585503460f94b5156c/html5/thumbnails/50.jpg)
Doing all this manuallyDoing all this manually
![Page 51: ScalaMeter Performance regression testing framework Aleksandar Prokopec, Josh Suereth](https://reader035.vdocuments.net/reader035/viewer/2022070408/56649e585503460f94b5156c/html5/thumbnails/51.jpg)
ScalaMeterScalaMeter
Does all this analysis automatically, highly configurable.
Plus, it detects performance regressions.
And generates reports.
![Page 52: ScalaMeter Performance regression testing framework Aleksandar Prokopec, Josh Suereth](https://reader035.vdocuments.net/reader035/viewer/2022070408/56649e585503460f94b5156c/html5/thumbnails/52.jpg)
ScalaMeter exampleScalaMeter example
object ListTest extends PerformanceTest.Microbenchmark {
A range of predefined benchmark types
![Page 53: ScalaMeter Performance regression testing framework Aleksandar Prokopec, Josh Suereth](https://reader035.vdocuments.net/reader035/viewer/2022070408/56649e585503460f94b5156c/html5/thumbnails/53.jpg)
ScalaMeter exampleScalaMeter example
object ListTest extends PerformanceTest.Microbenchmark { val sizes = Gen.range("size”)(500000, 1000000, 100000)
Generators provide input data for tests
![Page 54: ScalaMeter Performance regression testing framework Aleksandar Prokopec, Josh Suereth](https://reader035.vdocuments.net/reader035/viewer/2022070408/56649e585503460f94b5156c/html5/thumbnails/54.jpg)
ScalaMeter exampleScalaMeter example
object ListTest extends PerformanceTest.Microbenchmark { val sizes = Gen.range("size”)(500000, 1000000, 100000) val lists = for (sz <- sizes) yield (0 until sz).toList
Generators can be composed a la ScalaCheck
![Page 55: ScalaMeter Performance regression testing framework Aleksandar Prokopec, Josh Suereth](https://reader035.vdocuments.net/reader035/viewer/2022070408/56649e585503460f94b5156c/html5/thumbnails/55.jpg)
ScalaMeter exampleScalaMeter example
object ListTest extends PerformanceTest.Microbenchmark { val sizes = Gen.range("size”)(500000, 1000000, 100000) val lists = for (sz <- sizes) yield (0 until sz).toList
using(lists) in { xs => xs.groupBy(_ % 10) }}
Concise syntax to specify and group tests
![Page 56: ScalaMeter Performance regression testing framework Aleksandar Prokopec, Josh Suereth](https://reader035.vdocuments.net/reader035/viewer/2022070408/56649e585503460f94b5156c/html5/thumbnails/56.jpg)
ScalaMeter exampleScalaMeter example
object ListTest extends PerformanceTest.Microbenchmark { val sizes = Gen.range("size”)(500000, 1000000, 100000) val lists = for (sz <- sizes) yield (0 until sz).toList
measure method “groupBy” in { using(lists) in { xs => xs.groupBy(_ % 10) } using(ranges) in { xs => xs.groupBy(_ % 10) } }}
![Page 57: ScalaMeter Performance regression testing framework Aleksandar Prokopec, Josh Suereth](https://reader035.vdocuments.net/reader035/viewer/2022070408/56649e585503460f94b5156c/html5/thumbnails/57.jpg)
Automatic regression Automatic regression testingtesting
using(lists) in { xs => var sum = 0 xs.foreach(x => sum += x) }
![Page 58: ScalaMeter Performance regression testing framework Aleksandar Prokopec, Josh Suereth](https://reader035.vdocuments.net/reader035/viewer/2022070408/56649e585503460f94b5156c/html5/thumbnails/58.jpg)
Automatic regression Automatic regression testingtesting
using(lists) in { xs => var sum = 0 xs.foreach(x => sum += x) }
[info] Test group: foreach[info] - foreach.Test-0 measurements:[info] - at size -> 2000000, 1 alternatives: passed[info] (ci = <7.28, 8.22>, significance = 1.0E-10)
![Page 59: ScalaMeter Performance regression testing framework Aleksandar Prokopec, Josh Suereth](https://reader035.vdocuments.net/reader035/viewer/2022070408/56649e585503460f94b5156c/html5/thumbnails/59.jpg)
Automatic regression Automatic regression testingtesting
using(lists) in { xs => var sum = 0 xs.foreach(x => sum += math.sqrt(x)) }
![Page 60: ScalaMeter Performance regression testing framework Aleksandar Prokopec, Josh Suereth](https://reader035.vdocuments.net/reader035/viewer/2022070408/56649e585503460f94b5156c/html5/thumbnails/60.jpg)
Automatic regression Automatic regression testingtesting
using(lists) in { xs => var sum = 0 xs.foreach(x => sum += math.sqrt(x)) }
[info] Test group: foreach[info] - foreach.Test-0 measurements:[info] - at size -> 2000000, 2 alternatives: failed[info] (ci = <14.57, 15.38>, significance = 1.0E-10)[error] Failed confidence interval test: <-7.85, -6.60> [error] Previous (mean = 7.75, stdev = 0.44, ci = <7.28, 8.22>) [error] Latest (mean = 14.97, stdev = 0.38, ci = <14.57, 15.38>)
![Page 61: ScalaMeter Performance regression testing framework Aleksandar Prokopec, Josh Suereth](https://reader035.vdocuments.net/reader035/viewer/2022070408/56649e585503460f94b5156c/html5/thumbnails/61.jpg)
Automatic regression Automatic regression testingtesting
- configurable: ANOVA (analysis of variance) or confidence interval testing
- can apply noise to make unstable tests more solid- various policies on keeping the result history
![Page 62: ScalaMeter Performance regression testing framework Aleksandar Prokopec, Josh Suereth](https://reader035.vdocuments.net/reader035/viewer/2022070408/56649e585503460f94b5156c/html5/thumbnails/62.jpg)
Report generationReport generation
![Page 63: ScalaMeter Performance regression testing framework Aleksandar Prokopec, Josh Suereth](https://reader035.vdocuments.net/reader035/viewer/2022070408/56649e585503460f94b5156c/html5/thumbnails/63.jpg)
Tutorials online!Tutorials online!
http://axel22.github.com/scalameter
Questions?