timelapse interactive record/replay for web apps
DESCRIPTION
Timelapse Interactive record/replay for web apps. Brian Burg. Richard J. Bailey. Andrew J. Ko. Michael D. Ernst. Computer Science and Engineering University of Washington. “It’s hard to debug failures in the field”. Distributed across time and space H ardware and software variation - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Timelapse Interactive record/replay for web apps](https://reader036.vdocuments.net/reader036/viewer/2022062305/5681620d550346895dd239d0/html5/thumbnails/1.jpg)
1
Timelapse Interactive record/replay for web
apps
Brian Burg Andrew J. Ko Michael D. Ernst
Computer Science and EngineeringUniversity of Washington
Richard J. Bailey
![Page 2: Timelapse Interactive record/replay for web apps](https://reader036.vdocuments.net/reader036/viewer/2022062305/5681620d550346895dd239d0/html5/thumbnails/2.jpg)
2
“It’s hard to debug failures in the field”
Distributed across time and space
Hardware and software variation
Users are not software testers
![Page 3: Timelapse Interactive record/replay for web apps](https://reader036.vdocuments.net/reader036/viewer/2022062305/5681620d550346895dd239d0/html5/thumbnails/3.jpg)
3
Users can’t report failures accurately
Works for me..
“a piece doesn’t rotate
properly!!”end-user encounters bug in production code
files bug report with
ad-hoc information
developer unable to reproduce the
bug
![Page 4: Timelapse Interactive record/replay for web apps](https://reader036.vdocuments.net/reader036/viewer/2022062305/5681620d550346895dd239d0/html5/thumbnails/4.jpg)
4
Users can’t report failures accurately
“The most severe problems were errors in steps to reproduce and incomplete information.”
“What makes a good bug report”. Zimmerman et al. TSE Vol. 36, No. 5. 2010.
“a piece doesn’t rotate
properly!!”
Works for me..
![Page 5: Timelapse Interactive record/replay for web apps](https://reader036.vdocuments.net/reader036/viewer/2022062305/5681620d550346895dd239d0/html5/thumbnails/5.jpg)
5
Users can’t report failures accurately
Bug reporters and developers want better tool support for reproducing buggy behavior.“What makes a good bug report”. Zimmerman et al. TSE Vol. 36, No. 5.
2010.
“a piece doesn’t rotate
properly!!”
Works for me..
![Page 6: Timelapse Interactive record/replay for web apps](https://reader036.vdocuments.net/reader036/viewer/2022062305/5681620d550346895dd239d0/html5/thumbnails/6.jpg)
6
Existing tools are imprecise and hard to use
Selenium/WebDriver
CoScripterLeshed et al, CHI 2008
Sikuli ScriptYeh et al, UIST 2009
Capture and simulate user input. Designed for test and task automation.
macro replay(CoScripter, Selenium, Sikuli)
![Page 7: Timelapse Interactive record/replay for web apps](https://reader036.vdocuments.net/reader036/viewer/2022062305/5681620d550346895dd239d0/html5/thumbnails/7.jpg)
7
Existing tools are imprecise and hard to use
macro replay(CoScripter, Selenium, Sikuli)
Capture and simulate user input. Designed for test and task automation.
deterministic replay(Mugshot, WaRR)
Nondeterministic. Requires extra setup ahead of time. Can’t use with a debugger.Save and reuse nondeterministic inputs to exactly recreate a specific execution.Play/pause buttons only. Slows down execution. Can’t use with a debugger.
![Page 8: Timelapse Interactive record/replay for web apps](https://reader036.vdocuments.net/reader036/viewer/2022062305/5681620d550346895dd239d0/html5/thumbnails/8.jpg)
8
Timelapse: a precise, fast, integrated replay tool
This talk:• An interface for capturing and replaying program behavior
• Techniques for cheap, precise record/replay in web browsers
• How developers use record/replay during debugging tasks
![Page 9: Timelapse Interactive record/replay for web apps](https://reader036.vdocuments.net/reader036/viewer/2022062305/5681620d550346895dd239d0/html5/thumbnails/9.jpg)
9
How to capture program behavior
![Page 10: Timelapse Interactive record/replay for web apps](https://reader036.vdocuments.net/reader036/viewer/2022062305/5681620d550346895dd239d0/html5/thumbnails/10.jpg)
10
How to navigate a recording
![Page 11: Timelapse Interactive record/replay for web apps](https://reader036.vdocuments.net/reader036/viewer/2022062305/5681620d550346895dd239d0/html5/thumbnails/11.jpg)
11
Using replay while debugging
![Page 12: Timelapse Interactive record/replay for web apps](https://reader036.vdocuments.net/reader036/viewer/2022062305/5681620d550346895dd239d0/html5/thumbnails/12.jpg)
12
Browsers interpret input, render output
OutputBrowser Input(User, Network, Timers)
Web Interpreter(WebKit, Gecko)
![Page 13: Timelapse Interactive record/replay for web apps](https://reader036.vdocuments.net/reader036/viewer/2022062305/5681620d550346895dd239d0/html5/thumbnails/13.jpg)
13
Timelapse captures a browser’s inputs
Output
Inputs Log
Web Interpreter(WebKit, Gecko)
Browser Input(User, Network, Timers)
![Page 14: Timelapse Interactive record/replay for web apps](https://reader036.vdocuments.net/reader036/viewer/2022062305/5681620d550346895dd239d0/html5/thumbnails/14.jpg)
14
Timelapse replays a browser’s inputs
Output
Inputs Log
Web Interpreter(WebKit, Gecko)
![Page 15: Timelapse Interactive record/replay for web apps](https://reader036.vdocuments.net/reader036/viewer/2022062305/5681620d550346895dd239d0/html5/thumbnails/15.jpg)
15
Browsers have layered architectures
User input,
commands
policy decisions
Date.now, win.colorDepth
event loop callbacks
Web Interpreter(WebKit, Gecko)
PlatformsEmbedders
(Firefox, Safari, Chrome)
![Page 16: Timelapse Interactive record/replay for web apps](https://reader036.vdocuments.net/reader036/viewer/2022062305/5681620d550346895dd239d0/html5/thumbnails/16.jpg)
16
Timelapse intercepts input at layer boundaries
Embedders(Firefox, Safari, Chrome)
Web Interpreter(WebKit, Gecko)
Platforms
![Page 17: Timelapse Interactive record/replay for web apps](https://reader036.vdocuments.net/reader036/viewer/2022062305/5681620d550346895dd239d0/html5/thumbnails/17.jpg)
17
• Hardware interrupts• Nondeterministic
instructions• Instruction counts
VM record/replay
• Async callbacks• Nondeterministic
APIs• DOM event counts
Browser record/replay
Inspired by VM record/replay
![Page 18: Timelapse Interactive record/replay for web apps](https://reader036.vdocuments.net/reader036/viewer/2022062305/5681620d550346895dd239d0/html5/thumbnails/18.jpg)
18
Memoizing nondeterministic APIs
During normal execution, Date.now() returns the current time.
>> foo(a,b) { return a + b; } >bar(c) { return “now:”+ Date.now(); }
/* file: Source/wtf/DateMath.h */
inline double jsCurrentTime(){return floor(WTF::currentTimeMS());}
Web Interpreter Platform API
![Page 19: Timelapse Interactive record/replay for web apps](https://reader036.vdocuments.net/reader036/viewer/2022062305/5681620d550346895dd239d0/html5/thumbnails/19.jpg)
19Inputs Log
Memoizing nondeterministic APIs
During recording, the return value of Date.now() is saved.
>> foo(a,b) { return a + b; } >bar(c) { return “now:”+ Date.now(); }
/* file: Source/wtf/DateMath.h */
inline double jsCurrentTime(){return floor(WTF::currentTimeMS());}
Web Interpreter Platform API
![Page 20: Timelapse Interactive record/replay for web apps](https://reader036.vdocuments.net/reader036/viewer/2022062305/5681620d550346895dd239d0/html5/thumbnails/20.jpg)
Memoizing nondeterministic APIs
On replay, the logged return value of Date.now() is used.
>> foo(a,b) { return a + b; } >bar(c) { return “now:”+ Date.now(); }
/* file: Source/wtf/DateMath.h */
inline double jsCurrentTime(){return floor(WTF::currentTimeMS());}
Inputs LogWeb Interpreter Platform API
![Page 21: Timelapse Interactive record/replay for web apps](https://reader036.vdocuments.net/reader036/viewer/2022062305/5681620d550346895dd239d0/html5/thumbnails/21.jpg)
Making callbacks deterministic
enqueue()
while (true) { var event = queue.pop(); this.dispatchToListeners(event); }
Event Loop
timerFired()
Callback executes
Callback registered
Problem: accurately capturing and
simulating event loop dispatches.
![Page 22: Timelapse Interactive record/replay for web apps](https://reader036.vdocuments.net/reader036/viewer/2022062305/5681620d550346895dd239d0/html5/thumbnails/22.jpg)
timer 42, 34 DOM events
Making callbacks deterministic
enqueue()
timerFired()
Callback executes
Callback registered while (true) {
var event = queue.pop(); this.dispatch(event); }
Inputs Log
Event Loop
![Page 23: Timelapse Interactive record/replay for web apps](https://reader036.vdocuments.net/reader036/viewer/2022062305/5681620d550346895dd239d0/html5/thumbnails/23.jpg)
Making callbacks deterministic
enqueue()
Callback registered while (true) {
var event = queue.pop(); this.dispatchToListeners(event); }
Event Loop
timerFired()
Callback executesInputs Log
34 DOM events!
...
...
...
![Page 24: Timelapse Interactive record/replay for web apps](https://reader036.vdocuments.net/reader036/viewer/2022062305/5681620d550346895dd239d0/html5/thumbnails/24.jpg)
24
Runtime overheads are acceptable
JS Ray
tracer
Space
Inva
ders
Mozilla
.org
CodeM
irror
Colorpi
cker
DuckDuck
Go0
0.2
0.4
0.6
0.8
1
1.2
1.4
BaselineRecording1× ReplaySeeking
Run times(multiple of base-
line)
![Page 25: Timelapse Interactive record/replay for web apps](https://reader036.vdocuments.net/reader036/viewer/2022062305/5681620d550346895dd239d0/html5/thumbnails/25.jpg)
25
Recordings are small and compressible
JSLinu
x
JS Ray
tracer
Space
Inva
ders
Mozilla.
org
CodeM
irror
Colorpi
cker
DuckDuck
Go0
100200300400500600700800 In-memory
SerializedCom-pressed
Size (KB)
Siterecordin
g duration
(s)
resources on page
(KB)
log growth
(KB/sec)
JSLinux 10.5 4500 0.8
JS Raytracer 6.3 5.9 1.6Space
Invaders 25.8 712 2.2
Mozilla.org 22.3 2800 1.3
CodeMirror 16.6 168 1.0
Colorpicker 15.3 577 1.7DuckDuckG
o 14.1 1900 2.1
![Page 26: Timelapse Interactive record/replay for web apps](https://reader036.vdocuments.net/reader036/viewer/2022062305/5681620d550346895dd239d0/html5/thumbnails/26.jpg)
26
Page resources dominate recording size
JSLinu
x
Space
Inva
ders
CodeM
irror
DuckDuck
Go0
500100015002000250030003500400045005000 In-memory
SerializedCom-pressedResources
Size (KB)
Siterecordin
g duration
(s)
resources on page
(KB)
log growth
(KB/sec)
JSLinux 10.5 4500 0.8
JS Raytracer 6.3 5.9 1.6Space
Invaders 25.8 712 2.2
Mozilla.org 22.3 2800 1.3
CodeMirror 16.6 168 1.0
Colorpicker 15.3 577 1.7DuckDuckG
o 14.1 1900 2.1
![Page 27: Timelapse Interactive record/replay for web apps](https://reader036.vdocuments.net/reader036/viewer/2022062305/5681620d550346895dd239d0/html5/thumbnails/27.jpg)
27
How would developers use it?
Study Design20+ developers with industry experiencewithin-subjects, 2 tasks per person, 45 minutes per task, 4 treatments
Performance
RQ: changes to frequency/duration?
Reproduction
RQ: complete tasks more quickly? more successfully? Who, why?
![Page 28: Timelapse Interactive record/replay for web apps](https://reader036.vdocuments.net/reader036/viewer/2022062305/5681620d550346895dd239d0/html5/thumbnails/28.jpg)
28
How did developers use it?
Study Design20+ developers with industry experiencewithin-subjects, 2 tasks per person, 45 minutes per task, 4 treatments
Performance
Shorter and more frequent repro actions;Time spent unchanged (max. 25%; avg.15%)Successful developers quickly integrated replay into their existing workflows.Unsuccessful developers who used opportunistic strategies were distracted.
Reproduction
![Page 29: Timelapse Interactive record/replay for web apps](https://reader036.vdocuments.net/reader036/viewer/2022062305/5681620d550346895dd239d0/html5/thumbnails/29.jpg)
Current & Future Work
Visualizations Interaction histories aid navigation, but not program understanding.
Passive capturing
Precision and low overhead don’t matter if you forget to start capturing.Post-hoc analysis Developers can gather more runtime data without reproducing behavior:
Post-hoc logging, Post-hoc Whyline,Post-hoc SeeSS, Testcase extraction
![Page 30: Timelapse Interactive record/replay for web apps](https://reader036.vdocuments.net/reader036/viewer/2022062305/5681620d550346895dd239d0/html5/thumbnails/30.jpg)
Conclusion
Visualizations Interaction histories supported–but didn’t reduce–reproduction of program state.
Infrastructure Replay infrastructure enables new research, tools and workflows.
Record/Replay Virtual machine replay techniques work well when applied to web applications.
github.com/burg/timelapse
![Page 31: Timelapse Interactive record/replay for web apps](https://reader036.vdocuments.net/reader036/viewer/2022062305/5681620d550346895dd239d0/html5/thumbnails/31.jpg)
31
Replay fidelity and completeness
Web interpreters expose a large and ever-changing API.Timelapse doesn’t tame all sources of nondeterminism (yet).Excepting untamed sources, the DOM tree and JavaScript heap are identical for all recorded and replayed executions.Possible divergence is automatically detected when:• DOM event counts differ on capture and replay• Memoized inputs are overused or unused• Network request details differ unexpectedly• Known-bad APIs are used by client code
Divergence detection supports piecewise implementation.
![Page 32: Timelapse Interactive record/replay for web apps](https://reader036.vdocuments.net/reader036/viewer/2022062305/5681620d550346895dd239d0/html5/thumbnails/32.jpg)
32
Interpreter inputs by sourceUser: mouse, keyboard, scroll, resizeNetwork: images, scripts, HTML, AJAXCommands: page navigation
Internal nondeterminism:Animations, transitions, multimedia,async script and parser yields
Functions: Date.now, Math.random, etcCaching: resources, cookiesTimers: timer schedule
![Page 33: Timelapse Interactive record/replay for web apps](https://reader036.vdocuments.net/reader036/viewer/2022062305/5681620d550346895dd239d0/html5/thumbnails/33.jpg)
33
Shim: the thing in the middle
Shims are used to implement deterministic record/replay.
The hard part of implementing record/replay is designing and placing shims.
![Page 34: Timelapse Interactive record/replay for web apps](https://reader036.vdocuments.net/reader036/viewer/2022062305/5681620d550346895dd239d0/html5/thumbnails/34.jpg)
34
Embedding and platform APIs
Embedders
EMBE
DD
ING
AP
I
PLAT
FORM
AP
I
Web Interpreter(WebKit, Gecko)
Abstraction layers separate web interpreters from platforms/embedders.
Platforms
![Page 35: Timelapse Interactive record/replay for web apps](https://reader036.vdocuments.net/reader036/viewer/2022062305/5681620d550346895dd239d0/html5/thumbnails/35.jpg)
35
Embedding and platform APIs
Embedders
EMBE
DD
ING
AP
I
PLAT
FORM
AP
I
Web Interpreter(WebKit, Gecko)
Abstraction layers separate web interpreters from platforms/embedders.
Platforms
![Page 36: Timelapse Interactive record/replay for web apps](https://reader036.vdocuments.net/reader036/viewer/2022062305/5681620d550346895dd239d0/html5/thumbnails/36.jpg)
36
Embedding and platform APIs
Embedders
Web Interpreter(WebKit, Gecko)
Platforms
EMBE
DD
ING
AP
I
PLAT
FORM
AP
I
Shims sit between the web interpreter and abstraction layers.
![Page 37: Timelapse Interactive record/replay for web apps](https://reader036.vdocuments.net/reader036/viewer/2022062305/5681620d550346895dd239d0/html5/thumbnails/37.jpg)
37
Embedding and platform APIs
Embedders
Web Interpreter(WebKit, Gecko)
Platforms
EMBE
DD
ING
AP
I
PLAT
FORM
AP
I
Shims sit between the web interpreter and abstraction layers.