![Page 1: Identifying Hotspots in Software Build Processes](https://reader033.vdocuments.net/reader033/viewer/2022052909/5598a9d31a28ab64328b45e7/html5/thumbnails/1.jpg)
Identifying Hotspots in Software Build Processes
Shane McIntosh
Ahmed E. Hassan
Bram Adams
Meiyappan Nagappan
![Page 2: Identifying Hotspots in Software Build Processes](https://reader033.vdocuments.net/reader033/viewer/2022052909/5598a9d31a28ab64328b45e7/html5/thumbnails/2.jpg)
Source code
What is a build system?
!2
![Page 3: Identifying Hotspots in Software Build Processes](https://reader033.vdocuments.net/reader033/viewer/2022052909/5598a9d31a28ab64328b45e7/html5/thumbnails/3.jpg)
Source code
Deliverable
What is a build system?
!2
![Page 4: Identifying Hotspots in Software Build Processes](https://reader033.vdocuments.net/reader033/viewer/2022052909/5598a9d31a28ab64328b45e7/html5/thumbnails/4.jpg)
.tex
.c
.cc
.o
.o
.dvi
.a
.exe
.deb
Build systems describe how sources are translated into deliverables
!3
![Page 5: Identifying Hotspots in Software Build Processes](https://reader033.vdocuments.net/reader033/viewer/2022052909/5598a9d31a28ab64328b45e7/html5/thumbnails/5.jpg)
(4) Test
(1) Think
(2) Edit
(3) Build
The developer’s work cycle
All builds are equal…
!4
![Page 6: Identifying Hotspots in Software Build Processes](https://reader033.vdocuments.net/reader033/viewer/2022052909/5598a9d31a28ab64328b45e7/html5/thumbnails/6.jpg)
“
…But some builds are more equal than others
http://xkcd.com/303/
Why is the gtk build so abysmally slow? This hinders our ability to run gtk bots
and keep the gtk build green. :( ”!5
![Page 7: Identifying Hotspots in Software Build Processes](https://reader033.vdocuments.net/reader033/viewer/2022052909/5598a9d31a28ab64328b45e7/html5/thumbnails/7.jpg)
Refactoring can improve build performance
deliverable1 deliverable2
del1.o del2.olibrary.a
del1.c del2.cutil1.o util2.o
util1.c util2.c
!6
![Page 8: Identifying Hotspots in Software Build Processes](https://reader033.vdocuments.net/reader033/viewer/2022052909/5598a9d31a28ab64328b45e7/html5/thumbnails/8.jpg)
Refactoring can improve build performance
deliverable1 deliverable2
del1.o del2.olibrary.a
del1.c del2.cutil1.o util2.o
util1.c util2.c
!6
![Page 9: Identifying Hotspots in Software Build Processes](https://reader033.vdocuments.net/reader033/viewer/2022052909/5598a9d31a28ab64328b45e7/html5/thumbnails/9.jpg)
Refactoring can improve build performance
deliverable1 deliverable2
del1.o del2.olibrary.a
del1.c del2.cutil1.o util2.o
util1.c util2.c
!6
![Page 10: Identifying Hotspots in Software Build Processes](https://reader033.vdocuments.net/reader033/viewer/2022052909/5598a9d31a28ab64328b45e7/html5/thumbnails/10.jpg)
Refactoring can improve build performance
deliverable1 deliverable2
del1.o del2.olibrary.a
del1.c del2.cutil1.o util2.o
util1.c util2.c
!6
![Page 11: Identifying Hotspots in Software Build Processes](https://reader033.vdocuments.net/reader033/viewer/2022052909/5598a9d31a28ab64328b45e7/html5/thumbnails/11.jpg)
Refactoring can improve build performance
deliverable1 deliverable2
del1.o del2.olibrary.a
del1.c del2.cutil1.o util2.o
util1.c util2.c
!6
![Page 12: Identifying Hotspots in Software Build Processes](https://reader033.vdocuments.net/reader033/viewer/2022052909/5598a9d31a28ab64328b45e7/html5/thumbnails/12.jpg)
Refactoring can improve build performance
deliverable1 deliverable2
del1.o del2.olibrary.a
del1.c del2.cutil1.o util2.o
util1.c util2.c
!6
Before refactoring:!4 commands!
triggered
![Page 13: Identifying Hotspots in Software Build Processes](https://reader033.vdocuments.net/reader033/viewer/2022052909/5598a9d31a28ab64328b45e7/html5/thumbnails/13.jpg)
Refactoring can improve build performance
deliverable1 deliverable2
del1.o del2.olibrary.a
del1.c del2.cutil1.o util2.o
util1.c util2.c
!6
![Page 14: Identifying Hotspots in Software Build Processes](https://reader033.vdocuments.net/reader033/viewer/2022052909/5598a9d31a28ab64328b45e7/html5/thumbnails/14.jpg)
Refactoring can improve build performance
deliverable1 deliverable2
del1.o del2.o
del1.c del2.c
util1.o util2.o
util1.c util2.c
!7
![Page 15: Identifying Hotspots in Software Build Processes](https://reader033.vdocuments.net/reader033/viewer/2022052909/5598a9d31a28ab64328b45e7/html5/thumbnails/15.jpg)
Refactoring can improve build performance
deliverable1 deliverable2
del1.o del2.o
del1.c del2.c
util1.o util2.o
util1.c util2.c
!7
![Page 16: Identifying Hotspots in Software Build Processes](https://reader033.vdocuments.net/reader033/viewer/2022052909/5598a9d31a28ab64328b45e7/html5/thumbnails/16.jpg)
Refactoring can improve build performance
deliverable1 deliverable2
del1.o del2.o
del1.c del2.c
util1.o util2.o
util1.c util2.c
!7
![Page 17: Identifying Hotspots in Software Build Processes](https://reader033.vdocuments.net/reader033/viewer/2022052909/5598a9d31a28ab64328b45e7/html5/thumbnails/17.jpg)
Refactoring can improve build performance
deliverable1 deliverable2
del1.o del2.o
del1.c del2.c
util1.o util2.o
util1.c util2.c
!7
![Page 18: Identifying Hotspots in Software Build Processes](https://reader033.vdocuments.net/reader033/viewer/2022052909/5598a9d31a28ab64328b45e7/html5/thumbnails/18.jpg)
Refactoring can improve build performance
deliverable1 deliverable2
del1.o del2.o
del1.c del2.c
util1.o util2.o
util1.c util2.c
!7
After refactoring:!2 commands!
triggered
![Page 19: Identifying Hotspots in Software Build Processes](https://reader033.vdocuments.net/reader033/viewer/2022052909/5598a9d31a28ab64328b45e7/html5/thumbnails/19.jpg)
…But where should we focus refactoring effort?
Files that rebuild the slowest?
!8
![Page 20: Identifying Hotspots in Software Build Processes](https://reader033.vdocuments.net/reader033/viewer/2022052909/5598a9d31a28ab64328b45e7/html5/thumbnails/20.jpg)
…But where should we focus refactoring effort?
Files that rebuild the slowest?
…but they may only rarely change!
!8
![Page 21: Identifying Hotspots in Software Build Processes](https://reader033.vdocuments.net/reader033/viewer/2022052909/5598a9d31a28ab64328b45e7/html5/thumbnails/21.jpg)
…But where should we focus refactoring effort?
Files that rebuild the slowest?
Files that change the most often?
…but they may only rarely change!
!8
![Page 22: Identifying Hotspots in Software Build Processes](https://reader033.vdocuments.net/reader033/viewer/2022052909/5598a9d31a28ab64328b45e7/html5/thumbnails/22.jpg)
…But where should we focus refactoring effort?
Files that rebuild the slowest?
Files that change the most often?
…but they may only rarely change!
…but they may already be optimal!
!8
![Page 23: Identifying Hotspots in Software Build Processes](https://reader033.vdocuments.net/reader033/viewer/2022052909/5598a9d31a28ab64328b45e7/html5/thumbnails/23.jpg)
…But where should we focus refactoring effort?
Files that rebuild the slowest?
Files that change the most often?
…but they may only rarely change!
…but they may already be optimal!
Focus on build hotspots:
Files that rebuild slowly and change often
!8
![Page 24: Identifying Hotspots in Software Build Processes](https://reader033.vdocuments.net/reader033/viewer/2022052909/5598a9d31a28ab64328b45e7/html5/thumbnails/24.jpg)
(1)Dependency
Graph Construction
(2)Dependency
GraphAnalysis (3)
Build Hotspot
Detection
Graphanalysis results
(1)Dependency
Graph Construction (3)
Build Hotspot
DetectionQuadrant Plot
Build Activity
File
Chu
rn
(2)Dependency
GraphAnalysisDep. Graph
VersionControlSystem
Hotspot detection approach
!9
![Page 25: Identifying Hotspots in Software Build Processes](https://reader033.vdocuments.net/reader033/viewer/2022052909/5598a9d31a28ab64328b45e7/html5/thumbnails/25.jpg)
MAKAO tool extracts build dependency graphs
(1)Dependency
Graph Construction
(2)Dependency
GraphAnalysis
(3)Build
Hotspot Detection
Design recovery and maintenance of build systems B. Adams, Herman Tromp, Kris De Schutter, Wolfgang De Meuter
[ICSM 2007]
!10
![Page 26: Identifying Hotspots in Software Build Processes](https://reader033.vdocuments.net/reader033/viewer/2022052909/5598a9d31a28ab64328b45e7/html5/thumbnails/26.jpg)
The cost of traversing an edge is derived by timing its build command
(1)Dependency
Graph Construction
(2)Dependency
GraphAnalysis
(3)Build
Hotspot Detection
!11
Median of 10 repetitions
![Page 27: Identifying Hotspots in Software Build Processes](https://reader033.vdocuments.net/reader033/viewer/2022052909/5598a9d31a28ab64328b45e7/html5/thumbnails/27.jpg)
!12
Mining frequently changing files from version control
Terminal
$ git clone … … $ git checkout <some_tag> … $ git log --oneline <some_file>
(1)Dependency
Graph Construction
(2)Dependency
GraphAnalysis
(3)Build
Hotspot Detection
![Page 28: Identifying Hotspots in Software Build Processes](https://reader033.vdocuments.net/reader033/viewer/2022052909/5598a9d31a28ab64328b45e7/html5/thumbnails/28.jpg)
(1)Dependency
Graph Construction
(2)Dependency
GraphAnalysis
(3)Build
Hotspot Detection
Quadrant plots highlight build hotspots
Rebuild cost
Num
ber o
f cha
nges
!13
![Page 29: Identifying Hotspots in Software Build Processes](https://reader033.vdocuments.net/reader033/viewer/2022052909/5598a9d31a28ab64328b45e7/html5/thumbnails/29.jpg)
(1)Dependency
Graph Construction
(2)Dependency
GraphAnalysis
(3)Build
Hotspot Detection
Quadrant plots highlight build hotspots
Rebuild cost
Num
ber o
f cha
nges
!13
![Page 30: Identifying Hotspots in Software Build Processes](https://reader033.vdocuments.net/reader033/viewer/2022052909/5598a9d31a28ab64328b45e7/html5/thumbnails/30.jpg)
(1)Dependency
Graph Construction
(2)Dependency
GraphAnalysis
(3)Build
Hotspot Detection
Quadrant plots highlight build hotspots
Rebuild cost
Num
ber o
f cha
nges
!13
![Page 31: Identifying Hotspots in Software Build Processes](https://reader033.vdocuments.net/reader033/viewer/2022052909/5598a9d31a28ab64328b45e7/html5/thumbnails/31.jpg)
(1)Dependency
Graph Construction
(2)Dependency
GraphAnalysis
(3)Build
Hotspot Detection
Quadrant plots highlight build hotspots
Rebuild cost
Num
ber o
f cha
nges
Refactor
these first!
!13
![Page 32: Identifying Hotspots in Software Build Processes](https://reader033.vdocuments.net/reader033/viewer/2022052909/5598a9d31a28ab64328b45e7/html5/thumbnails/32.jpg)
Open source case studies!!
!
!
Rebuild cost 90 seconds
Number of changes Median
Thresholds
!14
![Page 33: Identifying Hotspots in Software Build Processes](https://reader033.vdocuments.net/reader033/viewer/2022052909/5598a9d31a28ab64328b45e7/html5/thumbnails/33.jpg)
0 750,000 1,500,000 2,250,000 3,000,000
2,752,225
38,102
Build dependency graph properties
0 35,000 70,000 105,000 140,000
60,170
121,710
5,131
3,375
# Nodes# Edges
!15
![Page 34: Identifying Hotspots in Software Build Processes](https://reader033.vdocuments.net/reader033/viewer/2022052909/5598a9d31a28ab64328b45e7/html5/thumbnails/34.jpg)
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●●
●●●
●●
●●●
●
●●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●●
●
●●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●●●
●●●
●
●
●
●
●●
●●
●
●●
●
●
●
●
●●●
●
●
●●
●
●
●
●
●
●
●
●
●
●●●●
●
●
●
●
●
●
●
●
●
●
●●●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●●
●●●
●
●
●
●
●
●
●●●
●●
●●●
●
●
●
●
●
●
●●
●●
●●●
●
●
●●
●
●
●●●
●●
●●●
●
●
●
●
●●
●
●
●●●
●
●
●
●●
●
●●
●
●●
●
●●●
●
●
●
●
●●
●
●
●●●●●
●
●
●●●●
●
●
●
●
●
●
●●
●●
●
0.25
0.50
0.75
1.00
0 50 100 150Build Time
Nor
mal
ized
File
Chu
rn
Rebuild cost (seconds)
Num
ber o
f cha
nges
(N
orm
aliz
ed)
65 hotspots!(7% of
source files)
!16
glib/glib.hglib/glib-object.h
Main culprits
Max: 148 seconds
![Page 35: Identifying Hotspots in Software Build Processes](https://reader033.vdocuments.net/reader033/viewer/2022052909/5598a9d31a28ab64328b45e7/html5/thumbnails/35.jpg)
Rebuild cost (seconds)
Num
ber o
f cha
nges
(N
orm
aliz
ed)
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●●
●
●
●
●
●
●●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●●●●
●●
●
●
●
●
●
●
●
●
●●
●●
●
●
●●
●
●
●●●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●●
●●●
●
●
●
●
●
●●
●●
●
●
●●
●●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●●
●●●●
●
●
●
●
●
●●
●●●
●
●●
●
●
●●
●
●
●
●
●
●●●
●●●
●
●
●
●
●
●●
●
●
●
●●●
●
●
●
●●
●
●
●●●
●●●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●●
●●●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●●
●
●
●●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●●●
●●
●
●
●
●
●
●●
●●●●●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●●●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●●
●●
●●●
●●●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●●
●
●●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●●
●
●●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●
●
●●
●
●
●
●
●●
●●
●●●●●●●●
●
●
●●●
●
●
●
●
●
●●
●●
●
●
●
●●
●
●●
●
●●
●●
●
●●
●●●
●●
●●
●
●●
●
●
●
●
●
●●●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●●●
●
●
●●●
●
●●
●
●
●●
●
●
●●
●
●●
●
●
●
●
●●
●
●●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
0.25
0.50
0.75
1.00
0 50 100 150 200Build Time (s)
Nor
mal
ized
File
Chu
rn27 hotspots!
(2% of source files)
!17
postgres.haccess/htup.h
Main culprits
access/genam.haccess/xact.hutils/builtins.h
Max: 191 seconds
![Page 36: Identifying Hotspots in Software Build Processes](https://reader033.vdocuments.net/reader033/viewer/2022052909/5598a9d31a28ab64328b45e7/html5/thumbnails/36.jpg)
Rebuild cost (seconds)
Num
ber o
f cha
nges
(N
orm
aliz
ed)
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●●
●
●
●
●
●●
●
●●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●●●
●●●●
●●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●●●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●●
●●●
●
●
●
●●●●●
●●
●
●●
●
●●
●
●
●●●●●●●●●●●
●●●●●●●●●●●
●●●●●●●●●
●●●●●●●●●●●●●●●
●●●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●●●●●
●
●
●
●
●
●
●
●
●
●●
●
●●
●●●●●
●●
●
●●●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●●●
●
●
●
●
●
●
●
●●●
●●
●●●●
●
●
●●●●●
●
●
●
●●●●●
●
●
●
●
●●●●●●●
●
●●●●●
●●●●●●●●
●
●
●
●●
●
●
●
●
●
●
●
●●●●●●●●
●
●
●
●
●
●
●
●
●●●●●●●
●
●●●●●●●
●
●
●
●●
●
●
●
●
●
●●
●
●●
●
●
●
●●●●●●●
●
●
●●
●●●●●●●●
●
●
●●
●●●●
●
●
●
●
●●
●
●●
●
●
●●
●
●●
●
●●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●●
●●●
●
●
●
●
●●●●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●●●
●
●●●●●●●●●●●●●
●
●
●
●●●●
●
●
●●●
●●
●
●
●
●
●●
●
●
●
●●
●
●●
●
●
●
●●●●●●●●●●●●●●
●●●
●●●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●●●
●
●●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●●●●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●●●
●
●●
●
●●
●●●
●
●●
●●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●●
●●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●●
●●●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●●
●●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●●●●●
●●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●●
●
●
●
●●●
●
●
●
●
●
●
●
●●
●
●
●●●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●●
●
●
●●
●●
●
●●
●
●
●
●
●
●●
●●●
●
●
●
●●
●
●
●●
●
●
●
●
●
●●
●
●
●●
●
●●
●
●
●
●
●
●●●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●●●
●
●●
●
●
●●
●
●
●
●
●
●
●●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●●●●●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●●
●
●●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●●●●●
●●●
●
●●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●●
●
●
●●●
●●
●●
●
●
●●
●
●●●●●
●
●
●
●●
●●●
●
●
●●
●
●
●●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●●
●
●
●
●
●
●●
●
●
●
●
●
●●●●
●●●●
●
●
●●●
●●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●●
●
●
●●
●
●
●
●●
●
●
●
●
●●
●●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●●●
●
●●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●●
●●
●
●
●
●
●●●
●
●●
●
●
●
●
●●●
●
●●
●
●
●
●
●
●●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●●
●●●
●
●
●●
●
●●●
●
●
●
●●
●●
●●●●
●
●
●
●
●
●
●
●
●●●●
●●
●
●●●●
●
●
●
●
●●
●
●
●●●
●
●
●
●
●●●
●
●
●
●●●●●
●
●
●
●
●
●
●●
●
●
●
●
●●●●●●
●●
●
●
●
●●
●
●
●
●
●
●●●
●
●
●●●
●●
●
●●
●
●
●
●●●
●
●
●●●●●
●
●
●
●●●●●
●
●
●
●●●●●
●
●●
●
●●●●●●●●
●
●
●●●●
●
●●●●
●
●●
●
●
●
●
●●●
●●●
●
●●●●●●●●●●●●●●●●
●
●
●
●●●
●
●
●
●
●●
●●●●●●●●●●
●
●
●
●
●●
●●
●
●
●●●●●
●
●●●●●
●
●●●●●●
●
●
●●●●
●
●
●
●
●
●●
●●●●●●●●●●●
●
●
●●●●●
●
●
●●●●●●●●
●
●
●
●
●●
●
●●●●●●
●
●
●
●
●
●
●●●
●
●
●
●
●
●●●
●
●
●
●●●
●
●
●
●
●
●
●●●●
●
●●●●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●●●
●●
●
●
●
●
●
●
●●
●
●●
●
●
●●●
●
●●●●●●
●●●●●
●
●●
●
●●●●●
●●
●
●
●
●
●●
●
●
●
●
●
●●
●●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●●
●
●
●
●
●●●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●●●●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●●●
●
●
●
●
●
●
●●
●
●
●●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●●
●
●●
●
●●
●
●
●
●●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●●●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●●●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●●
●
●
●
●
●
●●●
●●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●●●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●●
●
●
●●
●●
●●
●●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●●●●●●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●●●
●
●
●
●
●●
●
●
●●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●●
●●
●
●●●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●●
●●●
●
●
●
●
●
●
●
●
●●●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●●●
●●●
●●
●●●
●
●
●
●
●●
●●
●
●
●
●
●
●
●●●
●
●
●●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●●●
●●
●●●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●●●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●●
●
●●
●●●●●
●
●●
●
●
●
●
●
●●●
●●
●●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●●
●●●
●
●●
●
●
●●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●●
●
●●●
●●
●
●
●
●
●
●●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●●
●●
●
●
●
●
●●
●
●
●●●
●●●
●
●
●
●
●
●
●●
●
●●
●
●
●
●●●
●
●●
●●
●
●●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●●
●
●●
●
●●
●
●●
●
●
●
●
●●●●●●●●●●●●●●●
●●
●
●
●●●●●
●●
●
●●●●
●
●●
●●
●
●●
●●●
●
●●
●●
●●
●
●
●
●●
●●
●●●●
●
●
●●●●●
●●●
●
●
●●
●●
●●
●●
●
●●
●
●●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●●
●
●●
●
●
●●
●
●●
●
●
●
●●
●
●
●●
●
●
●●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●●
●●●●
●
●●
●
●
●●
●
●
●●
●
●
●●●●
●
●
●
●
●●●
●
●
●
●
●●●
●
●●●
●●
●●●
●●●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●●●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●●
●
●●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●●●●
●
●
●
●●
●
●
●
●
●
●●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●●
●
●
●●●
●
●
●●
●●
●
●
●
●
●
●●
●
●
●●
●●
●●
●
●●
●
●●●●●●
●●●●●
●
●
●●
●●●●
●
●
●
●
●●
●
●
●
●
●
●●●●●
●●●●●●
●
●●●
●
●●
●●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●●
●
●
●
●●
●
●
●
●
●●
●●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●●●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●●
●
●
●
●
●
●●
●
●
●
●
●
●●●
●
●●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●●
●
●●
●
●
●
●
●●
●
●
●
●
●●●●
●
●
●
●
●
●
●
●●
●
●●●
●
●
●●
●
●
●
●
●●●
●
●●●●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●●
●●●●●●●●●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●●
●●
●
●
●
●●●
●●
●
●
●
●●●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●●
●
●●●●
●●●
●
●
●●
●
●●●
●●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●●
●●
●●
●
●
●●
●●
●●●
●
●
●
●
●
●●
●
●●●●
●
●
●●●
●
●
●●
●●●●●
●●
●●●
●
●
●●
●●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●●
●
●
●
●
●
●●
●
●
●●
●●
●
●
●
●
●
●
●
●●
●●●
●●
●
●
●
●
●
●
●●●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●●●
●●
●●
●●
●
●
●
●
●
●
●
●
●●●
●
●
●
●●
●
●●
●
●
●
●
●
●
●●●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●●
●
●
●
●●
●●●●
●
●
●
●●
●
●●
●●●●
●
●
●
●●●
●
●
●
●●
●
●
●
●
●
●●
●
●
●●●
●●●●●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●●
●●
●
●●●
●●
●●●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●●
●●
●
●
●●
●
●
●
●●
●
●
●
●●●
●
●●
●●
●
●●●●●●●
●
●
●
●
●
●
●
●●
●
●●●
●
●
●
●
●●●
●
●
●●●
●●
●●●●
●●
●
●
●
●●●
●●
●
●
●
●
●
●
●
●
●
●●
●●
●●●
●
●
●
●
●●
●●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●●●
●
●●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●●
●
●
●
●
●●
●
●
●
●
●●
●
●●
●
●●
●
●
●
●
●●●
●
●●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●●●
●
●
●
●●●
●●
●
●
●●
●
●
●●●●●
●
●
●
●●●
●
●
●
●
●
●●
●●●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●●●
●●
●●
●
●
●●
●
●
●
●
●●●
●
●●●
●
●●●●●●●●●●●●
●●
●
●
●
●
●●●●●●●●
●●●
●●
●
●●
●
●
●●●
●●
●●
●
●●●●●●●●●●●●●
●
●
●●●
●●
●●●●●●●●●●
●●●●●●●●●
●●●
●●●
●
●
●●
●●
●●●
●●
●●
●
●●
●
●●●
●
●
●●●
●
●
●
●
●
●
●●
●●●
●
●
●
●
●
●
●
●
●
●●
●●
●
●●
●
●
●●●
●●
●●
●
●
●
●
●●
●
●●
●
●
●●
●
●
●
●●
●●
●●
●
●
●
●
●●
●●
●●
●●
●●
●●●
●●●●
●
●
●
●●
●
●●●
●
●
●
●
●
●
●
●●
●●
●●
●
●●
●●
●●●
●
●
●●
●
●
●
●●
●
●
●
●●
●
●
●●
●
●
●
●●
●●●
●●●
●●
●
●●●
●●●●●
●
●●●
●
●
●●
●
●
●
●●
●●●●
●●
●
●
●●
●●
●
●●●
●
●●
●
●
●●●
●
●●
●●●●
●
●●
●●
●●●●
●
●
●
●
●
●●●●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●●●
●
●
●●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●●●
●
●
●
●●●●
●●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●●
●
●
●
●●
●
●●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●●●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●●●●
●
●
●
●●
●
●
●●●
●
●
●
●●
●●●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●●●●●●●
●
●●
●●●●
●
●
●
●
●
●
●●●●●
●
●●
●●●●●●●●●●●●●●●●●●●●●
●
●●●●
●
●●
●●
●
●●●●
●
●
●●
●
●
●
●●●●●●●●●●●●●●●●●●●●●●
●●
●●●●
●
●
●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●
●●
●●●
●
●
●●●
●
●
●●●●
0.25
0.50
0.75
1.00
0 2000 4000 6000 8000Build Time (s)
Nor
mal
ized
File
Chu
rn 732 hotspots!(8% of source files)
!18
qtxmlpatterns
High hotspot!concentration
qtbase/…/corelib
Main culpritstmtm
Max: 2hrs 19min
![Page 37: Identifying Hotspots in Software Build Processes](https://reader033.vdocuments.net/reader033/viewer/2022052909/5598a9d31a28ab64328b45e7/html5/thumbnails/37.jpg)
hot.c
...1 2
hot.o
nn-1
!19
General trends Transitive property of build hotspots
![Page 38: Identifying Hotspots in Software Build Processes](https://reader033.vdocuments.net/reader033/viewer/2022052909/5598a9d31a28ab64328b45e7/html5/thumbnails/38.jpg)
hot.c
...1 2
hot.o
nn-1
!19
General trends Transitive property of build hotspots
![Page 39: Identifying Hotspots in Software Build Processes](https://reader033.vdocuments.net/reader033/viewer/2022052909/5598a9d31a28ab64328b45e7/html5/thumbnails/39.jpg)
x.h
hot.c
...1 2
hot.o
nn-1
!19
General trends Transitive property of build hotspots
![Page 40: Identifying Hotspots in Software Build Processes](https://reader033.vdocuments.net/reader033/viewer/2022052909/5598a9d31a28ab64328b45e7/html5/thumbnails/40.jpg)
x.h
hot.c
...1 2
hot.o
nn-1
!19
General trends Transitive property of build hotspots
![Page 41: Identifying Hotspots in Software Build Processes](https://reader033.vdocuments.net/reader033/viewer/2022052909/5598a9d31a28ab64328b45e7/html5/thumbnails/41.jpg)
x.h
hot.c
...1 2
hot.o
nn-1
General trends Transitive property of build hotspots
!20
![Page 42: Identifying Hotspots in Software Build Processes](https://reader033.vdocuments.net/reader033/viewer/2022052909/5598a9d31a28ab64328b45e7/html5/thumbnails/42.jpg)
x.h
hot.c
...1 2
hot.o
nn-1
General trends Transitive property of build hotspots
!20
![Page 43: Identifying Hotspots in Software Build Processes](https://reader033.vdocuments.net/reader033/viewer/2022052909/5598a9d31a28ab64328b45e7/html5/thumbnails/43.jpg)
hub.h
x.h
hot.c
...1 2
hot.o
nn-1
General trends Transitive property of build hotspots
!20
![Page 44: Identifying Hotspots in Software Build Processes](https://reader033.vdocuments.net/reader033/viewer/2022052909/5598a9d31a28ab64328b45e7/html5/thumbnails/44.jpg)
hub.h
x.h
hot.c
...1 2
hot.o
nn-1
a.h b.h y.h z.h
General trends Transitive property of build hotspots
!20
![Page 45: Identifying Hotspots in Software Build Processes](https://reader033.vdocuments.net/reader033/viewer/2022052909/5598a9d31a28ab64328b45e7/html5/thumbnails/45.jpg)
hub.h
x.h
hot.c
...1 2
hot.o
nn-1
a.h b.h y.h z.h
General trends Transitive property of build hotspots
!20
![Page 46: Identifying Hotspots in Software Build Processes](https://reader033.vdocuments.net/reader033/viewer/2022052909/5598a9d31a28ab64328b45e7/html5/thumbnails/46.jpg)
hub.h
x.h
hot.c
...1 2
hot.o
nn-1
a.h b.h y.h z.h
General trends Transitive property of build hotspots
!20
![Page 47: Identifying Hotspots in Software Build Processes](https://reader033.vdocuments.net/reader033/viewer/2022052909/5598a9d31a28ab64328b45e7/html5/thumbnails/47.jpg)
hub.h
x.h
hot.c
...1 2
hot.o
nn-1
Limit internal use of “header file hubs” as much as possible!
a.h b.h y.h z.h
General trends Transitive property of build hotspots
!20
![Page 48: Identifying Hotspots in Software Build Processes](https://reader033.vdocuments.net/reader033/viewer/2022052909/5598a9d31a28ab64328b45e7/html5/thumbnails/48.jpg)
![Page 49: Identifying Hotspots in Software Build Processes](https://reader033.vdocuments.net/reader033/viewer/2022052909/5598a9d31a28ab64328b45e7/html5/thumbnails/49.jpg)
![Page 50: Identifying Hotspots in Software Build Processes](https://reader033.vdocuments.net/reader033/viewer/2022052909/5598a9d31a28ab64328b45e7/html5/thumbnails/50.jpg)
![Page 51: Identifying Hotspots in Software Build Processes](https://reader033.vdocuments.net/reader033/viewer/2022052909/5598a9d31a28ab64328b45e7/html5/thumbnails/51.jpg)
![Page 52: Identifying Hotspots in Software Build Processes](https://reader033.vdocuments.net/reader033/viewer/2022052909/5598a9d31a28ab64328b45e7/html5/thumbnails/52.jpg)
![Page 53: Identifying Hotspots in Software Build Processes](https://reader033.vdocuments.net/reader033/viewer/2022052909/5598a9d31a28ab64328b45e7/html5/thumbnails/53.jpg)