itt 2015 - simon stewart - building android apps at speed and scale
TRANSCRIPT
Building Android Apps At Scale and Speed
Simon Stewart @shs96c
Software Engineer, Facebook
One Guiding Principle
We don't want people waiting on computers
<
We want computers waiting on people
<
Scalable Source Organisation
Server
Sharing Code Today
Android
c++
iOS
Server
Sharing Code Today
AndroidiOS
c++c++
Server
Sharing Code Today
AndroidiOS
c++ c++c++
Sharing Code Tomorrow
fbsource
c++
All our source in a single repository (a "monorepo")
Why a monorepo?
Easier: code sharing, versioning, contributing
Simpler: large scale changes
Consistent: uniform tooling
Scalable Source Control
Mercurial
>1M >100KSource control commands
per daycommits per week
Large Repos Are Slow
▪Lots of commits a day
Pulls are slow
Checkouts are slow
▪Years of history
Clones are slow
Requires lots of disk space
We want computers waiting on people
<
Shallow CheckoutsClone just the latest version
Work locally without complete history
Need more history? Download on demand
HEAD
HEAD-1
HEAD-2
HEAD-3
Sparse CheckoutsWork on just the files you need
Build system integrations knows how to check out more
~/fbsource ios/... common/...
~/fbsource/.hg
WatchmanOS independent filesystem change notification
Integrates with mercurial
File checking operations scale linearly with number of files changed
Remote FilelogClone and pull just download metadata
Download files when required
Clones and pulls: 10x faster
Large Repos Are Fast
50times faster than git
2000hg improvements
Up to More than
Scalable Builds
Buck▪Next generation build system
▪Designed for modern hardware
▪SSD
▪Plentiful RAM
▪Many cores
▪Automatically parallelizes builds
▪And it's Open Source
Build Files
java_binary( name = 'example-binary', deps = [ ':example', ], )
java_library( name = 'example', srcs = glob(['*.java']), deps = [ '//third-party/guava:guava', ], visibility = [ '//javatests/...', ], )
Build file. Typically named BUCK
Build Files
java_binary( name = 'example-binary', deps = [ ':example', ], )
java_library( name = 'example', srcs = glob(['*.java']), deps = [ '//third-party/guava:guava', ], visibility = [ '//javatests/...', ], )
Build file. Typically named BUCK
A build rule of type "java_binary"
Build Files
java_binary( name = 'example-binary', deps = [ ':example', ], )
java_library( name = 'example', srcs = glob(['*.java']), deps = [ '//third-party/guava:guava', ], visibility = [ '//javatests/...', ], )
Build file. Typically named BUCK
A build rule of type "java_binary"
A target defined in the same build file
Build Files
java_binary( name = 'example-binary', deps = [ ':example', ], )
java_library( name = 'example', srcs = glob(['*.java']), deps = [ '//third-party/guava:guava', ], visibility = [ '//javatests/...', ], )
Build file. Typically named BUCK
A build rule of type "java_binary"
A target defined in the same build file
Build rule of type "java_library"
Build Files
java_binary( name = 'example-binary', deps = [ ':example', ], )
java_library( name = 'example', srcs = glob(['*.java']), deps = [ '//third-party/guava:guava', ], visibility = [ '//javatests/...', ], )
Build file. Typically named BUCK
A build rule of type "java_binary"
A target defined in the same build file
A target defined in another build file
Build rule of type "java_library"
Build Files
java_binary( name = 'example-binary', deps = [ ':example', ], )
java_library( name = 'example', srcs = glob(['*.java']), deps = [ '//third-party/guava:guava', ], visibility = [ '//javatests/...', ], )
Build file. Typically named BUCK
A build rule of type "java_binary"
A target defined in the same build file
A target defined in another build file
Build rule of type "java_library"
All targets have visibility
Target and Action Graph
Target Graph: a naive representation of targets defined in build files.
Action Graph: an optimized graph of actions to perform as the build
Language-Aware Smartspublic class Foo {
private String someField;
public void doSomething(int times) {
System.out.println("Wow! " + times);
}
}
Buck calculates the ABI, and keys rebuilds of dependencies off that
Packaging rules that depend on this will always be re-run.
Language-Aware Smartspublic class Foo {
private boolean theCakeIsALie = true;
public void doSomething(int times) {
System.out.println("Wow! " + times);
}
}
Private field changed: ABI remains the same as before
Language-Aware Smartspublic class Foo {
private boolean theCakeIsALie = true;
public void doSomething(int num) {
System.out.println("Wow! " + num);
}
}
Parameter name changed: ABI remains the same as before.
Language-Aware Smartspublic class Foo {
private boolean theCakeIsALie = true;
public void doSomething(String num) {
System.out.println("Wow! " + num);
}
}
Parameter type changed: ABI changes too!
Language-Aware Smartspublic class Foo {
private boolean theCakeIsALie = true;
public void doSomething(String num) {
System.err.println("Wow! " + num);
}
}
Body of method changed: ABI remains the same.
Pervasive Caching• Only 3% of each build is unique.
• Local and distributed caches slash build times.
Exopackage▪Exopackage is an optimization for building and installing development builds of Android apps.
Buck Smarts
Build rule knows all of the inputs that can affect its output
Changes to dx and proguard for faster compilation
Java ABI smarts
Android resource smarts
Daemon to watch file changes
Scalable Continuous Integration
<10 MinutesFeedback to Developers
>1000test results per second
5 Yearsmachine work per day
Life Cycle of a Diff
Write Code Test Locally Diff and review Sandcastle Land
Sandcastle
RapidHigh-Signal Frequent
Sandcastle Design
Single Master
Master
Worker Worker WorkerWorker
Sandcastle Design
Single Master Distributed Queue
Master
Worker Worker Worker Worker
Worker
Worker
Worker
Worker
Worker
Worker
Distributed Queue
Test Runs
Flakiness SpeedCoverage
Landcastle
Land
Target Determinator
Test Test Test
Push
We don't want people waiting on computers
<
We want computers waiting on people
<
Summary
Tool OSS? Link!
Buck Yes https://facebook.github.io/buck/
Mercurial Yes http://mercurial.selenic.com/
Watchman Yes https://facebook.github.io/watchman/
hgwatchman Yes https://bitbucket.org/facebook/hgwatchman
remote filelog Yes https://bitbucket.org/facebook/remotefilelog
Sandcastle No