itt 2015 - simon stewart - building android apps at speed and scale

Building Android Apps At Scale and Speed

Simon Stewart @shs96c

Software Engineer, Facebook

One Guiding Principle

We don't want people waiting on computers

<

We want computers waiting on people

<

Scalable Source Organisation

Server

Sharing Code Today

Android

c++

iOS

Server

Sharing Code Today

AndroidiOS

c++c++

Server

Sharing Code Today

AndroidiOS

c++ c++c++

Sharing Code Tomorrow

fbsource

c++

All our source in a single repository (a "monorepo")

Why a monorepo?

Easier: code sharing, versioning, contributing

Simpler: large scale changes

Consistent: uniform tooling

Scalable Source Control

Mercurial

>1M >100KSource control commands

per daycommits per week

Large Repos Are Slow

▪Lots of commits a day

Pulls are slow

Checkouts are slow

▪Years of history

Clones are slow

Requires lots of disk space

We want computers waiting on people

<

Shallow CheckoutsClone just the latest version

Work locally without complete history

Need more history? Download on demand

HEAD

HEAD-1

HEAD-2

HEAD-3

Sparse CheckoutsWork on just the files you need

Build system integrations knows how to check out more

~/fbsource ios/... common/...

~/fbsource/.hg

WatchmanOS independent filesystem change notification

Integrates with mercurial

File checking operations scale linearly with number of files changed

Remote FilelogClone and pull just download metadata

Download files when required

Clones and pulls: 10x faster

Large Repos Are Fast

50times faster than git

2000hg improvements

Up to More than

Scalable Builds

Buck▪Next generation build system

▪Designed for modern hardware

▪SSD

▪Plentiful RAM

▪Many cores

▪Automatically parallelizes builds

▪And it's Open Source

Build Files

java_binary( name = 'example-binary', deps = [ ':example', ], )

java_library( name = 'example', srcs = glob(['*.java']), deps = [ '//third-party/guava:guava', ], visibility = [ '//javatests/...', ], )

Build file. Typically named BUCK

Build Files

java_binary( name = 'example-binary', deps = [ ':example', ], )

java_library( name = 'example', srcs = glob(['*.java']), deps = [ '//third-party/guava:guava', ], visibility = [ '//javatests/...', ], )

Build file. Typically named BUCK

A build rule of type "java_binary"

Build Files

java_binary( name = 'example-binary', deps = [ ':example', ], )

java_library( name = 'example', srcs = glob(['*.java']), deps = [ '//third-party/guava:guava', ], visibility = [ '//javatests/...', ], )

Build file. Typically named BUCK

A build rule of type "java_binary"

A target defined in the same build file

Build Files

java_binary( name = 'example-binary', deps = [ ':example', ], )

java_library( name = 'example', srcs = glob(['*.java']), deps = [ '//third-party/guava:guava', ], visibility = [ '//javatests/...', ], )

Build file. Typically named BUCK

A build rule of type "java_binary"

A target defined in the same build file

Build rule of type "java_library"

Build Files

java_binary( name = 'example-binary', deps = [ ':example', ], )

java_library( name = 'example', srcs = glob(['*.java']), deps = [ '//third-party/guava:guava', ], visibility = [ '//javatests/...', ], )

Build file. Typically named BUCK

A build rule of type "java_binary"

A target defined in the same build file

A target defined in another build file

Build rule of type "java_library"

Build Files

java_binary( name = 'example-binary', deps = [ ':example', ], )

java_library( name = 'example', srcs = glob(['*.java']), deps = [ '//third-party/guava:guava', ], visibility = [ '//javatests/...', ], )

Build file. Typically named BUCK

A build rule of type "java_binary"

A target defined in the same build file

A target defined in another build file

Build rule of type "java_library"

All targets have visibility

Target and Action Graph

Target Graph: a naive representation of targets defined in build files.

Action Graph: an optimized graph of actions to perform as the build

Language-Aware Smartspublic class Foo {

private String someField;

public void doSomething(int times) {

System.out.println("Wow! " + times);

}

}

Buck calculates the ABI, and keys rebuilds of dependencies off that

Packaging rules that depend on this will always be re-run.

Language-Aware Smartspublic class Foo {

private boolean theCakeIsALie = true;

public void doSomething(int times) {

System.out.println("Wow! " + times);

}

}

Private field changed: ABI remains the same as before

Language-Aware Smartspublic class Foo {

private boolean theCakeIsALie = true;

public void doSomething(int num) {

System.out.println("Wow! " + num);

}

}

Parameter name changed: ABI remains the same as before.

Language-Aware Smartspublic class Foo {

private boolean theCakeIsALie = true;

public void doSomething(String num) {

System.out.println("Wow! " + num);

}

}

Parameter type changed: ABI changes too!

Language-Aware Smartspublic class Foo {

private boolean theCakeIsALie = true;

public void doSomething(String num) {

System.err.println("Wow! " + num);

}

}

Body of method changed: ABI remains the same.

Pervasive Caching• Only 3% of each build is unique.

• Local and distributed caches slash build times.

Exopackage▪Exopackage is an optimization for building and installing development builds of Android apps.

Buck Smarts

Build rule knows all of the inputs that can affect its output

Changes to dx and proguard for faster compilation

Java ABI smarts

Android resource smarts

Daemon to watch file changes

Scalable Continuous Integration

<10 MinutesFeedback to Developers

>1000test results per second

5 Yearsmachine work per day

Life Cycle of a Diff

Write Code Test Locally Diff and review Sandcastle Land

Sandcastle

RapidHigh-Signal Frequent

Sandcastle Design

Single Master

Master

Worker Worker WorkerWorker

Sandcastle Design

Single Master Distributed Queue

Master

Worker Worker Worker Worker

Worker

Worker

Worker

Worker

Worker

Worker

Distributed Queue

Test Runs

Flakiness SpeedCoverage

Landcastle

Land

Target Determinator

Test Test Test

Push

We don't want people waiting on computers

<

We want computers waiting on people

<

Summary

Tool OSS? Link!

Buck Yes https://facebook.github.io/buck/

Mercurial Yes http://mercurial.selenic.com/

Watchman Yes https://facebook.github.io/watchman/

hgwatchman Yes https://bitbucket.org/facebook/hgwatchman

remote filelog Yes https://bitbucket.org/facebook/remotefilelog

Sandcastle No