Download - Uber Mobility Meetup: Mobile Testing
![Page 1: Uber Mobility Meetup: Mobile Testing](https://reader036.vdocuments.net/reader036/viewer/2022062523/58eebad01a28abff578b4713/html5/thumbnails/1.jpg)
![Page 2: Uber Mobility Meetup: Mobile Testing](https://reader036.vdocuments.net/reader036/viewer/2022062523/58eebad01a28abff578b4713/html5/thumbnails/2.jpg)
Evolution of Octopus
Bian Jiang, Nirav NagdaUber Mobility MeetupJuly 7, 2016
From cross-app/cross device testing, to network record/replay, to scenario-based testing
![Page 3: Uber Mobility Meetup: Mobile Testing](https://reader036.vdocuments.net/reader036/viewer/2022062523/58eebad01a28abff578b4713/html5/thumbnails/3.jpg)
Soon after we started at Uber, we encountered a fresh testing challenge. Can you guess what it is?
Our Uber Challenge
![Page 4: Uber Mobility Meetup: Mobile Testing](https://reader036.vdocuments.net/reader036/viewer/2022062523/58eebad01a28abff578b4713/html5/thumbnails/4.jpg)
Octopus: Our platform agnostic test runner
![Page 5: Uber Mobility Meetup: Mobile Testing](https://reader036.vdocuments.net/reader036/viewer/2022062523/58eebad01a28abff578b4713/html5/thumbnails/5.jpg)
GTAC 2015Octopus Takes on the Uber Challenge of Cross-app/Cross-device Testinghttps://www.youtube.com/watch?v=p6gsssppeT0
Uber Eng Bloghttps://eng.uber.com/rescued-by-octopus/
![Page 6: Uber Mobility Meetup: Mobile Testing](https://reader036.vdocuments.net/reader036/viewer/2022062523/58eebad01a28abff578b4713/html5/thumbnails/6.jpg)
EndToEnd (L)
Scenario-based Tests (M)
Unit Tests (S)
Top: small number of large end to end tests (dozens)Middle: middling amount of medium integration tests (hundreds)Bottom: large number of small unit tests (thousands)
PREREQUISITES: HEALTHY TEST PYRAMID
![Page 7: Uber Mobility Meetup: Mobile Testing](https://reader036.vdocuments.net/reader036/viewer/2022062523/58eebad01a28abff578b4713/html5/thumbnails/7.jpg)
Octopus Evolution(Hermetic Tests)
![Page 8: Uber Mobility Meetup: Mobile Testing](https://reader036.vdocuments.net/reader036/viewer/2022062523/58eebad01a28abff578b4713/html5/thumbnails/8.jpg)
Record mode:- Records the network responses
Replay mode:- Plays back the network responses from local files
HERMETIC TESTS: USING NETWORK RECORD/REPLAY
![Page 9: Uber Mobility Meetup: Mobile Testing](https://reader036.vdocuments.net/reader036/viewer/2022062523/58eebad01a28abff578b4713/html5/thumbnails/9.jpg)
ANDROID EXAMPLE: NETWORK RECORD/REPLAY
@Test@Replaypublic void whenDestinationPromptTap_shouldGoToLocationEditor() { HomeLayout .verifyLayout() .clickDestinationPrompt();}
(record = true)
![Page 10: Uber Mobility Meetup: Mobile Testing](https://reader036.vdocuments.net/reader036/viewer/2022062523/58eebad01a28abff578b4713/html5/thumbnails/10.jpg)
func testFavoriteLocationsScreen() { do { launchAppWithOctopusConfig(try OctopusConfigBuilder() .setNetworkReplayCache("rider_favorite_locations") .setFakeLocation("555 Market St") .build()) } catch { print("Error when launching app") } login(withUsername: "fake_user", password: "fake_password") let favoriteLocationsScreen = FavoriteLocaitonsScreen( testCase: self, application: application) favoriteLocationsScreen.waitForRequiredElements()}
IOS EXAMPLE: NETWORK RECORD/REPLAY
![Page 11: Uber Mobility Meetup: Mobile Testing](https://reader036.vdocuments.net/reader036/viewer/2022062523/58eebad01a28abff578b4713/html5/thumbnails/11.jpg)
Octopus Demo
![Page 13: Uber Mobility Meetup: Mobile Testing](https://reader036.vdocuments.net/reader036/viewer/2022062523/58eebad01a28abff578b4713/html5/thumbnails/13.jpg)
Octopus Evolution(Rules & Scenario-based testing)
![Page 14: Uber Mobility Meetup: Mobile Testing](https://reader036.vdocuments.net/reader036/viewer/2022062523/58eebad01a28abff578b4713/html5/thumbnails/14.jpg)
● What are rules?● How do they help?
○ Easier test code○ Shared setup/tearDown code logic○ Abstract out the complicated logic from tests
UBER RULES
![Page 15: Uber Mobility Meetup: Mobile Testing](https://reader036.vdocuments.net/reader036/viewer/2022062523/58eebad01a28abff578b4713/html5/thumbnails/15.jpg)
● Experiments● Location● Accounts● Animations● App reset
RULES EXAMPLES
![Page 16: Uber Mobility Meetup: Mobile Testing](https://reader036.vdocuments.net/reader036/viewer/2022062523/58eebad01a28abff578b4713/html5/thumbnails/16.jpg)
EXAMPLE: LOCATION RULE
@Location (device = TestLocation.CHICAGO_SUBURB)@Replay@Testpublic void whenValidAccount_shouldBeAbleToSignIn() {
WelcomeLayout.verifyLayout().clickLogin();
LoginLayout.verifyLayout().signInWith(Account.GENERIC.getEmail(),
Account.GENERIC.getPassword());
RideLayout.verifyLayout();}
![Page 17: Uber Mobility Meetup: Mobile Testing](https://reader036.vdocuments.net/reader036/viewer/2022062523/58eebad01a28abff578b4713/html5/thumbnails/17.jpg)
● What is scenario-based testing?● How does it help?
○ Speeds up the test run○ Easier to maintain○ Deterministic tests
SCENARIO-BASED TESTING
![Page 18: Uber Mobility Meetup: Mobile Testing](https://reader036.vdocuments.net/reader036/viewer/2022062523/58eebad01a28abff578b4713/html5/thumbnails/18.jpg)
SCENARIO-BASED TESTING: ANDROID EXAMPLE Only when recording
Actual test
public void cancelTripRequest() {
replayScenarioManager.doWhenRecording(new Interactions() { @Override public void perform() { AcceleratorsLayout.verifyLayout().clickItem(0); ProductSelectionLayout.verifyLayout().swipeToIndex(1) .clickProduct(R.string.uberx); ConfirmationLayout.verifyLayout().clickGo(); } });
// The test will start from here in replay mode. DispatchingLayout.verifyLayout().clickCancel(); ConfirmationLayout.verifyLayout(); }
![Page 19: Uber Mobility Meetup: Mobile Testing](https://reader036.vdocuments.net/reader036/viewer/2022062523/58eebad01a28abff578b4713/html5/thumbnails/19.jpg)
Special thanks to mobile-test-platform@, octopus-eng@, mobile-eng@ and all the amazing people at Uber :)Uber eng bloghttps://eng.uber.com/rescued-by-octopus/
GTAC talk on Octopushttps://www.youtube.com/watch?v=p6gsssppeT0
![Page 20: Uber Mobility Meetup: Mobile Testing](https://reader036.vdocuments.net/reader036/viewer/2022062523/58eebad01a28abff578b4713/html5/thumbnails/20.jpg)
![Page 21: Uber Mobility Meetup: Mobile Testing](https://reader036.vdocuments.net/reader036/viewer/2022062523/58eebad01a28abff578b4713/html5/thumbnails/21.jpg)
How an "ideal" CI setup for mobile looks like
Valera Zakharov
![Page 22: Uber Mobility Meetup: Mobile Testing](https://reader036.vdocuments.net/reader036/viewer/2022062523/58eebad01a28abff578b4713/html5/thumbnails/22.jpg)
![Page 23: Uber Mobility Meetup: Mobile Testing](https://reader036.vdocuments.net/reader036/viewer/2022062523/58eebad01a28abff578b4713/html5/thumbnails/23.jpg)
![Page 24: Uber Mobility Meetup: Mobile Testing](https://reader036.vdocuments.net/reader036/viewer/2022062523/58eebad01a28abff578b4713/html5/thumbnails/24.jpg)
![Page 25: Uber Mobility Meetup: Mobile Testing](https://reader036.vdocuments.net/reader036/viewer/2022062523/58eebad01a28abff578b4713/html5/thumbnails/25.jpg)
Ideal =
Reliable
Scalable
Performant
Controllable
Debuggable
![Page 26: Uber Mobility Meetup: Mobile Testing](https://reader036.vdocuments.net/reader036/viewer/2022062523/58eebad01a28abff578b4713/html5/thumbnails/26.jpg)
On Mobile
Different screen resolutions
Different OS versions
![Page 27: Uber Mobility Meetup: Mobile Testing](https://reader036.vdocuments.net/reader036/viewer/2022062523/58eebad01a28abff578b4713/html5/thumbnails/27.jpg)
On Android
Need Android Runtime to run instrumentation tests
$ adb shell am instrument -w <test_package_name>/<runner_class>
![Page 28: Uber Mobility Meetup: Mobile Testing](https://reader036.vdocuments.net/reader036/viewer/2022062523/58eebad01a28abff578b4713/html5/thumbnails/28.jpg)
Basic Building Blocks
Orchestration DeviceADB
Host (build node)
![Page 29: Uber Mobility Meetup: Mobile Testing](https://reader036.vdocuments.net/reader036/viewer/2022062523/58eebad01a28abff578b4713/html5/thumbnails/29.jpg)
Ideal Orchestration
Obtains/releases a device
Sets up the device for testing
Issues ‘adb shell am instrument’ calls for each test
Collects debug info
![Page 30: Uber Mobility Meetup: Mobile Testing](https://reader036.vdocuments.net/reader036/viewer/2022062523/58eebad01a28abff578b4713/html5/thumbnails/30.jpg)
Ideal Android Debug Bridge
Issue shell commands
Install apks
Push data to device
Pull data from device
Does all of the above reliably and quickly
![Page 31: Uber Mobility Meetup: Mobile Testing](https://reader036.vdocuments.net/reader036/viewer/2022062523/58eebad01a28abff578b4713/html5/thumbnails/31.jpg)
Device
![Page 32: Uber Mobility Meetup: Mobile Testing](https://reader036.vdocuments.net/reader036/viewer/2022062523/58eebad01a28abff578b4713/html5/thumbnails/32.jpg)
Ideal Test Device
Starts reliably and quickly
Runs as fast as top-of the line phone
Allows you to scale
Provides tests with control over system settings
Test Services
Prevents UI popups
IActivityController
![Page 33: Uber Mobility Meetup: Mobile Testing](https://reader036.vdocuments.net/reader036/viewer/2022062523/58eebad01a28abff578b4713/html5/thumbnails/33.jpg)
Android Vanilla Setup
Orchestration
Jenkins or another equivalent
./gradlew connectedCheck
Adb
Stock
Device
Locally connected emulator or physical device
![Page 34: Uber Mobility Meetup: Mobile Testing](https://reader036.vdocuments.net/reader036/viewer/2022062523/58eebad01a28abff578b4713/html5/thumbnails/34.jpg)
Vanilla Setup Problems
Orchestration
one process for all tests
no sharding
no separate logcat/debug info
Adb
Slow
Flaky
![Page 35: Uber Mobility Meetup: Mobile Testing](https://reader036.vdocuments.net/reader036/viewer/2022062523/58eebad01a28abff578b4713/html5/thumbnails/35.jpg)
Vanilla Setup Problems
Device
Not test friendly
Physical devices
not designed for CI usage
hard to maintain and scale
Emulator:
hard to configure properly
system images not well maintained
“fast” version can’t run on AWS
![Page 36: Uber Mobility Meetup: Mobile Testing](https://reader036.vdocuments.net/reader036/viewer/2022062523/58eebad01a28abff578b4713/html5/thumbnails/36.jpg)
Ideal =
Reliable
Scalable
Performant
Controllable
Debuggable
![Page 37: Uber Mobility Meetup: Mobile Testing](https://reader036.vdocuments.net/reader036/viewer/2022062523/58eebad01a28abff578b4713/html5/thumbnails/37.jpg)
Google’s Internal Setup
Orchestration
Custom (hooked into the internal build system)
Adb
adb.turbo ™
Device
Stock x86 emulator running in dedicated data-center machines
Properly configured/maintained
Config abstracted away by a script
Includes special services for testing
Custom ActivityController for dismissing UI popups
Test services for tweaking system settings, screenshots, etc
![Page 38: Uber Mobility Meetup: Mobile Testing](https://reader036.vdocuments.net/reader036/viewer/2022062523/58eebad01a28abff578b4713/html5/thumbnails/38.jpg)
![Page 39: Uber Mobility Meetup: Mobile Testing](https://reader036.vdocuments.net/reader036/viewer/2022062523/58eebad01a28abff578b4713/html5/thumbnails/39.jpg)
Existing services come up short
Firebase Google Cloud Test Lab
Runs all tests in one instrumentation
Virtual Devices are slow
Not debuggable
Xamarin
Doesn’t run instrumentation tests
AWS
Runs all tests in one instrumentation
Only real devices available
![Page 40: Uber Mobility Meetup: Mobile Testing](https://reader036.vdocuments.net/reader036/viewer/2022062523/58eebad01a28abff578b4713/html5/thumbnails/40.jpg)
So… now what
¯\_(ツ )_/¯
![Page 41: Uber Mobility Meetup: Mobile Testing](https://reader036.vdocuments.net/reader036/viewer/2022062523/58eebad01a28abff578b4713/html5/thumbnails/41.jpg)
Some thoughts...
Orchestration
Tools for discovering tests in an apk (based on dexdump)
An alternative to ./gradlew connectedCheck
android_test --apk my.apk --tests test.apk
ADB
adb.turbo (or equivalent) if planning to run at Google-scale
Device
Custom IActivityController
No go on physical devices
Test Services works well as a community project
Some parts won’t work on physical devices
To scale, need a fast emulator in the cloud
Genymotion cloud?
User Mode Linux for Android?
![Page 42: Uber Mobility Meetup: Mobile Testing](https://reader036.vdocuments.net/reader036/viewer/2022062523/58eebad01a28abff578b4713/html5/thumbnails/42.jpg)
![Page 43: Uber Mobility Meetup: Mobile Testing](https://reader036.vdocuments.net/reader036/viewer/2022062523/58eebad01a28abff578b4713/html5/thumbnails/43.jpg)
![Page 44: Uber Mobility Meetup: Mobile Testing](https://reader036.vdocuments.net/reader036/viewer/2022062523/58eebad01a28abff578b4713/html5/thumbnails/44.jpg)
Faster Tests with Test Parallelization JUSTIN MARTIN & DANIEL RIBEIRO
![Page 45: Uber Mobility Meetup: Mobile Testing](https://reader036.vdocuments.net/reader036/viewer/2022062523/58eebad01a28abff578b4713/html5/thumbnails/45.jpg)
Too many tests? No such thing!
![Page 46: Uber Mobility Meetup: Mobile Testing](https://reader036.vdocuments.net/reader036/viewer/2022062523/58eebad01a28abff578b4713/html5/thumbnails/46.jpg)
~150 minutes
~50 minutes
MACHINE TIME WALL TIME
Distributed vs. Non-distributed
![Page 47: Uber Mobility Meetup: Mobile Testing](https://reader036.vdocuments.net/reader036/viewer/2022062523/58eebad01a28abff578b4713/html5/thumbnails/47.jpg)
~150 minutes
~125 minutes
~50 minutes
MACHINE TIME WALL TIME
NON-DISTRIBUTED DISTRIBUTED
Distributed vs. Non-distributed
![Page 48: Uber Mobility Meetup: Mobile Testing](https://reader036.vdocuments.net/reader036/viewer/2022062523/58eebad01a28abff578b4713/html5/thumbnails/48.jpg)
~150 minutes
~125 minutes
~50 minutes
MACHINE TIME WALL TIME
~20 minutes
Distributed vs. Non-distributed
NON-DISTRIBUTED DISTRIBUTED
![Page 49: Uber Mobility Meetup: Mobile Testing](https://reader036.vdocuments.net/reader036/viewer/2022062523/58eebad01a28abff578b4713/html5/thumbnails/49.jpg)
Separation of Powers
BUILD TEST
![Page 50: Uber Mobility Meetup: Mobile Testing](https://reader036.vdocuments.net/reader036/viewer/2022062523/58eebad01a28abff578b4713/html5/thumbnails/50.jpg)
Separation of Powers
BUILD MAC PROS
TEST MAC MINIS
![Page 51: Uber Mobility Meetup: Mobile Testing](https://reader036.vdocuments.net/reader036/viewer/2022062523/58eebad01a28abff578b4713/html5/thumbnails/51.jpg)
![Page 52: Uber Mobility Meetup: Mobile Testing](https://reader036.vdocuments.net/reader036/viewer/2022062523/58eebad01a28abff578b4713/html5/thumbnails/52.jpg)
Reruns Are Even Faster!
PRO TIME MINI TIME
MACHINE TIME (RERUN)
WALL TIME (RERUN)
![Page 53: Uber Mobility Meetup: Mobile Testing](https://reader036.vdocuments.net/reader036/viewer/2022062523/58eebad01a28abff578b4713/html5/thumbnails/53.jpg)
XCKnife
FASTER DISTRIBUTED TESTS FOR iOSGITHUB.COM/SQUARE/XCKNIFE
SQU.RE/XCKNIFE
![Page 54: Uber Mobility Meetup: Mobile Testing](https://reader036.vdocuments.net/reader036/viewer/2022062523/58eebad01a28abff578b4713/html5/thumbnails/54.jpg)
![Page 55: Uber Mobility Meetup: Mobile Testing](https://reader036.vdocuments.net/reader036/viewer/2022062523/58eebad01a28abff578b4713/html5/thumbnails/55.jpg)
Each iteration takes a test class from the queue and assigns it to the machine that is the least busy
XCKnife Balancing
![Page 56: Uber Mobility Meetup: Mobile Testing](https://reader036.vdocuments.net/reader036/viewer/2022062523/58eebad01a28abff578b4713/html5/thumbnails/56.jpg)
Balanced Sharding
![Page 57: Uber Mobility Meetup: Mobile Testing](https://reader036.vdocuments.net/reader036/viewer/2022062523/58eebad01a28abff578b4713/html5/thumbnails/57.jpg)
Approximate
![Page 58: Uber Mobility Meetup: Mobile Testing](https://reader036.vdocuments.net/reader036/viewer/2022062523/58eebad01a28abff578b4713/html5/thumbnails/58.jpg)
![Page 59: Uber Mobility Meetup: Mobile Testing](https://reader036.vdocuments.net/reader036/viewer/2022062523/58eebad01a28abff578b4713/html5/thumbnails/59.jpg)
Key takeaways
![Page 60: Uber Mobility Meetup: Mobile Testing](https://reader036.vdocuments.net/reader036/viewer/2022062523/58eebad01a28abff578b4713/html5/thumbnails/60.jpg)
Disclaimer
![Page 61: Uber Mobility Meetup: Mobile Testing](https://reader036.vdocuments.net/reader036/viewer/2022062523/58eebad01a28abff578b4713/html5/thumbnails/61.jpg)
Diminishing Returns
![Page 62: Uber Mobility Meetup: Mobile Testing](https://reader036.vdocuments.net/reader036/viewer/2022062523/58eebad01a28abff578b4713/html5/thumbnails/62.jpg)
Testing Pyramid
![Page 63: Uber Mobility Meetup: Mobile Testing](https://reader036.vdocuments.net/reader036/viewer/2022062523/58eebad01a28abff578b4713/html5/thumbnails/63.jpg)
Testing Pyramid
![Page 64: Uber Mobility Meetup: Mobile Testing](https://reader036.vdocuments.net/reader036/viewer/2022062523/58eebad01a28abff578b4713/html5/thumbnails/64.jpg)
Testing Matrix
![Page 65: Uber Mobility Meetup: Mobile Testing](https://reader036.vdocuments.net/reader036/viewer/2022062523/58eebad01a28abff578b4713/html5/thumbnails/65.jpg)
Many devices
Testing Matrix:
![Page 66: Uber Mobility Meetup: Mobile Testing](https://reader036.vdocuments.net/reader036/viewer/2022062523/58eebad01a28abff578b4713/html5/thumbnails/66.jpg)
OS versions
Testing Matrix:
![Page 67: Uber Mobility Meetup: Mobile Testing](https://reader036.vdocuments.net/reader036/viewer/2022062523/58eebad01a28abff578b4713/html5/thumbnails/67.jpg)
Orientations
Testing Matrix:
![Page 68: Uber Mobility Meetup: Mobile Testing](https://reader036.vdocuments.net/reader036/viewer/2022062523/58eebad01a28abff578b4713/html5/thumbnails/68.jpg)
localizations
Testing Matrix:
![Page 69: Uber Mobility Meetup: Mobile Testing](https://reader036.vdocuments.net/reader036/viewer/2022062523/58eebad01a28abff578b4713/html5/thumbnails/69.jpg)
Test Explosion
![Page 70: Uber Mobility Meetup: Mobile Testing](https://reader036.vdocuments.net/reader036/viewer/2022062523/58eebad01a28abff578b4713/html5/thumbnails/70.jpg)
It is distributed
![Page 71: Uber Mobility Meetup: Mobile Testing](https://reader036.vdocuments.net/reader036/viewer/2022062523/58eebad01a28abff578b4713/html5/thumbnails/71.jpg)
Availability
![Page 72: Uber Mobility Meetup: Mobile Testing](https://reader036.vdocuments.net/reader036/viewer/2022062523/58eebad01a28abff578b4713/html5/thumbnails/72.jpg)
Operations matter
![Page 73: Uber Mobility Meetup: Mobile Testing](https://reader036.vdocuments.net/reader036/viewer/2022062523/58eebad01a28abff578b4713/html5/thumbnails/73.jpg)
Test your infrastructure
![Page 74: Uber Mobility Meetup: Mobile Testing](https://reader036.vdocuments.net/reader036/viewer/2022062523/58eebad01a28abff578b4713/html5/thumbnails/74.jpg)
Everything breaks
![Page 75: Uber Mobility Meetup: Mobile Testing](https://reader036.vdocuments.net/reader036/viewer/2022062523/58eebad01a28abff578b4713/html5/thumbnails/75.jpg)
When you fix, please open source it
![Page 76: Uber Mobility Meetup: Mobile Testing](https://reader036.vdocuments.net/reader036/viewer/2022062523/58eebad01a28abff578b4713/html5/thumbnails/76.jpg)
Toolchain
![Page 77: Uber Mobility Meetup: Mobile Testing](https://reader036.vdocuments.net/reader036/viewer/2022062523/58eebad01a28abff578b4713/html5/thumbnails/77.jpg)
UXMatters
![Page 78: Uber Mobility Meetup: Mobile Testing](https://reader036.vdocuments.net/reader036/viewer/2022062523/58eebad01a28abff578b4713/html5/thumbnails/78.jpg)
Bad actors
![Page 80: Uber Mobility Meetup: Mobile Testing](https://reader036.vdocuments.net/reader036/viewer/2022062523/58eebad01a28abff578b4713/html5/thumbnails/80.jpg)
![Page 81: Uber Mobility Meetup: Mobile Testing](https://reader036.vdocuments.net/reader036/viewer/2022062523/58eebad01a28abff578b4713/html5/thumbnails/81.jpg)
3x3: Speeding Up Mobile Releases
Drew Hannay (@drewhannay)
![Page 82: Uber Mobility Meetup: Mobile Testing](https://reader036.vdocuments.net/reader036/viewer/2022062523/58eebad01a28abff578b4713/html5/thumbnails/82.jpg)
Project VoyagerNew version of
flagship LinkedIn app
250+ committers across Android & iOS
~1 year of development
Investment in mobile infrastructure at LinkedIn
![Page 83: Uber Mobility Meetup: Mobile Testing](https://reader036.vdocuments.net/reader036/viewer/2022062523/58eebad01a28abff578b4713/html5/thumbnails/83.jpg)
Before Voyager12 releases per year
RC build + manual regression suite
Mad dash to commit code before the RC cutoffMissing the cutoff meant a long wait for the next release
Product & marketing plans were made around the monthly releases
Hard to iterate on member feedback
![Page 84: Uber Mobility Meetup: Mobile Testing](https://reader036.vdocuments.net/reader036/viewer/2022062523/58eebad01a28abff578b4713/html5/thumbnails/84.jpg)
3x3Release three times per day, no more than three hours from code commit to member
availability
![Page 85: Uber Mobility Meetup: Mobile Testing](https://reader036.vdocuments.net/reader036/viewer/2022062523/58eebad01a28abff578b4713/html5/thumbnails/85.jpg)
Why three hours?Not enough time for manual testing steps
Not enough time to test everythingThe goal isn’t 100% automation, it’s faster iterations
We don’t want engineers spending most of their time maintaining tests that break whenever a design changes
UI tests are chosen based on production-critical business flows
Faster iteration helps emphasize craftsmanshipDevs can take the extra time to write quality code since the next release is soon
![Page 86: Uber Mobility Meetup: Mobile Testing](https://reader036.vdocuments.net/reader036/viewer/2022062523/58eebad01a28abff578b4713/html5/thumbnails/86.jpg)
Commit Pipeline
CodeReview
StaticAnalysis
UnitTests
BuildReleaseArtifacts
UITests
AlphaRelease
FeatureDevelopment
Production Release
BetaRelease
![Page 87: Uber Mobility Meetup: Mobile Testing](https://reader036.vdocuments.net/reader036/viewer/2022062523/58eebad01a28abff578b4713/html5/thumbnails/87.jpg)
Commit Pipeline
CodeReview
StaticAnalysis
UnitTests
BuildReleaseArtifacts
UITests
AlphaRelease
FeatureDevelopment
Production Release
BetaRelease
![Page 88: Uber Mobility Meetup: Mobile Testing](https://reader036.vdocuments.net/reader036/viewer/2022062523/58eebad01a28abff578b4713/html5/thumbnails/88.jpg)
Static analysisCompile-time contract with API server using Rest.li
Rest.li data templates are shared between API server & clients
Provides static analysis checks that guarantee backwards compatibility
Client models are code generated for further safety
Java Checkstyle
Android LintOver 200 checks provided by Google
Several custom checks written for LinkedIn-specific patterns
Swift LintForked version of Realm’s SwiftLint
Added custom checks for LinkedIn patterns
![Page 89: Uber Mobility Meetup: Mobile Testing](https://reader036.vdocuments.net/reader036/viewer/2022062523/58eebad01a28abff578b4713/html5/thumbnails/89.jpg)
Building the codeOver 500k lines of code between Android & iOS
Building production binaries for a large codebase is slow
iOS & SwiftAt one point early on, Swift compilation took over two hours
Refactoring into sub-projects and modules lead to a more than 50% speed up
Android Split APKsSeparate binary for each combination of screen density and CPU architecture
Distributed buildsBuild the release binaries on separate machines while tests are running
![Page 90: Uber Mobility Meetup: Mobile Testing](https://reader036.vdocuments.net/reader036/viewer/2022062523/58eebad01a28abff578b4713/html5/thumbnails/90.jpg)
What do we test?Unit tests
Layout testsUnit tests for views
Stress test views with long strings, short strings
Make sure views don’t overlap, and render properly in right-to-left mode
Scenario testsValidate that key business metric flows are working properly
Usually flows that span multiple screens in the app
App gets mock data from a local fixture server
Not an exhaustive suite
![Page 91: Uber Mobility Meetup: Mobile Testing](https://reader036.vdocuments.net/reader036/viewer/2022062523/58eebad01a28abff578b4713/html5/thumbnails/91.jpg)
Test stabilityUI tests use Android Espresso & iOS KIF frameworks
Needed to create a consistent test environment across dev & build machines
AndroidSelf-contained, standardized Android emulator bundle
Custom test harness that runs one test per application launch
iOSKIF contained hard-coded waits that passed on dev boxes, but failed on slower build
servers
Forked KIF to convert to a checkpoint-based system,where the code tells the test when to proceed to the next step
![Page 92: Uber Mobility Meetup: Mobile Testing](https://reader036.vdocuments.net/reader036/viewer/2022062523/58eebad01a28abff578b4713/html5/thumbnails/92.jpg)
Test speedAndroid
Use Espresso’s IdlingResource API to avoid sleeps and waits
Run up to 16 Android emulators in parallel on a single build machine
Custom test harness allows optimal test parallelization
iOSRefactoring KIF cut UI testing time by more than 80%
Distributed testing -> Shard tests across multiple machinesSignificantly faster, but led to greater exposure to any tooling instability
Nontrivial overhead in starting a child job
![Page 93: Uber Mobility Meetup: Mobile Testing](https://reader036.vdocuments.net/reader036/viewer/2022062523/58eebad01a28abff578b4713/html5/thumbnails/93.jpg)
Android multi-emulator test run
![Page 94: Uber Mobility Meetup: Mobile Testing](https://reader036.vdocuments.net/reader036/viewer/2022062523/58eebad01a28abff578b4713/html5/thumbnails/94.jpg)
iOS KIF refactoring
![Page 95: Uber Mobility Meetup: Mobile Testing](https://reader036.vdocuments.net/reader036/viewer/2022062523/58eebad01a28abff578b4713/html5/thumbnails/95.jpg)
iOS multi-simulator testing
![Page 96: Uber Mobility Meetup: Mobile Testing](https://reader036.vdocuments.net/reader036/viewer/2022062523/58eebad01a28abff578b4713/html5/thumbnails/96.jpg)
Partner teamsHistorically, several partner teams validated
the build before a release
For example, we needed sign off from the localization team
Lint checks catch hardcoded or improperly formatted strings
Layout tests catch strings that are too long and RTL layout bugs
Semantic correctness of translations is still validated by translators manually
![Page 97: Uber Mobility Meetup: Mobile Testing](https://reader036.vdocuments.net/reader036/viewer/2022062523/58eebad01a28abff578b4713/html5/thumbnails/97.jpg)
Getting to membersEvery three hours, internal alpha testers get a new build
Mainly members of the Flagship team
Product managers, devs, and execs who want to see the latest code ASAP
Every week, the rest of the company gets a new beta buildiOS build is submitted to Apple for review
After a week of beta, the build is promoted to productionAssuming Apple’s review is complete, iOS is released
Take advantage of Google Play staged rollout for Android
![Page 98: Uber Mobility Meetup: Mobile Testing](https://reader036.vdocuments.net/reader036/viewer/2022062523/58eebad01a28abff578b4713/html5/thumbnails/98.jpg)
DogfoodingAndroid: Google Play alpha/beta channel
Easy upgrades for employees, even while off the corporate network
Somewhat difficult to get set up, but easy once registered
iOS: TestFlightNice, but limited number of users
iOS: Custom enterprise distributionScales to our number of users, but employees must be on corporate wifi to upgrade
Splash screen in the production app encourages employees to use beta builds
![Page 99: Uber Mobility Meetup: Mobile Testing](https://reader036.vdocuments.net/reader036/viewer/2022062523/58eebad01a28abff578b4713/html5/thumbnails/99.jpg)
Minimizing risk & enabling experimentsTake advantage of LinkedIn’s existing A/B testing infrastructure
New features are developed behind feature flagsCode can be ramped dynamically to different groups of members
Performance of new features or changes can be monitored
Dynamic configuration
Server-controlled kill switchCrashing or buggy code can often be disabled without a new build
![Page 100: Uber Mobility Meetup: Mobile Testing](https://reader036.vdocuments.net/reader036/viewer/2022062523/58eebad01a28abff578b4713/html5/thumbnails/100.jpg)
3x3 after 6 months: areas to improveRelease automation
Production uploads to the app stores are still a manual process
Getting release notes & translations is painful
Automated performance testingWe can sample performance of the app in production,
but don’t have a great way of catching issues before release
Android Monkey testingEnables wide range of API level & device coverage with very low overhead cost
iOS speed improvementsKeep up with Swift evolution
Bring 3x3 framework to other LinkedIn apps
![Page 101: Uber Mobility Meetup: Mobile Testing](https://reader036.vdocuments.net/reader036/viewer/2022062523/58eebad01a28abff578b4713/html5/thumbnails/101.jpg)
Questions
![Page 102: Uber Mobility Meetup: Mobile Testing](https://reader036.vdocuments.net/reader036/viewer/2022062523/58eebad01a28abff578b4713/html5/thumbnails/102.jpg)
3x3 blogs & videos3x3: Speeding up mobile releases
3x3: iOS Build Speed and StabilityTest Stability - How We Make UI Tests Stable
UI Automation: Keep it Functional - and Stable!
Consistent Android Testing Environments with Gradle (slides)
Effective Layout Testing Library for iOS
Managing iOS Continuous Integration at Enterprise Scale
![Page 103: Uber Mobility Meetup: Mobile Testing](https://reader036.vdocuments.net/reader036/viewer/2022062523/58eebad01a28abff578b4713/html5/thumbnails/103.jpg)