codefest 2014. eduardo bravo — google+, a deep dive into mobile testing
DESCRIPTION
TRANSCRIPT
Eduardo Bravo (Software Engineer in Test @ Google)
Google+, A Deep Dive into Mobile TestingCodefest 2014, Novosibirsk, Russia
Lets start with a question….. (‘cause life is full of them)
Is your product a “Mobile first” product?
Google+ became “Mobile first” overnight ..Testing teamManagers
“Yeah! Mobile first” Mobile testing ... what?
The Google+ Testing Picture
Google ApisServer
Foo backend
Bar backend
Many more
Google+ Backend
The Google+ Mobile Testing Picture
Google ApisServer
Foo backend
Bar backend
Many more
Google+ Backend
Mobile Testing
Google+ Android Mobile Test Infrastructure Unit tests
Hermetic UI tests
Monkeyrunner tests
Feature “Leak” tests
Android tooling
Android Unit TestingRobolectric
Why?Robolectric is designed to allow running android tests on the JVM, by using shadow objects and allowing resource loading. Robolectric tests can be run as normal java tests.
Advantages:● It allows us to shadow methods that can’t be mocked on the traditionally
android framework.(Exceptions apply, see powermock)● Decoupled from an android device or emulator.
Android Unit TestingRobolectric
All of a sudden your unit tests just got faster!
● There is no need to start an emulator (our tests run in the cloud)● Shorter build times● No need to install apks
Android UI TestingThe flakiness problem
Flaky tests are worse than having no tests
Android Hermetic UI TestsGoogle+ uses Espresso
Why?● Backwards compatibility: Froyo, Gingerbread, Ice Cream
Sandwich, Jelly Bean, Kit kat
● Handles Synchronization: Instrumentation tests run on a different (instrumentation) thread than UI operations (processed on the UI thread).
Android Hermetic UI TestsGoogle+ uses Espresso
Why?
● Rich debugging information, cleaner api, 95% test coverage, etc
Without synchronization of test operations with UI updates the tests will be prone to flakiness; Espresso takes care of this for us.
Android Emulator
Espresso Hermetic Tests
Fake Login Server
Normal flow Espresso + Hermetic flow
Google+ Backend
Login Server
Android Emulator
UI Tests
FakeGoogle+ Backend
Android Hermetic UI Tests
Android Monkey runnerWhat is it?● The Android Monkey is a tool that generates pseudo-
random user events such as clicks, touches, gestures, as well as a number of system-level events.
Purpose● Stress test applications
Android Monkey runnerSome details about our setup:
● 10 devices with different API levels running 24/7● Exceptions clustering● Over 100+ bugs found (yes, human reproducible ones’)
Android “Feature” Leak Testing Why?● Because people are curious and like reverse
engineering android apk’s to search for unreleased features.
Example from Android Police:[APK Teardown] Google Games, Anyone? Play Services Is Getting Real-Time And Turn-Based Multiplayer, Invitations, In-Game Chat, Lobbies, Leaderboards, And Achievements!
Android “Feature” Leak Testing Google+ 3-step strategy:1. Flags library that removes unreleased features at
compile time. (Provides separation between a developer build and a release one.)
2. Proguard every version sent to production. 3. Continuously run a leak test that reverse-engineers the
release apk, looking for “Feature” leaks.
Android Tooling● Android lint errors displayed at changelist review time.
● Continuous monitoring of APK size and alert system.
● Monitor method count (Don’t get caught by the 64K method reference limit, two days before shipping.)
iOS Mobile Test Infrastructure for Google+ app
Unit tests
Hermetic UI tests
Monkeyrunner tests
Feature “Leak” tests
iOS Unit TestingGTM + OCMock
GTM (google-toolbox-for-mac)● Unit test suite based on GTMSenTestcase (psst, we are
considering switching to xctest in the future).
Why OCMock?● Very powerful framework for mocking Objective-C code
and also very flexible.
iOS Hermetic UI TestingKIF (Keep it functional)
Why KIF?● More natural than UIAutomation (Javascript based)● Since tests are run in-process, it allows tests to be more
tightly coupled with the app under test, making the tests inherently more stable.
iOS Hermetic UI TestingHermetic environment
We swizzle methods that fire HTTP requests in the app, and instead have them read off of an in-memory NSDictionary to get response data.
@interface FakeGoogleApisServer : NSObject { …NSMutableDictionary *_sequentialResponses; …NSMutableDictionary *_fixedResponses;...}
KIF + Swizzling = Hermetic Google+ iOS tests
iOS Monkey Runner● Tool developed to randomly navigate an application in
order to test for crashes.● In house tool, but we are planning to open source it.● KIF Based.● The Google+ team runs it mostly on simulators.● More than 50+ bugs catched by monkey● Experimenting with a smarter monkey that can leverage
randomized and deterministic actions.
iOS “Feature” Leak TestingLooking for unreleased features in iOS apps is not a common practice like it is in android, we like to take precautions
We have a test that:- unzips a provided .zip or .ipa- Checks for leak terms in filenames- Uses the class-dump tool and makes sure no leak terms are found.
Backend Server TestingServer and Client have different release cyclesGoogle+ server releases new versions of services multiple times per week. Mobile clients (iOS and Android) usually twice a month.
Different mobile clients app versionsGoogle+ servers changes needs to be compatible against older mobile clients.
Backend Server TestingWe decided to test the protocolWe follow a replay-test approach. The approach simulates mobile clients requests and makes sure the Google+ backend server returns the response the mobile clients were expecting yesterday, and the response they were expecting 6 months ago.
Backend server testing
Fake Data
Спасибо
Вопросы?