Speed to Market in Mobile Development

Finding the right solution with continuous integration on real devices

Agenda Mobile trends & developers’ challenges Speed & quality through continuous integration

- Intuit Virtual Device Lab- Intuit mobile test automation

Takeaways

Aggressive mobile momentum

“2013 Internet Trends,” KPCB

Will surpass desktopby 2014

It should just work!

Proliferation of platforms, OS versions & device types

Developer challenges• Immense configuration matrix

– OS platforms (Android, iOS, Windows, phone vs. tablet, …)– Inputs (touch/gesture, real buttons, voice, camera, GPS receiver)– Outputs (portrait/landscape, screen resolution, language)

• Application delivery through the markets– App signing, provision (app & device), submission

• Offline experience– Network, different network bandwidth/latency, no-network– HTML5 offline app

• Media challenges– Some codecs don’t work on a device– Streaming, sound

• Security & privacy

Intuit mobile app portfolio todayApple

Google

Amazon

SMS

0 5 10 15 20 490

22

15

11

3

491 mobile apps for financial institutions

Ramping up Intuit’s innovation engine Pain: Takes days per build to test combinations of device/OS configurations physically & manually, one by one …QA can’t keep up with the changes

Goal: Reduce development iteration cycle from days to minutes on an ever-growing number of devices

Strategy: Speed & quality through mobile continuous integration

• iPhone 4• iPhone 4S• iPhone 5• iPad 2• iPad 3• iPad Mini

Devices

OS versions

• iOS 5.0• iOS 5.1 • iOS 6.0• iOS 6.1.x• iOS 7.0

Builds

X X

Daily builds

• Apple• Google• Amazon

Platforms

X

Continuous integration (CI)What is CI?

• Automate build, test & deployment process• Integrate & build early, often, daily, for every check-in• Best practice of Agile development• Jenkins is widely adopted, flexible & extensible

Benefits of CI in a mobile world

• Instant feedback on quality, functionality & system impact• Less complex, no integration hell• Detect and address device configuration issues early• Involve stakeholders early• Ultimately achieve shippable release any time

Test results

Speed and qualitythrough mobile CI

Key enablers:• Mobile device lab• Test automation

Test processorReal devices

Jenkins

SCM: source repository

Developers/IDE

Automatic test run

Submit the app to app store if passed

Check in

Check-in triggers

automated CI

Install the test app toreal devices in parallel

Run tests on real devices in parallel

Analyze test results

Trigger testing in mobile device lab

Process starts here

3rd-party vendor solutions didn’t work for us• Couldn’t handle E2E automation of testing pre-release apps

– Dependency on internal resources & infrastructure

• Vendor lock-in: customized CI & test automation solutions– No flexibility to use our existing tools & frameworks

• Disadvantages of vendors’ shared devices– Security concern on testing on shared devices– No access to corporate network– Cost-prohibitive to reserve devices

• Private cloud (on-site setup) with local devices not practical– With sensitive customer data, we need private cloud – Extensive use of devices for daily testing of hundreds of mobile apps is expensive

Why test on real devices?Most mobile features can be tested on simulators, but simulators have limitations:

• Device capabilities (GPS, camera, accelerometer, gyros), form factors & pixel density

• Real-world experience over the real connection(3G, 4G, crappy Wi-Fi, etc.)

• True performance, security tests• Testing on non-default Web browsers (e.g., Opera Mini)• Not all simulator/emulators have good quality

(don’t reflect real rendering/behavior, bugs)• Testing SMS & app integration

Intuit Virtual Device Lab

Intuit Virtual Device Lab (VDL)Gives access to real mobile devices using a browser

Intranet

VDL

User machine

User machine

User machine

3-click access

Always on

Integrated with dev infrastructure

Globally available

iPhoneiPad

GalaxyCaptivate

Optimus Nexus OneDesireIncredible

Manufacturers/devices:

DroidMilestoneMoto

VDL architecture – Android

Windows PC

Tomcat

Java image transferring application

Browser & Java applet

Internet/intranet

Device interaction app

Developer’s laptop

VDL

Image transfer

USB connection

VDL architecture – iOS

Mac with Xcode installed

Tomcat

Java VNC client

Port forwarding

Browser & Java applet

Internet/intranet

VNC server

USB connection

Developer’s laptop

VDL

Open source tools & technologies used in VDLKey capabilities Android iOS

Installing mobile applications into the real device

adb install <APK file> Fruitstraphttps://github.com/ghughes/fruitstrap

Transferring device screen images, controlling device

Screencast toolhttp://code.google.com/p/androidscreencast

Veency VNC app

Fetching device local files adb pull <filename> sftp command(jailbroken devices only)

VDL key features & functionality today Control the device through standard browser/RESTful APIs

Change orientation to landscape or portrait

Copy local file to the device from your PC

Get file from the device onto your local PC

Execute an ADB command on the device

See current running logs in your browser

Download history logs to your local PC

Continuous integration – how it works

Product builds

Trigger file launches test

VDL device inventoryResource

manager

Select a device

Test executionengine

Results

Monitor for new

builds

Mobile continuous integrationLeveraging the Virtual Device Lab for test automation on multiple, real devices simultaneously.

Submit, build, test … in minutes vs. hours

Select devices & tests1 Tests run automatically across selected devices with each build

2 See the results!3

Next step: over-the-air replaces USB connection

Fast onboarding

Scalable

Any device

Internet/intranet

App installation

Test execution

Results retrieval

VDL

Mobile Test Automation

Mobile test automation tools

Instrumentation-based

Non-instrumentation

AndroidiOS

MonkeyRunner

MOET

Sikuli

Android SDK

Robotium

TestDroid (BitBar)Calabash

(LessPainful)

Soasta

iOS SDK

UIAutomation

UISpec

Frank

MonkeyTalk

DA/PerfectoMob

ileeggPlantCross-platform solution

iOS Driver Appium

• Assessed as of 2012

Mobile test automation trends• Abstraction with domain-specific language (DSL)

– Common test script over different devices (e.g., Cucumber, MonkeyScript, MOET)– Support behavior-driven development (BDD)

• Client/server-based approach (remotely driven tests via HTTP)• Instrumentation & non-instrumentation solutions complement

each other• Mobile cloud testing environment

– From hardware (cradle) to software approach to drive tests onreal devices

• Automated OTA app installation & tests

Intuit automation framework of choice/recommendation

• Native apps:

– Cucumber + Calabash for BDD on iOS & Android

• Mobile Web:

– Selenium/WebDriver

What is Cucumber?It lets stakeholders (customers, QE or developers) describe how software should behave in plain text.• BDD: Written in business domain-specific language (DSL)

– Good abstraction over different device types• One format for multiple purposes

– UI feature specification – Automated tests

• Cucumber itself is NOT a test automation tool

– Add-on to any test automation framework– Examples: Calabash, Sikuli & Frank mobile test

frameworks

What is Calabash?• Open source cross-platform mobile test framework

– iOS + Android, simulator + real device, native + hybrid• Supports Cucumber for BDD• Application needs instrumentation that embeds a Calabash HTTP server

– Tests run from a remote client (in a Jenkins slave)– Tests can be driven via Wi-Fi (no USB wire)

• Ruby clients in API level (Java clients in experiment)• Works great in touch simulation & real device testing

Intuit shared test library – rapid test development

Internal open source repository

Teams focus on creating

Use shared test lib

… and then contribute to common test library

Tests

Contribute

Tests

Common test library Product tests

TestsTests

Globalization test framework(I18N/L10N)

Performance/load test framework

Security test framework

Native/hybrid test framework

High-level test domain-specific language(Cucumber steps)

Mobileback-end

Mobilefront-end

Codebuild

Deploy

Test

Results

Connect device

Notifyemail

Getdevices

Consolidatereport

Uploadreport

From: manual

50*5= 250

15 30

5

10

20

15

30 10

= 6 hours

Codebuild

Trigger Execute tests in parallel

Consolidatedreport in archive

Deploy on devices

0 5 10 0

= 15 mins

To: automated CI

TIME SAVED: 95%Release quality increased

*

Stunning results: reduced iteration cycle from hours to minutes

Tested on 5 devices

VDL cost saving over vendors’ private clouds (today)

Annual cost ($)

# of devices

Intuit Virtual Device Lab

150

3rd-party vendor 1 local device in cradle

Assumptions: 50 native apps & daily check-in

Note: As vendor pricing changes, so do the savings

Savings

25

3rd-party vendor 2local device w/o cradle(Software approach)

Savings

Private cloud setup behind corp firewall

Key takeawaysBuilding custom solution vs. 3rd-party vendor solution

Understand your technical/platform/global spread Many vendor solutions provide a larger set of devices/OS versions

in the global market Quite a few offer mobile test services – might be tightly coupled with their

device access solution & automation framework Connecting to your corporate network is a challenge – some provide private

cloud (on-site setup within your network) but it’s cost-prohibitive

Invest in automation and CI Automate tests with robust automation framework Enable rapid test development through shared test library Adopt CI in mobile development for speed & quality

Thank you!!