best practices for performance testing mobile apps
DESCRIPTION
With an ever increasing amount of traditional, web-based content being accessed from mobile devices performance testing of mobile apps has become critical. According to the 2012-2013 World Quality Report, 64% of firms identified performance as a focus for mobile testing over both functional testing and security. It’s no wonder why firms are placing this degree of focus on mobile performance. Consumers expect their mobile content to be delivered as fast as it would be through a desktop / high bandwidth connection and they won’t hesitate to find other options when it’s not. On the surface, performance testing of mobile apps may appear to be no different from web apps. However, applying the same testing techniques used for traditional web-based systems may produce inaccurate results, or fail to expose performance issues. Mobile apps present unique differences and challenges that must be addressed to ensure performance tests accurately represent production user behavior and load profiles. This session will discuss how to address unique mobile app challenges such as device / OS diversity, mobile user behavior and network variability. Attendees will walk away with an understanding of best practices for ensuring the performance of their business critical mobile applications.TRANSCRIPT
© Utopia Solutions
Best Practices for Performance Testing Mobile Apps
Lee Barnes, CTO Utopia Solutions
© Utopia Solutions
Apology
I don’t believe in “best practices”…
I do believe in guidelines and better approaches for a given situation
© Utopia Solutions 3
Key Questions
How is mobile different?
How can I adapt?
Why is performance of mobile important?
© Utopia Solutions 4
Why Should I Care?
M-Commerce
66% OF SHOPPERS ABANDON TRANSACTIONS
5 OUT OF 10 BECAUSE OF POOR PERFORMANCE
1 DELAY SECOND 7% DROP IN
CONVERSIONS equals a
2013 Mobile Commerce Insights study conducted by Jumio
© Utopia Solutions 5
Still Not Convinced?
2012-2013 World Quality Report
© Utopia Solutions 6
Evolution of Performance Testing
1990 2000 2010
2-Tier Client Server Web Mobile
Challenges
State of the Practice
Testing Large Loads
Complex / Integrated
Systems
Proprietary Protocols
Skilled Resources
In-Lab Testing
Tool Maturation
Open Source Tools
Specialized Resources
Cloud-based Testing
© Utopia Solutions
Mobile Challenges & Adaptations
© Utopia Solutions 8
Mobile Challenges – User Access
Native App Mobile Site Full / Responsive Site
Your users have options…
© Utopia Solutions 9
Assess the User and Load Profile
1. Determine the user profile
• Business processes • Native app, mobile site, full site • Preferred browser • Network conditions • Geographic location
2. Determine the load profile • Volume of users by business
process and location • Volume of users by access type • Frequency / throughput
Incorporate constant feedback from application logs and other monitoring solutions
© Utopia Solutions 10
Mobile Challenges - Network
Mobile User Connected User
3G 4G InATunnelG
Network bandwidth and quality affect both end user experience and system load
© Utopia Solutions 11
Assess Impact of Network Variability
1. Assess baseline performance (single user)
Observe UX with real devices behind network virtualization solutions
2. Assess performance under load
Put system under virtual user load
Observe UX with real devices behind network virtualization solutions
NV
Virtual Users
NV
Real Devices
© Utopia Solutions 12
Network Virtualization Tools
Single User (Device and/or Emulator) • Apple Network Link Conditioner • Android Emulator • Charles Proxy • Shunra NV
Multiple Virtual Users • Shunra NV • Load test tools (built-in) • Charles Proxy
© Utopia Solutions 13
Mobile Challenges – User Location and Volume
A high volume of global users is difficult to reproduce in a lab…
© Utopia Solutions 14
Utilize Cloud-based Testing Solutions
• Achieve production level loads • Test entire infrastructure (vs.
behind the firewall)
• Distribute load geographically • Assess user impact on real
devices
© Utopia Solutions 15
Mobile Challenges – Device Performance
Measuring System / Network Metrics is Not Enough…
Memory
CPU
Battery Drain
Graphics
I / O
© Utopia Solutions 16
Measuring Device Performance
Critical device metrics: • CPU • Memory • Battery • Storage
Mobile development platforms • Android DDMS • iOS Instruments
Mobile test automation tools • TouchTest (SOASTA) • Trust (Mobile Labs) • MonkeyTalk (Cloud Monkey)
Mobile lab / cloud solutions • Device Connect (Mobile Labs) • Keynote DeviceAnywhere • Perfecto Mobile
Capture metrics while exercising app
Include appropriate devices / configurations
Incorporate automation for consistency
Device Performance Measurement Solutions
© Utopia Solutions 17
Mobile Challenges – Agile Development
Analysis
Design
Develop & Unit Test
Acceptance Test
Final Acceptance
Test
Deploy
Source: Agile and Test Estimation Sharon Robson
How does performance testing fit?
© Utopia Solutions 18
Performance Testing in an Agile Environment
Plan Design Build Test Deploy
Waterfall
Performance Testing
Too late!
© Utopia Solutions 19
Performance Testing in an Agile Environment
Sprint 1 Sprint 2 Sprint 3 … Deploy
Agile
Performance Testing
Same Process…
Same Risk!
© Utopia Solutions 20
Performance Testing in an Agile Environment
Sprint 1 Sprint 2 Sprint 3 … Deploy
Agile
Performance Testing
Performance Testing
Performance Testing
Performance Testing
Waterfall Approach • Rigid • 4-6 week test cycle
Agile Approach • Flexible • 2 week sprints
+ =
© Utopia Solutions 21
Early Cycle / Ongoing
Performance Analysis
Performance Testing in an Agile Environment
Ultimate Goal…
Late Cycle Performance
Testing Shift Left
© Utopia Solutions 22
Performance Testing in an Agile Environment
How? Map business and end user requirements onto system components
Benchmark in development / QA environment
Look for trends in measurements vs. absolutes
Use service virtualization to stub out 3rd party or yet to be developed services
Execute unit performance tests as part of continuous integration
Unit / Component Performance Testing
© Utopia Solutions 23
Performance Testing in an Agile Environment
How?
Measure resource / performance profile of native apps
Use mobile site analysis services / tools to analyze services and pages
User APM solutions to profile code
Use device profiling solutions to understand / compare base device performance
Application / Code Profiling
© Utopia Solutions 24
Waterfall Agile Service Model Perform turnkey
performance test Timing End of development cycle Duration Weeks Tools Traditional load test tools
Environment Dedicated / production like Role of development
Throw system over wall Wait for results
Waterfall Agile Service Model Perform turnkey
performance test Provide mentorship to agile team
Timing End of development cycle Ongoing Duration Weeks Hours / days Tools Traditional load test tools Traditional load test tools
APM tools Service virtualization Continuous integration
Environment Dedicated / production like Development / desktop Role of development
Throw system over wall Wait for results
Constant focus on performance
Performance Testing in an Agile Environment
What about my Performance Testing Center of Excellence?
© Utopia Solutions 25
Summary
Key takeaways… • Mobile users are not the same as
connected users • Performance is more than backend /
network • Push performance analysis to the left
in the development cycle
© Utopia Solutions 26
Questions…
…. and Answers!
Direct future questions to: Lee Barnes Founder and CTO Utopia Solutions, Inc. Email: [email protected] Twitter: twitter.com/USI_LeeBarnes LinkedIn: linkedin.com/in/leebarnes Blog: www.RaiseYourMQ.com
?