mobile monitoring for peak events
TRANSCRIPT
Grand National 2015Mobile application performance insightsHow to get monitoring right with millions of users
Perfecto Mobile The world leader in mobile application continuous quality helping the world’s biggest brands to deliver better apps faster
“Perfecto Mobile currently has the strongest 3rd party position in the market” [ Thomas Murphy]
Intechnica• Vendor/Technology Independent• Enable Performance by Design• Provide Performance Management Services• Specialists in Agile Performance• Enable and deliver Performance Best Practice
Your presenters:
Larry HaigSenior Performance Consultant, [email protected]+44 (0)845 054 2381Contributing author: The Art of Application Performance Testing[Molyneaux, 2nd Ed 2014 Pub O’Reilly]
Amir RozenbergDirector, Product Management, Perfecto [email protected]
Agenda
Grand National: Fast Facts
Our Findings
The Next Race
Best Practices
Demonstration
vv v
v v
Grand National 2015 – fast facts
• OpenBet record 19,300,000 bets within 24 hours– Peak 45,000 bets pm– 54,000,000 account transactions [UK population c63m (2011)]
• + 24% y-o-y– Average bet size is £8.00 – 63% of remote bets via mobile, – [up from 54% 2014]
• UK Transaction volume +12.6% y-o-y– (+83.6% vs typical April trading day)
• UK Transaction value + 20.3% y-o-y Refs: OpenBet; Worldpay
Survey & caveats• Focus on Mobile perspectives, end user/client side issues
– Android and iOS– Complementary testing:
• Mobile emulation / public carrier (4G) / webkit browser (m. sites)• Real device testing (native apps)
• Raceday & 2 day buildup• High vs low traffic• Performance and Quality of Service• Caveats:
– Snapshot testing – indicative, not absolute / continuous– High level monitoring only
• No device metric correlation – Real device test redundancy limited
• Native App availability not quoted
Example – race day errors log
Summary of key findings• Overall performance improving y-o-y, BUT• Still evidence of traffic related stress• All m. sites (and some native apps) showed some evidence of Quality of Service
issues– 85%+ likely to be revenue relevant– Brand impact
• m.sites:– iPad2 Tablet slower than Smartphone– 3rd party content source of failure– Lack of traffic resilience– Transaction payload highly correlated with performance
Summary – traffic & payload:
Summary – traffic & payload:
NATIVE MOBILE APPLICATIONS
Native App Response Times by Device- The Good
4.23s 2.59s 2.91s 8.06s
5.16s 9.32s 5.67s 1.48s
Web Emulation = 1.15s
Poll
v
Impact of 5 second delay in a revenue transaction
• I would call customer support• I would complain on social networks• I would transition to competitor application• I would try again
Poll Results
v
Impact of 5 second delay in a revenue transaction
I would call customer support
5%
I would complain on social networks
5%
I would transition to a competitor application
56%
I would try again35%
Native mobile applications – maximized revenue?
Mobile Advertising on app launch takes user attention
Application behavior in absence of network- opportunity to optimize experience
Native applications Panel performance – ‘Time to Bet’
Panel # Android iOS
10 5.0
7 8.9
8 5.1 3.4
9 5.9 1.5
13 10.2 5.0
< 5 sec
>5 - < 10 sec
> 10 secRaceday median (seconds)
Performance consistency (raceday):
0600hrs 11 AprilTran
sacti
on re
spon
se (s
econ
ds)
Example root cause analysis
0
10
20
30
40
50
60
70
80
90
100
0
50000
100000
150000
200000
250000
300000
350000
400000
CPU User (%)#Device CPU Total (%)#DeviceBytes In - WLAN#Device
Rapid find & fix - root cause analysis - drilldown
• Network wait: Client side processing
• Large JavaScript D/L: Repeated downloads
• Significant advertising/tracking/3rd party activity
Brand impact [race day errors]:
PLANNING FOR THE NEXT PEAK EVENT
Digital Experience Unhappy PathNew Use Cases
What will the next race look like?
Quality of Experience- Adopt the user “glass”
Server Architecture Delivery Chain Devices & User
3rd party
Device Emulation UX Timer App-level CPU Peak
iPhone 6 0.9s 1.15s 67%
iPhone 4S 0.9s 8.9s 85%
Typical Application Launch Time
Recommendation 1
Daily Weekly
Sensors (NFC, BT)
Events
Incoming (Call/SMS/Alert)
Location
Rotation
WiFi On/Off
Airplane mode
Dynamic Backend Switching
Light/Heavy Load
Device Stress
Restricted Permissions
Environments
Network Conditions
Sensor allocation
UI AssetsOS & App-level Vitals
Optimization Directions
Reporting
Rich Media QoE
KPIs
Network Traffic
Continuous Quality Lab
Performance TestingAdvanced Functional Testing
Recommendation 2
Test, Optimize & Get Ready
Recommendation 3
Alerting Drives Early Awareness (MTTK) &
correction (MTTR)Real DevicesCoverage
Geos/Scenarios/Carriers
Remain in control:Know early, resolve early, and keep winning!
Key suggestions for monitoring1. Understand customer devices, locations, trends2. Monitor appropriately and manage all digital delivery channels
– PC, m.– Native mobile applications– Other (in store kiosks, etc.)
3. Lean is fast – manage page weights in m. sites4. Audit & optimise client side/application components5. Explicitly monitor object rendering6. Understand device specific effects 7. Manage 3rd party inclusions
– SLAs, monitoring8. Plan ahead
– Understand traffic load effects ahead of time– Baseline key metrics
www.intechnica.co.ukwww.perfectomobile.com
Thank you.