stressing a lte device with typical user behaviors
DESCRIPTION
Presented by: Artun Kutchuk on April 16, 2013 at 2013 North America LTE Forum by R&STRANSCRIPT
Stressing a LTE device with
Typical User Behaviors
April 16, 2013
Presented by:
Artun Kutchuk
Vice President, Business Development and Strategy, w2bi, Inc.
[email protected], +1.858.361.4870
© 2013 w2bi, inc.
Leading Provider of end-to-end automated device testing software solutions for LTE, UMTS/GSM, and CDMA handsets
to Tier 1 service providers and device manufacturers
Who is w2bi?
� Incorporated in 1997
� HQ in South Plainfield, NJ and R&D centers in NJ and India
� Sales presence in US, Canada, UK, Italy, Korea, Taiwan, and China
2013 North America LTE Forum 2
OEMs and ODMs
Mobile Operators OS and Chip ProvidersIndependent Labs
© 2013 w2bi, inc.
Smartphone Industry Challenges
� $6+ Billion/yr industry cost for device returns with No
Fault Found (NFF) or software Issue (67% of returns)*
� Cost of Quality (CoQ)�Impact on return rate �Impact
on margin
� Common Software Issues on returned devices:
• Device reset
• Application crashes
• Error messages
• Device lockup
• Launch failure
• Blank screen
2013 North America LTE Forum 3
Functionality
Application
OperatingSystem
Firmware
Key Reasons
*IWPC Device Return White Paper, Oct 2011
© 2013 w2bi, inc.
Mobile Technology Evolution & Diversity
2013 North America LTE Forum 4
Diverse Device Types
(HW/OS)
Diverse Services
Diverse Networks
Diverse Applications
This image cannot currently be displayed.
This image cannot currently be displayed.
This image cannot currently be displayed.
• 2012 - 70% mobile data traffic growth globally (885 peta bytes/month)*• 51% mobile video• 14% data on 4G (.9% of
connections = 60 million)• 18% of handsets =
smartphones
• 2017 - 45% data on 4G (10% of connections = 992 million)
*Cisco VNI: Global Mobile Data Traffic Forecast Update, 2012–2017, Feb 2013
© 2013 w2bi, inc.
Opportunities to assess impact on user QoE with cross layer variation
Current Test Methodologies
� App developers focused on software functionalities rather than
the device complexities or network impact
� Most of the apps are being tested on clean device with device
emulators or on static live networks
� Device testing for RF & Signalling conformance, data
throughput, SMS, core services, etc.
� Standards based testing focused on perceived user Quality of
Service (QoS) with no device variability
� Operator tests focused on their specific network and service
capabilities
2013 North America LTE Forum 5
R&D TestsLab
CertificationInteroperabi
lity TestField Test
Returned Device Test
Industry Certification
© 2013 w2bi, inc.
Impact of LTE Introduction on User Experience
2013 North America LTE Forum 6
� LTE advancements are heightening user expectations of
service quality and performance
� Device technology evolution is broadening
the types of mobile applications
� On the device cross layer
interactions are increasing
� What is the impact of small cells,
band combinations, VoLTE, Carrier
Aggregation, multi media collaboration, etc. under realistic
usage scenarios on device and user experience?
Opportunity to stress device and applications with realistic scenarios to discover software related issues early!
© 2013 w2bi, inc.
What is Device Stress Test?
� Real and practical user behavior scenarios executed
repetitively under different conditions:• Varying device memory and CPU loads, and battery levels
• Background and foreground device applications
• Live network (4G/3G, Wi-Fi)
• Emulated network
• Varying mobility and RF conditions, e.g. fading,
handoffs, carrier aggregation (CA), multiple bands
• Structured (i.e. predefined user behavior) or
unstructured (i.e. random)
� Capturing and verifying:• Failures (i.e. applications, device reset, etc.)
• Battery impact
• Network impact (i.e. data transfer, signaling)
• Quality of Experience (QoE) – responsiveness, buffering,
latency, throughput, jitter, etc.
2013 North America LTE Forum 7
© 2013 w2bi, inc.
Typical Stress Test Cycle
Choose Test Type
•Record a Test Plan
•Specific Sequence
•Randomly choose a sequence
Edit Test Sequence
•Add a setup script
•Edit specific test plan
•Edit order of plans
Add System Stress
•CPU/Memory Hogging
•Data Streaming, I/O
•RF, iRAT stress, etc.
Playback Setup
•Privacy sweep
•Capture existing configuration
•Load starting application and content configuration
Playback Execution
•Execute test plans
•Capture failure logs
•Capture KPIs
Playback Iteration
•Reset device to starting configuration
•Execute Playback
•Repeat X times•Repeat Y Devices
Re
co
rdP
lay
An
aly
ze
Run Basic Analytics
•MTBF/MTFF Reporting
•KPI Isolation
•CPU, Memory, Temp
•Batt. Level, Data Usage•RSSI, Processes
•KPI Comparison
Run Advanced Analytics
•Trending of failures
•By OS version
•By manufacturer
•By SKU•By hardware profile
•Isolation of specific failure trend
Proof of Correction
•Deep testing on isolated issue
•Work with OEM/ISV
•Verify isolation correction
2013 North America LTE Forum 8
© 2013 w2bi, inc.
Value of stress test during device lifecycle
2013 North America LTE Forum 9
Pre-Deployment Testing Post-Deployment Testing
R&DLab
CertificationPost-Market
ReturnsField
CertificationIn-Market
Triage
• Mobile Chipsets
• Mobile OS• Device
Integration• Mobile Apps
• Standards• Operator
Specific
• Interoperability• Operator
Specific• Core Services
& Applications
• Vertical Industry or Enterprise Specific
• Point-of-Sale Diagnostics
• On device self Diagnostics
• Device field data collection
• Repair CenterDiagnostics
Stress Test Emulated Emulated Live Live Emulated
Relative Duration
Network
© 2013 w2bi, inc.
What type of results should we expect?
� Rapid identification of device or application failures
under typical usage scenarios
� Impact of applications on the battery, CPU, memory
and network (e.g. data traffic, signalling, etc.)
� Identification of rogue applications on customer
commercial devices
� Comparison of device and network measurements
� Benchmarking of devices across: • Diverse networks and mobility conditions
• Varying device loads
• Different operating systems and versions
� Detailed device logs on failure conditions, or in
significant device or network changes
2013 North America LTE Forum 10
© 2013 w2bi, inc.
Sample Stress Test Results – Failures
Consistent App Problem
2013 North America LTE Forum 11
© 2013 w2bi, inc.
Sample Stress Test Results – App Device Impact
2013 North America LTE Forum 12
© 2013 w2bi, inc.
Sample Stress Test Results – App Network Impact
2013 North America LTE Forum 13
© 2013 w2bi, inc.
DUT automation
Live or Emulated
MHL / HDMI
stream
Typical Device Stress Test Setup
R&S VTEvideo
analysis
R&S CMW500LTE/HSPANetwork
under fadingconditions
w2bi QuikStress:DUT Application
Control,Benchmarking & Failure Analysis
2013 North America LTE Forum 14
EmulatorControl
© 2013 w2bi, inc.
Improve device quality and device return rate with NFF
2013 North America LTE Forum 15
� Adapt end user behavior scenarios in the overall smart device
test strategy
� Embrace the end-user device feedback into early part of
testing and certification process
� Take device application neutral testing approach to reduce
cycle time and integration
� Introduce stress tests at integration, field test, certification
and device logistics
phases