drive testing benchmarking v4b
DESCRIPTION
drive testTRANSCRIPT
-
NSO Service Delivery
Motorola Confidential
Page 1
UMTS Drive Testing & Benchmarking
-release 4-
Applies to TEMS 2.3.1 20 July 2004
Razvan Georgescu
-
NSO Service Delivery
Motorola Confidential
Page 2
Table of Content
Table of Content ......................................................................................................................................... 2 1. Introduction ...................................................................................................................................... 3 2. Executive Summary ....................................................................................................................... 3
2.1 Summary Report Format - Example ........................................................................................ 3 2.2 Deliverables ................................................................................................................................. 3 2.3 Tools ............................................................................................................................................. 3
3. Drive Test Setup ............................................................................................................................. 4 3.1 Call scenarios .............................................................................................................................. 4 3.2 Drive test routes .......................................................................................................................... 5 3.3 Duration ........................................................................................................................................ 5 3.4 Test Case Description ................................................................................................................ 5
4. Drive Test Analysis ......................................................................................................................... 7 4.1 TEMS Outputs ............................................................................................................................. 7 4.2 Formulas, Counters, Measurements ...................................................................................... 7 4.2.1 Formulas - Voice calls MO .................................................................................................... 7 4.2.2 Formulas - Data calls MO ..................................................................................................... 7 4.2.3 Voice calls MO TEMS Raw Output ................................................................................... 8 4.2.4 Data Calls TEMS Raw Output ........................................................................................... 9 4.2.5 Radio Health Indicators ...................................................................................................... 12 4.3 SMAP Outputs .......................................................................................................................... 14 4.4 NRG Outputs ............................................................................................................................. 14 4.5 Pareto Analyis Outputs ............................................................................................................ 14
5. Equipment List .............................................................................................................................. 15 6. Appendix ........................................................................................................................................ 16
-
NSO Service Delivery
Motorola Confidential
Page 3
1. Introduction
This document intends to give a general guideline for collecting drive test data with the primary scope of collecting KPI ( benchmarking) and secondary to collect basic call log data
2. Executive Summary
Summary Report Format - Example
CSSR (average) LOAD
RAB 0% Call Attempts Requirement
AMR MOC 98,25 343 97
AMR MTC 98,58 212 97
AMR MTM 97,78 270 97
AMR Total 98.18 825 97
CS 64 MTM 97,76 268 95
PS64 100,00 44 97
PS128 97.16 423 97
PS384 (static) 100,00 154 97
DCR (average) LOAD
RAB 0% Successful Call Setups GSD Req.
AMR 1,2219 1064 3
CS 64 0,57416 406 5
PS64 0 44 3
PS128 0.97 411 3
PS384 (static) 0 154 3
Deliverables
KPI counters
Logs for offline analyisis and post processing stored at a Motorola location and available on the intranet
Pareto for call failure causes
Tools
Note: At this date TEMS 2.4 is available, however no time for tests yet, release notes do not suggest changes in the event definition which could impact the KPI calculation described further in thei document.
-
NSO Service Delivery
Motorola Confidential
Page 4
Tools to be used:
TEMS Investigation WCDMA 2.3.1 , Motorola A835 phones with NETMON - UE logging and call counting
SMAP RNC logging NRG post processing
System Retainability & System Accessibility statistics will be generated from calls which are 2 minutes long for all types of calls, voice, video, data in order to align to the test calls done in Porto by GSD. This is to have KPIs deducted from the same type of calls in each location subjected to GSD aligned benchmarking. If this alignment is not needed, then the TEMS Practices document describes an alternative procedure which is also used by other major UMTS test equipment manufacturers.
3. Drive Test Setup
Call scenarios
All tests done in car and in motion. Voice calls & Video calls: 2 phones simultaneously MS1 calls MS2 both in the same car. This is an atipycal subscriber behaviour. This scenario is adopted though for testing purposes ,because we need to log with 1 TEMS kit both the call originating mobile and the paged mobile with the scope of collecting logs for debugging. KPIs only for paging can be deduced much easily only using UTRAN stats.
120 sec calls + 30 sec breaks
Data calls: - 1 phone
FTP, the data quantity adapted for each bearer speed. 2 minutes long + 30 seconds break
UL FTP tests also to be included
UDP as required
-
NSO Service Delivery
Motorola Confidential
Page 5
Drive test routes To ensure proper functionality validation of the UTRAN, the drive test routes for KPI benchmarking should be done in areas with proper coverage. For orientation we define the following values to be measured outdoor. o Good RF coverage, UTRAN Carrier RSSI > -70 dBm, CPICH Ec/Io > - 10 dB, 1/2/3 way SHO o Degraded RF coverage, UTRAN Carrier RSSI < -90 dBm, CPICH Ec/Io = -10 dB to 14 dB
Duration TBD. Depending on call volume. Can be from minimum 3 days to more than 1 week for the complete set of tests. Remark: Reports must be delivered weekly. This will be a live table where KPIs for different calls must be updated daily as the revelant test was performed.
Test Case Description All tests done in car and in motion. with the possible exception of 384K A description of the initial test cases is in the follwing table. Please treat this as a guideline to be adapted locally according to the needs and resources.
-
NSO Service Delivery
Motorola Confidential
Page 6
Which mobile
Scope of Calls
Call Sequence Description
Call Sequence Handling
Log Ue Log UTRAN
Post Processing
Minimum time needed for execution
Comments
Day 1, Voice Calls
5 hrs absolute minimum effective drive time
Practically takes whole day.
100 MTM voice call
MS1 Motion to MS2 Motion
CSSR, DCR, Pareto
120 sec call + 30 sec break
automatic
TEMS logging both mobiles
SMAP TEMSNRG
Day 2, Video Calls
5 hrs absolute minimum effective drive time
100 MTM video call
MS1 Motion to MS2 Motion
CSSR, DCR, Pareto
120 sec call + 30 sec break
manual
TEMS logging both mobiles
SMAP TEMS,NRG
Day 3, Voice Calls MO to fixed + Downloads
5 hrs absolute minimum effective drive time
100 MO voice call
MS1 Motion to fixed line (answering machine, or TEST number )
CSSR, DCR, Pareto
120 sec call + 30 sec break
automatic TEMS SMAP TEMSNRG
Need fixed line and answering machine or TEST number
100 Download sessions
MS2 Motion ftp ownload 128 K (or preffered service)
CSSR, DCR, Pareto
120 sec download + 30 sec break
automatic TEMS
Day 4, Uploads (+ optional MO to fixed)
5 hrs absolute minimum effective drive time
100 Upload sessions
MS2 Motion ftp upload 64 K
CSSR, DCR, Pareto
120 sec upload + 30 sec break
automatic SMAP TEMSNRG
TOTAL= 6 working days
-
NSO Service Delivery
Motorola Confidential
Page 7
Notes: 1. Day 5 & 6, Additional tests to be performed as needed on non preffeded data bearers - 64 K 384 K, UDP 2. TOTAL= 6 working days
3. The TEMS logging and UE control is done according to the general TEMS practices, for details please review Drive Testing and Benchmarking TEMS practices. 4. SMAP, NRG usage are subject to usage guidelines from GSD who are the tool developers. Any issues related to these tools should be directed to GSD. 5 .FULL BENCHMARKING of all services requires more than one working week with 1 team or alternative 2 teams or reduce call volume as per TEMS Practices if the performance is good and no alignment is required.
4. Drive Test Analysis
TEMS Outputs
Note: All these reports are generated using the Report Generator function in TEMS. Reports can be generated by anyone having TEMS instabbeld on his PC, without the dongle.
Formulas, Counters, Measurements
Formulas - Voice calls MO
CSSR = [Call Established / Call Initialization ]* 100 rounded at 2 decimals DCR = [Dropped Call / Call Established ] *100 rounded at 2 decimals
Formulas - Data calls MO
CSSR = [Session Start / RAS Dial ]* 100 rounded at 2 decimals DCR = [Session Error / Session Start] *100 rounded at 2 decimals
-
NSO Service Delivery
Motorola Confidential
Page 8
Voice calls MO TEMS Raw Output
Event #[no.of] #Log
Blocked Call 6 2
Call Established 75 1 ,2
Call Initialization 82 1 ,2
Call End 66 1 ,2
Call Setup 76 1 ,2
Dropped Call 9 2
Missing Neighbor Detection
0 -
Call Attempt 85 1 ,2
RRC Established 76 1 ,2
CSSR = [Call Established / Call Initialization]* 100 = [75 / 82 ]*100 = 91.5 % rounded at 2 decimals DCR = [Dropped Call / Call Established ] *100 = [9 / 75 ]*100 =12 % , rounded at 2 decimals
Setup Times
-
NSO Service Delivery
Motorola Confidential
Page 9
Data Calls TEMS Raw Output
Event #[no.of] #Log
Packet Service Attach 1 1
Packet Service Attach Failure
0 -
Packet Service Detach
0 -
PDP Context Activation
7 1
PDP Context Activation Failure
0 -
PDP Context Deactivation
7 1
RACH Error 0 -
RRC Connection Abnormal Release
5 1
RRC Connection Reject
1 1
RRC Established 6 1
RAS Error 0 -
RAS Dial 7 1
RAS Hangup 7 1
IP Address Assigned 7 1
Session Start 7 1
Session End 6 1
Session Error 1 1
A normal Call Sequence for Data Calls is:
DATA CALL SETUP PART
-
NSO Service Delivery
Motorola Confidential
Page 10
RAS Dial .. . based on the fact that the phone responds properly to the RAS Access from the PC PDP Context Activate IP Address Assigned Session Start .. .. FTP Get starts here .. I have successfully set up a Data Call, this is including also FTP server authentication which prooves that my data pipe is usable and I do not just have a blind PDP Context. If the call fails during server authentication before Session Start, this is technically a dropped call. The user may interpret it as a setup failure, that is why if the counters for IP Address Assigned and Session Start are not equal, these cases are recommended to be analysed by replaying the logs.
DATA CALL END PART Session End . The Data Transfer was successfull / or Session Error- Call is Dropped RAS Hangup release RAS from PC PDP Context Deactivated . end PS connection If Session Start counter = IP Address Assigned counter CSSR = [Session Start / RAS Dial ]* 100 =xy.z % rounded at 2 decimals If Session Start < IP Address Assigned , analyse and adjust the Call Setup figure and the DRC figure accordingly to compensate. Note: Low variance is to be expected here. Generally if IP Address Assigned but Session cannot Start then there is a functionality issue in the PS core or FTP server.
DCR = [Session Error / Session Start] *100 = xy.z % , rounded at 2 decimals Link Reestablishments :
these are meant to save the PS call if the RL fails, due to the specifics of the user data, such a procedure is not realistic for voice / video CS calls.
Reestablishments are not to be considered into the DRC statistics.
If the Reestablishments are too often, causes must be analysed. Visual investigation of the data transfer behaviour:
-
NSO Service Delivery
Motorola Confidential
Page 11
The 64 K bearer. On the graph it can be seen at 58 kbps app throughput (ftp) The 128 K bearer is reaching above 110 k the horizontal blue line is showing that. The 384K bearer is not really stable like the 128k and 64k. It suffers either from reconfiguration back to 128 triggered every few seconds ( without dropping, which is good) or when it is stable assigned the throughput stays usually above 250 kbps, mostly above 300. Setup Times Attach
-
NSO Service Delivery
Motorola Confidential
Page 12
PDP Context Activation
Radio Health Indicators
Below only some examples which can help you grossly evaluate the quality of the radio environment in a network.
-
NSO Service Delivery
Motorola Confidential
Page 13
-
NSO Service Delivery
Motorola Confidential
Page 14
SMAP Outputs
Documented by GSD on request.
NRG Outputs
Documented by GSD on request.
Pareto Analyis Outputs
Attached the original Pareto template click to open.
-
NSO Service Delivery
Motorola Confidential
Page 15
5. Equipment List
Item Equipment Description Qty Comments
1. Car 1 Renault Espace or similar MPV
2. DC/ AC 12V >> 100240 V inverter 1 For use with the equipment not running on accus or requiring a usage time longer than possible with own accus
3. USB Cables for A835 for Flash and Data 2 For Flashing and Data Transfer
4. Portable USB Hub 1
5. Serial GPS for TEMS 1
6. TEMS kits + 2 x A835 TEMS phones 1 For Drive Testing
7. A835 phones 1 per engineer
8. Digital Maps for TEMS
9. Laptops for drive testing 1 Min PIII with accus lasting more than 2 hrs.
10. Iu Analyser 1 For IuPS, CS, IuB Agilent
11. SIM cards 3 ( CS& 64 + CS&128 + CS&384)
12. Additional accus for A835 2 1 extra per phone
13. Dongles for RTA/PST/Radicom 4 1 per Motorola location
14. PSTN line 1 For Mobile Terminated Calls
15. NRG 1
16. SMAP 1
17. Laptop Holder 1 For incar
18.
-
NSO Service Delivery
Motorola Confidential
Page 16
6. Appendix Tools package available in compass at http://compass.mot.com/go/140807456