tems moving from pesq to polqa

MOVING FROM PESQ TO POLQAMOVING FROM PESQ TO POLQA The Next-Generation Mobile Voice Quality Testing Standard

Dr. Irina Cotanis, on Moving from PESQ to POLQA 1

MOVING FROM PESQ TO POLQA

Why is a move from PESQ to POLQA recommended? (And, why will the move eventually be necessary?)

What are the limitations of the POLQA technology? What are the main differences between POLQA and PESQ? What are the implications of moving to the new POLQA technology? What are the implications of moving to the new POLQA technology?


WHY MOVE FROM PESQ TO POLQA?

Reason 1. POLQA fixes known and recognized PESQ weaknesses.

WB measurements in general Inaccuracy with CDMA codecs (e.g., EVRC) and, therefore,

CDMA t k i lCDMA networks in general

Sensitivity to certain GSM/WCDMA network conditions (DTX/Voice Activity Detection in AMR codec) ( y )

VoIP limitations (variable delay up to 1sec) Strong linear distortions (e.g., phones frequency shaping)

as well as gain variations

Limitations with speech processing (enhancement) devices (noise reduction automatic gain control)(noise reduction, automatic gain control)



Reason 2. POLQA has new operational features.

Works in all three available bandwidths (NB, WB, SWB) and under different background noise conditions (street, car, restaurant, babble, impulse, etc.)

W k ith ll i l d t d di d d d i diff t Works with all new commercial and standardized codecs used in different technologies (GSM/WCDMA, LTE, VoIP, VoIP over IMS) such as AMR-WB, EVRC-WB, iLBC, AMB+, AAC, Skype / SLIK, G.711, G.729, etc.

Allows measurements at various combinations of interfaces: electrical-electrical, acoustical-acoustical, electrical-acoustical, and acoustical electrical providing the opportunity to test terminals andacoustical-electrical providing the opportunity to test terminals and hands-free applications



Reason 3. POLQA copes with new types of degradations.

New codec conditions Single speech codecs and speech codecs working in tandem, packet loss

d l t t t i (PS) f d bit ( i l )and concealment strategies (PS), frame errors and bit errors (wireless), effects of coding on pre-noised speech

Network specific degradationsp g Interruptions (e.g., unconcealed packet loss or handover in GSM),

front-end clipping (temporal clipping), amplitude clipping (overload, t ti ) ti i /ti li (V IP IMS)saturation), time warping/time scaling (VoIP over IMS)


Continued on next page


Reason 3. POLQA copes with new types of degradations. (continued)

Voice Enhancement Devices (speech processing systems) implemented both in the CN as well as in the terminals

Eff t f i d ti t d h ll l h Effects of noise reduction systems and echo cancellers on clean speech as well as on pre-noised speech

Terminals Influence of time variant linear distortions (spectral shaping), non-linear

distortions produced by microphone/transducer at acoustical interfaces

Reverberations caused by hands-free test setups in defined acoustical environments



Reason 4. POLQA has been proven to over-perform PESQ.

POLQA NB mode has shown increased performance of up to 25% as compared with PESQ NB

Similar values haven been observed for POLQA SWB modeas compared with PESQ WB


WHAT ARE POLQAS LIMITATIONS?

STRICT LIMITATIONS. POLQA should be not used at all for: Other dimensions of speech quality such as conversational aspects and talking quality Speech quality per call. POLQA is not intended to score longer sequences of speech.

It is focused on prediction of quality for shorter speech utterances of 6 to 12 secondsIt is focused on prediction of quality for shorter speech utterances of 6 to 12 seconds

Noisy listening environments. POLQA does not predict perceived speech quality in these environments; it is designed in accordance with P.800, ACR testing.

Music (including multimedia) Evaluation of performance or ranking of voice enhancement devices

(e.g., noise suppressors)

Other technologies or components such as speech storage formats or non-telephony applications such as public safety networks or professional mobile radio connections



WHAT ARE POLQAS LIMITATIONS? (continued)

POSSIBLE LIMITATIONS (scenarios on which POLQQ has not been tested/evaluated). In the following use cases, POLQA should be used carefully. Additional subjective testing might be needed for increased confidence.

Other languages (e g Arabic or others not used for POLQA development) Other languages (e.g., Arabic, or others not used for POLQA development) Longer speech samples Multi-dimensional audible speech quality diagnosis p q y g

NOTE. Work on Perceptual Approaches for Multi-Dimensional Analysis (PAMD) to predict perceptual dimensions of degradations linked to the overall speech quality in NB, WB, and SWB telecommunication scenarios is ongoing g p q y g gwithin ITU. PAMD aims at providing more detailed information about individual quality dimensions as additional information to the POLQA overall MOS. It should be noted that this is different than diagnosing network problems based on speech quality parameters as provided by POLQA.


WHAT ARE THE MAIN DIFFERENCES BETWEEN POLQA & PESQ?

Reference speech material: 48kHz vs. 8kHz sampling frequency MOS-LQO output: POLQA score directly represents estimated MOS values on

a subjective scale of 1 to 5 MOS, unlike PESQ, which used the P.862.1mapping function for conversion to the MOS domain

Dual operational modes: Dual operational modes: Narrowband NB (IRS filtering up to 3.4kHz; 8kHz sampling frequency of the

reference sample) For backwards compatibility to PESQ recommended to be run only for this scope

Super wideband SWB (Flat filtering up to 14kHz, 48kHz sampling frequency of the reference sample)of the reference sample)



WHAT ARE THE MAIN DIFFERENCES BETWEEN POLQA & PESQ? (continued)

Unique triple bandwidth output scale: SWB mode: NB, WB, and SWB tested samples presented on a single SWB

scale (bandwidth being part of the transmission chains degradation)scale (bandwidth being part of the transmission chains degradation)

This has the advantage of direct comparison of different bandwidth scores.NOTE: It is generally expected that the SWB unique scale (running a SWB reference sample)

has the following transparency:

NB: Maximum MOS value 4.25

WB: Maximum MOS value 4.5

SWB: Maximum MOS value 4.75

New extensively tested and validated languages: Chinese, Japanese, CzechAccuracy: POLQA proven to over perform PESQ (both NB and WB) Accuracy: POLQA proven to over-perform PESQ (both NB and WB)


WHAT ARE THE IMPLICATIONS OF MOVING TO POLQA?

Possibility to achieve slightly lower spatial-temporal granularity of the speech quality measurements

Longer processing time than PESQ due to algorithm complexityL i t d t 48kH li f Larger memory requirements due to 48kHz sampling frequency

Proper understanding of the SWB unique scale, which provides three bandwidth scores on the same scale

Backwards compatibility to PESQ Valid only for PESQ NB. Provides rankings rather than exact absolute values Comparison should be made only between PESQ NB and POLQA NB


CONCLUSIONS


CONCLUSIONS

The move from PESQ to POLQA is recommended (and will eventually be required) due to:

POLQA fixes well-known weaknesses of PESQ POLQA ith t d h d i h t d i POLQA copes with todays speech codecs, voice enhancement devices, and network conditions

POLQA offers significantly higher accuracyQ g y g y

POLQA technology has still some limitations that must be well understood and controlled



CONCLUSIONS (continued)

Main differences between POLQA and PESQ:

Reference speech sampling frequency Operational modes / bandwidths Unique triple bandwidth MOS LQO output scale Accuracy

Implications of moving to POLQA:

Backwards compatibility to PESQp y Possibly lower spacetime granularity than PESQ Understanding the unique SWB output scale as well as its difference from

the NB scale


THANK YOU!


tems moving from pesq to polqa

Documents