standards for speech and media quality...

Standards for Speech and media Quality assessment

Jean-Yves MonfortETSI TC STQ Chair

© ETSI 2010. All rights reserved

ETSI Workshop 3GPP release 8 IMS Implementation, Deployment & Testing

Outline

� Two topics� Testing Terminal Quality to enhance the end-to-end speech Quality� Quality assessment of Multiplay services

� How testing different VoIP (or DECT, Mobile) termin al implementations to prove and improve perceived qual ity and optimize performance?

2

� From the users point of views the quality shall be assessed including terminals

� ETSI standards defining terminal requirements are i ntended to ensure best possible quality

• Test events, Plugtests,… are based on standards for quality assessment• Results of test events provide feedbacks on require ments vs real

implemenations �� standards requirements evolvements.

� Criteria for quality assessment of multiplay servic es� Exemple of comparison of ISP VoIP: Interoperability testing� Future plans for IPTV

Part 1 : STQ Standards for speech terminals-Narrowband (300-3400Hz)- Wideband (150Hz- 7kHz)

3

Handset and headset HandsfreeTransmission requirements for VoIP terminals from a

QoS perspective as perceived by the user

Links with other standardization bodies and certification activities

� ETSI STQ has disseminated the deliverables to other bodies� TS DECT� 3GPP SA4� ITU-T SG12And also

4

And also� GCF� DECT Forum/CATIQ

� The only way to ensure a good end to end quality is � To ensure that all speech terminals (whatever acces s network) are

providing equivalent speech quality� To avoid as far as possible coding and speech proc essing

cascading/tandeming� To reduce the end-to end delay

Plugtest (STQE) Reports� At the end of the Test Event

� An anonymous report (also made available as a contr ibution to TC STQ activities)compiling the results

� A specific and confidential report given individual ly to each equipment manufacturer participating to the Plugtes t

� As ETSI deliverables:

5

� As ETSI deliverables:TR 102 648-Series : Test methodologies for ETSI test events and

results Part 1 : VoIP Speech Quality TestingOther parts : ETSI Plugtests Speech Quality Test Event Reports

Plugtest (STQE) methodologies� Test set-up for 5 th ETSI SQTE

NISTNet

Packet Loss

Delay

IP

PacketMonitor

Gateway

6

aethraD2000 PRO

Delay

Input/Output

Input/Output

IP

E1/ISDN DSS1

IP-Terminal(Phone or PC)

HEAD acoustics Test System

Plugtest (STQE) methodologies� Test set-up for 5 th ETSI SQTE

Sub-wooferSub-

woofer

IP

Packet LossDelay

“Interface” - Gateway

IPISDN

Simulator

ISDN

Simulator

NISTNet

Receive direction

Send direction

far endIP

Packet LossDelay

“Interface” - Gateway

IPISDN

Simulator

ISDN

Simulator

NISTNet

Receive direction

Send direction

far end

7

MFE

ListeningOnly Test

ETSI EG 202 396-3

S-MOS N-MOS G-MOS

unprocessed

processed

clean

processedMFE

ListeningOnly Test

ETSI EG 202 396-3

S-MOS N-MOS G-MOS

unprocessed

processed

clean

processed

Speech sample (example 1) for MOS testing

InRCV

OutRCV

InSND

OutSND

USB

MFE VI

Send direction

EC under testnear end

Test system

ACQUA

InRCV

OutRCV

InSND

OutSND

USB

MFE VI

Send direction

EC under testnear end

Test system

ACQUA

Test setup for Echo cancellor testing

Plugtest (STQE) methodologies� Variable parameters

� Packet loss %, Networtk delay

� Quality parameters� PLC (packet Loss Concealment) Performance� VAD (Voice Activity detection) performance� MOS-LQO Mean Opinion Score – Listening Speech Quality Objective

8

� MOS-LQO Mean Opinion Score – Listening Speech Quality Objective� G-MOS (Global quality with background noise –according to EG 202396-3 )

� Conversational parameters� Performance for handset use like one-way transmissi on parameter,

echo performance tests, double talk and background noise transmission tests,

� Tests covering hands-free capability ( frequency responses , echo performance tests , double talk capability and background noise transmission quality),

� Wideband tests in handset and hands-free mode using the G.722 speech coder.

Plugtest (STQE) methodology: The Pie Diagram

9

DT: Double talk

HFT: Hands Free Terminal

TCLw: Weighted terminal Coupling Loss

BGNT: Background Noise Transmission

Plugtest methodologies, results and lessonsElectrical-acoustical set-ups

� The “Quality Pies” of the tested IP phone implement ations :

G.711 3%

4

5

G.711 SND

3

4

5

WB HFT TMOS RCV

2

3

4

WB HS TMOS RCV

3

G.711 3%

4

5

G.711 SND

3

4

5

WB HFT TMOS RCV

2

3

4

WB HS TMOS RCV

3

10

G.711, Jitter

23

4

2

3

2

2

2

HFT TMOS RCV

23

45

HFT Double Talk

2c2b2a1

3

HFT Echo Performance

30%

70%

Ok

BGNT (far)

15

10

5

0

Double Talk

2c

2b

2a

1 Echo Performance

30%

70%

Ok

G.729 Jitter

23

4

G.711, Jitter

23

4

2

3

2

32

2

HFT TMOS RCV

23

45

HFT Double Talk

2c2b2a1

3


30%

70%

Ok

BGNT (far)

15

10

5

0

Double Talk

2c

2b

2a

1 Echo Performance

30%

70%

Ok

G.729 Jitter

23

4

SND: send RCV: Receive

Plugtest (STQE) results� The “Quality Pies” of the IP phone implementations (continued)

G.711, Jitter

G.711 3%

2

3

4

5

G.711 SND

2

3

4

5

WB HFT TMOS RCV

2

3

4

WB HS TMOS RCV

2

3

HFT TMOS RCV5

11

23

4

23

45

HFT Double Talk

2c2b2a1

3


30%

70%

Ok

BGNT (far)

15

10

5

0

Double Talk

2c

2b

2a

1 Echo Performance

30%

70%

Ok

G.729 Jitter

23

4

Plugtest Benefits

� For standardization bodies� Validation of test methodologies in interoperabilit y context� Validation / Update of the requirements for future versions of

standards

� For test equipment manufacturers and test laborator ies

12

� Implementation of the standards and test equipment development� Direct contact with the terminal manufacturers and test of new

terminal implementations� To provide diagnosis when identifying implementatio n not in

conformance with the standards

� For terminal equipment manufacturers� To test the new implementations both in laboratory and

interoperability contexts� To have diagnosis when no conformance for some para meters� To compare the terminal performance with other manu facturers

Part 2: STQ Standards For Multiplay services

� Voice Quality metrics Two ETSI Standards

� ES 202 765-2 QoS and network performance metrics and measurement methods; part 2: Transmission Quality Indicator

13

measurement methods; part 2: Transmission Quality Indicator combining Voice Quality Metrics

� ES 202 765-4 QoS and network performance metrics and measurement methods; part 4 : Indicators for supervision of multiplay services

Multiplay services assesments� Speech Quality Analysis. Several campaigns already done

The test methods and results are made available in� TR 102 716 Series: Guidelines, objectives and resul ts of speech

quality analysis in the context of interworking plu gtests for multiplay services

Pert1: Guidelines and objectives

14

• Pert1: Guidelines and objectives• Part 2: Results (one annex / test campaign)

� IPTV Quality� Results are expected in the next months.

Multiplay services assesments� VoIP Quality – Test set-up for interoperability assessment (6 ISP s)

IP Network of ISP 2HGW 1

HGW 2HGW 1

IP to IP intra-ISP Call

15

IP Network of ISP 1

IP Network of ISP 4

IP Network of ISP 3

HGW 1

HGW 3

HGW 4PSTNIP to IP inter-ISP Call

IP to PSTN Call

Multiplay services assesments� VoIP Quality – Metrics

� Post Dialling Delay� Listening speech quality� End to End delay� Listening speech quality stability

16

� Listening speech quality stability� Active level of speech signal at reception� Reception noise level� Speech signal attenuation (or gain) after transmiss ion� Talker Echo Delay� Talker Echo Attenuation� DTMF integrity

Multiplay services assesments (VoIP – results)� Post Dialing Delay (PDD)

PDDISP1 to ISP2

1.34

3

4

2

0PDD

ISP1 to ISP31.35

3

4

2

0

PDDISP1 to ISP4

PDDISP6 to ISP1

PDDISP7 to ISP1

1.66

3

4

2

0

PDDISP4 to ISP1

0.59

3

4

2

0PDD

ISP4 to ISP21.38

3

4

2

0

PDDISP4 to ISP3

1.49

PDDISP6 to ISP4

5.33

PDDISP7 to ISP4

2.33

3

4

2

0

PDDISP1 to PSTN

1.07

3

4

2

0

PDDISP2 to PSTN

1.68

3

4

2

0

PDDISP3 to PSTN

1.150

PDDPSTN to ISP6

1.10

PDDPSTN to ISP7

0.79

3

4

2

0

17

8

6

8

6

4 ISP1 to ISP40.72

3

86

42

0

1.33PDD

ISP1 to ISP6

3

86

42

0

1.33PDD

ISP1 to ISP7

3

8

6

4

2

0

1.47PDD

ISP2 to ISP1

3

8

6

4

2

0

0.76PDD

ISP3 to ISP1

3

8

6

4

2

0

0.6PDD

ISP4 to ISP1

3

86

42

0

ISP6 to ISP11.01

3

86

42

0

8

6

4

ISP 1

8

6

8

6

1.49

3

86

42

0

1.45PDD

ISP4 to ISP6

3

86

42

0

1.24PDD

ISP4 to ISP7

3

8

6

4

2

0

0.71PDD

ISP1 to ISP4

3

8

6

4

2

0

1.12PDD

ISP2 to ISP4

3

8

6

4

2

0

5.45PDD

ISP3 to ISP4

3

86

42

0

5.33

3

86

42

0

8

6

ISP 4ISP to PSTN

8

6

8

6

4

3

8

6

4

2

PDDISP4 to PSTN

1.0938 6 4 2 0

1.19PDD

ISP6 to PSTN

3

8

6

4

2

0

1.38PDD

ISP7 to PSTN

3

8

6

4

2

0

0.99PDD

PSTN to ISP1

3

8

6

4

2

02.21PDD

PSTN to ISP2

3

8

6

4

2

0

1.17PDD

PSTN to ISP3

3

8

6

4

2

0

0.72PDD

PSTN to ISP4

3 86420

3

8

6

4

2

8

6

4

Multiplay services assesments (VoIP – results)� Listening Quality (MOS-LQON)

MOS-LQONISP1 to PSTN

4.29

44

5


4.38

44

5


4.395

MOS-LQONPSTN to ISP6

4.535


4.42

44

5

MOS-LQONISP1 to ISP2

4.14

44

5MOS-LQONISP1 to ISP3

4.1

44

5

MOS-LQONMOS-LQON


4.34

44

5


4.32

44

5MOS-LQONISP7 to ISP2

4.45


4.355

18

ISP 1 ISP 6ISP to PSTN

3 3

4

3

4

5


4.3743 4 5

4.39MOS-LQON

ISP6 to PSTN

4

3

4

5

4.42MOS-LQON

ISP7 to PSTN

4

3

4

5

4.44MOS-LQON

PSTN to ISP1

4

3

4

54.41


4

3

4

5

4.36MOS-LQON

PSTN to ISP3

4

3

4

5

4.4MOS-LQON

PSTN to ISP4

4 345

4

3

4

5

33

4

3

4 MOS-LQONISP1 to ISP4

4.09

4

3

4

5

4.06MOS-LQONISP1 to ISP6

4

3

4

5


4

3

4

5


4

3

4

5


4

3

4

5


4

3

4

5


4.27

4

3

4

5

4

3

4

3

4

3

4 MOS-LQONISP7 to ISP3

4.39

4

3

4

5


4

3

4

5


4

3

4

5


4

3

4

5


4

3

4

5


4

3

4

5


4.26

4

3

4

5

4

3

4

Multiplay services assesments (VoIP – results)� End to End Delay

� Improvements are needed: • The end to end delay is in most cases too long (eg ISP1 and some

ISP/PSTN configurations)

� It is possible to reach shorter delays (See ISP3)Delay

ISP1 to ISP2235

0Delay

ISP1 to ISP3Delay

ISP67 to ISP1

DelayISP3 to ISP1

357

DelayISP1 to PSTN

152 DelayDelay

19

ISP 1 ISP 3ISP to PSTN

150

400

300

200

100

0 ISP1 to ISP3252

150

400

300

200

100

0

DelayISP1 to ISP4

165

150

400300

200100

0

218Delay

ISP1 to ISP6

150

400300

200100

0

219Delay

ISP1 to ISP7

150

400

300

200

100

0

244Delay

ISP2 to ISP1

150

400

300

200

100

0

346Delay

ISP3 to ISP1

150

400

300

200

100

0

195Delay

ISP4 to ISP1

150

400300

200100

0

DelayISP6 to ISP1

317

150

400300

200100

0

ISP67 to ISP1298

150

400

300

200

100

0

357

150

400

300

200

100

0Delay

ISP3 to ISP2268

150

400

300

200

100

0

DelayISP3 to ISP4

308

150

400300

200100

0

67Delay

ISP3 to ISP6

150

400300

200100

0

217Delay

ISP3 to ISP7

150

400

300

200

100

0

241Delay

ISP1 to ISP3

150

400

300

200

100

0

206Delay

ISP2 to ISP3

150

400

300

200

100

0

255Delay

ISP4 to ISP3

150

400300

200100

0

DelayISP6 to ISP3

71

150

400300

200100

0

DelayISP7 to ISP3

211

150

400

300

200

100

0

152

150

400

300

200

100

0

DelayISP2 to PSTN

204

150

400

300

200

100

0

DelayISP3 to PSTN

175

150

400

300

200

100

0

DelayISP4 to PSTN

158150400 300 200 100 0

168Delay

ISP6 to PSTN

150

400

300

200

100

0

118Delay

ISP7 to PSTN

150

400

300

200

100

0

194Delay

PSTN to ISP1

150

400

300

200

100

0141

DelayPSTN to ISP2

150

400

300

200

100

0

109Delay

PSTN to ISP3

150

400

300

200

100

0

146Delay

PSTN to ISP4

150 4003002001000

DelayPSTN to ISP6

95

150

400

300

200

100

0

DelayPSTN to ISP7

70

150

400

300

200

100

0

Multiplay services assesment (IPTV)

Internet

Internet service

20

IP core network

Audiovisual Service

platforms

VoIP Service

platforms

Accessand

Backhaul

networks

Game service

platforms

VoIP service

IPTV and VoD services

Game service

Multiplay services assesments� IPTV Quality – Metrics

� Channel Availability � Service Group Channel Availability� Video Quality ( no standardized algorithm )� Audio Quality

21

� Audio Quality� "Black Screen" Occurrences� Blockiness Occurrences� Frozen Picture Occurrences� Lip Desynchronization Occurrences� Zapping Delay� Transmission Delay� IPTV service boot delay

Multiplay services assesments� IPTV Quality –

Metrics – Example

� No result currently available

Channel 1Availability

98.2

98

97

98

99

100Channel 2Availability

100

9898

99

100

Channel 3Availability

Zapping delay

Boot delay

7893

10000

13334

6667

0

22

available

95

96

95

96

97Availability

99.2

98

9596

9798

99100

96.7Channel 4Availability

98

9596

9798

99100

2Audio Quality

10

21

14

7

0

13.7Blockiness

Occurrences

10

21

14

7

0

0.2Black ScreenOccurrences

10

21

14

7

0

2.8Frozen PictureOccurrences

10

2114

70

4225

3000

80006000

40002000

0

20001

13334

Thanks

�For additionnal information

23

�For additionnal information �[email protected]

Acknowledgments

�Thanks for their contributions to

24

�Thanks for their contributions to�H.G. Gierlich, Head Acoustics�Jean-Yves Le Saout, France Telecom

Annex: ETSI Standards listed in the presentation� Speech terminals

� ETSI ES 202 740 V1.3.2 (2010-09) Title: Speech and multimedia Transmission Quality (STQ);Transmission requirement s for wideband VoIP loudspeaking and handsfree terminals from a Qo S perspective as perceived by the user

� ETSI ES 202 739 V1.3.2 (2010-09) Title: Speech and multimediaTransmission Quality (STQ);Transmission r equirements

25

multimediaTransmission Quality (STQ);Transmission r equirements for wideband VoIP terminals (handset and headset) f rom a QoS perspective as perceived by the user

� ETSI ES 202 738 V1.3.2 (2010-09) Title: Speech and multimedia Transmission Quality (STQ);Transmission requirement s for narrowband VoIP loudspeaking and handsfree terminal s from a QoS perspective as perceived by the user

� ETSI ES 202 737 V1.3.2 (2010-09) Title: Speech and multimedia Transmission Quality (STQ);Transmission requirement s for narrowband VoIP terminals (handset and headset) fro m a QoS perspective as perceived by the user

Annex: ETSI Standards listed in the presentation

� Multiplay services

� ETSI ES 202 765-2 V1.1.4 (2010-07) Title: Speech an d multimedia Transmission Quality (STQ);QoS and network performa nce metrics and measurement methods;Part 2 : Transmission Quali ty Indicator combining Voice Quality Metrics

26

combining Voice Quality Metrics � ETSI ES 202 765-4 V1.1.1 (2010-10) Title: Speech an d multimedia

Transmission Quality (STQ);QoS and network performa nce metrics and measurement methods;Part 4: Indicators for supe rvision of Multiplay services

� ETSI deliverables may be downloaded fromhttp://pda.etsi.org/pda/queryform.asp

standards for speech and media quality...

Documents