objective evaluation of video quality over satellite …objective evaluation of video quality over...
TRANSCRIPT
September 10, 1999 International Symposium on Advanced Radio Technologies 1
Objective Evaluation of Video Qualityover Satellite Networks
Ray JenningsInstitute for Telecommunication Sciences
National Telecommunications and Information AdministrationBoulder, CO
September 10, 1999 International Symposium on Advanced Radio Technologies 2
Transportable Video QualityMeasurement System (TVQMS)
T Designed and built by NTIA / ITS
T Assembled from off-the-shelf hardware components
T Utilizes ITS–developed (and patented) software
September 10, 1999 International Symposium on Advanced Radio Technologies 3
Objectives
T Describe the TVQMST Explain measurements that can be made • Parameters Þ Objective Quality Rating
– Objective and Repeatable – Correlate well with subjective
measurements • Differential Mean Opinion Score (DMOS)T Illustrate one possible experiment scenarioT Identify possible measurement applications
September 10, 1999 International Symposium on Advanced Radio Technologies 4
Measurement Approach
(Non Real-Time)
System Under Test
TVQMS
VideoSource
Objective QualityParameters
MPEG-2Encoder
MPEG-2Decoder
DigitalChannel
VideoOutput
September 10, 1999 International Symposium on Advanced Radio Technologies 5
TVQMS Front-View
September 10, 1999 International Symposium on Advanced Radio Technologies 6
TVQMS Applications
System Under Test
Video Cassette Recorder (plays “reference” video) Encoder Decoder
DigitalChannel
In-Service Measurements
Video Cassette Recorder (records “test” video)
System UnderTest
Video Cassette Recorder (records “reference” video)
Video Cassette Recorder (records “test” video)
VideoSource
Out-of-Service Measurements
September 10, 1999 International Symposium on Advanced Radio Technologies 7
TVQMS Front-View
September 10, 1999 International Symposium on Advanced Radio Technologies 8
TVQMS Functional Diagram
Define Video Clip(Digital Disk Recorder)
Video Source
System/Network Under Test
Compare “Reference” and “Test” Features
Compute Performance Parameter Valuesand Objective Quality Rating
(Computer)
ObjectiveQuality Rating
Extract “Reference” Features(Computer)
Extract “Test” Features(Computer)
ParameterValues
September 10, 1999 International Symposium on Advanced Radio Technologies 9
“FEATURE”
! A quantity of information (summary statistics) associated with a specific Spatial–Temporal region of a video scene
! Three types—
T Spatial Feature: The activity of image edges or spatial gradients
T Temporal Feature: The activity of absolute motion differences or temporal gradients between frames
T Chrominance Feature: The activity of color saturation
September 10, 1999 International Symposium on Advanced Radio Technologies 10
Spatial-Temporal (S-T) Regions
Video Stream
Fk Fk+1 Fk+2 Fk+3 Fk+4 Fk+5
Temporal-Width (∆t) Horizontal-Width (∆h)
Vertical-Width (∆v)
Vertical-Width (∆v)
Horizontal-Width (∆h)
Temporal-Width (∆t)
September 10, 1999 International Symposium on Advanced Radio Technologies 11
Parameters
T Recommended to VQEG (by NTIA / ITS)
• Loss in spatial activity
• Gain in temporal activity
• Loss in chroma activity
• Gain in chroma activity
• Gain in spatial-temporal activity
T ITU Video Quality Experts’ Group (VQEG)
September 10, 1999 International Symposium on Advanced Radio Technologies 12
Interpretation ofParameter Values
3 experiments: 26 video systems and 36 test scenes
E[DMOS] = OQR = w1p1 + w2p2 + w3p3 + w4p4 + w5p5
-4
-3.5
-3
-2.5
-2
-1.5
-1
-0.5
0
0.5-3.5-3-2.5-2-1.5-1-0.500.5
Subjective DMOS
Ob
ject
ive
Qu
alit
y R
atin
g (
OQ
R)
Correlation = 0.95(Scene-System)
-3
-2.5
-2
-1.5
-1
-0.5
0
-3-2.5-2-1.5-1-0.50
Average Subjective DMOS
Ave
rag
e O
bje
ctiv
e Q
ual
ity
Rat
ing
(O
QR
)Correlation = 0.99
(System)
September 10, 1999 International Symposium on Advanced Radio Technologies 13
Possible Quality-Measurement Scenario
VideoRecorder(Source)
Modulator UpConverter Transmitter
VideoRecorder
De-Modulator
DownConverter Receiver
ATMEncoder
MPEG-2Encoder
ATMDecoder
MPEG-2Decoder
ATM SwitchTVQMSBit-LevelChannel
Simulator
ActualSatelliteChannel
Parametersand OQR(Output)
1
4
3
2
September 10, 1999 International Symposium on Advanced Radio Technologies 14
Summary
T TVQMS provides quantitative and repeatable quality (performance) results
T Measurements — in the lab or on-site • Non real-time now • Real-time capability under development • Metrics — easy to update
T Utilization / Application • Satisfy users’ needs for video performance information • Provide new information to ITU-R Recommendation on ATM Performance (e.g., Annex 2, Section 4) • Provide information to new ITU-R Recommendation on IP Performance