a new methodology to estimate the impact of h.264 artefacts on subjective video quality
DESCRIPTION
Presentation of my scientific paper to the Third International Workshop on Video Processing and Quality Metrics (VPQM2007).TRANSCRIPT
![Page 1: A new methodology to estimate the impact of H.264 artefacts on subjective video quality](https://reader034.vdocuments.net/reader034/viewer/2022052507/555840ced8b42acb078b4fc2/html5/thumbnails/1.jpg)
A new methodology to estimate the impact of H.264 artefacts on
subjective video quality
Stéphane Péchard, Patrick Le Callet, Mathieu Carnec, Dominique Barba
Université de Nantes – IRCCyN laboratory – IVC teamPolytech’Nantes, rue Christian Pauc, 44306 Nantes, France
Third International Workshop on Video Processing and Quality Metrics for Consumer ElectronicsScottsdale, 2007-01-26
![Page 2: A new methodology to estimate the impact of H.264 artefacts on subjective video quality](https://reader034.vdocuments.net/reader034/viewer/2022052507/555840ced8b42acb078b4fc2/html5/thumbnails/2.jpg)
2
Introduction
• Codec => Coding artefacts
• Quality loss due to artefacts
=> Useful for quality metrics or better coding …
• Possible practical approach– Artefact classification– Annoyance or quality loss contribution per artefact type
![Page 3: A new methodology to estimate the impact of H.264 artefacts on subjective video quality](https://reader034.vdocuments.net/reader034/viewer/2022052507/555840ced8b42acb078b4fc2/html5/thumbnails/3.jpg)
3
Farias and al. methodology Farias VPQM 05
• Artefacts type set (blockiness, blur, ringing, …)• Generation of synthetic artefacts
– Strength parameter– Applied with the same strength on a whole part of the
sequence• Subjective assessment => Annoyance curve per artefact type regarding the
strength => Content dependency
Alternative approach : VPQM07⇒ H.264 coding, Subjective assessment : quality scale, blur
scale, blockiness scale …⇒ No direct control of artefacts strength
![Page 4: A new methodology to estimate the impact of H.264 artefacts on subjective video quality](https://reader034.vdocuments.net/reader034/viewer/2022052507/555840ced8b42acb078b4fc2/html5/thumbnails/4.jpg)
4
Proposed approach
• H.264 artefacts due to quantization/decision • effects are different regarding the local content
(edge, texture, …)• different perceived annoyance depending on the
local spatio-temporal activity of the content
• H264 distortions only in selected coherent spatio-temporal regions => define content categories
• Subjective quality assessment⇒ Quality loss curve per local content category
(e.g. effects of H264 on each category)
⇒ Strength ?
![Page 5: A new methodology to estimate the impact of H.264 artefacts on subjective video quality](https://reader034.vdocuments.net/reader034/viewer/2022052507/555840ced8b42acb078b4fc2/html5/thumbnails/5.jpg)
5
Outline
• Spatio temporal segmentation • distorted sequences generation• subjective quality assessment of sequences• Quality assessment : Combining categories• Towards quality loss function per content
category
![Page 6: A new methodology to estimate the impact of H.264 artefacts on subjective video quality](https://reader034.vdocuments.net/reader034/viewer/2022052507/555840ced8b42acb078b4fc2/html5/thumbnails/6.jpg)
6
The approach
temporalsegmentation classification
H.264 codingC-distortedsequencesgeneration
unlabeledholes filling
bordersprocessing
source
categories masks sequence
partly-distorted sequencesusable for subjective tests
Ci
……
![Page 7: A new methodology to estimate the impact of H.264 artefacts on subjective video quality](https://reader034.vdocuments.net/reader034/viewer/2022052507/555840ced8b42acb078b4fc2/html5/thumbnails/7.jpg)
7
Spatio temporal classification
2 steps
- temporal segmentation :reliability regarding the motion => temporal tubes
- tube classification :Regarding spatial content
![Page 8: A new methodology to estimate the impact of H.264 artefacts on subjective video quality](https://reader034.vdocuments.net/reader034/viewer/2022052507/555840ced8b42acb078b4fc2/html5/thumbnails/8.jpg)
8
Segmentation of sequences
temporalsegmentation classification
H.264 codingClass-distorted
sequencesgeneration
unlabeledholes filling
bordersprocessing
source
partly-distorted sequencesusable for subjective tests
Ci
……
![Page 9: A new methodology to estimate the impact of H.264 artefacts on subjective video quality](https://reader034.vdocuments.net/reader034/viewer/2022052507/555840ced8b42acb078b4fc2/html5/thumbnails/9.jpg)
9
Segmentation of sequences
• per group of five successive frame, the center frame is divided into blocks
• motion estimation of each block using the two previous frames and the two next frames
(motion estimation performed on a multi-resolution representation)
i+1 i+2i-1 ii-2
![Page 10: A new methodology to estimate the impact of H.264 artefacts on subjective video quality](https://reader034.vdocuments.net/reader034/viewer/2022052507/555840ced8b42acb078b4fc2/html5/thumbnails/10.jpg)
10
Segmentation of sequences
• temporal tracking of each block of frame i defines a spatio-temporal “tube” over the five frames
• a tube is oriented along the local motion
i+1 i+2i-1 ii-2
![Page 11: A new methodology to estimate the impact of H.264 artefacts on subjective video quality](https://reader034.vdocuments.net/reader034/viewer/2022052507/555840ced8b42acb078b4fc2/html5/thumbnails/11.jpg)
11
Classification
temporalsegmentation classification
H.264 codingC-distortedsequencesgeneration
unlabeledholes filling
bordersprocessing
source
partly-distorted sequencesusable for subjective tests
Ci
……
![Page 12: A new methodology to estimate the impact of H.264 artefacts on subjective video quality](https://reader034.vdocuments.net/reader034/viewer/2022052507/555840ced8b42acb078b4fc2/html5/thumbnails/12.jpg)
12
Definition of content categories
• low luminance smooth areas;• high luminance smooth areas;• fine textured areas;• edges;• strong textured areas
HVS has different perception of impairments depending on the local spatio-temporal content.
![Page 13: A new methodology to estimate the impact of H.264 artefacts on subjective video quality](https://reader034.vdocuments.net/reader034/viewer/2022052507/555840ced8b42acb078b4fc2/html5/thumbnails/13.jpg)
13
Classification
• 4 spatial gradients means per tube (directions : 0, 90, 45 and 135°)
• plot in spatial space P (0 and 90°) => C1, C2, C3 and C4
• 2nd step : space P’ (45 and 135°) used to discriminate C5 in P
• frontier determined to obtain relevant classification
![Page 14: A new methodology to estimate the impact of H.264 artefacts on subjective video quality](https://reader034.vdocuments.net/reader034/viewer/2022052507/555840ced8b42acb078b4fc2/html5/thumbnails/14.jpg)
14
Classification
• global tracking of moving objects over the whole sequence
• tubes are classified then merged by categories
smooth areas with low luminancesmooth areas with high luminancefine textured areasedgesstrong textured areas
![Page 15: A new methodology to estimate the impact of H.264 artefacts on subjective video quality](https://reader034.vdocuments.net/reader034/viewer/2022052507/555840ced8b42acb078b4fc2/html5/thumbnails/15.jpg)
15
Unlabeled holes filling and tube intersections
temporalsegmentation classification
H.264 codingC-distortedsequencesgeneration
unlabeledholes filling
bordersprocessing
source
partly-distorted sequencesusable for subjective tests
Ci
……
![Page 16: A new methodology to estimate the impact of H.264 artefacts on subjective video quality](https://reader034.vdocuments.net/reader034/viewer/2022052507/555840ced8b42acb078b4fc2/html5/thumbnails/16.jpg)
16
Unlabeled holes filling and tube intersection
• every pixel of the source has one and only one label• unlabeled holes :
– gradient value => class– closest tube
• Insection pixels : same
![Page 17: A new methodology to estimate the impact of H.264 artefacts on subjective video quality](https://reader034.vdocuments.net/reader034/viewer/2022052507/555840ced8b42acb078b4fc2/html5/thumbnails/17.jpg)
17
Borders processing
temporalsegmentation classification
H.264 codingC-distortedsequencesgeneration
unlabeledholes filling
bordersprocessing
source
partly-distorted sequencesusable for subjective tests
Ci
……
![Page 18: A new methodology to estimate the impact of H.264 artefacts on subjective video quality](https://reader034.vdocuments.net/reader034/viewer/2022052507/555840ced8b42acb078b4fc2/html5/thumbnails/18.jpg)
18
Borders processing
• borders between original and distorted large regions are treated so as to smooth the transitions
beforeborders
processing
afterbordersprocessing
![Page 19: A new methodology to estimate the impact of H.264 artefacts on subjective video quality](https://reader034.vdocuments.net/reader034/viewer/2022052507/555840ced8b42acb078b4fc2/html5/thumbnails/19.jpg)
19
H.264 coding and class-distorted sequences generation
temporalsegmentation classification
H.264 codingC-distortedsequencesgeneration
unlabeledholes filling
bordersprocessing
source
category masks sequence
partly-distorted sequencesusable for subjective tests
Ci
……
![Page 20: A new methodology to estimate the impact of H.264 artefacts on subjective video quality](https://reader034.vdocuments.net/reader034/viewer/2022052507/555840ced8b42acb078b4fc2/html5/thumbnails/20.jpg)
20
Partly-distorted sequences generation
H.264 sequences at different bitrates
categories sequence
original sequence C1
C2
C3
C4
C5
5 sequences per bitrate
![Page 21: A new methodology to estimate the impact of H.264 artefacts on subjective video quality](https://reader034.vdocuments.net/reader034/viewer/2022052507/555840ced8b42acb078b4fc2/html5/thumbnails/21.jpg)
21
Original sequence (first frame)
![Page 22: A new methodology to estimate the impact of H.264 artefacts on subjective video quality](https://reader034.vdocuments.net/reader034/viewer/2022052507/555840ced8b42acb078b4fc2/html5/thumbnails/22.jpg)
22
One caregory distorted sequence (first frame)
![Page 23: A new methodology to estimate the impact of H.264 artefacts on subjective video quality](https://reader034.vdocuments.net/reader034/viewer/2022052507/555840ced8b42acb078b4fc2/html5/thumbnails/23.jpg)
23
Subjective quality assessment
• SAMVIQ protocol with at least 15 validated observers and normalized conditions
• 1920x1080 HDTV Philips LCD display
• Doremi V1-UHD 1080i HDTV player
![Page 24: A new methodology to estimate the impact of H.264 artefacts on subjective video quality](https://reader034.vdocuments.net/reader034/viewer/2022052507/555840ced8b42acb078b4fc2/html5/thumbnails/24.jpg)
24
Subjective quality assessment
• 11 sequences in a SAMVIQ session:– 5 Ci-only distorted at a certain bitrate B
– entirely distorted sequence at B– entirely distorted sequence at low bitrate– entirely distorted sequence at intermediate
bitrate– explicit and hidden references
![Page 25: A new methodology to estimate the impact of H.264 artefacts on subjective video quality](https://reader034.vdocuments.net/reader034/viewer/2022052507/555840ced8b42acb078b4fc2/html5/thumbnails/25.jpg)
25
Sequences uncompressed HDTV sequences from SVT
Above marathon Captain Dance in the woods Duck fly
C5 50 % C2 78 % C3 54 % C5 60 %
Fountain man Group disorder Rendezvous Ulriksdals
C2 71 % C2+C3+C1 95 % C5 56 % C2+C3 80 %
![Page 26: A new methodology to estimate the impact of H.264 artefacts on subjective video quality](https://reader034.vdocuments.net/reader034/viewer/2022052507/555840ced8b42acb078b4fc2/html5/thumbnails/26.jpg)
26
example on sequence Ulriksdalscoded at 1 Mbps
0
10
20
30
40
50
60
70
80
90
1 2 3 4 5
Classes MOS(Sj,Bk) MOSref
![Page 27: A new methodology to estimate the impact of H.264 artefacts on subjective video quality](https://reader034.vdocuments.net/reader034/viewer/2022052507/555840ced8b42acb078b4fc2/html5/thumbnails/27.jpg)
27
• ∆MOS(Ci, Sj ,Bk) = MOSref - MOS(Ci, Sj ,Bk) is the quality loss induced by distortions in category Ci
∆MOS(C4)
∆MOS(C5)
∆MOS(C3)
∆MOS(C1)
∆MOS(C2)
MOSref
MOS(Sj,Bk)
DMOS(Sj,Bk)
MOS(C4)
MOS(C5)
MOS(C3)MOS(C1)
MOS(C2)
• MOS(Ci, Sj ,Bk) for each sequence Sj, each category Ci at each bitrate Bk
• DMOS(Sj ,Bk) = MOSref – MOS(Sj ,Bk) is the quality difference between the reference and the entirely distorted sequence
DMOS and ∆MOS
![Page 28: A new methodology to estimate the impact of H.264 artefacts on subjective video quality](https://reader034.vdocuments.net/reader034/viewer/2022052507/555840ced8b42acb078b4fc2/html5/thumbnails/28.jpg)
28
• relations use sums of ∆MOS
0.5472∆MOS(C4)
0.7664∆MOS(C2)
0.7094∆MOS(C3)
0.6400∆MOS(C5)
……
0.5349∆MOS(C1)
0.9058∆MOS(C1) + ∆MOS(C2) + ∆MOS(C3)
+ ∆MOS(C4) + ∆MOS(C5)
0.9094∆MOS(C2) + ∆MOS(C3) + ∆MOS(C4)0.9440∆MOS(C2) + ∆MOS(C5)0.9485∆MOS(C2)+ ∆MOS(C4) + ∆MOS(C5)
CCCombination
Possible relation between global DMOS and category ∆MOS?
![Page 29: A new methodology to estimate the impact of H.264 artefacts on subjective video quality](https://reader034.vdocuments.net/reader034/viewer/2022052507/555840ced8b42acb078b4fc2/html5/thumbnails/29.jpg)
29
Non linear functions
• DMOSp = maxi(∆MOSi)
– CC = 0.9467• DMOSp = maxi(∆MOSi) + maxj(∆MOSj) with j≠i
– CC = 0.9530
• Correlation exists between global DMOS and category ∆MOS=> DMOS could be predicted from quality per
category
![Page 30: A new methodology to estimate the impact of H.264 artefacts on subjective video quality](https://reader034.vdocuments.net/reader034/viewer/2022052507/555840ced8b42acb078b4fc2/html5/thumbnails/30.jpg)
30
Towards a quality loss model
• How to control the distortion level of a given class ?– Farias approach :strength of synthetic artefact
• Factors implied in the quality loss of category Ci:– distortions themselves– motion– proportion of the category– spatial localisation (not considered here)
![Page 31: A new methodology to estimate the impact of H.264 artefacts on subjective video quality](https://reader034.vdocuments.net/reader034/viewer/2022052507/555840ced8b42acb078b4fc2/html5/thumbnails/31.jpg)
31
Distortion strength for category C1
• distortion strength = f(M,P,E)With all along the sequence :– M the mean motion of the category;– P the mean proportion of the category;– E the MSE on the category;
• M decreases the distortion strength while P and E increase DSproposed model for f
DS = (1 — M/Mt)×P×E
![Page 32: A new methodology to estimate the impact of H.264 artefacts on subjective video quality](https://reader034.vdocuments.net/reader034/viewer/2022052507/555840ced8b42acb078b4fc2/html5/thumbnails/32.jpg)
32
Quality loss function for category C1
• Psychometic function as a prediction of ∆MOS1
φ(DS) = (a×DSb)/(c+DSb)
• correlation between φ(DS) and ∆MOS1 : 0.9514
• RMSE = 5.25• good predictor of the loss of quality induced by
category C1
![Page 33: A new methodology to estimate the impact of H.264 artefacts on subjective video quality](https://reader034.vdocuments.net/reader034/viewer/2022052507/555840ced8b42acb078b4fc2/html5/thumbnails/33.jpg)
33
Quality loss function for class C1
=> Possible prediction of ∆MOS1
![Page 34: A new methodology to estimate the impact of H.264 artefacts on subjective video quality](https://reader034.vdocuments.net/reader034/viewer/2022052507/555840ced8b42acb078b4fc2/html5/thumbnails/34.jpg)
34
Conclusion
• design of a new methodology to estimate the impact of H.264 artefacts on subjective video quality
• One distortion type but– Effect related to local content– possibility to relate the global loss to loss per
category– quality loss function for category C1
• Other categories and objective models
![Page 35: A new methodology to estimate the impact of H.264 artefacts on subjective video quality](https://reader034.vdocuments.net/reader034/viewer/2022052507/555840ced8b42acb078b4fc2/html5/thumbnails/35.jpg)
35
Results: segmentation statistics
60.7016.750.3650.06C5 (%)
10.703.021.430.94C4 (%)
19.5053.856.8127.79C3 (%)
8.9722.5778.2617.45C2 (%)
0.133.8013.143.75C1 (%)
Duck flyDance in the woodsCaptainAbove marathon
Séquence
![Page 36: A new methodology to estimate the impact of H.264 artefacts on subjective video quality](https://reader034.vdocuments.net/reader034/viewer/2022052507/555840ced8b42acb078b4fc2/html5/thumbnails/36.jpg)
36
Results: segmentation statistics
3.3056.924.543.93C5 (%)
1.362.051.791.45C4 (%)
40.4819.8729.8013.37C3 (%)
41.3112.3838.5870.71C2 (%)
13.548.7825.2810.52C1 (%)
UlriksdalsRendezvousGroup disorderFountain man
Séquence