quality assessment of fractalized npr textures, apgv09

Bénard – Thollot – Sillion (LJK - INRIA) /32Bénard – Thollot – Sillion (LJK - INRIA)

Quality Assessment of Fractalized NPR Textures:a Perceptual Objective Metric

Pierre Bénard Joëlle Thollot François Sillion

Grenoble Universities and CNRS / LJK

INRIA

October 2, 2009

October 2, 2009Quality Assessment of Fractilized Textures 1


• Non-Photorealistic Rendering

Inspiration in traditional illustration (drawing, painting...)

Stylization of still images and animations

• Stylization of 3D scenes

3D scenes 2D media (pigments, strokes, paper...)

Temporal coherence artifacts (popping, sliding, deformations)


Introduction

Introduction

[Herz98]

[GTDS04]

«Il p

leu

t b

erg

ère

», Jé

rém

y D

ep

uyd

t(2

00

5)


• Common solution: Fractalization process (e.g. [KLK+00, CTP+03, BSM+07, BBT09])

Medium = texture

Computation of multiple scales

Alpha-blending

Self-similar texture


Introduction

Introduction

+


• Common solution: Fractalization process (e.g. [KLK+00, CTP+03, BSM+07, BBT09])

Self-similar texture


Introduction

Introduction

+

[BBT09]


• Issues:

New features / new frequencies

Global contrast loss

Deformations

Fractalized textures visually dissimilar to the original

How to evaluate this dissimilarity ?

• Artists / viewers = final judges of the perceived quality

User study suitable… but costly

• Quality assessment metric

automatic comparison of existing techniques

new optimization based approaches


Problem statement

Introduction

≠


• Input: pairs of 2D texture

• Definition:texture distortion = visual dissimilarity between original and transformed textures

• Goal: define a quantitative metric of this distortion

• Procedure:

User study ranking of texture pairs according to their distortion

Statistical analysis scale of perceived quality

Correlation investigation objective metric

• Restrictions:

No texture mapping

Static images


Problem statement

-1.0 -0.5 0.0 0.5 1.0

Introduction

1.2

1.0

0.8

0.6

0.4

Z-Scores

AC

E

originaltransformed


Previous Work

Experimental Framework

Statistical Analysis

Correlation with Objective Metrics


Outline


Previous Work

Perceptual Evaluation in NPR

Perceptual Experiment Methodologies





Outline

Previous Work


• Various methodologies:

Questionnaire [SSLR96]

Performances measurement [GRG04]

Eye tracking [SD04]

Observational study + objective metric [INC+06, MIA+08]


Perceptual Evaluation in NPR

Previous Work

[SSLR96]

[GRG04]

[SD04]

[INC+06 ,MIA+08]


• Rating – ex.: image and video quality assessment [SSB06, Win05]

Simple, well-understood

Large number of trials and participants, subjects training needed

• Paired comparisons – ex.: tone mapping comparison [LCTS05, ČWNA08]

Straightforward forced choices

Quadratic complexity effect of fatigue

• Ranking – ex.: high quality global illumination [SFWG04]

Least time consuming task, invariant under stretching

Complicated task


Perceptual Experiment Methodologies

Previous Work


Previous Work


Stimuli

Procedure




Outline



• 2 sets of 10 textures pairs of NPR media


Stimuli

(Near-)regular

patterns

Irregular

patternsGrid Dots Hatching

Cross-

hatchingPaper NoisePaint Pigments

S1

S2



• 2 sets of 10 textures pairs of NPR media

• Fractalized with 3 scales


Stimuli

(Near-)regular

patterns

Irregular

patternsGrid Dots Hatching

Cross-

hatchingPaper NoisePaint Pigments

S1

S2

Orig

inal

Tra

nsfo

rme

dO

rig

inal

Tra

nsf

orm

ed



• Dynamic web interface

Number of participants (103)

Diversity of their skills in computer graphics

Control on the experimental conditions

Assessment of the statistical validity of the resulting data


Procedure

Naive 58.0%

Amateur/professionalinfographists

8.5%

Researcher 22.4%

Unknown 11.1%


Bénard – Thollot – Sillion (LJK - INRIA) /32Bénard – Thollot – Sillion (LJK - INRIA) October 2, 2009Quality Assessment of Fractilized Textures 15

Procedure



Previous Work



Ranking Duration

Concordance among Raters

Interval Scale of Relative Perceived Distortion

Ranking Criteria



Outline



• 103 subjects

45 starting with S1 then S2

58 starting with S2 then S1

• Similar distribution of duration

• Comparable mean duration

No learning or fatigue effect


Ranking Duration


+

+

1

1

2

2


• Merged data analysis relevant

Kendall’s coefficient of concordance (Kendall’s W) [Ken75]

• Ranking not effectively random

Significance of these coefficients validated by test

• Strong variations among textures pair


Concordance among Raters



• Ordinal scale no quantification of perceived differences between pairs



S1

S2



• Ordinal scale no quantification of perceived differences between pairs

Thurstone’s law of comparative judgment [Tor58]




-1.5

Z-Scores

-1.0 -0.5 0.0 0.5 1.0

Grid DotsHatching Paper NoisePaintPigmentsRegular

patterns

Irregular

patterns

Cross-

hatching

-1.5

Z-Scores

-1.0 -0.5 0.0 0.5 1.0 1.5

GridDots Hatching Paper NoisePaint PigmentsNear-regular

patterns

Irregular

patterns

Cross-

hatching

S1

S2


• Unstructured textures more robust

• Textures with distinctive features more severely distorted




-1.5

Z-Scores

-1.0 -0.5 0.0 0.5 1.0 1.5


patterns

Irregular

patterns

Cross-

hatching

S1

S2

-1.5

Z-Scores

-1.0 -0.5 0.0 0.5 1.0


patterns

Irregular

patterns

Cross-

hatching


• Perceived similar distortion intensity significance

Wilcoxon rank sum test




-1.5

Z-Scores

-1.0 -0.5 0.0 0.5 1.0 1.5


patterns

Irregular

patterns

Cross-

hatching

S1

S2

-1.5

Z-Scores

-1.0 -0.5 0.0 0.5 1.0


patterns

Irregular

patterns

Cross-

hatching


Overall contrast of patterns

Feature shapes




S1

S2

-1.5

Z-Scores

-1.0 -0.5 0.0 0.5 1.0


patterns

Irregular

patterns

Cross-

hatching

-1.5

Z-Scores

-1.0 -0.5 0.0 0.5 1.0 1.5


patterns

Irregular

patterns

Cross-

hatching



Feature shapes




S1

S2

-1.5

Z-Scores

-1.0 -0.5 0.0 0.5 1.0


patterns

Irregular

patterns

Cross-

hatching

-1.5

Z-Scores

-1.0 -0.5 0.0 0.5 1.0 1.5


patterns

Irregular

patterns

Cross-

hatching

Paper NoisePaintPigments



Feature shapes




S1

S2

-1.5

Z-Scores

-1.0 -0.5 0.0 0.5 1.0


patterns

Irregular

patterns

Cross-

hatching

-1.5

Z-Scores

-1.0 -0.5 0.0 0.5 1.0 1.5


patterns

Irregular

patterns

Cross-

hatching

Irregular patterns Cross-hatchingDotsHatching



Feature shapes




S1

S2

-1.5

Z-Scores

-1.0 -0.5 0.0 0.5 1.0


patterns

Irregular

patterns

Cross-

hatching

-1.5

Z-Scores

-1.0 -0.5 0.0 0.5 1.0 1.5


patterns

Irregular

patterns

Cross-

hatching


Quite similar frequencies


Ranking Criteria


0%

5%

10%

15%

20%

25%

30%

35%

40%

contrast

sharpness

scale

other

empty

S1 S2

Fre

qu

en

cy a

t w

hic

h e

ach

cri

teri

on

has b

een

used


contrast

sharpness

scale

others

empty


• Irregular preferences for different textures


Ranking Criteria


0%

10%

20%

30%

40%

50%

60%

S2S1




• Top 3 additional criteria proposed by the participants:

Pattern coherence

Density

Shape


Ranking Criteria



Previous Work




Image Quality Metric / Global Image Statistic

Local Image Statistic


Outline



• 11 quality assessment metrics MetrixMux Matlab© package by Matthew Gaubatz

(http://foulard.ece.cornell.edu/gaubatz/metrix_mux/)

No significant correlation distortion too strong

• 3 global statistics

Inconclusive results consider simultaneously contrast, sharpness and scale


Image Quality Metrics / Global Image Statistic


peak signal-to-noise ratio (PSNR)

signal-to noise ratio (SNR)

structural similarity index (SSIM)

multi-scale SSIM index (MSSIM)

visual signal-to-noise ratio (VSNR)

visual information fidelity (VIF)

pixel-based VIF (VIFP)

information fidelity criterion (IFC)

universal quality index (UIQ)

noise quality measure (NQM)

weighted signal-to-noise ratio (WSNR)

Histograms

Power spectra

Distribution of contrast [BG93]

http://foulard.ece.cornell.edu/gaubatz/metrix_mux/


• Gray level co-occurrence matrix (GLCM) [HSD73]

Texture descriptor [TJ93]

Local image property

Match certain levels of human perception [JGSF76]

Linked to density and pattern coherence criteria

Parameters selection

• Average Co-occurrence Error [CRT01]

High correlation with the perceptual interval scale(Person’s correlation: 0.953 for S1 and 0.836 for S2 )





ACE relevant estimator of the distortion



S1

S2


-1.5

Z-Scores

-1.0 -0.5 0.0 0.5 1.0


patterns

Irregular

patterns

Cross-

hatching

-1.5

Z-Scores

-1.0 -0.5 0.0 0.5 1.0 1.5


patterns

Irregular

patterns

Cross-

hatching

AC

E

1.2

1.0

0.8

0.6

0.4

AC

E

1.4

1.2

1.0

0.8

0.6

r² = 0.6992

r² = 0.9075


• First step toward the evaluation of texture-based NPR techniques

10 classes of NPR medium

User-study framework

Dataset and analysis methodology

Quality assessment metric: ACE

• Future work:

Other texture or vision descriptors

Dynamic version of the fractalization process

Trade-off between temporal continuity and texture dissimilarity


Conclusions

Conclusions


• Project web page

http://artis.inrialpes.fr/~Pierre.Benard/TextureQualityMetric/

Dynamic web interface

Full-size figures

R scripts

• Acknowledgments

All the participants of the study

Jean-Dominique Gascuel, Olivier Martin, Fabrice Neyret, Pierre-ÉdouardLandes, Pascal Barla, Alexandrina Orzan and the anonymous reviewers


Thank you for your attention

Conclusions

http://artis.inrialpes.fr/~Pierre.Benard/TextureQualityMetric/


• Proportion of each pair:

• Conversion to z-Scores:

with the mean and the standard deviation of these proportions

• Empirically verified

Normal Q-Q plots

Shapiro-Wilk test

Better confidencefor S2 than S1


Thurstone’s law of comparative judgement

Normal distribution assumption

S1 S2


• Wilcoxon rank sum test

Non-parametric test

Assess if two independent samples of observations come from the same distribution

No assumption about this distribution

Null hypothesis H0:

“the two considered samples are drawn from a single population”

Group pairs for which H0 cannot be rejected


Wilcoxon rank sum test (Mann-Witney U test)



Feature shapes




S1

S2

-1.5

Z-Scores

-1.0 -0.5 0.0 0.5 1.0


patterns

Irregular

patterns

Cross-

hatching

-1.5

Z-Scores

-1.0 -0.5 0.0 0.5 1.0 1.5


patterns

Irregular

patterns

Cross-

hatching


contrast

sharpness

scale

others

empty




Ranking Criteria


0%

10%

20%

30%

40%

50%

60%

S2S1


• Proposed criteria:

• Additional criteria proposed by participants:


Ranking Criteria


contrast sharpness scale other empty

S1 21.96% 26.86% 24.83% 14.4% 8.95%

S2 28.27% 35.21% 21.72% 7.12% 7.07%

density 15%

shape 10%

pattern coherence 21%

frequency 4%

relief 2%


• Gray Level Co-occurrence Matrices

n matrices of size G x G

n = number of displacement vectors

G = gray-level quantization step

= number of occurrence of gray-level pair

a distance d apart




G

G

11

1


• Gray Level Co-occurrence Matrices

n matrices of size G x G

n = number of displacement vectors

G = gray-level quantization step

= number of occurrence of gray-level pair

a distance d apart




G

G

11

1

1


• Average Co-occurrence Error [CRT01]

Distance between 2 sets of GLCM





References I

[BBT09] Pierre Bénard, Adrien Bousseau, and Joëlle Thollot, Dynamic solid textures for real-time coherent stylization, ACM SIGGRAPH Symposium on Interactive 3D Graphics and Game, 2009.

[BG93] Rosario M Balboa and Norberto M Grzywacz, Power spectra and distribution of contrasts of natural images from different habitats, Vision Research (1993).

[BSM+07] Simon Breslav, Karol Szerszen, Lee Markosian, Pascal Barla, and Joëlle Thollot, Dynamic 2D patterns for shading 3D scenes, SIGGRAPH 07: ACM Transactions on Graphics (2007).

[CRT01] A.C. Copeland, G. Ravichandran, and M.M. Trivedi, Texture synthesis using gray-level co-occurrence models, algorithms, experimental analysis and psychophysical support, Optical Engineering (2001).

[CTP+03] Matthieu Cunzi, Joëlle Thollot, Sylvain Paris, Gilles Debunne, Jean-Dominique Gascuel, and Frédo Durand, Dynamic canvas for immersive non-photorealistic walkthroughs, Proceedings of Graphics Interface, 2003.

[CWNA08] Martin Cadík, Michael Wimmer, Laszlo Neumann, and Alessandro Artusi, Evaluation of HDR tone mapping methods using essential perceptual attributes, Computers & Graphics (2008).

[GRG04] Bruce Gooch, Erik Reinhard, and Amy Gooch, Human facial illustrations: Creation and psychophysical evaluation, ACM Trans. Graph. 23 (2004), no. 1, 27–44.

[HSD73] Robert M. Haralick, K. Shanmugam, and Its’Hak Dinstein, Textural features for image classification, Systems, Man and Cybernetics, IEEE Transactions on 3 (1973), no. 6, 610–621.

[INC+06] Tobias Isenberg, Petra Neumann, Sheelagh Carpendale, Mario Costa Sousa, and Joaquim A. Jorge, Non-photorealistic rendering in context: an observational study, NPAR ’06: Symposium on Non-photorealistic animation and rendering, ACM, 2006.

[JGSF76] B. Julesz, E. N. Gilbert, L. A. Shepp, and H. L. Frisch, Inability of humans to discriminate between visual textures that agree in second-order statistics –revisited, Perception (1976).

[Ken75] M. G. Kendall, Rank correlation methods, Hafner Publishing Company, Inc, 1975.


References II

[KLK+00] Allison W. Klein, Wilmot W. Li, Michael M. Kazhdan, Wagner T. Correa, Adam Finkelstein, and Thomas A. Funkhouser, Non-photorealistic virtual environments, Proceedings of SIGGRAPH 2001, ACM, 2000.

[LCTS05] Patrick Ledda, Alan Chalmers, Tom Troscianko, and Helge Seetzen, Evaluation of tone mapping operators using a high dynamic range display, SIGGRAPH 05 : ACM Transactions on Graphics (2005).

[MIA+08] Ross Maciejewski, Tobias Isenberg, William M. Andrews, David S. Ebert, Mario Costa Sousa, and Wei Chen, Measuringstipple aesthetics in hand-drawn and computer-generated images, IEEE Comput. Graph. Appl. (2008).

[SD04] Anthony Santella and Doug DeCarlo, Visual interest and NPR: an evaluation and manifesto, NPAR ’04: Symposium on Non-photorealistic animation and rendering, ACM, 2004.

[SFWG04] William A. Stokes, James A. Ferwerda, Bruce Walter, and Donald P. Greenberg, Perceptual illumination components: a new approach to efficient, high quality global illumination rendering, ACM Transactions on Graphics (2004).

[SSB06] H.R. Sheikh, M.F. Sabir, and A.C. Bovik, A statistical evaluation of recent full reference image quality assessmentalgorithms, IEEE Transactions on Image Processing (2006).

[SSLR96] Jutta Schumann, Thomas Strothotte, Stefan Laser, and Andreas Raab, Assessing the effect of non-photorealistic rendered images in cad, CHI ’96: Proceedings of the SIGCHI conference on Human factors in computing systems, ACM, 1996.

[TJ93] Mihran Tuceryan and Anil K. Jain, Texture analysis, The Handbook of pattern recognition & computer vision (1993).

[Tor58] W. S. Torgerson, Theory and methods of scaling, Wiley, 1958.

[Win05] Stefan Winkler, Perceptual video quality metrics – a review, Digital Video Image Quality and Perceptual Coding, CRC Press, 2005.

quality assessment of fractalized npr textures, apgv09

Technology

textures goal

video quality assessmentssb06

texture distortion

deformations introduction

objective metrics

original transformed

texture pairsaccording

statistical validity