Statistical Modelling of a Colour Naming SpaceRobert Benavente, Francesc Tous, Ramon Baldrich and Maria Vanrell

Computer Vision Center / Dept. Informàtica (UAB)Bellaterra, Barcelona/Spain

Abstract

In this paper, we deal with the colour naming visual taskin computer vision. Our goal is to develop a colournaming model to be included in a real surveillanceapplication, where images have to be automaticallyannotated with people clothes description, for a furthercontent-based image browsing.

Although colour naming has been a usual goal inpsychophysical research, it is a quite novel topic incomputer vision. Colour naming is posed as a fuzzy setproblem where each colour category is presented as afuzzy set with a characteristic function. Our goal is tofind a model which provides membership values assimilar as possible to the values that would give a realobserver.

To this end, we propose a Sigmoid-Gaussian modelas the membership function of the colour fuzzycategories. We analyse its properties and results toconfirm the suitability of this model versus most commonGaussian models. To test the results, we have developeda colour naming experiment that has provided a set ofmembership values for a set of colour samples. Althoughthe data used is far from being a complete learning set, ithas been a first step to evaluate the proposed model.

Introduction

The work presented in this paper is framed in a largesurveillance application that is being developed. The aimof this application is the retrieval of personal informationfrom a database where the identification data of people isjointly stored with their appearance description. Theappearance description is automatically inserted by thesystem from an image taken when the person is standingup in front of the reception desk. The application isdesigned to work in highly restricted access buildingswhere people must give their identification at theentrance. Since the security staff will make retrievalqueries, the appearance descriptions have to be stored interms of natural language, such as 'dark hair', 'red tie' or'blue shirt'. This work is concerned with the descriptionof the clothes colour; therefore, the colour namingproblem is the main topic of our research.

The frame presented above implies that our colournaming module has to act just as a human observer woulddo. Hence, two constraints have to be considered on thisapproach. Firstly, it has to present colour constancyabilities since the system has to give a stable output undernormal changing conditions of any hall of a building.This will take us to some considerations about the colourspace to use. Secondly, the colour naming learning set

has to take into account colour judgements of the systemusers. In this sense, we will present a preliminaryexperiment we have done to collect psychophysical data.Finally, we estimate the parameters of a Sigmoid-Gaussian function that will model colour membershipand will allow giving a complete colour description ofany image region. Before going deeply in these threesteps we will review basic ideas on colour naming.

Colour Naming

In this work, colour naming problem is posed as a fuzzy-based approach. Kay and McDaniel1 were the first inproposing a formulation of colour categories as fuzzysets that provide a useful framework for the colournaming problem.

According to fuzzy set theory, any colour category Cis characterised by a function fC which assigns to eachcolour sample x a value fC(x) within the [0,1] interval.This value represents the degree of membership of x tocolour term C. From this point of view, the goal of anycolour naming method will be the definition of amembership function for each colour category.

In this paper, we will model the membershipfunctions derived from psychophysical data collectedwith an experiment we will present in a section below.The set of stimulus used is acquired by our system, andwe will model the membership functions for a device-dependent colour space.

At this point, the main question to be answered isabout which is the function that can better model thecolour categories. Previous research on statisticalmodelling of colour data has been focused on Gaussianmodels2. An interesting approach for colour naminggoing in this direction was proposed by Lammens3, whoused a symmetric Gaussian model. In that case, the fittingprocess is done over the categories of the Munsell colourspace considering the boundaries and focuses for eachcategory defined by Berlin and Kay4 for the Englishlanguage. This process is based on a minimization thatfits the Gaussian function constrained to get maximumvalues for those points in the category focus, a certainthreshold value for those points in the category boundaryand a minimum value for those points in other categories.

Considering these previous steps and towards to getuseful colour descriptors that behave consistently withhuman categorization, we propose to collectpsychophysical data of the membership degree of eachstimulus to each colour category, and subsequently, to fitour model to this data.

Colour space

As it has been previously introduced, we will fitpsychophysical data with computational colourrepresentations that are derived from device-dependentspaces. We will use a set of images acquired by theequipment of the surveillance application in which thiswork is framed. Therefore, an RGB representation isdirectly obtained.

In order to deal with colour variability we haveapplied the comprehensive normalization defined byFinlayson et al.5. This normalization deals with changesin both the intensity and the colour of the illuminant.Dependence to intensity changes is removed by a pixelnormalization which projects RGB co-ordinates on thechromaticity diagram. Independence to illuminant colourchanges is achieved by a channel normalization. Due tospecific conditions provided by the surveillanceapplication, we can assume certain constancy on theimage content. Such assumption can be used to improvethis normalization as it has been proposed by Vanrell etal.6. Once the colour space has been normalized, colourinformation can be considered on a two-dimensionalspace without loose of information. Figure 1 shows anscheme of how this space, which we will denote as uvspace, is derived.

Figure 1. Colour transformations for uv space

Comprehensive normalization, removes intensityfrom colour information. However, for a colour namingtask the intensity component might be essential. Hence,we propose to add an intensity axis, denoted by I, whichwill correspond to the original intensity in the RGB spacenormalized by the mean intensity of the whole image.Again, the assumption of image content constancy allowsus to consider I as an intensity descriptor.

Psychophysical data

To build the learning set for this colour namingmodelling process, we have made a psychophysicalexperiment7 to obtain all the necessary data. In thisexperiment, 10 subjects were asked to distribute, for eachstimulus, 10 points between the eleven basic colour termsproposed by Berlin and Kay4 (black, white, red, green,yellow, blue, brown, purple, pink, orange and grey). Atotal of 422 colour samples were selected for theexperiment which was done twice by all 10 subjects. Thismakes a total number of 8440 observations. For eachsample, the results were averaged and normalized to the

[0,1] interval. Each sample was also labelled with thecolour term which had received the highest value.

The selection of only the eleven basic colour termswas made in order to reach a high degree of consistencyand consensus in the colour naming task, which is highlydesirable in our application. The analysis of the results ofthe experiment showed that subjects were consistent intheir two scorings for the same sample, 85.09% of thetime. The consensus between the ten subjects wasachieved for 72.99% of the samples.

The colour learning set we have built with thisexperiment will allow us to present a preliminaryapproach of the colour naming system. However, itshould be extended to a more complete set of sampleswhich cover a bigger extension of the chromaticitydiagram. It would be interesting to test the model onmore complete sets and on standard colour spaces.Unfortunately, up to this moment, we have not foundpublic available data of a similar experiment, sincecolour naming experiments does not usually ask formembership degrees to colour categories.

Modelling membership functions

As we have stated before, the goal of our work, is todefine a set of characteristic functions which provide asimilar response as the one that would give a user of thesurveillance system. Hence, for a given colour stimulusx=(u,v,I), we will obtain a colour descriptor CD with themembership values mi (i=1,…,k) for each colourcategory Ci:

CD(x) =(m1,…,mk) (1)

where k is the number of colour categories considered. Inour case, we will consider k=9 categories whichcorrespond to the eight chromatic basic colours proposedby Berlin and Kay, plus an achromatic category includingwhite, black and grey. For this category, denoted by theterm ‘grey’, a thresholding step on the intensity is appliedto distinguish between these three colours.

Our first attempt to model the colour naming spacewas to select a multivariate Gaussian function ascharacteristic function for each colour category:

G(x,µ,S)=exp{(-1/2)(x-µ)T S-1(x-µ)} (2)

where µ is the mean of the samples and S is thecovariance matrix. The values of µ and S were estimatedfor each colour category as:

µ =Σjxj/n (3)

S =(1/n)Σj(xj-µ)(xj-µ)T (4)

where j=1,….,n and n is the number of learningsamples for the category considered. Notice that in thisfirst approach, the membership values obtained in theexperiment, are not used. To evaluate the errorcommitted by this Gaussian model, we computed themean squared error (MSE) between the values given bythe model and the values obtained in the experiment. Theresulting MSE was 0.3485, which indicated that the

colour naming modelling was far of our goal and shouldbe improved.

Following the direction initiated by Lammens, oursecond approach was to use the knowledge about themembership values of learning data obtained from thepsychophysical experiment and estimate the parametersof the model, such that the MSE of the fit was minimized.The estimation of the parameters was performed as anon-linear least-squares data fitting by the Gauss-Newtonmethod. The MSE of the fit was reduced to 0.1105.Despite of this improvement, some facts indicate that theGaussian model is only adequate to model the greycategory. On the one hand, the MSE obtained for some ofthe categories should be improved. On the other hand, ifthe intensity component is eliminated, the membershipmaps over the uv plane derived from the experimentshow that the membership values for the remaindercategories (red, green, yellow, blue, brown, purple, pinkand orange), are not normal distributions.

These facts took us to look for other mathematicalmodels to represent the membership functions. Theprevious results allow us to define the desirableproperties that should fulfil a characteristic function,f(u,v), for the cited colour categories:

• f(u,v) ∈ [0,1], in order to be a membership function.• f has a plateau form.• f is a parametrical function with a parameter

controlling the slope of the surface.• f is a parametrical function with a parameter

allowing asymmetry with respect to the u=v axis.

Several functions were considered to achieve theabove constraints. However, the best results have beengot when using a combination of two well-knownfunctions. The first three constraints are fulfilled by usinga Sigmoid function. To reach the last one, we propose tomodulate the Sigmoid with a Gaussian function in thedirection perpendicular to u=v. The Sigmoid function inone dimension is defined as:

S(x,β) = 1/(1+exp(-βx)) (5)

where β is a parameter that controls the slope of thefunction. Hence, we define the Sigmoid function in twodimensions for our uv space as:

S(u,v,βu,βv) = (1/(1+exp(-βuu)))* (1/(1+exp(-βvv))) (6)

The form of the Sigmoid function in the uv space,for βu=βv=3 and βu=βv=20 can seen in figure 2. Noticethat S(u,v,βu,βv) has a plateau form in the first quadrant.

Figure 2.2D-Sigmoid function for βu=βv=3 and βu=βv=20.

Once the Sigmoid function has been defined, wedefine the 1D-Gaussian function which we have proposedto modulate the Sigmoid function as:

G(u,v,µ,σ) = exp{(-1/2)(((u-v/sqrt(2))-µ)/σ)2} (7)

where µ is the mean and σ the standard deviation.Figure 3 shows an scheme of the purpose of this function.

Figure 3.A 1D-Gaussian function is used to modulate theSigmoid in the direction u=v. For a point (u0 ,v0) the Sigmoid

function is modulated by the value of the Gaussian at theposition (u0 - v0)/sqrt(2).

Hence, the final expression for our proposed model,which will be used as characteristic function for eachcolor category is:

SG(u,v,βu,βv,µ,σ) = S(u,βu)·S(v,βv)·G(u,v,µ,σ) (8)

In figure 4, some examples of the Sigmoid-Gaussianfunction can be seen for different values of theparameters.

a) βu=10,βv=10,µ=0,σ=1.5 b) βu=20,βv=20,µ=0,σ=0.5

c) βu=5,βv=20,µ=0,σ=2 d) βu=20,βv=20,µ=0.5,σ=1.5

Figure 4. Examples of the Sigmoid-Gaussian function fordifferent values of the parameters.

As it has been shown, the Sigmoid-Gaussianfunction is defined on the uv space. Previous approacheswith Gaussian models have shown the usefulness ofintensity component for colour naming. Therefore, inorder to consider the intensity component, we have

divided the uvI colour space in three levels of intensity.The values that define the three levels have been chosenin order to isolate some categories in only one or two ofthe intensity levels (i.e. yellow is only present for highintensity, orange does not appear for low intensity,…).The values which divide the I component into threelevels are I=1.8 and I=2.2. For each one of the threelevels of intensity, all the samples inside the interval arerepresented on a 2D uv-plane.

Once our Sigmoid-Gaussian colour naming modelhas been defined, the next step is to estimate theparameters for each characteristic function. For a certaincolour category Ci in a given intensity level Ij, themodelling process works as follows.

Firstly, all the samples of the learning set withintensity co-ordinate included in the level Ij are selected.From this subset, we find the sample with label Ci whichis the nearest to the center of the chromaticity diagram inthe uv space. This sample is translated to the origin of theuv space and the same translation is applied to all thesamples in the subset for Ij. The next step is to rotate thelearning data in order to place the samples of the colourcategory which is being modelled in the first quadrant,where the Sigmoid-Gaussian has its plateau form. Aninitial fixed rotation for each category is applied, but thefinal value of the rotation α is also estimated. Thus, fiveparameters must be adjusted in the modelling process: α,βu, βv, µ and σ. When the samples of the category to bemodelled have been situated in the first quadrant, theparameter estimation is again performed as a non-linearleast-squares data fitting by the Gauss-Newton method.

The process described above is repeated for eachcolour category in each intensity level, except for thegrey category. As we have mentioned before, the greycategory is well modelled by a multivariate Gaussianfunction. Hence, after the eight chromatic categories havebeen modelled, the parameters of a 2D multivariateGaussian function are estimated for each intensity level.

The results of the parameter estimation performedfor the Sigmoid-Gaussian model are presented on tables1 to 4.

Table 1. Model parameters for I < 1.8Colour αααα ββββx ββββy µµµµ σσσσRed 1.18 40 69 0.06 0.01Brown 1.13 283 397 0.01 0.05Green -0.74 60 62 -0.02 7.54Blue -1.95 188 64 -0.05 0.03Purple 2.17 322 155 -0.01 0.18

Table 2. Model parameters for 1.8 < I < 2.2Colour αααα ββββx ββββy µµµµ σσσσRed 1.47 64 80 0.03 0.03Orange 0.89 50 99 0.04 0.06Brown 0.79 255 383 -0.04 0.07Green -0.39 52 164 0.23 17.56Blue -2.19 238 145 -0.01 0.02Purple 2.01 330 294 0.00 0.04Pink 2.00 61 68 -0.05 0.02

Table 3. Model parameters for I > 2.2Colour αααα ββββx ββββy µµµµ σσσσRed 1.64 107 14 0.00 0.03Orange 0.93 70 78 -0.02 4.04Brown 0.35 -16 38 0.00 0.02Yellow 0.37 477 111 0.01 0.01Green -0.44 75 409 -0.09 23.67Blue -2.41 149 174 -0.01 0.02Purple 2.16 464 209 0.01 0.03Pink 1.52 328 534 0.00 0.04

Table 4. Model parameters for the grey membershipfunctions with µµµµ=(µµµµ1, µµµµ2) and S=(σσσσ0 σσσσ1; σσσσ1 σσσσ2)Level µµµµ1 µµµµ2 σσσσ0 σσσσ1 σσσσ2

Int <1.8 0.64 0.42 0.0015 0.0001 0.00011.8<Int<2.2 0.65 0.43 0.0006 0.0001 0.0001Int >2.2 0.64 0.42 0.0006 0.0001 0.0001

Figure 5. Samples Learning set (left) and membership valuesfor intensity level I < 1.8

Figure 6. Samples Learning set (left) and membership valuesfor intensity level 1.8 < I < 2.2

Figure 7. Samples Learning set (left) and membership valuesfor intensity level I > 2.2

On the left part of figures 5, 6 and 7, there are thelearning subsets which have been used to estimate theparameters of the characteristic functions for each levelof intensity. On the right part of the same figures, wehave represented the membership values for eachcategory. As it can be seen in these figures, most of thecharacteristic functions occupy a narrow band in thechromaticity diagram, leaving gaps between the differentcolour categories. In most of these cases, the same gapscan be found in the diagram of the correspondinglearning subset. This fact indicates that more coloursamples should be included in the learning set, in order tocover a wider area of the chromaticity diagram with theestimated characteristic functions.

Results

In order to evaluate the results obtained by the statisticalmodelling of the colour naming space, we have done acomparative analysis between our Sigmoid-Gaussianmodel and the Gaussian-based models that have beenpresented before.

The psychophysical experiment described above, hasprovided us a set of membership values for all thesamples in the learning set. Hence, the experiment resultshave been considered the target values for the output ofthe characteristic functions. During the statisticalmodelling of the colour naming space, the mean-squarederror (MSE) between the values returned by the modelsand the ones obtained in the experiment has been used asa goodness measure of the model fit to the colour data.

However, as the main goal of a colour namingsystem is to assign a colour term to a colour stimulus, weshould also evaluate the validity of the model in terms ofthe number of samples which are correctly named. Thenaming of a colour sample is performed by just assigningthe label corresponding to the colour category with thehighest membership value.

The models considered in the analysis are theGaussian model estimated without taking into account theresults of the experiment, the Gaussian model fitted to theexperiment data and the Sigmoid-Gaussian model. Allthree models have been tested over the described uvIspace, but also over the uv space without considering theintensity component. Notice that the Sigmoid-Gaussianmodel over uvI space corresponds to the three intensitylevels scheme described in the previous section. Theresults obtained by the three models are presented intable 5.

Table 5. Results.Model Colour

SpaceMSE % of Correct

Colour Nam.Gaussian uv 0.2755 85.07%Gaussian uvI 0.3485 92.42%Fitted Gaussian uv 0.1634 86.02%Fitted Gaussian uvI 0.1105 91.94%Sigmoid-Gaussian uv 0.1396 85.55%Sigmoid-Gaussian uvI 0.0726 92.42%

Although for a given colour space, the percentage ofcorrect colour naming is approximately the same for thethree models, results in terms of the MSE show theimprovement provided by the use of the membershipvalues obtained from the experiment in the colournaming space modelling. Moreover, the fact that MSEfalls to 0.0726 for our Sigmoid-Gaussian model confirmthe suitability of the statistical modelling of the colournaming space by non-Gaussian models. Finally, weshould notice the fall in the performance of the models interms of the MSE and in the percentage of correct colournaming, when the intensity dimension is eliminated. Thisresult shows the importance of intensity component forcolour naming.

Conclusions

In this paper, we have proposed a non-Gaussian modelfor statistical modelling of a colour naming space. Thismodel, based on a 2D-Sigmoid function modulated by a1D-Gaussian function, is used as characteristic functionfor colour categories in a fuzzy-set approach to thecolour naming problem.

The model is fitted to a set of psychophysical dataobtained from an experiment we have done. Althoughthis learning set includes a wide set of colour samples, itis far from being a complete learning set. This fact, hascaused some problems which could be solved by buildinga more complete colour data set.

Nevertheless, results obtained in terms of MSEbetween the output of the model and the values obtainedfrom the experiment confirm the suitability of theSigmoid-Gaussian model as a colour membershipfunction for colour naming.

To go further with this work, it would be interestingto validate the modelled space with other sets ofpsychophysical data. There are several previous colournaming experiments with large sets of data where a basiccolour term is assigned to a given stimulus 8,9,10,11,12 butwe are not aware of any experiment asking formembership assignments to the basic colour terms.

Moreover, the model, should also be tested on a realtest set, in order to evaluate its performance in a realproblem.

Acknowledgements

This work has been partially supported by projects2FD97-1800 and TIC2000-0382 of Spanish CICYT, byUAB FIPD grant and by Casinos de Catalunya.

The authors would also like to acknowledge CristinaCañero, Dèbora Gil and Daniel Ponsa for their help andsuggestions during this work.

References

1. P. Kay and C.K. McDaniel, The linguistic significance ofthe meaning of basic color terms, Language, 3(54), 610(1978).

2. D. Alexander, Statistical Modelling of Colour Data andModel Selection for Region Tracking, PhD thesis,Department of Computer Science, University CollegeLondon, 1997.

3. J.M. Lammens, A Computational Model of ColorPerception and Color Naming, PhD thesis, University ofNew York, 1994.

4. B. Berlin and P. Kay, Basic Color Terms: TheirUniversality and Evolution, University of CaliformiaPress, Berkeley, 1969.

5. G.D. Finlayson, B. Schiele, and J.L. Crowley,Comprehensive colour image normalization, Proc.ECCV’98, pg. 475. (1998).

6. M. Vanrell, F. Lumbreras, A. Pujol, R. Baldrich, J. Lladósand J.J. Villanueva, Colour Normalisation based onbackground information. Proc. ICIP'01, V.I, pg. 874(2001).

7. R. Benavente and M. Vanrell, A colour namingexperiment, Technical Report #56, Computer VisionCenter, Barcelona, 2001.

8. R.M. Boynton and C.X. Olson, Locating basic colors inthe OSA space, Color Research and Application, 12(2), 94(1987).

9. R.M. Boynton and C.X. Olson, Salience of chromaticbasic color terms confirmed by three measures, VisionResearch, 30(9), 1311 (1990).

10. J. Sturges and T.W. Allan Whitfield, Locating basic colorsin the Munsell space, Color Research and Application,20(6), 364 (1995).

11. J. Sturges and T.W. Allan Whitfield, Salient features ofMunsell color space as a function of monolexemic namingand response latencies, Vision Research, 37(3), 307(1997).

12. S. Guest and D. Van Laar, The structure of colour namingspace, Vision Research, 40, 723 (2000).

Biography

Robert Benavente is a research fellow in the ComputerScience Department of the Universitat Autònoma deBarcelona and is affiliated of the Computer VisionCenter. He received his MSc degree in Computer Visionfrom the Computer Vision Center in 1999. In addition, heparticipates in the development of industrial applicationsrelated with quality control processes based on computervision. His research interests include colour perceptionand representation, and colour naming.