Colour Image Search for Graphical Design InspirationHenrietta Rydfalk

Linköpings UniversitetLinköping, Sweden

henry773@student.liu.seISRN: LIU-IDA/LITH-EX-G--19/057--SE

ABSTRACTThis thesis project aims to develop a search algorithmwhich uses the colour information in uncompleted graphicalwork to find inspirational images that utilizes similar colourschemes. The dominant colours are separated from theimage, and those are used to find other images with similarpalettes in a database. The system is intended as part of aninspirational mood board, and the user case this project isbased on handles the design of a book cover, and theselection of images used to be searched through consistsentirely of book covers. The development is done inOpenCV and C++, using the k-means cluster method. Theresults are inconsistent, and while the fault in the system isknown, it has yet to be mended.

Author KeywordsImage search; image analysis; graphic design; searchalgorithm; colour harmony; colour map;

INTRODUCTIONIn this day and age, software capabilities only grow, and wegain new possibilities constantly. Even within the morespecific subject of image recognition and analysis, we see awide variety of applications. From a search engines ‘findrelated images’ to applications within medicine. In thiscase, the relation to consider is colour.

Colour search and analysis is not new. There are alreadyexamples on searching by colour on the market today, suchas TinEye Labs[1]. The idea for behind this thesis work isto utilize colour search automatically during a designprocess of graphical work, while requiring minimal inputfrom the user. This to allow the user to focus on thegraphical design and not have to break their focus to findreference images.

A user case that I will be focusing on here is that of aninspirational moodboard. A graphical artist or team ofartists are going to design a book cover. The first search isthe usual one, based on keywords. Such as ‘book cover’,‘thriller’, ‘blue’ and ‘Victorian’. The algorithm thensearches for fitting images to display. These will bedisplayed in the background, or on a secondary screen, andsupply the artists with ideas and inspiration relevant to whatthey are doing. As the artists start working, the mood-boardsoftware will analyse the in-progress work and update the

displayed mood-board accordingly. If it notices that thecolour palette of the work updates, the inspiration imageswill be adapted accordingly. It is important that it is not toofast with replacements since, for example, starting overfrom a white canvas should not change all the inspirationalimages to white canvases.

The moodboard displays images in different sections. Thisis to allow for variety in the inspirational images, while stillnarrowing down each search to be as relevant as possible.The sections would include one section for colour palettesfor similarly coloured images.

PurposeThis thesis work is to design a piece of the moodboardsoftware. This software will be using the colour informationin the users work to find and expand the colour palette withexamples. The mood-board is intended to be displayed on asecondary display and require minimal user interaction.

Research questionHow can one extract dominant colours from an image inorder to find inspirational images with similar palettes?

BACKGROUNDThe driving force behind this project came about throughthe design of a room in Kopparhammaren 2, at theuniversity campus in Norrköping. It is a circular room withdisplays around the walls, and one table in the centre. Oncein use, it became clear that this room was not suited for anymeeting or conference where everyone in the room is meantto focus on the same point on one display. You want tolooks inward, at the centre of the table, not outward at thescreens.

When discussing better uses for this setup, the idea ofperipheral displays came about. The users focus on a centralpoint, and are surrounded by more general messages orimages in their peripheral vision. One implementation ofthis would be the mood-board. A mood-board providesinspiration during a design process, by showing imagesrelevant to what is being designed. It is desired that, as adesigner works, they should not have to search and findthese inspirations themselves, allowing more focused workwhile still always having relevant references around them.

- 1 -

RELATED WORKImage analysis has been a developing area for some time,one example is MindFinder [2] from Microsoft Research2010. The system in MindFinder uses a combination ofgraphical information and labels to achieve better searchresult than keywords alone. It allows the user to draw on acanvas and label the symbols drawn. Such as a circlesomewhere on the canvas labelled ‘sunset’. The systemdoes take colour into account if it is added.

Another article from the same time and research centrefocuses instead on colour maps [3]. In this system, the userfills in broad colours on a grid canvas, and the systemsearches for images that matches these colours. Anotheroption is to select an already existing image, and then maskthe cells on the grid that is to be used for colour matching.Note that one effect of the grid is that layout information istaken into account. The authors conclude that while helpfulin certain ways, the system is limited when determiningshapes. However, in this project colour can be used andanalysed separately from the specific motives on the image,and so the lack of shape recognition is not a drawback.

Generating and expanding colour palettes based on colourtheory has been done as well. There are numerous freeapplications for generating colours such as [4,5]. These areapplications, and the theory used for generation is notimmediately offered. A more theoretical approach toautomatically finding good colour palettes has been done in[6]. One issue with automatic colour palette generators, andrelying on guidelines of colour harmony is that manyfactors influence the way humans experience colour. Theauthors of the article admit to the inherent subjectivity ofcolour harmony. Stating that there are many factors on whatany given person finds pleasing.

THEORY

Colour HarmonyThe choice of harmonious colour combination is an ancientfield, going back to Aristotle, and how colours are viewedhas changed over time [7]. Today, much of the theory ofcolour harmony centre around the colour wheel andcombinations around it.

In order for an algorithm to make suggestions on colourpalettes it needs a formal system to calculate these. In onearticle, O’Connor [8] brings up that many previous theoriesand attempts to formalize colour harmony are notscientifically sound. Humans experience colour and formopinions on colours based on many often very personalfactors, such as age, sex, culture, personal preference andmental state. Hård and Sivik[9] exclude aesthetic pleasureand beauty from their formalised model of colourcombinations for this reason.

Colour mapsTo suggest and expand on colour palettes it would bepossible to rely only on the colour map with empty areas as‘any’, as is described in [3].

A colour map divides the image into cells, 8x8 is suggestedby [3]. This would bring the maximum amount of coloursto 64. The map can then be used to match different imageswith similar colours.

The colour map may contain empty cells, which do not takepart in matching. If The input images includes a largeamount of unused canvas, then the background canvascolour (usually white) can be considered empty cells.

One potential issue with this is that it is possible for theartist to lay down only one baseline colour to start, whichrisks limiting the colour map to the point it is no longerproperly helpful. Another possibility is that if the artist startwith small islands of colour, there are too many options forthe algorithm, and its choices will be chaotic. Thirdly, it ispossible for large swatches of white space to be a desiredpart of the design.

Extracting colour paletteIn order to extract the colour palette from a coloured image,generally the number of colours have to be reduced. Inimages using few block colours, as is often the case withcompany logos, for example, those colour can be extracteddirectly. Anything painted or only using gradients howeverwill have colours that are similar, but still differ in pixelvalue.

However, colours may be similar to each other. If an imageis mainly blue, but also includes red and yellow, we do notwant the colour palette to only show shades of blue. It is forthis reason the k-means cluster method is used[10].

K is the number of clusters centres used. Initially, clustercentres are placed randomly. Each data pint will be attachedto its closest cluster centre. Then, the centres are moveduntil the clusters are as dense as they can be.

In the case where the data points are pixel colour values,each cluster centre. represents a single colour.

METHODThe system for finding inspirational images based on colourpalette follows the general structure of [11] The system willfirst need to extract the colour palette from that input. Thenthe palette is run through the palette editor. Finally, thecolour palette is compared to the colour palettes of theimages in the database. Both the palette editor utilises acolour comparison. This is shown graphically in Figure 1.

The development was done using OpenCV, a free libraryfor computer vision [12].

- 2 -

Removes similar coloursRemoves whites and greys

Extract palette

Input image

Comparepalettes

Compare colours

Parameters:Min saturtionMax lightnessColour cutoff

Output images

K-means methodColour map

DB

Parameters:Kmxm

Edit palette

Parameters:HueDiff weightSatDiff weightValDiff weight

Figure 1: A graph of the system, and the parameters that areused.

PaletteFor the extracting the colour palette from the input image,two approaches were tested. First, the colour map approach.An image was loaded, and divided into a grid of 4x4 or 8x8.Because OpenCV uses BGR (Blue-Green-Red) as itsstandard colour model, the image will be converted to HSV(Hue, Saturation, Value), which is more intuitive.

From each cell of the grid, a dominant colour wascalculated using a the k-means method for k=1. This resultsin the mean value of all colours in that cell. Then the wholecell was given the resulting colour mean. The resulting gridis the colour map.

In the second, more direct approach, the colour palette wasacquired immediately with k-means. Where k is the size ofthe palette. The use of k=4, k=6 and k=8 were tested, butultimately, k=6 was chosen as the best suited with thiscurrent setup,

Colour palettes were also calculated for the comparisonimages in the database. The tree most dominant colourswere selected with the k-means method for k=3. The threecolours were then saved.

For the database, one set of book covers from InternetArchives Open Library[13] was used. From this, a smallerselection was picked out for the testing and adjustingparameters. This was done for faster calculation, and tohave a reasonably sized data set so that the preferred imagescould be chosen from it without having to personally lookand judge each of the examples.

The palette comparison starts at the first colour from theinput palette, and compares it with all three colours of thecurrent item in the list of searchable images. Thedifferences found are combined into one value. This is thendone for each colour in the input palette. The images withthe smallest difference were then displayed.

High value and low saturation colours were removed fromthe input palette. This is to let the algorithm to ignore say ablank canvas that has not been utilized yet, and also toassure it does put too much weight in the most neutralcolours. The palette editor removes colours of lightnesshigher than x, and saturation lower than y.

Design decisionsOne problem with creating formal system of colourharmony is that we deal with only aesthetics, and what isconsidered aesthetically pleasing differ between ages,cultures, individual people, their mood and the context inwhich the image is seen. For this reason, formalisedsystems to achieve colour harmony will not be completelyrelied on in this thesis work.

A potential work around in using a selection of images suchas book covers. This adds one more layer of quality controlinto the process. The covers of published books aregenerally made by professional designers, and thereforetheir colour composition have at least been judgedacceptable by both an artist and a publisher. It is possiblethat, in order to find more structure, simple aesthetic rulesof this time and culture can still be used for a decent effect,even without relying on them too much.

Palette matching between the input and the database imageswill be done through the main 3 colours of the comparedimage palette. The intent is that three colours will allow forthe large scale colour information to be taken into account,while still allowing variety in how they are used, and foraccent colours in the result which may not appear in theinput. It is not reasonable to expect the HSV values tomatch exactly, therefore a way of calculating if two coloursare close enough to each other will be required. The highestsensitivity will be on hue, second to saturation and lastlyvalue. This means that hues have to be very similar tomatch, but allows more variety in value. The exactweighting for a good result will be determined iteratively.

Using a limited number of colours for the matching forgreater variety in the search results, while still keeping themrelevant.

- 3 -

Colour comparisonThe comparison in colours is done from HSV. The hue,value and saturation values are compared separately.Saturation and value are straight forward, and a simpleabsolute division can be used. With hue, one must be awarethat it is circular. The lowest and highest value are thesame, making the maximum difference only half the rangeof the h value.

Then a weight was applied to each value difference, andthese were then combined to make the final colourdifference value. The standard values for these were hue-difference*5, saturation-difference*2 and value-difference*1. Since humans are the most sensitive todifference in hue, the system puts the most weight on hue,followed by saturation and lastly value.

Judging resultsThe algorithm will be working on a test case with one set ofinput images, and one larger data set of images to chooseform. Each input image is given a small number ofpreferred results. This is to have some measure of thesuccess rate of the system. Examples of input and preferredoutput follow in Figures 2 to 4. Figures marked ‘a’ are theinput images and following are preferred choices forinspiration.

Figure 2a: Input image A.A generally pale browncolour scheme with a redfigure in the centre.

Figure 2b:Apreferredresult forimage A.

Figure 2c: Apreferred resultfor image A.

The first example, image A, shown unedited in Figure 2agoes mainly in browns but shows also distinct red. It isdesired that the results show browns and reds. Figure 2band 2c show two examples of what is desired to see in theresults.

Figure 3a: Input image B. An unfinished drawing withuncoloured canvas and twodistinct colours central.

Figure 3b: Apreferred result forimage B.

The second example, image B is shown unedited in Figure3a. It displays an unfinished drawing, the canvas is mostlyuncoloured, but with a figure in the centre coloured withblue and red. It is desired that both of those colours show inthe results. An example of a desired result is need in Figure3b.

Figure 4a: Input image C. An artwork byAlfonso Mucha

Figure 4b: Apreferred result forimage C.

The final example image used here, shown in Figure 4a isan artwork using mainly light colours. White, yellow andorange, with some green and blue also. An example of adesired result is shown in figure 4b. The image of 4b goesin mainly yellow and orange, but also relies on more bluetones than the input.

- 4 -

RESULTTests where done on a set of input images, including thosediscussed in the previous section. Shown here are aselection in order to illustrate the general results.

The colour map technique using grids gave results as seenin Figures 5 to 10.

The first two used a 4x4 grid. Input colours with a valueabove 200 or saturation below 30 were not considered. Thecolour comparison prioritised hue the highest, as describedin the Method section.

Figure 4 shows the 4x4 colour map of image A. Figure 5shows the first result of the search. Compare this to thereferences in Figure 2b and 2c. The result is not among thepreferred results for image A. The colour map goes mainlyin shades of brown, with some red and grey, and the resultsfollows a brown colour scheme.

Figure 5: Colour map of image Ausing a 4x4 grid.

Figure 6: Firstmatch result

The same technique for image B is shown in Figure 6. It ismainly white with darker grey colours in the centre, wherethe coloured figure is placed in the original. The originalcolours of red and blue do not show.

The same test was done again, with a 8x8 grid, shown inFigure 8. This increases the number of colours in the initialpalette from 16 to 64. That is, before reducing the palette bythe least saturated and most similar colours.

Using an 8x8 grid, blue and red shades start to appear.Looking closely, it can be seen that the most saturatedcolours are not the same red and blue as in the input image,but both go towards purple. The examples following thiswill all be using the 8x8 grid.

Figure 7: Colour map of image B using a4x4 grid.

Figure 8: Colour map of image B using an8x8 grid.

Figure 9: Search results for image B using an8x8 grid.

Out of the three firsts results from running image B throughthe system with an 8x8 grid, two of them are mainly blueand the third is a red-toned brown. See Figure 9. The firstresult is mainly a desaturated blue, but also include smallareas of red. The second is entirely in blue and white anddisplays no signs of the red colour. The third is theopposite, showing more red tones, but no sign of blue.

- 5 -

Figure 10: Search results for image C using an 8x8 grid.

Applying the same process to image C, as seen in Figure10, the first result goes in shades of brown, the second inbrowns and blue, and the third mainly in green and white.All of these colours are part of the oroginal, though thenoticeable orange of the original is absent here. None of theresults for B or C include their preferred results.

Following this, the second method was applied to the sameimages. This time, all colours of the image were reduced toa set number with the k means method. The same weightswere used for all of this and they are made with k=6. Figure11 shows image A with its colours reduced with k=6. Thecolours seen there are the colours of the palette, then usedfor comparison with the images of the database.

Figures 12, 13 and 14 show the results of input A B and Crespectively.

Figure 11: Colour reduction of image A using the k-meansmethod with k=6.

Figure 12: Search results for image A, reduced to 6 colours

Figure 13: Search results for image B reduced to 6 colours

- 6 -

Figure 14: Search results for image C reduced to 6 colours

For image A, the first two results are brown with some redtones. The second result is one of those initially chosen as apreferred result. The last one is brown and blue. There is noblue in the original image.

The three results for image B all contain some of the blue.The second image in entirely black and blue going towardspurple. The first image also contains more red tones andyellow. The third is heavy on the yellow tones. There is noyellow in the original image.

The first result for image C goes in shades of yellows andbrowns, and the second in brown.

DISCUSSIONOnly once did a preferred image show up in the results.This shows an issue in implementation, and also causesdifficulty when trying to judge the remaining results. Theresults still notably different, so their relative quality haveto be determined by personal judgment.

First the initial image was divided into 16 cells, and thedominant colour of each cell was selected with the k-meansmethod. This worked acceptably as long as the each colourfilled a large continuous area, see Figures 5 and 6. Howeverthis does not work in cases such as in Figure 7 wherenotable colours take up a relatively small space. In thiscase, the red and blue from the input image share a cellalong with much grey and white. This causes their meanvalue to become washed out and grey. The notable coloursin the original image dissapearing completely. The 8x8version seen in Figure 8 is an improvement. However, thecolours shows in the coloured cells are still not the samecolours as are displaued in the original. The use of a meanvalue still wash out the colours that share a cell with white,and if the red and blue hues occur in the same cell, this willnudge that cell towards purple.

One problem which has remained through multipleitterations is the one seen in Figures 15 and 16. Figure 15goes entirely in shades of blue, and the colours with nosaturation (in this case the black edges) should have beenremoved. Yet there is still a grey image included in theresults in Figure 16, and another result is mainly purple. Fora palette like that in Figure 15, I would have expected allresults to feature the blue.

Figure 15: Colour map which goes entirely in shades ofblue.

Figure 16: Search results for Figure 15

ParametersThe parameters of the colour compare, the weights on hue,saturation and value, have not been discussed much in thispaper previously. This is because, despite having assumedthat these would be the parameters that needed the mostadjustment, changes to the colour compare made very littledifference in the final results.

As long as hue had the highest priority, smaller changesmade no difference. If the order changed, and for examplevalue was given the highest priority, that would make theresults loose any semblance of connection to the input.Within the given priority, changes to the weights did notchange the results at all.

- 7 -

The minimum saturation and maximum value wereincluded to handle cases where the input had large amountsof colourless space. This did improve the results for imageB, which does include much uncoloured canvas. (Figure 3a)The values were set to 200 for maximum value and 30 forminimum saturation. A smaller cutoff made no difference,and larger cutoff started cutting too many colours. It wasthe value maximum that had the greatest effect there.

Working with these parameters only got me to a pointwhere the input had some visible similarity to the results. Itwas late in development that I realised that what reallyhindered the quality of the results was the palettecomparison.

Therefore, the palette comparison had to be suddenlyrewritten at the end of theproject. The effect was that aslong as any colour in the input palette matched well with asingle colour in the reference, it would score well, no matterif all others matched badly.

Now, instead the palette comparsion loops over the threecolours in the reference, and picks the best fitting colour inthe input palette for each. This combined value is then usedas the comparitor.

This is not optimal either. There are still examples whereone colour in the original is completely absent.

During my attempts to find the reason for these faults, Ihave come across information warning me against thespecific route I have taken. Shettini et.al write that wheninvestigating clustering as a method of colour reduction,one problem found was the need for a sophisticatedsimilarity measure[14]. The similarity measure I have used,while now free from its initial error, is still simple. Theauthors write that, when using methods similar to my own”...the risk of serious error should not be underestimated.”

I consider this a fault of my own preparatory work. Much ofthis was focused on colour harmony, which couldunfortunately not be implemented to the extent I hadplanned and hoped for, due to time constraints. In hindsight,I likely should have used a predefined palette to reduce thecolours of the images, rather than reducing by clusters. Thereliance on a simplistic similarity measure likelycompounds this problem.

Search efficiencyTwo versions of the searchable set was used, one selectionof about 150 images, and a larger set of just over 8000. Thelarger set was somewhat edited in order to remove some ofthe duplicates and sets of covers that were not judged to beuseful. Such as dictionaries between various languages, allusing approximately the same colour scheme. Even withvery limited information saved about each image, and onlythree colours extracted, it still takes hours for the programto run through the entire database and save the information.As the database manager is supposed to be run rarely andseparately, and not the focus of the work, this was deemedacceptable.

The time search for matches in this list is not as notable.There is a delay, but in seconds, not minutes. And as thefinal version of this product is intended for backgroundfunctionality, a delay in result will not be jarring. The userwill likely not know when new searches are being made atall.

Method validitySome choices were made early on, with the intention ofkeeping search times low. In hindsight, long search timenever caused a problem, even with the larger set of imagesused as database. I might have been better served by storingmore information and this having more information to use,in order to gain better results.

The decision to use book covers only for output is based onthe idea that these are already vetted, and this more likely tobe pleasing to the eye, and therefore useful as inspiration. Ifthe system itself could make a more specific vettingprocess, then this would not be necessary, or desired.

The images used for input has very little practical basis.One of them, image B, (see Fgure 3a) is a work in progress,but not for a book cover design, and not necessarilyrepresentative of how artists and designers in general work.In that regard, the system is not tested for the type ofimages it will likely to use.

- 8 -

Context and ethical considerationsThis article is, but own admission, dealing with a specificcase and not likely to have much effect outside of its ownpotential user base. The techniques are useful for thepurpose it was designed. No threats against individualprivacy has been proposed. The details will depend onapplication in the finished product, but it is worth to keep inmind that, since the application is intended to do the searchwith a users work in progress, any rights to those works inprogress should remain the property of the user. A softwarethat claims ownership of the images it uses to search wouldneed to be considered from an ethical standpoint, as thismay infringe on the ownership rights of the artists that usesit.

CONCLUSIONThe image finding system that has been developed, it takesan input image, extracts a colour palette from it by reducingthe number of colours, and compares the palette to otherpalettes in a database, extraced using similar means. Thecolour reduction is done with a cluster method basedaround mean values. The comparison between coloursutilizes the HSV colour system. Each component in thesystem can be adjusted.

The image finder runs and finds inspiration images from thedatabase based on the colour palette of the input. However,most of the results are unsatisfactory.

Despite the difference in hue weighting the heaviest, missesin hue are still very noticeable in the results. This can beseen especially in the third picture in Figure 12, where blueshows up without having a reason to, in Figure 13 whichfeatures more yellow than the original image would indicateand in Figure 16, where the results would be expected to allheavily feature blue, though not all do.

The main issue at fault is palette comparison, which is toosimplistic. Especially given the palette extraction method,which would require a more sophisticated palettecomparison in order to work well. While the structure of thesystem is sound, better results would likely be achieved ifeither a predetermined palette was used for colourdiscretization of the input images, or if the method ofpalette comparison was made more sophisticated to matchthe current colour palettes.

Future workThe first thing to be done is repairing the issue with thecolour compare. No other adaptations are likely to improveresults until this is done.

Only after this, would it be appropriate to start workingintentional colour suggestions. Currently, the coloursuggestions is only based on the fact that the images in thedatabase are saved with their three most dominant colours.Any other colour will supply variety the palette. It is theintention that in the future, implementation of colour theorywill be used to determine what extended colour paletteshould be chosen as a result.

It is also worth remembering, that the quality of the resultsis currently based on personal judgement and preference. Itwould be advantageous to bring in potential users of thesystem, and have a survey of their opinions as basis forwhat is a good result and not. On that note, input imageswould likely offer more value, if they were truly based onthe works in progress of potential users. It would be usefulto ask artists and designers who might used the finishedsystem what their works in progress look like, since this iswhat the system will be using in the end.

- 9 -

REFERENCES[1] TinEye. 2019. Multicolr: Search by color[https://labs.tineye.com/multicolr/] Accessed 4 April 2019.

[2] Y. Cao, H. Wang, C. Wang, Z. Li, L. Zhang, and L.Zhang. Mindfinder: Interactive sketch-based image searchon millions of images. In ACM International Conference onMulti-media, 2010

[3] J. Wang, X. Hua. Interactive Image Search by ColorMap. ACM TIST, 2011.

[4] Michaela Lavicka. 2017. ColorSpace[https://mycolor.space/] Accessed 4 April 2019.

[5] Fabrzio Bianchi. 2018. Coolors [https://coolors.co/]Accessed 4 April 2019.

[6] Lara‐Alvarez, C, Reyes, T. A geometric approach toharmonic color palette design. Color Res Appl. 2019

[7] Stokley, Steele R., Historic Look on Color Theory"Honors Theses - Providence Campus. 30. 2018

[8] O'Connor, Z. Colour harmony revisited. Color Res.2010

[9] Hård A, Sivik L. A theory of colours in combination—A descriptive model related to the NCS colour-ordersystem. Color Res Appl 2001.

[10]MacQueen, J. Some methods for classification andanalysis of multivariate observations. Proceedings of theFifth Berkeley Symposium on Mathematical Statistics andProbability, Volume 1: Statistics, 281--297, University ofCalifornia Press, Berkeley, Calif., 1967.

[11]Mehtre, B. Kankanhalli, M. Narasimhalu, D. GuoChang Man. Color Matching for Image Retrieval. PatternRecognition Letters Volume 16 Issue 3. 1995.

[12] OpenCVteam. 2019. OpenCV [https://opencv.org/]Accessed 28 April 2019

[13] InternetArchives 2019. Open Library Covers API[https://openlibrary.org/dev/docs/api/covers] Accessed 22May 2019

[14] Shettini R, Ciocca G, Zuffi S. A Survey of Methodsfor Colour Image Indexing and Retrieval in ImageDatabases, 2002

- 10 -