Mobile Photo Collage

Man Hee Lee†maninara@inha.edu

Nitin Singhal‡n.singhal@samsung.com

Sungdae Cho‡s-d.cho@samsung.com

In Kyu Park†pik@inha.ac.kr

†School of Information and Communication Engineering, Inha University, Incheon 402-751, Korea‡DMC R&D Center, Samsung Electronics Co. Ltd., Suwon 443-742, Korea

Abstract

In this paper, we propose an efficient technique for creat-ing a visually appealing collage on a mobile platform froma set of input images. The proposed algorithm consists offour main modules, namely image ranking, region of inter-est (ROI) selection, packing, and blending. Each of the fourmodules is designed using a greedy and localized approach.The modules are further optimized during implementationfor efficient porting on a mobile phone processor. Exper-imental results show the effectiveness of the proposed al-gorithm with visually appealing results on an off-the-shelfmobile phone.

1. Introduction

Recent advance in digital imaging technologies enablespeople to have high resolution digital cameras in mobilephones. Today, camera phones with 3 ∼ 10 mega pix-els and with video capture capability are quite common.The modern mobile phone is also a computing powerhouse.It has a capable CPU, high quality color display, and co-processors or DSPs for image/video encoding and decod-ing. The growth in camera technology coupled with the in-crease in storage capacity on modern mobile phones, facil-itates taking striking photographs without the need to eraseany. However, visualizing hundreds of these photographs,while browsing or moderating is a long and tedious task.

With the explosion of image files number, the ability tomanage the content of a huge number of images has becomea key skill. In order to speed up the process of visualizing,collage framework is proposed as a technique to summarizehuge image libraries. A collage represents an image libraryas a set of meaningful snapshots that allows to summarizethe content of the library. Rother et al. [6] proposed an auto-matic procedure for constructing a collage from a set of in-put images using image processing and global optimizationtechniques. Their work is an extension of Digital Tapestry[7]. In [1], an authoring tool which supports efficient con-

(a) (b) (c)

(d)

Figure 1. The procedure of running mobile photo collage on asmart phone (Samsung Omnia II). (a) Selecting a directory. (b)Image thumbnails. (c) Image selection. (d) Created photo collage.

struction of a collage using high level interaction and anautomatic layout procedure using a pre-designed templateis proposed. In Picture Collage [9], they define the packingand ordering modules by employing a Bayesian frameworkand adopting Markov chain Monte Carlo (MCMC) algo-rithm for the energy minimization. Yang et. al. [12] pro-posed a dynamic collage using the picture collage frame-work with an option to add image dynamically using be-

InputImages

ImageRanking

ROISelection

ROIPacking

ImageBlending Collage

Figure 2. Block diagram of the whole procedure of the proposedmethod.

lief propagation. Google’s Picassa [2] presents a fairlysimple photo collage design which is able to organize andedit a number of pictures. On the video front, Video Col-lage [10, 4] is proposed as a tool to summarize video se-quences.

However, all of the above applications are targeted onthe personal computer (PC) and are designed to utilize hugememory and processor resources. The complexity of thealgorithm and the need for huge memory resource serves asa major bottleneck in efficient porting of these algorithmson modern mobile phones.

In this paper, we propose an optimized technique for cre-ating a visually appealing collage from a set of input im-ages. The base framework is similar to AutoCollage andour major contribution rests in the design of a greedy and lo-calized algorithm for efficient porting on the mobile phoneplatform. The proposed algorithm consists of four mainmodules, namely image ranking, region of interest (ROI) se-lection, packing, and blending. Unlike AutoCollage, whichuses global optimization in its main algorithm, each mod-ule in the proposed algorithm is designed using a greedyapproach, which is low in complexity but still shows com-parable results. The modules are further optimized duringimplementation on the mobile phone. Experimental resultsshow the effectiveness of the proposed algorithm with vi-sually appealing results on an off-the-shelf mobile smartphone.

2. Proposed Photo Collage AlgorithmThe proposed algorithm consists of four main modules:

(i) image ranking; (ii) ROI selection; (iii) ROI packing; and(iv) blending. Firstly, the input images in a folder are rankedusing a numerical importance factor to select the most dis-tinct and informative images. This is followed by croppingthe salient region, also known as region of interest (ROI)from each of the selected images. Next, we arrange thecropped ROI images onto the collage canvas with the con-straint that no two ROIs overlap and the canvas is coveredmaximally by the images. Finally, we blend the imageson the canvas using a modified alpha blending algorithm togenerate the final photo collage. Figure 2 shows the blockframework of the proposed algorithm.

2.1. Image Ranking

The goal of image ranking is to select the most distinc-tive and informative images from a set of input images bycomputing the importance of images. The importance of an

(a) (b) (c)

Figure 3. Examples of local importance map. Brighter pixel hashigher importance. (a) Input image. (b) Using local histogramentropy. (c) Using saliency map.

image is measured as the sum of the self-importance and therelative importance.

We define two factors to estimate the self-importance ofan image. First, we use the local importance map by com-puting the pixelwise importance and taking the mean. If theimage includes a face, we set the importance of the pixelsin the face region to the maximum. Note that we employthe face detection algorithm [8] to find face regions. If oneor more face is detected, the normalized area of the face re-gion(s) serves as the second factor. The self-importance isdefined as the sum of these two factors.

For calculating the local importance map, the frameworkprovides different options with different degree of algorithmcomplexity. There is a tradeoff between complexity and ac-curacy. The low complexity one calculates the importancemap using local histogram entropy for a and b channel inLab color space. The high complexity algorithm employsthe saliency estimation [3]. The algorithm creates a Gaus-sian pyramid for L and ab space, and calculates the im-portance map by adding Gaussian pyramid across differentscales. Figure 3 shows an example of the importance mapsobtained by the two algorithms. As it is shown, the saliencymap performs better since it has higher importance value onthe foreground object with uniform color. Note that the im-portance map is displayed without detected faces to observethe difference more clearly.

In order to compute the relative importance between im-ages, it is necessary to measure the image-to-image simi-larity. In collage generation, an image is relatively moreimportant if it is distinct from others images. Therefore, therelative importance of the image gets higher if the distanceto other images (defined as the minimum distance among allthe distances) is higher. For computing the image-to-imagedistance, the hybrid graph representation [5] is employed inour approach.

In the ranking process, the top ranked image which hasthe largest importance value is selected first. Then, the se-lection procedure is performed iteratively until all imagesare selected and ranked. Note that, in each iteration, al-ready selected images are not considered in computing therelative importance. Alternatively, the top ranked image is

(a) (b) (c)

(d) (e) (f)

Figure 4. Examples of ROI selection. The rectangle denotes theROI. (a)(d) Input image. (b)(e) Using local histogram entropy.(c)(f) Using saliency map.

selected as the one with maximum number of faces. Thefollowing rank images are then selected in the same manneras described above.

2.2. ROI Selection

From the local importance map which contains pixelwiseimportance, the ROI is obtained in an iterative and greedyregion-growing manner. The initial ROI for the iterationis set as follows. If the image includes a face, we choosethe bounding box of the face regions as the starting region.On the other hand, if no face is detected, then the imageis divided into regular grids and a grid cell with maximumimportance is set as the starting region. In the iterative pro-cedure, an expanding direction among the four possible di-rections (up, down, left, and right) is selected such that ityields the maximum increase of the importance. The iter-ation continues until the expanded region reaches a prede-fined threshold (70% in the current implementation) of thetotal image importance. The expanded region is selected asthe ROI of the image.

Figure 4 shows several examples of ROI selection us-ing different local importance maps. Given an image inFig. 4 (a), the resultant ROIs are similar as shown inFig. 4 (b) and (c), when the high frequency features are dis-tributed across the image uniformly. However, if the dis-tribution is not uniform as an example in Fig. 4 (d), it ismore accurate to use the saliency map than local entropyhistogram as shown in Fig. 4 (e) and (f). Consequently, inour implementation, we decide to use the saliency based al-gorithm to calculate the importance map, even though thecomplexity of the algorithm is a bit higher.

(a) (b) (c)

(d) (e) (f)

Figure 5. An example of ROI packing procedure. (a) Result ofK-means clustering. (b) Initial position of each ROI’s center. (c)Initial layout of each ROI. (d) Each ROI is shifted to the centerof its movable area. (e) Each ROI expands to its maximum sizewithout overlapping with neighboring ROIs. (f) Final result aftera few iterations of ‘shift and expansion’.

2.3. ROI Packing

In the packing process, the K-means clustering algorithmis applied to the canvas area to deduce the initial locationand size of each ROI, which is shown in Fig. 5(a) and (b).In K-means clustering, we set the location of the top rankedimage to the cluster which is closest to the canvas center.The lower rank ROIs are placed in a greedy manner suchthat they are placed according to the similarity between theaspect ratio of the ROI and the bounding box of the cluster.

In our approach, packing is performed by iterativeexpand-and-shift procedure as shown in Fig. 5(d)∼(f). Eachof the ROI is maximally expanded while keeping the as-pect ratio until any two of them overlap. Then, each ROI isshifted to the middle location of its movable area. This pro-cedure is repeated until the amount of shift is below a prede-fined threshold. In this way, it is possible to maximize theROI region while keeping the non-overlapping constraint.Note that the top ranked image is allowed to expand firstsuch that it occupies the largest area than other ROIs.

However, the greedy packing algorithm can generateholes since maximizing the ROI size does not assure thateverywhere on canvas is always covered by at least a sin-gle image. To overcome this problem, we introduce anotherconstraint on the size of input image and the location of ROIduring the expansion step. If there is uncovered area aroundan ROI after shifting, then the ROI is free to move to fillthe hole while no new hole appears at the opposite side. Ifthe size of image area is smaller than the area that its ROIcan move, then the image moves to the center of the spacebetween neighboring images.

(a) (b)

Figure 6. An example of local smoothing map. The rectangle de-notes the ROI. (a) Input image. (b) Local smoothing map.

Figure 7. An example of mutual smoothing map.

2.4. Image Blending

Final step of the proposed algorithm is blending. Wepropose a customized alpha blending technique to smooththe transition area between images. Firstly, we define a localsmoothing map al of each image such that the pixel insidethe ROI has the maximum alpha value, 1.0, and it decreasesto the minimum, 0.0, as it gets close to the image boundary.Figure 6 shows an example of the local smoothing map.

Second, the mutual smoothing map am is defined by us-ing the relative distance to the neighboring ROIs. For thepixel position (xc, yc) in the canvas coordinate, we com-pute the distance to the ROI, d, and the minimum distancesto the neighboring ROIs, dmin. The mutual smoothing fac-tor is then defined as

am(xc, yc) = max(

0.0, 1.0− d− dmin

tm

), (1)

where tm is a constant (20 in the current implementation) tolimit the range of smoothing. Figure 7 shows an example ofthe mutual smoothing map. Note that the mutual smoothingmap has the same size as the canvas size but defined to allthe images on the canvas.

The final alpha map is obtained as the pixelwise multi-plication of the local and mutual smoothing maps, in whichthe local smoothing map is first translated to its image lo-cation in the canvas coordinates. Finally, all the images are

Table 1. Result of algorithm optimization on the mobile phone (inseconds). Number of test images = 12, Resolution = 800× 533.

Canvas Size Module Before Optimization After Optimization

1024×768

Ranking 3.952 1.268

Packing 10.134 0.523

Blending 2.938 2.938

Total 19.028 6.773

640×480

Ranking 3.952 1.268

Packing 2.146 0.358

Blending 1.176 1.176

Total 8.828 4.354

blended using the the alpha maps and the corresponding im-ages on the canvas to produce the final photo collage.

3. Mobile Implementation and OptimizationThe mobile phone used in our experiment is Samsung

Omnia II. The main processor of the phone is SamsungARM 1176 based S3C6410 mobile application processor.The processor operates at 800 MHz and equips 256 MB ofRAM. We focus on Microsoft’s Visual Studio .NET 2005integrated development environment (IDE) for developingthe application.

3.1. Algorithm Optimization

Since the computational complexity of the algorithm ishigh, it needs to be optimized to achieve high performance.Firstly, we measure the runtime of each module. Differentmodules of the collage algorithm have different degree ofcomplexity. The runtime of each module is listed in Ta-ble 1. From Table 1, it is shown that the packing algorithmis the most time consuming part. Image ranking and blend-ing follow the packing algorithm in complexity.

In what follows, we provide the software optimizationsthat we consider to allow efficient implementation of theproposed collage algorithm. We begin with algorithm opti-mization, which is specific to the collage algorithm. This isfollowed by general-purpose optimization techniques. It isworth mentioning that general-purpose optimization tech-niques can also be applied to other tasks that are computa-tionally intensive and desired to be run on mobile phones.Table 1 shows the runtime of each module before and afteroptimization.

3.1.1 ROI Packing Optimization

Scaling The goal of ROI packing is to cover maximumarea on the collage canvas using the selected images andtheir respective ROIs. As described in Section 2.3, the pack-ing algorithm iteratively scales ROIs to find the best loca-tion and size of each ROI. To limit the amount of search,the search space or the canvas width and height are scaled

down with the scale factor of 2. Thereafter, the ROI pack-ing is performed with the scale down resolution. Finally,the position and size of each of the ROI in the final canvasare scale up by 2 prior to image blending.

K-Means Clustering The K-means clustering algorithmis used to find the starting location of each ROI in the col-lage canvas. In the original algorithm, starting local iscomputed dynamically using a random seed. The cluster-ing algorithm is the most time consuming task in the ROIpacking, which amounts to 90% of the total packing time.However, the clustering algorithm is independent of imagedata and thus can be pre-computed offline. In our imple-mentation, we compute initial ROI locations for each con-figuration (number of selected images) and store them inthe lookup tables. The algorithm automatically chooses theright entry in the lookup table depending on the number ofselected images.

3.1.2 Local Importance Map Optimization

Estimating local importance map is the second most com-plex operation in the collage framework. In our implemen-tation, we propose a low-accuracy but high-speed local im-portance map generation. We employ face detection andGaussian mask as an estimate for local entropy. The Gaus-sian mask is given by

D =(x− w

2

)2

+(

y − h

2

)2

, (2)

M =(D << 8)

w ∗ h, (3)

W = 255− ((M ∗ 5) >> 2), (4)E(x, y) = (W & Ox80000000) ? 0 : W, (5)

where (x, y) is the 2D pixel coordinate, E is the local im-portance map, and w, h are the width and height of input im-age, respectively. In this high-speed version, we replace thesaliency map with the Gaussian mask and estimate the ROIusing the method described in Section 2.2. Figure 8 showsthe local importance map, the ROI selection, and comparesit with the ROI obtained using the original saliency map.

3.2. Code Optimization

3.2.1 Fixed Point Implementation

The ARM1176 CPU does not support floating point opera-tions in hardware. It just supports software emulated float-ing point, which is very slow. In our implementation, fixedpoint math is used to calculate floating point operations withinteger arithmetic. For example,

2.4680.512

= 4.8203,

(a) (b)

(c) (d)Figure 8. Local entropy optimization. (a) Original image. (b) Lo-cal importance map using Gaussian mask; (c) ROI selection usingGaussian mask. (d) ROI selected using original saliency map.

is equivalent to2468512

= 4.8203,

where the terms in the fraction are scaled by a factor of1000. The ARM CPU works with the base of two. Ina typical implementation, the input range is first enlargedby a factor, for example 210. This operations can be doneefficiently on ARM CPU using the bit shift operator, �(left shift) in one clock cycle. During intermediate com-putations, the fixed point representation is held until the fi-nal output. The right shift operator is used to scale backto the original integer representation, denoted by � (rightshift). For example, let V0 and V1 be left shifted by tenbits. All the arithmetic operations like addition, subtrac-tion, multiplication, and division can be done in followingways:Addition: V0 + V1

Subtraction: V0 − V1

Multiplication: (V0 ∗ V1)� 10Division: (V0 � 10)/V1

Division by 2n: V0 � n

3.2.2 Lookup Table for Complex Functions

The ARM CPU lacks support for log function used in cal-culating local importance map. In our implementation, apre-computed lookup table containing the log values forall probability combinations is used. Same approach canbe applied to other complex mathematical functions liketrigonometric and exponential functions.

3.3. Memory Optimization

The memory space available is always an issue on anymobile platform. Memory should be used carefully andsmartly in any mobile application because the amount avail-able is insufficient compared with the application’s demand.In our implementation, following optimization steps aremade:

Scaling If the width (or height) of an input image isgreater than 1024, the original image is scaled down to aresolution less than 1024×1024, while keeping the aspectratio same. The scaled down image is stored in a separateresized image buffer, followed by deleting the original im-age buffer. The scaling facilitates the use of input imagewith resolution as high as several mega pixels.

Maximum Number of Images The maximum number ofselected images is limited to a predefined constant, which is15 in the current implementation.

4. Experimental ResultsIn order to evaluate the performance, we implement and

evaluate the proposed algorithm on both PC and mobilesmart phone. Test images are downloaded from Flickr [11]using friends and travel keywords, in which 30 images withdifferent resolution between 1296×972 to 2048×1536 areselected. In entropy calculation and ROI selection step, theinput images are resized to a resolution below 1024×1024.Figure 9 (a) and (b) show the result of ROI packing us-ing first 7 and 13 high-ranked images, respectively. Fig-ure 9 (c) and (d) show the generated photo collages. Inthis example, image loading, entropy calculation and ROIselection take about 40 seconds, and packing and blend-ing procedure take 1∼2 seconds on an Intel Core2 Quad2.66GHz CPU. To compare the results with previous work,we run AutoCollage [6] and Picasa [2] for the same test im-ages, which is shown in Fig. 10 (a) and (b), respectively.The result of AutoCollage is similar to that of the proposedmethod. The noticeable difference is the size of face in thecollage. Note that the proposed algorithm focuses more onthe face in entropy calculation and ROI selection step. Auto-Collage’s color transition on the segment boundary is moresmooth and natural since its packing algorithm is based onthe global optimization considering color similarity and ob-ject segmentation together.

As we described in Section 3, we implement the pro-posed algorithm on an off-the-shelf smart phone (SamsungOmnia II). Figure 11 shows a few samples of created col-lages on the mobile platform. The whole procedure takes9.4 seconds with twelve images (middle example). In thisexperiment, the importance map is calculated using saliencymap.

(a) (b)

(c) (d)

Figure 9. Results of ROI packing and blending with 30 input im-ages. (a)(c) Using 7 high-ranked images. (b)(d) Using 13 high-ranked images.

(a) (b)

Figure 10. Results of the previous work. (a) AutoCollage [6] byMicrosoft. (b) Picasa [2] by Google.

Figure 12 shows the result of local importance map op-timization, in which Fig. 12 (a) and (b) are obtained usingsaliency map and only Gaussian mask as the importancemap. It is observed that, although the ROI selection usingGaussian mask is less accurate than using the saliency map,the collage results show little noticeable different.

5. ConclusionIn this paper, we proposed the optimized technique for

creating a visually appealing collage from a set of input im-ages on the mobile phone. The proposed algorithm consistsof four main modules, namely image ranking, ROI selec-tion, packing, and blending. Each module in the proposedalgorithm was designed using a greedy approach, which islow in complexity but still showed comparable results. Themodules were further optimized during the implementation

Figure 11. Collage examples on the smart phone (Samsung OmniaII).

on the mobile phone. Experimental results have proven theeffectiveness of the proposed algorithm with visually ap-pealing results on the off-the-shelf smart phone.

Acknowledgement

This work was supported by Samsung Electronics Co.Ltd. and supported partly by the National Research Founda-tion of Korea (NRF) grant funded by the Korea government(MEST) (No. 2009-0083945).

(a) (b)

Figure 12. Local entropy optimization. Collage is obtained using(a) Saliency map and (b) Gaussian mask.

References[1] N. Diakopoulos and I. Essa. Mediating photo collage

authoring. In Proc. ACM Symposium on User Inter-face Software and Technology, pages 183–186, Octo-ber 2005.

[2] Google. Picasa. http://picasa.google.com.[3] L. Itti, C. Koch, and E. Niebur. A model of saliency

based visual attention for rapid scene analysis. IEEETransactions on Pattern Analysis and Machine Intelli-gence, 20(11):1254–1259, November 1998.

[4] T. Mei, B. Yang, S. Q. Yang, and X. S. Hua. Videocollage: presenting a video sequence using a singleimage. The Visual Computer, 25(1):39–51, January2009.

[5] I. K. Park, I. D. Yun, and S. U. Lee. Color imageretrieval using hybrid graph representation. Image andVision Computing, 17(7):465–474, May 1999.

[6] C. Rother, L. Bordeaux, Y. Hamadi, and A. Blake.Autocollage. In ACM Transactions on Graphics, vol-ume 25, pages 847–852, July 2006.

[7] C. Rother, S. Kumar, V. Kolmogorov, and A. Blake.Digital tapestry. In Proc. IEEE CVPR, volume 1,pages 589–596, June 2005.

[8] P. Viola and M. Jones. Robust real-time face detection.International Journal of Computer Vision, 57(2):137–154, May 2004.

[9] J. Wang, J. Sun, L. Quan, X. Tang, and H. Y. Shum.Picture collage. In Proc. IEEE CVPR, volume 1, pages347–354, June 2006.

[10] T. Wang, X. S. Hua, X. L. Liu, and H. Q. Zhou. Videocollage: A novel presentation of video sequence. InProc. IEEE ICME, pages 1479–1482, July 2007.

[11] Yahoo. Flickr. http://www.flickr.com.[12] Y. Yang, Y. Wei, C. Liu, Q. Peng, and Y. Mat-

sushita. An improved belief propagation method fordynamic collage. The Visual Computer, 25(5-7):431–439, March 2009.