here today. gone tomorrow aaron mcclennon-sowchuk, michail greshischev

Here today. Gone TomorrowAaron McClennon-Sowchuk, Michail Greshischev

Objectives

remove an object from a set of images by using information (pixels) from other images in the set.

The images must be of the same scene but can vary in

time of taken and/or perspective of scene. The allowed variance in time means objects may change location from one image to the next.

Applications: stock photography, video surveillance, etc.

Steps

1. Read Images

2. Project images in same perspective

3. Align the images

4. Identify differences

5. Infill objects

Reading Images

How are images represented?– Matrices (M x N x P)

– M is the width of the image – N is the height of the image – P is 1 or 3 depend on quality of image

1: binary (strictly or white) or gray-scale images3: coloured images (3 components of colour: R,G,B)

What tools are capable of processing images?– Many to choose from but MatLab is ideal for matrices.

– Hence the name Mat(rix) Lab(oratory)

Identifying differences

Possible Methods:1. Direct subtraction2. Structural Similarity Index (SSIM) 3. Complex Waveform SSIM

Identifying differences

1. Direct subtraction– Too good to be true! (way too much noise)

Identifying differences

2. Structural Similarity Index (SSIM)– Number 0-1 indicating how “similar” two pixels are.

– 1 indicates perfect match, 0 indicates no similarities at all

– Number calculated based on:

– Luminance, function of the mean intensity for gray-scale image

– Contrast, function of std.dev of intensity for gray-scale image

Identifying differences

Once again, way too much noise.

SSIM map: 0 black pixel 1 white pixel

– Concerns:– Identify regions to copy

• Calculate a bounding box (smallest area surrounding entire blob)

– How to distinguish noise from actual objects?• Area - those blobs with area below threshold are ignored • location - those blobs along an edge of image are ignored.

– Copying method • Direct – images from same perspectives• Manipulated pixels – images from different perspectives.

Infilling the objects

Infilling the objects

Original bounding box results:

Matlab returnsLeft positionTop positionWidth and Height of each box

Infilling the objects

Result with small blobs and blobs along edges ignored:

Left: 119 Top: 52 Width: 122 Height: 264

Infilling the objects

Once regions identified, how can pixels be copied?– Same perspective – direct copy is possible.

Infilling the objects

Result of direct copying

Infilling the objects

Different perspectives– Goal: remove black trophy from left image

Infilling the objects

Direct copying produces horrendous results!

Rectified image Result

Work to come...

Copying techniques – Need better method for infilling objects between images in

different perspectives. Perhaps use same alignment matrix.

Anti-Aliasing – Method to smooth the edges around pixels copied from one

image to another– example looks alright but could improve other test cases

User friendly interface– Current state: a dozen different MatLab scripts. – In the perfect world, we’d have a nice interface to let user load

images and clearly displa

Conclusions

References

Z. Wang and A. C. Bovik, “Image quality assessment: from error visibility to structural similarity,” IEEE Trans. Image Processing, vol. 13, pp. 600 – 612, Apr. 2004. www.ece.uwaterloo.ca/~z70wang/publications/ssim.html