reconfigurable processor for energy-scalable computational photography

Reconfigurable Processor for Energy-Reconfigurable Processor for Energy-Scalable Computational Photography Scalable Computational Photography

M. R. Azarbad

Department of Electrical and Computer Engineering, University of Tehran, Tehran, Iran

In the Name of God,In the Name of God,the Compassionate, the Mercifulthe Compassionate, the Merciful

Class presentation for the course Custom Implementation of DSP Systems based on the ISSCC 2013 paper [1]:

All the materials are copy rights of their respective authors as listed in references

R. Rithe, P. Raina, N. Ickes, S. V. Tenneti, and A. P. Chandrakasan

M. R. AzarbadM. R. Azarbad Custom Implementation of DSP Systems Course PresentationCustom Implementation of DSP Systems Course Presentation

OutlineOutline

IntroductionIntroduction

Basic ConceptsBasic Concepts

This WorkThis Work

Chip FeaturesChip Features

Conclusions Conclusions

ReferencesReferences

22


IntroductionIntroduction

33

Computational photography enable capture and synthesis of images Computational photography enable capture and synthesis of images that could not be captured with a traditional camera [1]that could not be captured with a traditional camera [1]

Non-linear filtering techniques like bilateral filtering in computational Non-linear filtering techniques like bilateral filtering in computational photographyphotography [1] [1]

These techniques have a wide range of applications, including High-These techniques have a wide range of applications, including High-Dynamic Range (HDR) imaging, Low-Light Enhanced (LLE) imaging Dynamic Range (HDR) imaging, Low-Light Enhanced (LLE) imaging and video enhancementand video enhancement [1] [1]

The The real-time processingreal-time processing necessitates fast hardware necessitates fast hardware implementationsimplementations [1] [1]



44

Filtering in image Filtering in image processingprocessing

Function of the values of the Function of the values of the input image in a small input image in a small neighborhood [2]neighborhood [2]

Gaussian low-pass filtering Gaussian low-pass filtering computes a weighted computes a weighted average of pixel values in the average of pixel values in the neighborhood (the weights neighborhood (the weights decrease with distance from decrease with distance from the neighborhood center) [2]the neighborhood center) [2] Fig. 1: The mechanism of spatial filtering [3]



55

Images typically vary slowly over space, so near pixels are likely to Images typically vary slowly over space, so near pixels are likely to have similar values [2]have similar values [2]

Appropriate to average them together [2]Appropriate to average them together [2]

The noise values that corrupt these nearby pixels are mutually less The noise values that corrupt these nearby pixels are mutually less correlated than the signal values [2]correlated than the signal values [2]

Noise is averaged away while signal is preserved [2]Noise is averaged away while signal is preserved [2]



66

How can we prevent averaging across edges, while still averaging within smooth regions?How can we prevent averaging across edges, while still averaging within smooth regions?

Iteration may raise issues of stability and, depending on the Iteration may raise issues of stability and, depending on the computational architecture, efficiency [2]computational architecture, efficiency [2]

bilateral filtering [2]bilateral filtering [2]

bilateral filteringbilateral filtering

The idea is to do in the range of an image (vs. traditional filters domain) The idea is to do in the range of an image (vs. traditional filters domain) [2][2]

Range filters are nonlinear (weight dependency on image intensity or Range filters are nonlinear (weight dependency on image intensity or color) [2]color) [2]

Combination of range and domain filtering [2]Combination of range and domain filtering [2]



77

bilateral gridbilateral grid

The bilateral grid is a three dimensional array [3]The bilateral grid is a three dimensional array [3]

The first two dimensions (x, y) correspond to 2D position in the image The first two dimensions (x, y) correspond to 2D position in the image plane and form the spatial domain [3]plane and form the spatial domain [3]

The third dimension z corresponds to a reference range(typically, the The third dimension z corresponds to a reference range(typically, the range axis is image intensity) [3]range axis is image intensity) [3]


This WorkThis Work

88

Implementation of bilateral filtering using a reconfigurable grid [1]Implementation of bilateral filtering using a reconfigurable grid [1]

scheduling the filtering engine so that only two grid rows scheduling the filtering engine so that only two grid rows need to be stored at a time [1]need to be stored at a time [1]

Reduces the storage requirement to 21.5kB (vs. 65MB for a Reduces the storage requirement to 21.5kB (vs. 65MB for a 10Mpixel image) [1]10Mpixel image) [1]

The implementation is flexible to allow varying grid sizes for The implementation is flexible to allow varying grid sizes for energy/resolution scalable image processing [1]energy/resolution scalable image processing [1]


This WorkThis Work

99

Computes the bilateral grid [1]Computes the bilateral grid [1] The input image is partitioned into blocks of size σs×σs [1]The input image is partitioned into blocks of size σs×σs [1]

Grid Assignment EngineGrid Assignment Engine

Fig. 2: The Grid Assignment Engine [1]


This WorkThis Work

1010

Convolves the grid intensities and weights with a 3×3×3 Gaussian Convolves the grid intensities and weights with a 3×3×3 Gaussian kernel, which is equivalent to bilateral filtering in the image domain kernel, which is equivalent to bilateral filtering in the image domain [6], and returns the normalized intensity [1][6], and returns the normalized intensity [1]

Convolution EngineConvolution Engine

Fig. 3: The Concolution Engine [1] Fig. 4: The Interpolation Engine [1]


This WorkThis Work

1111

Fig. 5: Outputs of HDR imaging, LLE imaging and Glare reduction[1]


This WorkThis Work

1212

Fig. 6: Energy vs. performance trade-off for a 10Mpixel image and the frequency of operation for a range of voltages. Run times for different image sizes

at 98MHz, 0.9V [1]


Chip FeatureChip Feature

1313

The grid processing tasks are scheduled to minimize local storage The grid processing tasks are scheduled to minimize local storage requirements and memory traffic [1]requirements and memory traffic [1]

Fig. 7: Die photo of the testchip [1]


ConclusionsConclusions

1414

bilateral filtering using a reconfigurable grid, which reduces the storage bilateral filtering using a reconfigurable grid, which reduces the storage requirement to 21.5kB [1]requirement to 21.5kB [1]

The implementation is flexible to allow varying grid sizes for The implementation is flexible to allow varying grid sizes for energy/resolution scalable image processing [1]energy/resolution scalable image processing [1]

A 10Mpixel image in 771ms with 17.8mW power consumption while A 10Mpixel image in 771ms with 17.8mW power consumption while operating at 98MHz, 0.9V [1]operating at 98MHz, 0.9V [1]


ReferencesReferences

1515

[1] R. Rithe, P. Raina, N. Ickes, S. V. Tenneti, and A. P. Chandrakasan, “Reconfigurable Processor for Energy-Scalable Computational Photography,” ISSCC Dig. Tech. Papers, 2013.

[2] C. Tomasi, R. Manduchi, “Bilateral Filtering for Gray and Color Images,” IEEE International Conf. Computer Vision, pp. 839-846, 1998.

[3] R.C. Gonzalez, and R.E. Woods, “ Digital image processing,” 2nd edition, Prentice-Hall, 2002.

[4] F. Durand, J. Dorsey, “Fast Bilateral Filtering for the Display of High- Dynamic-Range Images,” ACM Trans. on Graphics, vol. 21, no. 3, pp. 257-266, 2002.

[5] G. Petschnigg, M. Agrawala, H. Hoppe, R. Szeliski, M. Cohen, K. Toyama, “Digital Photography with Flash and No-Flash Image Pairs,” ACM Trans. Graphics, vol. 23, no. 3, pp. 664-672, 2004.

[6] J. Oh, G. Kim, J. Park, I. Hong, H.-J. Yoo, “A 320mW 342GOPS Real-Time Moving Object Recognition Processor for HD 720p Video Streams,” ISSCC Dig. Tech. Papers, pp. 220-221, 2012.

[7] J. Chen, S. Paris, F. Durand, “Real Time Edge-Aware Image Processing With the Bilateral Grid,” ACM Trans. Graphics, vol. 26, no. 3, article 103, 2007.

[8] http://en.wikipedia.org

M. R. AzarbadM. R. Azarbad Custom Implementation of DSP Systems Course PresentationCustom Implementation of DSP Systems Course Presentation 1616

Thank Thank You!You!

reconfigurable processor for energy-scalable computational photography

Documents

domain filtering

filtering techniques

filtering engine

image intensity

image processingfunction

image plane

input image

mpixel image