Design and Simulation of High SpeedMulti-Processing CMOS Image Sensor

Jerome Dubois, Dominique Ginhac, Michel PaindavoineLaboratoire LE2I - UMR CNRS 5158

Universite de Bourgogne21078 Dijon Cedex - FRANCE

E-mail : (jerome.dubois,dginhac,paindav)@u-bourgogne.fr

Abstract— A high speed Analog VLSI Image acquisition andpre-processing system is described in this paper. A 64×64 pixelretina is used to extract the magnitude and direction of spatialgradients from images. So, the sensor implements some low-level image processing in a massively parallel strategy in eachpixel of the sensor. Spatial gradients, various convolutions asSobel filter or Laplacian are described and implemented on thecircuit. The retina implements in a massively parallel way, atpixel level, some various treatments based on a four-quadrantsmultipliers architecture. Each pixel includes a photodiode, anamplifier, two storage capacitors and an analog arithmetic unit.A maximal output frame rate of about 10 000 frames persecond with only image acquisition and 2000 to 5000 frames persecond with image processing is achieved in a 0.35 µm standardCMOS process. The retina provides address-event coded outputon three asynchronous buses, one output is dedicated to thegradient and both other to the pixel values. A prototype basedon this principle, has been designed. Simulation results fromMentor GraphicsTMsoftware and AustriaMicrosystem Design kitare presented.

keywords : CMOS Image Sensor, Parallel architecture, High-speed image processing, Analog arithmetic unit.

I. INTRODUCTION

Today, improvements continue to be made in the growing di-gital imaging world with two main image sensor technologies :the charge coupled devices (CCD) and CMOS sensors. Thecontinuous advances in CMOS technology for processors andDRAMs have made CMOS sensor arrays a viable alternativeto the popular CCD sensors. New technologies provide thepotential for integrating a significant amount of VLSI elec-tronics onto a single chip, greatly reducing the cost, powerconsumption and size of the camera [1]–[4]. This advantageis especially important for implementing full image systemsrequiring significant processing such as digital cameras andcomputational sensors [5]. Most of the work on complexCMOS systems talks about the integration of sensors providinga processing unit at chip level or column level [6][7]. Indeed,pixel level processing is generally dismissed because pixelsizes are often too large to be of practical use. However,integrating a processing element at each pixel or group ofneighbouring pixels seems to be promising. More significantly,employing a processing element per pixel offers the opportu-nity to achieve massively parallel computations. This benefitstraditional high speed image capture [8][9][10] and enablesthe implementation of several complex applications at standard

video rates [11][12].Vision chips are designed based on the concept that analog

VLSI systems with low precision are sufficient for imple-menting many low level vision algorithms. The precision inanalog VLSI systems is affected by many factors, which arenot usually controllable. As a result, if the algorithm doesnot account for these inaccuracies, the processing reliabilitymay be severely affected. Vision chips implement a specificalgorithm in a limited silicon area. They are always fullcustom designed which is a challenging task, known to betime consuming and error prone. One should consider issuesfrom visual processing algorithms to low level circuit designproblems, from photo transduction principles to high-levelVLSI architectural issues[13].

This paper describes the design and the implementation ofa 64 × 64 active pixel sensor with per-pixel programmableprocessing element. Both the circuit design and layout aretargeted for manufacturing in a standard 0.35 µm double-polyquadruple-metal CMOS technology. The main objectives ofour design are : (1) to evaluate the speed of the sensor, and, inparticular, to reach a 10 000 frames/s rate, (2) to demonstrate aversatile and programmable processing unit at pixel level, (3)to provide a original platform dedicated to embedded imageprocessing.

The rest of this paper is organized as follows : the section IIdescribes the main characteristics of the overall architecture.The section III is dedicated to the description of the operationalprinciple at pixel level in the sensor. In the following, thesections IV and V respectively describe the details of the pixeldesign and the Analog Arithmetic Unit embedded in eachpixel. Finally, some simulation results of high speed imageacquisition with processing at pixel level are presented in thelast section of this paper.

II. OVERALL ARCHITECTURE

The core circuit is, obviously, the bidimensional array ofpixels. This array is organized into a cartesian arrangementof 64×64 pixels and contains 160 000 transistors on a 3.5mm × 3.5 mm die. The full layout of the retina is shown inFigure 1 and the main chip characteristics are listed in Table I.Each pixel contains a photodiode for the light-to-voltage trans-duction and 38 transistors integrating all the analog circuitrynecessary to implement the algorithm described in section III.

This amount of electronics includes a preloading circuit, two“Analog Memory, Amplifier and Multiplexer” ([AM]2) andan “Analog Arithmetic Unit” (A2U) based on a four-quadrantmultiplier architecture. The full pixel size is 35 µm × 35 µmwith a 25 % fill factor.

The left part of the sensor is dedicated to a row decoderfor adressing the successive rows of pixels. Below the chipcore are the readout circuits with the three asynchronousbuses described in section IV. The chip also contains teststructures used for detailed characterization of the photodiodesand processing units. The test structures can be seen on thebottom left of the chip.

Fig. 1. Layout of the Retina in a standard 0.35 µm CMOS technology

TABLE ICHIP CHARACTERISTICS

Technology 0.35µm 4-metal CMOS

Array size 64 × 64

Chip size 11 mm2

Number of transistors 160 000

Number of transistors / pixel 38

Pixel size 35 µm × 35 µm

Sensor Fill Factor 25 %

Dynamics power consumption 250 mW

Supply voltage 3.3 V

Frame rate 10 000 fps

III. OPERATIONAL PRINCIPLE

A. Photodiode Structure

In a standard design, a pixel includes a photodiode (calledPD in our chip) and a processing unit (implemented in the zonecalled free surface) as shown in Figure 2(a). With the proposedapproach, we focus on the optimization of the processingunit mapping. Each pixel integrates image processing based

on neighborhood. So, each processing unit must easily haveaccess to the nearest neighbors that’s why an original structurewas chosen as shown in Figure 2(b). The major advantage ofthis structure is the possibility to limit the length of metalinterconnection between adjacent pixels and the processingunits, contributing to a better fill factor.

(a) (b)

Fig. 2. (a) Photosite classical structure, (b) Considered approach in oursystem

N-type photodiodes consist of a n+-type diffusion in ap-type silicon substrate. During the preloaded period, thedepletion region is formed in the neighborhood of photodiodecathode. However, considering the layout constraints, choosinga cross shape for the photodiode appears to be not realistic.Finally, a quasi-octagonal structure, shown in Figure 3, wasselected because of three main properties :

1) The reserved surface dedicated to the interconnectionsis about 12% lower compared to a square shape,

2) The depletion region is more efficient at the edges ofthe photodiode

3) This shape, based on 45 ˚ structures, is technologicallyrealizable by the founder.

(a) (b)

Fig. 3. (a) Photodiode shape, (b) Photodiode layout

B. Spatial Gradients

The structure of our processing unit is perfectly adapted tothe computation of spatial gradients.The main idea for evaluating these gradients [14] in-situ isbased on the definition of the first-order derivative of a 2-Dfunction performed in the vector direction

→ξ , which can be

expressed as :

∂V (x, y)

∂→ξ

=∂V (x, y)

∂x′cos(β) +

∂V (x, y)∂y′

cos(β) (1)

where β is the vector’s angle. A discretization of the equation1 at the pixel level, according to the Figure 5, would be givenby :

∂V5

∂→ξ

= (V1 − V3) cos(β) + (V2 − V4) sin(β) (2)

where Vi, i ∈ {1; 4} is the luminance at the pixel i, i.e.,the photodiode output. In this way, the local derivative in thedirection of vector

→ξ is continuously computed as a linear

combination of two basis functions, the derivatives in the x’and y’ directions.

Fig. 4. Implementation of four multipliers

Using a four-quadrant multiplier, shown in Figure 4, theproduct of the derivatives by a circular function in cosinescan be easily computed :

P = V1 cos(β) + V2 sin(β)− V3 sin(β)− V4 cos(β) (3)

According to the Figure 5, the processing element imple-mented at the pixel level carries out a linear combination ofthe four adjacent pixels by the four associated weights (coefi,i ∈ {1; 4}). To evaluate the equation 3, the following valueshave to be given to the coefficients :

(coef1 coef2coef3 coef4

)=

(sin(β) cos(β)−sin(β) −cos(β)

)(4)

C. Sobel operator

The structure of our architecture is also adapted to variousalgorithms based on convolutions using binary masks on aneighborhood of pixels. As example, the Sobel operator isdescribed here. With our chip, the evaluation of the Sobel al-gorithm leads to the result directly centered on the photosensorand directed along the natural axes of the image according tothe Figure 5. In order to compute the mathematical operation,

Fig. 5. Array architecture

a 3x3 neighborhood is applied on the whole image. To carryout the derived operation discretized in two dimensions, alongthe horizontal and vertical axes, it is necessary to build two3x3 matrices called h1 and h2 :

h1 =

−1 0 1−2 0 2−1 0 1

h2 =

−1 −2 −10 0 01 2 1

(5)

Within the four processing elements numbered from 1 to 4(see Figure 5), two 2x2 masks are locally applied. Accordingto the equation 5, this allows the evaluation of the followingseries of operations :

h1 :I11 = I3 + 2I6 + I9

I13 = I1 + 2I4 + I7h2 :

I22 = I1 + 2I2 + I3

I24 = I7 + 2I8 + I9(6)

with the I1k and I2k provided by the processing element (k).Then, from these trivial operations, the discrete amplitudesof the derivatives along the vertical axis (Ih1=I11-I13) andthe horizontal axis (Ih2=-I22+I24) can be computed. All theseoperations can be carried out in two cycle’s retina. We callcycle’s retina, all operations of treatments carried out at theend of the screen of acquisition.

D. Second-order detector : Laplacian

Edge detection based on some second-order derivatives asthe Laplacian can also be implemented on our architecture.Unlike spatial gradients previously described, the Laplacian isa scalar (see equation 7) and does not provide any indicationabout the edge direction.

δ =

0 1 01 −4 10 1 0

(7)

From this 3x3 mask, the following operation can be ex-tracted according to the principles used previously for theevaluation of the Sobel operator ; δ=I1+I2+I3+I4-4.I5. Thisoperation can be carried out in only one cycle’s retina.

IV. ARCHITECTURE AND OPERATION

The proposed chip is based on a classical architecturewidely used in the literature as shown on the Figure 6. TheCMOS image sensor consists of an array of pixels that are

typically selected a row at a time by a row decoder. The pixelsare read out to vertical column buses that connect the selectedrow of pixels to an output multiplexer. The chip includesthree asynchronous output buses, the first one is dedicated tothe image processing results whereas the two others providesparallel outputs for full high rate acquisition of raw images.

Fig. 6. Image sensor system architecture

As a classical APS, all reset transistor gates are connectedin parallel, so that the whole array is reset when is activethe reset line. In order to supervise the integration period(i.e., the time when incident light on each pixel generatesa photocurrent that is integrated and stored as a voltage inthe photodiode), the global output called Out int provides theaverage incidental illumination of the whole matrix of pixels.So, if the average level of the image is too low, the exposuretime may be increased. On the contrary, if the scene is tooluminous, the integration period may be reduced.

Fig. 7. Block diagram of reading structure

To increase the algorithmic possibilities of the architecture,the acquisition of the light inside the photodiode and thereadout of the stored value at pixel level are dissociatedand can be independently executed. So, two specific circuits,including an analog memory, an amplifier and a multiplexerare implemented at pixel level as shown in Figure 7.

Fig. 8. Parallelism between capture sequence and readout sequence

With these circuits called [AM]2 (Analog Memory, Am-plifier and Multiplexer), the capture sequence can be madein the first memory in parallel with a readout sequence and/orprocessing sequence of the previous image stored in the secondmemory (see Figure 8). With this strategy, the frame rate canbe increased without reducing the exposure time. Indeed, it ispossible to double the speed of acquisition. Simulations of thechip show that frame rates up to 10 000 frames per second canbe achieved with a brightness superior to 15 000 luxes.

Fig. 9. Frame capture circuit schematic with one analog memory

The chip operates at a single 3.3 V power supply. Ineach pixel, as seen on Figure 9, the photosensor is a NMOSphotodiode associated with a PMOS transistor reset, whichrepresents the first stage of the capture circuit. The pixel arrayis held in a reset state until the “reset” signal goes high. Then,the photodiode discharges according to incidental luminousflow. This signal is polarized around of VDD/2, so the halfpower supply voltage. While the “read” signal remains high,the analog switch is open and the capacitor CAM of the analogmemory stores the pixel value. The CAM capacitors are ableto store, during the frame capture, the pixel values, eitherfrom the switch 1 or the switch 2. The following inverter,polarized on VDD/2, serves as an amplifier of the stored valueand provides a level of tension proportional to the incidentalillumination to the photosite.

Fig. 10. Architecture of the A2U

V. ANALOG ARITHMETIC UNIT : A2U

The analog arithmetic unit (A2U) represents the centralpart of the pixel and includes four multipliers (M1, M2, M3and M4), as illustrated on the Figures 10 and 11. The fourmultipliers are all interconnected with a diode-connected load(i.e., a NMOS transistor with gate connected to drain). Theoperation result at the “node” point is a linear combination ofthe four adjacent pixels.

(a) (b)

Fig. 11. (a) Multiplier schematic, (b) Transistor sizes in µm

Figures 12 and 13 show the simulation results of thismultiplier structure with cosine signals as inputs, i.e.

coefi = A.cos(2πf1) with f1 = 2.5kHz (8)Vi = B.cos(2πf2) with f2 = 20kHz (9)

In this case, the output Node value can be written asfollowing :

Node =A.B

2[cos(2π(f2 − f1)) + cos(2π(f2 + f1))] (10)

The signal’s spectrum, represented on Figure 13 containstwo frequencies (17.5 kHz and 22.5 kHz) around the carrierfrequency. The residues which appear in the spectrum areknown as “inter modulation products”. They are mainly dueto the nonlinearity of the structure (around 10 kHz and 30kHz) and the defects in input pads insulation (at 40 kHz).However, the amplitude of these inter modulations productsare significantly lower than the two main frequencies.

Fig. 12. Multiplier of two cosine signals

Furthermore, in order to obtain the best linearity of themultiplier, the amplitude of the signal Vi is limited to a rangeof 0.6-2.6 V. In the full chip, the signal Vi corresponds to thevoltage coming from the pixel and can be easily included inthe range described before.

Fig. 13. Output Node spectrum in dB

VI. LAYOUT AND PRELIMINARY RESULTS

The layout of a 2x2 pixel block is shown in Figure 14.This layout is symmetrically built in order to reduce fixedpattern noise among the four pixels and to ensure uniformspatial sampling. An experimental 64×64 pixel image sensorhas been developed in a 0.35µm, 3.3 V, standard CMOStechnology with poly-poly capacitors. This prototype hasbeen sent to foundry at the beginning of 2006 and will beavailable at the end of the second quarter of the year. TheFigure 15 represents a simulation of the capture operation.Various levels of illumination are simulated by activatingdifferent readout signals (read 1 and read 2). The two outputs(output 1 and output 2) give the levels between GND andVDD, corresponding to incidental illumination on the pixels.The calibration of the structure is ensured by the biasing(Vbias=1,35V). Moreover, in this simulation, the “node” out-put is the result of the difference operation (out1-out2). Thefactors were fixed at the following values : coef1=-coef2=VDD

Fig. 14. Layout of a pixel

and coef3= coef4=VDD/2. MOS transistors operate in sub-threshold region. There is no energy spent for transferringinformation from one level of processing to another level.According to the simulation’s results, the voltage gain of theamplifier stage of the two [AM]2 is Av=10 and the disparitieson the output levels are about 4.3 %.

Fig. 15. High speed sequence capture with basic processing

VII. CONCLUSION

An experimental pixel sensor implemented in a stan-dard digital CMOS 0.35µm process was described. Each35µm×35µm pixel contains 38 transistors implementing acircuit with photo-current’s integration, two [AM]2 (AnalogMemory, Amplifier and Multiplexer), and a A2U (AnalogArithmetic Unit).

Chip simulations reveal that raw images acquisition at10 000 frames per second can be easily achieved using theparallel A2U implemented at pixel level. With basic imageprocessing, the maximal frame rate slows to reach about5000 fps.

The next step in our research will be the characterizationof the real circuit as soon as the chip comes back from thefoundry. Furthermore, we focus on the development of a fastanalog to digital converter (ADC). The integration of this ADCon future chips will allow us to provide new and sophisticatedvision systems on chip dedicated to digital embedded imageprocessing at thousands of frames per second.

REFERENCES

[1] E. R. Fossum, “Active pixel sensors : Are CCDs dinosaurs ?” Inter-national Society for Optical Engineering (SPIE), vol. 1900, pp. 2–14,1993.

[2] ——, “CMOS image sensor : Electronic camera on a CHIP,” IEEETransactions on Electron Devices, vol. 44, no. 10, pp. 1689–1698,October 1997.

[3] D. Litwiller, “Ccd vs. cmos : Facts and fiction,” Photonics Spectra, pp.154–158, January 2001.

[4] P. Seitz, “Solid-state image sensing,” Handbook of computer Vision andApplications, vol. 1, pp. 165–222, 2000.

[5] A. El Gamal, D. Yang, and B. Fowler, “Pixel level processing – Why,What and How ?” in Proceedings of the SPIE Electronic Imaging ’99conference, vol. 3650, January 1999, pp. 2–13.

[6] M. Loinaz, K. Singh, A. Blanksby, D. Inglis, K. Azadet, and B. Ackland,“A 200mv 3.3v cmos color camera ic producing 352x288 24b video at30 frames/s,” IEEE Journal of Solid-State Circuits, vol. 33, pp. 2092–2103, 1998.

[7] S. Smith, J. Hurwitz, M. Torrie, D. Baxter, A. Holmes, M. Panaghiston,R. Henderson, A. Murrayn, S. Anderson, and P. Denyer, “A single-chip306x244-pixel cmos ntsc video camera,” in In ISSCC Digest of technicalpapers, San Fransisco, CA, 1998, pp. 170–171.

[8] B. Coaker, N. Xu, R. Latham, and F. Jones, “High-speed imagingof pulsed-field flashover of an alumina ceramic in vacuum,” IEEETransactions on Dielectrics and Electrical Insulation, vol. 2, no. 2, pp.210–217, April 1995.

[9] B. Turko and M. Fardo, “High-speed imaging with a tapped solid statsensor,” IEEE Transactions on Nuclear Science, vol. 37, pp. 320–325,1990.

[10] S. Kleinfelder, S. Lim, X. Liu, and A. E. Gamal, “A 10 000 frames/sCMOS digital pixel sensor,” IEEE Journal of Solid-State Circuits,vol. 36, no. 12, pp. 2049–2059, December 2001.

[11] D. Yang, A. El Gamal, B. Fowler, and H. Tian, “A 640 x 512 CMOSimage sensor with ultra wide dynamix range floating-point pixel-levelADC,” IEEE Journal of Solid-State Circuits, vol. 34, pp. 1821–1834,December 1999.

[12] S. Lim and A. El Gamal, “Integrating image capture and processing– beyond single chip digital camera,” in Proceedings of the SPIEElectronic Imaging ’2001 conference, vol. 4306, San Jose, CA, january2001.

[13] A. Moini, Vision Chips. Norwell, MA, USA : Kluwer AcademicPublishers, 1999.

[14] M. Barbaro, P. Burgi, A. Mortara, P. Nussbaum, and F. Heitge, “A100x100 pixel silicon retina for gradient extraction with steering filtercapabilities and temporal output coding,” IEEE Journal of Solid-StateCircuits, vol. 37, no. 2, February 2002.