lecture 29: image sensors...cmos aps (active pixel) sensor. cs184/284a ren ng anatomy of the active...
TRANSCRIPT
![Page 1: Lecture 29: Image Sensors...CMOS APS (Active Pixel) Sensor. CS184/284A Ren Ng Anatomy of the Active Pixel Sensor Photodiode ... Microlenses on a CMOS sensor Microlenses on CCD pixel](https://reader036.vdocuments.net/reader036/viewer/2022063015/5fd3a84b323355709074020f/html5/thumbnails/1.jpg)
Computer Graphics and Imaging UC Berkeley CS184/284A
Lecture 29:
Image Sensors
![Page 2: Lecture 29: Image Sensors...CMOS APS (Active Pixel) Sensor. CS184/284A Ren Ng Anatomy of the Active Pixel Sensor Photodiode ... Microlenses on a CMOS sensor Microlenses on CCD pixel](https://reader036.vdocuments.net/reader036/viewer/2022063015/5fd3a84b323355709074020f/html5/thumbnails/2.jpg)
Photon Capture
![Page 3: Lecture 29: Image Sensors...CMOS APS (Active Pixel) Sensor. CS184/284A Ren Ng Anatomy of the Active Pixel Sensor Photodiode ... Microlenses on a CMOS sensor Microlenses on CCD pixel](https://reader036.vdocuments.net/reader036/viewer/2022063015/5fd3a84b323355709074020f/html5/thumbnails/3.jpg)
Ren NgCS184/284A
The Photoelectric Effect
Einstein’s Nobel Prize in 1921 “for his services to Theoretical Physics, and especially for his discovery of the law of the photoelectric effect"
Albert Einstein(wikipedia)
Incident photons
Ejected electrons
![Page 4: Lecture 29: Image Sensors...CMOS APS (Active Pixel) Sensor. CS184/284A Ren Ng Anatomy of the Active Pixel Sensor Photodiode ... Microlenses on a CMOS sensor Microlenses on CCD pixel](https://reader036.vdocuments.net/reader036/viewer/2022063015/5fd3a84b323355709074020f/html5/thumbnails/4.jpg)
Ren NgCS184/284A
Charge Coupled Devices (CCD)
Developed by Wilford Boyle (L) and George Smith (R) at Bells Labs in 1969Nobel Prize 2009 - "for the invention of an imaging semiconductor circuit – the CCD sensor"
![Page 5: Lecture 29: Image Sensors...CMOS APS (Active Pixel) Sensor. CS184/284A Ren Ng Anatomy of the Active Pixel Sensor Photodiode ... Microlenses on a CMOS sensor Microlenses on CCD pixel](https://reader036.vdocuments.net/reader036/viewer/2022063015/5fd3a84b323355709074020f/html5/thumbnails/5.jpg)
Ren NgCS184/284A
Charge Coupled Devices (CCD)
![Page 6: Lecture 29: Image Sensors...CMOS APS (Active Pixel) Sensor. CS184/284A Ren Ng Anatomy of the Active Pixel Sensor Photodiode ... Microlenses on a CMOS sensor Microlenses on CCD pixel](https://reader036.vdocuments.net/reader036/viewer/2022063015/5fd3a84b323355709074020f/html5/thumbnails/6.jpg)
CCD[N
ikon MicroscopyU]
Interline CCD
![Page 7: Lecture 29: Image Sensors...CMOS APS (Active Pixel) Sensor. CS184/284A Ren Ng Anatomy of the Active Pixel Sensor Photodiode ... Microlenses on a CMOS sensor Microlenses on CCD pixel](https://reader036.vdocuments.net/reader036/viewer/2022063015/5fd3a84b323355709074020f/html5/thumbnails/7.jpg)
Ren NgCS184/284A
Frame Transfer CCD: Global Shutter
![Page 8: Lecture 29: Image Sensors...CMOS APS (Active Pixel) Sensor. CS184/284A Ren Ng Anatomy of the Active Pixel Sensor Photodiode ... Microlenses on a CMOS sensor Microlenses on CCD pixel](https://reader036.vdocuments.net/reader036/viewer/2022063015/5fd3a84b323355709074020f/html5/thumbnails/8.jpg)
Ren NgCS184/284A
CMOS APS (Active Pixel) Sensor
![Page 9: Lecture 29: Image Sensors...CMOS APS (Active Pixel) Sensor. CS184/284A Ren Ng Anatomy of the Active Pixel Sensor Photodiode ... Microlenses on a CMOS sensor Microlenses on CCD pixel](https://reader036.vdocuments.net/reader036/viewer/2022063015/5fd3a84b323355709074020f/html5/thumbnails/9.jpg)
Ren NgCS184/284A
Anatomy of the Active Pixel Sensor Photodiode
http://www.olympusmicro.com/primer/digitalimaging/cmosimagesensors.html
![Page 10: Lecture 29: Image Sensors...CMOS APS (Active Pixel) Sensor. CS184/284A Ren Ng Anatomy of the Active Pixel Sensor Photodiode ... Microlenses on a CMOS sensor Microlenses on CCD pixel](https://reader036.vdocuments.net/reader036/viewer/2022063015/5fd3a84b323355709074020f/html5/thumbnails/10.jpg)
Ren NgCS184/284A
CCD & CMOS Response Functions Are Linear
Photoelectric effect in silicon:
• Response function from photons to electrons is linear
• May have some nonlinearity close to 0 due to noise, and near pixel saturation
![Page 11: Lecture 29: Image Sensors...CMOS APS (Active Pixel) Sensor. CS184/284A Ren Ng Anatomy of the Active Pixel Sensor Photodiode ... Microlenses on a CMOS sensor Microlenses on CCD pixel](https://reader036.vdocuments.net/reader036/viewer/2022063015/5fd3a84b323355709074020f/html5/thumbnails/11.jpg)
Ren NgCS184/284A
Quantum Efficiency
Not all photons will produce an electron
• Depends on quantum efficiency of the device
• Human vision: ~15%
• Typical digital camera: < 50%
• Mobile camera: 60%
• Best back-thinned CCD: > 90%
• Scientific CMOS (sCMOS) 95%
QE =# electrons# photons
Meynants et al. IISW 2013
QE of a 24MP CMOS full-frame sensor
![Page 12: Lecture 29: Image Sensors...CMOS APS (Active Pixel) Sensor. CS184/284A Ren Ng Anatomy of the Active Pixel Sensor Photodiode ... Microlenses on a CMOS sensor Microlenses on CCD pixel](https://reader036.vdocuments.net/reader036/viewer/2022063015/5fd3a84b323355709074020f/html5/thumbnails/12.jpg)
Color Architectures
![Page 13: Lecture 29: Image Sensors...CMOS APS (Active Pixel) Sensor. CS184/284A Ren Ng Anatomy of the Active Pixel Sensor Photodiode ... Microlenses on a CMOS sensor Microlenses on CCD pixel](https://reader036.vdocuments.net/reader036/viewer/2022063015/5fd3a84b323355709074020f/html5/thumbnails/13.jpg)
Ren NgCS184/284A
Color Filter Arrays (Mosaics)
Why more green pixels than red or blue?
• Because humans are most sensitive in the green portion of the visible spectrum
• Sensitivity given by the human luminous efficiency curve
Bayer pattern (most common)
Sony RGB+Ewider color gamut
Kodak RGB+Whigher dynamic range
(Stone)
R G
G B
![Page 14: Lecture 29: Image Sensors...CMOS APS (Active Pixel) Sensor. CS184/284A Ren Ng Anatomy of the Active Pixel Sensor Photodiode ... Microlenses on a CMOS sensor Microlenses on CCD pixel](https://reader036.vdocuments.net/reader036/viewer/2022063015/5fd3a84b323355709074020f/html5/thumbnails/14.jpg)
Ren NgCS184/284A
Demosaicking Algorithms
![Page 15: Lecture 29: Image Sensors...CMOS APS (Active Pixel) Sensor. CS184/284A Ren Ng Anatomy of the Active Pixel Sensor Photodiode ... Microlenses on a CMOS sensor Microlenses on CCD pixel](https://reader036.vdocuments.net/reader036/viewer/2022063015/5fd3a84b323355709074020f/html5/thumbnails/15.jpg)
Ren NgCS184/284A
Demosaicking Algorithms
Interpolate sparse color samples into RGB at every output image pixel Simple algorithm: bilinear interpolation
• Average 4 nearest neighbors of the same color Consumer cameras use more sophisticated techniques
• Try to avoid interpolating across edges Due to demosaicking, 2/3 of image data is “made up”!
![Page 16: Lecture 29: Image Sensors...CMOS APS (Active Pixel) Sensor. CS184/284A Ren Ng Anatomy of the Active Pixel Sensor Photodiode ... Microlenses on a CMOS sensor Microlenses on CCD pixel](https://reader036.vdocuments.net/reader036/viewer/2022063015/5fd3a84b323355709074020f/html5/thumbnails/16.jpg)
Ren NgCS184/284A
3-Sensor Color Architecture
• Prismatic optics
• No demosaicking
• Three (smaller) sensors and optical alignment
![Page 17: Lecture 29: Image Sensors...CMOS APS (Active Pixel) Sensor. CS184/284A Ren Ng Anatomy of the Active Pixel Sensor Photodiode ... Microlenses on a CMOS sensor Microlenses on CCD pixel](https://reader036.vdocuments.net/reader036/viewer/2022063015/5fd3a84b323355709074020f/html5/thumbnails/17.jpg)
Ren NgCS184/284A
Philips Total Internal Reflection Dichroic Prism
Light G-sensor
B-sensor
R-sensor
Dichroic coating and air gap
Dichroic coating
![Page 18: Lecture 29: Image Sensors...CMOS APS (Active Pixel) Sensor. CS184/284A Ren Ng Anatomy of the Active Pixel Sensor Photodiode ... Microlenses on a CMOS sensor Microlenses on CCD pixel](https://reader036.vdocuments.net/reader036/viewer/2022063015/5fd3a84b323355709074020f/html5/thumbnails/18.jpg)
Ren NgCS184/284A
Wavelengths Penetrate to Different Depths
Long-wavelength photons penetrate deeper than short in silicon The spectral response of electrons at the surface differs from electrons deeper in the material
![Page 19: Lecture 29: Image Sensors...CMOS APS (Active Pixel) Sensor. CS184/284A Ren Ng Anatomy of the Active Pixel Sensor Photodiode ... Microlenses on a CMOS sensor Microlenses on CCD pixel](https://reader036.vdocuments.net/reader036/viewer/2022063015/5fd3a84b323355709074020f/html5/thumbnails/19.jpg)
Pixel Structure & Micro Optics
![Page 20: Lecture 29: Image Sensors...CMOS APS (Active Pixel) Sensor. CS184/284A Ren Ng Anatomy of the Active Pixel Sensor Photodiode ... Microlenses on a CMOS sensor Microlenses on CCD pixel](https://reader036.vdocuments.net/reader036/viewer/2022063015/5fd3a84b323355709074020f/html5/thumbnails/20.jpg)
Courtesy R. Motta, Pixim
Front-Side-Illuminated (FSI) CMOSBuilding up the CMOS imager layers
![Page 21: Lecture 29: Image Sensors...CMOS APS (Active Pixel) Sensor. CS184/284A Ren Ng Anatomy of the Active Pixel Sensor Photodiode ... Microlenses on a CMOS sensor Microlenses on CCD pixel](https://reader036.vdocuments.net/reader036/viewer/2022063015/5fd3a84b323355709074020f/html5/thumbnails/21.jpg)
Pixel pitch: A few microns
Photodiodes ~50% Fill Factor
Courtesy R. Motta, Pixim
![Page 22: Lecture 29: Image Sensors...CMOS APS (Active Pixel) Sensor. CS184/284A Ren Ng Anatomy of the Active Pixel Sensor Photodiode ... Microlenses on a CMOS sensor Microlenses on CCD pixel](https://reader036.vdocuments.net/reader036/viewer/2022063015/5fd3a84b323355709074020f/html5/thumbnails/22.jpg)
Polysilicon & Via 1
Courtesy R. Motta, Pixim
![Page 23: Lecture 29: Image Sensors...CMOS APS (Active Pixel) Sensor. CS184/284A Ren Ng Anatomy of the Active Pixel Sensor Photodiode ... Microlenses on a CMOS sensor Microlenses on CCD pixel](https://reader036.vdocuments.net/reader036/viewer/2022063015/5fd3a84b323355709074020f/html5/thumbnails/23.jpg)
Metal 1
Courtesy R. Motta, Pixim
![Page 24: Lecture 29: Image Sensors...CMOS APS (Active Pixel) Sensor. CS184/284A Ren Ng Anatomy of the Active Pixel Sensor Photodiode ... Microlenses on a CMOS sensor Microlenses on CCD pixel](https://reader036.vdocuments.net/reader036/viewer/2022063015/5fd3a84b323355709074020f/html5/thumbnails/24.jpg)
Metal 2
Courtesy R. Motta, Pixim
![Page 25: Lecture 29: Image Sensors...CMOS APS (Active Pixel) Sensor. CS184/284A Ren Ng Anatomy of the Active Pixel Sensor Photodiode ... Microlenses on a CMOS sensor Microlenses on CCD pixel](https://reader036.vdocuments.net/reader036/viewer/2022063015/5fd3a84b323355709074020f/html5/thumbnails/25.jpg)
Metal 3
Courtesy R. Motta, Pixim
![Page 26: Lecture 29: Image Sensors...CMOS APS (Active Pixel) Sensor. CS184/284A Ren Ng Anatomy of the Active Pixel Sensor Photodiode ... Microlenses on a CMOS sensor Microlenses on CCD pixel](https://reader036.vdocuments.net/reader036/viewer/2022063015/5fd3a84b323355709074020f/html5/thumbnails/26.jpg)
Metal 4
Courtesy R. Motta, Pixim
![Page 27: Lecture 29: Image Sensors...CMOS APS (Active Pixel) Sensor. CS184/284A Ren Ng Anatomy of the Active Pixel Sensor Photodiode ... Microlenses on a CMOS sensor Microlenses on CCD pixel](https://reader036.vdocuments.net/reader036/viewer/2022063015/5fd3a84b323355709074020f/html5/thumbnails/27.jpg)
Color filter array
Courtesy R. Motta, Pixim
![Page 28: Lecture 29: Image Sensors...CMOS APS (Active Pixel) Sensor. CS184/284A Ren Ng Anatomy of the Active Pixel Sensor Photodiode ... Microlenses on a CMOS sensor Microlenses on CCD pixel](https://reader036.vdocuments.net/reader036/viewer/2022063015/5fd3a84b323355709074020f/html5/thumbnails/28.jpg)
Ren NgCS184/284A
Pixel Fill Factor
Photodiode area Non photosensitive (circuitry)
Fraction of pixel area that integrates incoming light.
![Page 29: Lecture 29: Image Sensors...CMOS APS (Active Pixel) Sensor. CS184/284A Ren Ng Anatomy of the Active Pixel Sensor Photodiode ... Microlenses on a CMOS sensor Microlenses on CCD pixel](https://reader036.vdocuments.net/reader036/viewer/2022063015/5fd3a84b323355709074020f/html5/thumbnails/29.jpg)
Ren NgCS184/284A
Pixel Fill Factor
Fraction of pixel area that integrates incoming light.
Optimize with per-pixel microlenses.
Microlenses on CCD pixelMicrolenses on a CMOS sensor
![Page 30: Lecture 29: Image Sensors...CMOS APS (Active Pixel) Sensor. CS184/284A Ren Ng Anatomy of the Active Pixel Sensor Photodiode ... Microlenses on a CMOS sensor Microlenses on CCD pixel](https://reader036.vdocuments.net/reader036/viewer/2022063015/5fd3a84b323355709074020f/html5/thumbnails/30.jpg)
Ren NgCS184/284A
Pixel Fill Factor
Leica M9
Shifted microlenses on M9 sensor.
![Page 31: Lecture 29: Image Sensors...CMOS APS (Active Pixel) Sensor. CS184/284A Ren Ng Anatomy of the Active Pixel Sensor Photodiode ... Microlenses on a CMOS sensor Microlenses on CCD pixel](https://reader036.vdocuments.net/reader036/viewer/2022063015/5fd3a84b323355709074020f/html5/thumbnails/31.jpg)
Ren NgCS184/284A
Optical Cross-Talk
http://gmpphoto.blogspot.com/2012/09/the-new-leica-max-24mp-cmos-sensor.html
![Page 32: Lecture 29: Image Sensors...CMOS APS (Active Pixel) Sensor. CS184/284A Ren Ng Anatomy of the Active Pixel Sensor Photodiode ... Microlenses on a CMOS sensor Microlenses on CCD pixel](https://reader036.vdocuments.net/reader036/viewer/2022063015/5fd3a84b323355709074020f/html5/thumbnails/32.jpg)
Ren NgCS184/284A
Pixel Optics for Minimizing Cross-Talk
http://gmpphoto.blogspot.com/2012/09/the-new-leica-max-24mp-cmos-sensor.html
![Page 33: Lecture 29: Image Sensors...CMOS APS (Active Pixel) Sensor. CS184/284A Ren Ng Anatomy of the Active Pixel Sensor Photodiode ... Microlenses on a CMOS sensor Microlenses on CCD pixel](https://reader036.vdocuments.net/reader036/viewer/2022063015/5fd3a84b323355709074020f/html5/thumbnails/33.jpg)
Ren NgCS184/284A
Image Example of Cross-Talk
Kohyama et al. IISW 2009
Color desaturation due to pixel cross-talk
![Page 34: Lecture 29: Image Sensors...CMOS APS (Active Pixel) Sensor. CS184/284A Ren Ng Anatomy of the Active Pixel Sensor Photodiode ... Microlenses on a CMOS sensor Microlenses on CCD pixel](https://reader036.vdocuments.net/reader036/viewer/2022063015/5fd3a84b323355709074020f/html5/thumbnails/34.jpg)
Ren NgCS184/284A
Recall: FSI (Front-Side Illuminated) Pixel Structure
Humrick & Yankulin, tomshardware.com
FSI
![Page 35: Lecture 29: Image Sensors...CMOS APS (Active Pixel) Sensor. CS184/284A Ren Ng Anatomy of the Active Pixel Sensor Photodiode ... Microlenses on a CMOS sensor Microlenses on CCD pixel](https://reader036.vdocuments.net/reader036/viewer/2022063015/5fd3a84b323355709074020f/html5/thumbnails/35.jpg)
Ren NgCS184/284A
BSI (Back-Side Illumination) Sensor Fabrication Process
Humrick & Yankulin, tomshardware.com
![Page 36: Lecture 29: Image Sensors...CMOS APS (Active Pixel) Sensor. CS184/284A Ren Ng Anatomy of the Active Pixel Sensor Photodiode ... Microlenses on a CMOS sensor Microlenses on CCD pixel](https://reader036.vdocuments.net/reader036/viewer/2022063015/5fd3a84b323355709074020f/html5/thumbnails/36.jpg)
Ren NgCS184/284A
FSI vs BSI Pixel Structure
Humrick & Yankulin, tomshardware.com
FSI BSI
![Page 37: Lecture 29: Image Sensors...CMOS APS (Active Pixel) Sensor. CS184/284A Ren Ng Anatomy of the Active Pixel Sensor Photodiode ... Microlenses on a CMOS sensor Microlenses on CCD pixel](https://reader036.vdocuments.net/reader036/viewer/2022063015/5fd3a84b323355709074020f/html5/thumbnails/37.jpg)
Ren NgCS184/284A
Majority of CMOS Sensors are BSI Today
Smartphones Some cameras
Good BSI sensors can provide higher QE and lower cross-talk.
![Page 38: Lecture 29: Image Sensors...CMOS APS (Active Pixel) Sensor. CS184/284A Ren Ng Anatomy of the Active Pixel Sensor Photodiode ... Microlenses on a CMOS sensor Microlenses on CCD pixel](https://reader036.vdocuments.net/reader036/viewer/2022063015/5fd3a84b323355709074020f/html5/thumbnails/38.jpg)
Pixel Aliasing, Antialiasing
![Page 39: Lecture 29: Image Sensors...CMOS APS (Active Pixel) Sensor. CS184/284A Ren Ng Anatomy of the Active Pixel Sensor Photodiode ... Microlenses on a CMOS sensor Microlenses on CCD pixel](https://reader036.vdocuments.net/reader036/viewer/2022063015/5fd3a84b323355709074020f/html5/thumbnails/39.jpg)
Ren NgCS184/284A
Pixel Fill Factor
Photodiode area Non photosensitive (circuitry)
Fraction of pixel area that integrates incoming light.
![Page 40: Lecture 29: Image Sensors...CMOS APS (Active Pixel) Sensor. CS184/284A Ren Ng Anatomy of the Active Pixel Sensor Photodiode ... Microlenses on a CMOS sensor Microlenses on CCD pixel](https://reader036.vdocuments.net/reader036/viewer/2022063015/5fd3a84b323355709074020f/html5/thumbnails/40.jpg)
Ren NgCS184/284A
Pixel Sampling & Aliasinglystit.com
What is going wrong in the image on the right?Simulation of pixels with 25% fill factor
![Page 41: Lecture 29: Image Sensors...CMOS APS (Active Pixel) Sensor. CS184/284A Ren Ng Anatomy of the Active Pixel Sensor Photodiode ... Microlenses on a CMOS sensor Microlenses on CCD pixel](https://reader036.vdocuments.net/reader036/viewer/2022063015/5fd3a84b323355709074020f/html5/thumbnails/41.jpg)
Ren NgCS184/284A
Pixel Sampling & Aliasing
R G
G B
Source of aliasing includes imperfect fill-factor, and color subsampling in color filter array.
Discussed techniques to improve fill-factor (e.g. microlenses)
![Page 42: Lecture 29: Image Sensors...CMOS APS (Active Pixel) Sensor. CS184/284A Ren Ng Anatomy of the Active Pixel Sensor Photodiode ... Microlenses on a CMOS sensor Microlenses on CCD pixel](https://reader036.vdocuments.net/reader036/viewer/2022063015/5fd3a84b323355709074020f/html5/thumbnails/42.jpg)
Ren NgCS184/284A
Antialiasing Filter
Optical low-pass filter
• Use layer of birefringent material, splits each ray into two that overlaps each pixel
• Use two layers oriented at 90 degrees to split each ray over 2x2 pixels
Birefringence
Effect of one birefringent OLPF layer (2D cross-section)
SensorOLPF
![Page 43: Lecture 29: Image Sensors...CMOS APS (Active Pixel) Sensor. CS184/284A Ren Ng Anatomy of the Active Pixel Sensor Photodiode ... Microlenses on a CMOS sensor Microlenses on CCD pixel](https://reader036.vdocuments.net/reader036/viewer/2022063015/5fd3a84b323355709074020f/html5/thumbnails/43.jpg)
Ren NgCS184/284A
With and Without Antialiasing Filter
http://kenrockwell.com/nikon/d800/vs-d800e.htm
![Page 44: Lecture 29: Image Sensors...CMOS APS (Active Pixel) Sensor. CS184/284A Ren Ng Anatomy of the Active Pixel Sensor Photodiode ... Microlenses on a CMOS sensor Microlenses on CCD pixel](https://reader036.vdocuments.net/reader036/viewer/2022063015/5fd3a84b323355709074020f/html5/thumbnails/44.jpg)
To Be Continued
![Page 45: Lecture 29: Image Sensors...CMOS APS (Active Pixel) Sensor. CS184/284A Ren Ng Anatomy of the Active Pixel Sensor Photodiode ... Microlenses on a CMOS sensor Microlenses on CCD pixel](https://reader036.vdocuments.net/reader036/viewer/2022063015/5fd3a84b323355709074020f/html5/thumbnails/45.jpg)
Ren NgCS184/284A
Acknowledgments
Many thanks to Marc Levoy, Brian Wandell, and Pat Hanrahan, who created many of these slides.