1

L. Yaroslavsky

FROM PHOTOGRAPHY TO *.GRAPHIES:UNCONVENTIONAL IMAGING TECHNIQUES

A short course at Tampere University of Technology,Tampere, Finland, Sept. 3 – Sept. 14, 2001

Lecture 1.EVOLUTION OF IMAGING:

DIRECT IMAGE PLANE IMAGING

2

Lecture 1. EVOLUTION OF IMAGING: DIRECT IMAGE PLANE IMAGING

First imaging system was invented by the Nature

Image discretization was also first invented by the Nature as well

(From: J. S. Lim, Ywo-dimensional Signal and Image Processing, Prentice Hall, Englewood Cliffs, N.J., 1990)

Light

Light

3

Fly eye

(From: Richard Dawkins, Climbing Mount Unprobable, W. W. Nortom Co, New York, 1998 )

3_D vision in the nature: indirect, such as stereoscopic vision.

Cup eyes from around the animalkingdom. (a) - flatworm; (b) –bevalve mollusc; (c) – polychaetworm; (d) - limpet

A range ofinvertebrate eyes:(a) – nautilus pinholeeye; (b) - marinesnail; (c) – bivalvemollusc;(d) – abalone;(e) - ragworm

4

Hexagonal arrangement of cones and rods in retina

H. Hofer, D. R. Williams, The Eye’s Mechanisms for Autocalibration, Optics andPhotonic News, January, 2002, p. 34-39

5

Direct image plane imaging was then adopted by men:

Image was regarded as a point by point projection of object on an image planeFirst imaging devices were:Painter

- A woodcut by Albrecht Dürer showing the relationship between a scene, a center of

projection and the picture planeandCamera-obscura (pinhole camera) (Ibn Al Haytam, X century):

Camera-obscura

6

Magnifying glass (Graeco-Roman times); Eye-glasses or spectacles – (End of 13th

century Salvino degli Armati, 1299):

Optical Microscope (A spectacle-maker Joannes and his son Zacharios, before1590.)

Microscope of Hooke (R. Hooke, Micrographia, 1665)

Object

Image

Image plane

7

Telescope (Zacharias Joannides of Middleburg, 1590; Galileo, 1609 year)Huygens (“Dioptrica, de telescopiis) held the view that only a superhuman geniuscould have invented the telescope on the basis of theoretical considerations, but thefrequent use of spectacles and lenses of various shapes over a period of 300 yearscontributed to its chance invention.

Newton’s telescope-refractor

The scientific impetus produced by the great discoveries made with the telescope canbe gauged from the enthusiastic manner in which Huygens in the “Dioptrica” speaksof these discoveries. He describes how Galileo was able to see the mountains andvalleys of the moon, to observe sun-spots and determine the rotation of the sun, todiscover Jupiter’s satellites and the phases of Venus, to resolve the Milky Way intostars, and to establish the differences in apparent diameter of the planets and fixedstars (after E. Mach, The principles of Physical Optics, Dover Publ., 1926).

8

Photographic camera: a revolutionary step(H. Nieps, 1826; J. Dagherr, 1836; W. F. Talbot, 1844. First public report waspresented by F. Arago, 19.8.1839 at a meeting of L’Institut, Paris)

Imaging optics + photo sensitive materialPhotographic plate/film combines three functions: image recording, imagestorage and image display

X-ray imaging(Wilhelm Conrad Röntgen, Nov. 8,1895; Institute of Physics, University of Würzburg.1-st Nobel Prize, 1901):

X-ray point source+ photographic film or photo-luminescent screen

Wilhelm Conrad Röntgen One of the first medical X-ray images (ahand with small shots in it)

Video camera:imaging optics + photo-electronic converter

A bit of history~1910, Boris Lvovich Rosing, CRT as a display device~1920, Vladimir Kozmich Zvorykin –iconoscope& kinescope, David Sarnov: invited Zvorykin to RCAand gave him $50.000.000~1935 : first regular TV broadcasting, BritainAn important step: image discretization. Modern CCD and CMOS cameras

Radar (~1935), Sonar:beam forming antenna + space scanning mechanism + CRT as a display

9

Gamma-camera:Gamma-ray collimator + Gamma-ray-to light converter + photo sensitive array

+ CRT as a display

Collimator plays role of a lens to separate rays from different object points; this goalis achieved by the expense of energy losses.

10

Electron microscope (1931)Electron optics + luminescent screen or electron sensitive array + CRT display

11

Transmission Electron Microscope: Atoms of gold (Au_clusters) on MoS2.

Scanning electron microscope image: white region is bare LTG GaAs; dark region ispattened region of 2-D array of single crystal AU clusters on LTG:GaAs substrate(from http://www.sst.ic.ac.uk/intro/AFM.htm )

12

Acoustic microscope (1950-th, after R. Bracewell, Two-dimensional Imaging, Prentice Hall,1995):

A monochromatic sound pulse can be focused to a point on the solid surface of anobject by a lens (sapphire rode), and the reflection will return to the lens to begathered by a receiver. The strength of the reflection depends on the acousticalimpedance looking into the solid surface relative to the impedance of the propagatingmedia. If the focal point performs a raster scan over the object, a picture of the surfaceimpedance is formed. Acoustic impedance of a medium depends on its density andelastic rigidity. Acoustic energy that is not reflected at the surface but enters the solidmay be only lightly attenuated and then reflect from surface discontinuities to revealan image of the invisible interior. With such a device, an optical resolution can beachieved. A major application is in the semiconductor industry for inspectingintegrated circuits.

The idea of focusing an acoustic beam was originally suggested by Rayleigh. Theapplication of scanning acoustic microscopes goes back to 1950.

A scanning optical microscope can also be made on the same principle. It has value asa means of imaging an extended field without aberrations associated with a lens.

ElectricOscillator

Receiver

Sapphir(Al2O3) rode

Piezo-electrictransducer

(niobium titanate)

Movable specimen(immersed in a liquid)

13

Scanned-proximity probe (SPP) microscopes.SPP- microscopes work by measuring a local property - such as height, optical

absorption, or magnetism - with a probe or "tip" placed very close to the sample. Thesmall probe-sample separation (on the order of the instrument's resolution) makes itpossible to take measurements over a small area. To acquire an image the microscoperaster-scans the probe over the sample while measuring the local property in question.Scanned-probe systems do not use lenses, so the size of the probe rather thandiffraction effects generally limit their resolution

Tunnel microscope (1980-th; Nobel prize)

Schematic of the physical principle and initial technical realization of ScanningTunnel Microscope. (a) shows apex of the tip (left) and the sample surface (right) at amagnification of about 108. The solid circles indicate atoms, the dotted lines electrondensity contours. The path of the tunnel current is given by the arrow. (b) Scaleddown by factor of 104. The tip (left) appears to touch the surface (right). (c) STM withrectangular piezo drive X,Y,Z of the tunnel tip at left and “loose” L (electrostatic“motor”) for rough positioning (µm to cm range) of the sample S (from G. Binning, H.Rohrer: Physica 127B, 37, 1984)

A conductive sample and a sharp metal tip, which acts as a local probe, arebrought within a distance of a few ångstroms, resulting in a significant overlap of theelectronic wave functions (see figure). With applied bias voltage (typically between1mV and 4V), a tunelling current (typically between 0.1nA and 10 nA) can flow fromthe occupied electronic states near the Fermi level of one electrode into theunoccupied states of the other electrode. By using a piezo-electric drive system of thetip and a feedback loop, a map of the surface topography can be obtained. Theexponential dependence of the tunneling current on the tip-to-sample spacing hasproven to be the key for the high spatial resolution which can be achieved with theSTM. Under favorable conditions, a vertical resolution of hundredths of an ångstromand the lateral resolution of about one ångstrom can be reached. Therefore, STM canprovide real-space images of surfaces of conducting materials down to the atomicscale. (from R. Wiesendanger and H.-J. Güntherodt, Introduction, Scanniing Tunneling Microscopy I,General Principles and Applications to Clean and Absorbate-Covered Surfaces, Springer Verlag,Berlin, 1994)

14

Scanning tunnel microscope image of silicon surface. The image shows two singlelayer steps (the jagged interfaces) separating three terraces. Because of the tetrahedralbonding configuration in the silicon lattice, dimer tow directions are orthogonal onterraces joined by a single layer step. The area pictured is 30x30 nm

15

Atomic force microscope (after http://stm2.nrl.navy.mil/how-afm/how-afm.html).

The atomic force microscope is one of about two dozen types of.

Figure 1. Concept of AFM and the optical lever: (left) a cantilever touching a sample;(right) the optical lever. Scale drawing; the tube scanner measures 24 mm in diameter,while the cantilever is 100 µm long.

AFM operates by measuring attractive or repulsive forces between a tip andthe sample. In its repulsive "contact" mode, the instrument lightly touches a tip at theend of a leaf spring or "cantilever" to the sample. As a raster-scan drags the tip overthe sample, some sort of detection apparatus measures the vertical deflection of thecantilever, which indicates the local sample height. Thus, in contact mode the AFMmeasures hard-sphere repulsion forces between the tip and sample. In noncontactmode, the AFM derives topographic images from measurements of attractive forces;the tip does not touch the sample.

AFMs can achieve a resolution of 10 pm, and unlike electron microscopes, canimage samples in air and under liquids. To achieve this most AFMs today use theoptical lever. The optical lever (Figure 1) operates by reflecting a laser beam off thecantilever. Angular deflection of the cantilever causes a twofold larger angulardeflection of the laser beam. The reflected laser beam strikes a position-sensitivephotodetector consisting of two side-by-side photodiodes. The difference between thetwo photodiode signals indicates the position of the laser spot on the detector and thusthe angular deflection of the cantilever. Image acquisition times is of about oneminute.

16

AFMs have two standard imaging modes

While topographic imaging uses the up-and-down deflection of the cantilever, frictionimaging uses torsional deflection.

2.5 x 2.5 nm simultaneous topographic andfriction image of highly oriented pyrolytic graphic(HOPG). Each bump represents one carbon atom.As the tip moves from right to left, it bumps intoan atom and gets stuck behind it. The scannercontinues to move and lateral force builds up untilthe tip slips past the atom and sticks behind thenext one.

“Steps”

The ability of AFM to image at atomic resolution, combined with its ability to imagea wide variety of samples under a wide variety of conditions, has created a great dealof interest in applying it to the study of biological structures. Images have appeared inthe literature showing DNA, single proteins, structures such as gap junctions, andliving cells.

17

Atomic force microscope, University of Konstanz, May 1991

Samples of ATM images with characteristic stripes due to the mechanical scanning

18

Linear tomography (~1930-th)

Schematic diagram of linear tomography. Due to the synchronous movement of theX-ray source and X-ray sensor, certain plane cross-section of the object is alwaysprojected in the same place of the sensor while others are projected with adisplacement and therefore will appear blurred in the resulting image.

Application in dentistry

Moving stage with a X-ray sensor

O1 O2 O3

Moving X-ray point source

Focal plane

Image 3 Image 2 Image 1

19

Laminography

The principle of laminography (http://lca.kaist.ac.kr/Research/2000/X_lamino.html)

X-ray point source moving in the source plane over a circular trajectory projectsobject onto X-ray detector plane. The detector moves synchronously to the source insuch a way as to secure that a specific object layer is projected on the same place onthe detector array for whatever position of the source. The plane of this selected layeris called “focal plane’. Projections of other object layers located above or beneath ofthe “focal plane” will, for different position of the source, be displaced. Therefore ifone sums up all projections obtained for different positions of the source, projectionsof the focal plane layers will be accumulated coherently producing a sharp image ofthis layer while other layers projected with different displacement in differentprojections will produce a blurred background image. The more projections areavailable the lower will be the contribution of this background into high frequencycomponents of the output image.

Illustration of restoration of different layers of a printed circuit board

20

Optical interferometry.Shape measurement by mean of structured illumination.

Schematic diagram of optical interferometry

The relationship between the interferogram and the object optical thickness:

( ) ( ) ( ) ( )xa~zxzcosCziexpAxziexpAxa rob =

−+∝

+

=

λπ

λπ

λπ 0

20 222

Image reconstruction principle:

( ) ( )( )( )02

zCxa~arccosunwrapxz +−

=π

λ ,

where ( ).unwrap is an unwrapping operator of the phase that can be measured onlyby modulo π2 .

x

z

Semi-transparentmirror

Semi-transparentmirror

Mirror Mirror

Photographicplate

( )xa

21

Interferograms without (left) and with (right) spatial carrier

Moire (Fringe) techniques

Schematic diagram of shape measurement by mean of structured light illumination(1 – fringe image; 2 – image sensor; 3 – illumination source; 4- support; 5 - object)

Object’s profile

22

Direct image plane imaging devices: mathematical models- InpImg(x,y) as a 2-D function is postulated- Imaging system is treated as a linear transformation system that converts InpImg

into OutImg:

( ) ( ) ( ) ( )y,xndd,;y,xh,InpImgy,xOutImg += ∫ ∫ ηξηξηξ

where ( )y,xn is a random image sensor’s noise and function ( )ηξ ,;y,xh is animaging system point spread function (PSF). Fourier Transform of PSF:

( ) =yxyx p,p;f,fH

( ) ( )[ ]∫ ∫ ∫ ∫∞

∞−

∞

∞−

∞

∞−

∞

∞−

−+− ηξηξπηξ ddxdydpyfpxfiexp,;y,xh yyxx2

is called Frequency Transfer Function of the imaging system (Modulation TransferFunction).

An important special case: space invariant imaging systems are modeled byconvolution integral

( ) ( ) ( ) ( ) =+−−= ∫ ∫∞

∞−

∞

∞−

y,xnddy,xh,InpImgy,xOutImg ηξηξηξ

( ) ( ) ( )y,xndd,hy,xInpImg +−−∫ ∫∞

∞−

∞

∞−

ηξηξηξ ;

( ) ( ) ( )[ ]∫ ∫∞

∞−

∞

∞−

+= dxdyyfxfiexpy,xhf,fH yxyx π2

23

Test objects to check PSF of imaging systems: grids of different orientation andperiod (LSF – line spread function); edge of different orientation (ESF – edge spreadfunction)

Optical mira for testing optical system resolving power

Rayleigh’s criterion of resolution of two point sources: two point sources areconsidered resolved if minimum between two peaks of PSF from these sources does

not exceed about 80% of the peak maxima

∆x≈0.2

24

Image sharpness

25

Noise models:

The simplest and the most popular sensor’s noise model is that of signalindependent random zero mean process with normal distribution. It can be fullyspecified by its variance and correlation function. Most frequently it is assumed thatthe noise is delta-correlated (“white” noise model).

Noise in low sensitivity photographic films caused by quantum effects ismultiplicative:

( ) ( )( ) ( )y,xInpImgy,xny,xOutImg += 1 ,

where ( )y,xn is zero mean random process. For a review on noise models in imagerestoration in astronomy see R. Molina, J. Núñez, F. J. Cortijo, J. Mateos, ImageRestoration in Astronomy, IEEE Signal Processing Magazine, March 2001)

For modeling imperfections and faults in electronic image plane imagingdevices, such as CCD and CMOS cameras and discrete and digital image storage andtransmission systems, useful are impulse noise model

( ) ( )( ) ( ) ( ) ( )y,xny,xpy,xInpImgy,xpy,xOutImg +−= 1 ,

where ( )y,xp is a binary signal independent random process (p=0,1) andquantization model:

( ) ( )

−−

=minmax

miny,xInpImg*Qfixy,xOutImg ,

where Q is number of quantization levels and max and min are left and right bordersof the image signal dynamic range.

An important issue that defines image quality is noise visibility. Noise visibilityhighly depends on the type of the noise and is determined by not only noise variancebut all noise statistical parameters.

26

Examples of random interferences in imaging systems

Noise free image Additive noise, stdev=20/256

Impulse noise, Pe=0.06, stdev=20/256 Moire noise, stdev=20/256

Quantization noise, Q=4, stdev=21/256 Multiplicative noise: (1+0.2*randn)

27

Image “restoration”:An important step in the evolution of imaging: imaging systems are complementedwith image processing

( )OutImgRestorInpImg rest =

The possibility of image restoration is completely determined by a priori knowledgeon images and imaging devices. For direct image plane imaging devices, a prioriknowledge on images is directly formulated in tterms of a priori knowledge onimaged objects.

Two large classes of image restoration methods:- Linear filtering (image deblurring and denoising)A classsical example is the Wiener filter:

( ) ( )( )( )yx

yx

yxyxrest f,fSNR

f,fSNRf,fH

f,fH+

=1

1 ( ) ( )( )yx

yxyx f,fsNoiseSpDen

f,fSignSpDensf,fSNR =

- Nonlinear filters that work in image and image transform domains (mostly imagedenoising)

Primary Imagingsystem

Imageprocessing

Sensorsignal OutImg InpImgrest

Effective imaging system

28

Examples of denoising:

Stdev_noiseRGB=20/256 txt512;Pnoise=0.5; filtrimp(*,35,5)

An example of “blind” image deconvolution by means of nonlinear modification ofimage local spectra

29

Test questions1. What direct image plane imaging devices you know?2. What are function of photographic plate (film) in photographic cameras?3. Explain principles of moiré-graphy4. List basic units and draw a general block-diagram of modern image plane imaging

devices5. Describe and explain mathematical model of direct image plane imaging devices6. Describe noise models for image plane imaging devices7. How can one certify imaging devices?

30

Exam questions:

1. Resolution power of direct image plane imaging devices is specified by their1.1 Point Spread Function1.2 Frequency response1.3 Line Spread Function1.4 Edge Spread Function1.5 Either of above1.6 All of above

2. Point spread function of Earth based telescopes is defined by2.1 The size of telescope’s optics2.2 The atmosphere turbulence2.3 The resolution power of the image sensor2.4 All of above

3. The capability of imaging system to resolve objects depends3.1 On its PSF3.2 On its noise level3.3 On both, PSF and noise level

4. Noise in direct image plane imaging is5.1 White Gaussian and signal independent5.2 White Guassian signal dependent5.3 Multiplicative5.4 “Salt &Pepper”5.5 Impulse5.6 Quantization5.7 Modeled by either or combination of above

31

Appendix(В 1931 Б.Л. Розинга репрессировали и сослали в Архангельск. Ему с трудом удалось добитьсяразрешения работать в Архангельском лесотехническом Институте. Профессор по имениПокотило носил ему из дома бидончики с супом, чтобы учёный не голодал. Однажды Б.Л. сэтими бидончиками ехал домой на трамвае (а поселился он на улице русского изобретателярадио А. Попова и его квартирную хозяйку звали Адександрой. Поповой). Трамвай на поворотекачнуло, суп разлился и испачкал одежду стоявшей рядом пассажирки. Она подняла визг:«Ездют тут всякие». Розинг бросился к ней извиняться и очистить платье. А вернувшисьдомой, лёг на кровать, отвернулся лицом к стене и через несколько дней умер)- по материаламкниг Ник. Голядкина и Алекс-ра Рохлина, изданных Интстутом повышения квалификацииработников телеыидения и радиовещания, - из Комс. Правды, 10 авг. 2001

Electron micrograph of two 100 µm longV-shaped cantilevers (by Jean-Paul Revel,Caltech; cantilevers from Park ScientificInstruments, Sunnyvale, CA).

The nanoscale size of an AFM tip (after H.Kempen and G. Walls, IBM)

A close enough inspection of any AFM tip reveals that it is rounded off. Thereforeforce microscopists generally evaluate tips by determining their "end radius." Incombination with tip-sample interaction effects, this end radius generally limits theresolution of AFM.

Three common types of AFM tip. (a) normal tip (3 µm tall); (b) supertip; (c)Ultralever (also 3 µm tall). Electron micrographs by Jean-Paul Revel, Caltech. Tipsfrom Park Scientific Instruments; supertip made by Jean-Paul Revel.

32

AFM: References

• Albrecht, T.R., Akamine, S., Carver, T.E., and Quate, C.F. (1990) Microfabrication ofcantilever styli for the atomic force microscope. J. Vac. Sci. Technol. A 8(4), 3386-3396

• Albrecht, T.R., Grütter, P., Horne, D., and Rugar, D. (1991) Frequency modulation detectionusing high-Q cantilevers for enhanced force microscope sensitivity. J. Appl. Phys. 69(2), 668-673

• Alexander, S., Hellemans, L., Marti, O., Schneir, J., Elings, V., Hansma, P.K., Longmiro, M.,and Gurley, J. (1989) An atomic-resolution atomic-force microscope implemented using anoptical lever. J. Appl. Phys. 65(1), 164-167

• Binnig, G., Quate, C.F., and Gerber, Ch. (1986) Atomic force microscope. Phys. Rev. Lett.56(9), 930-933

• Gallego-Juárez, J.A. (1989) Piezoelectric ceramics and ultrasonic transducers. J. Phys. E: Sci.Instrum. 22, 804-816

• Hoh, J.H. and Hansma, P.K. (1992) Atomic force microscopy for high-resolution imaging incell biology. Trends Cell Bio. 2, 208-213

• Keller, D.J. and Chih-Chung, C. (1992) Imaging steep, high structures by scanning forcemicroscopy with electron beam deposited tips. Surf. Sci. 268, 333-339

• Meyer, G. and Amer, N.M. (1988) Novel optical approach to atomic force microscopy. Appl.Phys. Lett. 53(12), 1045-1047

• Meyer, G. and Amer, N.M. (1990) Simultaneous measurement of lateral and normal forceswith an optical-beam-deflection atomic force microscope. Appl. Phys. Lett. 57(20), 2089-2091

• Putman, C.A.J., De Grooth, B.G., Van Hulst, N.F., and Greve, J. (1992) A detailed analysis ofthe optical beam deflection technique for use in atomic force microscopy. J. App. Phys. 72(1),6-12

• Weisenhorn, A.L., Hansma, P.K., Albrecht, T.R., and Quate, C.F. (1989) Forces in atomicforce microscopy in air and water. Appl. Phys. Lett. 54(26), 2651-2653