Digital In-Line HolographicMicroscope Algorithms forMicro-Organisms Detection andThree Dimension Tracking

Ahmed AYOUB, PhDSzabolcs TOKES, PhD

ICENCO 2007

Computer and Automation Research InstituteHungarian Academy of Science

Benha High Institute of TechnologyBenha University

Click t

o buy NOW!

PDF-XChange

www.docu-track.com Clic

k to buy N

OW!PDF-XChange

www.docu-track.com

Contents

Motivation

Principles of Digital Holographic MicroscopyRecent ResultsThree-D Object TrackingFuture Work

Click t

o buy NOW!

PDF-XChange

www.docu-track.com Clic

k to buy N

OW!PDF-XChange

www.docu-track.com

Motivation

Click t

o buy NOW!

PDF-XChange

www.docu-track.com Clic

k to buy N

OW!PDF-XChange

www.docu-track.com

Holography

Recording Reconstruction

Developmentof photo plate

Click t

o buy NOW!

PDF-XChange

www.docu-track.com Clic

k to buy N

OW!PDF-XChange

www.docu-track.com

Principle

Record the interference pattern(hologram) by an optical sensor, e.g.CCD or CMOSReconstruct using fast digitalalgorithms

Click t

o buy NOW!

PDF-XChange

www.docu-track.com Clic

k to buy N

OW!PDF-XChange

www.docu-track.com

Goal

Create a lensless microscopeRecord 3D interference patterninformationSeveral images of several depths can bereconstructed digitally

Click t

o buy NOW!

PDF-XChange

www.docu-track.com Clic

k to buy N

OW!PDF-XChange

www.docu-track.com

Digital Holographic Microscopy

Click t

o buy NOW!

PDF-XChange

www.docu-track.com Clic

k to buy N

OW!PDF-XChange

www.docu-track.com

The Optical Setup

Laser wavelength: 635 mCCD camera resolution: 10.6 m (1024 1024)Objective: several 3D objectsPinhole size: 1 m

Click t

o buy NOW!

PDF-XChange

www.docu-track.com Clic

k to buy N

OW!PDF-XChange

www.docu-track.com

The Optical Setup

CCD camerap=10.6 mN= 1024

3D Object

Pinhole

Microlens

Laser source(Red) 635nm

Click t

o buy NOW!

PDF-XChange

www.docu-track.com Clic

k to buy N

OW!PDF-XChange

www.docu-track.com

DHM Project

Experimental Setup ReconstructionAlgorithm

Diffractive Optics Recognition

Click t

o buy NOW!

PDF-XChange

www.docu-track.com Clic

k to buy N

OW!PDF-XChange

www.docu-track.com

László Orzó, PhD

Digital Holography Project

Ahmed Ayoub, PhD

Péter Divós

Sándor Tóth

Szabolcs T kés, PhD

Click t

o buy NOW!

PDF-XChange

www.docu-track.com Clic

k to buy N

OW!PDF-XChange

www.docu-track.com

System Calibration

Click t

o buy NOW!

PDF-XChange

www.docu-track.com Clic

k to buy N

OW!PDF-XChange

www.docu-track.com

Fundamental Formulas2

12 21L L LdNp Np

21

1 1r

L L L LLL L

2 2

min1 112 2r

p NpL Np

Resolution calculated from Numerical apprature

Relation between object-distance andreconstruction distance

Nyquist criteria

… laser wavelengthp … CCD camera pixel pitchN … Number of Pixel in one rowa = N*p … CCD row widthL … CCD distance from PinholeL1 … Object distance from PinholeL1max … Maximum allowed object distance (limited

by Nyquist criteria) from PinholeLr … Reconstruction distanceLrmin …Minimum allowed reconstruction distance

(limited by Nyquist criteria) from Pinholed … reciprocal of the ResolutionM … Magnification at the L1max plane

Click t

o buy NOW!

PDF-XChange

www.docu-track.com Clic

k to buy N

OW!PDF-XChange

www.docu-track.com

Derived Formulas

2

1max 2

2LLNp

2

1max

12

L NpML L

Allowed L1max as function of L

Magnification at L1max plane

Click t

o buy NOW!

PDF-XChange

www.docu-track.com Clic

k to buy N

OW!PDF-XChange

www.docu-track.com

Recording Dimensions

L

L1max

L1

a=N

*p

CCD plane

Maximum distance toplace an object!

Pinhole plane

Object plane

a… side length of the CCDN… number of pixelsp… pixel pitch

Click t

o buy NOW!

PDF-XChange

www.docu-track.com Clic

k to buy N

OW!PDF-XChange

www.docu-track.com

Reconstruction Dimensions

Reconstruction zoneRecording zone

L1L1max

LLr

L2

3D Object

CCD planePinhole plane

Reconstructedimage

Click t

o buy NOW!

PDF-XChange

www.docu-track.com Clic

k to buy N

OW!PDF-XChange

www.docu-track.com

Fundamental Formulas

L1min as a function of Lp=10.6um, N=1024, =635nm

0

5

10

15

20

25

30

0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5 4.0L1min [mm]

L [m

m]

Click t

o buy NOW!

PDF-XChange

www.docu-track.com Clic

k to buy N

OW!PDF-XChange

www.docu-track.com

Fundamental Formulas

Resolution as a function of Lp=10.6 m, N=1024, =635nm

0.0

1.0

2.0

3.0

4.0

5.0

6.0

7.0

0 10 20 30 40 50 60

L [mm]

d [u

m]

Click t

o buy NOW!

PDF-XChange

www.docu-track.com Clic

k to buy N

OW!PDF-XChange

www.docu-track.com

Focus and Magnification Maps

The present in-line digital holographic microscope (DHM) shows a linear behaviorfor the focus (small deviation because of human observation error) but anexponential one for the magnification.

Click t

o buy NOW!

PDF-XChange

www.docu-track.com Clic

k to buy N

OW!PDF-XChange

www.docu-track.com

Magnification Map

The present in-line holographic microscope (DHM) experimental magnificationmap behaves similarly to its computed one.

2

1max

12

L NpML L

Click t

o buy NOW!

PDF-XChange

www.docu-track.com Clic

k to buy N

OW!PDF-XChange

www.docu-track.com

Recent Results

Click t

o buy NOW!

PDF-XChange

www.docu-track.com Clic

k to buy N

OW!PDF-XChange

www.docu-track.com

Reconstruction Algorithm

Loading hologram imagematrix (H)

Loading reference beamimage matrix (R)

=H-R

Convolve with G

G… Kernel function of reconstructingplane wave Fourier transform

Focus Reached?

Savereconstructed

image

Click t

o buy NOW!

PDF-XChange

www.docu-track.com Clic

k to buy N

OW!PDF-XChange

www.docu-track.com

Reconstruction Program

Loadhologram

Hardwaresettings

Reconstruction settings

status

Progressimage

Outputsettings

Navigation settings

Click t

o buy NOW!

PDF-XChange

www.docu-track.com Clic

k to buy N

OW!PDF-XChange

www.docu-track.com

Result 1 - Test Pattern

Minimum Resolution(Group 0, Element 1)

Maximum ResolutionFinest line thickness

2.2 m(Group 7, Element 6)

Group Number

Element Number

Original test pattern1951 USAF resolution chart

www.edmundoptics.com

Click t

o buy NOW!

PDF-XChange

www.docu-track.com Clic

k to buy N

OW!PDF-XChange

www.docu-track.com

Digital Focusing for Group 7, Element 6

ReconstructionResolution: 1024x1024

June 13, 2006

Original test pattern1951 USAF resolution chart

Click t

o buy NOW!

PDF-XChange

www.docu-track.com Clic

k to buy N

OW!PDF-XChange

www.docu-track.com

Reconstruction

•Resolution: 1024x1024•Microscope focus: 3.75mm•Time of reconstruction: 2sec/layer

Group 7, Element 6(2.2 m)

Click t

o buy NOW!

PDF-XChange

www.docu-track.com Clic

k to buy N

OW!PDF-XChange

www.docu-track.com

Result 2 - Detecting Cat Cortex NeuronsClick t

o buy NOW!

PDF-XChange

www.docu-track.com Clic

k to buy N

OW!PDF-XChange

www.docu-track.com

Detecting Cat Cortex Neurons

By our present DHMResolution: 1024x1024

June 13, 2006

Cat brain neurons with optical600x Magnification

Neurons

Synapses?

Click t

o buy NOW!

PDF-XChange

www.docu-track.com Clic

k to buy N

OW!PDF-XChange

www.docu-track.com

CNN-Like Processing

Blur Diffuse Edges Contour

Click t

o buy NOW!

PDF-XChange

www.docu-track.com Clic

k to buy N

OW!PDF-XChange

www.docu-track.com

Result 6 – Eurytemora

Detected EurytemoraResolution: 1024x1024

Click t

o buy NOW!

PDF-XChange

www.docu-track.com Clic

k to buy N

OW!PDF-XChange

www.docu-track.com

Video Reconstruction

1. Record a video of holograms’ series (B)2. Decompose the holograms into a series of single ones (C)3. Reconstruct every hologram at several depths (A)4. Combine reconstructed images of the same reconstruction

depth in one output video (A)i. enhance the reconstructed frames of video (C)ii. Label the video – date, resolution, wavelength,

reconstruction depth (C)iii. Object detection step will be applied later to follow an

object moving in depth.

Tools:A. our present DHM

reconstructionalgorithm

B. record.exeC. VideoMach.exe

Decompose intotime frames

Time

Depth

Recorded video ofholograms

(4D information)

Series of Singleholograms

(3D contents)

Each set is a series ofreconstructed 2D images froma single hologram at different

depths. Focusing in depth.

Pick up time series ofholograms

Reconstruct eachframe in depth

Combine frames atcertain depth

Reconstructed video atcertain depth

PAST

Future

Click t

o buy NOW!

PDF-XChange

www.docu-track.com Clic

k to buy N

OW!PDF-XChange

www.docu-track.com

Video ReconstructionClick t

o buy NOW!

PDF-XChange

www.docu-track.com Clic

k to buy N

OW!PDF-XChange

www.docu-track.com

Three-D object Tracking

Click t

o buy NOW!

PDF-XChange

www.docu-track.com Clic

k to buy N

OW!PDF-XChange

www.docu-track.com

Objectives

Short term objective» Track micro organisms moving in depth

Long term objective» Construct 3D model of micro organisms

Click t

o buy NOW!

PDF-XChange

www.docu-track.com Clic

k to buy N

OW!PDF-XChange

www.docu-track.com

Frame#3 Frame#NFrame#2

Objectives

Frame#1

Time, frame

3D Object Tracking

4.5

5

5.5

6

6.5

7

7.5

8

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33

Time, Frame

Dep

th, u

m

In-depthreconstructions Detected

object

Click t

o buy NOW!

PDF-XChange

www.docu-track.com Clic

k to buy N

OW!PDF-XChange

www.docu-track.com

Algorithm

No

Yes

Load frame#1 asprevious frame (PF)

Start a counter, i,with i=2

Load frame# i ascurrent frame (CF)

DF=CF-CP

Extract template TFfrom CF

CGj= TF NFj

Threshold CGj’s

Find NFjmax

Add NFjmax to outputfile

Set PF= NFjmax

Set i= jmax

Lastframe?

Close output file

A

A

Click t

o buy NOW!

PDF-XChange

www.docu-track.com Clic

k to buy N

OW!PDF-XChange

www.docu-track.com

Extract Largest Object Template

Template

Click t

o buy NOW!

PDF-XChange

www.docu-track.com Clic

k to buy N

OW!PDF-XChange

www.docu-track.com

Extract Largest Object TemplateClick t

o buy NOW!

PDF-XChange

www.docu-track.com Clic

k to buy N

OW!PDF-XChange

www.docu-track.com

Correlate Template with Neighborhood

Correlate the extracted template with 9neighboring frames (3 spatial x 3temporal)» Spatial frames are those of the next 3 depth

reconstructions at certain time» Temporal frames are those of the next 3

frames in time at the same depthThe highest correlation peak shows wherethe object had moved toThe detection process repeats

Click t

o buy NOW!

PDF-XChange

www.docu-track.com Clic

k to buy N

OW!PDF-XChange

www.docu-track.com

Daphnia are members of a collection ofanimals that are broadly termed as"water fleas".Get their name from their jerkymovement through the water.Very sensitive to poor water conditions:» halide (e.g. chloride or fluoride)» metal ions (e.g. sodium, potassium,

magnesium and calcium)» dissolved toxins (e.g. copper and zinc)

Increased concentrations can causeimmobility and death.Daphnia is often used to monitor waterquality

Biological sensor

Daphnia

Image of Daphnia using transmission diode(courtesy: The Acadia Centre for Microstructural Analysis)

Click t

o buy NOW!

PDF-XChange

www.docu-track.com Clic

k to buy N

OW!PDF-XChange

www.docu-track.com

Recent Results

Reconstructed and labeled imageof Daphnia using the present

algorithm.

Anatomy of Daphnia(courtesy: Minnesotadepartment of natural

resources)

Click t

o buy NOW!

PDF-XChange

www.docu-track.com Clic

k to buy N

OW!PDF-XChange

www.docu-track.com

Recent ResultsClick t

o buy NOW!

PDF-XChange

www.docu-track.com Clic

k to buy N

OW!PDF-XChange

www.docu-track.com

Challenges and Open Problems

Quality» Higher magnification factor (presently ~500x)» Higher resolution (presently ~2 m)

Speed» Faster digital reconstruction algorithm» Dedicated hardware for real time applications

(FPGA is recommended)Recognition and 3D modeling» Autofocus algorithm» CNN UM operations

Click t

o buy NOW!

PDF-XChange

www.docu-track.com Clic

k to buy N

OW!PDF-XChange

www.docu-track.com

General Applications

Protein Crystal Growth MonitoringCharged Particle Trace DetectionNeurons Activity

Click t

o buy NOW!

PDF-XChange

www.docu-track.com Clic

k to buy N

OW!PDF-XChange

www.docu-track.com

Other Workgroups’ Results

Click t

o buy NOW!

PDF-XChange

www.docu-track.com Clic

k to buy N

OW!PDF-XChange

www.docu-track.com

Free-Drifting Digital Holographic Imaging System

Thin Layers of Harmful Algal Blooms

www.me.jhu.edu/~lefd/shc/HoloSub_site/Digital_Holosub.htm

Laboratory for Experimental Fluid DynamicsMechanical Engineering Department

Johns Hopkins University, USA

Click t

o buy NOW!

PDF-XChange

www.docu-track.com Clic

k to buy N

OW!PDF-XChange

www.docu-track.com

Free-Drifting Digital Holographic Imaging System

www.me.jhu.edu/~lefd/shc/HoloSub_site/Digital_Holosub.htm

Laboratory forExperimental Fluid

DynamicsMechanical Engineering

DepartmentJohns Hopkins University,

USA

Click t

o buy NOW!

PDF-XChange

www.docu-track.com Clic

k to buy N

OW!PDF-XChange

www.docu-track.com

Copepod Nauplius goes after a dinoflagellate but the dinoescapes!

www.me.jhu.edu/~lefd/shc/HoloSub_site/Digital_Holosub.htm

Click t

o buy NOW!

PDF-XChange

www.docu-track.com Clic

k to buy N

OW!PDF-XChange

www.docu-track.com

Dalhousie University

Dalhousie University, CanadaDepartment of Physics and Atmopsheric Science

Figure A a contrast hologram of a test pattern plate.Figure B the reconstructed image of the plateFigure C the image has been zoomed in on the central portion

The smallest 3 bars of the test pattern (near the right of figure C) are 11 microns across.

fizz.phys.dal.ca/~kreuzer/DIH.html

Click t

o buy NOW!

PDF-XChange

www.docu-track.com Clic

k to buy N

OW!PDF-XChange

www.docu-track.com

Woods Hole Oceanographic Institute

REMUS (Remote Environmental Monitoring Units) 100 is a compact, light-weight, autonomous underwater vehicle designed for operation in coastalenvironments up to 100 meters in depth. The REMUS 100 can be configuredto include a wide variety of standard and/or customer specified sensors andsystem options to meet your unique autonomous mission requirements

Click t

o buy NOW!

PDF-XChange

www.docu-track.com Clic

k to buy N

OW!PDF-XChange

www.docu-track.com