optoelectronic stochastic parallel processors for real time image processing groupe physique des...
TRANSCRIPT
![Page 1: Optoelectronic Stochastic Parallel Processors for real time image processing Groupe Physique des images. Ph. Lalanne, J.C.Rodier et P.Chavel Institut d’Optique](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649e895503460f94b8d8d6/html5/thumbnails/1.jpg)
Optoelectronic Stochastic Parallel Processors for real time image processing
Groupe Physique des images. Ph. Lalanne, J.C.Rodier et P.ChavelInstitut d’Optique Théorique et appliquée, CNRS URA 14
A.Dupret, E.Belhaire and P.GardaInstitut d’Electronique Fondamentale, CNRS URA 22.
Alvaro CASSINELLI
I.Glaser and A.A.Friesem Weizmann Institute, Physics of Complex Systems.
![Page 2: Optoelectronic Stochastic Parallel Processors for real time image processing Groupe Physique des images. Ph. Lalanne, J.C.Rodier et P.Chavel Institut d’Optique](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649e895503460f94b8d8d6/html5/thumbnails/2.jpg)
2Plan
I) Low level image processing
Mathematical Model (energy function)
Combinatorial problem and computational load (stochastic optimization).
II) Parallel Processor Implementation
First prototype on CMOS : SPIE600 (24x25)
Noise cleaning demonstration using SPIE600
Demonstrator enhanced with Optical Interconnections
Motion Detection Demonstrator
III) Modeling of an hybrid Silicon/AsGa Parallel Processor
Monolithic convolution : proposed architecture
Performance evaluation
IV) Further research
![Page 3: Optoelectronic Stochastic Parallel Processors for real time image processing Groupe Physique des images. Ph. Lalanne, J.C.Rodier et P.Chavel Institut d’Optique](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649e895503460f94b8d8d6/html5/thumbnails/3.jpg)
I) Optimization and Low-level image processing
Bayesian approach to image restoration
Optimisation problem
![Page 4: Optoelectronic Stochastic Parallel Processors for real time image processing Groupe Physique des images. Ph. Lalanne, J.C.Rodier et P.Chavel Institut d’Optique](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649e895503460f94b8d8d6/html5/thumbnails/4.jpg)
4
Inverse problem :
Bestimation...
observation...
A?
…there are too many possible solutions
…introducing a priori knowledge enables us to consider only the more probable !
)()/().()/(
BPxBPxPBxP
This is a “probabilistic regularization" technique (Bayesian restoration)
a priori term:- Favor smooth images(MRF field)
Likelihood term : - Favor images which “looks like the observation”
- it is constant on x
(a posteriori probability )
We look for the image A maximizing P(x/B)...
Image restoration : Model
![Page 5: Optoelectronic Stochastic Parallel Processors for real time image processing Groupe Physique des images. Ph. Lalanne, J.C.Rodier et P.Chavel Institut d’Optique](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649e895503460f94b8d8d6/html5/thumbnails/5.jpg)
5
Many low level image processing
tasks …
… can be seen as an optimization problem (absolute minimum of an energy function)
bad
good
Conclusion
Optimisation problemAlvaro Cassinelli:
REM: Il existe bien d'autres formes d'estimateurs que le MAP.
Le modèle bayesien est aussi utile pour grand nombre de problèmes en traitement d'images autres que les problèmes de restauration, car il représente une façon formelle pour introduire des informations a priori dans le problème.
Alvaro Cassinelli:
REM: Il existe bien d'autres formes d'estimateurs que le MAP.
Le modèle bayesien est aussi utile pour grand nombre de problèmes en traitement d'images autres que les problèmes de restauration, car il représente une façon formelle pour introduire des informations a priori dans le problème.
)(),()/( xregUBxobs
UBxU +=
From observation model
),(exp1)/( BxU
ZBxP
rewriting the a posteriori probability as :
we define the « energy » of the image :
From regularisation model
![Page 6: Optoelectronic Stochastic Parallel Processors for real time image processing Groupe Physique des images. Ph. Lalanne, J.C.Rodier et P.Chavel Institut d’Optique](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649e895503460f94b8d8d6/html5/thumbnails/6.jpg)
6Exemple : binary images
… if image is binary :
.:::
....010
....110
....101
x
),( ),(),(),(
1.2.121.2.12)/(ji jiVlk
klijji
ijij xxBxBxU
Interaction neighborhood
),( ),(),(
2
),(
2)/(ji jiVlk
klijji
ijij xxBxBxU
Energy
Image B Image x
a priori : avoid discontinuities
Constraint to the data
Interpretation
Constraint to the data term Regularisation term
![Page 7: Optoelectronic Stochastic Parallel Processors for real time image processing Groupe Physique des images. Ph. Lalanne, J.C.Rodier et P.Chavel Institut d’Optique](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649e895503460f94b8d8d6/html5/thumbnails/7.jpg)
7Practical Resolution : Deterministic updating
image 256x256 Energy function has 65536 variables !Problem :
Exhaustive search ? Impossible ! (2256x256 1020000 images)
First idea : deterministic updating
Random initialisation :
For each pixel (i,j)…
U0=U(x0)=U(1,0,1,1,0,….)
U(1,0,?,1,0,….)
…choose the value that makes the energy decreases :
ijij UF = U(…, xij= 0, ...) - U(…, xij=1, ...)
" Local FORCE "Fij>0
New state 1
New state 0
Fij<0
![Page 8: Optoelectronic Stochastic Parallel Processors for real time image processing Groupe Physique des images. Ph. Lalanne, J.C.Rodier et P.Chavel Institut d’Optique](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649e895503460f94b8d8d6/html5/thumbnails/8.jpg)
8Practical resolution : stochastic optimization
The problem :
Deterministic optimization
local minima xO of the energy !
xO
U<0 U>0
x=A
…image is “trapped” in a
- controlled “mistakes” are allowed :
Simulated Annealing
Stochastic Optimization
Solution :
![Page 9: Optoelectronic Stochastic Parallel Processors for real time image processing Groupe Physique des images. Ph. Lalanne, J.C.Rodier et P.Chavel Institut d’Optique](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649e895503460f94b8d8d6/html5/thumbnails/9.jpg)
9
Repeat by slowly decreasing the « temperature » :
For each pixel (i,j), compute the local force :
… and choose more likely the state which makes the energy decrease, (but not systematically) :
T
Fx
ijij
exp1
1]1Pr[
Stochastic updating of image :
-4 -2 0 2 40
0,5
1
T=
T=0
Force
Pro
bab
iliy
ijij UF = U(…, xij= 0, ...) - U(…, xij=1, ...)
Practical resolution : Simulated Annealing Algorithm
![Page 10: Optoelectronic Stochastic Parallel Processors for real time image processing Groupe Physique des images. Ph. Lalanne, J.C.Rodier et P.Chavel Institut d’Optique](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649e895503460f94b8d8d6/html5/thumbnails/10.jpg)
10Practical resolution : Parallel Updating
For many applications, computation of the force is a local operation :
ji,)lk,(
klijij 1.212V
xBF
Parallel Processor
Interconnection topology
![Page 11: Optoelectronic Stochastic Parallel Processors for real time image processing Groupe Physique des images. Ph. Lalanne, J.C.Rodier et P.Chavel Institut d’Optique](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649e895503460f94b8d8d6/html5/thumbnails/11.jpg)
11Conclusion
Low-level Image processing
Optimization problem
Stochastic
Processor
P P P P S S
Energy Function
Parallel
Updating
Local force computation
Optoelectronics ...
OO
![Page 12: Optoelectronic Stochastic Parallel Processors for real time image processing Groupe Physique des images. Ph. Lalanne, J.C.Rodier et P.Chavel Institut d’Optique](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649e895503460f94b8d8d6/html5/thumbnails/12.jpg)
II) Optoelectronic implementation
• Motion detection demonstrator
• First prototype : CMOS 24x25 SPIE600 processor
• Demonstration : restoration of binary images
• Enhanced demonstrator using optical interconnects
![Page 13: Optoelectronic Stochastic Parallel Processors for real time image processing Groupe Physique des images. Ph. Lalanne, J.C.Rodier et P.Chavel Institut d’Optique](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649e895503460f94b8d8d6/html5/thumbnails/13.jpg)
13SPIE600 parallel processor
CMOS circuit 0,8 m
one bit memory xij
local force computation :
deterministic updating ! (comparator)
optical (2 PD) electrical (currents)
)ji,()lk,(
klijij 1.212V
xBF FC FS
Elementary processor
xij
FS
FC
![Page 14: Optoelectronic Stochastic Parallel Processors for real time image processing Groupe Physique des images. Ph. Lalanne, J.C.Rodier et P.Chavel Institut d’Optique](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649e895503460f94b8d8d6/html5/thumbnails/14.jpg)
Noise cleaning demonstrator
(on binary images)
![Page 15: Optoelectronic Stochastic Parallel Processors for real time image processing Groupe Physique des images. Ph. Lalanne, J.C.Rodier et P.Chavel Institut d’Optique](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649e895503460f94b8d8d6/html5/thumbnails/15.jpg)
15Noise cleaning demontrator : architecture
Laser source
CCD
SLM SPIE600
Corrupted binary image
IN
Restored image
OUT
pixel Bij
FS (-)
+-
FS (+)FC (+)
FC (-)
(dual rail-encoding)
PEij
xij
)ji,()lk,(
kijij 1.212V
lxBF FC FS
…deterministic updating !
![Page 16: Optoelectronic Stochastic Parallel Processors for real time image processing Groupe Physique des images. Ph. Lalanne, J.C.Rodier et P.Chavel Institut d’Optique](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649e895503460f94b8d8d6/html5/thumbnails/16.jpg)
16…stochastic updating
+
=
Speckle generator
Laser diode
multimode fiber
Rotating diffuser
Deterministic comparator (fixed threshold)
xij
FS (+)
FS (-)
FC (+)
FC (-)
“Stochastic” comparator T=0
T2>T1>0
Fij>0Fij<0P
rob
ab
ilité
Laser power T force locale
(seuil aléatoire)
FijT xij
![Page 17: Optoelectronic Stochastic Parallel Processors for real time image processing Groupe Physique des images. Ph. Lalanne, J.C.Rodier et P.Chavel Institut d’Optique](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649e895503460f94b8d8d6/html5/thumbnails/17.jpg)
17Optoelectronic Simulated Annealing
- 50 temperature steps
- several thousand updating cycles
- … everything in 40 ms
Simulated Annealing :
Image CCD (annealing)
Laser source
CCD
SLM SPIE600
from Speckle generator
![Page 18: Optoelectronic Stochastic Parallel Processors for real time image processing Groupe Physique des images. Ph. Lalanne, J.C.Rodier et P.Chavel Institut d’Optique](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649e895503460f94b8d8d6/html5/thumbnails/18.jpg)
18Noise cleaning demonstrator : view of the setup
![Page 19: Optoelectronic Stochastic Parallel Processors for real time image processing Groupe Physique des images. Ph. Lalanne, J.C.Rodier et P.Chavel Institut d’Optique](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649e895503460f94b8d8d6/html5/thumbnails/19.jpg)
19Noise cleaning demonstrator : some experimental results
Slow computer interface «only» video rate (23 images/sec)
Without speckle (deterministic)
With speckle (stochastic)
![Page 20: Optoelectronic Stochastic Parallel Processors for real time image processing Groupe Physique des images. Ph. Lalanne, J.C.Rodier et P.Chavel Institut d’Optique](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649e895503460f94b8d8d6/html5/thumbnails/20.jpg)
Enhanced demonstrator with
Optical interconnections
![Page 21: Optoelectronic Stochastic Parallel Processors for real time image processing Groupe Physique des images. Ph. Lalanne, J.C.Rodier et P.Chavel Institut d’Optique](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649e895503460f94b8d8d6/html5/thumbnails/21.jpg)
21
Electrical interconnections : cumbersome, short range, fixed...
Shift-invariant neighborhood optical convolution !
Optical interconnections : optical convolutionAlvaro Cassinelli:
à partir de ce moment, les interconnexions électriques ne sont plus utilisées
Alvaro Cassinelli:
à partir de ce moment, les interconnexions électriques ne sont plus utilisées
Dammann gratings (2-level phase)
SLM (B) SPIE600
(4 neighboors)
(8 neighboors)
4-f architecture
+
-
+
-
+
-
+
-
PE (i,j-1)
PE (i+1,j)
PE (i-1,j)
+
-
PE (i,j+1)
Optical neighborhood
![Page 22: Optoelectronic Stochastic Parallel Processors for real time image processing Groupe Physique des images. Ph. Lalanne, J.C.Rodier et P.Chavel Institut d’Optique](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649e895503460f94b8d8d6/html5/thumbnails/22.jpg)
22Optical interconnection : test of optical loop
4 nearest neighborhood interconnection topology :
8 nearest neighborhood interconnection topology :
Alvaro Cassinelli:
-aberrations
-il faut détruire la coherence spatiale des sources laser des SLMs
- de toute manière, les irrégularités du voisinage n'affecteront pas outre mesure le résultat du traitement (voisinage)
Alvaro Cassinelli:
-aberrations
-il faut détruire la coherence spatiale des sources laser des SLMs
- de toute manière, les irrégularités du voisinage n'affecteront pas outre mesure le résultat du traitement (voisinage)
![Page 23: Optoelectronic Stochastic Parallel Processors for real time image processing Groupe Physique des images. Ph. Lalanne, J.C.Rodier et P.Chavel Institut d’Optique](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649e895503460f94b8d8d6/html5/thumbnails/23.jpg)
23
source laser
CCD
SLM (A)SPIE600SPIE600
From laser diode fiber A
FC
SLM (A)
Optically interconnected demonstrator : optical “force”
)ji,()lk,(
klijij 1.212V
xBF
SLM A
FC
SLM B + grating
FS
diode B power
diode A power =
Speckle generator
From laser diode fiber B grating
FS
SLM (B)
Electronic feed-back loop
(corrupted image)
input B
Aoutput
(restored image)
![Page 24: Optoelectronic Stochastic Parallel Processors for real time image processing Groupe Physique des images. Ph. Lalanne, J.C.Rodier et P.Chavel Institut d’Optique](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649e895503460f94b8d8d6/html5/thumbnails/24.jpg)
24
SLM-A
SLM-B
CCDSPIE600
Speckle generator input
Dammann grating
Laser diodes
21 cm
35 cm
14 cm
Optically interconnected demonstrator
![Page 25: Optoelectronic Stochastic Parallel Processors for real time image processing Groupe Physique des images. Ph. Lalanne, J.C.Rodier et P.Chavel Institut d’Optique](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649e895503460f94b8d8d6/html5/thumbnails/25.jpg)
Motion detection demonstrator
(with optical interconnections)
![Page 26: Optoelectronic Stochastic Parallel Processors for real time image processing Groupe Physique des images. Ph. Lalanne, J.C.Rodier et P.Chavel Institut d’Optique](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649e895503460f94b8d8d6/html5/thumbnails/26.jpg)
26Motion detection : principles
Goal :Input data : gray level image sequence
Output : image segmented in two motion regions (Binary Mask)
Hypothesis : Still camera
Quasi-constant scene illumination...
Relationship between luminance and motion
I(t)
MB(t)
![Page 27: Optoelectronic Stochastic Parallel Processors for real time image processing Groupe Physique des images. Ph. Lalanne, J.C.Rodier et P.Chavel Institut d’Optique](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649e895503460f94b8d8d6/html5/thumbnails/27.jpg)
27
- Previous field BM(t-1) is known…
- Observed field (input) CD(t) ...MB(t) ?
?
Changes Detection Binary field
I(t)
I(t-1)
BM(t)
-
BM(t-1)
?
Motion detection : algorithm
…without noise :
CD =1 : flip motion label
CD =0 : save motion label
MB(t)= CS(t) XOR MB(t-1)
XOR
…with noise :
!?
B(t)= CD(t) XOR BM(t-1) BM(t)
Restoration of the binary «image» :
XOR
Rest.
![Page 28: Optoelectronic Stochastic Parallel Processors for real time image processing Groupe Physique des images. Ph. Lalanne, J.C.Rodier et P.Chavel Institut d’Optique](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649e895503460f94b8d8d6/html5/thumbnails/28.jpg)
28
Detected motion(Binary Mask)
Motion detection : experimental results
Performances :
Input sequence Change Detection
24 sec/image (optical 8 nearest neigh.)
5 sec/image (optical 4 nearest neigh.)
0,04 sec/image (electronic 4 nearest neigh.)
![Page 29: Optoelectronic Stochastic Parallel Processors for real time image processing Groupe Physique des images. Ph. Lalanne, J.C.Rodier et P.Chavel Institut d’Optique](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649e895503460f94b8d8d6/html5/thumbnails/29.jpg)
III) Modeling of an hybrid Si/AsGa Processor
• Hybrid technology and monolithic convolution
• Performance estimation
![Page 30: Optoelectronic Stochastic Parallel Processors for real time image processing Groupe Physique des images. Ph. Lalanne, J.C.Rodier et P.Chavel Institut d’Optique](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649e895503460f94b8d8d6/html5/thumbnails/30.jpg)
30Hybrid prototype : “monolithic” convolution
…slow optical loop because of electronics !!
«Hybrid» technology
CMOS circuit : computation (threshold and memory)
MQW AsGa array : input/output
bonding pads
Modulation/detection
GaAs
Alvaro Cassinelli:
-la puce en AsGa : détecte et module
-la puce Silicium : calcule
Alvaro Cassinelli:
-la puce en AsGa : détecte et module
-la puce Silicium : calcule
“monolithic” feed-back ?
Source/detector pixel array !!
![Page 31: Optoelectronic Stochastic Parallel Processors for real time image processing Groupe Physique des images. Ph. Lalanne, J.C.Rodier et P.Chavel Institut d’Optique](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649e895503460f94b8d8d6/html5/thumbnails/31.jpg)
31Hybride CMOS/MQW prototype : evaluated architecture
(the image to be processed is entered electronically)
Global architecture
Hybrid circuit
grating
Read laser beam
Array illuminator
Elementary processor
modulators detectors
+-
xij
xij FS(+)
FS(-)
xij
Bij
- pair of modulators
- pair of detectors
- 2 bits : xij et Bij
![Page 32: Optoelectronic Stochastic Parallel Processors for real time image processing Groupe Physique des images. Ph. Lalanne, J.C.Rodier et P.Chavel Institut d’Optique](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649e895503460f94b8d8d6/html5/thumbnails/32.jpg)
32Prototype en technologie hybride : bilan optoélectronique
System performances are mainly limited by the optical power
0 10 20 30 40 50 60 70 80 90100
101
102
103
104
Heat dissipation
Q =5W
cadence
0 10 20 30 40 50 60 70 80 90100
101
102
103
104
Electrical power
P =5W
cadence
83,3 MHz
0 10 20 30 40 50 60 70 80 90100
101
102
103
104
105
106
Updating frequency (MHz)
Ptot= 1W
f4=73.3 MHz
f8=66.8 MHz
f12 =64.1 MHz
Hybrid array 16x32 PEs
Optical Power (mW)
![Page 33: Optoelectronic Stochastic Parallel Processors for real time image processing Groupe Physique des images. Ph. Lalanne, J.C.Rodier et P.Chavel Institut d’Optique](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649e895503460f94b8d8d6/html5/thumbnails/33.jpg)
33Hybrid CMOS/MQW prototype : final performances
In the worst case, 500 times video rate ! (independently on the array size)
Système de voisinage Nombre de recuits par seconde
SPIE600 (voisinage electronique) 23
SPIE600 (voisinage optique) < 0,02
Circuit hybride (voisinage 4) 36700
Circuit hybride (voisinage 8) 16700
Circuit hybride (voisinage 12) 12800
![Page 34: Optoelectronic Stochastic Parallel Processors for real time image processing Groupe Physique des images. Ph. Lalanne, J.C.Rodier et P.Chavel Institut d’Optique](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649e895503460f94b8d8d6/html5/thumbnails/34.jpg)
IV) Conclusion
Results
Possible research directions
![Page 35: Optoelectronic Stochastic Parallel Processors for real time image processing Groupe Physique des images. Ph. Lalanne, J.C.Rodier et P.Chavel Institut d’Optique](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649e895503460f94b8d8d6/html5/thumbnails/35.jpg)
35Results
Performances :
Optically interconnected system
Dedicated Machines for image processing
Optics in computing :
Optoelectronic demonstrator :
- noise cleaning, motion detection- Real time simulated annealing
- Silicon chip : 5 seconds/image- Hybrid circuit : tens of thousand images/sec
Optical random number generation
Compact
![Page 36: Optoelectronic Stochastic Parallel Processors for real time image processing Groupe Physique des images. Ph. Lalanne, J.C.Rodier et P.Chavel Institut d’Optique](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649e895503460f94b8d8d6/html5/thumbnails/36.jpg)
36
Inte
rco
nn
ecit
on
op
tica
l ban
dw
ith
Compared performances
10 101 10 2 103 104 10 5 106 10 7 1080
Number of optical interconnections in the system
1010
10
10
10
10
10
10
2
4
6
8
10
12
SYSTEM performances (bits/sec)
1012
1014
108
HERE/98
BS/98
CITR/95
AT&T/93
SPARCL/95
AT&T/91
PPOS4
E4O4
O8 O12
PPOS12
OHM4 2002OHM4 2007
FET-SEED
CMOS-SEED
S-SEED
CLF/CI + PnpN
CLF/CI + Si
VCSEL/MSM
Multi-chip
demonstrators
PPOS8
Tera Bit /sec
![Page 37: Optoelectronic Stochastic Parallel Processors for real time image processing Groupe Physique des images. Ph. Lalanne, J.C.Rodier et P.Chavel Institut d’Optique](https://reader035.vdocuments.net/reader035/viewer/2022062407/56649e895503460f94b8d8d6/html5/thumbnails/37.jpg)
37Possible research directions...
Rapid and local reconfigurable interconnects :
Optical random number generator :
OSPP with reconfigurable optical interconnects :
…a general purpose optoelectronic stochastic parallel processor for low-level image processing ?
…stochastic neural networks (Boltzmann Machines)
...cryptograpy, telecom channel simulation, Monte Carlo methods...
Alvaro Cassinelli:
(speckle : Gb/s contre Mb/s pour puce VLSI, 1998)
Alvaro Cassinelli:
(speckle : Gb/s contre Mb/s pour puce VLSI, 1998) Micro-optic integration.