![Page 1: [1] Pixel-Level Fusion of Active/Passive Data for Real-Time Composite Feature Extraction and Visualization Alan Steinberg and Robert Pack Space Dynamics](https://reader030.vdocuments.net/reader030/viewer/2022032606/56649e945503460f94b9921b/html5/thumbnails/1.jpg)
[1]
Pixel-Level Fusionof Active/Passive Data
for Real-Time Composite Feature Extraction and Visualization
Alan Steinberg and Robert Pack
Space Dynamics Laboratory/Utah State University
Presented atNATO IST-036/RWS-005
Halden, Norway
10-13 September 2002
![Page 2: [1] Pixel-Level Fusion of Active/Passive Data for Real-Time Composite Feature Extraction and Visualization Alan Steinberg and Robert Pack Space Dynamics](https://reader030.vdocuments.net/reader030/viewer/2022032606/56649e945503460f94b9921b/html5/thumbnails/2.jpg)
[2]
Concept of Pixel-Level Fusion
• Active Sensor – LADAR
• Passive Sensor – Electro-Optic (EO) Radiation
• Fused into One Data Set
RANGE DATA FROM MULTI-
CHANNEL LADAR
PASSIVE IR IMAGE DATA
+
FUSED RANGE IMAGE
![Page 3: [1] Pixel-Level Fusion of Active/Passive Data for Real-Time Composite Feature Extraction and Visualization Alan Steinberg and Robert Pack Space Dynamics](https://reader030.vdocuments.net/reader030/viewer/2022032606/56649e945503460f94b9921b/html5/thumbnails/3.jpg)
[3]
Enables Observation Through Partial Obscuration
![Page 4: [1] Pixel-Level Fusion of Active/Passive Data for Real-Time Composite Feature Extraction and Visualization Alan Steinberg and Robert Pack Space Dynamics](https://reader030.vdocuments.net/reader030/viewer/2022032606/56649e945503460f94b9921b/html5/thumbnails/4.jpg)
[4]
Employment Concept
IndividualPixel
r2
r3
r4
r1
IndividualPixel
r1
r2
r3r4
r5
Cloud Layers Cloud Layers
Ground Clutter
Upper Canopy
Lower Canopy
![Page 5: [1] Pixel-Level Fusion of Active/Passive Data for Real-Time Composite Feature Extraction and Visualization Alan Steinberg and Robert Pack Space Dynamics](https://reader030.vdocuments.net/reader030/viewer/2022032606/56649e945503460f94b9921b/html5/thumbnails/5.jpg)
[5]
Fused LADAR/EO Image 1(Co-Registered and Georeferenced at the Pixel Level)
LADAR Image EO Image
![Page 6: [1] Pixel-Level Fusion of Active/Passive Data for Real-Time Composite Feature Extraction and Visualization Alan Steinberg and Robert Pack Space Dynamics](https://reader030.vdocuments.net/reader030/viewer/2022032606/56649e945503460f94b9921b/html5/thumbnails/6.jpg)
[6]
LADAR Image EO Image
Fused LADAR/EO Image 2(Co-Registered and Georeferenced at the Pixel Level)
![Page 7: [1] Pixel-Level Fusion of Active/Passive Data for Real-Time Composite Feature Extraction and Visualization Alan Steinberg and Robert Pack Space Dynamics](https://reader030.vdocuments.net/reader030/viewer/2022032606/56649e945503460f94b9921b/html5/thumbnails/7.jpg)
[7]
Orthoprojection of the Two 3D Images Combined in Geographic Space
First Image
Second Image
![Page 8: [1] Pixel-Level Fusion of Active/Passive Data for Real-Time Composite Feature Extraction and Visualization Alan Steinberg and Robert Pack Space Dynamics](https://reader030.vdocuments.net/reader030/viewer/2022032606/56649e945503460f94b9921b/html5/thumbnails/8.jpg)
[8]
Profile Views in 3D Viewer
First LADAR/EO Image
First and Second LADAR/EO Images Combined
![Page 9: [1] Pixel-Level Fusion of Active/Passive Data for Real-Time Composite Feature Extraction and Visualization Alan Steinberg and Robert Pack Space Dynamics](https://reader030.vdocuments.net/reader030/viewer/2022032606/56649e945503460f94b9921b/html5/thumbnails/9.jpg)
[9]
Visualization Concept -
Data Volume Exploration and Clutter Removal
![Page 10: [1] Pixel-Level Fusion of Active/Passive Data for Real-Time Composite Feature Extraction and Visualization Alan Steinberg and Robert Pack Space Dynamics](https://reader030.vdocuments.net/reader030/viewer/2022032606/56649e945503460f94b9921b/html5/thumbnails/10.jpg)
[10]
3D Images of Cluttered Environment
Image Plane
Vegetation Canopy
Ground Plane
Terrain (per DTED or detected ground)
![Page 11: [1] Pixel-Level Fusion of Active/Passive Data for Real-Time Composite Feature Extraction and Visualization Alan Steinberg and Robert Pack Space Dynamics](https://reader030.vdocuments.net/reader030/viewer/2022032606/56649e945503460f94b9921b/html5/thumbnails/11.jpg)
[11]
Predict Ground
Image Plane
Bottom of Canopy Layer(Terrain + 3m)
Ground Plane
Terrain (per DTED or detected ground)
![Page 12: [1] Pixel-Level Fusion of Active/Passive Data for Real-Time Composite Feature Extraction and Visualization Alan Steinberg and Robert Pack Space Dynamics](https://reader030.vdocuments.net/reader030/viewer/2022032606/56649e945503460f94b9921b/html5/thumbnails/12.jpg)
[12]
Slice Volume at Bottom of Canopy
Image Plane
Bottom of Canopy Layer(Terrain + 3m)
Ground Plane
Terrain (per DTED or detected ground)
Residual Shadowed Regions: PD 0
![Page 13: [1] Pixel-Level Fusion of Active/Passive Data for Real-Time Composite Feature Extraction and Visualization Alan Steinberg and Robert Pack Space Dynamics](https://reader030.vdocuments.net/reader030/viewer/2022032606/56649e945503460f94b9921b/html5/thumbnails/13.jpg)
[13]
Example of Canopy Slicing
Orthoprojected Using LADAR – Trees RemovedOblique EO Scene
Residual Shadowed Region: PD 0
Ground with Trees Removed
![Page 14: [1] Pixel-Level Fusion of Active/Passive Data for Real-Time Composite Feature Extraction and Visualization Alan Steinberg and Robert Pack Space Dynamics](https://reader030.vdocuments.net/reader030/viewer/2022032606/56649e945503460f94b9921b/html5/thumbnails/14.jpg)
[14]
Two Shots Combined – Trees Removed
Residual Shadowed Region: PD 0
Corresponding EO Image
![Page 15: [1] Pixel-Level Fusion of Active/Passive Data for Real-Time Composite Feature Extraction and Visualization Alan Steinberg and Robert Pack Space Dynamics](https://reader030.vdocuments.net/reader030/viewer/2022032606/56649e945503460f94b9921b/html5/thumbnails/15.jpg)
[15]
Target Hidden Under Tree
Hidden Target
![Page 16: [1] Pixel-Level Fusion of Active/Passive Data for Real-Time Composite Feature Extraction and Visualization Alan Steinberg and Robert Pack Space Dynamics](https://reader030.vdocuments.net/reader030/viewer/2022032606/56649e945503460f94b9921b/html5/thumbnails/16.jpg)
[16]
Target Exposed After Tree Removal
Exposed Target
![Page 17: [1] Pixel-Level Fusion of Active/Passive Data for Real-Time Composite Feature Extraction and Visualization Alan Steinberg and Robert Pack Space Dynamics](https://reader030.vdocuments.net/reader030/viewer/2022032606/56649e945503460f94b9921b/html5/thumbnails/17.jpg)
[17]
Conclusions (1):Pixel-Level Fusion of Active/Passive Data for Real-Time
Composite Feature Extraction & Visualization
• Pixel-Level fusion of LADAR + EO data facilitates target detection & recognition in highly occluded environments
• Clutter
– tree canopies, clouds, etc. –
can be removed through geometric filters
![Page 18: [1] Pixel-Level Fusion of Active/Passive Data for Real-Time Composite Feature Extraction and Visualization Alan Steinberg and Robert Pack Space Dynamics](https://reader030.vdocuments.net/reader030/viewer/2022032606/56649e945503460f94b9921b/html5/thumbnails/18.jpg)
[18]
Visualization Implication
![Page 19: [1] Pixel-Level Fusion of Active/Passive Data for Real-Time Composite Feature Extraction and Visualization Alan Steinberg and Robert Pack Space Dynamics](https://reader030.vdocuments.net/reader030/viewer/2022032606/56649e945503460f94b9921b/html5/thumbnails/19.jpg)
[19]
Some Visualization Implications
• Target Detection & Recognition
> Probabilities of Detection
> Probabilities of Target Presence (prior & posterior)
> Exploitation of Negative Data
• Temporal Evolution of Estimation and Expectations
> Out-of Sequence Data
> Situation Projection
> Expected Updates
![Page 20: [1] Pixel-Level Fusion of Active/Passive Data for Real-Time Composite Feature Extraction and Visualization Alan Steinberg and Robert Pack Space Dynamics](https://reader030.vdocuments.net/reader030/viewer/2022032606/56649e945503460f94b9921b/html5/thumbnails/20.jpg)
[20]
Prior & Posterior Detection Probabilities; e.g. Presentation of Negative Data
Target Detectability
Map
0.75 - 1.00
0 - 0.25
0.25 – 0.50
0.50 – 0.75
pD(x,t+n|t,A)
Prior or Posterior Probability Density
(e.g. Track Projection Uncertainty)
x = Target state (type & location, etc.)
t, t+n = Update times
A = Action plan (sensor route, sensor controls, processing controls, etc.)
p(x,t+n|t)
ExpectationsMap
Likely false alarm
Likely missed
detections
p(x,t+n|t)pD(x,t+n|t,A)
![Page 21: [1] Pixel-Level Fusion of Active/Passive Data for Real-Time Composite Feature Extraction and Visualization Alan Steinberg and Robert Pack Space Dynamics](https://reader030.vdocuments.net/reader030/viewer/2022032606/56649e945503460f94b9921b/html5/thumbnails/21.jpg)
[21]
Some Visualization Implications
• Target Detection & Recognition
> Probabilities of Detection
> Probabilities of Target Presence (prior & posterior)
> Exploitation of Negative Data
• Temporal Evolution of Estimation and Expectations
> Out-of Sequence Data
> Situation Projection
> Expected Updates
![Page 22: [1] Pixel-Level Fusion of Active/Passive Data for Real-Time Composite Feature Extraction and Visualization Alan Steinberg and Robert Pack Space Dynamics](https://reader030.vdocuments.net/reader030/viewer/2022032606/56649e945503460f94b9921b/html5/thumbnails/22.jpg)
[22]
Revised Estimates of Track Histories (MTI + Imagery)
TimeELEMENTARY SCHOOL
MISSILE ASSEMBLY FACILITY
MTI Reports
Imagery Reports
Latency Issues:Out of Sequence Data
Initial Estimates of Track Histories (MTI only)
TimeELEMENTARY SCHOOL
MISSILE ASSEMBLY FACILITY
![Page 23: [1] Pixel-Level Fusion of Active/Passive Data for Real-Time Composite Feature Extraction and Visualization Alan Steinberg and Robert Pack Space Dynamics](https://reader030.vdocuments.net/reader030/viewer/2022032606/56649e945503460f94b9921b/html5/thumbnails/23.jpg)
[23]
Adaptive Information ExploitationProcess
• Acknowledge uncertainties
• Estimate length of time for which decision can be deferred
• Estimate probability of resolving ambiguity in time
> Given current collection plan
> By redirecting current resources
> By adding resource
• Estimate Cost & Net Utility of Candidate Action Plans
• Select & Implement Revised Plan
![Page 24: [1] Pixel-Level Fusion of Active/Passive Data for Real-Time Composite Feature Extraction and Visualization Alan Steinberg and Robert Pack Space Dynamics](https://reader030.vdocuments.net/reader030/viewer/2022032606/56649e945503460f94b9921b/html5/thumbnails/24.jpg)
[24]
Estimated History Estimated Present Situation Projection Situation Estimated History Estimated Present Situation Projection Situation Situation Time
-40 -30 -20 -10 0 +10 +20 +30 +40
Received Reports Expected CoverageReceived Reports Expected Coverage
Tgt Selection Decision Tgt Engagement Decision
Visualizing Time-Varying Situation Estimation & Expectation (1 of 3)
Current Estimate of Present Situation
Current Estimate of Present Situation
Report Time (Plan A)
![Page 25: [1] Pixel-Level Fusion of Active/Passive Data for Real-Time Composite Feature Extraction and Visualization Alan Steinberg and Robert Pack Space Dynamics](https://reader030.vdocuments.net/reader030/viewer/2022032606/56649e945503460f94b9921b/html5/thumbnails/25.jpg)
[25]
Estimated History Estimated Present Situation Projection Situation Estimated History Estimated Present Situation Projection Situation Situation Time
-40 -30 -20 -10 0 +10 +20 +30 +40
Received Reports Expected CoverageReceived Reports Expected Coverage
Tgt Selection Decision Tgt Engagement Decision
Visualizing Time-Varying Situation Estimation & Expectation (2 of 3)
Current Projection of Future Situation
Current Projection of Future Situation
Report Time (Plan A)
![Page 26: [1] Pixel-Level Fusion of Active/Passive Data for Real-Time Composite Feature Extraction and Visualization Alan Steinberg and Robert Pack Space Dynamics](https://reader030.vdocuments.net/reader030/viewer/2022032606/56649e945503460f94b9921b/html5/thumbnails/26.jpg)
[26]
Visualizing Time-Varying Situation Estimation & Expectation (3 of 3)
Report Time (Plan A)Estimated History Estimated Present Situation Projection Situation
Estimated History Estimated Present Situation Projection Situation Situation Time
-40 -30 -20 -10 0 +10 +20 +30 +40
Received Reports Expected CoverageReceived Reports Expected Coverage
Tgt Selection Decision Tgt Engagement Decision
Expected Update of Present Situation
Expected Update of Present Situation
![Page 27: [1] Pixel-Level Fusion of Active/Passive Data for Real-Time Composite Feature Extraction and Visualization Alan Steinberg and Robert Pack Space Dynamics](https://reader030.vdocuments.net/reader030/viewer/2022032606/56649e945503460f94b9921b/html5/thumbnails/27.jpg)
[27]
Conclusions
• Pixel-Level fusion of LADAR + EO data facilitates target detection & recognition in highly occluded environments> Clutter – tree canopies, clouds, etc. – can be removed over
time through geometric filters
• Enables visualization of time-varying situation estimation & expectation> Presentation of Detection Probabilities (prior & posterior):
e.g. Presenting Negative Data
> Presentation of Temporal Evolution of Estimates and of Expectations
– Out-of Sequence Data
– Situation Projection
– Expected Updates & Information Evolution
![Page 28: [1] Pixel-Level Fusion of Active/Passive Data for Real-Time Composite Feature Extraction and Visualization Alan Steinberg and Robert Pack Space Dynamics](https://reader030.vdocuments.net/reader030/viewer/2022032606/56649e945503460f94b9921b/html5/thumbnails/28.jpg)
[28]
Thank You
Mange Takk!