Topic 5: Level 0

David L. Hall

Topic Objectives

• Introduce concepts of level 0 processing

• Identify categories of techniques for level 0 processing (e.g., signal and image processing)

• Extend the concept of traditional signal and image processing and conditioning to meta-data generation

Level 0 Processing

Comments on Level 0 Processing

• Level 0 processing concerns processing each source of data independently to obtain the most useful information possible; we want to “squeeze out” useful information

• In general, input data to a fusion system may include scalar or vector data, signal data, image data, or textual information. Each of these classes of information may entail an entire range of potentially applicable techniques (e.g., signal processing, image processing, text-based processing)

• It is beyond the scope of this lecture (and indeed this entire course) to address these subjects – however, we seek to make the student aware of the need to consider such processing

• Finally, an emerging new area is the generation of semantic meta-data (e.g., automatic semantic labeling of images) as a new powerful process to improve data fusion.

Metasensor Processing

METASENSOR PROCESSING

CA

TEG

OR

YFU

NC

TIO

N

TEC

HN

I QU

E

Sensor Validation

and Calibration

Parametric

Modeling

• Spectral analysis• Wavelet processing• Short-time Fourier transform• AR modeling• System I/D analysis

Entity

Detection

• Array processing - Synthetic aperture (SA) - Inverse SA• Moving target indicator (MTI) • Hypothesis testing• Pre-detection fusion• Distributed fusion

• Phase and gain calibration• Sensor data validation

Non-Parametric

Analysis

• Time-series analysis• Dynamical systems analysis• Image analysis• Neural networks• Trend analysis• Thresholding and integration• Energy detection

Level 0 Processing by Source Type

Scalar orVectorSensor

Signalsensor

ImagingSensor

Text-based information

yt

Semantic processing

Signal processing

Image processing

Data Conditioning

Original Source Data plus transformed & added meta-data

Source Type: Scalar & Vector Data• Representative techniques

– Scaling methods and transformations– Coordinate transformations and rotations– Smoothing, filtering, averaging– Thresholding– Feature extraction and representations

y y + z Transformation

Original data Original data Alternate representations

Source Type: Signal Data• Representative

Parametric Modeling Techniques

• Spectral analysis• Wavelet processing• Short-time Fourier

transform• Auto-Regression modeling• System Identification

analysis

• Representative Non-Parametric Modeling Techniques

• Time-series analysis• Dynamical systems analysis• Neural networks• Trend analysis• Threshold & integration• Energy detection

Transform + characterization vector, y

Original signalTransformed signal

Examples of Signal Processing• Signal processing and

enhancement are standard on most radios & receivers– High & low frequency filtering– Boost of selected frequency

ranges– “Coloring” of signals

• Active Noise Cancelation devices– Active cancellation of

background noise (e.g., airline engine noise)

Extensive commercial tools are available such as MATLAB, Mathematica, etc

Source Type: Image Data• Classification• Feature Extraction• Pattern Recognition• Projection• Multi-Scale Signal Analysis

• Principal Components Analysis

• Independent Component Analysis

• Self-Organizing Maps• Hidden Markov Models• Neural Nets

+ Meta-Data

Original Image Transformed image

ImageProcessing

An enormous amount of methods exist to process and transform image data from the individual pixel level, to object level to complete image level. Commercial tools such as Photoshop Pro make many of these available to casual users

Examples of Image Processing

http://web.uct.ac.za/depts/physics/laser/hanbury/intro_ip.html

Contrast enhancement

Image sharpening using wavelets

http://www.phasespace.com.au/wavelet_ex.htm

Deconvolution

http://www.phasespace.com.au/wavelet_ex.htm

Source Type: Text Data• Representative techniques

– Key word extraction– Name disambiguation– Link of semantics to parametric data (e.g. location)– Syntactic processing– Thesaurus processing– Semantic distance calculations– Links to other documents/

Textual input via human reports or web info.

Textual input via human reports or web info.

Meta Data - key words - lexicon data - identified parameters - links to related documents

+TextualProcessing

Concept of Semantic Meta Data Generation• Signal & images represent a data

level view of patterns, features & characteristics

• In human pattern recognition, we identify labels to represent signals & data

• New techniques are being developed to train computers to perform a similar function

Problem: How to effectively query on a very large collection of satellite imagery for semantically meaningful regions ?

Use semantic categorization combined with CBIR on small “patches” of the images (Dr. James Wang)

Query Formulation

Query “patch” – pertaining to some semantics, e.g. mountains

Satellite Image Database Ranked

ResultsPurpose ?

Geography - Find mountainous regions with snow-caps (low-level semantics).

Forestry – Find forests of a certain density, analyze deforestation (mid-level semantics).

Military – Find air-bases in certain regions of the world (high-level semantics).

Limitations of Current Approaches

• Difficulty in defining classes Continuum of variety Subjectivity

• No major improvement in classification accuracy G. G. Wilkinson, “Results and Implications of a Study of Fifteen

Years of Satellite Image Classification Experiments,” IEEE Trans. On Geoscience and Remote Sensing, 2005.

• Recognition of higher-order semantics More focus on spectral than spatial dimensions

• Scalability of complex querying and browsing

• Flexibility to handle diverse applications

Automatic Semantic Categorization

Summary of Approach

• Practical implementation on large-scale archives

• Flexibility to adapt to various applications of satellite imagery, e.g., Geography, Military, Metallurgy, Agriculture etc.

• Exploiting spectral and spatial information using a generative model for semantic categorization (supervised learning)

• Handling of untrained classes (supervised learning)

• Using a scalable CBIR system for efficient querying and browsing (unsupervised)

Overview of the System

• Classification– Generative Classifier: Two-dimensional Multi-resolution Hidden Markov

Models (2-D MHMM)

• Handling untrained classes– Discriminative Classifier: Support Vector Machines

• Querying for fast retrieval– Integrated Region Matching (IRM) measure used in SIMPLIcity system

K trained 2D-MHMMs

40 x K training patches

Query patch

Patch Class

Random samples from trained and untrained classes

K Likelihood scores Trained

biased SVM

Database search using IRM

Query Results

Details of the Architecture

Efficient database design

Pre-computation of classes using 2D-MHMM for faster response

Suitable user interface and visualization (Common CBIR issue)

Training process

Automatic Semantic Categorization

Retrieval Results

http://riemann.ist.psu.edu/image

Retrieval Results

http://riemann.ist.psu.edu/image

Topic 5 Assignments

• Preview the on-line topic 5 materials• Read Farid paper (2008)

Data Fusion Tip of the Week

Level 0 processing is a vital part of the data fusion process - in essence pre-processing or conditioning data from individual sources and sensors. This kicks off all subsequent processing. It is necessary to do the best job possible at this stage without either introducing spurious information into the observed data, nor failing to extract and enhance the data “at the source”. It is not possible to use “down-stream” data fusion processing to make up for failures at the level 0 stage.