neural networks in measurement systemshome.mit.bme.hu/~horvath/ltp/ltp_slides.pdf19. 07. 2002. nato...

19. 07. 2002. NATO ASI LTP Leuven, Belgium 1

Neural Networks in Measurement Systems

Gábor Horváth

Department of Measurement and Information Systems


OutlineOutlineMeasurement systemsSensors, transducers, signal processingModelingThe role of neural networksThe importance of high-speed, low cost implementationCMAC: an alternative to MLPModeling capabilityGeneralization capability, analytical results CMAC with improved generalization capabilityGeneral cases, SVM versus CMAC


MeasurementMeasurementCan be defined in relation to modelingMeasurement is an autonomous phase of modeling; it is embedded in modeling


MMeasurementeasurementEmpirical process

to obtain experimental data (observations), primary information collection, orto obtain additional information to the a priori one.

to use the experimental data for obtaining (determining) the free parameters (features) of a model. to validate the model


MeasurementMeasurementThe goal of modelingThe goal of modelingThe goal of modeling

Collecting a priori knowledgeCollecting a priori knowledge

A priori modelA priori model

Experiment designExperiment design

Observations, determiningfeatures, parameters

Observations, determiningfeatures, parameters

Model validationModel validation

Final modelFinal model

CorrectionCorrection

Measurement

Modeling


MMeasurementeasurementTo obtain experimental data (observations),

sensing (primary information collection)To process information

signal conversionsignal processing


MMeasurementeasurement

Object formeasurement

Signal transmission

channel

Measuring device

input

output noise

noise

observationmeasurementresult

prior information


MeasurementMeasurementSpecial tasks with sensors’ signals

Sensor linearizationnonlinear sensor characteristics

Sensor fusionincomplete or unreliable sensors

Virtual sensingthe quantity to be measured cannot be sensed directly, other related quantity can be measured

Remote sensingindirect measurement (e.g. reflected light: motion of a mechanical system)


MeasurementMeasurementSensor fusion (fusion is a process of combining information from different sensors, when no physical law indicating the correct way to combine this information)

the system cannot rely on a single sensor to provide sufficient informationit must rely on fusion of multiple sensor measurements to obtainmore complete and accurate information

Sensors’ signalsuncertain

limited resolutionrandom measurement noiseinaccurate conversion

incomplete


MeasurementMeasurementSensor fusion

complementary fusione.g. several visual sensors pointed in different directions

competitive fusione.g. measuring of a distance: laser range sensor, acoustic

(ultrasonic) sensor pointed at the same object

cooperative fusione.g. fusion of physical measurements

2D images →3D representation


ExampleExampleMeasuring a distance


ExampleExampleSensor fusion for uncertainty reduction


ExampleExampleSensor fusion

common internal representation


ModelingModelingWhy do we need models?What models can be built?How to build models?


ModelingModelingWhat is a model?

Some (formal) description of a system, a separable part of the world. Represents essential aspects of a system

Key concepts: separation, selection, parsimonyMain features:

• All models are imperfect. Only some aspects are taken into consideration, while many other aspects are neglected.

• Easier to work with models than with the real systems


ModelingModelingSeparation:

the boundaries of the system have to be defined. system is separated from all other parts of the world

Selection:Only certain aspects are taken into consideration e.g.

information relation, interactions energy interactions

Parsimony:It is desirable to use as simple model as possible

Occam’s razor The most likely hypothesis is the simplest one that is

consistent with all observationsThe simpler of two theories, two models is to be preferred.


ModelingModelingWhy do we need models?

To simulate a systemto predict the behaviour of the system (prediction, forecasting),to determine faults and the cause of malfunctions, fault diagnosis, error detectionto control the system to obtain prescribed behaviour, operationsto increase observability: to estimate such parameters which are not directly observable (indirect measurement) system optimization


ModelingModelingWhat models can be built?

Approachesfunctional models

• parts and its connections based on the functional role in the system

physical models• based on physical laws, analogies (e.g. electrical analog

circuit model of a mechanical system)

mathematical models• mathematical expressions (algebraic, differential

equations, logic functions, etc.)


ModelingModelingWhat models can be built?

A priori informationphysical models, “first principle” models models based on observations (experiments)

Aspectsstructural modelsinput-output (behavioral) models


Model classesModel classesBased on the system characteristics

Static – dynamic Deterministic – stochastic Continuous-time – discrete-time Lumped parameter – distributed parameter Linear – non-linear Time invariant – time variant …


Model classesModel classesBased on the modeling approach

parametricknown model structurelimited number of unknown parameters

nonparametricno definite model structuredescribed in many points (frequency characteristics, impulse response)

semi-parametric


Model classesModel classesBased on the a priori information (physical insight)

Black-box

White-box

Structure ParametersStructure Parameters



Structure ParametersStructure ParametersGrey-box


Known Missing (Unknown)

Case of classical measurement


ModelingModelingMain steps

collect information

model set selectionexperiment design and data collectiondetermine model parameters (parameter estimation, decision)

model validation


ModelingModelingCollect information

physical insight (a priori information)understanding the physical behaviour

only observations or experiments can be designed application

what operating conditions• one operating point• a large range of different conditions

what purpose• scientific

– basic research, • engineering

– to study the behavior of a system, – to detect faults, – to design control systems,– etc.


ModelingModelingModel set selection

static – dynamiclinear – non-linear non-linear

linear - in - the - parametersnon-linear - in - the - parameters

parametric – non-parametricwhite-box – black-box


ModelingModelingModel structure selection

known model structure (available a priori information)no physical insights, general model structure

general rule: always use as simple model as possible

• linear• feed-forward

•••


Experiment design and data collectionExperiment design and data collectionExcitation

input signal selectiondesign of excitation

time domain or frequency domain identification (random signal, multi-sine excitation, impulse response, frequency characteristics)persistent excitation

Measurement of input-output data no possibility to design excitation signal

noisy data, missing data, distorted data non-representing data


Modeling (some examples)Modeling (some examples)Resistor modelingModel of a ductModel of a steel converter (model of a complex industrial process)Model of a signal (time series modeling)


Modeling (example)Modeling (example)Resistor modeling

the goal of modeling: to get a description of a physical system (electrical component)parametric model

linear modelconstant parameter

variant model

frequency dependent

RRIU =

I

U

I R(I)IIRU )(=

U

DC

cR AC

12)(

)()()()()()(

+===

CRfjRfZ

fIfUfZfIfZfU

π



nonparametric model

Z

fI

Unonlinear

I

Ulinear

AC

frequency dependent

DC



parameter estimation based on noisy measurements

I

Ulinear

OutputSystem

+ +nunI

Input

I U

System

nu

OutputInput

+nI

I

+

U

+

System noise

Measurement noise

+

Input noise


Modeling (example)Modeling (example)Model of a duct

the goal of modeling: to design a controller for noise compensation. active noise control problem


Modeling (example)Modeling (example)



physical modeling: general knowledge about acoustical effects; propagation of sound, etc.

no physical insight. Input: sound pressure, output: sound pressure what signals: stochastic or deterministic: periodic, non-periodic, combined, etc. what frequency range time invariant or notfixed solution, adaptive solution. Model structure is fixed, model parameters are estimated and adjusted: adaptive solution



nonparametric (linear) model of the duct (H1)FIR filter with 10-100 coefficients

0 200 400 600 800 1000-45

-40

-35

-30

-25

-20

-15

-10

-5

0

5

frequency (Hz)

mag

nitu

de (d

B)


Modeling (example)Modeling (example)Nonparametric models: impulse responses


Modeling (example)Modeling (example)Model of a steel converter (LD converter)


Modeling (example)Modeling (example)Model of a steel converter (LD converter)

the goal of modeling: to control steel-making process to get predetermined quality steelphysical insight:

complex physical-chemical process with many inputsheat balance, mass balancemany unmeasurable (input) parameters

no physical insight: there are input-output measurement data


The modeling taskThe modeling task

System

NeuralModel

Σε

components(parameters)

temperature

predictedtemperature

+

-

oxygen

measuredtemperature

components(parameters)

Σε

-

+

Copy ofModel

predictedoxygen

InverseModel

model outputtemperature


Neural Networks in MeasurementsNeural Networks in MeasurementsTasks to be solved in measurement systems

Non-linear transformation (linearization)Signal fusion (combining information from manysensors)Non-linear static or dynamic input-output mappingData compression, data separation


Neural Networks in MeasurementsNeural Networks in MeasurementsMain features of neural networks

Neural networks are universal modeling devices(universal approximators) Implement (static or dynamic) linear or non-linear mappinglinearization, static/dynamic (nonlinear) system modeling, sensor fusion

Their parameters are determined by learning fromexamples (measurement observations)Adaptive system (real-time adaptation) Data transformers (data compression, data separation)


Neural Networks in MeasurementNeural Networks in MeasurementNetwork function:

non-linear static mapping:• MLP, • RBF, • CMAC,• SVMnon-linear dynamic mapping• extended networks: local or global memory,

local or global feedbacklinear or non-linear data compression• MLP • PCA• ICA


Neural Networks for Embedded SystemsNeural Networks for Embedded SystemsEmbedded system:

small size, low cost, low power consumption

Real-time operation: high-speed learning and recall

Hardware implementation:analog

critical: accuracy, learning (weight updating)

digitalcritical: multipliers, activation function


Neural networks for Embedded SystemsNeural networks for Embedded SystemsEfficient implementation of the critical elements Special architecture without

multipliersnonlinear activation function


CMAC Neural NetworkCMAC Neural NetworkCerebellar Model Articulation Controller Albus 1975

Σ

discrete input space(quantized input)

aa

i+i+3

u a a y

y

wwwww

ii+i+i+

21

3

C=4

wa

association vector weight vector

u1

2

i+

i+4

5u

u

2

3

aa

aa

j+

j+

123

j j

j+

w

wwww

j+j+j+

1

32

ai

a

a

a

i+1

i+i+

45

CMAC: no multipliers, no activation functions, only encoder, memory, adder

∑=

=1)(:

)(u

ujaj

jwy


CMAC Neural NetworkCMAC Neural Network

a =

Σ

aa

==

Discrete input space

u

aaai+

1

===

111

00

a =a =M

M

0

00

z

y

y

M

-1-2

12

i+

i+

2

C= 4

aassociation compressed weight

ai+3=1z

i+

i+

i+z

w

w

w

w

i+

i+

i+

z

z

2

3 3

2

1 1

ii i

wz

u a a z

vector association vectorvector


CMAC as a basis function networkCMAC as a basis function network

Basis functions


CMAC as a basis function networkCMAC as a basis function networkC overlays of basis functionsBasis functions are arranged in overlays: one-dimensional case

C=8 dtrain=4

C=4


CMAC as a basis function networkCMAC as a basis function networkBasis functions are arranged in overlays: two-dimensional case

Basis functions

)1(1

01

−+= ∏

=−

N -1

iiN CR

CM

C overlays


CMAC modeling capabilityCMAC modeling capability

One-dimensional case: can learn any training data set exactly

Multi-dimensional case: can learn any non-linear mapping from the additive function set (consistency equations)


CMAC generalization capabilityCMAC generalization capability

Important parameters:

C generalization parameterdtrain distance between adjacent training data

Interesting behaviorC=l*dtrain : linear interpolation between the

training points

C≠l*dtrain : significant generalization error non-smooth output


CMAC generalization errorCMAC generalization error

=


C/2



Analytical model of Analytical model of the generalization errorthe generalization error

When solving a given problem, try to avoid solving a more general problem as an intermediate step(V. Vapnik: The Nature of Statistical Learning Theory)


Analytical model of Analytical model of the generalization errorthe generalization error

Awy =y(u)=a(u)Tw.

→←

→←→←

=

TL

T

T

)(

)2(

)1(

a

aa

AM

=

011111111...000000000000

0...001111111010000000000...000000010111111000000...00000000000111101111

MMM

A

( ) 1† −= TT AAAAdyAw †=∗


Analytical solutionAnalytical solution

dTT yAATATwy 1)( −∗ == d

T yBTAy = ( ) 1−= TAAB

dT

d yAAByv 1)( −==

dTjjv yb=

=

1 1 1 1 1 10... 0 0 0 0 0 0

0...0 0 0 0 0 1 1 1 1 1 1 1 1 000...0 0 0 0 0 0 1 1 1 1 1 1 1 1 00...0 0 0 0 0 0 0 1 1 1 1 1 1 1 1

L

MMM

T

( ) vGyBTA idiT

iy ==

( ) dyTAPh †−= P: interpolation matrix


Analytical solutionAnalytical solution

−

+−−−

−−−

−+

−+

−−−−

+−−−

=

cdc

dccdcc

dcc

cdcd

cd

cdccdc

dccdcc

T

000

11200310

11420

0210011100122

00211001

00011000

MMMMM

MMMMM

LL

MMMMM

LL

MMMMM

TA


FilterFilter--bank model of the CMACbank model of the CMAC

yd

B

G0

Gd-1

y

G0=AAT=B-1

( ) 1−= TAAB

Gi interpolation filters


Analytical resultAnalytical resultThe key problem: to find the inverse of AAT (symmetrical, Toeplitz, banded)

***************

**********

**********

**********

MO

L

−−−

−−−−−−−−−

−−−

=

CdCdCzdC

zdCdCdCCdCdCzdCdCdCCdC

zdCdCdCC

T

2...0...00

0...0...220...0...20...00...2

MMM

AA

Conjecture: the inner part of the inverse of a matrix is the inverse of the inner part of a matrix

=

dCz


Analytical resultAnalytical resultExtension to get a cyclic matrix

−−−

−−−−−−−−−

−−−

=

CdCdCzdC

zdCdCdCCdCdCzdCdCdCCdC

zdCdCdCC

T

2...0...00

0...0...220...0...20...00...2

MMM

AA

−−−−

−−−−−−−−−−−−−−−−

=

CdCzdCdCdC

dCzdCdCdCCdCdCdCzdCdCdCCdC

dCdCzdCdCdCC

Tcyc

...0...2

3...0...222...0...2

2...00...2

MMM

AA


Analytical resultAnalytical result

Using the spectral representation of

TcycAA

nlk

lk en

)1)(1(2

,1

−−

=π

FF FH


Analytical resultAnalytical resultUsing the spectral representation the general form of the inverse can be written as:

HTcyc FΛFAA =

( ) Hk

Tcyc ff FFAA λ=)(

H

k

Tcyc FFAA

=−

λ1)( 1

( ) ( ) ( )12cos24cos222cos21 −

−++

−+

−+=

nzkzdC

nkdC

nkdCC

k

πππλ

K



( ) 1−= TAAB

=

dCz

∑−

= −−+++−

−−

=1

0, 2cos)()1(2cos)(

1)2cos()(2cos1 n

kji

dnkπCdzd

nzkπzdC

nkπ

nijkπ

nB

According to numerical results the conjecture is truewith high accuracy


Analytical resultAnalytical resultMore special case of z=1

−

−−−−

−

=

CdC

dCCdCdCCdC

dCC

T

0...000

0...0000...0000...0

MMM

AA



[ ]

−−−

−−−

=−

xx

xx

x

x

10001100

0010011

L

L

MM

L

L

L

KI

The inverse can be determined using [ ] 1−−= KIP x

− 001



dCCx−

−=

( ) ( )[ ] PKIAAdC

dCCT−

=−−= −− 111

2−<−

−=dC

Cx

P

Θchx 2−=

ifif


Analytical resultAnalytical resultIf n is large enough we can use

P

( )l

llii b

rCr

−−=+ 2

11 )(,B 22 4bCr −= dCb −=


Generalization error (max)Generalization error (max)

1 2 3 4 5 6 7 80

0.02

0.04

0.06

0.08

0.1

0.12

0.14

0.16

0.18

0.2

C/dtrain

Abs. error


CMAC gCMAC geometrical interpretationeometrical interpretationWeight activation


CMAC gCMAC geometrical interpretationeometrical interpretationWeight activations


Error reductionError reductionHow to reduce this error?

The real reason of the erroneous behavior: non-uniform distribution of the weight values

Regularization

( ) ( ) ( ) ( ) ( )

−++=+ kw

Ckykkkwkw i

dii

)(1 λεµ a

( ) ( ) ( )( ) ( ) ( )2

2

221

−+−= kw

Cky

kykykC id

dλ


ResultsResults

a.) b.) c.)


ResultsResults


More general casesMore general casesInput positions: periodical by two-samples

Basis functions

Training samples


More general casesMore general casesInput positions: periodical by two-samples; superposition


More general casesMore general casesRandom input positions

Red: periodical by two-samples, blue: random


More general casesMore general casesMultidimensional case

without regularization with regularization


HigherHigher--order CMACorder CMACFirst order Bspline basis function

C=8

02

46

8

02

46

80

0.2

0.4

0.6

0.8

1

continuous input space discrete input space


Relation to SVM with Relation to SVM with splinespline Kernel Kernel functionfunction

The response of an SVM with linear Bspline kernelC=ldtrain C≠ ldtrain



The effect of the training points in their neighborhood can be described by a linearly decreasing proximity function with 2C support. Binary CMAC corresponds to an SVM with linear Bspline kernel function

BCMAC: primal space SVM with linear Bspline kernel: dual space



SVM (with Bspline

kernel)

CMAC(binary or higher-order)

No. of basis function

Support vectors (sparse

approximation)

Training rule Quadratic programming

LMS

Learning speed,on-line adaptation

Low?

Highpossible

Generalization General results Special results

Embedded system ? Easy-to-implement

)1(1

11

−+= ∏

=−

N

iiN CR

CM


ConclusionsConclusions• Measurement as an autonomous step of modeling• Sensors + signal processing• Different tasks require nonlinear mapping• Mapping is learned from observations• The importance of embedded system• Network architecture suitable for embedded systems: CMAC • Properties of CMAC from the point of view of:

implementationmodeling capabilitygeneralization capability

• Detailed mathematical analysis of the performance • Modified learning algorithm to improve generalization

capability.


ReferencesReferencesB. Pataki – G. Horváth – Gy. Strausz – Zs. Talata "Inverse Neural Modeling of a Linz-Donawitz Steel Converter" e & i

Elektrotechnik und Informationstechnik, Vol. 117. No. 1. 2000. pp. 13-17.J. Sjöberg, Q. Zhang, L. Ljung, A. Benveniste, B. Delyon, P.-Y. Glorennec, H. Hjalmarsson, and A. Juditsky: "Non-linear

Black-box Modeling in System Identification: a Unified Overview", Automatica, 31:1691-1724, 1995. G. Horváth: “Neural Networks in System Identification” Chapter 3. In: V. Piuri (Ed.) Neural Networks in Measurement Systems

NATO ASI NIMIA, Crema, Italy 2001. Oct. IOS Press, in print.J. Van Dam: “Environment Modelling of Mobile robots. Neural Learning for Sensor Fusion” Ph.D thesis Universiteit van

Amsterdam, 1998.S. Thrun: “Learning Maps for Indoor Robot Navigation” Artificial Intelligence 1998. J. S. Albus: "A New Approach to Manipulator Control: The Cerebellar Model Articulation Controller (CMAC)", Transaction of

the ASME, Sep. 1975. pp. 220-227. M. Brown – C. J. Harris – P. Parks: "The Interpolation Capability of the Binary CMAC", Neural Networks, Vol. 6. No. 3. 1993.

pp. 429-440. G. Horváth – F. Deák: "Hardware Implementation of Neural Networks Using FPGA Elements", Proc. of The International

Conference on Signal Processing Application and Technology, Santa Clara Vol. II. pp. 60-65. 1993. T. Szabó – G. Horváth: "CMAC and its Extensions for Efficient System Modelling", International Journal of Applied

Mathematics and Computer Science, 1999. Vol. 9. No. 3. pp. 571-598. T. Szabó, G. Horváth: "Improving the generalization capability of the binary CMAC", Proc. of the International Joint

Conference on Neural Networks, IJCNN'2000. Como Italy. Vol. III. pp.85-90. G. Horváth, T. Szabó: “CMAC Neural Network with Improved Generalization Capability for System Modeling” Proc. Of the

IEEE Conference on Instrumentation and Measurement, Anchorage, AK. 2002. Vol. 2. pp. 1603-1608.S. Gunn: “Support Vector Machines for Classification and Regression” ISIS Technical Report and MATLAB toolbox, University

of Southampton, 1998.V. Vapnik: “The Nature of Statistical Learning Theory”, Springer, 1995.

neural networks in measurement systemshome.mit.bme.hu/~horvath/ltp/ltp_slides.pdf19. 07. 2002. nato...

Documents