Biology Inspired Approximate Data Representation for

Signal Processing, Soft Computing and Control Applications

Emil M. Petriu, Dr. Eng., FIEEESchool of Information Technology and Engineering

University of OttawaOttawa, ON., K1N 6N5 Canada

http://www.site.uottawa.ca/~petriu

WISP’2007 - IEEE Int. Symposium on Intelligent Signal Processing, Alcalá de Henares, Spain, 3-5 Oct.2007

Abstract

This paper reviews basics, similarities, and applications of two biology inspired approximate data representation modalities: stochastic data representation and fuzzy linguistic variables.

Stochastic Data Representation

Biological Neurons

Incoming signals to a dendrite may be inhibitory or excitatory.The strength of any input signal is determined by the strength ofits synaptic connection. A neuron sends an impulse down its axonif excitation exceeds inhibition by a critical amount (threshold/offset/bias) within a time window (period of latent summation).

Biological neurons are rather slow (10-3 s) when compared with the modern electronic circuits. ==> The brain is faster than an electronic computer because of its massively parallel structure. The brain has approximately 1011 highly connected neurons (approx. 104 connections per neuron).

Dendrites carry electrical signals in into the neuron body. The neuron body integrates and thresholds the incoming signals.The axon is a single long nerve fiber that carries the signal fromthe neuron body to other neurons. A synapse is the connection between dendrites of two neurons.

Memories are formed by the modification of the synaptic strengths which can change during the entire life of the neural systems.

Body

Axon

Dendrites

Synapse

Looking for a model to prove that algebraic operations with analog variables can be performed by logic gates, von Neuman advanced in 1956 the idea of representing analog variables by the mean rate of random-pulse streams [J. von Neuman, “Probabilistic logics and the synthesis of reliable organisms from unreliable components,” in Automata Studies, (C.E. Shannon, Ed.), Princeton, NJ, Princeton University Press, 1956].

The “random-pulse machine” concept, [S.T. Ribeiro, “Random-pulse machines,” IEEE Trans. Electron. Comp., vol. EC-16, no. 3, pp. 261-276,1967], a.k.a. "noise computer“, "stochastic computing“, “dithering” deals with analog variables represented by the mean rate of random-pulse streams allowing to use digital circuits to perform arithmetic operations. This concept presents a good tradeoff between the electronic circuit complexity and the computational accuracy. The resulting neural network architecture has a high packing density and is well suited for very large scale integration (VLSI).

FS+VFS-V

FS FSXQ

p.d.f. of VR

1

2 FS.

-FS

+FS

1

V

X

0-1

VRQ1-BIT QUANTIZER

X-FS

+FS

XQ

X

0

1

-1

XQ

CLOCK CLK

VRP

ANALOG RANDOM SIGNAL GENERATOR

-FS +FS0R

p(R)1

2 FS

++

VRV

R

Analog/Random-Pulse Conversion

The deterministic component of the random-pulse sequence, conveniently unbiased and rescaled for this purpose to take values +1 and -1 (instead of 1 and respectively 0) , can be calculated as a statistical estimation from the quantization diagram:

E[VRP] = (+1) .p[VR>=0] + (-1) .p[VR<0] = p(VRP) - p(VRP’) = (FS+V)/(2.FS) - (FS-V)/(2.FS )= V/FS;

This finally gives the deterministic analog value V associated with the binary VRP sequence:

V = [p(VRP) - p(VRP')] . FS ; where the apostrophe ( ' ) denotes a logical inversion.

Random-Pulse/Digital Conversion

V*

N-bit SHIFTREGISTER

Up

(n+1) bit UP/DOWNCOUNTER

Down

VRP

Clock

V*

N-bit SHIFTREGISTER

N-bit SHIFTREGISTER

Up

(n+1) bit UP/DOWNCOUNTER

Down

VRP

Clock

Random-pulse/digital converter using the moving average algorithm .

;**

);(11*

01

1

11

NVRPVRPVV

VRPVRPN

VRPN

V

NNN

N

N

ii

N

iiN

32 266 5003.2

1

1.2

x2is

x2ditis

x2RQis4

2

dZis

dHis

dLis

MAVx2RQis

is

(1) Analog Input (2) Analog Input + Dither Noise

(4) Estimation of the Analog Input recovered by the moving-average “Random-Pulse/Digital Conversion”

(3) Random-Pulse Sequence produced by the “Analog/Random-Pulse Conversion”

Sample Index32 266 5003.2

1

1.2

x2is

x2ditis

x2RQis4

2

dZis

dHis

dLis

MAVx2RQis

is

(1) Analog Input (2) Analog Input + Dither Noise

(4) Estimation of the Analog Input recovered by the moving-average “Random-Pulse/Digital Conversion”

(3) Random-Pulse Sequence produced by the “Analog/Random-Pulse Conversion”

Sample Index

Analog/random-pulse and random-pulse/digital conversion of a “step” input signal

1 OUT_OF mDEMULTIPLEXER

RANDOM NUMBERGENERATOR

S1SjSm

CLK

Y = (X1+...+Xm)/m

y

x1

xj

xm

X1

Xj

Xm

Random-Pulse Addition

32 266 5008.2

3.5

1.2

x1is

x1RQis

41.5

MAVx1RQis

dZ1is

x2is 3

x2RQis

44.5

MAVx2RQis 3

dZ2is

x1is x2is 6

SUMRQXis

47.5

MAVSUMRQXis 6

dZSis

dHis

dLis

is

Random-pulse addition

32 144 2569.2

4

1.2

x1is

x1ditis

x1RQis

42

dZis

dHis

dLis

w1is 3.5

dZis 3.5

W1is4

5

x1W1RQis4

6.5

MAVx1W1RQ is 8

dZis 8

is

Random-pulse multiplication

Generalized b-bit analog/stochastic-data conversion and its quantization characteristics

VR

V

RVRQ

CLOCKCLK

VRP

b-BIT

QUANTIZER

X XQ

ANALOG RANDOM

SIGNAL

GENERATOR

-D/2 0

R

p(R)

1/D

+D/2

+

+

.(k+0.5) D(k-0.5) D.

XQ

X

k

k+1

k-1

0

b D.

1/ Dp.d.f.

of VR

D /2D /2

b D. (1- b ) D.

.V= (k- b ) D

k D.

Stochastic Data Representation

The ideal estimation over an infinite number of samples of the stochastic data sequence VRD is:

E[VRD] = (k-1). p[(k-1.5) ≤ VR < (k-0.5) ] + k . p[(k-0.5) ≤ VR< (k+0.5)] = (k-1) . + k . (1-) = k -

The estimation accuracy of the recovered value for V depends on thequantization resolution, the finite number of samples that are actually averaged, and on the statistical properties of the analog dither.

Quantization levels

Relative mean square error

2 72.23

3 5.75

4 2.75

... ...

8 1.23

... ...

analog 1

0 10 20 30 40 50 60 700

0.02

0.04

0.06

0.08

0.1

0.12

0.14

0.16

0.18

Moving average window size

Mea

n sq

uare

err

or

1-Bit

2-Bit

Mean square errors function of the moving average window size

RANDOMNUMBERGENERATOR

1-OUT OF-mDEMULTIPLEXER

...

...

CLK

... ... Sm

S1

Si

mX

1X

iX Z = (X +...+X )/mmi

b

b

b

b

b

b

b

Stochastic-data addition.

2-bit stochastic-data multiplier.

YX

101

-110

00010-1

-110

101

000011

000

000

000000

100100

-110

XLSB

XMSB Z

LSB

ZMSB

YLSB

YLSB

0 100 200 300 400 500-2

-1

0

1

2multiplication

0 100 200 300 400 500-2

-1

0

1

2

weightinput

product

Example of 2-bit stochastic-data multiplication.

Correlator Architectures Using Stochastic Data

Representation

V1 VRD1(n.t)

DELAY LINE

MULTIPLIER

STOCHASTIC DATA / DIGITAL

V2

ANALOG / STOCHASTIC DATA

VRD2((n-r).t ) VRD2(n.t )

VRD1(n.t) . VRD2((n-r).t)

CORv1v2 (r.t)

ANALOG / STOCHASTIC DATA

))((2)(11)(1

021 tknVtnV

NtkCOR

N

nVV

Parallel architecture of a 4-point correlator using random-pulse data representation.

for k = 0,1,2,3. ;))(()(1)(1

0tknytnx

NtkCOR

N

nxy

Each calculated correlation point has a 8-bit resolution. In order to reduce the number of interface lines, the 8-bit wide outputs of thefour random-pulse/digital converters are multiplexed. When more modules are connected in larger structures the 1-bit XRQ input lines of all modules are connected together while the 1-bit YRQ output of each module is connected to the 1-bit YRQ input of the next module creating a longer delay line.

Autocorrelation function of a sinusoid calculated by a 32-point parallel random-pulse correlator with a 8-bit/point resolution.

Correlation function of two sinusoids with the same frequency but different amplitudes and phases, calculated by a 32-point parallel random-pulse

correlator with a 8-bit/point resolution.

Correlation function between a sinusoid and another sinusoid of the same frequency but corrupted by white noise, calculated by a32-point parallel random-pulse correlator with 8-bit/point resolution.

Neural Network Architectures Using

Stochastic Data Representation

F

Y = F [ w X ]j=1

m.

j iij

SYN

APS

E

SYN

APS

E

SYN

APS

E

. . .. . . X mX 1 X i

w mjw ijw 1j

Neuron structure

SYNAPSE ADDRESS DECODER

S mpS ijS 11

N-STAGE DELAY LINE

......

wij

DT = w Xij ij.

i

SYNAPSE

MODE

DATIN SYNADD X i

MULTIPLICATION

b

b

b

b

b

... ...

RANDOM-DATA ADDER

DT mj DT ij DT 1j

RANDOM-DATA / DIGITAL

CLK

DIGITAL / RANDOM-DATA

ACTIVATION FUNCTION F

Y = F [ w X ] j j=1

m .

i ij

Multi-bit stochastic-data implementation of a neuron body.

Multi-bit stochastic-data implementation of a synapse

Auto-associative memory NN architecture

P1, t1 P2, t2 P3, t3

Training set

30

P

30x1

30x30

n

30x1

a

30x1W

)*hardlim( PWa

Recovery of 30% occluded patterns

Fuzzy Logic Control

Pioneered by Zadeh in the mid ‘60s fuzzy logic provides the formalism for modeling the approximate reasoning mechanisms specific to the human brain.

“In more specific terms, what is central about fuzzy logic is that, unlike classical logical systems, it aims at modeling the imprecise modes of reasoning that play an essential role in the remarkable human ability to make rational decisions in an environment of uncertainty and imprecision. This ability depends, in turn, on our ability to infer an approximate answer to a question based on a store of knowledge that is inexact, incomplete, or not totally reliable.” [ “Fuzzy Logic,” IEEE Computer Magazine, April 1988, pp. 83-93: ]

Fuzzy Logic

Fuzzy Logic Control

ANALOG (CRISP) -TO-FUZZY INTERFACE FUZZIFICATION

FUZZY-TO- ANALOG (CRISP) INTERFACE DEFUZZIFICATION

SENSORS ACTUATORS

INFERENCE MECHANISM (RULE EVALUATION)

FUZZY RULE BASE

PROCESS

The basic idea of “fuzzy logic control” (FLC) was suggested by L.A. Zadeh, “A rationale for fuzzy control,” J. Dynamic Syst. Meas. Control, vol.94, series G, pp.3-4,1972.

FLC provides a non analytic alternative to the classical analytic control theory. ==> “But what is striking is that its most important and visible application today is in a realm not anticipated when fuzzy logic was conceived, namely, the realm of fuzzy-logic-based process control,” [L.A. Zadeh, “Fuzzy logic,” IEEE Computer Mag., pp. 83-93, Apr. 1988].

Early FLCs were reported by Mamdani and Assilian in 1974, and Sugeno in 1985.

INPUT

OUTPUT

x*

y*

Classical control systems are based on a detailed I/O function OUTPUT= F (INPUT) mapping each high-resolution

quantization interval of the input domain into a high-resolution quantization interval of the output domain

Fuzzy control is based on a much simpler functional description of the desired I/O behavior mapping each low-resolution quantization interval of the input domain into a low-resolution quantization interval of the output domain

INPUT

OUTPUT

y*

x*

Defuzzific

ation

Fuzzification

Membership functions and the quantization characteristics for a 3-set (N, Z, and P) fuzzy partition of the domain where the analog variable x is defined. XFQ are the crisp analog values recovered after a defuzzification of the fuzzy converted value x. It also shows the truncated information XQ recovered from an A/D converter with 3 quantization levels defined over the same domain for x.

The key benefit of FLC is that the desired system behavior can be described with simple “if-then” relations based on very low-resolution models able to incorporate empirical engineering knowledge. FLCs have found many practical applications in the context of complex ill-defined processes that can be controlled by skilled human operators: water quality control, automatic train operation control, elevator control, etc.,

Fuzzy Controller for Truck and Trailer Docking

DOCK

d

NL NM NS PS PM PLZE

AB(

[deg.] -110 -95 -35 -20 -10 0 10 20 35 95 110

NL NM NS PS PM PLZE

GAMMA

[deg.] -85 -55 -30 -15 -10 0 10 15 30 55 85

NEAR LIMITFAR

DIST ( d )

[m] 0.05 0.1 0.75 0.90

INPUT MEMBERSHIP FUNCTIONS

SPEED

[ % ] 16 24 30

STEER (

[deg.] -48 -38 -20 0 20 38 48

LH LM LS ZE RS RM RH

SLOW MED FAST

REV FWD

DIRN

[arbitrary] - +

OUTPUT MEMBERSHIP FUNCTIONS

“STEER / DIRN” RULE BASE

LM/F LS/F RS/F RM/F

LM ZE RM

RM

RH

RH

RHRM RH

RH/F

RH/F

RH/F

RH/F

RH/F

RH/FRH/FRH/FRM/F

LS/R RS/R

RS/R

RS/R RM/R

ZE/R

ZE PS PM PL

LH/F LH/F LH/F

LH

LH

LH

LM

LM

LH

ZE

LH/F

LH/F

LH/F

LH/F

LH/F

LM/F RS/FLS/F

LM/R

LS/R

LS/R

NM

NL

NS

ZE

PS

PM

PL

NL NM NS

GAMMA ()

AB( )

F-R F-R F-R F-R F-R

F-R

F-R

F-R

F-R F-R F-R F-R F-R

F-R

F-R

F-R

There is a hysteresis ring around the center of the rule base table for the DIRN output. This means that when the vehicle reaches a state within this ring, it will continue to drive in the same direction, F (forward) or R (reverse), as it did in the previous state outside this ring.

A hysteresis was purposefully introduced to increase the robustness of the FLC.

DEFFUZIFICATION

The crisp value of the steering angle is obtained by the modified “centroidal” deffuzification (Mamdani inference):

207

47

20

63

63

0

I/O characteristic of th Fuzzy Logic Controller for truck and trailer docking.

LH . LH + LM . LM + LS. LS + ZE . ZE

+ RS . RS + RM . RM + RH . RH ) / (LH + LM + LS + ZE + RS + RM + RL)

There is tenet of common wisdom that FLCs are meant to successfully deal with uncertain data. According to this, FLCs are supposed to make do with “uncertain” data coming from (cheap) low-resolution and imprecise sensors. However, experiments show that the low resolution of the sensor data results in rough quantization of of the controller's I/O characteristic:

207

47

20

63

63

0

0

16

16

0

207

47

1disp

4-bit sensors

7-bit sensors

I/O characteristics of the FLC for truck & trailer docking for 4-bit sensor data () and 7-bit sensor data.

STOCHASTIC-DATA FUZZY LOGIC CONTROLLERS

d

Loading Dock

( , )x y Front Wheel

Back Wheel

(0,0)x

y

The truck backing-up problem

Design a Fuzzy Logic Controller (FLC) able to back up a truck into a docking station from any initial position that has enough clearance from the docking station.

0 4 5 15 20-4-5-15-20-50 50x-position

900 100080060030000-90 27001200 1500 1800

truck angle

00-250-350-450 250 350 450

LE LC CE RC RI

RB RU RV VE LV LU LB

NL NM NS ZE PS PM PL

steering angle

0.0

1.0

1.0

0.0

Membership functions for the truck backer-upper FLC

PS

NS

NM

NM

NL

NL

NL

PM PM

PM

PL PL

NL

NL

NM

NM

NS

PS

NM

NM

NS

PS

NM

NS

PS

PM

PM

PL

NS

PS

PM

PM

PL

PL

RL

RU

RV

VE

LV

LU

LL

LE LC CE RC RI

x

ZE

1 2 3 4 5

6 7

18

31 35343332

30

The FLC is based on the Sugeno-style fuzzy inference.

The fuzzy rule base consists of 35 rules.

0 10 20 30 40 50 60 70-40

-30

-20

-10

0

10

20

30

Time (s)

[deg]

0 10 20 30 40 50 60-50

-40

-30

-20

-10

0

10

20

30

40

Time (s)

[deg]

Time diagram of digital FLC's output during a docking experiment when the input variables, j and x are analog and respectively quantizied with a 4-bit bit resolution

A/D

A/D

Dither

Dither

Low-PassFilter

Low-PassFilter

DigitalFLC

Analog Input

Analog Input

DitheredAnalog Input

High ResolutionDigital Outputs

DitheredDigital Input

DitheredDigital Input

High ResolutionDigital Input

High ResolutionDigital Input

DitheredAnalog Input

FLC architecture using 4-bit stochastic data representation with low-pass filters placed immediately after the input A/D converters

A/D

A/D

Dither

Dither

Low-PassFilter

Low-PassFilter

DigitalFLC

AnalogInput

AnalogInput

DitheredAnalog Input

High ResolutionDigital Output

Low-ResolutionDithered DigitalInput

High ResolutionDigital Output

Low-ResolutionDithered DigitalInput

DitheredAnalog Input

It offers a better performance than the previous one because a final low-pass filter can also smooth the non-linearity caused by the min-max composition rules of the FLC.

FLC architecture using 4-bit stochastic data representation with low-pass filters placed at the FLC’s outputs.

Time diagram of the stochastic FLC's output during a docking experiment when 4-bit A/D converters are used to quantize the dithered inputs and the

low-pass filter is placed at the FLC's output

0 10 20 30 40 50 60 70-50

-40

-30

-20

-10

0

10

20

30

Q [deg]

Time (s)

-50 500

10

20

30

40

50

X

Y

(a)

(b)

(c)

[dock]

initial position

(-30,25)

0

Truck trails for different FLC architectures: (a) analog ; (b) digital without dithering; (c) stochastic data representation with uniform

dithering and 20-unit moving average filter

Stochastic FLCDigital FLC

Analog FLC

Conclusions

Due to its relatively low hardware complexity and high internal noise immunity, the von Neumann stochastic data representation represents an attractive alternative to the analog and high resolution digital data processing techniques for many statistical signal processing and soft computing applications.

Because of the smooth linear transitions in the membership of overlapping fuzzy sets, the fuzzy partition of the analog FLC inputs will not introduce any quantization noise. However, digital FLCs cannot make do with low-resolution data. It is shown that dithering can offer a solution to significantly improve the resolution of the reduced word-length digital FLCs.

• E. Pop, E. Petriu, “Influence of Reference Domain Instability Upon the Precision of Random Reference Quantizer with Uniformly Distributed Auxiliary Source,” Signal Processing, North Holland, Vol. 5, 1983, pp. 87-96.• G. Eatherley, Autonomous Vehicle Docking Using Fuzzy Logic, M.A.Sc. Thesis, 1994. • G. Eatherley, E.M. Petriu, "A Fuzzy Controller for Vehicle Rendezvous and Docking," IEEE Trans. Instr. Meas., Vol. 44, No. 3, pp. 810-814, 1995. • E.M. Petriu, G. Eatherley, “Fuzzy Systems in Instrumentation: Fuzzy Control,” Proc. IMTC/95, IEEE Instr. Meas. Technol. Conf., pp.1-5, Waltham, MA, 1995. • E. Petriu, K. Watanabe, T. Yeap, “Applications of Random-Pulse Machine Concept to Neural Network Design,” IEEE Tr.. Instr. Meas., Vol. 45, No.2, 1996, pp. 665-669.• L. Zhao, Random Pulse Artificial Neural Network Architecture, M.A.Sc. Thesis, University of Ottawa, Canada, 1998 • J. Mao, Reduction of the Quantization Error in Fuzzy Logic Controllers by Dithering, M.A.Sc. Thesis, University of Ottawa, Canada, 1998.• E.M. Petriu, J. Mao, “Fuzzy Sensing and Control for a Truck,” Proc. VIMS-2000, IEEE Workshop on Virtual and Intelligent Measurement Systems, Annapolis, MD, April 2000, pp. 27-32.• E. M. Petriu, L. Zhao, S.R. Das, V.Z. Groza, A. Cornell, “Instrumentation Applications of Multibit Random-Data Representation,” IEEE Tr. Instr. Meas., Vol. 52, No. 1, 2003, pp. 175- 181.• M. Dostaler, Multi-Level Random Data Based Correlator Model, M.A.Sc. Thesis, 2005.

AcknowledgementThis paper represents a synthesis of the work carried out by the author and his collaborators published over the years in conference proceedings and journals:

Acknowledgement

The work reported in this paper was funded in part by Communications and Information Technology Ontario (CITO) and the Natural Sciences and Engineering Research Council (NSERC) of Canada.

Thank you!Thank you!