Analog Circuits for Self-organizing Neural Networks Based on Mutual

Information

Janusz Starzyk and Jing Liang

 

School of Electrical Engineering and Computer Science

Ohio University

Athens, OH 45701 USA

Index

Introduction

System description

Analog counter

Mutual information principle

Conclusion

Index

IntroductionIntroduction

System description

Analog counter

Mutual information principle

Conclusion

Introduction

Index

Introduction

System descriptionSystem description

Analog counter

Mutual information principle

Conclusion

System DescriptionSystem Description

Fig. 1 System Description

MbyNMUX

+/-

EBE EBE EBE

ThresholdValue

MbyNMUX

+/-

EBE EBE EBE

ThresholdValue

MbyNMUX

+/-

EBE EBE EBE

ThresholdValue

MbyNMUX

+/-

EBE EBE EBE

ThresholdValue

MbyNMUX

+/-

EBE EBE EBE

ThresholdValue

MbyNMUX

+/-

EBE EBE EBE

ThresholdValue

MbyNMUX

+/-

EBE EBE EBE

ThresholdValue

MbyNMUX

+/-

EBE EBE EBE

ThresholdValue

EBE

ThresholdValue

Index

IntroductionSystem descriptionAnalog counterAnalog counter Circuit structureCircuit structure Analysis of the operationAnalysis of the operation Simulation and resultsSimulation and results

Mutual information principleConclusion

Circuit StructureCircuit Structure

C1

C2

M1

M2

M3

M4

M5

Clk_in

V_ref

Clr

V_out

Vdd

Fig. 2 The analog counter

Analysis of the OperationAnalysis of the Operation

34

33

44MM I

LW

LWI

dt

dVCII C

Mo2

4 2

)( 32

1

4

3

3

42 tprefddC VVV

C

C

L

L

W

WV

C1

C2

M1

M2

M3

M4

M5

Clk_in

V_ref

Clr

V_out

Vdd

Fig. 2 The analog counter

dt

dVCI C

C1

1 1

Simulation and ResultsSimulation and Results

Fig. 3. Simulation result of analog counter’s linearity and dynamic range.

Index

IntroductionSystem descriptionAnalog counterMutual information principleMutual information principle Mutual informationMutual information Approximations of mutual information functionApproximations of mutual information function Test data preparationTest data preparation Simulation resultsSimulation results

Conclusion

The entropy based mutual information is defined as follows:

(4.1)

Where

(4.2)And

(4.3)

Mutual InformationMutual Information

1

0

1

0 1

)log()log(a

aaa

n

cacac ppppE

c

cn

ccc ppE

1max )log(

max

1E

EI

Any classification problem can be decomposed into a number of classification problems of two classes:

(4.4)

(4.5)

(4.6)

Approximations of Mutual Approximations of Mutual Information FunctionInformation Function

1

0

1,1,1,1, )1log(1)log(a a

ca

a

ca

a

ca

a

ca

p

p

p

p

p

p

p

pE

)1log()1(log 0000max cccc PPPPE

)1log()1(log)( pppppI

Approximations of Mutual Approximations of Mutual Information FunctionInformation Function

Different approximations can be used in order to implement it in analog circuits. Two different approximation are used here. One is the linear approximation:

(4.7)

The other one is quadratic approximation:

(4.8)Their responses with respect to class

probability and its compare with original function is shown in Fig. 4.

Approximations of Mutual Approximations of Mutual Information FunctionInformation Function

)5.021(2log)( ppI

)1()2(log4)( pppI

Test Data PreparationTest Data Preparation

In order to compare the final decisions made by using the original mutual information function and its approximations, we randomly generated test data with normal distributions.

The mean value and covariance matrix are different for each class date set. To better shown the distribution of data, these data generated with two dimensions, although we only use one of the dimensions. Each class can have different ellipse shape with major axis in different directions.

 

Simulation ResultsSimulation Results

Simulation ResultsSimulation Results

Simulation ResultsSimulation Results

Index

Introduction

System description

Analog counter

Mutual information principle

ConclusionConclusion

ConclusionConclusion

We presented a new self-organizing neural network concept based on mutual information. This network is to be implemented in analog circuits in order to get better performance. To implement the most important circuit in this network, the EBE, an analog counter is presented as a building block to perform statistical calculation and simulated. Satisfying results are achieved.

Furthermore, different approximation approaches are used to implement the mutual information calculation; the simulation result shows that it is possible to implement it by linear circuits or quadratic circuits.