a pulsed neural network capable of universal approximation

7
308 IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 3, NO. 2, MARCH 1992 A Pulsed Neural Network Capable of Universal Approximation Neil E. Cotter and Omar N. Mian Abstract- This paper describes a pulsed network version of the cerebellar model articulation controller (CMAC), popularized by Albus. The network produces output pulses whose times of occurrence are a function of input pulse intervals. Within limits imposed by causality conditions, this function can approximate any bounded measurable function on a compact domain. Simu- lation results demonstrate the viability of training the network with a least mean squares algorithm. I. INTRODUCTION N biological nervous systems, information typically is con- I veyed by the timing of pulses or “action potentials,” whose shapes are relatively invariant. For some time, researchers have debated the importance of single intervals between action potentials. Modelers of neural networks routinely average the intervals to obtain variables representing pulse rates of neurons [l], [2], but some researchers have suggested that detailed timing information is important in biological nervous sys- tems [3]. In this paper we present a network that outputs pulses and accepts pulses as inputs. Because it is an extension of the cerebellar model articulation controller (CMAC), developed by Albus [4], we call this neural network a pulsed CMAC (PCMAC). The timing of output pulses from the PCMAC is a function of timing intervals between incoming pulses. For a sufficiently large PCMAC and a compact domain, this function can approximate any bounded measurable function that satis- fies unavoidable causality conditions. Thus, the network we present here is capable of a form of universal approximation and serves as an existence proof that pulse intervals employed by biological systems are suitable variables for representing information in a network. Our approach contrasts with past research that has em- phasized three prominent perspectives on pulsed networks: synchrony [5] - [9], detailed physiological models of neurons [lo], [ 111, and abstract mathematical methods of processing with pulses [12]-[15]. Regarding synchrony, we observe that phase locking entails a reduction of information. If neurons are synchronized, the same information is represented a number of times and degrees of freedom are lost. Such mechanisms may be useful for pattern recognition or reliable communications in a noisy environment [16], and synchrony may be common in biological information processing systems. Our interest, however, is in pulse processing with the fewest possible timing constraints. We also desire uniformity of input and output coding. The PCMAC encodes both inputs and outputs as pulse Manuscript received February 14, 1991; revised October 9, 1991. The authors are with the Electrical Engineering Department, University of Utah, Salt Lake City, UT 84112. IEEE Log Number 9105240. Synaptic Response lnDul Waveform Svnavtic Pulses Weights Threshold Detector O A k Pulse Output Generator Pulse (.rigger 1 --+ t Fig. 1. Model of pulsing neuron. (Synapse response reset mechanism not shown.) intervals. Hence, we can readily cascade PCMAC’s to form larger systems. Regarding physiological models, we observe that overly detailed models may obscure useful mathematical principles for designing pulsed networks. Although our network is based on physiology, we include only salient features of real neurons. An important ingredient of our model is the inclusion of synapse response waveforms. However, we treat these as gen- eral mathematical functions rather than attempting to model their physiological origins. This approach avoids massive calculations and allows us to focus on mathematical analyses of pulse-position processing. Regarding abstract mathematical methods of processing with pulses, we observe that our network retains more physi- ological detail than abstract models based on pulse tallies [12] or integrate-and-fire mechanisms [ 171. Consequently, we gain realism and computational power while avoiding excessive detail. The resulting PCMAC is a compromise that is complex enough to be capable of universal approximation but simple enough to be analyzed. 11. PULSE PROCESSING NEURON A. Time Coordinates Fig. 1 illustrates a two-input example of our model neuron. Most of our discussion in this paper focuses on a two- input neuron with a triangular synapse response waveform. Nevertheless, the conclusions and mathematical derivations extend to neurons with any number of input lines. In this section, results also generalize to arbitrary synapse response waveforms. We consider a neuron with N + 1 inputs, and we assume a time window in which each input is pulsed only once. The pulse on input i occurs at time t,. Thus, the pulse times, to :... t,v, are ordered by input line number rather than the temporal order of occurrence. In response to the input pulses, 1045-9227/92$03.00 0 1992 IEEE

Upload: on

Post on 11-Mar-2017

213 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: A pulsed neural network capable of universal approximation

308 IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 3, NO. 2, MARCH 1992

A Pulsed Neural Network Capable of Universal Approximation Neil E. Cotter and Omar N. Mian

Abstract- This paper describes a pulsed network version of the cerebellar model articulation controller (CMAC), popularized by Albus. The network produces output pulses whose times of occurrence are a function of input pulse intervals. Within limits imposed by causality conditions, this function can approximate any bounded measurable function on a compact domain. Simu- lation results demonstrate the viability of training the network with a least mean squares algorithm.

I. INTRODUCTION

N biological nervous systems, information typically is con- I veyed by the timing of pulses or “action potentials,” whose shapes are relatively invariant. For some time, researchers have debated the importance of single intervals between action potentials. Modelers of neural networks routinely average the intervals to obtain variables representing pulse rates of neurons [l] , [2], but some researchers have suggested that detailed timing information is important in biological nervous sys- tems [3].

In this paper we present a network that outputs pulses and accepts pulses as inputs. Because it is an extension of the cerebellar model articulation controller (CMAC), developed by Albus [4], we call this neural network a pulsed CMAC (PCMAC). The timing of output pulses from the PCMAC is a function of timing intervals between incoming pulses. For a sufficiently large PCMAC and a compact domain, this function can approximate any bounded measurable function that satis- fies unavoidable causality conditions. Thus, the network we present here is capable of a form of universal approximation and serves as an existence proof that pulse intervals employed by biological systems are suitable variables for representing information in a network.

Our approach contrasts with past research that has em- phasized three prominent perspectives on pulsed networks: synchrony [ 5 ] - [9], detailed physiological models of neurons [lo], [ 111, and abstract mathematical methods of processing with pulses [12]-[15]. Regarding synchrony, we observe that phase locking entails a reduction of information. If neurons are synchronized, the same information is represented a number of times and degrees of freedom are lost. Such mechanisms may be useful for pattern recognition or reliable communications in a noisy environment [16], and synchrony may be common in biological information processing systems. Our interest, however, is in pulse processing with the fewest possible timing constraints. We also desire uniformity of input and output coding. The PCMAC encodes both inputs and outputs as pulse

Manuscript received February 14, 1991; revised October 9, 1991. The authors are with the Electrical Engineering Department, University of

Utah, Salt Lake City, UT 84112. IEEE Log Number 9105240.

Synaptic Response

lnDul Waveform Svnavtic Pulses Weights

Threshold Detector

O A k Pulse Output

Generator Pulse

(.rigger 1 --+ t

Fig. 1. Model of pulsing neuron. (Synapse response reset mechanism not shown.)

intervals. Hence, we can readily cascade PCMAC’s to form larger systems.

Regarding physiological models, we observe that overly detailed models may obscure useful mathematical principles for designing pulsed networks. Although our network is based on physiology, we include only salient features of real neurons. An important ingredient of our model is the inclusion of synapse response waveforms. However, we treat these as gen- eral mathematical functions rather than attempting to model their physiological origins. This approach avoids massive calculations and allows us to focus on mathematical analyses of pulse-position processing.

Regarding abstract mathematical methods of processing with pulses, we observe that our network retains more physi- ological detail than abstract models based on pulse tallies [12] or integrate-and-fire mechanisms [ 171. Consequently, we gain realism and computational power while avoiding excessive detail. The resulting PCMAC is a compromise that is complex enough to be capable of universal approximation but simple enough to be analyzed.

11. PULSE PROCESSING NEURON

A . Time Coordinates

Fig. 1 illustrates a two-input example of our model neuron. Most of our discussion in this paper focuses on a two- input neuron with a triangular synapse response waveform. Nevertheless, the conclusions and mathematical derivations extend to neurons with any number of input lines. In this section, results also generalize to arbitrary synapse response waveforms.

We consider a neuron with N + 1 inputs, and we assume a time window in which each input is pulsed only once. The pulse on input i occurs at time t,. Thus, the pulse times, to :... t,v, are ordered by input line number rather than the temporal order of occurrence. In response to the input pulses,

1045-9227/92$03.00 0 1992 IEEE

Page 2: A pulsed neural network capable of universal approximation

IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 3, NO. 2, MARCH 1992 309

Synaptic Response

t t 0 + 2

Fig. 2. Summation of synaptic responses to pulses at times t o and tl causes neuron to output a pulse at time T.

the neuron outputs a pulse at time r. Pulses may be absent from some inputs, and if input activity is insufficient the neuron may fail to produce an output pulse.

Because the neuron has no access to a universal time stan- dard, only the intervals between pulses, rather than absolute pulse times, are relevant to a mathematical description of the neuron’s behavior. For convenience, we choose to measure all intervals relative to the arrival time, t o , for a pulse on the first input to the neuron. The relative time, r - t o , of the output pulse is thus a function of one less argument than the number of inputs:

7- - t o = f ( t l - t o . ‘ ’ . . t,y - t o ) . (1)

Locally, as in a time window containing only a single pulse on each input, the domain of f is homeomorphic to a region of RN.

B. Model Neuron with Synapses

Our model neuron has two stages: synapses and pulse generator. To simplify descriptions, we assume that neuron input and output pulses are extremely narrow and serve only as trigger signals for synapse responses. Identical results are obtained for other pulse shapes so long as the resulting shapes of synaptic response waveforms described below are unaltered.

A synapse responds to an incoming pulse by producing a “synapse response waveform.” For the neuron shown in Fig. 1, this waveform has a triangular shape, s ( t ) , scaled by a synaptic weight, wi. Synaptic responses sum, as shown in Fig. 2. At the moment, 7, when the total summed response exceeds a threshold activation level, A, the neuron generates an output pulse. The output pulse time, r , is implicitly defined by

When the neuron generates a pulse, we extinguish all synapse response waveforms. Thus, a neuron remains inactive until a new set of pulses arrives at its inputs. This reset action

produces an artificial refractory period after firing and splits input pulses into groups that do not interfere with one another.

The triangular synapse response waveform, s ( t ) , of Fig. 2 is a simplification of an excitatory postsynaptic potential (EPSP) as observed, for example, in motor neurons responding to the neurotransmitter acetylcholine [18]. It has long been known that the EPSP is a summation of miniature end-plate potentials (MEPP’s) whose number depends on the amount of neurotransmitter present in the synaptic cleft [19], [20]. We assume that the EPSP scales linearly with the number of MEPP’s, and we identify the height of an EPSP as the synaptic weight, wi, for input i .

C. Output Pulse Time Function

Because the output pulse time r is embedded in (2), we cannot always solve for r explicitly. Nevertheless, we can calculate the derivative of 7- with respect to input pulse arrival time t k . We begin by differentiating both sides of (2) with respect to t k :

.v i)s(r - t L ) = 0. cwl 1=0 dtl, ( 3 )

Since 7- is a function of t k , we use the chain rule to evaluate the derivative of .s( ):

where d(7- - t i ) is defined as follows:

We observe that i) t , /dt , = 0 except when i = I ; , and we substitute (4) into (3) to obtain the final formula:

- (6) 37- - W k S ’ ( 7 - t k )

- i ) t k

W , d ( T - t L ) 2=0

Without loss of generality, we henceforth assume t o = 0, allowing us to use r in place of the time interval 7- - t o . In Fig. 3, we use (2 ) and (6) as aids in plotting 7- versus tl for a two-input neuron with A = 1.5, WO = 1, and triangular s ( t ) as in Fig. 2. The axis crossings in the plot were found by making drawings similar to Fig. 2, and checking the results with (2). The slopes of the lines follow from (6). For triangular synaptic responses, the derivatives inside the summation in (6) are f 1 or 0, simplifying calculations. For the derivative at the peak of the triangular response we use +1 for a negative perturbation in t k and -1 for a positive perturbation in t k . The features of the plot in Fig. 3 are characteristic of response delay functions:

1) Curves for smaller values of synaptic weight w1 vanish outside of finite intervals. These intervals get shorter as w1 gets smaller because the synaptic responses are smaller in amplitude. Smaller responses mean pulses must be nearly synchronous or they fail to generate activity exceeding threshold. The lines on the plot end where this failure occurs.

Page 3: A pulsed neural network capable of universal approximation

310 IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 3, NO. 2, MARCH 1992

7 to be reached sooner. We use this fact later on to derive a A learning algorithm. -_

111. TWO-LAYER ARCHITECTURE FOR COMPUTING ARBITRARY FUNCTIONS

A. Conventional CMAC

w1 = .75 Before introducing the PCMAC, we briefly review the conventional CMAC. In the CMAC, as described by Albus [4], each hidden-layer neuron (association cell) responds to some subset of input signals. These subsets, or regions, cover the input domain and overlap extensively. Thus, any input value

w = I 1

causes several neurons to respond. Preferably, the number of neurons responding is the same for any input.

Active hidden-layer neurons output a 1; inactive hidden- * t l

1 w, = 1.5 w = -

Plot of output pulse time versus input pulse interval and synaptic weight wl when 11’0 = 1 and threshold .4 = 1.5.

Curves for larger values of synaptic weight ?U] exist over infinite intervals. When 701 2 A = 1.5, the second input alone can cause the neuron to fire. If the input pulses are nonoverlapping this gives rise to a constant delay between tl and r. This delay appears as a line of unitary slope on the plot, meaning r is slaved to t l . Curves for smaller values of synaptic weight 1u1 some- times have negative slope. This means the neuron output pulse is advanced in time if the pulse on the tl input is delayed in time. This phenomenon arises if the t l synapse response is decaying at the time threshold is reached. Delaying the pulse on the t l input then pro- duces a larger total synaptic response, and threshold is reached earlier. The curve for synaptic weight 7111 equal to infinity has unity slope and passes through the origin. For infinitely large 1111 the synaptic response becomes a tall triangle whose leading edge crosses threshold immediately after the arrival of an input pulse. This gives rise to zero delay between t l and r. Hence, r is equal to t l .

D. Synaptic Weights and Output Pulse Time

Following a procedure similar to that used to derive (6), we can derive a formula indicating how r is affected by changes in synaptic weights:

~ (7) - - s (r - t k ) -

37 awl; A.

7uts’(r - t t ) a=O

In both (6) and (7), the denominator is the slope of the total synaptic response when it first crosses threshold. This slope must be positive. Otherwise, threshold previously would have been exceeded. In this paper we also restrict our attention to a strictly positive s ( t ) , as in Fig. 1. We conclude that the numerator of (7) is negative and that I?J/dWk is negative. Thus, increasing a synaptic weight always causes threshold

layer neurons output a 0. The hidden-layer outputs are mul- tiplied by synaptic weights of a summing neuron (response cell) in the output layer. The set of active synaptic weights is unique to the intersection of overlapping response regions for active hidden-layer neurons. For an input domain with two or more dimensions, the number of response region intersections may be much larger than the number of hidden-layer neurons. Thus, a small number of output-neuron synapses may account for a much larger number of possible network output values.

B. Pulsed CMAC

Fig. 4 illustrates a two-input PCMAC, and Table I lists the network parameters used for simulations. The hidden- layer neurons, numbered zero through eight, are pulse interval detectors, with fixed synaptic weight values of 1.0. From the response characteristics for the w1 = 1 curve in Fig. 2 it follows that these neurons fire only when they see input pulses arriving within f 0 . 5 time units of each other. Delay lines bring pulses that are widely separated in time at the network inputs into coincidence at neuron inputs, This allows each neuron to respond to a different range of network input pulse intervals. Thus, we create the pulsed equivalent of CMAC association cells. Given the delays of 0.125 time units, input pulses separated by less than 310.75 time units activate four neurons.

Pulse separations wider than f0 .75 time units cause fewer hidden-layer neurons, near either end of the delay lines, to be active. These “edge effects” are caused by intervals lying near the boundary of the input domain. As in the conventional CMAC, edge effects may be dealt with in several ways: by ignoring the infrequent errors they cause, by doubling up neurons at the ends of the delay line, by greatly increasing the number of neurons in the hidden layer, or by restricting the range of input pulse separations. To retain the simplicity of our example, we have chosen the latter course.

The output neuron is identical to the hidden-layer neurons except that it has a stretched synaptic response waveform, s ( t / D ) , where D > 0 is large. This expands the range of possible pulse positions at the network output by a factor of D. The derivative of s ( t / D ) is correspondingly small, being 0 or f l / D .

Page 4: A pulsed neural network capable of universal approximation

IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 3, NO. 2, MARCH 1992 311

TABLE I PARAMETERS FOR PULSED NETWORK SIMULATION

Symbol Description Value

+ 1 Number of network inputs 2 Number of hidden-layer neurons 9

t o Starting time of pulse on network input 0 0 Starting time of pulse on network input 1 Starting time of hidden-layer output pulses Starting time of network output pulse

- t l T -

- TO Td Desired starting time of network output pulse ~

- Domain (range of t l - t o values) (-1.lj E Squared error for output pulse time - f Total squared error for output pulse times -

A T Time spacing of input delay line taps 0. I 25 Synaptic response waveform (triangular) - .$( j

- Duration of hidden-layer synaptic response 1 .o - Hidden-layer synaptic weights (fixed) 1 .o

D Duration of output-layer synaptic response 10.0 tr.,(Oj Initial output-layer synaptic weights 1 .0 ’? Step size for backward error .propagation 0.05

.4 Threshold for both hidden and output layer 1.5

Fig. 4. PCMAC network example.

One can show that the output pulses from the hidden layer of our example network start at exactly the same time. This property, which is peculiar to our particular example of the PCMAC, greatly simplifies our analysis without invalidating the more general conclusions we reach. Straightforward calcu- lations yield a simple formula for the hidden-layer pulse time T for our special case:

T = 1.25 + 0.5tl . (8)

Since hidden-layer outputs pulse simultaneously we con- clude that the output neuron’s threshold is crossed while all output neuron synaptic responses have an upward slope. Given the stretched triangular shape of s ( t ) , we have

4 [ t o - 7 1 / q = (70 - 7 ) / D (9)

where T~ is the output neuron pulse time. Substituting (9) into (2), we obtain an equation for the output

neuron:

i active

where the w , are output neuron synaptic weights. Since only four neurons in the hidden layer are active simultaneously, we take the sum in (10) over the four corresponding synapses in the output neuron. Combining (9) and (10) yields a closed- form expression for the PCMAC output pulse time function:

A D r,, = 1.25 + 0 3 1 + ~ c 7ui

This equation reveals that plots of r, versus tl always have a slope of 0.5 over intervals where the same group of hidden-layer neurons is active. Simulation results in Fig. 5 illustrate that these intervals are 0.25 time units in width for our example. When the boundary of one of these intervals is crossed, the set of active weights changes in the summation term in (10) and (11). This change allows the network to shift the value of ‘ T ~ up or down over the new interval.

Iv. LEARNING IN THE PULSED NETWORK

In (7), above, we presented a derivative formula relating synaptic weights to output pulse times. The equivalent formula for the output neuron is as follows:

Since hidden-layer neurons pulse simultaneously and threshold for the output neuron occurs on the upward slope of synaptic responses, we have

s’([r, - T ] / D ) = 1 / D .

Page 5: A pulsed neural network capable of universal approximation

312

7 , initial

input pulse time, t ,

Fig. 5. Simulation results of training PCMAC for 1000 pulses. € is total squared error, and ~d is desired output pulse time.

Using (9), (lo), and (13), we obtain a simplified version of (12):

From (14) we derive a least mean squares learning algo- rithm. We start with a definition of squared output error, E , for the network:

1 2

E = - [ r d - r0]*

where r d is the desired output pulse time. Weights in the output neuron are updated by the gradient descent rule:

Substituting (14) and (15) into (16) gives the learning rule used for simulations:

(17)

This learning rule is applied to each active synapse after each output pulse. Table I lists parameter values for the simulation. As in a conventional CMAC, weights in the hidden layer of the network are fixed.

Simulation Results: Fig. 5 illustrates the result of training the pulsed network to reproduce a cosine function:

r d = 4 f C O S T t l .

This function has both positive and negative slopes. To match this nonmonotonic function, output pulse delays must in some cases grow shorter as the input pulse interval increases. Nevertheless, after 1000 training pulses the network matched the desired function quite well. Noticeable errors occur near

(18)

IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 3, NO. 2, MARCH 1992

0 200 400 600 800 1000 1200

number of training pulses

Fig. 6. Time course of total squared error for training of PCMAC

the ends of the interval, however, owing to edge effects described earlier.

The constant offset in (18) compensates for delays incurred in the hidden layer of the network. According to (8) the largest possible delay is 1.75 time units for tl in the range [-1,1], whereas the smallest value of r d is 3.0. The difference of 1.25 between these values is a margin of safety that reduces the likelihood that the learning algorithm will attempt to create extremely large weights.

Conversely, the total variation of r d plus the margin of safety just described must be less than the time, D , for synaptic responses in the output neuron to reach their peak. In (18) the total variation is 2.0, and the margin of safety is 1.25, giving a total of 3.25, which is comfortably less than the value of 10.0 for D. The difference between these values is a second margin of safety that reduces the likelihood that the learning algorithm will attempt to make output-layer weights so small that the output neuron fails to reach threshold.

The total squared error, &, for the curves in Fig. 5 is defined as

& = ]mil -1

where E is defined in (15), and (11) specifies the relationship between T~ and t l . Because (19) is readily integrated, we used a closed-form expression to evaluate & based on the values of T at the centers of intervals. Fig. 6 illustrates the time course o f f computed after every ten pulses for the data in Fig. 5. The results are similar to what one would expect for a conventional CMAC trained with a least mean squares algorithm.

V. APPROXIMATING ARBITRARY MEASURABLE FUNCTIONS

To prove that the PCMAC is capable of universal approxi- mation on a compact domain, we observe that hidden-layer neurons act as coincident pulse detectors and that an output neuron adds variable delays to pulses. Input delay lines allow us to position a response region anywhere in a function's do- main. Consequently, our strategy for approximating functions has two stages. First, we create small hidden-layer response

Page 6: A pulsed neural network capable of universal approximation

IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 3, NO. 2 , MARCH l Y Y 2 313

regions that cover the domain of input pulse intervals. Over a response region, the output pulse time for a neuron varies only slightly. Second, we select output-neuron synaptic weights that add nearly constant delays, corresponding to nearly constant function values, for each response region.

A. Single-Variable Functions

Straightforward modifications improve the ability of the two-input PCMAC to approximate a single-variable function. By increasing the number of hidden-layer neurons and raising the hidden-layer threshold, the intervals spanning the domain of the output pulse time function r, may be made as narrow and as numerous as desired. Over each such interval, r, is a tilted line segment, as in Fig. 5. If we use hidden-layer neurons with nonoverlapping response regions, then we can cover the domain exactly once and the line segments may be moved up or down independently of each other.

We now show that r, can approximate an arbitrary con- tinuous function r,l bounded below by causality conditions. The extension from continuous to measurable functions is a straightforward exercise in real analysis [21]. Interested readers will find the necessary arguments in [22] and [23].

By definition, a continuous Tcf must have limited variations in amplitude over small intervals. Around any point in the domain, we can find an interval over which the value of r d

changes by less than a specified E > 0. Around any point in the domain, we can also design a neuron response interval over which the value of r, changes by less than E . Thus, by making our PCMAC response intervals sufficiently narrow, r, can approximate r(1 with an error less than 2~ everywhere on the domain. Because we assume a compact domain, the required number of intervals is finite.

As we shrink E we obtain a sequence of r<] functions. This sequence converges uniformly to r(1, and we conclude that we can approximate a continuous r d . A constraint on the function we approximate, as discussed in the preceding section, is the lower bound on r,f induced by causality. This bound appears in the guise of a safety margin and is caused by necessary delays in the network.

B. Multivariable Functions

We expand a two-input PCMAC to three or more inputs by increasing the number of synaptic inputs to hidden-layer neurons and by adding multidimensional input delay lines. If we desire multiple outputs, we need only add output neurons. As in the two-input PCMAC, output neurons receive inputs from all of the hidden-layer neurons.

Except for the changes listed below, the conceptual ar- guments presented above for single-variable approximations apply to multivariable approximations.

1) Rather than being intervals, the response regions of multiple-input hidden-layer neurons are ( N + 1)- dimensional volumes that we refer to as granules.

2) Rather than having hidden-layer output functions that are tilted line segments, we have hidden-layer output functions that are composites of tilted hyperplanes.

3) Rather than having granules that cover the domain ex- actly once, we have granules that overlap if N + 1 > 5.

4) Rather than finding an interval over which r d changes by less than E, we find granules such that r d changes by less than E over the union of all granules covering any given point.

5) Rather than having hidden-layer neurons whose out- puts pulse simultaneously, we have hidden-layer neurons whose outputs pulse within an interval of duration less than 2 ~ .

As before, we start with neurons having triangular synaptic response waveforms s ( t ) and unitary synaptic weights. For N + 1 inputs, we make the response region granules small by choosing a threshold close to N + I:

where

The boundary of each granule consists of points where the peak of the summed synaptic responses is exactly equal to threshold. For N + 1 odd, we find that the peak response occurs at the peak time t , + 1 of the pulse occupying the middle position in the order of pulse arrival. The case for N + 1 even is similar but requires more cumbersome notation. Hence, we only consider the case of N + 1 odd.

By measuring pulse intervals with respect to t,,, we obtain an equation specifying the boundary of a granule:

Granules have the following properties: 1) Their size scales with E but their geometrical shape is

2) They are symmetric in the variables t o . . . . . t x . 3) They form tubes in (N + 1)-dimensional space, having

axes of symmetry in the ( t o :... t . ~ ) = (1 :... 1) direction. The axis corresponds to delaying all input pulses by the same amount, leaving interpulse intervals unchanged.

4) Their cross sections are convex parallelepipeds in N - dimensional space.

We duplicate hidden-layer neurons to create more granules, and we use input delays to offset granules and cover the domain. Because the domain is compact, a finite number of granules covers it. Appropriate delay vectors for packing granules edge to edge have the following form:

invariant.

(At" :... At,) = ~ ~ ( - ~ E / N . ~ E / N , O : . . , O ) + k2(0. - 2 ~ l N . ~ E I N . 0. . . ' , 0) + . . + k.y(O.. . . 0. - ~ E / N . ~ E / N ) + K(1, ' . ' . 1) (23)

where ICl , . ' . . kAv and K are integers. Each set of integers k l . . . . . k , ~ corresponds to a unique hidden-layer neuron, and

Page 7: A pulsed neural network capable of universal approximation

314 IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 3, NO. 2, MARCH 1992

(23) tells us how to construct delay lines for hidden-layer inputs. We avoid negative delays by adding the same positive delay K to every input of every hidden-layer neuron.

Except for a set of measure 0 consisting of points on granule boundaries, the covering dictated by (23) is uniform: each point in the domain is contained by exactly m granules. For N + 1 2 5 we find that rn > 1, implying that the granules overlap. Nevertheless, all the hidden-layer neurons responding to a set of input pulses will produce output pulses within an interval of duration less than ZE. We define r to be the center of this interval. We can use the same value of r everywhere in the domain of input pulse intervals.

If the duration D of output neuron synaptic responses is suf- ficiently large, then threshold occurs while all output-neuron synaptic responses have an upward slope. Consequently, the uncertainty of in hidden-layer output pulse times leads to an uncertainty of * E in network output pulse time r,. Thus, (24) replaces (10) as a description of the network output:

r o - r - - c i active

Since rri output-layer synapses are active at one time, we choose output-layer weights w, such that each contributes about equally to the summed response:

1 AD W I Z - -

m r d 7 - r

where Tdd.1 is the desired output pulse time for a point in the center of hidden-layer granule i.

Recall that we assume the variation in desired output pulse time r d to be small over any granule:

(25)

I r d - rd71 < (26)

Assuming every 20, takes on the minimum value allowed by (26) or the maximum value allowed by (26), we obtain an indirect bound on the network output pulse time:

a active I i active I

By combining (24) and (27) we obtain a bound on the approximation error for the network:

Thus, ro can approximate r d with an error less than 2~ everywhere on the domain, and the remainder of the proof is identical to the single-variable case.

VI. CONCLUSION

In this paper we have shown how to construct the PCMAC, a pulsed network based on the CMAC architecture. Simulation results demonstrate that the PCMAC can learn to produce out- put pulses at desired times relative to input pulses. The network used in the simulation is relatively small but may be expanded

to increase the accuracy of output pulses or accommodate more inputs. Within bounds dictated by delays in the hidden layer and by the duration of synaptic responses in the output neuron, a sufficiently large PCMAC can approximate any measurable function.

REFERENCES

[ l ] J. J . Hopfield, “Neurons with graded response have collective computa- tional properties like those of two-state neurons,” Proc. Nnt. Acad. Sci. US., vol. 81, pp. 3088-3092, 1984.

[2] D. E. Rumelhart and J. L. McClelland, Eds., Parallel Distributed Pro- cessing: Exploring the Microstructures of Cognition, vol. 1. Cam- bridge, MA: M.I.T. Press, 1986.

[3] L.M. Optican and B.J. Richmond, “Temporal encoding of two- dimensional patterns by single units in primate inferior temporal cortex 111: Information theoretic analysis,” J . Neurophys., vol. 57, no. 1, pp. 162-178, 1987.

[4] J . S. Albus, Brains, Behavior, and Robotics. Peterborough NH: Mc- Graw Hill, 1981.

[5] F. C. Hoppensteadt, A n Introduction to the Mathematics of Neurons. Cambridge: Cambridge University Press, 1986.

[6] T.B. Schillen and P. Konig, “Coherency detection and response seg- regation by synchronizing and desynchronizing delay connections in a neuronal oscillator model,” in Proc. IEEE Int. Conf Neural Networks (San Diego, CA), June 17-21, 1990, vol. 11, pp. 387-395.

[7] V. Menon and D. S. Tang, “Synchronization in distributed neural sys- tems,” in Proc. IEEE Inr. Conj Neural Networks (San Diego, CA), June 17-21, 1990, vol. 11, pp. 397-402.

181 G.N. Borisyuk, R. M. Borisyuk, A. B. Kirillov, V.I. Kryukov, and W. Singer, “Modeling of oscillatory activity of neuron assemblies of the visual cortex,” in Proc. IEEE Int. Conf Neural Networks (San Diego, CA), June 17-21, 1990, vol. 11, pp. 431-434.

[9] W. J. Freeman, “Simulation of chaotic EEG patterns with a dynamic model of the olfactory system,” Biol. Cybern., vol. 56, pp. 139- 150, 1987.

[ 101 R. J. MacGregor, Neural and Brain Modeling. San Diego, CA: Aca- demic, 1987.

[ I l l W. Rall, “Dendritic spines and synaptic potency,” in Studies in Neu- rophysiology, R. Porter, Ed. Cambridge: Cambridge University Press, 1978, pp. 203-209.

[ 121 J. E. Dayhoff, “Regularity properties in pulse transmission networks,” in Proc. IEEE Int. Conf Neural Networks (San Diego, CA), June 17-21, 1990, vol. 111, pp. 621-626.

131 N. Goerke, M. Schone, B. Kreimeier, and R. Eckmiller, “A network with pulse processing neurons for generation of arbitrary temporal sequences,” in Proc. IEEE Int. Conf Neural Networks (San Diego, CA), June 17-21, 1990, vol. 111, pp. 315-320.

141 M. Gluck, D. B. Parker, and E. S. Reifsnider, “Learning with temporal derivatives in pulse-coded neuronal systems,” in Advances in Neural In- formation Processing Systems 1, D. Touretzky, Ed. New York: Morgan Kaufmann, 1989, pp. 195-203.

151 D. C. Tam, “Decoding of firing intervals in a temporal-coded spike train using a topographically mapped neural network,” in Proc. IEEE Int. Cotit Neural Networks (San Diego, CA), June 17-21, 1990, vol. 111, pp. 627-632.

[16] J. E. Dayhoff, “Detection of favored patterns in the temporal structure of nerve cell connections,” in Proc. IEEE First Int. Conf Neural Networks (San Diego, CA), June 21-24, 1987, vol. 111, pp. 63-77.

[17] J. P. Keener, F.C. Hoppensteadt, and J. Rinzel, “Integrate and fire models of nerve membrane response to oscillatory input,” SIAMJ. Appl. Marh., vol. 41, pp. 503-517, 1981.

[18] E. R. Kandel and J. H. Schwartz, Principles of Neural Science. 2nd ed. New York: Elsevier, 1985.

[19] P. Fatt and B. Katz, “Spontaneous subthreshold activity at moter nerve endings,” J . Physiol. (London), vol. 117, pp. 109-128, 1952.

[20] D. Junge, Nerve and Muscle Excitation. 2nd ed. Sunderland MA: Sinauer, 1981.

[21] H. L. Royden, Real Analysis, 2nd ed. [22] K. Hornik, M. Stinchcombe, and H. White, “Multilayer feedfonvard

networks are universal approximators,” Neural Nefworks, vol. 2, pp. 359-366, 1989.

1231 N. E. Cotter, “The Stone-Weierstrass theorem and its application to neural networks,” IEEE Trans. Neural Networks, vol. 1, pp. 290-295, Dec. 1990.

New York: Macmillan, 1968.