Synaptic Devices and Neuron Circuits

for Neuron-Inspired NanoElectronics

Byung-Gook Park Inter-university Semiconductor Research Center &

Department of Electrical and Computer Engineering Seoul National University

ECE & ISRC

Human vs. AlphaGo

ECE & ISRC

AlphaGo - Software

B. G. Park

Monte Carlo Tree Search (MCTS)

Neural Networks

<Silver, Nature (2016)>

ECE & ISRC

AlphaGo - Hardware

1920 CPU

280 GPU

B. G. Park

Supercomputer Performance

4

ECE & ISRC

Comparison

B. G. Park

Human Brain Digital Computer

5

■ neuron + synapse

■ massively parallel

■ ~ ms speed

■ low power

■ recognition/reasoning

■ self-organized

■ CPU + memory

■ serial

■ ~ ns speed

■ high power

■ computation

■ manufacturing

ECE & ISRC

Interconnection Bottleneck

Human Brain (Face Recognition) Computer System

Bottleneck!!

Bottleneck!!

B. G. Park 6

ECE & ISRC

Human Brain and Its Building Blocks

Processor: neuron (~1011)

B. G. Park

Memory: Synapse (~1015)

7

ECE & ISRC

Neuron (1)

B. G. Park

Structure of a neuron

8

ECE & ISRC

Neuron (2)

B. G. Park

Potential change due to K+ ion channel

Structure of a neuron

9

V (mV)

t(ms) 5

50

100

0

ECE & ISRC

Electrochemical signal transmission in a synapse

Neurotransmitter is emitted in the unit of a vesicle

conveyed to the post-synaptic cell through the cleft

Synapse (1)

B. G. Park 10

ECE & ISRC

Short term memory and long term memory

Synapse (2)

B. G. Park 11

ECE & ISRC

Spike-timing-dependent plasticity – learning mechanism

Spike-Timing-Dependent Plasticity

B. G. Park 12

ECE & ISRC

Building Blocks: Biological vs. Electronic

B. G. Park 13

ECE & ISRC

Artificial Neural Network (ANN)

Concept of neural network

B. G. Park

** Weights are calculated by the back-propagation (BP)

method (1986).

<perceptron>

(1958)

14

ECE & ISRC

Deep Neural Network (DNN)

Multiple hidden layers

B. G. Park

** Vanishing gradient problem (VGP) unsupervised pre-training

(RBM) + modified activation function (ReLU)

15

ECE & ISRC

1st Breakthrough (2006)

Pre-training with restricted Boltzmann machine (RBM)

B. G. Park 16

Regard each pair of layers as RBM

']/)'(exp[

]/)(exp[)(

xx

xx

E

EP

ij

ijijW

PWW

)ln(

WxxxbxTTE

2

1)(

ECE & ISRC

2nd Breakthrough (2010)

Rectified Linear Unit (ReLU)

B. G. Park 17

** Vanishing gradient problem due to saturating

activation function

sigmoid

hyperbolic tangent

** ReLU solves the vanishing gradient problem!!

(+ Concept of signal intensity included)

ECE & ISRC

Comparison: ReLU vs. Sigmoid

<ReLU>

B. G. Park 18

<Sigmoid>

Speed of Learning: 8:1 Compression

<Original>

Epoch = 20

MSE = 0.00177

Epoch = 20

MSE = 0.00441

** MSE (mean square error)

512x512

Image

Epoch = 40

MSE = 0.00116

Epoch = 40

MSE = 0.00403

Epoch = 60

MSE = 0.00104

Epoch = 60

MSE = 0.00395

Epoch = 80

MSE = 0.00097

Epoch = 80

MSE = 0.00392

Epoch = 100

MSE = 0.00095

Epoch = 100

MSE = 0.00380

Epoch = 200

MSE = 0.00094

Epoch = 200

MSE = 0.00250

Epoch = 400

MSE = 0.00093

Epoch = 400

MSE = 0.00194

Epoch = 800

MSE = 0.00093

Epoch = 800

MSE = 0.00142

ECE & ISRC

Recurrent Neural Network (RNN)

Directed cycles in connections

B. G. Park

** Complicated dynamics

and difficulty in training

Sequential data modeling

19

ECE & ISRC

Long Short-Term Memory (LSTM)

RNN with LSTM

B. G. Park

LSTM block

** Hidden layer units with LSTM can

store and access information over

a long period of time.

20

ECE & ISRC

Spiking Neural Network (SNN) (1)

B. G. Park 21

3rd generation neural network model

- input/output: spikes

- signal intensity: firing rates

Sequence of spikes

Output

spikes

S Sequence of spikes

Sequence of spikes

Weighted sum

of decayed PSPs

V(t)

PSP(t)

Fj

t(f)j

Unit j

ECE & ISRC

Spiking Neural Network (SNN) (2)

B. G. Park 22

Learning mechanism

- error back-propagation with time coding

- spike-timing-dependent plasticity (STDP)

Spre Spost

SpreSpost

ECE & ISRC

Spiking Neural Network (SNN) (3)

Structure

B. G. Park

Supervised learning

23

before

learning

after

learning random fixed weights

learning with

modified rule

ECE & ISRC

STDP and Error Back-propagation

B. G. Park 24

<Bengio, arXiv.org, (2016)>

jiijW

STDP BP

jiij xxW

( : spike rate) (x : neuron output)

ECE & ISRC

Floating-body Synaptic Transistor (1)

Cross-section in A A’

direction

B. G. Park 25

Structure TEM image

Gate1Gate2

Fin

O/N/O~ 3.5/5/8 nm

BOX

O ~ 3.5 nm

Sidewall

(MTO)

50 nm

S

D

FB G2 G1

ONO

O A A’

ECE & ISRC

Floating-body Synaptic Transistor (2)

Impact-generated holes are

temporarily stored in the body.

Without further inputs, these

holes gradually disappear

through recombination

process.

Hot holes are programmed

to the floating gate.

Even without further inputs,

these charges do not

disappear without special

erasing actions.

B. G. Park 26

Short-term memorization Long-term memorization

Charge trap

layer

Charge trap

layer

ECE & ISRC

Floating-body Synaptic Transistor (3)

B. G. Park 27

0 20 40 60 80 100

0.0

0.5

1.0

1.5

Sou

rce

Cu

rren

t [

A]

Time [s]

0

2

Inp

ut

Pu

lse

Hei

ght

[V]

0 200 400 600 800 1000

Time [s]

100 μs interval 10 μs interval

Transient response of FST to spikes

ECE & ISRC

Floating-body Synaptic Transistor (4)

B. G. Park 28

10-9

10-7

10-5

10-3

10-1

0

50

100

150

Sou

rce

curr

ent

[nA

]

Time [s]10

-710

-510

-310

-1

Time [s]

N = 1~ 10

No Transition

100 μs interval

N = 1~ 10

one-time increase

per step

10 μs interval

forgetting

long-term

memory due

to hole

trapping

short-term memory

Short-term to long-term memory transition

ECE & ISRC

Floating-body Synaptic Transistor (5)

B. G. Park 29

-5 -4 -3 -2 -1 0 1 2 3 4 5-0.15

-0.10

-0.05

0.00

0.05

0.10

0.15

Sou

rce

Cu

rren

t [

A]

t [s]

STDP characteristic

Spre Spost

t

ECE & ISRC

Neuron Circuits (1)

B. G. Park 30

<spike generation part> <synaptic integration part>

Integrate-and-fire neuron circuit with capacitor integration

ECE & ISRC

Neuron Circuits (2)

B. G. Park 31

Implementation with discrete devices

<PCB layout> <Output of neuron>

ECE & ISRC

Neuron Circuits (3)

B. G. Park 32

-6 -4 -2 0 2 4 6-30

-20

-10

0

10

20

30

40

50STDP chracteristic (measuremt)

Time difference t [s]

Cu

rren

t ch

an

ge

I

[nA

]

Measured STDP characteristics

< Transient current > <STDP characteristic>

ECE & ISRC

Neuron Circuits (4)

B. G. Park 33

Integrate-and-fire neuron circuit with a floating-body MOSFET

ECE & ISRC

Neuron Circuits (5)

B. G. Park 34

Role of floating-body MOSFET in the circuit

<transient current> <spike generation>

- Temporal integration of input spike signal by the floating body

ECE & ISRC

Neuron Circuits (6)

B. G. Park 35

Shapes of spikes and STDP characteristic

<spike shapes> <STDP characteristic>

- Output and feedback spike shapes are different !!

ECE & ISRC

Integration of Neurons and Synapses

B. G. Park 36

Stacking of neuron and synapse arrays

Synapse Array

Neuron Array

Neuron Array

Synapse Array

Neuron Array

Synapse Array

Neuron Array

<primary sensory cortex> <neuromorphic system>

ECE & ISRC

q The recent advancement of ANNs has been

achieved by imitating the biological neural networks

(BNNs) more closely. Spiking neural networks with

STDP weight adjustment is the closest to the BNN.

q Combining the capacitor-less DRAM and SONOS flash

memory, we have developed floating-body synaptic transistors (FSTs), which show short- and long-term memory and STDP.

q Integrate-and-fire neuron circuits with capacitor and

floating-body integration are proposed and

implemented.

Conclusions

B. G. Park 37

ECE & ISRC B. G. Park 38

Thank you for

your attention !