establishing the equivalence between recurrent neural networks and turing machines. ritesh kumar...
TRANSCRIPT
Establishing the Equivalence between Recurrent Neural Networks and Turing Machines.
Ritesh Kumar Sinha(02d05005)
Kumar Gaurav Bijay(02005013)
“No computer has ever been designed that is everaware of what it’s doing; but most of the time, wearen’t either”
– Marvin Minsky.
Introduction
Plan : Establish Equivalence between recurrent neural network and turing machine History of Neurons Definitions Constructive Proof of equivalence
Approach : Conceptual Understanding
Motivation
Understanding the learning patterns of human brain – concept of neuron
Is Turing Machine the ultimate computing machine ?
How powerful are neural networks – DFA, PDA, Turing Machine, still higher …
Brain
The most complicated human organ sense, perceive, feel, think, believe, remember, utter Information processing Centre
Neurons : information processing units
MCP Neuron
McCulloch and Pitts gave a model of a neuron in 1943
But it's only a highly simplified model of real neuron Positive weights (activators) Negative weights (inhibitors)
Artificial Neural Neworks (ANN)
Interconnected units : model neurons
Modifiable weights (models synapse)
Types of ANN
Feed-forward Networks Signals travel in one way only Good at computing static functions
Neuro-Fuzzy Networks Combines advantages of both fuzzy reasoning
and Neural Networks. Good at modeling real life data.
Recurrent Networks
Recurrent Neural Networks
Activation Function:f(x) = x , if x > 0 0 , otherwise
Turing Machine and Turing Complete Languages
Turing Machine: As powerful as any other computer
Turing Complete Language: Programming language that can compute any
function that a Turing Machine can compute.
A language L
Four basic operations: No operation : V V Increment : V V+1 Decrement : V max(0,V-1) Conditional Branch : IF V != 0 GOTO j
(V is any variable having positive integer values, and j stands for line numbers)
L is Turing Complete
Turing Machine can encode Neural Network
Turing machine can compute any computable function, by definition
The activation function, in our case, is a simple non-linear function
Turing machine can therefore simulate our recurrent neural net
Intuitive, cant we write code to simulate our neural net
An example
Function that initiates: Y=X
L1: X X - 1
Y Y + 1
if X != 0 goto L1
An example
Function that initiates: Y=X
if X != 0 goto L1
X X + 1
if X != 0 goto L2
L1: X X - 1
Y Y + 1
if X != 0 goto L1
L2: Y Y
L is Turing Complete : Conceptual Understanding
Idea: Don’t think of C++, think of 8085 ;)
Subtraction: Y = X1 - X2
L is Turing Complete : Conceptual Understanding
Idea: Don’t think of C++, think of 8085 ;)
Subtraction: Y = X1 - X2 Yes, decrement X, increment Y, when Y=0,
stop
Multiplication, division:
L is Turing Complete : Conceptual Understanding
Idea: Don’t think of C++, think of 8085 ;)
Subtraction: Y = X1 - X2
Yes, decrement X, increment Y, when Y=0, stop
Multiplication, division:
Yes, think of the various algos you studied in Hardware Class :)
L is Turing Complete : Conceptual Understanding
If: if X=0 goto L
L is Turing Complete : Conceptual Understanding
If: if X=0 goto L
Yes,
if X != 0 goto L2
Z Z + 1 // Z is dummy
if Z != 0 goto L
L2: ...
Constructing a Perceptron Network for Language L
For each variable V : entry node NV
For each program row i : instruction node Ni
Conditional branch instruction on row i :
2 transition nodes Ni’ and Ni”
Constructions
Variable V :
NV NV
No Operation:
Ni Ni+1
Constructions Continued
Increment Operation :
Ni Ni+1
Ni NV
Decrement Operation:
Ni Ni+1
Ni NV
Constructions Continued
Conditional Branch Operation :
Ni Ni’
Ni Ni’’
Nv Ni’’
Ni’’ Ni+1
Ni’ Nj
Ni’’ Nj
Definitions
Legal State: Transition Nodes (Ni’ and Ni’’) : 0 outputs
Exactly one Ni has output=1
Final State : All instruction nodes have 0 outputs.
Properties
If yi = 1 then program counter is on line i
V = output of Nv
Changes in network state are activated by non-zero nodes.
Proof
•If row i is V V
•If row i is V V - 1
•For V V + 1, behavior is similar
Proof Continued
•If row i is : if V != 0 GOTO j
Proof Continued
•Thus a legal state leads to another legal state.
What about other activation functions?
• Feed-forward Net With Binary State Function is DFA
• Recurrent Neural Network with:1) Sigmoid is Turing Universal2) Saturated Linear Function is
Turing Universal
Conclusion
Turing machine can encode Recurrent Neural Network.
Recurrent neural net can simulate a Turing complete language.
So, Turing machines are Recurrent Neural Networks !
Project
Implementation of Push down automata (PDA) using Neural Networks.
References
[1] McCulloch , W. S., and Pitts, W. : A logical calculus of the ideas immanent in nervous activity. Bulletin of Mathematical Biophysics 5, 1943 :115-133.
[2] Hyotyniemi, H : Turing Machines are Recurrent Neural Networks. In SYmposium of Artificial Networks, Vaasa, Finland, Aug 19-23, 1926, pp. 13-24
[3] Davis, Martin D.Weyuker, Elaine J. : Computability, complexity, and languages : fundamentals of theoretical computer science, New York : Academic Press, 1983.
References Continued
[4] J. E. Hopcroft and J. D. Ullman, Introduction to Automata Theory, Languages and Computation. Addison-Wesley, 1979.
[5] Arbib, M. : Turing Machnies, Finite Automata and Neural Nets. Journal of the ACM (JACM), NY, USA, Oct 1961, Vol 8 , Issue 4 , pp. 467 - 475.