neural networks part i the multilayer perceptron

Neural Networks Part I

The Multilayer Perceptron

Neural Networks History Lesson

2

1962: Rosenblatt, Principles of Neurodynamics: Perceptrons and the Theory of Brain Mechanisms

o First neuron-based learning algorithm

o Allegedly “could learn anything that you could program”


3

1962: Rosenblatt, Principles of Neurodynamics: Perceptrons and the Theory of Brain Mechanisms

o First neuron-based learning algorithm

o Allegedly “could learn anything that you could program”

1969: Minsky & Papert, Perceptron: An Introduction to Computational Geometry

o First real complexity analysis

o Showed, in principle, many things that perceptrons can't learn to do

o Shut down any interest in neural networks


4

1986: Rumelhart, Hinton & Williams, Back Propagation

o Overcame many difficulties raised by Minsky, et al

o Neural Networks wildly popular again (for a while)


5

1999-2005:

Shift to Bayesian Methods: o Best ideas from neural networks

o Direct statistical computing

Support Vector Machines:o Nice mathematical properties

o Global optima vs local optima (like NNs)


6

1999-2005:

Shift to Bayesian Methods: o Best ideas from neural networks

o Direct statistical computing

Support Vector Machines:o Nice mathematical properties

o Global optima vs local optima (like NNs)

A few people still playing with NNs: o Hinton (Univ. Toronto and Google)

o LeCun (NYU and Facebook)

o Bengio (Univ. Montreal)


7

2005-2010:

o Core group of people continue to make improvements

o Developed various tricks to make NN learning practical

2010-Present:o Went back to the NN methods of the 1980s

o Have been wildly successful


8

2005-2010:

o Core group of people continue to make improvements

o Developed various tricks to make NN learning practical

2010-Present:o Went back to the NN methods of the 1980s

o Have been wildly successful

Question for You: Why? What made a 1980s algorithm suddenly amazing 30 years later?


9

Question for You: Why? What made a 1980s algorithm suddenly amazing 30 years later?

Efficient algorithms (Back Propagation)

Raw computing power

Massive amounts of training data:o Deep Neural Nets in particular have a massive amount of parameters

o Need tons of data to effectively train accurate models


10

The history of ML has been very cyclic

But will they be 20 years from now?

¯\_(�)_/¯

The Perceptron

11

In its simplest form, a perceptron takes some binary inputs, and predicts a binary output

Given some training data, learn weights associated with edges that make predictions accurate

o →

2→ Go , I }

-3( →

The Perceptron

12


y =

⇢0 if wTx threshold1 if wTx > threshold

<latexit sha1_base64="pLSlOG7zC6X4LedQAUAnUMB7WHg=">AAACnnichVFNb9QwEHXCR0v46AJHOBi2IE6rpJeWA6iCA5VoRZG6dKX1snKcycaq4wR7Ao2s/C1+SG/9N/VuIwEtiJEsPb03b2Y8k9ZKWozj8yC8cfPW7bX1O9Hde/cfbAwePvpiq8YIGItKVWaScgtKahijRAWT2gAvUwXH6cn7pX78HYyVlT7CtoZZyRda5lJw9NR88HMzYgVH13b0DWUKcmQuYikspHbcGN52zpguiulLyhBO0ZSOypx21LE0pz+6r0crcNotzd9+5WBhwBaVyrzAouT/9rf/8jLQWT9KxIxcFDiKNueDYTyKV0Gvg6QHQ9LH4XxwxrJKNCVoFIpbO03iGme+LkqhwFduLNRcnPAFTD3UvAQ7c6v9dvSFZzKaV8Y/jXTF/u5wvLS2LVOfWXIs7FVtSf5NmzaY78yc1HWDoMVlo7xRFCu6PBbNpAGBqvWACyP9rFQU3HCB/qSRX0Jy9cvXwXhr9HoUf94a7m7321gnT8hz8ookZJvskj1ySMZEBE+Dd8HHYD98Fn4ID8JPl6lh0Hsekz8inFwA8ZHLsg==</latexit><latexit sha1_base64="pLSlOG7zC6X4LedQAUAnUMB7WHg=">AAACnnichVFNb9QwEHXCR0v46AJHOBi2IE6rpJeWA6iCA5VoRZG6dKX1snKcycaq4wR7Ao2s/C1+SG/9N/VuIwEtiJEsPb03b2Y8k9ZKWozj8yC8cfPW7bX1O9Hde/cfbAwePvpiq8YIGItKVWaScgtKahijRAWT2gAvUwXH6cn7pX78HYyVlT7CtoZZyRda5lJw9NR88HMzYgVH13b0DWUKcmQuYikspHbcGN52zpguiulLyhBO0ZSOypx21LE0pz+6r0crcNotzd9+5WBhwBaVyrzAouT/9rf/8jLQWT9KxIxcFDiKNueDYTyKV0Gvg6QHQ9LH4XxwxrJKNCVoFIpbO03iGme+LkqhwFduLNRcnPAFTD3UvAQ7c6v9dvSFZzKaV8Y/jXTF/u5wvLS2LVOfWXIs7FVtSf5NmzaY78yc1HWDoMVlo7xRFCu6PBbNpAGBqvWACyP9rFQU3HCB/qSRX0Jy9cvXwXhr9HoUf94a7m7321gnT8hz8ookZJvskj1ySMZEBE+Dd8HHYD98Fn4ID8JPl6lh0Hsekz8inFwA8ZHLsg==</latexit><latexit sha1_base64="pLSlOG7zC6X4LedQAUAnUMB7WHg=">AAACnnichVFNb9QwEHXCR0v46AJHOBi2IE6rpJeWA6iCA5VoRZG6dKX1snKcycaq4wR7Ao2s/C1+SG/9N/VuIwEtiJEsPb03b2Y8k9ZKWozj8yC8cfPW7bX1O9Hde/cfbAwePvpiq8YIGItKVWaScgtKahijRAWT2gAvUwXH6cn7pX78HYyVlT7CtoZZyRda5lJw9NR88HMzYgVH13b0DWUKcmQuYikspHbcGN52zpguiulLyhBO0ZSOypx21LE0pz+6r0crcNotzd9+5WBhwBaVyrzAouT/9rf/8jLQWT9KxIxcFDiKNueDYTyKV0Gvg6QHQ9LH4XxwxrJKNCVoFIpbO03iGme+LkqhwFduLNRcnPAFTD3UvAQ7c6v9dvSFZzKaV8Y/jXTF/u5wvLS2LVOfWXIs7FVtSf5NmzaY78yc1HWDoMVlo7xRFCu6PBbNpAGBqvWACyP9rFQU3HCB/qSRX0Jy9cvXwXhr9HoUf94a7m7321gnT8hz8ookZJvskj1ySMZEBE+Dd8HHYD98Fn4ID8JPl6lh0Hsekz8inFwA8ZHLsg==</latexit>

WTX = THRESH

WTX - THRESH = o

o→ -

b

¥s y

= 0I →

The Perceptron

13


y =

⇢0 if wTx+ b 01 if wTx+ b > 0

<latexit sha1_base64="d3Pw5q+8jIlGM4xpzrnffbc+yco=">AAACgXicjVFNb9QwEHVCgRK+FjhyGbEtQiAFZy8FIVAlLhyL1KWV1svK8U52rTpOsCfQKMoP4W9x49eAd5sDtBwYydLTe/PlN3lttCfOf0bxtZ3rN27u3kpu37l77/7owcNPvmqcwqmqTOVOc+nRaItT0mTwtHYoy9zgSX72fqOffEXndWWPqa1xXsqV1YVWkgK1GH3fS8RaUtf28BaEwYJEl4gcV9p20jnZ9p1zfcLhKQjCc3JlB7qAHjqRF/Ct/3y8Bec9vIAcNh2+AAchIMn+r+Qd8ESgXQ7TEuH0ak1psrcYjXnKtwFXQTaAMRviaDH6IZaVakq0pIz0fpbxmuahL2llMHRuPNZSnckVzgK0skQ/77YW9rAfmCUUlQvPEmzZPys6WXrflnnILCWt/WVtQ/5LmzVUvJp32tYNoVUXg4rGAFWwuQcstUNFpg1AKqfDrqDW0klF4WpJMCG7/OWrYDpJX6f842R8eDC4scsesyfsGcvYATtkH9gRmzLFfkX7URq9jHfi5zGPJxepcTTUPGJ/RfzmN2SzvSc=</latexit><latexit sha1_base64="d3Pw5q+8jIlGM4xpzrnffbc+yco=">AAACgXicjVFNb9QwEHVCgRK+FjhyGbEtQiAFZy8FIVAlLhyL1KWV1svK8U52rTpOsCfQKMoP4W9x49eAd5sDtBwYydLTe/PlN3lttCfOf0bxtZ3rN27u3kpu37l77/7owcNPvmqcwqmqTOVOc+nRaItT0mTwtHYoy9zgSX72fqOffEXndWWPqa1xXsqV1YVWkgK1GH3fS8RaUtf28BaEwYJEl4gcV9p20jnZ9p1zfcLhKQjCc3JlB7qAHjqRF/Ct/3y8Bec9vIAcNh2+AAchIMn+r+Qd8ESgXQ7TEuH0ak1psrcYjXnKtwFXQTaAMRviaDH6IZaVakq0pIz0fpbxmuahL2llMHRuPNZSnckVzgK0skQ/77YW9rAfmCUUlQvPEmzZPys6WXrflnnILCWt/WVtQ/5LmzVUvJp32tYNoVUXg4rGAFWwuQcstUNFpg1AKqfDrqDW0klF4WpJMCG7/OWrYDpJX6f842R8eDC4scsesyfsGcvYATtkH9gRmzLFfkX7URq9jHfi5zGPJxepcTTUPGJ/RfzmN2SzvSc=</latexit><latexit sha1_base64="d3Pw5q+8jIlGM4xpzrnffbc+yco=">AAACgXicjVFNb9QwEHVCgRK+FjhyGbEtQiAFZy8FIVAlLhyL1KWV1svK8U52rTpOsCfQKMoP4W9x49eAd5sDtBwYydLTe/PlN3lttCfOf0bxtZ3rN27u3kpu37l77/7owcNPvmqcwqmqTOVOc+nRaItT0mTwtHYoy9zgSX72fqOffEXndWWPqa1xXsqV1YVWkgK1GH3fS8RaUtf28BaEwYJEl4gcV9p20jnZ9p1zfcLhKQjCc3JlB7qAHjqRF/Ct/3y8Bec9vIAcNh2+AAchIMn+r+Qd8ESgXQ7TEuH0ak1psrcYjXnKtwFXQTaAMRviaDH6IZaVakq0pIz0fpbxmuahL2llMHRuPNZSnckVzgK0skQ/77YW9rAfmCUUlQvPEmzZPys6WXrflnnILCWt/WVtQ/5LmzVUvJp32tYNoVUXg4rGAFWwuQcstUNFpg1AKqfDrqDW0klF4WpJMCG7/OWrYDpJX6f842R8eDC4scsesyfsGcvYATtkH9gRmzLFfkX7URq9jHfi5zGPJxepcTTUPGJ/RfzmN2SzvSc=</latexit>

ob

: v

The Perceptron

14

Simple Example: Learning ORy =



x1 0 1 0 1x2 0 0 1 1

x1 OR x2 0 1 1 1<latexit sha1_base64="UM8xfazAVtlDjQbsqgXmCAzAUIo=">AAACd3icbVHJTsMwEHXCHrYCx14sWhCnKumF5YTEhRuLKCA1VeW409bCcSJ7gqhCvoF/48aHcOGE2wYBhZFGfn5vxjOeiVIpDPr+m+POzS8sLi2veKtr6xubla3tW5NkmkOLJzLR9xEzIIWCFgqUcJ9qYHEk4S56OBvrd4+gjUjUDY5S6MRsoERfcIaW6lZevHrdCyMYCJUzrdmoyPkzt1Z4T92A7lPf+vcZhl44HNeiVm6W9FfIjGyvCE+ImNOLa1rQ74TgVwKoXlna9tKt1PyGPzH6FwQlqJHSLruV17CX8CwGhVwyY9qBn2LHPoiCSyi8MDOQMv7ABtC2ULEYTCefDK6ge5bp0X6irSukE/ZnRs5iY0ZxZCNjhkMzq43J/7R2hv2jTi5UmiEoPi3UzyTFhI63QHtCA0c5soBxLWyvlA+ZZhztrjw7hGD2y39Bq9k4bvhXzdrpSTmNZVIlu+SABOSQnJJzcklahJN3p+rUnT3nw6XuvnswDXWdMmeH/DI3+ATd3bFz</latexit><latexit sha1_base64="UM8xfazAVtlDjQbsqgXmCAzAUIo=">AAACd3icbVHJTsMwEHXCHrYCx14sWhCnKumF5YTEhRuLKCA1VeW409bCcSJ7gqhCvoF/48aHcOGE2wYBhZFGfn5vxjOeiVIpDPr+m+POzS8sLi2veKtr6xubla3tW5NkmkOLJzLR9xEzIIWCFgqUcJ9qYHEk4S56OBvrd4+gjUjUDY5S6MRsoERfcIaW6lZevHrdCyMYCJUzrdmoyPkzt1Z4T92A7lPf+vcZhl44HNeiVm6W9FfIjGyvCE+ImNOLa1rQ74TgVwKoXlna9tKt1PyGPzH6FwQlqJHSLruV17CX8CwGhVwyY9qBn2LHPoiCSyi8MDOQMv7ABtC2ULEYTCefDK6ge5bp0X6irSukE/ZnRs5iY0ZxZCNjhkMzq43J/7R2hv2jTi5UmiEoPi3UzyTFhI63QHtCA0c5soBxLWyvlA+ZZhztrjw7hGD2y39Bq9k4bvhXzdrpSTmNZVIlu+SABOSQnJJzcklahJN3p+rUnT3nw6XuvnswDXWdMmeH/DI3+ATd3bFz</latexit><latexit sha1_base64="UM8xfazAVtlDjQbsqgXmCAzAUIo=">AAACd3icbVHJTsMwEHXCHrYCx14sWhCnKumF5YTEhRuLKCA1VeW409bCcSJ7gqhCvoF/48aHcOGE2wYBhZFGfn5vxjOeiVIpDPr+m+POzS8sLi2veKtr6xubla3tW5NkmkOLJzLR9xEzIIWCFgqUcJ9qYHEk4S56OBvrd4+gjUjUDY5S6MRsoERfcIaW6lZevHrdCyMYCJUzrdmoyPkzt1Z4T92A7lPf+vcZhl44HNeiVm6W9FfIjGyvCE+ImNOLa1rQ74TgVwKoXlna9tKt1PyGPzH6FwQlqJHSLruV17CX8CwGhVwyY9qBn2LHPoiCSyi8MDOQMv7ABtC2ULEYTCefDK6ge5bp0X6irSukE/ZnRs5iY0ZxZCNjhkMzq43J/7R2hv2jTi5UmiEoPi3UzyTFhI63QHtCA0c5soBxLWyvlA+ZZhztrjw7hGD2y39Bq9k4bvhXzdrpSTmNZVIlu+SABOSQnJJzcklahJN3p+rUnT3nw6XuvnswDXWdMmeH/DI3+ATd3bFz</latexit>

ow=f, }b=o

I

1 wt⇒ly1t,b=t'1511,1+0--2>0

The Perceptron

15

Simple Example: Learning ANDy =



x1 0 1 0 1x2 0 0 1 1

x1 AND x2 0 0 0 1<latexit sha1_base64="jXeA1icTPY9SMyKMFtVfAhfiWlw=">AAACd3icbVFNT9tAEF2bFqihEOCYy6pxopwiOxc+TiB66KlKpaYgxVG03kySVdZra3eMEhn/Bv4bN35ILz2xCa4ooSON9u1787E7E2dSGAyCJ8fd+vBxe2f3k7e3//ngsHZ0/MukuebQ56lM9W3MDEihoI8CJdxmGlgSS7iJ59cr/eYOtBGp+onLDIYJmyoxEZyhpUa1B9/3ohimQhVMa7YsC37PrZXeYhTSFg2sv55R5EWzVStq5W5F/w3ZkO0VYYGIBb36/pWW9G3Ga0FQ46q35/ujWiPoBGuj70FYgQaprDeqPUbjlOcJKOSSGTMIgwyHtiAKLqH0otxAxvicTWFgoWIJmGGxHlxJm5YZ00mqrSuka/bfjIIlxiyT2EYmDGdmU1uR/9MGOU7OhoVQWY6g+EujSS4ppnS1BToWGjjKpQWMa2HfSvmMacbR7sqzQwg3v/we9Lud807wo9u4vKimsUvq5Atpk5CckkvyjfRIn3Dy26k7vtN0/rjUbbntl1DXqXJOyBtzw2c1q7GZ</latexit><latexit sha1_base64="jXeA1icTPY9SMyKMFtVfAhfiWlw=">AAACd3icbVFNT9tAEF2bFqihEOCYy6pxopwiOxc+TiB66KlKpaYgxVG03kySVdZra3eMEhn/Bv4bN35ILz2xCa4ooSON9u1787E7E2dSGAyCJ8fd+vBxe2f3k7e3//ngsHZ0/MukuebQ56lM9W3MDEihoI8CJdxmGlgSS7iJ59cr/eYOtBGp+onLDIYJmyoxEZyhpUa1B9/3ohimQhVMa7YsC37PrZXeYhTSFg2sv55R5EWzVStq5W5F/w3ZkO0VYYGIBb36/pWW9G3Ga0FQ46q35/ujWiPoBGuj70FYgQaprDeqPUbjlOcJKOSSGTMIgwyHtiAKLqH0otxAxvicTWFgoWIJmGGxHlxJm5YZ00mqrSuka/bfjIIlxiyT2EYmDGdmU1uR/9MGOU7OhoVQWY6g+EujSS4ppnS1BToWGjjKpQWMa2HfSvmMacbR7sqzQwg3v/we9Lud807wo9u4vKimsUvq5Atpk5CckkvyjfRIn3Dy26k7vtN0/rjUbbntl1DXqXJOyBtzw2c1q7GZ</latexit><latexit sha1_base64="jXeA1icTPY9SMyKMFtVfAhfiWlw=">AAACd3icbVFNT9tAEF2bFqihEOCYy6pxopwiOxc+TiB66KlKpaYgxVG03kySVdZra3eMEhn/Bv4bN35ILz2xCa4ooSON9u1787E7E2dSGAyCJ8fd+vBxe2f3k7e3//ngsHZ0/MukuebQ56lM9W3MDEihoI8CJdxmGlgSS7iJ59cr/eYOtBGp+onLDIYJmyoxEZyhpUa1B9/3ohimQhVMa7YsC37PrZXeYhTSFg2sv55R5EWzVStq5W5F/w3ZkO0VYYGIBb36/pWW9G3Ga0FQ46q35/ujWiPoBGuj70FYgQaprDeqPUbjlOcJKOSSGTMIgwyHtiAKLqH0otxAxvicTWFgoWIJmGGxHlxJm5YZ00mqrSuka/bfjIIlxiyT2EYmDGdmU1uR/9MGOU7OhoVQWY6g+EujSS4ppnS1BToWGjjKpQWMa2HfSvmMacbR7sqzQwg3v/we9Lud807wo9u4vKimsUvq5Atpk5CckkvyjfRIn3Dy26k7vtN0/rjUbbntl1DXqXJOyBtzw2c1q7GZ</latexit>: w=[1,1 & b= -1

The Perceptron

16

Simple Example: Learning NANDy =



x1 0 1 0 1x2 0 0 1 1

x1 NAND x2 1 1 1 0<latexit sha1_base64="2vKppGvl1XGFf2Q25MWaYD7QWT4=">AAACeXicbVFNT+MwEHUCy0dYoLscEZJFgUUcqqQcluUEggMnBBIFpKaqHHfaWnWcyJ4gqpAfwV/jxi/hwgG3DYItjDTy83sznvFMlEph0PefHXdm9sfc/MKit/RzeWW18uv3tUkyzaHBE5no24gZkEJBAwVKuE01sDiScBMNTkb6zR1oIxJ1hcMUWjHrKdEVnKGl2pVHb2vLCyPoCZUzrdmwyPkDt1Z49+2A7lDf+scZhl7YH9WiVq6X9HvIlGyvCPeImNPz4/NTWtBJSvDJ/XEKqE5Z3HbTrlT9mj82+hUEJaiS0i7alaewk/AsBoVcMmOagZ9iyz6IgksovDAzkDI+YD1oWqhYDKaVj0dX0G3LdGg30dYV0jH7OSNnsTHDOLKRMcO+mdZG5HdaM8PuQSsXKs0QFJ8U6maSYkJHe6AdoYGjHFrAuBa2V8r7TDOOdlueHUIw/eWvoFGv/av5l/Xq0WE5jQWyTjbJLgnIX3JEzsgFaRBOXpwNZ8f547y6m+6uuzcJdZ0yZ438Z+7+Gzi0sgc=</latexit><latexit sha1_base64="2vKppGvl1XGFf2Q25MWaYD7QWT4=">AAACeXicbVFNT+MwEHUCy0dYoLscEZJFgUUcqqQcluUEggMnBBIFpKaqHHfaWnWcyJ4gqpAfwV/jxi/hwgG3DYItjDTy83sznvFMlEph0PefHXdm9sfc/MKit/RzeWW18uv3tUkyzaHBE5no24gZkEJBAwVKuE01sDiScBMNTkb6zR1oIxJ1hcMUWjHrKdEVnKGl2pVHb2vLCyPoCZUzrdmwyPkDt1Z49+2A7lDf+scZhl7YH9WiVq6X9HvIlGyvCPeImNPz4/NTWtBJSvDJ/XEKqE5Z3HbTrlT9mj82+hUEJaiS0i7alaewk/AsBoVcMmOagZ9iyz6IgksovDAzkDI+YD1oWqhYDKaVj0dX0G3LdGg30dYV0jH7OSNnsTHDOLKRMcO+mdZG5HdaM8PuQSsXKs0QFJ8U6maSYkJHe6AdoYGjHFrAuBa2V8r7TDOOdlueHUIw/eWvoFGv/av5l/Xq0WE5jQWyTjbJLgnIX3JEzsgFaRBOXpwNZ8f547y6m+6uuzcJdZ0yZ438Z+7+Gzi0sgc=</latexit><latexit sha1_base64="2vKppGvl1XGFf2Q25MWaYD7QWT4=">AAACeXicbVFNT+MwEHUCy0dYoLscEZJFgUUcqqQcluUEggMnBBIFpKaqHHfaWnWcyJ4gqpAfwV/jxi/hwgG3DYItjDTy83sznvFMlEph0PefHXdm9sfc/MKit/RzeWW18uv3tUkyzaHBE5no24gZkEJBAwVKuE01sDiScBMNTkb6zR1oIxJ1hcMUWjHrKdEVnKGl2pVHb2vLCyPoCZUzrdmwyPkDt1Z49+2A7lDf+scZhl7YH9WiVq6X9HvIlGyvCPeImNPz4/NTWtBJSvDJ/XEKqE5Z3HbTrlT9mj82+hUEJaiS0i7alaewk/AsBoVcMmOagZ9iyz6IgksovDAzkDI+YD1oWqhYDKaVj0dX0G3LdGg30dYV0jH7OSNnsTHDOLKRMcO+mdZG5HdaM8PuQSsXKs0QFJ8U6maSYkJHe6AdoYGjHFrAuBa2V8r7TDOOdlueHUIw/eWvoFGv/av5l/Xq0WE5jQWyTjbJLgnIX3JEzsgFaRBOXpwNZ8f547y6m+6uuzcJdZ0yZ438Z+7+Gzi0sgc=</latexit>I wtxtbs El - 1 3 'Ll, ) t 1.5

The Perceptron

17

Simple Example: Can we learn XOR?y =



x1 0 1 0 1x2 0 0 1 1

x1 XOR x2 0 1 1 0<latexit sha1_base64="7KwtoelifvaVe1T374Vf/hf8/1A=">AAACd3icbVFLT9tAEF67LQ9DaWiPXFbEQZwiOxcKJ6ReuPFQQyLFUbTeTJIV67W1O0ZExr+h/40bP4QLp66DadqkI4322++bx+5MnElhMAieHffDx08bm1vb3s7u570vjf2vtybNNYcuT2Wq+zEzIIWCLgqU0M80sCSW0IvvflR67x60Ean6ifMMhgmbKjERnKGlRo1fvu9FMUyFKpjWbF4W/JFbK72HUUiPaGB9eUaRF82qVtTKnZp+D1mR7RXhAREL2r+8oSVdZoR/ilYZoMZ1b8/3R41m0A4WRtdBWIMmqe1q1HiKxinPE1DIJTNmEAYZDm1BFFxC6UW5gYzxOzaFgYWKJWCGxWJwJW1ZZkwnqbaukC7YvzMKlhgzT2IbmTCcmVWtIv+nDXKcfB8WQmU5guJvjSa5pJjSagt0LDRwlHMLGNfCvpXyGdOMo92VZ4cQrn55HXQ77dN2cN1pnp/V09giB+SQHJOQnJBzckGuSJdw8uIcOL7Tcl5d6h65x2+hrlPnfCP/mBv+BnbdscA=</latexit><latexit sha1_base64="7KwtoelifvaVe1T374Vf/hf8/1A=">AAACd3icbVFLT9tAEF67LQ9DaWiPXFbEQZwiOxcKJ6ReuPFQQyLFUbTeTJIV67W1O0ZExr+h/40bP4QLp66DadqkI4322++bx+5MnElhMAieHffDx08bm1vb3s7u570vjf2vtybNNYcuT2Wq+zEzIIWCLgqU0M80sCSW0IvvflR67x60Ean6ifMMhgmbKjERnKGlRo1fvu9FMUyFKpjWbF4W/JFbK72HUUiPaGB9eUaRF82qVtTKnZp+D1mR7RXhAREL2r+8oSVdZoR/ilYZoMZ1b8/3R41m0A4WRtdBWIMmqe1q1HiKxinPE1DIJTNmEAYZDm1BFFxC6UW5gYzxOzaFgYWKJWCGxWJwJW1ZZkwnqbaukC7YvzMKlhgzT2IbmTCcmVWtIv+nDXKcfB8WQmU5guJvjSa5pJjSagt0LDRwlHMLGNfCvpXyGdOMo92VZ4cQrn55HXQ77dN2cN1pnp/V09giB+SQHJOQnJBzckGuSJdw8uIcOL7Tcl5d6h65x2+hrlPnfCP/mBv+BnbdscA=</latexit><latexit sha1_base64="7KwtoelifvaVe1T374Vf/hf8/1A=">AAACd3icbVFLT9tAEF67LQ9DaWiPXFbEQZwiOxcKJ6ReuPFQQyLFUbTeTJIV67W1O0ZExr+h/40bP4QLp66DadqkI4322++bx+5MnElhMAieHffDx08bm1vb3s7u570vjf2vtybNNYcuT2Wq+zEzIIWCLgqU0M80sCSW0IvvflR67x60Ean6ifMMhgmbKjERnKGlRo1fvu9FMUyFKpjWbF4W/JFbK72HUUiPaGB9eUaRF82qVtTKnZp+D1mR7RXhAREL2r+8oSVdZoR/ilYZoMZ1b8/3R41m0A4WRtdBWIMmqe1q1HiKxinPE1DIJTNmEAYZDm1BFFxC6UW5gYzxOzaFgYWKJWCGxWJwJW1ZZkwnqbaukC7YvzMKlhgzT2IbmTCcmVWtIv+nDXKcfB8WQmU5guJvjSa5pJjSagt0LDRwlHMLGNfCvpXyGdOMo92VZ4cQrn55HXQ77dN2cN1pnp/V09giB+SQHJOQnJBzckGuSJdw8uIcOL7Tcl5d6h65x2+hrlPnfCP/mBv+BnbdscA=</latexit>

1= =

41 wtx = +1 )

L

Wtxtb EO

The Perceptron

18

Simple Example: Can we learn XOR?

NOPE!

y =



x1 0 1 0 1x2 0 0 1 1


The Perceptron

19

Question: Why can’t we learn XOR?

OR AND

NAND XOR

W

The Perceptron

20

Question: Why can’t we learn XOR?

Answer: The perceptron is a linear classifier. Can only learn things that are linearly separable

Question: How can we fix this?


21

Question: How can we fix this?

Answer: The Multilayer Perceptron

Anatomy: o Input Layer

o Hidden Layer

o Output Layera.12 bz

onW 22=


22

Anatomy: Input Layer to Hidden Layer

o Define intermediate activities z

o Define activations a

The indicator I is the activation function

z1 = w111x1 + w1

12x2 + b11, a1 = I(z1 > 0)

z2 = w121x1 + w1

22x2 + b12, a2 = I(z2 > 0)<latexit sha1_base64="LbR5oqBU2cyj5PiNOTK0lNmSgd8=">AAACkHicbZHdTtswFMedDFgXBnTjcjfWyiQQqIqtSdBJsE7cMK6KtAJSWyLHccHC+Zh9stFFeZ89z+54G5y0Qm3ZkSz/z9fP9nGYKWnA9x8d99XK6trrxhtv/e3G5lbz3ftLk+aaiz5PVaqvQ2aEkonogwQlrjMtWBwqcRXen1b5q19CG5kmP2CSiVHMbhM5lpyBDQXNvzs73p+A4GP8OygIKW/Ig91LvF/7dOrTyg8DckMO8PBnziLM6pYhiAfQcfG93K0YJ9jfw54FejWUTqF0CUoXoXQeSpeh9BkaNFt+268NvxRkJlpoZr2g+W8YpTyPRQJcMWMGxM9gVDANkitResPciIzxe3YrBlYmLBZmVNQTLfEnG4nwONV2JYDr6HxHwWJjJnFoK2MGd2Y5VwX/lxvkMD4aFTLJchAJnx40zhWGFFffgyOpBQc1sYJxLe1dMb9jmnGwn+jZIZDlJ78UfdrutP0L2up+mU2jgT6gj2gXEXSIuugM9VAfcWfT+ewcOyfutttxv7rfpqWuM+vZRgvmnj8BEEq6wQ==</latexit><latexit sha1_base64="LbR5oqBU2cyj5PiNOTK0lNmSgd8=">AAACkHicbZHdTtswFMedDFgXBnTjcjfWyiQQqIqtSdBJsE7cMK6KtAJSWyLHccHC+Zh9stFFeZ89z+54G5y0Qm3ZkSz/z9fP9nGYKWnA9x8d99XK6trrxhtv/e3G5lbz3ftLk+aaiz5PVaqvQ2aEkonogwQlrjMtWBwqcRXen1b5q19CG5kmP2CSiVHMbhM5lpyBDQXNvzs73p+A4GP8OygIKW/Ig91LvF/7dOrTyg8DckMO8PBnziLM6pYhiAfQcfG93K0YJ9jfw54FejWUTqF0CUoXoXQeSpeh9BkaNFt+268NvxRkJlpoZr2g+W8YpTyPRQJcMWMGxM9gVDANkitResPciIzxe3YrBlYmLBZmVNQTLfEnG4nwONV2JYDr6HxHwWJjJnFoK2MGd2Y5VwX/lxvkMD4aFTLJchAJnx40zhWGFFffgyOpBQc1sYJxLe1dMb9jmnGwn+jZIZDlJ78UfdrutP0L2up+mU2jgT6gj2gXEXSIuugM9VAfcWfT+ewcOyfutttxv7rfpqWuM+vZRgvmnj8BEEq6wQ==</latexit><latexit sha1_base64="LbR5oqBU2cyj5PiNOTK0lNmSgd8=">AAACkHicbZHdTtswFMedDFgXBnTjcjfWyiQQqIqtSdBJsE7cMK6KtAJSWyLHccHC+Zh9stFFeZ89z+54G5y0Qm3ZkSz/z9fP9nGYKWnA9x8d99XK6trrxhtv/e3G5lbz3ftLk+aaiz5PVaqvQ2aEkonogwQlrjMtWBwqcRXen1b5q19CG5kmP2CSiVHMbhM5lpyBDQXNvzs73p+A4GP8OygIKW/Ig91LvF/7dOrTyg8DckMO8PBnziLM6pYhiAfQcfG93K0YJ9jfw54FejWUTqF0CUoXoXQeSpeh9BkaNFt+268NvxRkJlpoZr2g+W8YpTyPRQJcMWMGxM9gVDANkitResPciIzxe3YrBlYmLBZmVNQTLfEnG4nwONV2JYDr6HxHwWJjJnFoK2MGd2Y5VwX/lxvkMD4aFTLJchAJnx40zhWGFFffgyOpBQc1sYJxLe1dMb9jmnGwn+jZIZDlJ78UfdrutP0L2up+mU2jgT6gj2gXEXSIuugM9VAfcWfT+ewcOyfutttxv7rfpqWuM+vZRgvmnj8BEEq6wQ==</latexit>

a

:#=¥ I


23

Vectorized Anatomy: Input Layer to Hidden Layer

becomes

where

z1 = w111x1 + w1

12x2 + b11, a1 = I(z1 > 0)z2 = w1

21x1 + w122x2 + b12, a2 = I(z2 > 0)

<latexit sha1_base64="if1dw1K6ibWbAraQM+La//QnF5A=">AAACqHicbZHLbtQwFIadlEsJlw6wZGMxAyoqGsXecJGKKrEBiUUrkU7RZBo5zpmpVccJttN2iPJuPAM73gbnsminHMnyf26f7eO0lMLYMPzr+Vt37t67v/0gePjo8ZOd0dNnx6aoNIeIF7LQJykzIIWCyAor4aTUwPJUwiw9/9zmZxegjSjUd7suYZGzlRJLwZl1oWT0ezIJ4hRWQtVMa7ZuatkEvxKC9/FlUhPSnJIrtzd4r/Np79PWTxNySt7i+GfFMsy6ltjCldV5/bXZbRmfcPgGx7Hj0Z5HN3j0Jo9e59FNHu15QQwqGy4bTCbJaBxOw87wbUEGMUaDHSajP3FW8CoHZblkxsxJWNqFA1rBJTRBXBkoGT9nK5g7qVgOZlF3k27wKxfJ8LLQbimLu+j1jprlxqzz1FXmzJ6ZzVwb/F9uXtnl+0UtVFlZULw/aFlJbAvcfhvOhAZu5doJxrVwd8X8jGnGrfvcwA2BbD75tojo9MM0PKLjg4/DNLbRC/QS7SKC3qED9AUdoghx77X3zYu8Y3/PP/Jn/o++1PeGnufohvnpPxh3x1U=</latexit><latexit sha1_base64="if1dw1K6ibWbAraQM+La//QnF5A=">AAACqHicbZHLbtQwFIadlEsJlw6wZGMxAyoqGsXecJGKKrEBiUUrkU7RZBo5zpmpVccJttN2iPJuPAM73gbnsminHMnyf26f7eO0lMLYMPzr+Vt37t67v/0gePjo8ZOd0dNnx6aoNIeIF7LQJykzIIWCyAor4aTUwPJUwiw9/9zmZxegjSjUd7suYZGzlRJLwZl1oWT0ezIJ4hRWQtVMa7ZuatkEvxKC9/FlUhPSnJIrtzd4r/Np79PWTxNySt7i+GfFMsy6ltjCldV5/bXZbRmfcPgGx7Hj0Z5HN3j0Jo9e59FNHu15QQwqGy4bTCbJaBxOw87wbUEGMUaDHSajP3FW8CoHZblkxsxJWNqFA1rBJTRBXBkoGT9nK5g7qVgOZlF3k27wKxfJ8LLQbimLu+j1jprlxqzz1FXmzJ6ZzVwb/F9uXtnl+0UtVFlZULw/aFlJbAvcfhvOhAZu5doJxrVwd8X8jGnGrfvcwA2BbD75tojo9MM0PKLjg4/DNLbRC/QS7SKC3qED9AUdoghx77X3zYu8Y3/PP/Jn/o++1PeGnufohvnpPxh3x1U=</latexit><latexit sha1_base64="if1dw1K6ibWbAraQM+La//QnF5A=">AAACqHicbZHLbtQwFIadlEsJlw6wZGMxAyoqGsXecJGKKrEBiUUrkU7RZBo5zpmpVccJttN2iPJuPAM73gbnsminHMnyf26f7eO0lMLYMPzr+Vt37t67v/0gePjo8ZOd0dNnx6aoNIeIF7LQJykzIIWCyAor4aTUwPJUwiw9/9zmZxegjSjUd7suYZGzlRJLwZl1oWT0ezIJ4hRWQtVMa7ZuatkEvxKC9/FlUhPSnJIrtzd4r/Np79PWTxNySt7i+GfFMsy6ltjCldV5/bXZbRmfcPgGx7Hj0Z5HN3j0Jo9e59FNHu15QQwqGy4bTCbJaBxOw87wbUEGMUaDHSajP3FW8CoHZblkxsxJWNqFA1rBJTRBXBkoGT9nK5g7qVgOZlF3k27wKxfJ8LLQbimLu+j1jprlxqzz1FXmzJ6ZzVwb/F9uXtnl+0UtVFlZULw/aFlJbAvcfhvOhAZu5doJxrVwd8X8jGnGrfvcwA2BbD75tojo9MM0PKLjg4/DNLbRC/QS7SKC3qED9AUdoghx77X3zYu8Y3/PP/Jn/o++1PeGnufohvnpPxh3x1U=</latexit>

z2 = W 1x1 + b1

a2 = I(z2 > 0) = I(W 1x1 + b1 > 0)<latexit sha1_base64="bFJTMLtnFcASWTm+WeQFNfaTtV8=">AAACiXicdVHLTttAFB275VFDIYUlm1ETEBUisrPgJYGQ2LQ7KjUNUpxE48l1GDEeWzPXFanlj+GX2PE3jI2r8mivNNLROec+5t4ok8Kg7z847rv3C4tLyx+8ldWPa+utTxs/TZprDn2eylRfRcyAFAr6KFDCVaaBJZGEQXRzUemDX6CNSNUPnGcwSthMiVhwhpaatO46HS+MYCZUwbRm87LQXJZeEUYx/V2Oe3TndIcOxkFN3JbjgO7RGkcVDsMnJ/vjDBFuUSfFt3L3b4kz6n+hp8+1/xasrF4IatpM43U6k1bb7/p10LcgaECbNHE5ad2H05TnCSjkkhkzDPwMR7YgCi6h9MLcQMb4DZvB0ELFEjCjol5lSbctM6Vxqu1TSGv2eUbBEmPmSWSdCcNr81qryH9pwxzjo1EhVJYjKP7UKM4lxZRWd6FToYGjnFvAuBZ2VsqvmWYc7fU8u4Tg9Zffgn6ve9z1v/fa5yfNNpbJFvlMdklADsk5+UouSZ9wZ8nZdw6cQ3fV7blHbuN1nSZnk7wI9+IRCR69hg==</latexit><latexit sha1_base64="bFJTMLtnFcASWTm+WeQFNfaTtV8=">AAACiXicdVHLTttAFB275VFDIYUlm1ETEBUisrPgJYGQ2LQ7KjUNUpxE48l1GDEeWzPXFanlj+GX2PE3jI2r8mivNNLROec+5t4ok8Kg7z847rv3C4tLyx+8ldWPa+utTxs/TZprDn2eylRfRcyAFAr6KFDCVaaBJZGEQXRzUemDX6CNSNUPnGcwSthMiVhwhpaatO46HS+MYCZUwbRm87LQXJZeEUYx/V2Oe3TndIcOxkFN3JbjgO7RGkcVDsMnJ/vjDBFuUSfFt3L3b4kz6n+hp8+1/xasrF4IatpM43U6k1bb7/p10LcgaECbNHE5ad2H05TnCSjkkhkzDPwMR7YgCi6h9MLcQMb4DZvB0ELFEjCjol5lSbctM6Vxqu1TSGv2eUbBEmPmSWSdCcNr81qryH9pwxzjo1EhVJYjKP7UKM4lxZRWd6FToYGjnFvAuBZ2VsqvmWYc7fU8u4Tg9Zffgn6ve9z1v/fa5yfNNpbJFvlMdklADsk5+UouSZ9wZ8nZdw6cQ3fV7blHbuN1nSZnk7wI9+IRCR69hg==</latexit><latexit sha1_base64="bFJTMLtnFcASWTm+WeQFNfaTtV8=">AAACiXicdVHLTttAFB275VFDIYUlm1ETEBUisrPgJYGQ2LQ7KjUNUpxE48l1GDEeWzPXFanlj+GX2PE3jI2r8mivNNLROec+5t4ok8Kg7z847rv3C4tLyx+8ldWPa+utTxs/TZprDn2eylRfRcyAFAr6KFDCVaaBJZGEQXRzUemDX6CNSNUPnGcwSthMiVhwhpaatO46HS+MYCZUwbRm87LQXJZeEUYx/V2Oe3TndIcOxkFN3JbjgO7RGkcVDsMnJ/vjDBFuUSfFt3L3b4kz6n+hp8+1/xasrF4IatpM43U6k1bb7/p10LcgaECbNHE5ad2H05TnCSjkkhkzDPwMR7YgCi6h9MLcQMb4DZvB0ELFEjCjol5lSbctM6Vxqu1TSGv2eUbBEmPmSWSdCcNr81qryH9pwxzjo1EhVJYjKP7UKM4lxZRWd6FToYGjnFvAuBZ2VsqvmWYc7fU8u4Tg9Zffgn6ve9z1v/fa5yfNNpbJFvlMdklADsk5+UouSZ9wZ8nZdw6cQ3fV7blHbuN1nSZnk7wI9+IRCR69hg==</latexit>

W 1 =

w1

11 w112

w121 w1

22

�, b1 =

b11b12

�

<latexit sha1_base64="0kMo3/CYxOAQiq06yYUmE5mNBR8=">AAACnXicbZHfatswFMZld1s77V/WXfZGLNnYxQi2b9YOCi2DMtYyOliWQuwZST5ORGXZleSWYPxYfZHd7W0mJ4ZtbQ4IfnznnO9IR6ySwtgg+O35Ww8ePtreeYyfPH32/MXg5e4PU9aaw4SXstQXjBqQQsHECivhotJACyZhyi4/dfnpNWgjSvXdLitICjpXIhecWielg9vRCE9/huSQ4FhCbmc4ZjAXqqFa02XbcN7im7QJw9YVvSUdRh3GcSdHf+Wok3EMKutbcazFfGGT9875qqYZwU3McsLa1bTNw1rM0nDtztLIwSZDPBqlg2EwDlZB7kPYwxD1cZ4OfsVZyesClOWSGjMLg8omztgKLsFZ1wYqyi/pHGYOFS3AJM1qvS1545SM5KV2R1myUv/taGhhzLJgrrKgdmHu5jpxU25W23w/aYSqaguKrwfltSS2JN1fkUxo4FYuHVCuhbsr4QuqKbfuR7FbQnj3yfdhEo0PxsG3aHj0sd/GDtpDr9E7FKIP6Ah9Rudogri35x17X7xTn/gn/pn/dV3qe33PK/Rf+NM/KSnFow==</latexit><latexit sha1_base64="0kMo3/CYxOAQiq06yYUmE5mNBR8=">AAACnXicbZHfatswFMZld1s77V/WXfZGLNnYxQi2b9YOCi2DMtYyOliWQuwZST5ORGXZleSWYPxYfZHd7W0mJ4ZtbQ4IfnznnO9IR6ySwtgg+O35Ww8ePtreeYyfPH32/MXg5e4PU9aaw4SXstQXjBqQQsHECivhotJACyZhyi4/dfnpNWgjSvXdLitICjpXIhecWielg9vRCE9/huSQ4FhCbmc4ZjAXqqFa02XbcN7im7QJw9YVvSUdRh3GcSdHf+Wok3EMKutbcazFfGGT9875qqYZwU3McsLa1bTNw1rM0nDtztLIwSZDPBqlg2EwDlZB7kPYwxD1cZ4OfsVZyesClOWSGjMLg8omztgKLsFZ1wYqyi/pHGYOFS3AJM1qvS1545SM5KV2R1myUv/taGhhzLJgrrKgdmHu5jpxU25W23w/aYSqaguKrwfltSS2JN1fkUxo4FYuHVCuhbsr4QuqKbfuR7FbQnj3yfdhEo0PxsG3aHj0sd/GDtpDr9E7FKIP6Ah9Rudogri35x17X7xTn/gn/pn/dV3qe33PK/Rf+NM/KSnFow==</latexit><latexit sha1_base64="0kMo3/CYxOAQiq06yYUmE5mNBR8=">AAACnXicbZHfatswFMZld1s77V/WXfZGLNnYxQi2b9YOCi2DMtYyOliWQuwZST5ORGXZleSWYPxYfZHd7W0mJ4ZtbQ4IfnznnO9IR6ySwtgg+O35Ww8ePtreeYyfPH32/MXg5e4PU9aaw4SXstQXjBqQQsHECivhotJACyZhyi4/dfnpNWgjSvXdLitICjpXIhecWielg9vRCE9/huSQ4FhCbmc4ZjAXqqFa02XbcN7im7QJw9YVvSUdRh3GcSdHf+Wok3EMKutbcazFfGGT9875qqYZwU3McsLa1bTNw1rM0nDtztLIwSZDPBqlg2EwDlZB7kPYwxD1cZ4OfsVZyesClOWSGjMLg8omztgKLsFZ1wYqyi/pHGYOFS3AJM1qvS1545SM5KV2R1myUv/taGhhzLJgrrKgdmHu5jpxU25W23w/aYSqaguKrwfltSS2JN1fkUxo4FYuHVCuhbsr4QuqKbfuR7FbQnj3yfdhEo0PxsG3aHj0sd/GDtpDr9E7FKIP6Ah9Rudogri35x17X7xTn/gn/pn/dV3qe33PK/Rf+NM/KSnFow==</latexit>

b2,

=

× - [ It ] WE

WE

[ wiwii ]b2= Ibf }


24

Vectorized Anatomy: Hidden Layer to Output Layer

where

z3 = W 2a2 + b2

y = I(z3 > 0) = I(W 2a2 + b2 > 0)<latexit sha1_base64="ASe3O1wbpXNcLSB0W+Iovcd8mnM=">AAACiHicdVFdb9MwFHUCjJEBK/DIi0XLNASqkiLYhzQ0wQu8DYnSSXVbOe5Na81xIvsGLUTZb+E38ca/wckiMTa4kqWjc8798L1xrqTFMPzl+bdu39m4u3kv2Lr/4OF279HjrzYrjICxyFRmTmNuQUkNY5So4DQ3wNNYwSQ++9Dok29grMz0FyxzmKV8pWUiBUdHLXo/BoOAxbCSuuLG8LKujFB1ULE4od/r+Wu6c7RDJ/NRS/B6PqIvaYvjBjMWsDXHqqwvWiNDOEeTVp/q3T8V3tHwBT26qv23XmMNGOhlN0wwGCx6/XAYtkFvgqgDfdLFyaL3ky0zUaSgUShu7TQKc5y5giiFgjpghYWcizO+gqmDmqdgZ1W7yZo+d8ySJplxTyNt2asZFU+tLdPYOVOOa3tda8h/adMCk/1ZJXVeIGhx2SgpFMWMNmehS2lAoCod4MJINysVa264QHe8wC0huv7lm2A8Gh4Mw8+j/vFht41N8pQ8I7skInvkmHwkJ2RMhLfhvfLeeG/9LT/y9/yDS6vvdTlPyF/hv/8NMF69sQ==</latexit><latexit sha1_base64="ASe3O1wbpXNcLSB0W+Iovcd8mnM=">AAACiHicdVFdb9MwFHUCjJEBK/DIi0XLNASqkiLYhzQ0wQu8DYnSSXVbOe5Na81xIvsGLUTZb+E38ca/wckiMTa4kqWjc8798L1xrqTFMPzl+bdu39m4u3kv2Lr/4OF279HjrzYrjICxyFRmTmNuQUkNY5So4DQ3wNNYwSQ++9Dok29grMz0FyxzmKV8pWUiBUdHLXo/BoOAxbCSuuLG8LKujFB1ULE4od/r+Wu6c7RDJ/NRS/B6PqIvaYvjBjMWsDXHqqwvWiNDOEeTVp/q3T8V3tHwBT26qv23XmMNGOhlN0wwGCx6/XAYtkFvgqgDfdLFyaL3ky0zUaSgUShu7TQKc5y5giiFgjpghYWcizO+gqmDmqdgZ1W7yZo+d8ySJplxTyNt2asZFU+tLdPYOVOOa3tda8h/adMCk/1ZJXVeIGhx2SgpFMWMNmehS2lAoCod4MJINysVa264QHe8wC0huv7lm2A8Gh4Mw8+j/vFht41N8pQ8I7skInvkmHwkJ2RMhLfhvfLeeG/9LT/y9/yDS6vvdTlPyF/hv/8NMF69sQ==</latexit><latexit sha1_base64="ASe3O1wbpXNcLSB0W+Iovcd8mnM=">AAACiHicdVFdb9MwFHUCjJEBK/DIi0XLNASqkiLYhzQ0wQu8DYnSSXVbOe5Na81xIvsGLUTZb+E38ca/wckiMTa4kqWjc8798L1xrqTFMPzl+bdu39m4u3kv2Lr/4OF279HjrzYrjICxyFRmTmNuQUkNY5So4DQ3wNNYwSQ++9Dok29grMz0FyxzmKV8pWUiBUdHLXo/BoOAxbCSuuLG8LKujFB1ULE4od/r+Wu6c7RDJ/NRS/B6PqIvaYvjBjMWsDXHqqwvWiNDOEeTVp/q3T8V3tHwBT26qv23XmMNGOhlN0wwGCx6/XAYtkFvgqgDfdLFyaL3ky0zUaSgUShu7TQKc5y5giiFgjpghYWcizO+gqmDmqdgZ1W7yZo+d8ySJplxTyNt2asZFU+tLdPYOVOOa3tda8h/adMCk/1ZJXVeIGhx2SgpFMWMNmehS2lAoCod4MJINysVa264QHe8wC0huv7lm2A8Gh4Mw8+j/vFht41N8pQ8I7skInvkmHwkJ2RMhLfhvfLeeG/9LT/y9/yDS6vvdTlPyF/hv/8NMF69sQ==</latexit>

W 2 =⇥w2

11 w212

⇤, b2 =

⇥b21

⇤<latexit sha1_base64="VTIiy0b6DlHmPd9FAKlM8oFfRz4=">AAACfXicbVFNj9MwEHXC12K+Chy5GFpWCFZVkssCEtJKXDguEqUrNdnIdiattY4T7AmoivIz+GPc+C1ccLo5wG5HsvT05s2z/UY0WjmMot9BeOPmrdt3Du7Se/cfPHw0efzkq6tbK2Eha13bM8EdaGVggQo1nDUWeCU0LMXFx6G//A7Wqdp8wW0DWcXXRpVKcvRUPvk5m9HlecI+MJpqKHFFUwFrZTpuLd/2nZQ9/ZF3cdx70SEbYDJAmoIpRhFNrVpvMDvy7LeWF4x2qSiZ6He++217KvL4PNlnQ2ezfDKN5tGu2HUQj2BKxjrNJ7/SopZtBQal5s6t4qjBzBujkhq8deug4fKCr2HloeEVuKzbxdezl54pWFlbfwyyHfvvRMcr57aV8MqK48Zd7Q3kvt6qxfJt1inTtAhGXl5UtpphzYZdsEJZkKi3HnBplX8rkxtuuUS/MepDiK9++TpYJPN38+hzMj15P6ZxQJ6RF+QVickxOSGfyClZEEn+BM+D18GbkISH4VE4v5SGwTjzlPxX4fFfU0a8Og==</latexit><latexit sha1_base64="VTIiy0b6DlHmPd9FAKlM8oFfRz4=">AAACfXicbVFNj9MwEHXC12K+Chy5GFpWCFZVkssCEtJKXDguEqUrNdnIdiattY4T7AmoivIz+GPc+C1ccLo5wG5HsvT05s2z/UY0WjmMot9BeOPmrdt3Du7Se/cfPHw0efzkq6tbK2Eha13bM8EdaGVggQo1nDUWeCU0LMXFx6G//A7Wqdp8wW0DWcXXRpVKcvRUPvk5m9HlecI+MJpqKHFFUwFrZTpuLd/2nZQ9/ZF3cdx70SEbYDJAmoIpRhFNrVpvMDvy7LeWF4x2qSiZ6He++217KvL4PNlnQ2ezfDKN5tGu2HUQj2BKxjrNJ7/SopZtBQal5s6t4qjBzBujkhq8deug4fKCr2HloeEVuKzbxdezl54pWFlbfwyyHfvvRMcr57aV8MqK48Zd7Q3kvt6qxfJt1inTtAhGXl5UtpphzYZdsEJZkKi3HnBplX8rkxtuuUS/MepDiK9++TpYJPN38+hzMj15P6ZxQJ6RF+QVickxOSGfyClZEEn+BM+D18GbkISH4VE4v5SGwTjzlPxX4fFfU0a8Og==</latexit><latexit sha1_base64="VTIiy0b6DlHmPd9FAKlM8oFfRz4=">AAACfXicbVFNj9MwEHXC12K+Chy5GFpWCFZVkssCEtJKXDguEqUrNdnIdiattY4T7AmoivIz+GPc+C1ccLo5wG5HsvT05s2z/UY0WjmMot9BeOPmrdt3Du7Se/cfPHw0efzkq6tbK2Eha13bM8EdaGVggQo1nDUWeCU0LMXFx6G//A7Wqdp8wW0DWcXXRpVKcvRUPvk5m9HlecI+MJpqKHFFUwFrZTpuLd/2nZQ9/ZF3cdx70SEbYDJAmoIpRhFNrVpvMDvy7LeWF4x2qSiZ6He++217KvL4PNlnQ2ezfDKN5tGu2HUQj2BKxjrNJ7/SopZtBQal5s6t4qjBzBujkhq8deug4fKCr2HloeEVuKzbxdezl54pWFlbfwyyHfvvRMcr57aV8MqK48Zd7Q3kvt6qxfJt1inTtAhGXl5UtpphzYZdsEJZkKi3HnBplX8rkxtuuUS/MepDiK9++TpYJPN38+hzMj15P6ZxQJ6RF+QVickxOSGfyClZEEn+BM+D18GbkISH4VE4v5SGwTjzlPxX4fFfU0a8Og==</latexit>

zs = W2A2+b2T

a3=yn=I( Z ?> 0) p.ee#svres NEWFEATVRS

( Inputs ) ( HIDDEN )


25

Example: Can we learn XOR? x1 0 1 0 1x2 0 0 1 1



26

Example: Can we learn XOR? x1 0 1 0 1x2 0 0 1 1

(x1 OR x2) AND (x1 NAND x2) 0 1 1 0<latexit sha1_base64="2EOcTsTdKqsK/cnJtMgylgc25pQ=">AAACoXicbVFNT9tAEF0bCtRtIdBjD10Rt6KXyM6FjxOorYR6CAGRghRH0XozSVas19buGBG5/mH9G73137BOjCDASKN9eu/N7O5MnElhMAj+O+7K6pu19Y233rv3Hza3Gts7v02aaw49nspUX8fMgBQKeihQwnWmgSWxhKv45nulX92CNiJVlzjLYJCwiRJjwRlaatj46/teFMNEqIJpzWZlwf9wG6V3NwzpVxrYfDyjyIum1VXUyu2afrAsyXtVeYRwh4gFPbugJbUV3x6pk84Pyy3bOgtybnzat8JVb1Cj+pWe7w8bzaAVzIO+BGENmqSO7rDxLxqlPE9AIZfMmH4YZDiwDVFwCaUX5QYyxm/YBPoWKpaAGRTzEZf0i2VGdJxqmwrpnH1aUbDEmFkSW2fCcGqeaxX5mtbPcXwwKITKcgTFFxeNc0kxpdW+6Eho4ChnFjCuhX0r5VOmGUe7Vc8OIXz+5Zeg124dtoLzdvP4qJ7GBvlEdskeCck+OSanpEt6hDufnZ9OxzlzffeX23UvFlbXqWs+kqVw+/dfgr63</latexit><latexit sha1_base64="2EOcTsTdKqsK/cnJtMgylgc25pQ=">AAACoXicbVFNT9tAEF0bCtRtIdBjD10Rt6KXyM6FjxOorYR6CAGRghRH0XozSVas19buGBG5/mH9G73137BOjCDASKN9eu/N7O5MnElhMAj+O+7K6pu19Y233rv3Hza3Gts7v02aaw49nspUX8fMgBQKeihQwnWmgSWxhKv45nulX92CNiJVlzjLYJCwiRJjwRlaatj46/teFMNEqIJpzWZlwf9wG6V3NwzpVxrYfDyjyIum1VXUyu2afrAsyXtVeYRwh4gFPbugJbUV3x6pk84Pyy3bOgtybnzat8JVb1Cj+pWe7w8bzaAVzIO+BGENmqSO7rDxLxqlPE9AIZfMmH4YZDiwDVFwCaUX5QYyxm/YBPoWKpaAGRTzEZf0i2VGdJxqmwrpnH1aUbDEmFkSW2fCcGqeaxX5mtbPcXwwKITKcgTFFxeNc0kxpdW+6Eho4ChnFjCuhX0r5VOmGUe7Vc8OIXz+5Zeg124dtoLzdvP4qJ7GBvlEdskeCck+OSanpEt6hDufnZ9OxzlzffeX23UvFlbXqWs+kqVw+/dfgr63</latexit><latexit sha1_base64="2EOcTsTdKqsK/cnJtMgylgc25pQ=">AAACoXicbVFNT9tAEF0bCtRtIdBjD10Rt6KXyM6FjxOorYR6CAGRghRH0XozSVas19buGBG5/mH9G73137BOjCDASKN9eu/N7O5MnElhMAj+O+7K6pu19Y233rv3Hza3Gts7v02aaw49nspUX8fMgBQKeihQwnWmgSWxhKv45nulX92CNiJVlzjLYJCwiRJjwRlaatj46/teFMNEqIJpzWZlwf9wG6V3NwzpVxrYfDyjyIum1VXUyu2afrAsyXtVeYRwh4gFPbugJbUV3x6pk84Pyy3bOgtybnzat8JVb1Cj+pWe7w8bzaAVzIO+BGENmqSO7rDxLxqlPE9AIZfMmH4YZDiwDVFwCaUX5QYyxm/YBPoWKpaAGRTzEZf0i2VGdJxqmwrpnH1aUbDEmFkSW2fCcGqeaxX5mtbPcXwwKITKcgTFFxeNc0kxpdW+6Eho4ChnFjCuhX0r5VOmGUe7Vc8OIXz+5Zeg124dtoLzdvP4qJ7GBvlEdskeCck+OSanpEt6hDufnZ9OxzlzffeX23UvFlbXqWs+kqVw+/dfgr63</latexit>

\ µaBor : T 1 11

T°3=Wl ,bl

-

ORnano : [t.IT/l.5/

/ And :[ 1 1 ) [ -1 ]

NAND W2 b2


27

Example: XOR Check

x1 0 1 0 1x2 0 0 1 1


W 1 =

1 1

�1 �1

�, b1 =

0

1.5

�

<latexit sha1_base64="dtFRWrcBCPwkdLpADAsQUDkV7Fo=">AAACfXicbVFNb9QwEHXCVzFfCxx7MexSIWijuBJaOFSqxIVjkVi20iasHGeya9Vxgj1BWkX5GfwxbvyWXupsc4B2n2Tp6c3Ms/0mq7VyGMd/g/DO3Xv3H+w9pI8eP3n6bPT8xXdXNVbCTFa6sueZcKCVgRkq1HBeWxBlpmGeXXzu6/NfYJ2qzDfc1JCWYmVUoaRALy1HvycTOv/B2QmjiYYCFzTJYKVMK6wVm661tqOcHTDOkoQe9eyI0wRMPjTQxKrVGtNDP/+zETmjbZIVLOu2nrstOxr3bjz6wHZZ0clkORrHUbwFu034QMZkwNly9CfJK9mUYFBq4dyCxzWm3hiV1OCtGwe1kBdiBQtPjSjBpe02vo698UrOisr6Y5Bt1X8nWlE6tykz31kKXLubtV7cVVs0WHxMW2XqBsHI64uKRjOsWL8LlisLEvXGEyGt8m9lci2skOg3Rn0I/OaXb5PZcfQpir8ej0+nQxp7ZJ+8Jm8JJ1NySr6QMzIjklwGr4J3wfuQhAfhYRhdt4bBMPOS/IdwegVA8rlr</latexit><latexit sha1_base64="dtFRWrcBCPwkdLpADAsQUDkV7Fo=">AAACfXicbVFNb9QwEHXCVzFfCxx7MexSIWijuBJaOFSqxIVjkVi20iasHGeya9Vxgj1BWkX5GfwxbvyWXupsc4B2n2Tp6c3Ms/0mq7VyGMd/g/DO3Xv3H+w9pI8eP3n6bPT8xXdXNVbCTFa6sueZcKCVgRkq1HBeWxBlpmGeXXzu6/NfYJ2qzDfc1JCWYmVUoaRALy1HvycTOv/B2QmjiYYCFzTJYKVMK6wVm661tqOcHTDOkoQe9eyI0wRMPjTQxKrVGtNDP/+zETmjbZIVLOu2nrstOxr3bjz6wHZZ0clkORrHUbwFu034QMZkwNly9CfJK9mUYFBq4dyCxzWm3hiV1OCtGwe1kBdiBQtPjSjBpe02vo698UrOisr6Y5Bt1X8nWlE6tykz31kKXLubtV7cVVs0WHxMW2XqBsHI64uKRjOsWL8LlisLEvXGEyGt8m9lci2skOg3Rn0I/OaXb5PZcfQpir8ej0+nQxp7ZJ+8Jm8JJ1NySr6QMzIjklwGr4J3wfuQhAfhYRhdt4bBMPOS/IdwegVA8rlr</latexit><latexit sha1_base64="dtFRWrcBCPwkdLpADAsQUDkV7Fo=">AAACfXicbVFNb9QwEHXCVzFfCxx7MexSIWijuBJaOFSqxIVjkVi20iasHGeya9Vxgj1BWkX5GfwxbvyWXupsc4B2n2Tp6c3Ms/0mq7VyGMd/g/DO3Xv3H+w9pI8eP3n6bPT8xXdXNVbCTFa6sueZcKCVgRkq1HBeWxBlpmGeXXzu6/NfYJ2qzDfc1JCWYmVUoaRALy1HvycTOv/B2QmjiYYCFzTJYKVMK6wVm661tqOcHTDOkoQe9eyI0wRMPjTQxKrVGtNDP/+zETmjbZIVLOu2nrstOxr3bjz6wHZZ0clkORrHUbwFu034QMZkwNly9CfJK9mUYFBq4dyCxzWm3hiV1OCtGwe1kBdiBQtPjSjBpe02vo698UrOisr6Y5Bt1X8nWlE6tykz31kKXLubtV7cVVs0WHxMW2XqBsHI64uKRjOsWL8LlisLEvXGEyGt8m9lci2skOg3Rn0I/OaXb5PZcfQpir8ej0+nQxp7ZJ+8Jm8JJ1NySr6QMzIjklwGr4J3wfuQhAfhYRhdt4bBMPOS/IdwegVA8rlr</latexit>

W 2 =⇥1 1

⇤, b2 =

⇥�1

⇤<latexit sha1_base64="NxNR37Ppw14yi1R4phhQWUATR6g=">AAACbHicbVFNb9NAEF0b6McWigvqoVSVViRUHCCycykckCJx4VgkQirFJlqvx8mq67W7O64UWb7xC7nxF7jwF9ikPpQ0I6309N7Mm923aaWkxTD87fmPHj/Z2d3bpwdPnx0+D45efLdlbQSMRalKc5VyC0pqGKNEBVeVAV6kCibp9eeVPrkFY2Wpv+GygqTgcy1zKTg6ahb87Pfp5MeQfWI0VpDjlMYpzKVuuDF82TZCtDRi5yxyOuiso2ls5HyByTvH3tQ8Y7SJ05yl7dppu1FL30fbPGi/Pwt64SBcF3sIog70SFeXs+BXnJWiLkCjUNzaaRRWmDhjlEKBs64tVFxc8zlMHdS8AJs067Ra9sYxGctL445GtmbvTzS8sHZZpK6z4Liwm9qK3KZNa8w/JI3UVY2gxd2ivFYMS7aKnmXSgEC1dIALI91dmVhwwwW6D6IuhGjzyQ/BeDj4OAi/Dnujiy6NPXJKXpO3JCIXZES+kEsyJoL88QLvxHvl/fWP/VP/7K7V97qZl+S/8s//ATYBtkk=</latexit><latexit sha1_base64="NxNR37Ppw14yi1R4phhQWUATR6g=">AAACbHicbVFNb9NAEF0b6McWigvqoVSVViRUHCCycykckCJx4VgkQirFJlqvx8mq67W7O64UWb7xC7nxF7jwF9ikPpQ0I6309N7Mm923aaWkxTD87fmPHj/Z2d3bpwdPnx0+D45efLdlbQSMRalKc5VyC0pqGKNEBVeVAV6kCibp9eeVPrkFY2Wpv+GygqTgcy1zKTg6ahb87Pfp5MeQfWI0VpDjlMYpzKVuuDF82TZCtDRi5yxyOuiso2ls5HyByTvH3tQ8Y7SJ05yl7dppu1FL30fbPGi/Pwt64SBcF3sIog70SFeXs+BXnJWiLkCjUNzaaRRWmDhjlEKBs64tVFxc8zlMHdS8AJs067Ra9sYxGctL445GtmbvTzS8sHZZpK6z4Liwm9qK3KZNa8w/JI3UVY2gxd2ivFYMS7aKnmXSgEC1dIALI91dmVhwwwW6D6IuhGjzyQ/BeDj4OAi/Dnujiy6NPXJKXpO3JCIXZES+kEsyJoL88QLvxHvl/fWP/VP/7K7V97qZl+S/8s//ATYBtkk=</latexit><latexit sha1_base64="NxNR37Ppw14yi1R4phhQWUATR6g=">AAACbHicbVFNb9NAEF0b6McWigvqoVSVViRUHCCycykckCJx4VgkQirFJlqvx8mq67W7O64UWb7xC7nxF7jwF9ikPpQ0I6309N7Mm923aaWkxTD87fmPHj/Z2d3bpwdPnx0+D45efLdlbQSMRalKc5VyC0pqGKNEBVeVAV6kCibp9eeVPrkFY2Wpv+GygqTgcy1zKTg6ahb87Pfp5MeQfWI0VpDjlMYpzKVuuDF82TZCtDRi5yxyOuiso2ls5HyByTvH3tQ8Y7SJ05yl7dppu1FL30fbPGi/Pwt64SBcF3sIog70SFeXs+BXnJWiLkCjUNzaaRRWmDhjlEKBs64tVFxc8zlMHdS8AJs067Ra9sYxGctL445GtmbvTzS8sHZZpK6z4Liwm9qK3KZNa8w/JI3UVY2gxd2ivFYMS7aKnmXSgEC1dIALI91dmVhwwwW6D6IuhGjzyQ/BeDj4OAi/Dnujiy6NPXJKXpO3JCIXZES+kEsyJoL88QLvxHvl/fWP/VP/7K7V97qZl+S/8s//ATYBtkk=</latexit>

z2 = W 1x1+b1 =

1 1

�1 �1

� �+

0

1.5

�=

�, a2 = I(z2 > 0) =

�

<latexit sha1_base64="1OIxmbCc/a4EtkhLEvGxQfJ4e6A=">AAADm3iclVJbb9MwFHYTLsPcOnhCSOiIFjQ0qOJKaPAwmAQPG+JhSIROatrKcZzWWuIE20ErUf4UP4U3/g1OGomxtQ87kq3P5/Kd408nzBOhjef96Tjutes3bm7dwrfv3L13v7v94JvOCsW4z7IkUych1TwRkvtGmISf5IrTNEz4KDz9UMdHP7jSIpNfzTLnk5TOpYgFo8a6ZtudX/0+LoMwhp/VdAj7MJqS5nlWTQnsQoPDGu8DDhIemzEOQj4XsqRK0WVVKlVhAs+BQBDgVzWyl83lMmpTcKDEfGEmG+orDEG+oNJkaUkqS3L+uY4H7FwbuTxLQAav19bhjX+48gwvLdP3gkYr7ehKu8DwM6PS8qja+SfpO/BebBbvyo1xvz/r9ryB1xhcBqQFPdTa8az7O4gyVqRcGpZQrcfEy83EEhvBEm6pC81zyk7pnI8tlDTlelI2y1XBM+uJIM6UPdJA4z1fUdJU62Ua2syUmoW+GKud62LjwsRvJqWQeWG4ZKtGcZGAyaDeVIiE4swkSwsoU8LOCmxBFWXG7jO2IpCLX74M/OHg7cD7Muwd7LVqbKHH6CnaQQTtoQN0iI6Rj5jzyHnvHDpH7hP3o/vJ/bxKdTptzUP0n7n+X7G5HyE=</latexit><latexit sha1_base64="1OIxmbCc/a4EtkhLEvGxQfJ4e6A=">AAADm3iclVJbb9MwFHYTLsPcOnhCSOiIFjQ0qOJKaPAwmAQPG+JhSIROatrKcZzWWuIE20ErUf4UP4U3/g1OGomxtQ87kq3P5/Kd408nzBOhjef96Tjutes3bm7dwrfv3L13v7v94JvOCsW4z7IkUych1TwRkvtGmISf5IrTNEz4KDz9UMdHP7jSIpNfzTLnk5TOpYgFo8a6ZtudX/0+LoMwhp/VdAj7MJqS5nlWTQnsQoPDGu8DDhIemzEOQj4XsqRK0WVVKlVhAs+BQBDgVzWyl83lMmpTcKDEfGEmG+orDEG+oNJkaUkqS3L+uY4H7FwbuTxLQAav19bhjX+48gwvLdP3gkYr7ehKu8DwM6PS8qja+SfpO/BebBbvyo1xvz/r9ryB1xhcBqQFPdTa8az7O4gyVqRcGpZQrcfEy83EEhvBEm6pC81zyk7pnI8tlDTlelI2y1XBM+uJIM6UPdJA4z1fUdJU62Ua2syUmoW+GKud62LjwsRvJqWQeWG4ZKtGcZGAyaDeVIiE4swkSwsoU8LOCmxBFWXG7jO2IpCLX74M/OHg7cD7Muwd7LVqbKHH6CnaQQTtoQN0iI6Rj5jzyHnvHDpH7hP3o/vJ/bxKdTptzUP0n7n+X7G5HyE=</latexit><latexit sha1_base64="1OIxmbCc/a4EtkhLEvGxQfJ4e6A=">AAADm3iclVJbb9MwFHYTLsPcOnhCSOiIFjQ0qOJKaPAwmAQPG+JhSIROatrKcZzWWuIE20ErUf4UP4U3/g1OGomxtQ87kq3P5/Kd408nzBOhjef96Tjutes3bm7dwrfv3L13v7v94JvOCsW4z7IkUych1TwRkvtGmISf5IrTNEz4KDz9UMdHP7jSIpNfzTLnk5TOpYgFo8a6ZtudX/0+LoMwhp/VdAj7MJqS5nlWTQnsQoPDGu8DDhIemzEOQj4XsqRK0WVVKlVhAs+BQBDgVzWyl83lMmpTcKDEfGEmG+orDEG+oNJkaUkqS3L+uY4H7FwbuTxLQAav19bhjX+48gwvLdP3gkYr7ehKu8DwM6PS8qja+SfpO/BebBbvyo1xvz/r9ryB1xhcBqQFPdTa8az7O4gyVqRcGpZQrcfEy83EEhvBEm6pC81zyk7pnI8tlDTlelI2y1XBM+uJIM6UPdJA4z1fUdJU62Ua2syUmoW+GKud62LjwsRvJqWQeWG4ZKtGcZGAyaDeVIiE4swkSwsoU8LOCmxBFWXG7jO2IpCLX74M/OHg7cD7Muwd7LVqbKHH6CnaQQTtoQN0iI6Rj5jzyHnvHDpH7hP3o/vJ/bxKdTptzUP0n7n+X7G5HyE=</latexit>

z3 = W 2a2 + b2 =⇥1 1

⇤ �+⇥�1

⇤=

⇥ ⇤, y = I(z3 > 0) =

⇥ ⇤

<latexit sha1_base64="4x4wKmONk0RqxcJrhKAYXwr6uSw=">AAADanichVJNb9NAEN3YfJQFSkoPgHpZEYOKgMgOh8KhqBIXuBWJkEpxEq3X63jV9drsjhHG8oG/yI1/wIX/wMa1BG0TdaSV3rzZN1+aqJDCgO//6jnutes3bm7dwrfv3N2+19+5/9nkpWZ8zHKZ65OIGi6F4mMQIPlJoTnNIskn0em7VXzylWsjcvUJqoLPMrpUIhGMgqUWO70fnofrMErI92b+ihySyXzUurSZj8hz0uJohQ8JDiVPYIrDiC+FqqnWtGpqrRsckKcksHGu4o7GoRbLFGYbNA0mYZFSBXlWB00YnnPX5SG2l425XgZrJXhjy82V9V5Y6ZeSxjhMKdRVY8cPgX8DndUfmv1/C3tL/GebV3N1Hex5i/7AH/qtkcsg6MAAdXa86P8M45yVGVfAJDVmGvgFzGxiEExym7o0vKDslC751EJFM25mdXsrDXlimZgkubZPAWnZ/xU1zYypssj+zCik5mJsRa6LTUtIXs9qoYoSuGJnhZJSEsjJ6vBILDRnICsLKNPC9kpYSjVlYM8T2yUEF0e+DMaj4Zuh/3E0ODrotrGF9tBjtI8CdICO0Ht0jMaI9X47284D56Hzx911H7l7Z1+dXqfZRefM9f4CM6wQ+A==</latexit><latexit sha1_base64="4x4wKmONk0RqxcJrhKAYXwr6uSw=">AAADanichVJNb9NAEN3YfJQFSkoPgHpZEYOKgMgOh8KhqBIXuBWJkEpxEq3X63jV9drsjhHG8oG/yI1/wIX/wMa1BG0TdaSV3rzZN1+aqJDCgO//6jnutes3bm7dwrfv3N2+19+5/9nkpWZ8zHKZ65OIGi6F4mMQIPlJoTnNIskn0em7VXzylWsjcvUJqoLPMrpUIhGMgqUWO70fnofrMErI92b+ihySyXzUurSZj8hz0uJohQ8JDiVPYIrDiC+FqqnWtGpqrRsckKcksHGu4o7GoRbLFGYbNA0mYZFSBXlWB00YnnPX5SG2l425XgZrJXhjy82V9V5Y6ZeSxjhMKdRVY8cPgX8DndUfmv1/C3tL/GebV3N1Hex5i/7AH/qtkcsg6MAAdXa86P8M45yVGVfAJDVmGvgFzGxiEExym7o0vKDslC751EJFM25mdXsrDXlimZgkubZPAWnZ/xU1zYypssj+zCik5mJsRa6LTUtIXs9qoYoSuGJnhZJSEsjJ6vBILDRnICsLKNPC9kpYSjVlYM8T2yUEF0e+DMaj4Zuh/3E0ODrotrGF9tBjtI8CdICO0Ht0jMaI9X47284D56Hzx911H7l7Z1+dXqfZRefM9f4CM6wQ+A==</latexit><latexit sha1_base64="4x4wKmONk0RqxcJrhKAYXwr6uSw=">AAADanichVJNb9NAEN3YfJQFSkoPgHpZEYOKgMgOh8KhqBIXuBWJkEpxEq3X63jV9drsjhHG8oG/yI1/wIX/wMa1BG0TdaSV3rzZN1+aqJDCgO//6jnutes3bm7dwrfv3N2+19+5/9nkpWZ8zHKZ65OIGi6F4mMQIPlJoTnNIskn0em7VXzylWsjcvUJqoLPMrpUIhGMgqUWO70fnofrMErI92b+ihySyXzUurSZj8hz0uJohQ8JDiVPYIrDiC+FqqnWtGpqrRsckKcksHGu4o7GoRbLFGYbNA0mYZFSBXlWB00YnnPX5SG2l425XgZrJXhjy82V9V5Y6ZeSxjhMKdRVY8cPgX8DndUfmv1/C3tL/GebV3N1Hex5i/7AH/qtkcsg6MAAdXa86P8M45yVGVfAJDVmGvgFzGxiEExym7o0vKDslC751EJFM25mdXsrDXlimZgkubZPAWnZ/xU1zYypssj+zCik5mJsRa6LTUtIXs9qoYoSuGJnhZJSEsjJ6vBILDRnICsLKNPC9kpYSjVlYM8T2yUEF0e+DMaj4Zuh/3E0ODrotrGF9tBjtI8CdICO0Ht0jMaI9X47284D56Hzx911H7l7Z1+dXqfZRefM9f4CM6wQ+A==</latexit>

O112.2 > o )

0 0 0

0 1.5 g

°

, o o


28

Example: XOR Check

x1 0 1 0 1x2 0 0 1 1


W 1 =

1 1

�1 �1

�, b1 =

0

1.5

�


W 2 =⇥1 1

⇤, b2 =

⇥�1


z2 = W 1x1+b1 =

1 1

�1 �1

� �+

0

1.5

�=

�, a2 = I(z2 > 0) =

�


z3 = W 2a2 + b2 =⇥1 1

⇤ �+⇥�1

⇤=

⇥ ⇤, y = I(z3 > 0) =

⇥ ⇤


O

1 c (0 05 1

'

, 1 1


29

Example: XOR Check

x1 0 1 0 1x2 0 0 1 1


W 1 =

1 1

�1 �1

�, b1 =

0

1.5

�


W 2 =⇥1 1

⇤, b2 =

⇥�1


z2 = W 1x1+b1 =

1 1

�1 �1

� �+

0

1.5

�=

�, a2 = I(z2 > 0) =

�


z3 = W 2a2 + b2 =⇥1 1

⇤ �+⇥�1

⇤=

⇥ ⇤, y = I(z3 > 0) =

⇥ ⇤



30

Example: XOR Check

x1 0 1 0 1x2 0 0 1 1


W 1 =

1 1

�1 �1

�, b1 =

0

1.5

�


W 2 =⇥1 1

⇤, b2 =

⇥�1


z2 = W 1x1+b1 =

1 1

�1 �1

� �+

0

1.5

�=

�, a2 = I(z2 > 0) =

�


z3 = W 2a2 + b2 =⇥1 1

⇤ �+⇥�1

⇤=

⇥ ⇤, y = I(z3 > 0) =

⇥ ⇤


1 2 1

1 - .5 0

f o o


31

OK, so the multilayer perceptron is cool and all, but when do we get to REAL neural networks??


32

OK, so the multilayer perceptron is cool and all, but when do we get to REAL neural networks??

We’re pretty much already there.

Things we’ll explore moving forward:

o Non-Binary Features

o Better activation functions

o How to choose architectures

o How to train these things

o Regression or Classification (Answer: Yes)

33

Neural Networks I Wrap-Up

33

o Simplest network is just a regular old perceptron (a linear classifier)

o We can learn non-linear decision boundaries by chaining perceptrons

o We can represent these complicated interactions using linear algebra

Next Time: o The Feed-Forward Neural Network

neural networks part i the multilayer perceptron

Documents