neural networks part i the multilayer perceptron

Neural Networks Part I

The Multilayer Perceptron

Neural Networks History Lesson

1962: Rosenblatt, Principles of Neurodynamics: Perceptrons and the Theory of Brain Mechanisms

o First neuron-based learning algorithm

o Allegedly “could learn anything that you could program”

1962: Rosenblatt, Principles of Neurodynamics: Perceptrons and the Theory of Brain Mechanisms

o First neuron-based learning algorithm

o Allegedly “could learn anything that you could program”

1969: Minsky & Papert, Perceptron: An Introduction to Computational Geometry

o First real complexity analysis

o Showed, in principle, many things that perceptrons can't learn to do

o Shut down any interest in neural networks

1986: Rumelhart, Hinton & Williams, Back Propagation

o Overcame many difficulties raised by Minsky, et al

o Neural Networks wildly popular again (for a while)

1999-2005:

Shift to Bayesian Methods: o Best ideas from neural networks

o Direct statistical computing

Support Vector Machines:o Nice mathematical properties

o Global optima vs local optima (like NNs)

1999-2005:

Shift to Bayesian Methods: o Best ideas from neural networks

o Direct statistical computing

Support Vector Machines:o Nice mathematical properties

o Global optima vs local optima (like NNs)

A few people still playing with NNs: o Hinton (Univ. Toronto and Google)

o LeCun (NYU and Facebook)

o Bengio (Univ. Montreal)

2005-2010:

o Core group of people continue to make improvements

o Developed various tricks to make NN learning practical

2010-Present:o Went back to the NN methods of the 1980s

o Have been wildly successful

2005-2010:

o Core group of people continue to make improvements

o Developed various tricks to make NN learning practical

2010-Present:o Went back to the NN methods of the 1980s

o Have been wildly successful

Question for You: Why? What made a 1980s algorithm suddenly amazing 30 years later?

Efficient algorithms (Back Propagation)

Raw computing power

Massive amounts of training data:o Deep Neural Nets in particular have a massive amount of parameters

o Need tons of data to effectively train accurate models

The history of ML has been very cyclic

But will they be 20 years from now?

¯\_(�)_/¯

The Perceptron

In its simplest form, a perceptron takes some binary inputs, and predicts a binary output

Given some training data, learn weights associated with edges that make predictions accurate

2→ Go , I }

-3( →

The Perceptron

⇢0 if wTx threshold1 if wTx > threshold

<latexit sha1_base64="pLSlOG7zC6X4LedQAUAnUMB7WHg=">AAACnnichVFNb9QwEHXCR0v46AJHOBi2IE6rpJeWA6iCA5VoRZG6dKX1snKcycaq4wR7Ao2s/C1+SG/9N/VuIwEtiJEsPb03b2Y8k9ZKWozj8yC8cfPW7bX1O9Hde/cfbAwePvpiq8YIGItKVWaScgtKahijRAWT2gAvUwXH6cn7pX78HYyVlT7CtoZZyRda5lJw9NR88HMzYgVH13b0DWUKcmQuYikspHbcGN52zpguiulLyhBO0ZSOypx21LE0pz+6r0crcNotzd9+5WBhwBaVyrzAouT/9rf/8jLQWT9KxIxcFDiKNueDYTyKV0Gvg6QHQ9LH4XxwxrJKNCVoFIpbO03iGme+LkqhwFduLNRcnPAFTD3UvAQ7c6v9dvSFZzKaV8Y/jXTF/u5wvLS2LVOfWXIs7FVtSf5NmzaY78yc1HWDoMVlo7xRFCu6PBbNpAGBqvWACyP9rFQU3HCB/qSRX0Jy9cvXwXhr9HoUf94a7m7321gnT8hz8ookZJvskj1ySMZEBE+Dd8HHYD98Fn4ID8JPl6lh0Hsekz8inFwA8ZHLsg==</latexit><latexit sha1_base64="pLSlOG7zC6X4LedQAUAnUMB7WHg=">AAACnnichVFNb9QwEHXCR0v46AJHOBi2IE6rpJeWA6iCA5VoRZG6dKX1snKcycaq4wR7Ao2s/C1+SG/9N/VuIwEtiJEsPb03b2Y8k9ZKWozj8yC8cfPW7bX1O9Hde/cfbAwePvpiq8YIGItKVWaScgtKahijRAWT2gAvUwXH6cn7pX78HYyVlT7CtoZZyRda5lJw9NR88HMzYgVH13b0DWUKcmQuYikspHbcGN52zpguiulLyhBO0ZSOypx21LE0pz+6r0crcNotzd9+5WBhwBaVyrzAouT/9rf/8jLQWT9KxIxcFDiKNueDYTyKV0Gvg6QHQ9LH4XxwxrJKNCVoFIpbO03iGme+LkqhwFduLNRcnPAFTD3UvAQ7c6v9dvSFZzKaV8Y/jXTF/u5wvLS2LVOfWXIs7FVtSf5NmzaY78yc1HWDoMVlo7xRFCu6PBbNpAGBqvWACyP9rFQU3HCB/qSRX0Jy9cvXwXhr9HoUf94a7m7321gnT8hz8ookZJvskj1ySMZEBE+Dd8HHYD98Fn4ID8JPl6lh0Hsekz8inFwA8ZHLsg==</latexit><latexit sha1_base64="pLSlOG7zC6X4LedQAUAnUMB7WHg=">AAACnnichVFNb9QwEHXCR0v46AJHOBi2IE6rpJeWA6iCA5VoRZG6dKX1snKcycaq4wR7Ao2s/C1+SG/9N/VuIwEtiJEsPb03b2Y8k9ZKWozj8yC8cfPW7bX1O9Hde/cfbAwePvpiq8YIGItKVWaScgtKahijRAWT2gAvUwXH6cn7pX78HYyVlT7CtoZZyRda5lJw9NR88HMzYgVH13b0DWUKcmQuYikspHbcGN52zpguiulLyhBO0ZSOypx21LE0pz+6r0crcNotzd9+5WBhwBaVyrzAouT/9rf/8jLQWT9KxIxcFDiKNueDYTyKV0Gvg6QHQ9LH4XxwxrJKNCVoFIpbO03iGme+LkqhwFduLNRcnPAFTD3UvAQ7c6v9dvSFZzKaV8Y/jXTF/u5wvLS2LVOfWXIs7FVtSf5NmzaY78yc1HWDoMVlo7xRFCu6PBbNpAGBqvWACyP9rFQU3HCB/qSRX0Jy9cvXwXhr9HoUf94a7m7321gnT8hz8ookZJvskj1ySMZEBE+Dd8HHYD98Fn4ID8JPl6lh0Hsekz8inFwA8ZHLsg==</latexit>

WTX = THRESH

WTX - THRESH = o

o→ -

= 0I →

The Perceptron

⇢0 if wTx+ b 01 if wTx+ b > 0

<latexit sha1_base64="d3Pw5q+8jIlGM4xpzrnffbc+yco=">AAACgXicjVFNb9QwEHVCgRK+FjhyGbEtQiAFZy8FIVAlLhyL1KWV1svK8U52rTpOsCfQKMoP4W9x49eAd5sDtBwYydLTe/PlN3lttCfOf0bxtZ3rN27u3kpu37l77/7owcNPvmqcwqmqTOVOc+nRaItT0mTwtHYoy9zgSX72fqOffEXndWWPqa1xXsqV1YVWkgK1GH3fS8RaUtf28BaEwYJEl4gcV9p20jnZ9p1zfcLhKQjCc3JlB7qAHjqRF/Ct/3y8Bec9vIAcNh2+AAchIMn+r+Qd8ESgXQ7TEuH0ak1psrcYjXnKtwFXQTaAMRviaDH6IZaVakq0pIz0fpbxmuahL2llMHRuPNZSnckVzgK0skQ/77YW9rAfmCUUlQvPEmzZPys6WXrflnnILCWt/WVtQ/5LmzVUvJp32tYNoVUXg4rGAFWwuQcstUNFpg1AKqfDrqDW0klF4WpJMCG7/OWrYDpJX6f842R8eDC4scsesyfsGcvYATtkH9gRmzLFfkX7URq9jHfi5zGPJxepcTTUPGJ/RfzmN2SzvSc=</latexit><latexit sha1_base64="d3Pw5q+8jIlGM4xpzrnffbc+yco=">AAACgXicjVFNb9QwEHVCgRK+FjhyGbEtQiAFZy8FIVAlLhyL1KWV1svK8U52rTpOsCfQKMoP4W9x49eAd5sDtBwYydLTe/PlN3lttCfOf0bxtZ3rN27u3kpu37l77/7owcNPvmqcwqmqTOVOc+nRaItT0mTwtHYoy9zgSX72fqOffEXndWWPqa1xXsqV1YVWkgK1GH3fS8RaUtf28BaEwYJEl4gcV9p20jnZ9p1zfcLhKQjCc3JlB7qAHjqRF/Ct/3y8Bec9vIAcNh2+AAchIMn+r+Qd8ESgXQ7TEuH0ak1psrcYjXnKtwFXQTaAMRviaDH6IZaVakq0pIz0fpbxmuahL2llMHRuPNZSnckVzgK0skQ/77YW9rAfmCUUlQvPEmzZPys6WXrflnnILCWt/WVtQ/5LmzVUvJp32tYNoVUXg4rGAFWwuQcstUNFpg1AKqfDrqDW0klF4WpJMCG7/OWrYDpJX6f842R8eDC4scsesyfsGcvYATtkH9gRmzLFfkX7URq9jHfi5zGPJxepcTTUPGJ/RfzmN2SzvSc=</latexit><latexit sha1_base64="d3Pw5q+8jIlGM4xpzrnffbc+yco=">AAACgXicjVFNb9QwEHVCgRK+FjhyGbEtQiAFZy8FIVAlLhyL1KWV1svK8U52rTpOsCfQKMoP4W9x49eAd5sDtBwYydLTe/PlN3lttCfOf0bxtZ3rN27u3kpu37l77/7owcNPvmqcwqmqTOVOc+nRaItT0mTwtHYoy9zgSX72fqOffEXndWWPqa1xXsqV1YVWkgK1GH3fS8RaUtf28BaEwYJEl4gcV9p20jnZ9p1zfcLhKQjCc3JlB7qAHjqRF/Ct/3y8Bec9vIAcNh2+AAchIMn+r+Qd8ESgXQ7TEuH0ak1psrcYjXnKtwFXQTaAMRviaDH6IZaVakq0pIz0fpbxmuahL2llMHRuPNZSnckVzgK0skQ/77YW9rAfmCUUlQvPEmzZPys6WXrflnnILCWt/WVtQ/5LmzVUvJp32tYNoVUXg4rGAFWwuQcstUNFpg1AKqfDrqDW0klF4WpJMCG7/OWrYDpJX6f842R8eDC4scsesyfsGcvYATtkH9gRmzLFfkX7URq9jHfi5zGPJxepcTTUPGJ/RfzmN2SzvSc=</latexit>

The Perceptron

Simple Example: Learning ORy =

x1 0 1 0 1x2 0 0 1 1

x1 OR x2 0 1 1 1<latexit sha1_base64="UM8xfazAVtlDjQbsqgXmCAzAUIo=">AAACd3icbVHJTsMwEHXCHrYCx14sWhCnKumF5YTEhRuLKCA1VeW409bCcSJ7gqhCvoF/48aHcOGE2wYBhZFGfn5vxjOeiVIpDPr+m+POzS8sLi2veKtr6xubla3tW5NkmkOLJzLR9xEzIIWCFgqUcJ9qYHEk4S56OBvrd4+gjUjUDY5S6MRsoERfcIaW6lZevHrdCyMYCJUzrdmoyPkzt1Z4T92A7lPf+vcZhl44HNeiVm6W9FfIjGyvCE+ImNOLa1rQ74TgVwKoXlna9tKt1PyGPzH6FwQlqJHSLruV17CX8CwGhVwyY9qBn2LHPoiCSyi8MDOQMv7ABtC2ULEYTCefDK6ge5bp0X6irSukE/ZnRs5iY0ZxZCNjhkMzq43J/7R2hv2jTi5UmiEoPi3UzyTFhI63QHtCA0c5soBxLWyvlA+ZZhztrjw7hGD2y39Bq9k4bvhXzdrpSTmNZVIlu+SABOSQnJJzcklahJN3p+rUnT3nw6XuvnswDXWdMmeH/DI3+ATd3bFz</latexit><latexit sha1_base64="UM8xfazAVtlDjQbsqgXmCAzAUIo=">AAACd3icbVHJTsMwEHXCHrYCx14sWhCnKumF5YTEhRuLKCA1VeW409bCcSJ7gqhCvoF/48aHcOGE2wYBhZFGfn5vxjOeiVIpDPr+m+POzS8sLi2veKtr6xubla3tW5NkmkOLJzLR9xEzIIWCFgqUcJ9qYHEk4S56OBvrd4+gjUjUDY5S6MRsoERfcIaW6lZevHrdCyMYCJUzrdmoyPkzt1Z4T92A7lPf+vcZhl44HNeiVm6W9FfIjGyvCE+ImNOLa1rQ74TgVwKoXlna9tKt1PyGPzH6FwQlqJHSLruV17CX8CwGhVwyY9qBn2LHPoiCSyi8MDOQMv7ABtC2ULEYTCefDK6ge5bp0X6irSukE/ZnRs5iY0ZxZCNjhkMzq43J/7R2hv2jTi5UmiEoPi3UzyTFhI63QHtCA0c5soBxLWyvlA+ZZhztrjw7hGD2y39Bq9k4bvhXzdrpSTmNZVIlu+SABOSQnJJzcklahJN3p+rUnT3nw6XuvnswDXWdMmeH/DI3+ATd3bFz</latexit><latexit sha1_base64="UM8xfazAVtlDjQbsqgXmCAzAUIo=">AAACd3icbVHJTsMwEHXCHrYCx14sWhCnKumF5YTEhRuLKCA1VeW409bCcSJ7gqhCvoF/48aHcOGE2wYBhZFGfn5vxjOeiVIpDPr+m+POzS8sLi2veKtr6xubla3tW5NkmkOLJzLR9xEzIIWCFgqUcJ9qYHEk4S56OBvrd4+gjUjUDY5S6MRsoERfcIaW6lZevHrdCyMYCJUzrdmoyPkzt1Z4T92A7lPf+vcZhl44HNeiVm6W9FfIjGyvCE+ImNOLa1rQ74TgVwKoXlna9tKt1PyGPzH6FwQlqJHSLruV17CX8CwGhVwyY9qBn2LHPoiCSyi8MDOQMv7ABtC2ULEYTCefDK6ge5bp0X6irSukE/ZnRs5iY0ZxZCNjhkMzq43J/7R2hv2jTi5UmiEoPi3UzyTFhI63QHtCA0c5soBxLWyvlA+ZZhztrjw7hGD2y39Bq9k4bvhXzdrpSTmNZVIlu+SABOSQnJJzcklahJN3p+rUnT3nw6XuvnswDXWdMmeH/DI3+ATd3bFz</latexit>

ow=f, }b=o

1 wt⇒ly1t,b=t'1511,1+0--2>0

The Perceptron

Simple Example: Learning ANDy =

x1 0 1 0 1x2 0 0 1 1

x1 AND x2 0 0 0 1<latexit sha1_base64="jXeA1icTPY9SMyKMFtVfAhfiWlw=">AAACd3icbVFNT9tAEF2bFqihEOCYy6pxopwiOxc+TiB66KlKpaYgxVG03kySVdZra3eMEhn/Bv4bN35ILz2xCa4ooSON9u1787E7E2dSGAyCJ8fd+vBxe2f3k7e3//ngsHZ0/MukuebQ56lM9W3MDEihoI8CJdxmGlgSS7iJ59cr/eYOtBGp+onLDIYJmyoxEZyhpUa1B9/3ohimQhVMa7YsC37PrZXeYhTSFg2sv55R5EWzVStq5W5F/w3ZkO0VYYGIBb36/pWW9G3Ga0FQ46q35/ujWiPoBGuj70FYgQaprDeqPUbjlOcJKOSSGTMIgwyHtiAKLqH0otxAxvicTWFgoWIJmGGxHlxJm5YZ00mqrSuka/bfjIIlxiyT2EYmDGdmU1uR/9MGOU7OhoVQWY6g+EujSS4ppnS1BToWGjjKpQWMa2HfSvmMacbR7sqzQwg3v/we9Lud807wo9u4vKimsUvq5Atpk5CckkvyjfRIn3Dy26k7vtN0/rjUbbntl1DXqXJOyBtzw2c1q7GZ</latexit><latexit sha1_base64="jXeA1icTPY9SMyKMFtVfAhfiWlw=">AAACd3icbVFNT9tAEF2bFqihEOCYy6pxopwiOxc+TiB66KlKpaYgxVG03kySVdZra3eMEhn/Bv4bN35ILz2xCa4ooSON9u1787E7E2dSGAyCJ8fd+vBxe2f3k7e3//ngsHZ0/MukuebQ56lM9W3MDEihoI8CJdxmGlgSS7iJ59cr/eYOtBGp+onLDIYJmyoxEZyhpUa1B9/3ohimQhVMa7YsC37PrZXeYhTSFg2sv55R5EWzVStq5W5F/w3ZkO0VYYGIBb36/pWW9G3Ga0FQ46q35/ujWiPoBGuj70FYgQaprDeqPUbjlOcJKOSSGTMIgwyHtiAKLqH0otxAxvicTWFgoWIJmGGxHlxJm5YZ00mqrSuka/bfjIIlxiyT2EYmDGdmU1uR/9MGOU7OhoVQWY6g+EujSS4ppnS1BToWGjjKpQWMa2HfSvmMacbR7sqzQwg3v/we9Lud807wo9u4vKimsUvq5Atpk5CckkvyjfRIn3Dy26k7vtN0/rjUbbntl1DXqXJOyBtzw2c1q7GZ</latexit><latexit sha1_base64="jXeA1icTPY9SMyKMFtVfAhfiWlw=">AAACd3icbVFNT9tAEF2bFqihEOCYy6pxopwiOxc+TiB66KlKpaYgxVG03kySVdZra3eMEhn/Bv4bN35ILz2xCa4ooSON9u1787E7E2dSGAyCJ8fd+vBxe2f3k7e3//ngsHZ0/MukuebQ56lM9W3MDEihoI8CJdxmGlgSS7iJ59cr/eYOtBGp+onLDIYJmyoxEZyhpUa1B9/3ohimQhVMa7YsC37PrZXeYhTSFg2sv55R5EWzVStq5W5F/w3ZkO0VYYGIBb36/pWW9G3Ga0FQ46q35/ujWiPoBGuj70FYgQaprDeqPUbjlOcJKOSSGTMIgwyHtiAKLqH0otxAxvicTWFgoWIJmGGxHlxJm5YZ00mqrSuka/bfjIIlxiyT2EYmDGdmU1uR/9MGOU7OhoVQWY6g+EujSS4ppnS1BToWGjjKpQWMa2HfSvmMacbR7sqzQwg3v/we9Lud807wo9u4vKimsUvq5Atpk5CckkvyjfRIn3Dy26k7vtN0/rjUbbntl1DXqXJOyBtzw2c1q7GZ</latexit>: w=[1,1 & b= -1

The Perceptron

Simple Example: Learning NANDy =

x1 0 1 0 1x2 0 0 1 1

x1 NAND x2 1 1 1 0<latexit sha1_base64="2vKppGvl1XGFf2Q25MWaYD7QWT4=">AAACeXicbVFNT+MwEHUCy0dYoLscEZJFgUUcqqQcluUEggMnBBIFpKaqHHfaWnWcyJ4gqpAfwV/jxi/hwgG3DYItjDTy83sznvFMlEph0PefHXdm9sfc/MKit/RzeWW18uv3tUkyzaHBE5no24gZkEJBAwVKuE01sDiScBMNTkb6zR1oIxJ1hcMUWjHrKdEVnKGl2pVHb2vLCyPoCZUzrdmwyPkDt1Z49+2A7lDf+scZhl7YH9WiVq6X9HvIlGyvCPeImNPz4/NTWtBJSvDJ/XEKqE5Z3HbTrlT9mj82+hUEJaiS0i7alaewk/AsBoVcMmOagZ9iyz6IgksovDAzkDI+YD1oWqhYDKaVj0dX0G3LdGg30dYV0jH7OSNnsTHDOLKRMcO+mdZG5HdaM8PuQSsXKs0QFJ8U6maSYkJHe6AdoYGjHFrAuBa2V8r7TDOOdlueHUIw/eWvoFGv/av5l/Xq0WE5jQWyTjbJLgnIX3JEzsgFaRBOXpwNZ8f547y6m+6uuzcJdZ0yZ438Z+7+Gzi0sgc=</latexit><latexit sha1_base64="2vKppGvl1XGFf2Q25MWaYD7QWT4=">AAACeXicbVFNT+MwEHUCy0dYoLscEZJFgUUcqqQcluUEggMnBBIFpKaqHHfaWnWcyJ4gqpAfwV/jxi/hwgG3DYItjDTy83sznvFMlEph0PefHXdm9sfc/MKit/RzeWW18uv3tUkyzaHBE5no24gZkEJBAwVKuE01sDiScBMNTkb6zR1oIxJ1hcMUWjHrKdEVnKGl2pVHb2vLCyPoCZUzrdmwyPkDt1Z49+2A7lDf+scZhl7YH9WiVq6X9HvIlGyvCPeImNPz4/NTWtBJSvDJ/XEKqE5Z3HbTrlT9mj82+hUEJaiS0i7alaewk/AsBoVcMmOagZ9iyz6IgksovDAzkDI+YD1oWqhYDKaVj0dX0G3LdGg30dYV0jH7OSNnsTHDOLKRMcO+mdZG5HdaM8PuQSsXKs0QFJ8U6maSYkJHe6AdoYGjHFrAuBa2V8r7TDOOdlueHUIw/eWvoFGv/av5l/Xq0WE5jQWyTjbJLgnIX3JEzsgFaRBOXpwNZ8f547y6m+6uuzcJdZ0yZ438Z+7+Gzi0sgc=</latexit><latexit sha1_base64="2vKppGvl1XGFf2Q25MWaYD7QWT4=">AAACeXicbVFNT+MwEHUCy0dYoLscEZJFgUUcqqQcluUEggMnBBIFpKaqHHfaWnWcyJ4gqpAfwV/jxi/hwgG3DYItjDTy83sznvFMlEph0PefHXdm9sfc/MKit/RzeWW18uv3tUkyzaHBE5no24gZkEJBAwVKuE01sDiScBMNTkb6zR1oIxJ1hcMUWjHrKdEVnKGl2pVHb2vLCyPoCZUzrdmwyPkDt1Z49+2A7lDf+scZhl7YH9WiVq6X9HvIlGyvCPeImNPz4/NTWtBJSvDJ/XEKqE5Z3HbTrlT9mj82+hUEJaiS0i7alaewk/AsBoVcMmOagZ9iyz6IgksovDAzkDI+YD1oWqhYDKaVj0dX0G3LdGg30dYV0jH7OSNnsTHDOLKRMcO+mdZG5HdaM8PuQSsXKs0QFJ8U6maSYkJHe6AdoYGjHFrAuBa2V8r7TDOOdlueHUIw/eWvoFGv/av5l/Xq0WE5jQWyTjbJLgnIX3JEzsgFaRBOXpwNZ8f547y6m+6uuzcJdZ0yZ438Z+7+Gzi0sgc=</latexit>I wtxtbs El - 1 3 'Ll, ) t 1.5

The Perceptron

Simple Example: Can we learn XOR?y =

x1 0 1 0 1x2 0 0 1 1

x1 XOR x2 0 1 1 0<latexit sha1_base64="7KwtoelifvaVe1T374Vf/hf8/1A=">AAACd3icbVFLT9tAEF67LQ9DaWiPXFbEQZwiOxcKJ6ReuPFQQyLFUbTeTJIV67W1O0ZExr+h/40bP4QLp66DadqkI4322++bx+5MnElhMAieHffDx08bm1vb3s7u570vjf2vtybNNYcuT2Wq+zEzIIWCLgqU0M80sCSW0IvvflR67x60Ean6ifMMhgmbKjERnKGlRo1fvu9FMUyFKpjWbF4W/JFbK72HUUiPaGB9eUaRF82qVtTKnZp+D1mR7RXhAREL2r+8oSVdZoR/ilYZoMZ1b8/3R41m0A4WRtdBWIMmqe1q1HiKxinPE1DIJTNmEAYZDm1BFFxC6UW5gYzxOzaFgYWKJWCGxWJwJW1ZZkwnqbaukC7YvzMKlhgzT2IbmTCcmVWtIv+nDXKcfB8WQmU5guJvjSa5pJjSagt0LDRwlHMLGNfCvpXyGdOMo92VZ4cQrn55HXQ77dN2cN1pnp/V09giB+SQHJOQnJBzckGuSJdw8uIcOL7Tcl5d6h65x2+hrlPnfCP/mBv+BnbdscA=</latexit><latexit sha1_base64="7KwtoelifvaVe1T374Vf/hf8/1A=">AAACd3icbVFLT9tAEF67LQ9DaWiPXFbEQZwiOxcKJ6ReuPFQQyLFUbTeTJIV67W1O0ZExr+h/40bP4QLp66DadqkI4322++bx+5MnElhMAieHffDx08bm1vb3s7u570vjf2vtybNNYcuT2Wq+zEzIIWCLgqU0M80sCSW0IvvflR67x60Ean6ifMMhgmbKjERnKGlRo1fvu9FMUyFKpjWbF4W/JFbK72HUUiPaGB9eUaRF82qVtTKnZp+D1mR7RXhAREL2r+8oSVdZoR/ilYZoMZ1b8/3R41m0A4WRtdBWIMmqe1q1HiKxinPE1DIJTNmEAYZDm1BFFxC6UW5gYzxOzaFgYWKJWCGxWJwJW1ZZkwnqbaukC7YvzMKlhgzT2IbmTCcmVWtIv+nDXKcfB8WQmU5guJvjSa5pJjSagt0LDRwlHMLGNfCvpXyGdOMo92VZ4cQrn55HXQ77dN2cN1pnp/V09giB+SQHJOQnJBzckGuSJdw8uIcOL7Tcl5d6h65x2+hrlPnfCP/mBv+BnbdscA=</latexit><latexit sha1_base64="7KwtoelifvaVe1T374Vf/hf8/1A=">AAACd3icbVFLT9tAEF67LQ9DaWiPXFbEQZwiOxcKJ6ReuPFQQyLFUbTeTJIV67W1O0ZExr+h/40bP4QLp66DadqkI4322++bx+5MnElhMAieHffDx08bm1vb3s7u570vjf2vtybNNYcuT2Wq+zEzIIWCLgqU0M80sCSW0IvvflR67x60Ean6ifMMhgmbKjERnKGlRo1fvu9FMUyFKpjWbF4W/JFbK72HUUiPaGB9eUaRF82qVtTKnZp+D1mR7RXhAREL2r+8oSVdZoR/ilYZoMZ1b8/3R41m0A4WRtdBWIMmqe1q1HiKxinPE1DIJTNmEAYZDm1BFFxC6UW5gYzxOzaFgYWKJWCGxWJwJW1ZZkwnqbaukC7YvzMKlhgzT2IbmTCcmVWtIv+nDXKcfB8WQmU5guJvjSa5pJjSagt0LDRwlHMLGNfCvpXyGdOMo92VZ4cQrn55HXQ77dN2cN1pnp/V09giB+SQHJOQnJBzckGuSJdw8uIcOL7Tcl5d6h65x2+hrlPnfCP/mBv+BnbdscA=</latexit>

41 wtx = +1 )

Wtxtb EO

The Perceptron

Simple Example: Can we learn XOR?

x1 0 1 0 1x2 0 0 1 1

The Perceptron

Question: Why can’t we learn XOR?

OR AND

NAND XOR

The Perceptron

Question: Why can’t we learn XOR?

Answer: The perceptron is a linear classifier. Can only learn things that are linearly separable

Question: How can we fix this?

Answer: The Multilayer Perceptron

Anatomy: o Input Layer

o Hidden Layer

o Output Layera.12 bz

onW 22=

Anatomy: Input Layer to Hidden Layer

o Define intermediate activities z

o Define activations a

The indicator I is the activation function

z1 = w111x1 + w1

12x2 + b11, a1 = I(z1 > 0)

z2 = w121x1 + w1

22x2 + b12, a2 = I(z2 > 0)<latexit sha1_base64="LbR5oqBU2cyj5PiNOTK0lNmSgd8=">AAACkHicbZHdTtswFMedDFgXBnTjcjfWyiQQqIqtSdBJsE7cMK6KtAJSWyLHccHC+Zh9stFFeZ89z+54G5y0Qm3ZkSz/z9fP9nGYKWnA9x8d99XK6trrxhtv/e3G5lbz3ftLk+aaiz5PVaqvQ2aEkonogwQlrjMtWBwqcRXen1b5q19CG5kmP2CSiVHMbhM5lpyBDQXNvzs73p+A4GP8OygIKW/Ig91LvF/7dOrTyg8DckMO8PBnziLM6pYhiAfQcfG93K0YJ9jfw54FejWUTqF0CUoXoXQeSpeh9BkaNFt+268NvxRkJlpoZr2g+W8YpTyPRQJcMWMGxM9gVDANkitResPciIzxe3YrBlYmLBZmVNQTLfEnG4nwONV2JYDr6HxHwWJjJnFoK2MGd2Y5VwX/lxvkMD4aFTLJchAJnx40zhWGFFffgyOpBQc1sYJxLe1dMb9jmnGwn+jZIZDlJ78UfdrutP0L2up+mU2jgT6gj2gXEXSIuugM9VAfcWfT+ewcOyfutttxv7rfpqWuM+vZRgvmnj8BEEq6wQ==</latexit><latexit sha1_base64="LbR5oqBU2cyj5PiNOTK0lNmSgd8=">AAACkHicbZHdTtswFMedDFgXBnTjcjfWyiQQqIqtSdBJsE7cMK6KtAJSWyLHccHC+Zh9stFFeZ89z+54G5y0Qm3ZkSz/z9fP9nGYKWnA9x8d99XK6trrxhtv/e3G5lbz3ftLk+aaiz5PVaqvQ2aEkonogwQlrjMtWBwqcRXen1b5q19CG5kmP2CSiVHMbhM5lpyBDQXNvzs73p+A4GP8OygIKW/Ig91LvF/7dOrTyg8DckMO8PBnziLM6pYhiAfQcfG93K0YJ9jfw54FejWUTqF0CUoXoXQeSpeh9BkaNFt+268NvxRkJlpoZr2g+W8YpTyPRQJcMWMGxM9gVDANkitResPciIzxe3YrBlYmLBZmVNQTLfEnG4nwONV2JYDr6HxHwWJjJnFoK2MGd2Y5VwX/lxvkMD4aFTLJchAJnx40zhWGFFffgyOpBQc1sYJxLe1dMb9jmnGwn+jZIZDlJ78UfdrutP0L2up+mU2jgT6gj2gXEXSIuugM9VAfcWfT+ewcOyfutttxv7rfpqWuM+vZRgvmnj8BEEq6wQ==</latexit><latexit sha1_base64="LbR5oqBU2cyj5PiNOTK0lNmSgd8=">AAACkHicbZHdTtswFMedDFgXBnTjcjfWyiQQqIqtSdBJsE7cMK6KtAJSWyLHccHC+Zh9stFFeZ89z+54G5y0Qm3ZkSz/z9fP9nGYKWnA9x8d99XK6trrxhtv/e3G5lbz3ftLk+aaiz5PVaqvQ2aEkonogwQlrjMtWBwqcRXen1b5q19CG5kmP2CSiVHMbhM5lpyBDQXNvzs73p+A4GP8OygIKW/Ig91LvF/7dOrTyg8DckMO8PBnziLM6pYhiAfQcfG93K0YJ9jfw54FejWUTqF0CUoXoXQeSpeh9BkaNFt+268NvxRkJlpoZr2g+W8YpTyPRQJcMWMGxM9gVDANkitResPciIzxe3YrBlYmLBZmVNQTLfEnG4nwONV2JYDr6HxHwWJjJnFoK2MGd2Y5VwX/lxvkMD4aFTLJchAJnx40zhWGFFffgyOpBQc1sYJxLe1dMb9jmnGwn+jZIZDlJ78UfdrutP0L2up+mU2jgT6gj2gXEXSIuugM9VAfcWfT+ewcOyfutttxv7rfpqWuM+vZRgvmnj8BEEq6wQ==</latexit>

:#=¥ I

Vectorized Anatomy: Input Layer to Hidden Layer

becomes

z1 = w111x1 + w1

12x2 + b11, a1 = I(z1 > 0)z2 = w1

21x1 + w122x2 + b12, a2 = I(z2 > 0)

<latexit sha1_base64="if1dw1K6ibWbAraQM+La//QnF5A=">AAACqHicbZHLbtQwFIadlEsJlw6wZGMxAyoqGsXecJGKKrEBiUUrkU7RZBo5zpmpVccJttN2iPJuPAM73gbnsminHMnyf26f7eO0lMLYMPzr+Vt37t67v/0gePjo8ZOd0dNnx6aoNIeIF7LQJykzIIWCyAor4aTUwPJUwiw9/9zmZxegjSjUd7suYZGzlRJLwZl1oWT0ezIJ4hRWQtVMa7ZuatkEvxKC9/FlUhPSnJIrtzd4r/Np79PWTxNySt7i+GfFMsy6ltjCldV5/bXZbRmfcPgGx7Hj0Z5HN3j0Jo9e59FNHu15QQwqGy4bTCbJaBxOw87wbUEGMUaDHSajP3FW8CoHZblkxsxJWNqFA1rBJTRBXBkoGT9nK5g7qVgOZlF3k27wKxfJ8LLQbimLu+j1jprlxqzz1FXmzJ6ZzVwb/F9uXtnl+0UtVFlZULw/aFlJbAvcfhvOhAZu5doJxrVwd8X8jGnGrfvcwA2BbD75tojo9MM0PKLjg4/DNLbRC/QS7SKC3qED9AUdoghx77X3zYu8Y3/PP/Jn/o++1PeGnufohvnpPxh3x1U=</latexit><latexit sha1_base64="if1dw1K6ibWbAraQM+La//QnF5A=">AAACqHicbZHLbtQwFIadlEsJlw6wZGMxAyoqGsXecJGKKrEBiUUrkU7RZBo5zpmpVccJttN2iPJuPAM73gbnsminHMnyf26f7eO0lMLYMPzr+Vt37t67v/0gePjo8ZOd0dNnx6aoNIeIF7LQJykzIIWCyAor4aTUwPJUwiw9/9zmZxegjSjUd7suYZGzlRJLwZl1oWT0ezIJ4hRWQtVMa7ZuatkEvxKC9/FlUhPSnJIrtzd4r/Np79PWTxNySt7i+GfFMsy6ltjCldV5/bXZbRmfcPgGx7Hj0Z5HN3j0Jo9e59FNHu15QQwqGy4bTCbJaBxOw87wbUEGMUaDHSajP3FW8CoHZblkxsxJWNqFA1rBJTRBXBkoGT9nK5g7qVgOZlF3k27wKxfJ8LLQbimLu+j1jprlxqzz1FXmzJ6ZzVwb/F9uXtnl+0UtVFlZULw/aFlJbAvcfhvOhAZu5doJxrVwd8X8jGnGrfvcwA2BbD75tojo9MM0PKLjg4/DNLbRC/QS7SKC3qED9AUdoghx77X3zYu8Y3/PP/Jn/o++1PeGnufohvnpPxh3x1U=</latexit><latexit sha1_base64="if1dw1K6ibWbAraQM+La//QnF5A=">AAACqHicbZHLbtQwFIadlEsJlw6wZGMxAyoqGsXecJGKKrEBiUUrkU7RZBo5zpmpVccJttN2iPJuPAM73gbnsminHMnyf26f7eO0lMLYMPzr+Vt37t67v/0gePjo8ZOd0dNnx6aoNIeIF7LQJykzIIWCyAor4aTUwPJUwiw9/9zmZxegjSjUd7suYZGzlRJLwZl1oWT0ezIJ4hRWQtVMa7ZuatkEvxKC9/FlUhPSnJIrtzd4r/Np79PWTxNySt7i+GfFMsy6ltjCldV5/bXZbRmfcPgGx7Hj0Z5HN3j0Jo9e59FNHu15QQwqGy4bTCbJaBxOw87wbUEGMUaDHSajP3FW8CoHZblkxsxJWNqFA1rBJTRBXBkoGT9nK5g7qVgOZlF3k27wKxfJ8LLQbimLu+j1jprlxqzz1FXmzJ6ZzVwb/F9uXtnl+0UtVFlZULw/aFlJbAvcfhvOhAZu5doJxrVwd8X8jGnGrfvcwA2BbD75tojo9MM0PKLjg4/DNLbRC/QS7SKC3qED9AUdoghx77X3zYu8Y3/PP/Jn/o++1PeGnufohvnpPxh3x1U=</latexit>

z2 = W 1x1 + b1

a2 = I(z2 > 0) = I(W 1x1 + b1 > 0)<latexit sha1_base64="bFJTMLtnFcASWTm+WeQFNfaTtV8=">AAACiXicdVHLTttAFB275VFDIYUlm1ETEBUisrPgJYGQ2LQ7KjUNUpxE48l1GDEeWzPXFanlj+GX2PE3jI2r8mivNNLROec+5t4ok8Kg7z847rv3C4tLyx+8ldWPa+utTxs/TZprDn2eylRfRcyAFAr6KFDCVaaBJZGEQXRzUemDX6CNSNUPnGcwSthMiVhwhpaatO46HS+MYCZUwbRm87LQXJZeEUYx/V2Oe3TndIcOxkFN3JbjgO7RGkcVDsMnJ/vjDBFuUSfFt3L3b4kz6n+hp8+1/xasrF4IatpM43U6k1bb7/p10LcgaECbNHE5ad2H05TnCSjkkhkzDPwMR7YgCi6h9MLcQMb4DZvB0ELFEjCjol5lSbctM6Vxqu1TSGv2eUbBEmPmSWSdCcNr81qryH9pwxzjo1EhVJYjKP7UKM4lxZRWd6FToYGjnFvAuBZ2VsqvmWYc7fU8u4Tg9Zffgn6ve9z1v/fa5yfNNpbJFvlMdklADsk5+UouSZ9wZ8nZdw6cQ3fV7blHbuN1nSZnk7wI9+IRCR69hg==</latexit><latexit sha1_base64="bFJTMLtnFcASWTm+WeQFNfaTtV8=">AAACiXicdVHLTttAFB275VFDIYUlm1ETEBUisrPgJYGQ2LQ7KjUNUpxE48l1GDEeWzPXFanlj+GX2PE3jI2r8mivNNLROec+5t4ok8Kg7z847rv3C4tLyx+8ldWPa+utTxs/TZprDn2eylRfRcyAFAr6KFDCVaaBJZGEQXRzUemDX6CNSNUPnGcwSthMiVhwhpaatO46HS+MYCZUwbRm87LQXJZeEUYx/V2Oe3TndIcOxkFN3JbjgO7RGkcVDsMnJ/vjDBFuUSfFt3L3b4kz6n+hp8+1/xasrF4IatpM43U6k1bb7/p10LcgaECbNHE5ad2H05TnCSjkkhkzDPwMR7YgCi6h9MLcQMb4DZvB0ELFEjCjol5lSbctM6Vxqu1TSGv2eUbBEmPmSWSdCcNr81qryH9pwxzjo1EhVJYjKP7UKM4lxZRWd6FToYGjnFvAuBZ2VsqvmWYc7fU8u4Tg9Zffgn6ve9z1v/fa5yfNNpbJFvlMdklADsk5+UouSZ9wZ8nZdw6cQ3fV7blHbuN1nSZnk7wI9+IRCR69hg==</latexit><latexit sha1_base64="bFJTMLtnFcASWTm+WeQFNfaTtV8=">AAACiXicdVHLTttAFB275VFDIYUlm1ETEBUisrPgJYGQ2LQ7KjUNUpxE48l1GDEeWzPXFanlj+GX2PE3jI2r8mivNNLROec+5t4ok8Kg7z847rv3C4tLyx+8ldWPa+utTxs/TZprDn2eylRfRcyAFAr6KFDCVaaBJZGEQXRzUemDX6CNSNUPnGcwSthMiVhwhpaatO46HS+MYCZUwbRm87LQXJZeEUYx/V2Oe3TndIcOxkFN3JbjgO7RGkcVDsMnJ/vjDBFuUSfFt3L3b4kz6n+hp8+1/xasrF4IatpM43U6k1bb7/p10LcgaECbNHE5ad2H05TnCSjkkhkzDPwMR7YgCi6h9MLcQMb4DZvB0ELFEjCjol5lSbctM6Vxqu1TSGv2eUbBEmPmSWSdCcNr81qryH9pwxzjo1EhVJYjKP7UKM4lxZRWd6FToYGjnFvAuBZ2VsqvmWYc7fU8u4Tg9Zffgn6ve9z1v/fa5yfNNpbJFvlMdklADsk5+UouSZ9wZ8nZdw6cQ3fV7blHbuN1nSZnk7wI9+IRCR69hg==</latexit>

11 w112

w121 w1

�, b1 =

b11b12

<latexit sha1_base64="0kMo3/CYxOAQiq06yYUmE5mNBR8=">AAACnXicbZHfatswFMZld1s77V/WXfZGLNnYxQi2b9YOCi2DMtYyOliWQuwZST5ORGXZleSWYPxYfZHd7W0mJ4ZtbQ4IfnznnO9IR6ySwtgg+O35Ww8ePtreeYyfPH32/MXg5e4PU9aaw4SXstQXjBqQQsHECivhotJACyZhyi4/dfnpNWgjSvXdLitICjpXIhecWielg9vRCE9/huSQ4FhCbmc4ZjAXqqFa02XbcN7im7QJw9YVvSUdRh3GcSdHf+Wok3EMKutbcazFfGGT9875qqYZwU3McsLa1bTNw1rM0nDtztLIwSZDPBqlg2EwDlZB7kPYwxD1cZ4OfsVZyesClOWSGjMLg8omztgKLsFZ1wYqyi/pHGYOFS3AJM1qvS1545SM5KV2R1myUv/taGhhzLJgrrKgdmHu5jpxU25W23w/aYSqaguKrwfltSS2JN1fkUxo4FYuHVCuhbsr4QuqKbfuR7FbQnj3yfdhEo0PxsG3aHj0sd/GDtpDr9E7FKIP6Ah9Rudogri35x17X7xTn/gn/pn/dV3qe33PK/Rf+NM/KSnFow==</latexit><latexit sha1_base64="0kMo3/CYxOAQiq06yYUmE5mNBR8=">AAACnXicbZHfatswFMZld1s77V/WXfZGLNnYxQi2b9YOCi2DMtYyOliWQuwZST5ORGXZleSWYPxYfZHd7W0mJ4ZtbQ4IfnznnO9IR6ySwtgg+O35Ww8ePtreeYyfPH32/MXg5e4PU9aaw4SXstQXjBqQQsHECivhotJACyZhyi4/dfnpNWgjSvXdLitICjpXIhecWielg9vRCE9/huSQ4FhCbmc4ZjAXqqFa02XbcN7im7QJw9YVvSUdRh3GcSdHf+Wok3EMKutbcazFfGGT9875qqYZwU3McsLa1bTNw1rM0nDtztLIwSZDPBqlg2EwDlZB7kPYwxD1cZ4OfsVZyesClOWSGjMLg8omztgKLsFZ1wYqyi/pHGYOFS3AJM1qvS1545SM5KV2R1myUv/taGhhzLJgrrKgdmHu5jpxU25W23w/aYSqaguKrwfltSS2JN1fkUxo4FYuHVCuhbsr4QuqKbfuR7FbQnj3yfdhEo0PxsG3aHj0sd/GDtpDr9E7FKIP6Ah9Rudogri35x17X7xTn/gn/pn/dV3qe33PK/Rf+NM/KSnFow==</latexit><latexit sha1_base64="0kMo3/CYxOAQiq06yYUmE5mNBR8=">AAACnXicbZHfatswFMZld1s77V/WXfZGLNnYxQi2b9YOCi2DMtYyOliWQuwZST5ORGXZleSWYPxYfZHd7W0mJ4ZtbQ4IfnznnO9IR6ySwtgg+O35Ww8ePtreeYyfPH32/MXg5e4PU9aaw4SXstQXjBqQQsHECivhotJACyZhyi4/dfnpNWgjSvXdLitICjpXIhecWielg9vRCE9/huSQ4FhCbmc4ZjAXqqFa02XbcN7im7QJw9YVvSUdRh3GcSdHf+Wok3EMKutbcazFfGGT9875qqYZwU3McsLa1bTNw1rM0nDtztLIwSZDPBqlg2EwDlZB7kPYwxD1cZ4OfsVZyesClOWSGjMLg8omztgKLsFZ1wYqyi/pHGYOFS3AJM1qvS1545SM5KV2R1myUv/taGhhzLJgrrKgdmHu5jpxU25W23w/aYSqaguKrwfltSS2JN1fkUxo4FYuHVCuhbsr4QuqKbfuR7FbQnj3yfdhEo0PxsG3aHj0sd/GDtpDr9E7FKIP6Ah9Rudogri35x17X7xTn/gn/pn/dV3qe33PK/Rf+NM/KSnFow==</latexit>

× - [ It ] WE

[ wiwii ]b2= Ibf }

Vectorized Anatomy: Hidden Layer to Output Layer

z3 = W 2a2 + b2

y = I(z3 > 0) = I(W 2a2 + b2 > 0)<latexit sha1_base64="ASe3O1wbpXNcLSB0W+Iovcd8mnM=">AAACiHicdVFdb9MwFHUCjJEBK/DIi0XLNASqkiLYhzQ0wQu8DYnSSXVbOe5Na81xIvsGLUTZb+E38ca/wckiMTa4kqWjc8798L1xrqTFMPzl+bdu39m4u3kv2Lr/4OF279HjrzYrjICxyFRmTmNuQUkNY5So4DQ3wNNYwSQ++9Dok29grMz0FyxzmKV8pWUiBUdHLXo/BoOAxbCSuuLG8LKujFB1ULE4od/r+Wu6c7RDJ/NRS/B6PqIvaYvjBjMWsDXHqqwvWiNDOEeTVp/q3T8V3tHwBT26qv23XmMNGOhlN0wwGCx6/XAYtkFvgqgDfdLFyaL3ky0zUaSgUShu7TQKc5y5giiFgjpghYWcizO+gqmDmqdgZ1W7yZo+d8ySJplxTyNt2asZFU+tLdPYOVOOa3tda8h/adMCk/1ZJXVeIGhx2SgpFMWMNmehS2lAoCod4MJINysVa264QHe8wC0huv7lm2A8Gh4Mw8+j/vFht41N8pQ8I7skInvkmHwkJ2RMhLfhvfLeeG/9LT/y9/yDS6vvdTlPyF/hv/8NMF69sQ==</latexit><latexit sha1_base64="ASe3O1wbpXNcLSB0W+Iovcd8mnM=">AAACiHicdVFdb9MwFHUCjJEBK/DIi0XLNASqkiLYhzQ0wQu8DYnSSXVbOe5Na81xIvsGLUTZb+E38ca/wckiMTa4kqWjc8798L1xrqTFMPzl+bdu39m4u3kv2Lr/4OF279HjrzYrjICxyFRmTmNuQUkNY5So4DQ3wNNYwSQ++9Dok29grMz0FyxzmKV8pWUiBUdHLXo/BoOAxbCSuuLG8LKujFB1ULE4od/r+Wu6c7RDJ/NRS/B6PqIvaYvjBjMWsDXHqqwvWiNDOEeTVp/q3T8V3tHwBT26qv23XmMNGOhlN0wwGCx6/XAYtkFvgqgDfdLFyaL3ky0zUaSgUShu7TQKc5y5giiFgjpghYWcizO+gqmDmqdgZ1W7yZo+d8ySJplxTyNt2asZFU+tLdPYOVOOa3tda8h/adMCk/1ZJXVeIGhx2SgpFMWMNmehS2lAoCod4MJINysVa264QHe8wC0huv7lm2A8Gh4Mw8+j/vFht41N8pQ8I7skInvkmHwkJ2RMhLfhvfLeeG/9LT/y9/yDS6vvdTlPyF/hv/8NMF69sQ==</latexit><latexit sha1_base64="ASe3O1wbpXNcLSB0W+Iovcd8mnM=">AAACiHicdVFdb9MwFHUCjJEBK/DIi0XLNASqkiLYhzQ0wQu8DYnSSXVbOe5Na81xIvsGLUTZb+E38ca/wckiMTa4kqWjc8798L1xrqTFMPzl+bdu39m4u3kv2Lr/4OF279HjrzYrjICxyFRmTmNuQUkNY5So4DQ3wNNYwSQ++9Dok29grMz0FyxzmKV8pWUiBUdHLXo/BoOAxbCSuuLG8LKujFB1ULE4od/r+Wu6c7RDJ/NRS/B6PqIvaYvjBjMWsDXHqqwvWiNDOEeTVp/q3T8V3tHwBT26qv23XmMNGOhlN0wwGCx6/XAYtkFvgqgDfdLFyaL3ky0zUaSgUShu7TQKc5y5giiFgjpghYWcizO+gqmDmqdgZ1W7yZo+d8ySJplxTyNt2asZFU+tLdPYOVOOa3tda8h/adMCk/1ZJXVeIGhx2SgpFMWMNmehS2lAoCod4MJINysVa264QHe8wC0huv7lm2A8Gh4Mw8+j/vFht41N8pQ8I7skInvkmHwkJ2RMhLfhvfLeeG/9LT/y9/yDS6vvdTlPyF/hv/8NMF69sQ==</latexit>

W 2 =⇥w2

11 w212

⇤, b2 =

⇥b21

⇤<latexit sha1_base64="VTIiy0b6DlHmPd9FAKlM8oFfRz4=">AAACfXicbVFNj9MwEHXC12K+Chy5GFpWCFZVkssCEtJKXDguEqUrNdnIdiattY4T7AmoivIz+GPc+C1ccLo5wG5HsvT05s2z/UY0WjmMot9BeOPmrdt3Du7Se/cfPHw0efzkq6tbK2Eha13bM8EdaGVggQo1nDUWeCU0LMXFx6G//A7Wqdp8wW0DWcXXRpVKcvRUPvk5m9HlecI+MJpqKHFFUwFrZTpuLd/2nZQ9/ZF3cdx70SEbYDJAmoIpRhFNrVpvMDvy7LeWF4x2qSiZ6He++217KvL4PNlnQ2ezfDKN5tGu2HUQj2BKxjrNJ7/SopZtBQal5s6t4qjBzBujkhq8deug4fKCr2HloeEVuKzbxdezl54pWFlbfwyyHfvvRMcr57aV8MqK48Zd7Q3kvt6qxfJt1inTtAhGXl5UtpphzYZdsEJZkKi3HnBplX8rkxtuuUS/MepDiK9++TpYJPN38+hzMj15P6ZxQJ6RF+QVickxOSGfyClZEEn+BM+D18GbkISH4VE4v5SGwTjzlPxX4fFfU0a8Og==</latexit><latexit sha1_base64="VTIiy0b6DlHmPd9FAKlM8oFfRz4=">AAACfXicbVFNj9MwEHXC12K+Chy5GFpWCFZVkssCEtJKXDguEqUrNdnIdiattY4T7AmoivIz+GPc+C1ccLo5wG5HsvT05s2z/UY0WjmMot9BeOPmrdt3Du7Se/cfPHw0efzkq6tbK2Eha13bM8EdaGVggQo1nDUWeCU0LMXFx6G//A7Wqdp8wW0DWcXXRpVKcvRUPvk5m9HlecI+MJpqKHFFUwFrZTpuLd/2nZQ9/ZF3cdx70SEbYDJAmoIpRhFNrVpvMDvy7LeWF4x2qSiZ6He++217KvL4PNlnQ2ezfDKN5tGu2HUQj2BKxjrNJ7/SopZtBQal5s6t4qjBzBujkhq8deug4fKCr2HloeEVuKzbxdezl54pWFlbfwyyHfvvRMcr57aV8MqK48Zd7Q3kvt6qxfJt1inTtAhGXl5UtpphzYZdsEJZkKi3HnBplX8rkxtuuUS/MepDiK9++TpYJPN38+hzMj15P6ZxQJ6RF+QVickxOSGfyClZEEn+BM+D18GbkISH4VE4v5SGwTjzlPxX4fFfU0a8Og==</latexit><latexit sha1_base64="VTIiy0b6DlHmPd9FAKlM8oFfRz4=">AAACfXicbVFNj9MwEHXC12K+Chy5GFpWCFZVkssCEtJKXDguEqUrNdnIdiattY4T7AmoivIz+GPc+C1ccLo5wG5HsvT05s2z/UY0WjmMot9BeOPmrdt3Du7Se/cfPHw0efzkq6tbK2Eha13bM8EdaGVggQo1nDUWeCU0LMXFx6G//A7Wqdp8wW0DWcXXRpVKcvRUPvk5m9HlecI+MJpqKHFFUwFrZTpuLd/2nZQ9/ZF3cdx70SEbYDJAmoIpRhFNrVpvMDvy7LeWF4x2qSiZ6He++217KvL4PNlnQ2ezfDKN5tGu2HUQj2BKxjrNJ7/SopZtBQal5s6t4qjBzBujkhq8deug4fKCr2HloeEVuKzbxdezl54pWFlbfwyyHfvvRMcr57aV8MqK48Zd7Q3kvt6qxfJt1inTtAhGXl5UtpphzYZdsEJZkKi3HnBplX8rkxtuuUS/MepDiK9++TpYJPN38+hzMj15P6ZxQJ6RF+QVickxOSGfyClZEEn+BM+D18GbkISH4VE4v5SGwTjzlPxX4fFfU0a8Og==</latexit>

zs = W2A2+b2T

a3=yn=I( Z ?> 0) p.ee#svres NEWFEATVRS

( Inputs ) ( HIDDEN )

Example: Can we learn XOR? x1 0 1 0 1x2 0 0 1 1

(x1 OR x2) AND (x1 NAND x2) 0 1 1 0<latexit sha1_base64="2EOcTsTdKqsK/cnJtMgylgc25pQ=">AAACoXicbVFNT9tAEF0bCtRtIdBjD10Rt6KXyM6FjxOorYR6CAGRghRH0XozSVas19buGBG5/mH9G73137BOjCDASKN9eu/N7O5MnElhMAj+O+7K6pu19Y233rv3Hza3Gts7v02aaw49nspUX8fMgBQKeihQwnWmgSWxhKv45nulX92CNiJVlzjLYJCwiRJjwRlaatj46/teFMNEqIJpzWZlwf9wG6V3NwzpVxrYfDyjyIum1VXUyu2afrAsyXtVeYRwh4gFPbugJbUV3x6pk84Pyy3bOgtybnzat8JVb1Cj+pWe7w8bzaAVzIO+BGENmqSO7rDxLxqlPE9AIZfMmH4YZDiwDVFwCaUX5QYyxm/YBPoWKpaAGRTzEZf0i2VGdJxqmwrpnH1aUbDEmFkSW2fCcGqeaxX5mtbPcXwwKITKcgTFFxeNc0kxpdW+6Eho4ChnFjCuhX0r5VOmGUe7Vc8OIXz+5Zeg124dtoLzdvP4qJ7GBvlEdskeCck+OSanpEt6hDufnZ9OxzlzffeX23UvFlbXqWs+kqVw+/dfgr63</latexit><latexit sha1_base64="2EOcTsTdKqsK/cnJtMgylgc25pQ=">AAACoXicbVFNT9tAEF0bCtRtIdBjD10Rt6KXyM6FjxOorYR6CAGRghRH0XozSVas19buGBG5/mH9G73137BOjCDASKN9eu/N7O5MnElhMAj+O+7K6pu19Y233rv3Hza3Gts7v02aaw49nspUX8fMgBQKeihQwnWmgSWxhKv45nulX92CNiJVlzjLYJCwiRJjwRlaatj46/teFMNEqIJpzWZlwf9wG6V3NwzpVxrYfDyjyIum1VXUyu2afrAsyXtVeYRwh4gFPbugJbUV3x6pk84Pyy3bOgtybnzat8JVb1Cj+pWe7w8bzaAVzIO+BGENmqSO7rDxLxqlPE9AIZfMmH4YZDiwDVFwCaUX5QYyxm/YBPoWKpaAGRTzEZf0i2VGdJxqmwrpnH1aUbDEmFkSW2fCcGqeaxX5mtbPcXwwKITKcgTFFxeNc0kxpdW+6Eho4ChnFjCuhX0r5VOmGUe7Vc8OIXz+5Zeg124dtoLzdvP4qJ7GBvlEdskeCck+OSanpEt6hDufnZ9OxzlzffeX23UvFlbXqWs+kqVw+/dfgr63</latexit><latexit sha1_base64="2EOcTsTdKqsK/cnJtMgylgc25pQ=">AAACoXicbVFNT9tAEF0bCtRtIdBjD10Rt6KXyM6FjxOorYR6CAGRghRH0XozSVas19buGBG5/mH9G73137BOjCDASKN9eu/N7O5MnElhMAj+O+7K6pu19Y233rv3Hza3Gts7v02aaw49nspUX8fMgBQKeihQwnWmgSWxhKv45nulX92CNiJVlzjLYJCwiRJjwRlaatj46/teFMNEqIJpzWZlwf9wG6V3NwzpVxrYfDyjyIum1VXUyu2afrAsyXtVeYRwh4gFPbugJbUV3x6pk84Pyy3bOgtybnzat8JVb1Cj+pWe7w8bzaAVzIO+BGENmqSO7rDxLxqlPE9AIZfMmH4YZDiwDVFwCaUX5QYyxm/YBPoWKpaAGRTzEZf0i2VGdJxqmwrpnH1aUbDEmFkSW2fCcGqeaxX5mtbPcXwwKITKcgTFFxeNc0kxpdW+6Eho4ChnFjCuhX0r5VOmGUe7Vc8OIXz+5Zeg124dtoLzdvP4qJ7GBvlEdskeCck+OSanpEt6hDufnZ9OxzlzffeX23UvFlbXqWs+kqVw+/dfgr63</latexit>

\ µaBor : T 1 11

T°3=Wl ,bl

ORnano : [t.IT/l.5/

/ And :[ 1 1 ) [ -1 ]

NAND W2 b2

Example: XOR Check

x1 0 1 0 1x2 0 0 1 1

�1 �1

�, b1 =

<latexit sha1_base64="dtFRWrcBCPwkdLpADAsQUDkV7Fo=">AAACfXicbVFNb9QwEHXCVzFfCxx7MexSIWijuBJaOFSqxIVjkVi20iasHGeya9Vxgj1BWkX5GfwxbvyWXupsc4B2n2Tp6c3Ms/0mq7VyGMd/g/DO3Xv3H+w9pI8eP3n6bPT8xXdXNVbCTFa6sueZcKCVgRkq1HBeWxBlpmGeXXzu6/NfYJ2qzDfc1JCWYmVUoaRALy1HvycTOv/B2QmjiYYCFzTJYKVMK6wVm661tqOcHTDOkoQe9eyI0wRMPjTQxKrVGtNDP/+zETmjbZIVLOu2nrstOxr3bjz6wHZZ0clkORrHUbwFu034QMZkwNly9CfJK9mUYFBq4dyCxzWm3hiV1OCtGwe1kBdiBQtPjSjBpe02vo698UrOisr6Y5Bt1X8nWlE6tykz31kKXLubtV7cVVs0WHxMW2XqBsHI64uKRjOsWL8LlisLEvXGEyGt8m9lci2skOg3Rn0I/OaXb5PZcfQpir8ej0+nQxp7ZJ+8Jm8JJ1NySr6QMzIjklwGr4J3wfuQhAfhYRhdt4bBMPOS/IdwegVA8rlr</latexit><latexit sha1_base64="dtFRWrcBCPwkdLpADAsQUDkV7Fo=">AAACfXicbVFNb9QwEHXCVzFfCxx7MexSIWijuBJaOFSqxIVjkVi20iasHGeya9Vxgj1BWkX5GfwxbvyWXupsc4B2n2Tp6c3Ms/0mq7VyGMd/g/DO3Xv3H+w9pI8eP3n6bPT8xXdXNVbCTFa6sueZcKCVgRkq1HBeWxBlpmGeXXzu6/NfYJ2qzDfc1JCWYmVUoaRALy1HvycTOv/B2QmjiYYCFzTJYKVMK6wVm661tqOcHTDOkoQe9eyI0wRMPjTQxKrVGtNDP/+zETmjbZIVLOu2nrstOxr3bjz6wHZZ0clkORrHUbwFu034QMZkwNly9CfJK9mUYFBq4dyCxzWm3hiV1OCtGwe1kBdiBQtPjSjBpe02vo698UrOisr6Y5Bt1X8nWlE6tykz31kKXLubtV7cVVs0WHxMW2XqBsHI64uKRjOsWL8LlisLEvXGEyGt8m9lci2skOg3Rn0I/OaXb5PZcfQpir8ej0+nQxp7ZJ+8Jm8JJ1NySr6QMzIjklwGr4J3wfuQhAfhYRhdt4bBMPOS/IdwegVA8rlr</latexit><latexit sha1_base64="dtFRWrcBCPwkdLpADAsQUDkV7Fo=">AAACfXicbVFNb9QwEHXCVzFfCxx7MexSIWijuBJaOFSqxIVjkVi20iasHGeya9Vxgj1BWkX5GfwxbvyWXupsc4B2n2Tp6c3Ms/0mq7VyGMd/g/DO3Xv3H+w9pI8eP3n6bPT8xXdXNVbCTFa6sueZcKCVgRkq1HBeWxBlpmGeXXzu6/NfYJ2qzDfc1JCWYmVUoaRALy1HvycTOv/B2QmjiYYCFzTJYKVMK6wVm661tqOcHTDOkoQe9eyI0wRMPjTQxKrVGtNDP/+zETmjbZIVLOu2nrstOxr3bjz6wHZZ0clkORrHUbwFu034QMZkwNly9CfJK9mUYFBq4dyCxzWm3hiV1OCtGwe1kBdiBQtPjSjBpe02vo698UrOisr6Y5Bt1X8nWlE6tykz31kKXLubtV7cVVs0WHxMW2XqBsHI64uKRjOsWL8LlisLEvXGEyGt8m9lci2skOg3Rn0I/OaXb5PZcfQpir8ej0+nQxp7ZJ+8Jm8JJ1NySr6QMzIjklwGr4J3wfuQhAfhYRhdt4bBMPOS/IdwegVA8rlr</latexit>

W 2 =⇥1 1

⇤, b2 =

⇥�1

⇤<latexit sha1_base64="NxNR37Ppw14yi1R4phhQWUATR6g=">AAACbHicbVFNb9NAEF0b6McWigvqoVSVViRUHCCycykckCJx4VgkQirFJlqvx8mq67W7O64UWb7xC7nxF7jwF9ikPpQ0I6309N7Mm923aaWkxTD87fmPHj/Z2d3bpwdPnx0+D45efLdlbQSMRalKc5VyC0pqGKNEBVeVAV6kCibp9eeVPrkFY2Wpv+GygqTgcy1zKTg6ahb87Pfp5MeQfWI0VpDjlMYpzKVuuDF82TZCtDRi5yxyOuiso2ls5HyByTvH3tQ8Y7SJ05yl7dppu1FL30fbPGi/Pwt64SBcF3sIog70SFeXs+BXnJWiLkCjUNzaaRRWmDhjlEKBs64tVFxc8zlMHdS8AJs067Ra9sYxGctL445GtmbvTzS8sHZZpK6z4Liwm9qK3KZNa8w/JI3UVY2gxd2ivFYMS7aKnmXSgEC1dIALI91dmVhwwwW6D6IuhGjzyQ/BeDj4OAi/Dnujiy6NPXJKXpO3JCIXZES+kEsyJoL88QLvxHvl/fWP/VP/7K7V97qZl+S/8s//ATYBtkk=</latexit><latexit sha1_base64="NxNR37Ppw14yi1R4phhQWUATR6g=">AAACbHicbVFNb9NAEF0b6McWigvqoVSVViRUHCCycykckCJx4VgkQirFJlqvx8mq67W7O64UWb7xC7nxF7jwF9ikPpQ0I6309N7Mm923aaWkxTD87fmPHj/Z2d3bpwdPnx0+D45efLdlbQSMRalKc5VyC0pqGKNEBVeVAV6kCibp9eeVPrkFY2Wpv+GygqTgcy1zKTg6ahb87Pfp5MeQfWI0VpDjlMYpzKVuuDF82TZCtDRi5yxyOuiso2ls5HyByTvH3tQ8Y7SJ05yl7dppu1FL30fbPGi/Pwt64SBcF3sIog70SFeXs+BXnJWiLkCjUNzaaRRWmDhjlEKBs64tVFxc8zlMHdS8AJs067Ra9sYxGctL445GtmbvTzS8sHZZpK6z4Liwm9qK3KZNa8w/JI3UVY2gxd2ivFYMS7aKnmXSgEC1dIALI91dmVhwwwW6D6IuhGjzyQ/BeDj4OAi/Dnujiy6NPXJKXpO3JCIXZES+kEsyJoL88QLvxHvl/fWP/VP/7K7V97qZl+S/8s//ATYBtkk=</latexit><latexit sha1_base64="NxNR37Ppw14yi1R4phhQWUATR6g=">AAACbHicbVFNb9NAEF0b6McWigvqoVSVViRUHCCycykckCJx4VgkQirFJlqvx8mq67W7O64UWb7xC7nxF7jwF9ikPpQ0I6309N7Mm923aaWkxTD87fmPHj/Z2d3bpwdPnx0+D45efLdlbQSMRalKc5VyC0pqGKNEBVeVAV6kCibp9eeVPrkFY2Wpv+GygqTgcy1zKTg6ahb87Pfp5MeQfWI0VpDjlMYpzKVuuDF82TZCtDRi5yxyOuiso2ls5HyByTvH3tQ8Y7SJ05yl7dppu1FL30fbPGi/Pwt64SBcF3sIog70SFeXs+BXnJWiLkCjUNzaaRRWmDhjlEKBs64tVFxc8zlMHdS8AJs067Ra9sYxGctL445GtmbvTzS8sHZZpK6z4Liwm9qK3KZNa8w/JI3UVY2gxd2ivFYMS7aKnmXSgEC1dIALI91dmVhwwwW6D6IuhGjzyQ/BeDj4OAi/Dnujiy6NPXJKXpO3JCIXZES+kEsyJoL88QLvxHvl/fWP/VP/7K7V97qZl+S/8s//ATYBtkk=</latexit>

z2 = W 1x1+b1 =

�1 �1

� �+

�, a2 = I(z2 > 0) =

<latexit sha1_base64="1OIxmbCc/a4EtkhLEvGxQfJ4e6A=">AAADm3iclVJbb9MwFHYTLsPcOnhCSOiIFjQ0qOJKaPAwmAQPG+JhSIROatrKcZzWWuIE20ErUf4UP4U3/g1OGomxtQ87kq3P5/Kd408nzBOhjef96Tjutes3bm7dwrfv3L13v7v94JvOCsW4z7IkUych1TwRkvtGmISf5IrTNEz4KDz9UMdHP7jSIpNfzTLnk5TOpYgFo8a6ZtudX/0+LoMwhp/VdAj7MJqS5nlWTQnsQoPDGu8DDhIemzEOQj4XsqRK0WVVKlVhAs+BQBDgVzWyl83lMmpTcKDEfGEmG+orDEG+oNJkaUkqS3L+uY4H7FwbuTxLQAav19bhjX+48gwvLdP3gkYr7ehKu8DwM6PS8qja+SfpO/BebBbvyo1xvz/r9ryB1xhcBqQFPdTa8az7O4gyVqRcGpZQrcfEy83EEhvBEm6pC81zyk7pnI8tlDTlelI2y1XBM+uJIM6UPdJA4z1fUdJU62Ua2syUmoW+GKud62LjwsRvJqWQeWG4ZKtGcZGAyaDeVIiE4swkSwsoU8LOCmxBFWXG7jO2IpCLX74M/OHg7cD7Muwd7LVqbKHH6CnaQQTtoQN0iI6Rj5jzyHnvHDpH7hP3o/vJ/bxKdTptzUP0n7n+X7G5HyE=</latexit><latexit sha1_base64="1OIxmbCc/a4EtkhLEvGxQfJ4e6A=">AAADm3iclVJbb9MwFHYTLsPcOnhCSOiIFjQ0qOJKaPAwmAQPG+JhSIROatrKcZzWWuIE20ErUf4UP4U3/g1OGomxtQ87kq3P5/Kd408nzBOhjef96Tjutes3bm7dwrfv3L13v7v94JvOCsW4z7IkUych1TwRkvtGmISf5IrTNEz4KDz9UMdHP7jSIpNfzTLnk5TOpYgFo8a6ZtudX/0+LoMwhp/VdAj7MJqS5nlWTQnsQoPDGu8DDhIemzEOQj4XsqRK0WVVKlVhAs+BQBDgVzWyl83lMmpTcKDEfGEmG+orDEG+oNJkaUkqS3L+uY4H7FwbuTxLQAav19bhjX+48gwvLdP3gkYr7ehKu8DwM6PS8qja+SfpO/BebBbvyo1xvz/r9ryB1xhcBqQFPdTa8az7O4gyVqRcGpZQrcfEy83EEhvBEm6pC81zyk7pnI8tlDTlelI2y1XBM+uJIM6UPdJA4z1fUdJU62Ua2syUmoW+GKud62LjwsRvJqWQeWG4ZKtGcZGAyaDeVIiE4swkSwsoU8LOCmxBFWXG7jO2IpCLX74M/OHg7cD7Muwd7LVqbKHH6CnaQQTtoQN0iI6Rj5jzyHnvHDpH7hP3o/vJ/bxKdTptzUP0n7n+X7G5HyE=</latexit><latexit sha1_base64="1OIxmbCc/a4EtkhLEvGxQfJ4e6A=">AAADm3iclVJbb9MwFHYTLsPcOnhCSOiIFjQ0qOJKaPAwmAQPG+JhSIROatrKcZzWWuIE20ErUf4UP4U3/g1OGomxtQ87kq3P5/Kd408nzBOhjef96Tjutes3bm7dwrfv3L13v7v94JvOCsW4z7IkUych1TwRkvtGmISf5IrTNEz4KDz9UMdHP7jSIpNfzTLnk5TOpYgFo8a6ZtudX/0+LoMwhp/VdAj7MJqS5nlWTQnsQoPDGu8DDhIemzEOQj4XsqRK0WVVKlVhAs+BQBDgVzWyl83lMmpTcKDEfGEmG+orDEG+oNJkaUkqS3L+uY4H7FwbuTxLQAav19bhjX+48gwvLdP3gkYr7ehKu8DwM6PS8qja+SfpO/BebBbvyo1xvz/r9ryB1xhcBqQFPdTa8az7O4gyVqRcGpZQrcfEy83EEhvBEm6pC81zyk7pnI8tlDTlelI2y1XBM+uJIM6UPdJA4z1fUdJU62Ua2syUmoW+GKud62LjwsRvJqWQeWG4ZKtGcZGAyaDeVIiE4swkSwsoU8LOCmxBFWXG7jO2IpCLX74M/OHg7cD7Muwd7LVqbKHH6CnaQQTtoQN0iI6Rj5jzyHnvHDpH7hP3o/vJ/bxKdTptzUP0n7n+X7G5HyE=</latexit>

z3 = W 2a2 + b2 =⇥1 1

⇤ �+⇥�1

⇥ ⇤, y = I(z3 > 0) =

⇥ ⇤

<latexit sha1_base64="4x4wKmONk0RqxcJrhKAYXwr6uSw=">AAADanichVJNb9NAEN3YfJQFSkoPgHpZEYOKgMgOh8KhqBIXuBWJkEpxEq3X63jV9drsjhHG8oG/yI1/wIX/wMa1BG0TdaSV3rzZN1+aqJDCgO//6jnutes3bm7dwrfv3N2+19+5/9nkpWZ8zHKZ65OIGi6F4mMQIPlJoTnNIskn0em7VXzylWsjcvUJqoLPMrpUIhGMgqUWO70fnofrMErI92b+ihySyXzUurSZj8hz0uJohQ8JDiVPYIrDiC+FqqnWtGpqrRsckKcksHGu4o7GoRbLFGYbNA0mYZFSBXlWB00YnnPX5SG2l425XgZrJXhjy82V9V5Y6ZeSxjhMKdRVY8cPgX8DndUfmv1/C3tL/GebV3N1Hex5i/7AH/qtkcsg6MAAdXa86P8M45yVGVfAJDVmGvgFzGxiEExym7o0vKDslC751EJFM25mdXsrDXlimZgkubZPAWnZ/xU1zYypssj+zCik5mJsRa6LTUtIXs9qoYoSuGJnhZJSEsjJ6vBILDRnICsLKNPC9kpYSjVlYM8T2yUEF0e+DMaj4Zuh/3E0ODrotrGF9tBjtI8CdICO0Ht0jMaI9X47284D56Hzx911H7l7Z1+dXqfZRefM9f4CM6wQ+A==</latexit><latexit sha1_base64="4x4wKmONk0RqxcJrhKAYXwr6uSw=">AAADanichVJNb9NAEN3YfJQFSkoPgHpZEYOKgMgOh8KhqBIXuBWJkEpxEq3X63jV9drsjhHG8oG/yI1/wIX/wMa1BG0TdaSV3rzZN1+aqJDCgO//6jnutes3bm7dwrfv3N2+19+5/9nkpWZ8zHKZ65OIGi6F4mMQIPlJoTnNIskn0em7VXzylWsjcvUJqoLPMrpUIhGMgqUWO70fnofrMErI92b+ihySyXzUurSZj8hz0uJohQ8JDiVPYIrDiC+FqqnWtGpqrRsckKcksHGu4o7GoRbLFGYbNA0mYZFSBXlWB00YnnPX5SG2l425XgZrJXhjy82V9V5Y6ZeSxjhMKdRVY8cPgX8DndUfmv1/C3tL/GebV3N1Hex5i/7AH/qtkcsg6MAAdXa86P8M45yVGVfAJDVmGvgFzGxiEExym7o0vKDslC751EJFM25mdXsrDXlimZgkubZPAWnZ/xU1zYypssj+zCik5mJsRa6LTUtIXs9qoYoSuGJnhZJSEsjJ6vBILDRnICsLKNPC9kpYSjVlYM8T2yUEF0e+DMaj4Zuh/3E0ODrotrGF9tBjtI8CdICO0Ht0jMaI9X47284D56Hzx911H7l7Z1+dXqfZRefM9f4CM6wQ+A==</latexit><latexit sha1_base64="4x4wKmONk0RqxcJrhKAYXwr6uSw=">AAADanichVJNb9NAEN3YfJQFSkoPgHpZEYOKgMgOh8KhqBIXuBWJkEpxEq3X63jV9drsjhHG8oG/yI1/wIX/wMa1BG0TdaSV3rzZN1+aqJDCgO//6jnutes3bm7dwrfv3N2+19+5/9nkpWZ8zHKZ65OIGi6F4mMQIPlJoTnNIskn0em7VXzylWsjcvUJqoLPMrpUIhGMgqUWO70fnofrMErI92b+ihySyXzUurSZj8hz0uJohQ8JDiVPYIrDiC+FqqnWtGpqrRsckKcksHGu4o7GoRbLFGYbNA0mYZFSBXlWB00YnnPX5SG2l425XgZrJXhjy82V9V5Y6ZeSxjhMKdRVY8cPgX8DndUfmv1/C3tL/GebV3N1Hex5i/7AH/qtkcsg6MAAdXa86P8M45yVGVfAJDVmGvgFzGxiEExym7o0vKDslC751EJFM25mdXsrDXlimZgkubZPAWnZ/xU1zYypssj+zCik5mJsRa6LTUtIXs9qoYoSuGJnhZJSEsjJ6vBILDRnICsLKNPC9kpYSjVlYM8T2yUEF0e+DMaj4Zuh/3E0ODrotrGF9tBjtI8CdICO0Ht0jMaI9X47284D56Hzx911H7l7Z1+dXqfZRefM9f4CM6wQ+A==</latexit>

O112.2 > o )

0 1.5 g

Example: XOR Check

x1 0 1 0 1x2 0 0 1 1

�1 �1

�, b1 =

W 2 =⇥1 1

⇤, b2 =

⇥�1

z2 = W 1x1+b1 =

�1 �1

� �+

�, a2 = I(z2 > 0) =

z3 = W 2a2 + b2 =⇥1 1

⇤ �+⇥�1

⇥ ⇤, y = I(z3 > 0) =

⇥ ⇤

1 c (0 05 1

Example: XOR Check

x1 0 1 0 1x2 0 0 1 1

�1 �1

�, b1 =

W 2 =⇥1 1

⇤, b2 =

⇥�1

z2 = W 1x1+b1 =

�1 �1

� �+

�, a2 = I(z2 > 0) =

z3 = W 2a2 + b2 =⇥1 1

⇤ �+⇥�1

⇥ ⇤, y = I(z3 > 0) =

⇥ ⇤

Example: XOR Check

x1 0 1 0 1x2 0 0 1 1

�1 �1

�, b1 =

W 2 =⇥1 1

⇤, b2 =

⇥�1

z2 = W 1x1+b1 =

�1 �1

� �+

�, a2 = I(z2 > 0) =

z3 = W 2a2 + b2 =⇥1 1

⇤ �+⇥�1

⇥ ⇤, y = I(z3 > 0) =

⇥ ⇤

1 - .5 0

OK, so the multilayer perceptron is cool and all, but when do we get to REAL neural networks??

We’re pretty much already there.

Things we’ll explore moving forward:

o Non-Binary Features

o Better activation functions

o How to choose architectures

o How to train these things

o Regression or Classification (Answer: Yes)

Neural Networks I Wrap-Up

o Simplest network is just a regular old perceptron (a linear classifier)

o We can learn non-linear decision boundaries by chaining perceptrons

o We can represent these complicated interactions using linear algebra

Next Time: o The Feed-Forward Neural Network

neural networks part i the multilayer perceptron

Documents

neural networks and introduction to bishop (1995) : neural...

multilayer perceptron = feedforward neural network€¦ ·...

artificial neural networks (cont.) chapter 4 perceptron...

artificial neural networks · artificial neural networks:...

multilayer perceptron

15 machine learning multilayer perceptron

multilayer perceptron architecture optimization using

6.- supervised neural networks: multilayer...

using multilayer perceptron artificial neural...

artificial neural networks the brain brain vs. computers the...

on the use of neural networks to solve problems. from...

multilayer perceptron and neural networks - semantic...

comparative application of radial basis function and ......

neural networks: multilayer perceptron

implementasi metode multilayer perceptron …

supervised learning neural networks multilayer perceptron...

multilayer perceptrons lecture...

multilayer perceptron perceptron.pdf · multilayer...

a multilayer perceptron neural network based model for...

copy artificial neural - science · presents a method for...